Minor refactor of AddPeer/RemovePeer.

@xiangli-cmu Simplifying the `AddPeer()` and `RemovePeer()` functions. There's a bunch of red/green in the diff but the function is pretty straightforward if you look at the file itself.