Remove a peer if the pointer isn't nil