Refactor determining the remote IP
Fix returning more than the requested number of peers, and don't return
a user's peers to themselves