Clean up peer tracking some

Now it's a map that tracks by peer_id, so we stop getting duplicate
peers in our list. We also try to reduce sending peers links to peers
that are themselves.