PeerGroup: add/remove listeners from active/pending peers as well as the internal list.