Implement function to remove all fetchers from a peer.

This function is required if a peer is destroyed.