Close sockets of peers we update or remove