Remove friends from the map when the server stops reporting them