Remove the spatial map entry if there's nothing left at that location.