Don't do the work to clear if the map is empty.