Remove items from submap with the correct index for each direction.