Remove pointless code.

The map never contained the empty arrays at all.