Improve column map performance

Makes column iterators a bit faster