Refactor column reader

This simplifies the column reader and yields a slight perf improvement.