Move the seek optimization into the CSV reader.
We don't need an immutable view into the buffer.