An optimization that doesn't seek if it isn't necessary.

This avoids an extra seek and re-buffering, but you still have to
pay for the seek+read on the index.