Fix pointer arithmetic when seeking buffers

It's possible the sample type will change, so best to not use
a fixed value.