buffer: make nearest sampler fast path catch more cases

The logic to perform conversions if the buffers format is not same as desired
are already in place.