Handle binary buffers by doing the encoding ourselves