Allow setting a buffers host mem