Only copy buffer to direct buffer if pooled allocator is used