Don't force MemoryBlocks on classes that don't care.

Only MappedByteBuffer and DirectByteBuffer need to know
about this abstraction.

Change-Id: I293284c8b8a73ac01463337dc8a1e22d16cd378d