ENH: make memory respect mmap_mode at first call too

Simple fix to make the dump / load cycle more consistent when `mmap_mode` is not None.