Package: mmap-ninja
mmap.ninjaInstall with:pipinstallmmap_ninjaMicrolib docs can be foundhere.ContentsQuick exampleWhat is it?When to use it?When not to use it?How it works?API guideFAQI want to contributeQuick exampleimportnumpyasnpimportmatplotlib.imageasmpimgfromtqdmimporttqdmfrompathlibimportPathfrommmap_ninja.raggedimportRaggedMmapcoco_path=Path('<PATH TO IMAGE DATASET>')# Once per project, convert the images to a memory mapRaggedMmap.from_generator(# Directory in which the memory map will be persistedout_dir='images_mmap',# Something that yields np.ndarraysample_generator=map(mpimg.imread,coco_path.iterdir()),# Maximum number of samples to keep in memory before flushing to diskbatch_size=1024,# Show/hide progress barverbose=True)# Open the memory mapimages_mmap=RaggedMmap('images_mmap')# This iteration takes 0.2s on COCO val 2017# This iteration takes 35s without memory-mappingforiintqdm(range(len(images_mmap))):img:np.ndarray=images_mmap[i]Back to ContentsWhat is it?Accelerate the iteration over your machine learning dataset by up to20 times!mmap_ninjais a library for storing your datasets in memory-mapped files,
which leads to a dramatic speedup in the training time.The only dependencies arenumpyandtqdm.You can usemmap_ninjawith any training framework (such asTensorflow,PyTorch,MxNet), etc.,
as it stores your dataset as a memory-mapped numpy array.Amemory mapped fileis a file that is physically present on disk in a way that the correlation between the file
and the memory space permits applications to treat the mapped portions as if it were primary memory, allowing very fast
I/O!When working on a machine learning project, one of the most time-consuming parts is the model's training.
However, a large portion of the training time actually consists of just iterating over your dataset and filesystem I/O!This library,mmap_ninjaprovides high-level, easy to use, well tested API for using memory maps for your
datasets, reducing the time needed for training.Memory maps would usually take a little more disk space though, so if you are willing to trade some disk space
for fast filesystem to memory I/O, this is your library!Back to ContentsWhen do I use it?Use it whenever you want to store a sequence ofnp.ndarrays (of varying shapes) that you are going to
read from at random positions very often.mmap_ninjacan work with any type of data that can be stored as anp.ndarray, as the
memory map is initialized with a generator that yields samples.In the table below, you can see concrete examples, but beware that those are just examples,mmap_ninjahas no specific logic to handle images or videos or something like that.It just storesnp.ndarrayand it is up to you to decide what this array represents.Use caseNotebookBenchmarkClass/ModuleImageCOCO 2017from mmap_ninja.ragged import RaggedMmapText20 newsgroupsfrom mmap_ninja.string import StringsMmapVideoComing soon!from mmap_ninja import numpy as RaggedMmapBack to ContentsMemory mapping images with different shapesYou can create a newRaggedMmmapfrom one of its class methods:RaggedMmmap.from_lists,RaggedMmap.from_generator.Create a memory map from generator, flushing to disk every 1024 images (so that you don't have to keep it all in memory at once):importmatplotlib.pyplotaspltfrommmap_ninja.raggedimportRaggedMmapfrompathlibimportPathcoco_path=Path('<PATH TO IMAGE DATASET>')val_images=RaggedMmap.from_generator(out_dir='val_images',sample_generator=map(plt.imread,coco_path.iterdir()),batch_size=1024,verbose=True)Once created, you can open the map by simply supplying the path to the memory map:frommmap_ninja.raggedimportRaggedMmapval_images=RaggedMmap('val_images')print(val_images[3])# Prints the ndarray image, e.g. with shape (387, 640, 3)You can also extend an already existing memory map easily by using the.extendmethod.In the table show the time needed for initial loading, one iteration over the COCO validation 2017 dataset,
the memory usage of every method and the disk usage.Initial load (s)Time for iteration (s)Memory usage (GB)Disk usage (GB)in_memory1.3560770.0004033.818741 GB3.819034 GBragged_mmap0.0020540.0578580.001144 GB3.819114 GBimread_from_disk0.00000022.2083850.001144 GB0.758753 GBYou can see that once created, theRaggedMmapis383 timesfaster for iterating over the
dataset.
It does require 4 times more disk space though, so if you are willing to trade 4 times more disk space
for383 timesspeedup (and less memory usage), you definitely should use theRaggedMmap!This makes theRaggedMmapa fantastic choice for your computer vision, image-based machine learning datasets!Memory mapping text documentsYou can create a newStringsMmmapfrom one of its class methods:StringsMmmap.from_strings,StringsMmap.from_generator.
Once it's created, you can open it by just supplying the path to the memory map.An example of creating a memory map for thesklearn's 20newsgroups dataset:frommmap_ninja.stringimportStringsMmapfromsklearn.datasetsimportfetch_20newsgroupsdata=fetch_20newsgroups()memmap=StringsMmap.from_strings('20newsgroups',data['data'],verbose=True)Opening an already existingStringsMmmap:frommmap_ninja.stringimportStringsMmaptexts=StringsMmap('20newsgroups')print(texts[123])# Prints the 123-th textYou can also extend an already existing memory map easily by using the.extendmethod.In the table show the time needed for initial loading, 100 iterations over thesklearn's 20newsgroups dataset,
the memory usage of every method and the disk usage.Initial load (s)Time for iteration (s)Memory usage (GB)Disk usage (GB)in_memory0.1746260.0689950.09 MB45 MBragged_mmap0.0037012.0526590.07 MB22 MBread_from_disk0.00000013.9967380.07 MB45 MBYou can see that once created, theStringsMmapis nearly7 timesfaster compared to reading.txtfiles
from disk one by one.
Moreover, it takes2 timesless disk space (this is true only forStringsMmap, in general for other types the memory map
would take more disk space).
This makes theStringsMmmapa fantastic choice for your NLP, text-based machine learning datasets!When not to use it?Very frequently,mmap_ninjatakes more disk space than traditional approaches.
For example, for jpeg images, it takes 4 times more disk space.For this reason, do not usemmap_ninjain the following cases:You are low on disk spaceYou want to send the data over a network - use a compressed format insteadThere are other cases in whichmmap_ninjais not a good choice:When you want to concurrently append to the memory map (use a queue like RabbitMQ and append from a subscriber instead)If you want to frequently delete samples from the memory map - this will require a new copy of the whole objectand so on.Back to ContentsHow it worksComing soonBack to ContentsAPI guideRead the docsBack to ContentsFAQQ: Can I use it with Tensorflow/TF?
A: Of course. You can use it with any framework that can work with numpy arrays. Here's anend-to-end exampleBack to ContentsI want to contributeComing soon!Back to Contents