Package: jagged
Efficient storage of same-type, uneven-size arraysJaggedis an ongoing amateur project exploring the storage panorama
for datasets containing (large amounts of) arrays with the same type
and number of columns, but varying number of rows. Examples of such
datasets for whichjaggedhas been used are collections of multivariate
timeseries (short animal behaviour snippets) and collections of molecules
(represented as varying length strings).Jagged aims to help analyzing data in the laptop and the cluster, in batch
or interactively, providing a very lightweight store. Jagged provides fast
retrieval of array subsets for many-GB datasets containing millions of rows.By-design constraintsFocus is on fast retrieval of arbitrary batch queries.Jagged stores are append only.There is no transaction, replication or distribution.
It is all files in your local or network disks.Not important efforts have been given yet to optimize
(although some backends work quite smoothly).At the moment, everything is simple algorithms implemented in pure python.InstallationIt should suffice to use pip:pip install jaggedJagged stores builds on top of several high quality python libraries: numpy, blosc,
bloscpack, bcolz and joblib. It also needs whatami and python-future.
Testing relies on pytest (you need to install all dependencies to test at the moment,
this will change soon).ShowcaseUsing jagged is simple. There are different implementations that provide
two basic methods:appendadds a new array to the store,getretrieves
collections of arrays identified by their insertion order in the store.importos.pathasopimportnumpyasnpfromjagged.mmap_backendimportJaggedByMemmap# A Jagged instance is all you needjagged=JaggedByMemmap(op.expanduser(path='~/jagged-example/mmap'))# You can drop here any you want to# Generate a random datasetrng=np.random.RandomState(0)max_length=2000num_arrays=100originals=[rng.randn(rng.randint(0,max_length),50)for_inrange(num_arrays))# Add these to the store (context is usually optional but recommended)withjagged:indices=map(jagged.append,originals)# What do we have in store?print('Number of arrays:%d, number of rows:%d'%(jbmm.narrays,jbmm.nrows))print('Jagged shape=%r, dtype=%r, order=%r'%(jagged.shape,jagged.dtype,jagged.order))# Check roundtriproundtripped=jagged.get(indices)print('The store has%darrays')# Jagged stores self-identified themselves (using whatami)print(jagged.what().id())# Jagged stores can be iterated in chunks# See iter# Jagged stores can be populated from other jagged stores# Some jagged stores allow to retrieve arbitrary rows as fast# as arbitrary arrays.BackendsAlthough rapidly changing,jaggedalready provides the following storage backends
that can be considered as working and stable. Other backends are planned.BackendcompchunkcolumnmmaplinlazycontJaggedByBloscXXJaggedByCarrayXXXXJaggedByH5PyXXXXXJaggedByJoblibXXJaggedByMemMapXXXXJaggedByNPYJaggedByBloscpackXJaggedByPickleXXcomp: can be compressedchunk: can be chunkedcolumn: stores columns of the array contiguously (can be easily implemented by using a store per column)mmap: can open a memmap to the datalin: can retrieve any row without the need to retrieve the wholearray it contains itlazy: the arrays are not fetched immediatly; this can mean also that they can be managedas virtual-memory by the OS (JaggedByMemMap only)cont: retrieved arrays can be forced to lie in contiguous memory segmentsBenchmarksWhat backend and parameters work best depends on whether your data is compressible or not and the
sizes of the arrays. We have a good idea of what works best for our data and are working at
providing a benchmarking framework. Find here apreview.