Package: daf
DAF 0.1.0 - Data in Axes in FormatsThe daf package provides a uniform generic interface for accessing 1D and 2D data arranged along some set of axes. This
is a much-needed generalization of theAnnDatafunctionality. The key features
are:Support both in-memory and persistent data storage of “any” format (given an adapter implementation).Out of the box, allow storing the data in memory, inAnnDataobjects (e.g., usingh5adfiles), directly insideH5FSfiles (usingh5py), or as a collection of simple
memory-mapped files in a directory.The data model is based on (1) some axes with named entries, (2) 1-D data indexed by a single axis, (3) 2-D
data indexed by a pair of axes, and also (4) 0-D data items (anything not tied to some axis).There is explicit control over 2D data layout (row or column major), and support for both dense and sparse matrices,
both of which are crucial for performance.Allows accessing the data as either plain numpy arrays, scipy csr/csc sparse matrices, or as pandas series/frames.Is a pure Python package so should run “anywhere” (as long as its dependencies ara available, most notablypandas,numpy,scipyandanndata).MotivationThedafpackage was created to overcome the limitations of theAnnDatapackage. LikeAnnData,dafwas
created to support code analyzing single-cell RNA sequencing data (“scRNA-seq”), but should be useful for other problem
domains.The main issue we had withAnnDatais that it restricts all the stored data to be described by two axes
(“observations” and “variables”). E.g., in single-cell data, every cell would be an “observation” and every gene would
be a “variable”. As a secondary annoyance,AnnDatagives one special “default” per-observation-per-variable data
layer the uninformative nameX, and only allows meaningful names for any additional data layers, which use a
completely different access method.This works pretty well until one starts to perform higher level operations:(Almost) everyone groups cells into “type” clusters. This requires storing per-cluster data (e.g. its name and its
color).Since “type” clusters (hopefully) correspond to biological states, which map to gene expression levels, this requires
also storing per-cluster-per-gene data.Often such clusters form at least a two-level hierarchy, so we need per-sub-cluster data and per-sub-cluster-per-gene
data as well.We’d like to keep both the UMIs count and the normalized gene fractions (and possibly their log2 values for quick
fold factor computations).Sure, it is possible to use a set ofAnnDataobjects, each with its own distinct set of “observations” (for cell,
clusters, and sub-clusters). We can reduce confusion about whatXis in each data set by always using it for UMIs,
though that may not make much sense for some of the data sets. We’ll also need to replicate simple per-gene data across
the data sets, and keep it in sync, or just store each such data in one of the data sets, and remember in which.In short, we’d end up writing some problem-specific code to manage the multipleAnnDataobjects for us, which kind
of defeats the purpose of usingAnnDatain the first place. Instead, we have chosen to createdaf, which is a
general-purpose solution that embraces the existence of arbitrary multiple axes in the same data set, and enforces no
opaque default names, to make it easy for us store explicitly named data per-whatever-we-damn-please all in a single
place.When it comes to storage,dafmakes it as easy as possible to write adapters to allow storing the data in your
favorite format; in particular,dafsupportsAnnData(or a set ofAnnData) as a storage format, which in
turn allows the data to be stored in various disk file formats such ash5ad. Sinceh5adis restricted by theAnnDatadata model, we also allow directly storingdafdata in anh5fsfile in a more efficient way (that
is, inh5dffiles).That said, we find that, for our use cases, the use of complex single-file formats such ash5fsto be sub-optimal.
In effect they function as a file system, but offer only some of its functionality. For example, you need special APIs
to list the content of the data, copy or delete just parts of it, find out which parts have been changed when, and most
implementations do not support memory-mapping the data, which causes a large performance hit for large data sets.Therefore, as an option,dafalso supports a simple “files” storage format where every “annotation” is a separate
file (in a trivial format) inside a single directory. This allows for efficient memory-mapping of files, using standard
file system tools to list, copy and/or delete data, and using tools likemaketo automate incremental computations.
The main downside is that to send a data set across the network, one has to first collect it into atarorziparchive. This may actually be more efficient as this allows compressing the data for more efficient transmission or
archiving. Besides, due to the limitations ofAnnData, one has to send multiple files for a complete data set
anyway.Finally,dafalso provides a simple in-memory storage format, which is a lightweight container optimized fordafdata access (similar to an in-memoryAnnDataobject).It is possible to create views ofdafdata (slicing, renaming and hiding axes and/or specific annotations), and to
copydafdata from one data set to another (e.g., from a view of an in-memory data set into anAnnDatadata set
for exporting it into anh5adfile).Finally, thedafpackage also provides some convenience functionality out of the box, such as caching derived data
(different layouts of the same data, sums or other computations along axes, etc.). It is possible to disable such
caching (recomputing the derived data every time it is requested), or direct the cache to an arbitrary storage format
(e.g., keep the derived data cache in-memory on top of a disk-based data set, which is a common usage pattern).It is assumed thatdafdata will be processed in a single machine, that is,dafdoes not try to address the
issues of a distributed cluster of servers working on a shared data set. Today’s servers (as of 2022) can get very big
(~100 cores and ~1TB of RAM is practical), which means that all/most data sets would fit comfortably in one machine (and
memory mapped files are a great help here). In addition, if using the “files” storage, it is possible to have different
servers access the samedafdirectory, each computing a different independent additional annotation (e.g., one
server searching for doublets while another is searching for gene modules), and as long as only one server writes each
new “annotation”, this should work fine (one can do even better by writing more complex code). This is another example
of how simple files make it easy to provide functionality which is very difficult to achieve using a complex single-file
format such ash5fs.The bottom line is thatdafprovides a convenient abstraction layer above any “reasonable” storage format, allowing
efficient computation and/or visualization code to naturally access and/or write the data it needs, even for
higher-level analysis pipeline, for small to “very large” (but not for “ludicrously large”) data sets.Usageimportdaf# Open an existing DAF storage in the "files" format.data=daf.DafReader(daf.FilesReader("..."))# Access arbitrary 0D data.description=data.get_item("description")# Get a 1D numpy array by axis and name.metacell_types=data.get_vector("metacell;type")# Get a Pandas series by axis and name (index is the type names).type_colors=data.get_series("type;color")# Combine these to get a Pandas series of the color of each metacell.metacell_colors=type_colors[metacell_types]# Get a 2D matrix by two axes and a name.umis_matrix=data.get_matrix("cell,gene;UMIs")ifdaf.is_dense(umis_matrix):# Umis matrix is dense (2D numpy.ndarray)....else:assertdaf.is_sparse(umis_matrix)# Umis matrix is sparse (scipy.sparse.csr_matrix)....# Get a Pandas data frame with homogeneous elements by two axes and a name.type_marker_genes=data.get_frame("gene,type;marker")# Access the mask of marker genes for a specific type as a Pandas series.t_marker_genes=type_marker_genes["T"]# Get a Pandas data frame with multiple named (columns) of different types.genes_masks=data.get_columns("gene",["forbidden","significant"])# Access the mask of significant genes in the frame as a Pandas series.significant_genes_mask=genes_masks["significant"]# Get the total sum of UMIs per cell (and cache it for future requests).cells_umis_sum=data.get_vector("cell,gene;UMIs|Sum")#: Slice to include cells with a high number of UMIs and significant genes.strong_data=data.view(axes=dict(cells=cells_umis_sum>1000,genes=significant_genes_mask))See thedocumentationfor the full API details.InstallationIn short:pip install daf. Note thatdafrequires many “heavy” dependencies, most notablynumpy,pandas,scipyandanndata, whichpipshould automatically install for you. If you are running inside acondaenvironment, you might prefer to use it to first install these dependencies, instead of havingpipinstall them fromPyPI.License (MIT)Copyright © 2022 Weizmann Institute of SciencePermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the “Software”), to deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.History0.1.0First published version (alpha).