Package: dolomite-base
Save and load Bioconductor objects in PythonThedolomite-basepackage is the Python counterpart to thealabaster.baseR package
for language-agnostic reading and writing of Bioconductor objects (see theBiocPyproject).
This is a more robust and portable alternative to the typical approach of pickling Python objects to save them to disk.By separating the on-disk representation from the in-memory object structure, we can more easily adapt to changes in class definitions.
This improves robustness to Python environment updates.By using standard file formats like HDF5 and CSV, we ensure that the objects can be easily read from other languages like R and Javascript.
This improves interoperability between application ecosystems.By breaking up complex Bioconductor objects into their components, we enable modular reads and writes to the backing store.
We can easily read or update part of an object without having to consider the other parts.Thedolomite-basepackage defines the base generics to read and write the file structures along with the associated metadata.
Implementations of these methods for various Bioconductor classes can be found in the otherdolomitepackages likedolomite-rangesanddolomite-se.Quick startFirst, we'll install thedolomite-basepackage.
This package is available fromPyPIso we can use the standard installation process:pipinstalldolomite-baseThe simplest example involves saving aBiocFrameinside a staging directory.
Let's mock one up:importbiocframedf=biocframe.BiocFrame({"X":list(range(0,10)),"Y":["a","b","c","d","e","f","g","h","i","j"]})print(df)## BiocFrame with 10 rows and 2 columns##           X      Y##     <range> <list>## [0]       0      a## [1]       1      b## [2]       2      c## [3]       3      d## [4]       4      e## [5]       5      f## [6]       6      g## [7]       7      h## [8]       8      i## [9]       9      jWe save ourBiocFrameto a user-specified directory with thesave_object()function.
This function saves its input object to file according to the relevantspecification.importtempfileimportostmp=tempfile.mkdtemp()importdolomite_basepath=os.path.join(tmp,"my_df")dolomite_base.save_object(df,path)os.listdir(path)## ['basic_columns.h5', 'OBJECT']We load the contents of the directory back into a Python session by using theread_object()function.
Note that the exact Python types for theBiocFramecolumns may not be preserved by the round trip,
though the contents of the columns will be unchanged.out=dolomite_base.read_object(path)print(out)BiocFramewith10rowsand2columns##                    X            Y##     <ndarray[int32]> <StringList>## [0]                0            a## [1]                1            b## [2]                2            c## [3]                3            d## [4]                4            e## [5]                5            f## [6]                6            g## [7]                7            h## [8]                8            i## [9]                9            jCheck out theAPI referencefor more details.Supported classesThe saving/reading process can be applied to a range ofBiocPydata structures,
provided the appropriatedolomitepackage is installed.
Each package implements a saving and reading function for its associated classes,
which are automatically used fromdolomite-base'ssave_object()andread_object()functions, respectively.
(That is, there is no need to explicitlyimporta package when callingsave_object()orread_object()for its classes.)PackageObject typesPyPIdolomite-baseBiocFrame,list,dict,NamedListdolomite-matrixnumpy.ndarray,scipy.sparse.spmatrix,DelayedArraydolomite-rangesGenomicRanges,GenomicRangesListdolomite-seSummarizedExperiment,RangedSummarizedExperimentdolomite-sceSingleCellExperimentdolomite-maeMultiAssayExperimentAll of the listed packages are available from PyPI and can be installed with the usualpip installprocedure.
Alternatively, to install all packages in one go, users can install thedolomiteumbrella package.Operating on directoriesUsers can move freely rename or relocate directories and theread_object()function will still work.
For example, we can easily copy the entire directory to a new file system and everything will still be correctly referenced within the directory.
The simplest way to share objects is to justziportarthe staging directory forad hocdistribution,
though more serious applications will use storage systems like AWS S3 for easier distribution.# Mocking up an object:importbiocframedf=biocframe.BiocFrame({"X":list(range(0,10)),"Y":["a","b","c","d","e","f","g","h","i","j"]})# Saving to one location:importtempfileimportosimportdolomite_basetmp=tempfile.mkdtemp()path=os.path.join(tmp,"my_df")dolomite_base.save_object(df,path)# Reading from another location:alt_path=os.path.join(tmp,"foobar")os.rename(path,alt_path)alt_out=dolomite_base.read_object(alt_path)That said, it is unwise to manipulate the files inside the directory created bysave_object().
Reading functions will usually depend on specific file names or subdirectory structures within the directory, and fiddling with them may cause unexpected results.
Advanced users can exploit this by loading components from subdirectories if the full object is not required:# Creating a nested DF:nested=biocframe.BiocFrame({"A":df})nest_path=os.path.join(tmp,"nesting")dolomite_base.save_object(nested,nest_path)# Now reading in the nested DF:redf=dolomite_base.read_object(os.path.join(nest_path,"other_columns","0"))Validating filesEach Bioconductor class's on-disk representation is determined by the associatedtakanespecification.
For example,save_object()will save aBiocFrameaccording to thedata_framespecification.
More complex objects may be represented by multiple files, possibly including subdirectories with "child" objects.Each call tosave_object()will automatically enforce the relevant specification by validating the directory contents withdolomite-base'svalidate_object()function.
Successful validation provides some guarantees on the file structure within the directory, allowing developers to reliably implement readers in other frameworks.
Conversely, thealabastersuite applies the same validators on directories generated within an R session,
which ensures thatdolomite-baseis able to read those objects into a Python environment.Users can also callvalidate_object()themselves, if they have modified the directory after callingsave_object()and they want to check that the contents are still valid:# Mocking up an object:importbiocframedf=biocframe.BiocFrame({"X":list(range(0,10)),"Y":["a","b","c","d","e","f","g","h","i","j"]})# Saving to one location:importtempfileimportosimportdolomite_basetmp=tempfile.mkdtemp()path=os.path.join(tmp,"my_df")dolomite_base.save_object(df,path)# So far so good...dolomite_base.validate_object(path)# Deleting the file to make it invalid:os.remove(os.path.join(path,"basic_columns.h5"))dolomite_base.validate_object(path)## Traceback (most recent call last):## etc...Extending to new classesThedolomiteframework is easily extended to new classes by:Writing a method forsave_object().
This should accept an instance of the object and a path to a directory, and save the contents of the object inside the directory.
It should also produce anOBJECTfile that specifies the type of the object, e.g.,data_frame,hdf5_sparse_matrix.Writing a function forread_object()and registering it in theread_object_registry.
This should accept a path to a directory and read its contents to reconstruct the object.
The registered type should be the same as that used in theOBJECTfile.Writing a function forvalidate_object()and registering it in thevalidate_object_registry.
This should accept a path to a directory and read its contents to determine if it is a valid on-disk representation.
The registered type should be the same as that used in theOBJECTfile.(optional) Devleopers can alternatively formalize the on-disk representation by adding a specification to thetakanerepository.
This aims to provide C++-based validators for each representation, allowing us to enforce consistency across multiple languages (e.g., R).
Anytakanevalidator is automatically used byvalidate_object()so no registration is required.To illustrate, let's extenddolomiteto a new custom class:classCoffee:def__init__(self,beans:str,milk:bool):self.beans=beansself.milk=milkFirst we implement the saving method.
Note that we add a@validate_savesdecorator to instructsave_object()to automatically runvalidate_object()on the directory by theCoffeemethod.
This confirms that the output is valid according to our (yet to be added) validator method.importdolomite_baseimportosimportjson@dolomite_base.save_object.register@dolomite_base.validate_savesdefsave_object_for_Coffee(x:Coffee,path:str,**kwargs):os.mkdir(path)withopen(os.path.join(path,"bean_type"),"w")ashandle:handle.write(x.beans)withopen(os.path.join(path,"has_milk"),"w")ashandle:handle.write("true"ifx.milkelse"false")withopen(os.path.join(path,"OBJECT"),"w")ashandle:json.dump({"type":"coffee","coffee":{"version":"0.1"}},handle)Then we implement and register the reading method:fromtypingimportDictdefread_Coffee(path:str,metadata:Dict,**kwargs)->Coffee:metadata["coffee"]["version"]# possibly do something different based on versionwithopen(os.path.join(path,"bean_type"),"r")ashandle:beans=handle.read()withopen(os.path.join(path,"has_milk"),"r")ashandle:milk=(handle.read()=="true")returnCoffee(beans,milk)dolomite_base.read_object_registry["coffee"]=read_CoffeeAnd finally, the validation method:defvalidate_Coffee(path:str,metadata:Dict):metadata["coffee"]["version"]# possibly do something different based on versionwithopen(os.path.join(path,"bean_type"),"r")ashandle:beans=handle.read()ifnotbeansin["arabica","robusta","excelsa","liberica"]:raiseValueError("wrong bean type '"+beans+"'")withopen(os.path.join(path,"has_milk"),"r")ashandle:milk=handle.read()ifnotmilkin["true","false"]:raiseValueError("invalid milk '"+milk+"'")dolomite_base.validate_object_registry["coffee"]=validate_CoffeeLet's run them and see how it works:cup=Coffee("arabica",milk=False)importtempfiletmp=tempfile.mkdtemp()path=os.path.join(tmp,"stuff")dolomite_base.save_object(cup,path)cup2=dolomite_base.read_object(path)print(cup2.beans)## arabicaFor more complex objects that are composed of multiple smaller "child" objects, developers should consider saving each of their children in subdirectories ofpath.
This can be achieved by callingalt_save_object()andalt_read_object()in the saving and loading functions, respectively.
(We use thealt_*versions of these functions to respect application overrides, see below.)Creating applicationsDevelopers can also create applications that customize the machinery of thedolomiteframework for specific needs.
In most cases, this involves storing more metadata to describe the object in more detail.
For example, we might want to remember the identity of the author for each object.
This is achieved by creating an application-specific saving generic with the same signature assave_object():fromfunctoolsimportsingledispatchfromtypingimportAny,Dict,Optionalimportdolomite_baseimportjsonimportosimportgetpassimportbiocframedefdump_extra_metadata(path:str,extra:Dict):user_id=getpass.getuser()# File names with leading underscores are reserved for application-specific# use, so they won't clash with anything produced by save_object().metapath=os.path.join(path,"_metadata.json")withopen(metapath,"w")ashandle:json.dump({**extra,"author":user_id},handle)@singledispatchdefapp_save_object(x:Any,path:str,**kwargs):dolomite_base.save_object(x,path,**kwargs)# does the real workdump_extra_metadata(path,{})# adding some application-specific metadata@app_save_object.registerdefapp_save_object_for_BiocFrame(x:biocframe.BiocFrame,path:str,**kwargs):dolomite_base.save_object(x,path,**kwargs)# does the real work# We can also override specific methods to add object+application-specific metadata:dump_extra_metadata(path,{"columns":x.get_column_names().as_list()})In general, applications should avoid modifying the files created by thedolomite_base.save_object()call, to avoid violating anytakaneformat specifications
(unless the application maintainer really knows what they're doing).
Applications are free to write to any path starting with an underscore as this will not be used by any specification.Once a generic is defined, applications should callalt_save_object_function()to instructalt_save_object()to use it instead ofdolomite_base.save_object().
This ensures that the customizations are applied to all child objects, such as the nestedBiocFramebelow.# Create a friendly user-visible function to perform the generic override; this# is reversed on function exit to avoid interfering with other applications.defsave_for_application(x,path:str,**kwargs):old=dolomite_base.alt_save_object_function(app_save_object)try:dolomite_base.alt_save_object(x,path,**kwargs)finally:dolomite_base.alt_save_object_function(old)# Saving our nested BiocFrames with our overrides active.importbiocframedf=biocframe.BiocFrame({"A":[1,2,3,4],"B":biocframe.BiocFrame({"C":["a","b","c","d"]})})importtempfiletmp=tempfile.mkdtemp()path=os.path.join(tmp,"foobar")save_for_application(df,path)# Both the parent and child BiocFrames have new metadata.withopen(os.path.join(path,"_metadata.json"),"r")ashandle:print(handle.read())## {"columns": ["A", "B"], "author": "aaron"}withopen(os.path.join(path,"other_columns","1","_metadata.json"),"r")ashandle:print(handle.read())## {"columns": ["C"], "author": "aaron"}The reading function can be similarly overridden by settingalt_read_object_function()to instruct allalt_read_object()calls to use the override.
This allows applications to, e.g., do something with the metadata that we just added.defapp_read_object(path:str,metadata:Optional[Dict]=None,**kwargs):ifmetadataisNone:withopen(os.path.join(path,"OBJECT"),"r")ashandle:metadata=json.load(handle)# Print custom message based on the type and application-specific metadata.withopen(os.path.join(path,"_metadata.json"),"r")ashandle:appmeta=json.load(handle)print("I am a "+metadata["type"]+" created by "+appmeta["author"])ifmetadata["type"]=="data_frame":print("I have the following columns: "+", ".join(appmeta["columns"]))returndolomite_base.read_object(path,metadata=metadata,**kwargs)# Creating a user-friendly function to set the override before the read operation.defread_for_application(path:str,metadata:Optional[Dict]=None,**kwargs):old=dolomite_base.alt_read_object_function(app_read_object)try:returndolomite_base.alt_read_object(path,metadata=metadata,**kwargs)finally:dolomite_base.alt_read_object_function(old)# This diverts to the override with printing of custom messages.read_for_application(path)## I am a data_frame created by aaron## I have the following columns: A, B## I am a data_frame created by aaron## I have the following columns: CBy overriding the saving and reading process for one or more classes, each application can customize the behavior of thedolomiteframework to their own needs.