Package: reptar
reptarA tool for storing and analyzing manuscript-scale computational chemistry dataDocumentationMotivation•Installation•File Types•Key-value pairs•Workflow•LicenseMotivationThe computational chemistry community often fails to openly provide raw and/or processed data used to draw their scientific conclusions.For large projects, frameworks such asQCArchive,Materials Project,Pitt Quantum Repository,ioChem-BDand many others provide great storage solutions.
This approach would not be practical for fluid data pipelines and small-scale projects such as a single manuscript.Alternatively, you could use individual files in formats such as JSON, XML, YAML, npz, etc.
These are great options for customizable data storage with their own advantages and disadvantages.
However, you often must choose between (1) a standardized parser that might not support your workflow or (2) writing your own.Reptar is designed for easy data storage and analysis for individual projects.
Customizable parsers provide a simple way to extract new data without submitting issues and pull requests (although this is highly encouraged).
While files are the heart of reptar, it strives to be file-type agnostic by providing the same interface for all supported file types.
The result is a user-specified file streamlined for analysis in Python and archival on places such asGitHubandZenodo.InstallationYou can install reptar fromPyPIby usingpip install reptar.
Or, the latest development version can be installed directly from theGitHub repositoryor fromTestPyPI.gitclonehttps://github.com/aalexmmaldonado/reptarcdreptar
pipinstall.File typesReptar supports four file types with a single interface: exdir, zarr, JSON, and npz.
JSON is a text file for storing key-value pairs with few dimensions (i.e., no large arrays).
NumPy's npz format is useful for arrays; however, no nesting is possible and loading data often requires postprocessing for 0D arrays (e.g.,np.array('data')).Exdir is a simple, yet powerful open file format that mimics theHDF5format with metadata and data stored in directories with YAML and npy files instead of a single binary file.
For more detailed information, please read thisFront. Neuroinform.article about exdir.Zarris a similar hierarchical data format for chunked and compressed NumPy-like arrays and JSON attributes.
Both of these file types provide several advantages such as mixing human-readable and binary files, being easier for version control, and only loading requested portions of arrays into memory.Key-value pairsAll data is stored under akey-valuepair within the reptar framework.
Thekeytells reptar where the data is stored and is conceptually related to standard file paths (without file extensions).
Nested data is specified by separating the nested keys with a/.
For example,energy_pot,md_run/geometry, andentity_idsare all valid keys.
Note thatgradientsand/gradientswould translate to the same value (/species the "root" of the file).WorkflowStoring dataWe refer to a "reptar file" as any file that can be used with thereptar.Fileclass.
Creating a reptar file starts by having a set of data files generated from some calculation.
Paths to these data files are passed intoreptar.Creator.from_calcthat extracts information using areptar.parserclass.
Information parsed from these files,parsed_info, is then used to populate areptar.Fileobject.Data can also be manually added by usingFile.put(key, data)wherekeyis a string specifying where to store the data.Accessing dataData can be added or retrieved using the same interface regardless of the underlying file format (e.g., exdir, JSON, and npz).
The only thing required is the respectivekeyspecifying where it is stored.
Then,File.get(key)can retrieve the data.When working with JSON and npz files,File.save()must be explicitly called after any modification.Writing to other formatsOther packages often require data to be formatted in their own specific way.
Reptar provides ways to extract data from reptar files usingFile.get(key)and passing it into the desiredreptar.writerfunction.
Reptar currently automates the creation of:Atomic simulation environment (ASE) databases,Gaussian approximate potentials (GAP) extended XYZ files,Protein data bank (PDB) files,Schnetpack databases,XYZ files.LicenseDistributed under the MIT License. SeeLICENSEfor more information.