Package: dsch
dschIntroductionDsch provides a way to store data and its metadata in a structured, reliable
way. It is built upon well-known data storage engines, such as theHDF5file
format, providing performance and long-term stability.The core feature is the schema-based approach to data storage, which means that
a pre-defined schema specification is used to determine:which data fields are availablethe (hierarchical) structure of data fieldsmetadata of the stored values (e.g. physical units)expected data types and constraints for the stored valuesIn fact, this is similar to an API specification, but it can be attached to and
stored with the data. Programswritingdatasets benefit from data validation
and the high-level interface.Readingprograms can determine the given data’s
schema upfront, and process accordingly. This is especially useful with schemas
evolving over time.For persistent storage, dsch supports multiple storage engines via itsbackends, but all through a single, transparent interface. Usually, there are
no client code changes required to support a new backend, and custom backends
can easily be added to dsch.
Currently, backends exist for these storage engines:HDF5files (throughh5py)NumPy .npzfilesMATLAB .matfiles (throughSciPy)Note that dsch is only a thin layer, so that users can still benefit from the
performance of the underlying storage engine. Also, files created with dsch can
always be opened directly (i.e. without dsch) and still provide all relevant
information, even the metadata!ReasoningDsch is a response to the challenges in low-level data acquisition scenarios,
which are commonly found in labs at universities or R&D departments. Frequent
changes in both hardware and software are commonplace in these environments, and
since those changes are often made by different people, the data acquisition
hardware, software and data consumption software tend to get out of sync. At the
same time, datasets are often stored (and used!) for many years, which makes
backwards-compatibility a significant issue.Dsch aims to counteract these problems by making the data exchange process more
explicit. Using pre-defined schemas ensures backward-compatibility as long as
possible, and when it can no longer be retained, provides a clear way to detect
(and properly handle) multiple schema versions. Also, schema based validation
allows to detect possible errors upfront, so that most non-security-related
checks do not have to be re-implemented in data consuming applications.Note that dsch is targeted primarily at these low-level applications. When using
high-level data processing or even data science and machine learning techniques,
data is often pre-processed and aggregated with regard to a specific
application, which often eliminates the need for some of dsch’s features, such
as the metadata storage. One might think of dsch as the tool to handle databeforeit is filled into something likepandas.ChangelogThis project follows the guidelines ofKeep a changelogand adheres toSemantic versioning.0.3.0- 2021-02-12AddedNewdata_treemethod for exporting data as nesteddict/liststructures.ChangedImprove documentation.Improve testsFixedMinor updates to handleh5pydeprecations.0.2.1- 2018-02-02Changedh5pyandscipy, needed for HDF5 and MAT file support, respectively,
are now listed as extras / optional dependencies in setup.py.FixedFix missing type conversion forScalarininmembackend that causes
validation to incorrectly fail in some cases.0.2.0- 2018-02-01AddedNew node type forbytesdata.In-memory backend, for handling data without needing e.g. a file on disk.Support for copying data between different storages.Support for creating new storages from existing ones, aka. “save as”.PseudoStorageabstraction class for unified data access in libraries.Human-readable tree-representation of data nodes for use in interactive
sessions.Support==operator for schema nodes.ChangedData nodes in Compilations and Lists can no longer be overwritten
accidentally when trying to overwrite their stored value.Improve structure and conciseness of docs.Change List to evaluateempty-ness recursively.Replace generic exceptions likeTypeErrorby custom dsch exceptions.0.1.3- 2018-01-11ChangedAttempting to open a non-existent file now shows a sensible error message.Attempting to create an existing file now shows a sensible error message.FixedFix error when handling partially filled compilations.Fix typo in documentation.0.1.2- 2017-08-25FixedFix incorrect ordering of list items.0.1.1- 2017-06-09AddedCover additional topics in documentation.FixedFix error when handling single-element lists withmatbackend.0.1.0- 2017-05-18AddedFirst preview release.