Package: datafold
Source repository|Contributing and feedback|PyPI|Documentation|Tutorials|Scientific literatureWhat isdatafold?datafoldis aMIT-licensedPython package containing operator-theoretic, data-driven models to identify dynamical
systems from time series data and to infer geometrical structures in point clouds.The package includes:Implementations and variants of the Dynamic Mode Decomposition as data-driven methods to
identify and analyze dynamical systems from time series collection data. This incldues:DMDFullorDMDEcoas standard methods of DMDOnlineDMDorStreamingDMDmodify the DMD to handle streaming dataDMDControlaugments the DMD to handle additional control inputEDMD- The Extended-DMD, which allows setting up a highly flexible dictionary to
decompose and embed time series data and thereby handle nonlinear dynamics within the
Koopman operator framework.EDMDwraps an arbitrary DMD variation for the decomposition.
The key advantage of this is, that theEDMDdirectly profits from the above
functionalities.EDMDcan be used in control or streaming settings. Furthermore, the
dictionary can also be learnt from the data, corresponding to the EDMD-DL.An efficient implementation of theDiffusionMapsmodel to infer geometric
meaningful structures from (time series) data, such as the eigenfunctions of the
Laplace-Beltrami operator. As a distinguishing factor to other implementations, the
model can handle a sparse kernel matrix and allows setting an arbitrary kernel,
including the standard Gaussian kernel,continuous k-nearest neighbor kernel, ordynamics-adapted cone kernel.Cross-validation. The methodEDMDCVallows model parameters to be optimized with
cross-validation splittings that account for the temporal order in time series data.Methods to perform Model Predictive Control (MPC) with Koopman operator-based methods (
mainly theEDMD).Regression models for high-dimensional data, which are commonly used for out-of-sample
extensions for the Diffusion Maps model. This includes the (auto-tuned) Laplacian Pyramids
or Geometric Harmonics to interpolate general function values on a point cloud manifold.A data structureTSCDataFrameto handle time series collection (TSC) data. It simplifies
model inputs/output and make it easier to describe various forms of time series data.See alsothis introduction page.
For a mathematical thorough introduction, we refer to thescientific literature.NoteThe project is under active development in a research-driven environment.Code quality varies from “experimental/early stage” to “well-tested”. Well tested
code is listed in thesoftware documentationand are directly accessible through the highest module level (e.g.from datafold import ...). Experimental code is
only accessible via “deep imports” (e.g.from datafol.dynfold.outofsample import ...) and may raise a warning when using
it.The interfaces withindatafoldare not stable. The software isnotintended for
production. Nevertheless, if we break something it is intentional and we hope that such
adaptations become less over time.There is no deprecation cycle. The software usessemantic versioningpolicy[major].[minor].[patch], i.e.major- making incompatible changes in the (documented) APIminor- adding functionality in a backwards-compatible mannerpatch- backwards-compatible bug fixesWe do not intend to indicate a feature complete milestone with version1.0.CiteIf you usedatafoldin your research, please citethis paperpublished in theJournal of Open Source Software(JOSS).Lehmberg et al., (2020). datafold: data-driven models for point clouds and time series on
manifolds. Journal of Open Source Software, 5(51), 2283,https://doi.org/10.21105/joss.02283BibTeX:@article{Lehmberg2020,
         doi       ={10.21105/joss.02283},
         url       ={https://doi.org/10.21105/joss.02283},
         year      ={2020},
         publisher ={The Open Journal},
         volume    ={5},
         number    ={51},
         pages     ={2283},
         author    ={Daniel Lehmberg and Felix Dietrich and Gerta K{\"o}ster and Hans-Joachim Bungartz},
         title     ={datafold: data-driven models for point clouds and time series on manifolds},
         journal   ={Journal of Open Source Software}}How to get it?Installation requiresPython>=3.9withpipandsetuptoolsinstalled (both packages ship with a
standard Python installation). The package dependencies
install automatically. The main dependencies and their usage indatafoldare listed
in the section “Dependencies” below.There are two ways to installdatafold:1. From PyPIThis is the standard way for users. The package is hosted on the official Python package
index (PyPI) and installs the core package (excluding tutorials and tests). The tutorial
files can be downloaded separatelyhere.To install the package and its dependencies withpip, runpython-mpipinstalldatafoldNoteIf you run Python in an Anaconda environment you can use pip from withinconda.
See alsoofficial instructions.condaactivatevenvcondainstallpippipinstalldatafold2. From sourceThis way is recommended if you want to access the latest (but potentially unstable)
development state, run tests or wish to contribute (see section “Contributing” for details).
Download or git-clone the source code repository.Download the repositoryIf you wish to contribute code, it is required to havegitinstalled. Clone the repository withgitclonehttps://gitlab.com/datafold-dev/datafold.gitIf you only want access to the source code (currentmasterbranch), download one
of the compressed file types
(zip,tar.gz,tar.bz2,tar)Install the package from the downloaded repositorypython-mpipinstall.ContributingAny contribution (code/tutorials/documentation improvements), question or feedback is
very welcome. Either use theissue trackerorEmailus.
Instructions to set updatafoldfor development can be foundhere.DependenciesThe dependencies of the core package are managed in the filerequirements.txtand install withdatafold. The tests, tutorials, documentation and code analysis
require additional dependencies which are managed inrequirements-dev.txt.datafoldintegrates with common packages from thePython scientific computing stack:NumPyNumPy is used throughoutdatafoldand is the default package for numerical
data and algorithms.pandasdatafolduses pandas’DataFrameas a base class forTSCDataFrameto capture various forms of time series data. The data
It includes specific time series collection functionality and is mostly compatible with
pandas’ rich functionality.scikit-learnAlldatafoldalgorithms that are part of the “machine learning pipeline” align
to the scikit-learnAPI.
This is done by deriving the models fromBaseEstimator.
and appropriateMixIns.datafolddefines ownMixInsthat align with the
API in a duck-typing fashion to allow identifying dynamical systems from temporal data
inTSCDataFrame.SciPyThe package is used for elementary numerical algorithms and data structures in
conjunction with NumPy. This includes (sparse) linear least
square regression, (sparse) eigenpairs solver and sparse matrices as
optional data structure for kernel matrices.How does it compare to other software?Note: This list covers only Python packages.scikit-learnprovides algorithms and models along the entire machine learning pipeline, with a
strong focus on static data (i.e. without temporal context).datafoldintegrates
into scikit-learn’ API and all data-driven models are subclasses ofBaseEstimator.
An important contribution ofdatafoldis theDiffusionMapsmodel as popular
framework for manifold learning, which is not contained in scikit-learn’sset of
algorithms.
Furthermore,datafoldincludes dynamical systems as a new model class that is
operable with scikit-learn - the attributes align to supervised learning tasks.
The key differences are that a model processes data of typeTSCDataFrameand instead of a one-to-one relation in the model’s input/output, the model can return
arbitrary many output samples (a time series) for a single input
(an initial condition).PyDMDprovides many variants of theDynamic Mode Decomposition (DMD).datafoldprovides a wrapper
to make models ofPyDMDaccessible. However, a limitation ofPyDMDis that it only
processes single coherent time series, seePyDMD issue 86. The DMD models that are directly included
indatafoldutilize the functionality of the data structureTSCDataFrameand can
therefore process time series collections - in an extreme case only containing snapshot
pairs.PySINDyspecializes on asparsesystem identification of nonlinear dynamical systems to
infer governing equations.