Package: tpcp
tpcp - Tiny Pipelines for Complex ProblemsA generic way to build object-oriented datasets and algorithm pipelines and tools to evaluate them.Easily installtpcpvia pip:pipinstalltpcpOr add it to your project withpoetry:poetryaddtpcpWhy?Evaluating Algorithms - in particular when they contain machine learning - is hard.
Besides understanding required concepts (cross validation, bias, overfitting, ...), you need to implement the required
steps and make them work together with your algorithms and data.
If you are doing something "regular" like training an SVM on tabular data, amazing libraries likesklearn,tslearn,pytorch, and many others, have your back.
By using their built-in tools (e.g.sklearn.evaluation.GridSearchCV) you prevent implementation errors, and you are
provided with a sensible structure to organize your code that is well understood in the community.However, often the problems we are trying to solve are not regular.
They arecomplex.
As an example, here is the summary of the method from one of ourrecent papers:We have continuous multi-dimensional sensor recordings from multiple participants from a hospital visit and multiple days at homeFor each participant we have global metadata (age, diagnosis) and daily annotationsWe want to train a Hidden-Markov-Model that can find events in the data streamsWe need to tune hyper-parameters of the algorithm using a participant-wise cross validationWe want to evaluate the final performance of the algorithm for the settings trained on the hospital data -> tested on home data and trained on home data -> tested on home dataUsing the same structure we want to evaluate a state-of-the-art algorithm to compare the resultsNone of the standard frameworks can easily abstract this problem, because here we have none-tabular data, multiple data
sources per participant, a non-traditional ML algorithm, and a complex train-test split logic.Withtpcpwe want to provide a flexible framework to approach such complex problems with structure and confidence.How?To maketpcpeasy to use, we try to focus on a couple of key ideas:Datasets are Python classes (think ofpytorch.datasets, but more flexible) that can be split, iterated over, and queried.Algorithms and Pipelines are Python classes with a simplerunandoptimizeinterface, that can be implemented to fit any problem.Everything is a parameter and everything is optimization: In regular ML we differentiatetrainingandhyper-parameter optimization.
Intpcpwe consider everything that modifies parameters or weights as anoptimization.
This allows to use the same concepts and code interfaces from simple algorithms that just require a grid search to optimize a parameter to neuronal network pipelines with hyperparameter tuning.Provide what is difficult, allow to change everything else:tpcpimplements complicated constructs like cross validation and grid search and, whenever possible, tries to catch obvious errors in your approach.
However, for the actual algorithm and dataset you are free to do whatever is required to solve your current research question.Should you use tpcp?DatasetsYes- the object-oriented Datasets have proven themselves to be a really nice and flexible way to encapsulate Datasets with data from multiple modalities.
There is a clear path of integrating lazy-loading, load-cashing, data filtering, or pre-processing on loading.
From our experience, even if you ignore all the other tpcp features, Datasets can greatly simplify how you interact with your data sources and can serve as a self-documenting API for your data.Learn more(Examples)Other projects using Datasets:gaitmap-datasetscold-face-test-analysiscarwatch-analysisParameter Optimization and Cross ValidationMaybe- All parameter optimization features in tpcp exist to provide a unified API, in case other specific frameworks are too specialised.
In cases where all your algorithms can be abstracted bysklearn,pytorch(with theskorchwrapper),tensorflow/Keras(with thescikeraswrapper),
or any other framework that provides a nice scikit-learn API, you will get all the features tpcp can provide with much less boilerplate by just usingsklearnandoptunadirectly.
Even, if you need to implement completely custom algorithms, we would encourage you to see if you can emulate a sklearn-like API to make use of its vast ecosystem.This will usually work well for all algorithms that can be abstracted by the fit-predict paradigm.
However, for more "traditional" algorithms with no "fit" step or complicated optimizations, therun(with optionalself_optimize) API of tpcp might be a better fit.
So if you are using or developing algorithms across library domains, that don't all work well with a sklearn API, thenYes, tpcp is a good choice.Learn more:General Concepts,Custom Algorithms,Parameter Optimization,Cross ValidationCitationIf you usetpcpin your research, we would appreciate a citation to our JOSS paper.
This helps us to justify putting time into maintaining and improving the library.Küderle et al., (2023). tpcp: Tiny Pipelines for Complex Problems - 
A set of framework independent helpers for algorithms development and evaluation. 
Journal of Open Source Software, 8(82), 4953,
https://doi.org/10.21105/joss.04953@article{Küderle2023,doi={10.21105/joss.04953},url={https://doi.org/10.21105/joss.04953},year={2023},publisher={The Open Journal},volume={8},number={82},pages={4953},author={Arne Küderle and Robert Richer and Raul C. Sîmpetru and Bjoern M. Eskofier},title={tpcp: Tiny Pipelines for Complex Problems -A set of framework independent helpers for algorithms development and evaluation},journal={Journal of Open Source Software}}ContributionThe entire development is managed viaGitHub.
If you run into any issues, want to discuss certain decisions, want to contribute features or feature requests, just
reach out to us byopening a new issue.Dev SetupWe are usingpoetryto manage dependencies andpoethepoetto run and manage dev tasks.To set up the dev environmentincludingthe required dependencies for using and developing ontpcprun the following
commands:gitclonehttps://github.com/mad-lab-fau/tpcpcdtpcp
poetryinstall--all-extrasAfterward, you can start to develop and change things.
If you want to run tests, format your code, build the docs, ..., you can run one of the followingpoethepoetcommandsCONFIGURED TASKS
  format            
  format_unsafe     
  lint              Lint all files with ruff.
  ci_check          Check all potential format and linting issues.
  test              Run Pytest with coverage.
  docs              Build the html docs using Sphinx.
  docs_clean        Remove all old build files and build a clean version of the docs.
  docs_preview      Preview the built html docs.
  versionby callingpoetryrunpoe<commandname>If you installedpoethepoetglobally, you can skip thepoetry runpart at the beginning.