Package: chai-sacred
Every experiment is sacredEvery experiment is greatIf an experiment is wastedGod gets quite irateSacred is a tool to help you configure, organize, log and reproduce experiments.
It is designed to do all the tedious overhead work that you need to do around
your actual experiment in order to:keep track of all the parameters of your experimenteasily run your experiment for different settingssave configurations for individual runs in a databasereproduce your resultsSacred achieves this through the following main mechanisms:Config ScopesA very convenient way of the local variables in a function
to define the parameters your experiment uses.Config Injection: You can access all parameters of your configuration
from every function. They are automatically injected by name.Command-line interface: You get a powerful command-line interface for each
experiment that you can use to change parameters and run different variants.Observers: Sacred provides Observers that log all kinds of information
about your experiment, its dependencies, the configuration you used,
the machine it is run on, and of course the result. These can be saved
to a MongoDB, for easy access later.Automatic seedinghelps controlling the randomness in your experiments,
such that the results remain reproducible.ExampleScript to train an SVM on the iris datasetThe same script as a Sacred experimentfromnumpy.randomimportpermutationfromsklearnimportsvm,datasetsC=1.0gamma=0.7iris=datasets.load_iris()perm=permutation(iris.target.size)iris.data=iris.data[perm]iris.target=iris.target[perm]clf=svm.SVC(C,'rbf',gamma=gamma)clf.fit(iris.data[:90],iris.target[:90])print(clf.score(iris.data[90:],iris.target[90:]))fromnumpy.randomimportpermutationfromsklearnimportsvm,datasetsfromsacredimportExperimentex=Experiment('iris_rbf_svm')@ex.configdefcfg():C=1.0gamma=0.7@ex.automaindefrun(C,gamma):iris=datasets.load_iris()per=permutation(iris.target.size)iris.data=iris.data[per]iris.target=iris.target[per]clf=svm.SVC(C,'rbf',gamma=gamma)clf.fit(iris.data[:90],iris.target[:90])returnclf.score(iris.data[90:],iris.target[90:])DocumentationThe documentation is hosted atReadTheDocs.InstallingYou can directly install it from the Python Package Index with pip:pip install sacredOr if you want to do it manually you can checkout the current version from git
and install it yourself:git clonehttps://github.com/IDSIA/sacred.gitcd sacredpython setup.py installYou might want to also install thenumpyand thepymongopackages. They are
optional dependencies but they offer some cool features:pip install numpy, pymongoTestsThe tests for sacred use thepytestpackage.
You can execute them by runningpytestin the sacred directory like this:pytestThere is also a config file fortoxso you
can automatically run the tests for various python versions like this:toxUpdate pyptest versionIf you update or change the pytest version, the following files need to be changed:dev-requirements.txttox.initest/test_utils.pysetup.pyContributingIf you find a bug, have a feature request or want to discuss something general you are welcome to open anissue. If you have a specific question related
to the usage of sacred, please ask a question on StackOverflow under thepython-sacred tag. We value documentation
a lot. If you find something that should be included in the documentation please
document it or let us know whats missing. If you are using Sacred in one of your projects and want to share
your code with others, put your repo in theProjects using Sacred <docs/projects_using_sacred.rst>_ list.
Pull requests are highly welcome!FrontendsAt this point there are three frontends to the database entries created by sacred (that I’m aware of).
They are developed externally as separate projects.OmniboardOmniboard is a web dashboard that helps in visualizing the experiments and metrics / logs collected by sacred.
Omniboard is written with React, Node.js, Express and Bootstrap.IncenseIncense is a Python library to retrieve runs stored in a MongoDB and interactively display metrics and artifacts
in Jupyter notebooks.SacredboardSacredboard is a web-based dashboard interface to the sacred runs stored in a
MongoDB.NeptuneNeptune is a metadata store for MLOps, built for teams that run a lot of experiments.
It gives you a single place to log, store, display, organize, compare, and query all your model-building metadata via API available for both Python and R programming languages:In order to log your sacred experiments to Neptune, all you need to do is add an observer:fromneptune.new.integrations.sacredimportNeptuneObserverex.observers.append(NeptuneObserver(api_token='<YOUR_API_TOKEN>',project='<YOUR_WORKSPACE/YOUR_PROJECT>'))For more info, check theNeptune + Sacred integration guide.SacredBrowserSacredBrowser is a PyQt4 application to browse the MongoDB entries created by
sacred experiments.
Features include custom queries, sorting of the results,
access to the stored source-code, and many more.
No installation is required and it can connect to a local
database or over the network.ProphetProphet is an early prototype of a webinterface to the MongoDB entries created by
sacred experiments, that is discontinued.
It requires you to runRestHeartto access the database.Related ProjectsSumatraSumatra is a tool for managing and tracking projects based on numericalsimulation and/or analysis, with the aim of supporting reproducible research.It can be thought of as an automated electronic lab notebook forcomputational projects.Sumatra takes a different approach by providing commandline tools to initialize
a project and then run arbitrary code (not just python).
It tracks information about all runs in a SQL database and even provides a nice browser tool.
It integrates less tightly with the code to be run, which makes it easily
applicable to non-python experiments.
But that also means it requires more setup for each experiment and
configuration needs to be done using files.
Use this project if you need to run non-python experiments, or are ok with the additional setup/configuration overhead.Future Gadget LaboratoryFGLab is a machine learning dashboard, designed to make prototypingexperiments easier. Experiment details and results are sent to a database,which allows analytics to be performed after their completion. The serveris FGLab, and the clients are FGMachines.Similar to Sumatra, FGLab is an external tool that can keep track of runs from
any program. Projects are configured via a JSON schema and the program needs to
accept these configurations via command-line options.
FGLab also takes the role of a basic scheduler by distributing runs over several
machines.CDEBy tracing system calls during program execution CDE creates a snapshot ofallused files and libraries to guarantee the ability to reproduce any unix
program execution. Itonlysolves reproducibility, but it does so thoroughly.LicenseThis project is released under the terms of theMIT license.Citing SacredK. Greff, A. Klein, M. Chovanec, F. Hutter, and J. Schmidhuber, ‘The Sacred Infrastructure for Computational Research’,
in Proceedings of the 15th Python in Science Conference (SciPy 2017), Austin, Texas, 2017, pp. 49–56.