Package: scalarstop
Keep track of your machine learning experiments with ScalarStop.ScalarStop is a Python framework for reproducible machine learning research.It was written and open-sourced atNeocrym, where it is used to train thousands of models every week.ScalarStop can help you:organize datasets and models withcontent-addressablenames.save/load datasets and models to/from the filesystem.record hyperparameters and metrics to a relational database.System requirementsScalarStop is a Python package that requires Python 3.8 or newer.Currently, ScalarStop only supports trackingtf.data.Datasetdatasets andtf.keras.Modelmodels. As such, ScalarStop requires TensorFlow 2.8.0 or newer.We encourage anybody that would like to add support for other
machine learning frameworks to ScalarStop. :)InstallationScalarStop isavailable on PyPI.Selecting a TensorFlow package variantIf you are using TensorFlow on a CPU, you can install ScalarStop with the command:python3-mpipinstallscalarstop[tensorflow]If you are using TensorFlow with GPUs, you can install ScalarStop with the command:python3-mpipinstallscalarstop[tensorflow-gpu]Selecting a PostgreSQL psycopg2 package variantIf you intend to use ScalarStop with PostgreSQL, you should also
install eitherpsycopg2-binary(which works out of the box)
orpsycopg2(which you compile from source).Therefore, your installation command could look like either:python3-mpipinstallscalarstop[tensorflow,psycopg2]python3-mpipinstallscalarstop[tensorflow,psycopg2-binary]python3-mpipinstallscalarstop[tensorflow-gpu,psycopg2]python3-mpipinstallscalarstop[tensorflow-gpu,psycopg2-binary]DevelopmentIf you would like to make changes to ScalarStop, you canclone the repositoryfrom GitHub.gitclonehttps://github.com/scalarstop/scalarstop.gitcdscalarstoppython3-mpipinstall.UsageRead theScalarStop Tutorialto learn the core concepts behind ScalarStop and how to structure your datasets and models.Afterwards, you might want to dig deeper into theScalarStop Documentation. In general, a typical ScalarStop workflow involves four steps:1. Organize your datasets withscalarstop.datablob.2. Describe your machine learning model architectures usingscalarstop.model_template.3. Load, train, and save machine learning models withscalarstop.model.4. Save hyperparameters and training metrics to a SQLite or PostgreSQL database usingscalarstop.train_store.Contributing to ScalarStopWe warmly welcome contributions to ScalarStop. Here are the technical details for getting started with adding code to ScalarStop.Getting startedFirst, clone this repository from GitHub. All development happens on themainbranch.gitclonehttps://github.com/scalarstop/scalarstop.gitThen, runmake installto install Python dependencies in a Poetry virtualenv.You can runmake helpto see the other commands that are available.Checking your codeRunmake fmtto automatically format code.Runmake lintto run Pylint and MyPy to check for errors.Generating documentationDocumentation is important! Here is how to add to it.Generating Sphinx documentationYou can generate a local copy of our Sphinx documentation atscalarstop.comwithmake docs.The generated documentation can be found atdocs/_build/dirhtml. To view it, you should start an HTTP server in this directory, such as:makedocscddocs/_build/dirhtmlpython3-mhttp.server5000Then visithttp://localhost:5000in your browser to preview changes to the documentation.If you want to use Sphinx’s ability to automatically generate hyperlinks to the Sphinx documentation of other Python projects, then you should configureintersphinxsettings at the path docs/conf.py. If you need to download an objects.inv file, make sure to update themakeupdate-sphinxcommand in theMakefile.Editing the tutorial notebookThe main ScalarStop tutorial isin a Jupyter notebook. If you have made changes to ScalarStop, you should rerun the Jupyter notebook on your machine with your changes to make sure that it still runs without error.Running unit testsRunmake testto run all unit tests.If you want to run a specific unit test, try runningpython3-mpoetry run python-munittest-k{name of your test}.Unit tests with SQLite3If you are running tests using a Python interpreter that does not have theSQLite3 JSON1 extension, thenTrainStoreunit tests involving SQLite3 will be skipped. This is likely to happen if you are using Python 3.8 on Windows. If you suspect that you are missing the SQLite3 JSON1 extension, theDjango documentation has some suggestionsfor how to fix it.Unit tests with PostgreSQLBy default, tests involving PostgreSQL are skipped. To enable PostgreSQL, runmake testin a shell where the environment variableTRAIN_STORE_CONNECTION_STRINGis set to aSQLAlchemy database connection URL–which looks something like"postgresql://scalarstop:changeme@localhost:5432/train_store". The connection URL should point to a working PostgreSQL database with an existing database and user.The docker-compose.yml file in the root of this directory can set up a PostgreSQL instance on your local machine. If you haveDockerandDocker Composeinstalled, you can start the PostgreSQL database by runningdocker-composeupin the same directory as the docker-compose.yml file.Measuring test coverageYou can runmaketest-with-coverageto collect Python line and branch coverage information. Afterwards, runmakecoverage-htmlto generate an HTML report of unit test coverage. You can view the report in a web browser at the pathhtmlcov/index.html.CreditsScalarStop’s documentation is built withSphinxusing@pradyunsg’sFurotheme and is hosted byRead the Docs.