Package: mltrace
mltracemltraceis a lightweight, open-source Python tool to get "bolt-on" observability in ML pipelines. It offers the following:interface to define data and ML tests for components in pipelinescoarse-grained lineage and tracingPython API to log versions of data and pipeline componentsdatabase to store information about component runsUI and CLI to show the trace of steps in a pipeline taken to produce an output, flag outputs for review, and identify what steps of the pipeline to investigate first when debuggingmltraceis designed specifically for Agile or multidisciplinary teams collaborating on machine learning or complex data pipelines. The prototype is very lofi, but thisREADMEcontains instructions on how to run the prototype on your machineif you are interested in developing.For general usage instructions, please see theofficial documentation. The accompanying blog post can be foundhere.QuickstartYou should have Docker installed on your machine. To get started, you will need to do 3 things:Set up the database and Flask serverRun some pipelines with loggingLaunch the UIIf you are interested in learning about specificmltraceconcepts, please readthis pagein the official docs.Database setup (server-side)We use Postgres-backed SQLAlchemy. Assuming you have Docker installed, you can run the following commands from the
root directory after cloning the most recent release:docker-compose build
docker-compose up [-d]And then to tear down the containers, you can rundocker-compose down.Run pipelines (client-side)To use the logging functions in dev mode, you will need to install various dependencies:pip install -r requirements.txt
pip install -e .Next, you will need to set the database URI. It is recommended to use environment variables for this. To set the database address, set theDB_SERVERvariable:export DB_SERVER=<SERVER'S IP ADDRESS>where<SERVER'S IP ADDRESS>is either the IP address of a remote machine orlocalhostif running locally. If, when you set up the server, you changed the URI indocker-compose.yaml, you can set theDB_URIvariable (which represents the entire database URI) client-side instead ofDB_SERVER.The files in theexamplesfolder contain sample scripts you can run. For instance, if you runexamples/industry_ml.py, you might get an output like:> python examples/industry_ml.py
Final output id: aafknvtoagAnd if you trace this output in the UI (trace aafknvtoag), you will get:You can also look atexamplesfor ways to integratemltraceinto your ML pipelines, or check out theofficial documentation.Launch UI (client-side)If you randocker-compose upfrom the root directory, you can just navigate to the server's IP address at port 8080 (orlocalhost:8080) in your browser. To launch a dev version of the UI, navigate to./mltrace/server/uiand executeyarn installthenyarn start. It should be served atlocalhost:3000. The UI is based oncreate-react-appandblueprintjs. Here's an example of what tracing an output would give:Commands supported in the UICommandDescriptionrecentShows recent component runs, also the home pagehistory COMPONENT_NAMEShows history of runs for the component name. Defaults to 10 runs. Can specify number of runs by appending a positive integer to the command, likehistory etl 15inspect COMPONENT_RUN_IDShows info for that component run IDtrace OUTPUT_IDShows a trace of steps for the output IDtag TAG_NAMEShows all components with the tag nameflag OUTPUT_IDFlags an output ID for further review. Necessary to see any results from thereviewcommand.unflag OUTPUT_IDUnflags an output ID. Removes this output ID from any results from thereviewcommand.reviewShows a list of output IDs flagged for review and the common component runs involved in producing the output IDs. The component runs are sorted from most frequently occurring to least frequently occurring.Launch without UI (client-side)If you want to launch database and api containers without the UI, you will rundocker-compose docker-compose-not-ui.yml upfrom the root directory. If running correctly, you should see nothing displayed in the server's IP address at port 8080 (orlocalhost:8080) but the database and API service should work unaffectedly:change directory to the level where docker-compose-not-ui.yml exist (root directory)rundocker-compose -f docker-compose-not-ui.yml buildto build the image (skip this step if image already exist)rundocker-compose -f docker-compose-not-ui.yml upto bring up the serviceUsing the CLI for queryingThe following commands are supported via CLI:historyrecenttraceflagunflagreviewYou can executemltrace --helpin your shell for usage instructions, or you can executemltrace command --helpfor usage instructions for a specific command.Future directionsThe following projects are in the immediate roadmap:API to log from any type of file, not just a Python filePrometheus integrations to monitor component output distributionsSupport for finer-grained lineage (at the record level)ContributingAnyone is welcome to contribute, and your contribution is greatly appreciated! Feel free to either create issues or pull requests to address issues.Fork the repoCreate your branch (git checkout -b YOUR_GITHUB_USERNAME/somefeature)Make changes and add files to the commit (git add .)Commit your changes (git commit -m 'Add something')Push to your branch (git push origin YOUR_GITHUB_USERNAME/somefeature)Make a pull request