Package: lightex
LightExLightExis a lightweight experiment framework to create, monitor and record your machine learning experiments. Targeted towards individual data scientists, researchers, small teams and, in general, resource-constrained experimentation. Compatible with all machine-learning frameworks.Project Status:AlphaUnlike most experiment frameworks,LightExsports a modular, and highly configurable design:dispatcher: run experiments usingprocess,dockercontainers orkubernetespods. Switch between modes seamlessly by minor changes to config.mulogger: log metrics and artifacts to multiple logger backends, using anunifiedAPI. Supportsmlflow,tensorboardandtrains— add new loggers easily asplugins.namedconf:  pythondataclassbased flexible andunifiedconfiguration specification for jobs, parameters and model architectures. Config instances arenamedand can belocally modified.qviz: query, compare and visualize your experiment results.The run environment and parameters for your experiments are specified using a config filelxconfig.pyin your project directory. Modify, inherit, and create newnamedconfig instances, on-the-fly, as you perform a series of experiments.Learn more about theanatomyof a ML experimentation frameworkhere. Also, why [yet another experiment framework] (#yet-another-experiment-framework) ?BenefitsStart with a basictrainorevalproject. In a few minutes,introduce systematic logging (multiple loggers) and visualization to your projectgo from running a single experiment to multiple parameterized experiments, e.g.,multiple training runs over a set of hyper-parameters.multipleefficient-netorberttrain or eval runs.a neural architecture search over multiple architectures in parallel.Installationpip install -U lightexQuick StartImagine we have an existing ML project, with the following model train command:train.py --data-dir ./data —-lr 0.1 -—hidden_dim 512Now, let us bring inlightexto organize and scale experiments for the project. In the main project directory, initializelightex:lx initThis creates fileslxconfig.pyandrun_expts.py. The filelxconfig.pycontains pre-defineddataclasses for specifyingnamedexperiment configs.The main classConfig, contains three fields:er,hpandrun(see the example below), which store configurations for logging and storageresources,hyper-parameters, andruntime, respectively.By configuring these three fields (depending on the project environment, parameters), we canseamlesslygenerate multiple experiments via different dispatchers and log the results to one or more logger backends.The config classesResources,HPandRunfor these fields are pre-defined  inlxconfig.py. Seeconfig.mdfor full description of the defined dataclasses.Configalso includes aget_experimentsfunction, which generates a list of experiment configs to be executed by the dispatcher. .@dataclassclassConfig:er:Resources#(Logging, Storage resources)hp:HP#(Hyper-parameters of model, training)run:Run#(Run-time config)defget_experiments(self):#required: generate a list of experiments to runexpts=[Experiment(er=self.er,hp=self.hp,run=self.run)]returnexptsTo create aConfiginstance for our existing project, we instantiateclassHPwith parameter values (lrandhidden_dim), andclassRun'scmdfield with the train commandtemplate. The template refers toExperimentobjects returned byget_experimentsfunction (of thisConfiginstance).Need not create the template manually. Useargparse_to_commandin your existing code to generate it.classResourceswith default values (stores experiments in/tmp/ltex).cmd="python train.py --data-dir {{run.data_dir}} --lr {{hp.lr}} --hidden_dim {{hp.hidden_dim}}"#template placeholders refer to fields of Experiment instanceRu1=Run(cmd=cmd,experiment_name="find_hp")H2=HP(lr=1e-2,hidden_dim=512)R1=Resources()C1=Config(er=R1,hp=H1,run=Ru1)Once the config instanceC1is defined, run your experiments as follows:python run_expts.py -c C1That's it!Now, your experiments, logs, metrics and models are executed and recorded systematically.Modify Experiment Parameters, Experiment GroupsModify configs from previous experiments quickly usingreplaceand run new experiments.Example: Create a newHPinstance and replace it inC1to create a newConfig.Recursivereplace also supported.H2=HP(lr=1e-3,hidden_dim=1024)C2=replace(C1,hp=H2)#inherit er=R1 and run=Ru1python run_expts.py -c C2Experiment GroupsWe can create multiple similar experiments programmatically and run them in parallel. To specify suchexperiment groups, specify a set ofHPs in aHPGroupand update theget_experimentsfunction (seescripts/lxconfig.pyfor an example). In the usual case, these experiments have different hyper-parameters but shareResourcesandRuninstances.Note: Although LightEx pre-defines the dataclass hierarchy, it allows the developer plenty of flexibility in defining the individual fields of classes, in particular, the fields of theHPclass.Adding Logging to your CodeUse the unifiedMultiLoggerAPIto log metrics and artifacts to multiple logger backends.Supported Loggers:mlflow,tensorboard,trains,wandb.fromlightex.muloggerimportMLFlowLogger,MultiLogger,PytorchTBLoggerlogger=MultiLogger(['mlflow','trains'])logger.start_run()# log to trains onlylogger.log('trains',ltype='hpdict',value={'alpha':alpha,'l1_ratio':l1_ratio})# log to mlflow onlylogger.log('mlflow',ltype='scalardict',value={'mae':mae,'rmse':rmse,'r2':r2},step=1)# log to alllogger.log('*',ltype='scalardict',value={'mae':mae,'rmse':rmse,'r2':r2},step=3)# log scalars and tensors, if supported by the logger backendlogger.log('trains',ltype='1d',name='W1',value=Tensor(..),histogram=True,step=4)logger.end_run()Or, use one of the existing loggers' API directly.logger=MLFlowLogger();mlflow=logger.mlflow# call mlflow APIlogger=PytorchTBLogger();writer=logger.writer#call tensorboard's API# Similarly, use TrainsLogger for trains and WBLogger for wandbNote: Except for changes in logging, no changes are required to your existing code! Setup the logger backend using scriptshere.Switch to DockerSetting up thelxconfiginstances pays off here!Now, add aDockerfileto your project which builds the runtime environment with all the project dependencies. Update theBuildinstance insideResourcesconfig. Seeexamples/sklearn, for example.python run_expts.py -c C2 -e dockerBoth your code and data are mounted on the container (no copying involved) — minimal disruption in your dev cycle.Advanced FeaturesMore advanced features are in development stage.Running Experiments on multiple nodes / serversIf you've setup a dockerswarmorkubernetescluster, few changes to the existing config instance allow changing the underlying experiment dispatcher.We need to virtualize code (by adding to Dockerfile) and storage.Create a shared NFS on your nodes. Switch storage config to the NFS partition. Setup scripts will be added.Better QViz module, Logger PluginsImprovements to qviz module and a better plugin system for loggers being developed.Setup SummaryIn summary,LightExinvolves the followingone-time setup:config values inlxconfig.pySetup backend logger servers (only the ones required). Instructionshere. (Optional)Update logging calls in your code to callmuloggerAPI. (Optional)Dockerfile for your project, if you want to use containers for dispatch. (Optional)WhileLightExis quick to start with, it is advisable to spend some time understanding theconfig schema.DependenciesPython > 3.6 (requiredataclasses, included during install). Other dependencies defined insetup.py.Design ChallengesThe configuration system allows you to abstract away the locations of thecode(project directory),datadirectory, and theoutputdirectory, via thecmdfield ofRun. This enables running your code as-is across all dispatcher environments (process, docker, kubernetes).A significant part of experiment manager design is about setting up and propagating a giant web of configuration variables.No optimal choice here:json,yaml,jsonnet— all formats have issues.Usingdataclasses, we can write complex config specs, with built-in inheritance and ability to dolocal updates. Tiny bit of a learning curve here, bound to python, but we gain a lot of flexibility.A unifiedmuloggerAPI to abstract away the API of multiple logging backends.Designing multiple dispatchers with similar API, enabling containers and varying storage options.Read more on challengeshere.ReferencesML Experiment Frameworks:kubeflow,mlflow,polyaxon, ...Loggers:sacred,trains,wandb,Trixi,ml_loggerMotivating Dataclassesintro,how-differentFlexible configurationin modeling: allennlp, gin, jiant.in orchestration:ksonnet, kubernetes-operatorOn the pains of ML experimentationan article fromwandbMost current (2019 Q3) tools focus on theloggercomponent and provide selectiveqvizcomponents.kubeflowandpolyaxonare tied to the (k8s)dispatcher. Every tool has its own version of config management — mostlyyamlbased, where config types are absent or have a non-nested config class. Config-specific languages have been also proposed (ksonnet, sonnet, gin).Yet Another Experiment FrameworkSystematic experimentation tools are essential for a data scientist. Unfortunately, many existing tools (kubeflow,mlflow,polyaxon) are too monolithic, kubernetes-first, cloud-first, target very diverse audiences and hence spread too thin, and yet lack important dev-friendly features.sacred's design is' tightly coupled and requires severalsacred-specific changes to your main code. Other tools cater only to a specific task , e.g.,tensorboardonly handles log recording and visualization. Also, contrasting different experiment frameworks is hard: there is no standardized expt-management architecture for machine learning and most open-source frameworks are undergoing a process of adhoc requirements discovery.AuthorNishant Sinha,OffNote Labs(nishant@offnote.co, @medium, @twitter)ContributorsAkarsh E S,github