Package: mlgym
a feature-rich deep learning framework providing full reproducibility of experiments.Reproducibility is a recurring issue in deep learning (research) with models often being implemented in Jupyter notebooks or entire training and evaluation pipelines implemented from scratch with every new project.
The lack of standardization and repetitive boilerplate code of experimental setups impede reproducibility.MLgym aims to increase reproducibility by separating the experimental setup from the code and providing the entire infrastructure for e.g., model training, model evaluation, experiment logging, checkpointing and experiment analysis.Specifically, MLgym provides an extensible set of machine learning components (e.g., trainer, evaluator, loss functions, etc.). The framework instantiates these components dynamically as specified and parameterized within a configuration file (see here, for an exemplary configuration) describing the entire experiment setup (i.e., training and evaluation pipeline). The separation of experimental setup and code maximizes the replicability and interpretability of ML experiments. The machine learning components cut down the implementational efforts significantly and lets your focus solely on your ideas.Additionally, MLgym provides the following key features:Component registry to register custom components and their dependencies.Warm starts allowing to resume training after crashCustomizable checkpointing strategiesMLboard webservice for experiment tracking / analysis (live and offline) by subscribing to the websocket logging environmentLarge scale, multi GPU training supporting grid search, nested cross validation and cross validationDistributed logging via websockets and event sourcing, allowing location-independent logging and full replicabilityDefinition of training and evaluation pipeline in a configuration file, achieving separation of experiment setup and code.Please note, that at the moment this code should be treated as experimental and is not production ready.Installthere are two options to install MLgym, the easiest way is to install the framework from the pip repository:pipinstallmlgymFor the latest version, one can directly install it from source bycdinto the root folder and then runningpipinstallsrc/UsageWe provide an easy-to-use example that lets you run a MLgymexperiment setup.Before running the experiments we need to setup the MLboard logging environment, i.e., the websocket service and the RESTful webservice. MLgym logs the training/evaluation progress and evaluation results via the websocket API, allowing the MLboard frontend to receive live updates. The RESTful webservice provides endpoints to receive checkpoints and experiment setups. For a full specification of both APIs seehere.We start the websocket service and the RESTful webservice on ports 5001 and 5002, respectively. Feel free to choose different ports if desired.
Similarly, we specify the folderevent_storageas the local event storage folder. Note, to access the websocket service from a different port, we need to specify theCORSallowed origins. In thise example, we only use the websocket service locally from 127.0.0.1:8080 via the MLboard frontend.ml_board_ws_endpoint--host127.0.0.1--port5002--event_storage_pathevent_storage--cors_allowed_originshttp://127.0.0.1:8080

ml_board_rest_endpoint--port5001--event_storage_pathevent_storageNext, we run the experiment setup. Wecdinto the example folder and runrun.pywith the respective config whose path is passed via the parametergs_config_path.
The parameterprocess_countspecifies the number of experiments that we run in parallel.num_epochslimits the maximum number of epochs to train a model. If the model performance does not improve substantially over time, the checkpointing strategy defined ings_config.ymlwill stop training prematurely.cdmlgym/example/grid_search_example

pythonrun.py--process_count3\--text_logging_pathgeneral_logging/\--num_epochs10\--websocket_logging_servershttp://127.0.0.1:5002\--gs_rest_api_endpointhttp://127.0.0.1:5001\train\--gs_config_pathgs_config.ymlTo visualize the live updates, we run the MLboard frontend. We specify the server host and port that delivers the frontend and the endpoints of the REST webservice and the websocket service. The parameterrun_idrefers to the experiment run that we want to analyze and differs in your case. Each experiment runs is stored in separate folders within theevent_storagepath. The folder names refer to the respective experiment run ids.ml_board--ml_board_host127.0.0.1--ml_board_port8080--rest_endpointhttp://127.0.0.1:5001--ws_endpointhttp://127.0.0.1:5002--run_id2022-11-06--17-59-10The script returns the parameterized URL pointing to the respective experiment run:====> ACCESS MLBOARD VIA http://127.0.0.1:8080?rest_endpoint=http%3A//127.0.0.1%3A5001&ws_endpoint=http%3A//127.0.0.1%3A5002&run_id=2022-11-06--17-59-10Note, that the Flask webservice delivers the compiled react files statically, which is why any changes to the frontend code will not be automatically reflected. As a solution, you can start the MLboard react app directly via yarn and call the URL with the respective URL search params in the browsercdmlgym/src/ml_board/frontend/dashboard

yarnstartTo this day, the MLboard frontend is still under development and not all features have been implemented, yet. Therefore, it is possible analyze the log files directly in the event storage. All messages are logged as specified within thewebsocket APITo see the messages livecdinto the event storage directory andtailtheevent_storage.logfile.cdevent_storage/2022-11-06--17-59-10/
tail-fevent_storage.logMLboardSince MLboard is still under heavy development, we would like to give you a sneak peek about what is going to come in the foreseeable future.CopyrightCopyright (c) 2020 Max LübberingFor license see:https://github.com/mlgym/mlgym/blob/master/LICENSE