Package: mle-monitor
Lightweight Experiment & Resource Monitoring 📺"Did I already run this experiment before? How many resources are currently available on my cluster?" If these are common questions you encounter during your daily life as a researcher, thenmle-monitoris made for you. It provides a lightweight API for tracking your experiments using a pickle protocol database (e.g. for hyperparameter searches and/or multi-configuration/multi-seed runs). Furthermore, it comes with built-in resource monitoring on Slurm/Grid Engine clusters and local machines/servers.mle-monitorprovides three core functionalities:MLEProtocol: A composable protocol database API for ML experiments.MLEResource: A tool for obtaining server/cluster usage statistics.MLEDashboard: A dashboard visualizing resource usage & experiment protocol.To get started I recommend checking out thecolab notebookand anexample workflow.MLEProtocol: Keeping Track of Your Experiments 📝frommle_monitorimportMLEProtocol# Load protocol database or create new one -> print summaryprotocol_db=MLEProtocol("mle_protocol.db",verbose=False)protocol_db.summary(tail=10,verbose=True)# Draft data to store in protocol & add it to the protocolmeta_data={"purpose":"Grid search",# Purpose of experiment"project_name":"MNIST",# Project name of experiment"experiment_type":"hyperparameter-search",# Type of experiment"experiment_dir":"experiments/logs",# Experiment directory"num_total_jobs":10,# Number of total jobs to run...}new_experiment_id=protocol_db.add(meta_data)# ... train your 10 (pseudo) networks/complete respective jobsforiinrange(10):protocol_db.update_progress_bar(new_experiment_id)# Wrap up an experiment (store completion time, etc.)protocol_db.complete(new_experiment_id)The meta data can contain the following keys:Search TypeDescriptionDefaultpurposePurpose of experiment'None provided'project_nameProject name of experiment'default'exec_resourceResource jobs are run on'local'experiment_dirExperiment log storage directory'experiments'experiment_typeType of experiment to run'single'base_fnameMain code script to execute'main.py'config_fnameConfig file path of experiment'base_config.yaml'num_seedsNumber of evaluations seeds1num_total_jobsNumber of total jobs to run1num_job_batchesNumber of jobs in single batch1num_jobs_per_batchNumber of sequential job batches1time_per_jobExpected duration: days-hours-minutes'00:01:00'num_cpusNumber of CPUs used in job1num_gpusNumber of GPUs used in job0Additionally you can synchronize the protocol with a Google Cloud Storage (GCS) bucket by providingcloud_settings. In this case also the results stored inexperiment_dirwill be uploaded to the GCS bucket, when you callprotocol.complete().# Define GCS settings - requires 'GOOGLE_APPLICATION_CREDENTIALS' env var.cloud_settings={"project_name":"mle-toolbox",# GCP project name"bucket_name":"mle-protocol",# GCS bucket name"use_protocol_sync":True,# Whether to sync the protocol to GCS"use_results_storage":True,# Whether to sync experiment_dir to GCS}protocol_db=MLEProtocol("mle_protocol.db",cloud_settings,verbose=True)TheMLEResource: Keeping Track of Your Resources 📉On Your Local Machinefrommle_monitorimportMLEResource# Instantiate local resource and get usage dataresource=MLEResource(resource_name="local")resource_data=resource.monitor()On a Slurm Clusterresource=MLEResource(resource_name="slurm-cluster",monitor_config={"partitions":["<partition-1>","<partition-2>"]},)On a Grid Engine Clusterresource=MLEResource(resource_name="sge-cluster",monitor_config={"queues":["<queue-1>","<queue-2>"]})TheMLEDashboard: Dashboard Visualization 🎞️frommle_monitorimportMLEDashboard# Instantiate dashboard with protocol and resourcedashboard=MLEDashboard(protocol,resource)# Get a static snapshot of the protocol & resource utilisation printed in consoledashboard.snapshot()# Run monitoring in while loop - dashboarddashboard.live()Installation ⏳A PyPI installation is available via:pip install mle-monitorIf you want to get the most recent commit, please install directly from the repository:pip install git+https://github.com/mle-infrastructure/mle-monitor.git@mainCiting the MLE-Infrastructure ✏️If you usemle-monitorin your research, please cite it as follows:@software{mle_infrastructure2021github,
  author = {Robert Tjarko Lange},
  title = {{MLE-Infrastructure}: A Set of Lightweight Tools for Distributed Machine Learning Experimentation},
  url = {http://github.com/mle-infrastructure},
  year = {2021},
}Development 👷You can run the test suite viapython -m pytest -vv tests/. If you find a bug or are missing your favourite feature, feel free to create an issue and/or startcontributing🤗.