Package: clearml-agent
ClearML Agent - ML-Ops made easyML-Ops scheduler & orchestration solution supporting Linux, macOS and WindowsClearML-AgentFormerly known as Trains AgentRun jobs (experiments) on any local or cloud based resourceImplement optimized resource utilization policiesDeploy execution environments with either virtualenv or fully docker containerized with zero effortLaunch-and-Forget service containersCloud autoscalingCustomizable cleanupAdvancedpipeline building and executionIt is a zero configuration fire-and-forget execution agent, providing a full ML/DL cluster solution.Full Automation in 5 stepsClearML Serverself-hostedorfree tier hostingpip install clearml-agent(installthe ClearML Agent on any GPU machine:
on-premises / cloud / ...)Create ajobor
addClearMLto your code with just 2 lines of codeChange theparametersin the UI & schedule forexecution(or
automate with anAutoML pipeline):chart_with_downwards_trend: :chart_with_upwards_trend: :eyes:  :beer:"All the Deep/Machine-Learning DevOps your research needs, and then some... Because ain't nobody got time for that"Try ClearML nowSelf HostedorFree tier HostingSimple, Flexible Experiment OrchestrationThe ClearML Agent was built to address the DL/ML R&D DevOps needs:Easily add & remove machines from the clusterReuse machines without the need for any dedicated containers or imagesCombine GPU resources across any cloud and on-premNo need for yaml / json / template configuration of any kindUser friendly UIManageable resource allocation that can be used by researchers and engineersFlexible and controllable scheduler with priority supportAutomatic instance spinning in the cloudUsing the ClearML Agent, you can now set up a dynamic cluster with *epsilon DevOps*epsilon - Because we are :triangular_ruler: and nothing is really zero workKubernetes Integration (Optional)We think Kubernetes is awesome, but it should be a choice. We designedclearml-agentso you can run bare-metal or
inside a pod with any mix that fits your environment.Find Dockerfiles in thedockerdir and a helm Chart inhttps://github.com/allegroai/clearml-helm-chartsBenefits of integrating existing K8s with ClearML-AgentClearML-Agent adds the missing scheduling capabilities to K8sAllowing for more flexible automation from codeA programmatic interface for easier learning curve (and debugging)Seamless integration with ML/DL experiment managerWeb UI for customization, scheduling & prioritization of jobsTwo K8s integration flavoursSpin ClearML-Agent as a long-lasting service pod:Useclearml-agentdocker imagemap docker socket into the pod (soon replaced bypodman)Allow the clearml-agent to manage sibling dockersBenefits: full use of the ClearML scheduling, no need to worry about wrong container images / lost pods etc.Downside: sibling containersKubernetes Glue, map ClearML jobs directly to K8s jobs:Run theclearml-k8s glueon
a K8s cpu nodeThe clearml-k8s glue pulls jobs from the ClearML job execution queue and prepares a K8s job (based on provided
yaml template)Inside the pod itself the clearml-agent will install the job (experiment) environment and spin and monitor the
experiment's processBenefits: Kubernetes full view of all running jobs in the systemDownside: No real scheduling (k8s scheduler), no docker image verification (post-mortem only)Using the ClearML AgentFull scale HPC with a click of a buttonThe ClearML Agent is a job scheduler that listens on job queue(s), pulls jobs, sets the job environments, executes the
job and monitors its progress.Any 'Draft' experiment can be scheduled for execution by a ClearML agent.A previously run experiment can be put into 'Draft' state by either of two methods:Using the'Reset'action from the experiment right-click context menu in the ClearML UI - This will clear any
results and artifacts the previous run had created.Using the'Clone'action from the experiment right-click context menu in the ClearML UI - This will create a new
'Draft' experiment with the same configuration as the original experiment.An experiment is scheduled for execution using the'Enqueue'action from the experiment right-click context menu in
the ClearML UI and selecting the execution queue.Seecreating an experiment and enqueuing it for execution.Once an experiment is enqueued, it will be picked up and executed by a ClearML Agent monitoring this queue.The ClearML UI Workers & Queues page provides ongoing execution information:Workers Tab: Monitor you clusterReview available resourcesMonitor machines statistics (CPU / GPU / Disk / Network)Queues Tab:Control the scheduling order of jobsCancel or abort job executionMove jobs between execution queuesWhat The ClearML Agent Actually DoesThe ClearML Agent executes experiments using the following process:Create a new virtual environment (or launch the selected docker image)Clone the code into the virtual-environment (or inside the docker)Install python packages based on the package requirements listed for the experimentSpecial note for PyTorch: The ClearML Agent will automatically select the torch packages based on the CUDA_VERSION
environment variable of the machineExecute the code, while monitoring the processLog all stdout/stderr in the ClearML UI, including the cloning and installation process, for easy debuggingMonitor the execution and allow you to manually abort the job using the ClearML UI (or, in the unfortunate case of a
code crash, catch the error and signal the experiment has failed)System Design & FlowInstalling the ClearML Agentpipinstallclearml-agentClearML Agent Usage ExamplesFull Interface and capabilities are available withclearml-agent--help
clearml-agentdaemon--helpConfiguring the ClearML Agentclearml-agentinitNote: The ClearML Agent uses a cache folder to cache pip packages, apt packages and cloned repositories. The default
ClearML Agent cache folder is~/.clearml.See full details in your configuration file at~/clearml.conf.Note: TheClearML Agentextends theClearMLconfiguration file~/clearml.conf.
They are designed to share the same configuration file, see examplehereRunning the ClearML AgentFor debug and experimentation, start the ClearML agent inforegroundmode, where all the output is printed to screen:clearml-agentdaemon--queuedefault--foregroundFor actual service mode, all the stdout will be stored automatically into a temporary file (no need to pipe).
Notice: with--detachedflag, theclearml-agentwill be running in the backgroundclearml-agentdaemon--detached--queuedefaultGPU allocation is controlled via the standard OS environmentNVIDIA_VISIBLE_DEVICESor--gpusflag (or disabled
with--cpu-only).If no flag is set, andNVIDIA_VISIBLE_DEVICESvariable doesn't exist, all GPUs will be allocated for
theclearml-agent.If--cpu-onlyflag is set, orNVIDIA_VISIBLE_DEVICES="none", no gpu will be allocated for
theclearml-agent.Example: spin two agents, one per GPU on the same machine:Notice: with--detachedflag, theclearml-agentwill run in the backgroundclearml-agentdaemon--detached--gpus0--queuedefault
clearml-agentdaemon--detached--gpus1--queuedefaultExample: spin two agents, pulling from dedicateddual_gpuqueue, two GPUs per agentclearml-agentdaemon--detached--gpus0,1--queuedual_gpu
clearml-agentdaemon--detached--gpus2,3--queuedual_gpuStarting the ClearML Agent in docker modeFor debug and experimentation, start the ClearML agent inforegroundmode, where all the output is printed to screenclearml-agentdaemon--queuedefault--docker--foregroundFor actual service mode, all the stdout will be stored automatically into a file (no need to pipe).
Notice: with--detachedflag, theclearml-agentwill run in the backgroundclearml-agentdaemon--detached--queuedefault--dockerExample: spin two agents, one per gpu on the same machine, with defaultnvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04docker:clearml-agentdaemon--detached--gpus0--queuedefault--dockernvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04
clearml-agentdaemon--detached--gpus1--queuedefault--dockernvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04Example: spin two agents, pulling from dedicateddual_gpuqueue, two GPUs per agent, with defaultnvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04docker:clearml-agentdaemon--detached--gpus0,1--queuedual_gpu--dockernvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04
clearml-agentdaemon--detached--gpus2,3--queuedual_gpu--dockernvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04Starting the ClearML Agent - Priority QueuesPriority Queues are also supported, example use case:High priority queue:important_jobs, low priority queue:defaultclearml-agentdaemon--queueimportant_jobsdefaultTheClearML Agentwill first try to pull jobs from theimportant_jobsqueue, and only if it is empty, the agent
will try to pull from thedefaultqueue.Adding queues, managing job order within a queue, and moving jobs between queues, is available using the Web UI, see
example on ourfree serverStopping the ClearML AgentTo stop aClearML Agentrunning in the background, run the same command line used to start the agent with--stopappended. For example, to stop the first of the above shown same machine, single gpu agents:clearml-agentdaemon--detached--gpus0--queuedefault--dockernvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04--stopHow do I create an experiment on the ClearML Server?IntegrateClearMLwith your codeExecute the code on your machine (Manually / PyCharm / Jupyter Notebook)As your code is running,ClearMLcreates an experiment logging all the necessary execution information:Git repository link and commit ID (or an entire jupyter notebook)Git diff (we’re not saying you never commit and push, but still...)Python packages used by your code (including specific versions used)HyperparametersInput artifactsYou now have a 'template' of your experiment with everything required for automated executionIn the ClearML UI, right-click on the experiment and select 'clone'. A copy of your experiment will be created.You now have a new draft experiment cloned from your original experiment, feel free to edit itChange the hyperparametersSwitch to the latest code base of the repositoryUpdate package versionsSelect a specific docker image to run in (see docker execution mode section)Or simply change nothing to run the same experiment again...Schedule the newly created experiment for execution: right-click the experiment and select 'enqueue'ClearML-Agent Services ModeClearML-Agent Services is a special mode of ClearML-Agent that provides the ability to launch long-lasting jobs that
previously had to be executed on local / dedicated machines. It allows a single agent to launch multiple dockers (Tasks)
for different use cases:Auto-scaler service (spinning instances when the need arises and the budget allows)Controllers (Implementing pipelines and more sophisticated DevOps logic)Optimizer (such as Hyperparameter Optimization or sweeping)Application (such as interactive Bokeh apps for increased data transparency)ClearML-Agent Services mode will spinanytask enqueued into the specified queue. Every task launched by
ClearML-Agent Services will be registered as a new node in the system, providing tracking and transparency capabilities.
Currently, clearml-agent in services-mode supports CPU only configuration. ClearML-Agent services mode can be launched
alongside GPU agents.clearml-agentdaemon--services-mode--detached--queueservices--create-queue--dockerubuntu:18.04--cpu-onlyNote: It is the user's responsibility to make sure the proper tasks are pushed into the specified queue.AutoML and Orchestration PipelinesThe ClearML Agent can also be used to implement AutoML orchestration and Experiment Pipelines in conjunction with the
ClearML package.Sample AutoML & Orchestration examples can be found in the
ClearMLexample/automationfolder.AutoML examples:Toy Keras training experimentIn order to create an experiment-template in the system, this code must be executed once manuallyRandom Search over the above Keras experiment-templateThis example will create multiple copies of the Keras experiment-template, with different hyperparameter
combinationsExperiment Pipeline examples:First step experimentThis example will "process data", and once done, will launch a copy of the 'second step' experiment-templateSecond step experimentIn order to create an experiment-template in the system, this code must be executed once manuallyLicenseApache License, Version 2.0 (see theLICENSEfor more information)