Package: trains-agent
Allegro Trains AgentDeep Learning DevOps For Everyone - Now supporting all platforms (Linux, macOS, and Windows)"All the Deep-Learning DevOps your research needs, and then some... Because ain't nobody got time for that"Help improve Trains by filling our 2-minuser surveyTrains Agent is an AI experiment cluster solution.It is a zero configuration fire-and-forget execution agent, which combined with trains-server provides a full AI cluster solution.Full AutoML in 5 stepsInstall theTrains Server(or use ouropen server)pip install trains-agent(installthe Trains Agent on any GPU machine: on-premises / cloud / ...)AddTrainsto your code with just 2 lines & run it once (on your machine / laptop)Change theparametersin the UI & schedule forexecution(or automate with anAutoML pipeline):chart_with_downwards_trend: :chart_with_upwards_trend: :eyes:  :beer:Using the Trains Agent, you can now set up a dynamic cluster with *epsilon DevOps*epsilon - Because we are scientists :triangular_ruler: and nothing is really zero work(Experience Trains live athttps://demoapp.trains.allegro.ai)Simple, Flexible Experiment OrchestrationThe Trains Agent was built to address the DL/ML R&D DevOps needs:Easily add & remove machines from the clusterReuse machines without the need for any dedicated containers or imagesCombine GPU resources across any cloud and on-premNo need for yaml/json/template configuration of any kindUser friendly UIManageable resource allocation that can be used by researchers and engineersFlexible and controllable scheduler with priority supportAutomatic instance spinning in the cloud(coming soon)But ... K8S?We think Kubernetes is awesome.Combined with KubeFlow it is a robust solution for production-grade DevOps.We've observed, however, that it can be a bit of an overkill as an R&D DL/ML solution.
If you are considering K8S for your research, also consider that you will soon be managinghundredsof containers...In our experience, handling and building the environments, having to package every experiment in a docker, managing those hundreds (or more) containers and building pipelines on top of it all, is very complicated (also, it’s usually out of scope for the research team, and overwhelming even for the DevOps team).We feel there has to be a better way, that can be just as powerful for R&D and at the same time allow integration with K8Swhen the need arises.(If you already have a K8S cluster for AI, detailed instructions on how to integrate Trains into your K8S cluster areherewith includedhelm chart)Using the Trains AgentFull scale HPC with a click of a buttonThe Trains Agent is a job scheduler that listens on job queue(s), pulls jobs, sets the job environments, executes the job and monitors its progress.Any 'Draft' experiment can be scheduled for execution by a Trains agent.A previously run experiment can be put into 'Draft' state by either of two methods:Using the'Reset'action from the experiment right-click context menu in the
Trains UI - This will clear any results and artifacts the previous run had created.Using the'Clone'action from the experiment right-click context menu in the
Trains UI - This will create a new 'Draft' experiment with the same configuration as the original experiment.An experiment is scheduled for execution using the'Enqueue'action from the experiment
right-click context menu in the Trains UI and selecting the execution queue.Seecreating an experiment and enqueuing it for execution.Once an experiment is enqueued, it will be picked up and executed by a Trains agent monitoring this queue.The Trains UI Workers & Queues page provides ongoing execution information:Workers Tab: Monitor you clusterReview available resourcesMonitor machines statistics (CPU / GPU / Disk / Network)Queues Tab:Control the scheduling order of jobsCancel or abort job executionMove jobs between execution queuesWhat The Trains Agent Actually DoesThe Trains Agent executes experiments using the following process:Create a new virtual environment (or launch the selected docker image)Clone the code into the virtual-environment (or inside the docker)Install python packages based on the package requirements listed for the experimentSpecial note for PyTorch: The Trains Agent will automatically select the
torch packages based on the CUDA_VERSION environment variable of the machineExecute the code, while monitoring the processLog all stdout/stderr in the Trains UI, including the cloning and installation process, for easy debuggingMonitor the execution and allow you to manually abort the job using the Trains UI (or, in the unfortunate case of a code crash, catch the error and signal the experiment has failed)System Design & Flow+-----------------+
                                                                              |  GPU  Machine   |
Development Machine                                                           |                 |
+------------------------+                                                    | +-------------+ |
|    Data Scientist's    |                            +--------------+        | |Trains Agent | |
|      DL/ML Code        |                            |    WEB UI    |        | |             | |
|                        |                            |              |        | | +---------+ | |
|                        |                            |              |        | | |  DL/ML  | | |
|                        |                            +--------------+        | | |  Code   | | |
|                        |       User Clones Exp #1  / . . . . . . . /        | | |         | | |
| +-------------------+  |           into Exp #2    / . . . . . . . /         | | +---------+ | |
| |      Trains       |  |         +---------------/-_____________-/          | |             | |
| +---------+---------+  |         |                                          | |      ^      | |
+-----------|------------+         |                                          | +------|------+ |
            |                      |                                          +--------|--------+
 Auto-Magically                    |                                                   |
 Creates Exp #1                    |                                      The Trains Agent
             \          User Change Hyper-Parameters                      Pulls Exp #2, setup the
             |                     |                                      environment & clone code.
             |                     |                                      Start execution with the
+------------|------------+        |            +--------------------+    new set of Hyper-Parameters.
|  +---------v---------+  |        |            |   Trains Server    |                 |
|  | Experiment #1     |  |        |            |                    |                 |
|  +-------------------+  |        |            |  Execution Queue   |                 |
|            ||           |        |            |                    |                 |
|  +-------------------+<----------+            |                    |                 |
|  |                   |  |                     |                    |                 |
|  | Experiment #2     |  |                     |                    |                 |
|  +-------------------<------------\           |                    |                 |
|                         |          ------------->---------------+  |                 |
|                         |  User Send Exp #2   | |Execute Exp #2 +--------------------+
|                         |  For Execution      | +---------------+  |
|     Trains Server       |                     |                    |
+-------------------------+                     +--------------------+Installing the Trains Agentpipinstalltrains-agentTrains Agent Usage ExamplesFull Interface and capabilities are available withtrains-agent--help
trains-agentdaemon--helpConfiguring the Trains Agenttrains-agentinitNote: The Trains Agent uses a cache folder to cache pip packages, apt packages and cloned repositories. The default Trains Agent cache folder is~/.trainsSee full details in your configuration file at~/trains.confNote: TheTrains agentextends theTrainsconfiguration file~/trains.confThey are designed to share the same configuration file, see examplehereRunning the Trains AgentFor debug and experimentation, start the Trains agent inforegroundmode, where all the output is printed to screentrains-agentdaemon--queuedefault--foregroundFor actual service mode, all the stdout will be stored automatically into a temporary file (no need to pipe)
Notice: with--detachedflag, thetrains-agentwill be running in the backgroundtrains-agentdaemon--detached--queuedefaultGPU allocation is controlled via the standard OS environmentNVIDIA_VISIBLE_DEVICESor--gpusflag (or disabled with--cpu-only).If no flag is set, andNVIDIA_VISIBLE_DEVICESvariable doesn't exist, all GPU's will be allocated for thetrains-agentIf--cpu-onlyflag is set, orNVIDIA_VISIBLE_DEVICESis an empty string (""), no gpu will be allocated for thetrains-agentExample: spin two agents, one per gpu on the same machine:
Notice: with--detachedflag, thetrains-agentwill be running in the backgroundtrains-agentdaemon--detached--gpus0--queuedefault
trains-agentdaemon--detached--gpus1--queuedefaultExample: spin two agents, pulling from dedicateddual_gpuqueue, two gpu's per agenttrains-agentdaemon--detached--gpus0,1--queuedual_gpu
trains-agentdaemon--detached--gpus2,3--queuedual_gpuStarting the Trains Agent in docker modeFor debug and experimentation, start the Trains agent inforegroundmode, where all the output is printed to screentrains-agentdaemon--queuedefault--docker--foregroundFor actual service mode, all the stdout will be stored automatically into a file (no need to pipe)
Notice: with--detachedflag, thetrains-agentwill be running in the backgroundtrains-agentdaemon--detached--queuedefault--dockerExample: spin two agents, one per gpu on the same machine, with default nvidia/cuda docker:trains-agentdaemon--detached--gpus0--queuedefault--dockernvidia/cuda
trains-agentdaemon--detached--gpus1--queuedefault--dockernvidia/cudaExample: spin two agents, pulling from dedicateddual_gpuqueue, two gpu's per agent, with default nvidia/cuda docker:trains-agentdaemon--detached--gpus0,1--queuedual_gpu--dockernvidia/cuda
trains-agentdaemon--detached--gpus2,3--queuedual_gpu--dockernvidia/cudaStarting the Trains Agent - Priority QueuesPriority Queues are also supported, example use case:High priority queue:important_jobsLow priority queue:defaulttrains-agentdaemon--queueimportant_jobsdefaultTheTrains Agentwill first try to pull jobs from theimportant_jobsqueue, only then it will fetch a job from thedefaultqueue.Adding queues, managing job order within a queue and moving jobs between queues, is available using the Web UI, see example on ouropen serverStopping the Trains AgentTo stop aTrains Agentrunning in the background, run the same command line used to start the agent with--stopappended.For example, to stop the first of the above shown same machine, single gpu agents:trains-agentdaemon--detached--gpus0--queuedefault--dockernvidia/cuda--stopHow do I create an experiment on the Trains Server?IntegrateTrainswith your codeExecute the code on your machine (Manually / PyCharm / Jupyter Notebook)As your code is running,Trainscreates an experiment logging all the necessary execution information:Git repository link and commit ID (or an entire jupyter notebook)Git diff (we’re not saying you never commit and push, but still...)Python packages used by your code (including specific versions used)Hyper-ParametersInput ArtifactsYou now have a 'template' of your experiment with everything required for automated executionIn the Trains UI, Right click on the experiment and select 'clone'. A copy of your experiment will be created.You now have a new draft experiment cloned from your original experiment, feel free to edit itChange the Hyper-ParametersSwitch to the latest code base of the repositoryUpdate package versionsSelect a specific docker image to run in (see docker execution mode section)Or simply change nothing to run the same experiment again...Schedule the newly created experiment for execution: Right-click the experiment and select 'enqueue'Trains-Agent Services ModeTrains-Agent Services is a special mode of Trains-Agent that provides the ability to launch long-lasting jobs
that previously had to be executed on local / dedicated machines. It allows a single agent to
launch multiple dockers (Tasks) for different use cases. To name a few use cases, auto-scaler service (spinning instances
when the need arises and the budget allows), Controllers (Implementing pipelines and more sophisticated DevOps logic),
Optimizer (such as Hyper-parameter Optimization or sweeping), and Application (such as interactive Bokeh apps for
increased data transparency)Trains-Agent Services mode will spinanytask enqueued into the specified queue.
Every task launched by Trains-Agent Services will be registered as a new node in the system,
providing tracking and transparency capabilities.
Currently trains-agent in services-mode supports cpu only configuration. Trains-agent services mode can be launched alongside GPU agents.trains-agentdaemon--services-mode--detached--queueservices--create-queue--dockerubuntu:18.04--cpu-onlyNote: It is the user's responsibility to make sure the proper tasks are pushed into the specified queue.AutoML and Orchestration PipelinesThe Trains Agent can also be used to implement AutoML orchestration and Experiment Pipelines in conjunction with the Trains package.Sample AutoML & Orchestration examples can be found in the Trainsexample/automationfolder.AutoML examplesToy Keras training experimentIn order to create an experiment-template in the system, this code must be executed once manuallyRandom Search over the above Keras experiment-templateThis example will create multiple copies of the Keras experiment-template, with different hyper-parameter combinationsExperiment Pipeline examplesFirst step experimentThis example will "process data", and once done, will launch a copy of the 'second step' experiment-templateSecond step experimentIn order to create an experiment-template in the system, this code must be executed once manuallyLicenseApache License, Version 2.0 (see theLICENSEfor more information)