Package: caliban
CalibanCaliban is a tool that helps researchers launch and track their numerical
experiments in an isolated, reproducible computing environment. It was developed
by machine learning researchers and engineers, and makes it easy to go from a
simple prototype running on a workstation to thousands of experimental jobs
running on Cloud.With Caliban, you can:Develop your experimental code locally and test it inside an isolated (Docker)
environmentEasily sweep over experimental parametersSubmit your experiments as Cloud jobs, where they will run in the same
isolated environmentControl and keep track of jobsQuickstartInstall Docker, make sure it's running, then install Caliban (you'll needPython >= 3.6):pipinstallcalibanTrain a simple deep learning model on your local machine:gitclonehttps://github.com/google/caliban.git&&cdcaliban/tutorials/basic
calibanrun--nogpumnist.pySweep over learning rates to find the best one (flags are specified in JSON format):echo'{"learning_rate": [0.01, 0.001, 0.0001]}'|calibanrun--experiment_configstdin--nogpumnist.pyNext:See how to submit the experiment to Cloud and use other Caliban features in"Getting Started with Caliban"SeeInstallationfor detailed installation instructionsRead theCommand Overviewfor info on Caliban commands.Full documentation for Caliban lives atRead The Docs.Dramatic Interlude“Be not afeard; the isle is full of noises,Sounds, and sweet airs, that give delight and hurt not.Sometimes a thousand twangling instrumentsWill hum about mine ears; and sometime voices,That, if I then had waked after long sleep,Will make me sleep again: and then, in dreaming,The clouds methought would open, and show richesReady to drop upon me; that, when I waked,I cried to dream again.”--Shakespeare, The TempestInstallation and PrerequisitesCaliban's prequisites areDockerandPython >= 3.6.Make sure your Python is up to date:$python--version
Python3.6.9# should be >=3.6.0If not, visit"Installing Python 3.6"before proceeding.Next, install Caliban viapip:pipinstall-Ucalibancheck if your installation worked by navigating to an empty folder and runningcaliban --help. You should see the usage dialogue:$caliban--help
usage:caliban[-h][--helpfull][--version]{shell,notebook,build,run,cloud,cluster,status,stop,resubmit}...DockerCaliban executes your code inside a "container", managed byDocker. To get Docker:On MacOS, follow the installation instructions atDocker
Desktopand
start the newly-installed Docker Desktop application.On Linux, visit theDocker installation
instructions.
(It's important that you configuresudo-less
Dockerand start Docker running on your machine.)Make sure Docker is correctly installed, configured and running by executing the
following command:dockerrunhello-worldYou should see output that looks like this:...
Hello from Docker!
This message shows that your installation appears to be working correctly.
...Python 3.6Make sure your Python version is up to date:$python--version
Python3.6.9# should be >=3.6.0If you need to upgrade:On MacOS, install the latest Python version frompython.org(direct
link).On Linux, runsudo apt-get update && sudo apt-get install python3.7.Cloud Submission and GPUsCaliban'sRead the Docsdocumentation has
instructions on:Installing thenvidia-docker2runtime,
so you can use Caliban to run jobs that use your Linux machine's GPU.Setting up a Google Cloud
accountso you can submit your code to Google'sCloud AI
Platformwithcaliban cloud.Getting Started with CalibanIn this section we will use Caliban to train an image classification network
(implemented inTensorFlow). We
will:Train a neural network on the local machineIncrease the model's accuracy by changing thelearning
ratewith a command-line flagSweep across a range of learning rates with Caliban'sexperiment
broadcastingfeatureTrain the model in the Cloud on Google'sAI
PlatformDevelop code interactively usingcaliban shellin the exact same
environment.Preparing your ProjectCreate an empty directory and usecurlto download apython
scriptthat trains a basic neural network.mkdir demo && cd demo
curl --output mnist.py https://raw.githubusercontent.com/google/caliban/master/tutorials/basic/mnist.pyCreate a file calledrequirements.txtto declaretensorflow-cpuas a dependency:echo"tensorflow-cpu">requirements.txtCaliban will automatically make any entry inrequirements.txtavailable when
you run your code. See"Declaring
Requirements"for more information.Training the NetworkRun this command to train your first ML model:calibanrun--nogpumnist.pyYou should see a stream of output ending in this:Training model with learning rate=0.1 for 3 epochs.
Epoch 1/3
1875/1875 - 3s - loss: 2.0989 - accuracy: 0.2506
Epoch 2/3
1875/1875 - 3s - loss: 1.9222 - accuracy: 0.2273
Epoch 3/3
1875/1875 - 3s - loss: 2.0777 - accuracy: 0.1938
Model performance:
313/313 - 0s - loss: 2.0973 - accuracy: 0.1858Your model was able to recognize digits from theMNISTdataset with 18.58%
accuracy. Can we do better?Improving the ModelThe default learning rate is0.1. Run the code again with a smaller learning
rate by passing a command-line flag, separated from your original command by--:$calibanrun--nogpumnist.py----learning_rate0.01<<elided>>Training modelwithlearningrate=0.01for3epochs.
Epoch1/31875/1875-4s-loss:0.2676-accuracy:0.9221
Epoch2/31875/1875-4s-loss:0.1863-accuracy:0.9506
Epoch3/31875/1875-4s-loss:0.1567-accuracy:0.9585
Modelperformance:313/313-0s-loss:0.1410-accuracy:0.964296% accuracy! Much better! Can we do better still?Experiment BroadcastingCaliban'sexperiment
broadcastingfeature will allow us to run many jobs with different sets of arguments.Create a file calledexperiment.jsonwith aJSONdictionary of the format{"flag_name": ["list", "of", "values"]}:echo'{"learning_rate": [0.01, 0.001, 0.0001]}'>experiment.jsonPass the config with--experiment_configand run again:calibanrun--experiment_configexperiment.json--nogpumnist.pyYou should see accuracies of roughly0.9493,0.9723and0.9537. Looks like0.001is a nice choice.Submitting to Cloud AI PlatformNow it's time to submit the job toCloud AI
Platform.(NOTE: This section requires a Google Cloud account. You can create a free
account with $300 of credit to get started. Follow Caliban's"Getting Started
with Google
Cloud"documentation, then come back here to proceed.)Submit the job to AI Platform by changing the wordruntocloud:calibancloud--nogpumnist.py----learning_rate0.01You should see output like this:I061519:57:43.3541724563361216core.py:161]Job1-jobId:caliban_totoro_1,image:gcr.io/research-3141/974a776e6037:latest
I061519:57:43.3547124563361216core.py:161]Job1-Accelerator:{'count':0,'type':'ACCELERATOR_TYPE_UNSPECIFIED'},machine:'n1-highcpu-32',region:'us-central1'I061519:57:43.3550824563361216core.py:161]Job1-Experimentarguments:['--learning_rate','0.01']I061519:57:43.3554404563361216core.py:161]Job1-labels:{'gpu_enabled':'false','tpu_enabled':'false','job_name':'caliban_totoro','learning_rate':'0_01'}I061519:57:43.3566214563361216core.py:324]Submittingrequest!
I061519:57:45.0783824563361216core.py:97]Requestforjob'caliban_totoro_20200615_195743_1'succeeded!
I061519:57:45.0789894563361216core.py:98]JobURL:https://console.cloud.google.com/ai-platform/jobs/caliban_totoro_20200615_195743_1?projectId=totoro-project
I061519:57:45.0795244563361216core.py:100]StreaminglogCLIcommand:$gcloudai-platformjobsstream-logscaliban_totoro_20200615_195743_1
Submittingcaliban_totoro_1:100%|####################################################################################################################################################################################| 1/1 [00:02<00:00,  2.65s/requests]I061519:57:45.4056004563361216core.py:673]I061519:57:45.4058194563361216core.py:676]Visithttps://console.cloud.google.com/ai-platform/jobs/?projectId=research-3141toseethestatusofalljobs.
I061519:57:45.4059594563361216core.py:677]This output means that Caliban has:built a Docker container with all of your codePushed that container up to Google Cloud'sContainer
RegistrySubmitted the job toAI Platform.You can now visit the link in the output that looks like:https://console.cloud.google.com/ai-platform/jobs/caliban_totoro_20200615_195743_1?projectId=totoro-projectto see all of your job's logs.Why do I need Cloud?With Google Cloud, you can use on-demandGPUsandTPUsand
train models on large datasets at very high speeds. You can also customize themachine
typethat AI Platform uses to run your job. You might need high memory or more CPU,
for example.See Caliban's"Customizing Machines and
GPUs"for more
information.Interactive Development withcaliban shellcaliban shelllets
you develop code interactively inside of the exact same environment that your
code will have available, locally duringcaliban runor in the Cloud withcaliban cloud.Run the following command to activate the shell:calibanshell--nogpuYou should see Caliban's terminal:I0611 12:33:17.551121 4500135360 docker.py:911] Running command: docker run --ipc host -w /usr/app -u 735994:89939 -v /Users/totoro/code/example:/usr/app -it --entrypoint /bin/bash -v /Users/totoro:/home/totoro ab8a7d7db868
   _________    __    ________  ___    _   __  __  __
  / ____/   |  / /   /  _/ __ )/   |  / | / /  \ \ \ \
 / /   / /| | / /    / // __  / /| | /  |/ /    \ \ \ \
/ /___/ ___ |/ /____/ // /_/ / ___ |/ /|  /     / / / /
\____/_/  |_/_____/___/_____/_/  |_/_/ |_/     /_/ /_/

You are running caliban shell as user with ID 735994 and group 89939,
which should map to the ID and group for your user on the Docker host. Great!

[totoro@6a9b28990757 /usr/app]$You're now living in an isolatedDocker
containerwith yourtensorflow-cpudependency available (and any othersyou've
declared).Run thepythoncommand and check thattensorflowis installed:$python
Python3.6.9(default,Nov72019,10:44:02)[GCC8.3.0]onlinux
Type"help","copyright","credits"or"license"formoreinformation.
>>>importtensorflowastf
>>>tf.__version__'2.2.0'Your home directory and the folder where you ran the command are both mounted
into this isolated environment, so any changes you make to either of those
directories will be reflected immediately.Any code you add to the current folder and edit on your computer will be
available in this special Caliban shell. Run the example from before like this:python mnist.py --learning_rate 0.01If your code runs incaliban shell, you can be almost certain that your code
will execute in a Cloud environment, with potentially many GPUs attached and
much larger machines available.What next?Read theOverviewfor more information on Caliban's subcommands,
then head over toCaliban's documentation
siteand check out the links on the
sidebar.If you find anything confusing, please feel free tocreate an
issueon ourGithub Issues
page, and we'll get you sorted out.Command OverviewCaliban provides seven subcommands that you run inside some project directory on
your machine:caliban shellgenerates a Docker image containing any dependencies you've declared in arequirements.txtand/orsetup.pyin the directory and opens an interactive
shell in that directory. Thecaliban shellenvironment is ~identical to the
environment that will be available to your code when you submit it to AI
Platform; the difference is that your current directory is live-mounted into
the container, so you can develop interactively.caliban notebookstarts a Jupyter notebook or lab instance inside of a Docker image containing
your dependencies; the guarantee about an environment identical to AI Platform
applies here as well.caliban runpackages your directory's code into the Docker image and executes it locally
usingdocker run. If you have a GPU, the instance will attach to it by
default - no need to install the CUDA toolkit. The Docker environment takes
care of all that. This environment is truly identical to the AI Platform
environment. The Docker image that runs locally is the same image that will
run in AI Platform.caliban cloudallows you tosubmit jobs to AI
Platformthat will run inside the same Docker image you used withcaliban run. You
can submit hundreds of jobs at once. Any machine type, GPU count, and GPU type
combination you specify will be validated client side, so you'll see an
immediate error with suggestions, rather than having to debug by submitting
jobs over and over.caliban buildbuilds
the Docker image used incaliban cloudandcaliban runwithout actually
running the container or submitting any code.caliban clustercreates GKE clusters and submits jobs to GKE clusters.caliban statusdisplays information about all jobs submitted by Caliban, and makes it easy to
interact with large groups of experiments. Usecaliban statuswhen you need
to cancel pending jobs, or re-build a container and resubmit a batch of
experiments after fixing a bug.DisclaimerThis is a research project, not an official Google product. Expect bugs and
sharp edges. Please help by trying out Caliban,reporting
bugs, and letting us know what you
think!Get Involved + Get SupportPull requests and bug reports are always welcome! Check out ourContributor's
Guidefor information on how to get started contributing to
Caliban.The TL;DR; is:send us a pull request,iterate on the feedback + discussion, andget a +1 from aCommitterin order to get your PR accepted.Issues should be reported on theGitHub issue
tracker.If you want to discuss an idea for a new feature or ask us a question,
discussion occurs primarily in the body ofGithub
Issues, though the project is growing
large enough that we may start a Gitter channel soon.The current list of active committers (who can +1 a pull request) can be found
here:COMMITTERS.mdA list of contributors to the project can be found at the project'sContributorspage.Citing CalibanIf Caliban helps you in your research, please consider citing Caliban's
associated academic paper:@article{Ritchie2020,
  doi = {10.21105/joss.02403},
  url = {https://doi.org/10.21105/joss.02403},
  year = {2020},
  publisher = {The Open Journal},
  volume = {5},
  number = {53},
  pages = {2403},
  author = {Sam Ritchie and Ambrose Slone and Vinay Ramasesh},
  title = {Caliban: Docker-based job manager for reproducible workflows},
  journal = {Journal of Open Source Software}
}LicenseCopyright 2020 Google LLC.Licensed under theApache License, Version 2.0.