Multi-Stage Docker Robot Deployments
Levon Avagyan and Xu Han

ROSCon2018

Multi-Stage Docker Robot Deployments
Levon Avagyan and Xu Han

ROSCon2018

“Developers these days don’t know how to deploy code,
so they just ship their development environment instead”- commenter on stackoverflow on using docker

ROSCon2018

❏

Keep update sizes as small as possible❏

Minimize the time and effort it takes to make a release❏

Make upgrades/downgrades hermetic, no accidental halfway upgrades

Goals for Release and Deployment

ROSCon2018

❏Make it easy to have multiple versions and developers on one robot❏

Make it so that developers can build a release locally❏

Make developer, testing, and production environments as similar as possible❏

Make setting up a robot development and testing environment seamless and easy❏

Minimize time spent compiling

Before multi-stage docker deploys

Goals for Development

ROSCon2018

❏Form of containerization❏

Encapsulates system dependencies, environment, executables, file systems,etc…

❏

Not a VM, uses underlying kernel and has minimal virtualization overhead (inmost cases)❏Dockerfiles - description files for configuring Docker Images❏

Docker Compose - description files for starting sets of Docker Containers❏

Relies on concept of image layers that can be cached, reused, and transferred

ROSCon2018

# Base Image
FROM ubuntu:18.04
RUN echo “hello world” > hello_world.txt
RUN date > date.txt
RUN cp date.txt date2.txt
RUN rm date.txt
RUN echo “I am done”

LayersROSCon2018

# Base Image
FROM ubuntu:18.04

CACHED

RUN echo “hello world” > hello_world.txt
RUN date > date.txt

CACHED

RUN cp date.txt date2.txt
RUN rm date.txt

CACHEDCACHED

RUN echo “I am done”

LayersCACHEDCACHEDROSCon2018

# Base Image
FROM ubuntu:18.04

CACHED

RUN echo “bye world” > hello_world.txt
RUN date > date.txt

BROKEN

RUN cp date.txt date2.txt
RUN rm date.txt

BROKENBROKEN

RUN echo “I am done”

LayersCHANGEDBROKENROSCon2018

# Base Image
FROM ubuntu:18.04

CACHED

RUN echo “bye world” > hello_world.txt
RUN date > date.txt

CACHED

RUN cp date.txt date3.txt
RUN rm date.txt

CHANGEDBROKEN

RUN echo “I am done”

LayersCACHEDBROKENROSCon2018

.
├── moveit.php├── data.txt├── Dockerfile└── robot.png

Context and Copy

ROSCon2018

# Base Image
FROM ubuntu:18.04

data.txt

COPY data.txt data.txt
RUN cat data.txt

Context and Copy

I am data

ROSCon2018

# Base Image
FROM ubuntu:18.04

CACHED

data.txt

COPY data.txt data.txt CACHED
RUN cat data.txt

CACHED

Context and Copy

I am data

ROSCon2018

# Base Image
FROM ubuntu:18.04

CACHED

data.txt

COPY data.txt data.txt CHANGED
RUN cat data.txt

BROKENContext and Copy

I am groot

ROSCon2018

❏Allows copying from intermediate stages without having those layers show up inthe final image❏

Cached separately from mainline docker image❏

Replaces bash scripts clobbering together different build steps

Multi-Stage Builds

ROSCon2018

# Copy StageFROM ubuntu:18.04 as copy_stage

data.txt

RUN echo “copy from me!”RUN echo “I am data” > data.txt

I am data# Base Image
FROM ubuntu:18.04
COPY --from=copy_stage data.txt data.txt
RUN cp date.txt date2.txt
RUN echo “I am done”

LayersROSCon2018

# Copy StageFROM ubuntu:18.04 as copy_stage
RUN echo “copy from me!”CACHED

data.txt

CACHED

RUN echo “I am data” > data.txt

I am data

CACHED# Base Image
FROM ubuntu:18.04 CACHEDCOPY --from=copy_stage data.txt data.txt
RUN cp date.txt date2.txt
RUN echo “I am done”

LayersCACHEDCACHEDCACHEDROSCon2018

# Copy StageFROM ubuntu:18.04 as copy_stage
RUN echo “I am changed!”

CACHED

data.txt

CHANGED

RUN echo “I am data” > data.txt

I am dataBROKEN# Base Image
FROM ubuntu:18.04 CACHEDCOPY --from=copy_stage data.txt data.txt
RUN cp date.txt date2.txt
RUN echo “I am done”

LayersCACHEDCACHEDCACHEDROSCon2018

# Copy StageFROM ubuntu:18.04 as copy_stage
RUN echo “I am changed!”

CACHED

data.txt

CACHED

RUN echo “I am groot” > data.txt

CHANGEDI am groot# Base Image
FROM ubuntu:18.04 CACHEDCOPY --from=copy_stage data.txt data.txt
RUN cp date.txt date2.txt
RUN echo “I am done”

LayersCHANGEDBROKENBROKENROSCon2018AB

B inherits from A

Changes very rarelyAB

B copies from A

Major releases

AB

B takes A as arg

All releases

SSH KeysStructure

ROSCon2018

❏Needs to be updated extremely rarely❏

Only needs to be built once and can be stored in a registry❏

Can either be in same, or different Dockerfile

SSH Keys

Frozen StageROSCon2018

FROM ubuntu:18.04# Install basic packages
RUN DEBIAN_FRONTEND=noninteractive apt-get update -y && \apt-get install software-properties-common curl -yFrozen StageROSCon2018

❏Allows the use of secrets without leaking them into output images❏

Can be combined with cache breaker stage to git clone private repos❏

You don’t have to share build keys, everyone can use their own (though the cachewill break)

SSH Keys

Secrets StageROSCon2018

# Cache Breaking StageFROM ubuntu:18.04 as secret_stage# Inject the secret we want to use
ARG SUPER_SECRET_PASSWORD# Download remote file that may change between builds
RUN wget --user roscon --password $SUPER_SECRET_PASSWORD
https://legitexample.com/requirements.txt# Work StageFROM python:3.7.0# We can copy the requirements file but the password ARG never interacts with this stage’sfilesystem
COPY --from=secret_stage /requirements.txt /requirements.txt
RUN pip install -r requirements.txt

docker build --build-arg SUPER_SECRET_PASSWORD="hunter2" -t robot:roscon

Secrets StageROSCon2018

❏Used to force a particular section to never be cached, without breaking the cachefor the entire Dockerfile❏

Can be used to force updates, download source code, get time/date related data❏

If the data copied out of this stage has not changed, the stage that copies from it willremain cached

SSH Keys

Cache Breaker StageROSCon2018

# Cache Breaking StageFROM ubuntu:18.04 as cache_breaker_stage# Breaks the cache every time this arg changes
ARG RANDOMIZER# Download remote file that may change between builds
RUN wget https://legitexample.com/requirements.txt# Work StageFROM python:3.7.0
COPY --from=cache_breaker_stage /requirements.txt /requirements.txt# This will be cached so long as requirements.txt has not changed
RUN pip install -r requirements.txt

docker build --build-arg RANDOMIZER="$(date|md5sum)" -t robot:roscon

Cache Breaker StageROSCon2018

❏Contains all runtime (production) requirements for the robot❏If you were to copy the built binaries into this stage, they should be able to run❏

Common base for dev, production, and testing images for a given release, andinherits from frozen stage

SSH Keys

Base StageROSCon2018

# Cache Breaking StageFROM ubuntu:18.04 as cache_breaker_stage
ARG SUPER_SECRET_PASSWORD
ARG RANDOMIZER# Download remote file that may change between buildsRUN wget --user roscon --password $SUPER_SECRET_PASSWORD
https://legitexample.com/requirements.txt# Base StageFROM frozen_stage as base_stage# Install all of our runtime requirements using pip, apt, curl, etc...
COPY --from=cache_breaker_stage /requirements.txt /requirements.txt
RUN pip install -r requirements.txt

Base StageROSCon2018

❏Contains all build time requirements for the robot, as well as source code❏

At the end of this stage, if you were to call build on any of your projects, they should
be successful❏

Can be used by developers for debugging build issues

SSH Keys

Dev Base StageROSCon2018

# Cache Breaking StageFROM ubuntu:18.04 as cache_breaker_stage
ARG SUPER_SECRET_PASSWORD
ARG RANDOMIZER# Download remote file that may change between builds
RUN wget --user roscon --password $SUPER_SECRET_PASSWORD
https://legitexample.com/requirements-dev.txt
RUN git clone git@github.com:sourcecode/isthirteen.git# Dev Base StageFROM base_stage as dev_base_stage
COPY --from=cache_breaker_stage /isthirteen /catkin_ws/src/isthirteen# Install all of our build/dev requirements using pip, apt, curl, etc...
COPY --from=cache_breaker_stage /requirements-dev.txt /requirements-dev.txt
RUN pip install -r requirements-dev.txt

Dev Base StageROSCon2018

❏Builds all of the source code, by the end of the stage you have all of your binaries
prepared❏

Gets cannibalized for binaries and then discarded❏

Will only be copied from, so you can inject secrets here as well for things such as
npm/bazel

SSH Keys

Build StageROSCon2018

# Base StageFROM dev_base_stage as dev_build_stage
ARG SUPER_SECRET_PASSWORD# Build the stuff!
RUN cd catkin_ws && catkin_make install

Build StageROSCon2018

❏Is your final dev image, you should be able to put this on a newly reimaged robot
and get testing right away❏

Can be used directly as a development environment, or as a base for CI/CD images❏Inherits from dev base stage so it has all of the runtime dependencies and build
dependencies

SSH Keys

Dev Stage

ROSCon2018

# Base StageFROM base_stage as prod_stage# We’re done!
COPY --from=dev_build_stage /catkin_ws/ /catkin_ws/

Dev Stage

ROSCon2018

❏Is your final production image, you should be able to put this on a newly reimaged
robot and call it a day❏

Copies over only the binaries, no need for the source code❏

Inherits from base stage so it has all of the runtime dependencies

SSH Keys

Prod StageROSCon2018

❏Is your final production image, you should be able to put this on a newly reimaged
robot and call it a day❏

Copies over only the binaries, no need for the source code❏

Inherits from base stage so it has all of the runtime dependencies# Base StageFROM base_stage as prod_stage# We’re done, but actually!
COPY --from=dev_build_stage /catkin_ws/install /catkin_ws/install# Set a command to automatically run when the container is started
CMD /bin/bash -c “source /catkin_ws/install/setup.bash && roslaunch isthirteen
is_it_thirteen.launch”

Prod StageROSCon2018

AB

B inherits from A

Changes very rarelyAB

B copies from A

Major releases

AB

B takes A as arg

All releases

SSH Keys

Optional Extensions

ROSCon2018

❏Developers can have different versions with mutually exclusive dependencies
running on the same robot❏

No more worries about robots other developers have snowflaked, your own
environment is portable and hermetic

Developer Benefits

ROSCon2018

❏Entire build pipeline for all production robot types is now just ~300 lines ofDockerfile❏

Releases are now nightly and on-demand, take under two hours instead of a day❏

Build pipeline is transparent and easy to understand and modify❏

Building a release candidate just takes github ssh keys and docker, no othertooling required

Release Benefits

ROSCon2018

❏Worry free testing in the field. If something goes wrong, just start a new container!

❏Upgrades/downgrades are one shot, no failed intermediate states

Deployment Benefits

ROSCon2018

❏

Layer and image size is now a first class concern and can affect update anddependency management strategies❏

Changing a layer high up in the chain can cause large updates, up to 2GB in ourcase❏

But, images can be downloaded in the background without interrupting
operation❏

Dev images with unstripped binaries can be very large (~20GB), pulling/pushing
to robots can take a while if you don’t have good wifi (still faster than pullingsource and recompiling, though!)❏Requires new infrastructure (local and cloud registries)

Considerations

ROSCon2018

Questions?
ROSCon2018