Project: Apache Mesos (Orchestration & Management / Scheduling & Orchestration)
File: container-image.md

title: Apache Mesos - Supporting Container Images in Mesos Containerizer layout: documentation Mesos currently supports several , notably the Mesos containerizer and the Docker containerizer. Mesos containerizer uses native OS features directly to provide isolation between containers, while Docker containerizer delegates container management to the Docker engine. Maintaining two containerizers is hard. For instance, when we add new features to Mesos (e.g., persistent volumes, disk isolation), it becomes a burden to update both containerizers. Even worse, sometimes the isolation on some resources (e.g., network handles on an agent) requires coordination between two containerizers, which is very hard to implement in practice. In addition, we found that extending and customizing isolation for containers launched by Docker engine is difficult, mainly because we do not have a way to inject logics during the life cycle of a container. Therefore, we made an effort to unify containerizers in Mesos (, a.k.a. the Universal Containerizer). We improved Mesos containerizer so that it now supports launching containers that specify container images (e.g., Docker/Appc images). To support container images, we introduced a new component in Mesos containerizer, called image provisioner. Image provisioner is responsible for pulling, caching and preparing container root filesystems. It also extracts runtime configurations from container images which will then be passed to the corresponding isolators for proper isolation. There are a few container image specifications, notably , , and (future). Currently, we support Docker and Appc images. More details about what features are supported or not can be found in the following sections. NOTE: container image is only supported on Linux currently. To enable container image support in Mesos containerizer, the operator will need to specify the `--image_providers` agent flag which tells Mesos containerizer what types of container images are allowed. For example, setting `--image_providers=docker` allow containers to use Docker images. The operators can also specify multiple container image types. For instance, `--image_providers=docker,appc` allows both Docker and Appc container images. A few isolators need to be turned on in order to provide proper isolation according to the runtime configurations specified in the container image. The operator needs to add the following isolators to the `--isolation` flag. `filesystem/linux`: This is needed because supporting container images involves changing filesystem root, and only `filesystem/linux` support that currently. Note that this isolator requires root permission. `docker/runtime`: This is used to provide support for runtime configurations specified in Docker images (e.g., Entrypoint/Cmd, environment variables, etc.). See more details about this isolator in . Note that if this isolator is not specified and `--image_providers` contains `docker`, the agent will refuse to start. In summary, to enable container image support in Mesos containerizer, please specify the following agent flags: $ sudo mesos-agent \ --containerizers=mesos \ --image_providers=appc,docker \ --isolation=filesystem/linux,docker/runtime We introduced a new protobuf message `Image` which allow frameworks to specify container images for their containers. It has two types right now: `APPC` and `DOCKER`, representing Appc and Docker images respectively. For Appc images, the `name` and `labels` are what described in the . For Docker images, the `name` is the Docker image reference in the following form (the same format expected by `docker pull`): `[REGISTRYHOST[:REGISTRYPORT]/]REPOSITORY[:TAG|@DIGEST]` message Image { enum Type { APPC = 1; DOCKER = 2; } message Appc { required string name = 1; optional Labels labels = 3; } message Docker { required string name = 1; } required Type type = 1; // Only one of the following image messages should be set to match // the

optional Appc appc = 2; optional Docker docker = 3; } The framework needs to specify `MesosInfo` in `ContainerInfo` in order to launch containers with container images. In other words, the framework needs to set the type to `ContainerInfo.MESOS`, indicating that it wants to use the Mesos containerizer. If `MesosInfo.image` is not specified, the container will use the host filesystem. If `MesosInfo.image` is specified, it will be used as the container image when launching the container. message ContainerInfo { enum Type { DOCKER = 1; MESOS = 2; } message MesosInfo { optional Image image = 1; } required Type type = 1; optional MesosInfo mesos = 5; } First, start the Mesos master: $ sudo sbin/mesos-master --work_dir=/tmp/mesos/master Then, start the Mesos agent: $ sudo GLOG_v=1 sbin/mesos-agent \ --master=<MASTER_IP>:5050 \ --isolation=docker/runtime,filesystem/linux \ --work_dir=/tmp/mesos/agent \ --image_providers=docker \ --executorenvironmentvariables="{}" Now, use Mesos CLI (i.e., mesos-execute) to launch a Docker container (e.g., redis). Note that `--shell=false` tells Mesos to use the default entrypoint and cmd specified in the Docker image. $ sudo bin/mesos-execute \ --master=<MASTER_IP>:5050 \ --name=test \ --docker_image=library/redis \ --shell=false Verify if your container is running by launching a redis client: $ sudo docker run -ti --net=host redis redis-cli 127.0.0.1:6379> ping PONG 127.0.0.1:6379> Image provisioner uses [Docker v2 registry API](https://docs.docker.com/registry/spec/api/) to fetch Docker images/layers. Both docker manifest and are supported (v2 schema2 is supported starting from 1.8.0). The fetching is based on `curl`, therefore SSL is automatically handled. For private registries, the operator needs to configure `curl` with the location of required CA certificates. Fetching requiring authentication is supported through the `--docker_config` agent flag. Starting from 1.0, operators can use this agent flag to specify a shared docker config file, which is used for pulling private repositories with authentication. Per container credential is not supported yet (coming soon). Operators can either specify the flag as an absolute path pointing to the docker config file (need to manually configure `.docker/config.json` or `.dockercfg` on each agent), or specify the flag as a JSON-formatted string. See [configuration documentation](configuration/agent.md) for detail. For example: --docker_config=file:///home/vagrant/.docker/config.json or as a JSON object, --docker_config="{ \ \"auths\": { \ \"https://index.docker.io/v1/\": { \ \"auth\": \"xXxXxXxXxXx=\", \ \"email\": \"username@example.com\" \ } \ } \ }" Private registry is supported either through the `--docker_registry` agent flag, or specifying private registry for each container using image name `<REGISTRY>/<REPOSITORY>` (e.g., `localhost:80/gilbert/inky:latest`). If `<REGISTRY>` is included as a prefix in the image name, the registry specified through the agent flag `--docker_registry` will be ignored. If the `--docker_registry` agent flag points to a local directory (e.g., `/tmp/mesos/images/docker`), the provisioner will pull Docker images from local filesystem, assuming Docker archives (result of `docker save`) are stored there based on the image name and tag. For example, the operator can put a `busybox:latest.tar` (the result of `docker save -o busybox:latest.tar busybox`) under `/tmp/mesos/images/docker` and launch the agent by specifying `--docker_registry=/tmp/mesos/images/docker`. Then the framework can launch a Docker container by specifying `busybox:latest` as the name of the Docker image. This flag can also point to an HDFS URI (experimental in Mesos 1.7) (e.g., `hdfs://localhost:8020/archives/`) to fetch images from HDFS if the `hadoop` command is available on the agent. If the `--switch_user` flag is set on the agent and the framework specifies a user (either `CommandInfo.user` or `FrameworkInfo.user`), we expect that user exists in the container image and its uid and gids matches that on the host. User namespace is not supported yet. If the user is not specified, `root` will be used by

The operator or the framework can limit the of the container by using the isolator. Currently, we support `host`, `bridge` and user defined networks (). `none` is not supported yet. We support the above networking modes in using the (Container Network Interface) standard. Please refer to the isolator document for more details about how to configure the network for the container. `--docker_registry`: The default URL for pulling Docker images. It could either be a Docker registry server URL (i.e: `https://registry.docker.io`), or a local path (i.e: `/tmp/docker/images`) in which Docker image archives (result of `docker save`) are stored. The default value is `https://registry-1.docker.io`. `--dockerstoredir`: Directory the Docker provisioner will store images in. All the Docker images are cached under this directory. The default value is `/tmp/mesos/store/docker`. `--docker_config`: The default docker config file for agent. Can be provided either as an absolute path pointing to the agent local docker config file, or as a JSON-formatted string. The format of the docker config file should be identical to docker's default one (e.g., either `$HOME/.docker/config.json` or `$HOME/.dockercfg`). Currently, only the root filesystem specified in the Appc image is supported. Other runtime configurations like environment variables, exec, working directory are not supported yet (coming soon). For image discovery, we current support a simple discovery mechanism. We allow operators to specify a URI prefix which will be prepend to the URI template `{name}-{version}-{os}-{arch}.{ext}`. For example, if the URI prefix is `file:///tmp/appc/` and the Appc image name is `example.com/reduce-worker` with `version:1.0.0`, we will fetch the image at `file:///tmp/appc/example.com/reduce-worker-1.0.0.aci`. `appcsimplediscoveryuriprefix`: URI prefix to be used for simple discovery of appc images, e.g., `http://`, `https://`, `hdfs://<hostname>:9000/user/abc/cde`. The default value is `http://`. `appcstoredir`: Directory the appc provisioner will store images in. All the Appc images are cached under this directory. The default value is `/tmp/mesos/store/appc`. A provisioner backend takes a set of filesystem layers and stacks them into a root filesystem. Currently, we support the following backends: `copy`, `bind`, `overlay` and `aufs`. Mesos will validate if the selected backend works with the underlying filesystem (the filesystem used by the image store `--dockerstoredir` or `--appcstoredir`) using the following logic table: ++--++ | Backend | Suggested on | Disabled on | ++--++ | aufs | ext4 xfs | btrfs aufs eCryptfs | | overlay | ext4 xfs* | btrfs aufs overlay overlay2 zfs eCryptfs | | bind | | N/A(`--sandbox_directory' must exist) | | copy | | N/A | ++--++ NOTE: `xfs` support on `overlay` is enabled only when `d_type=true`. Use `xfs_info` to verify that the `xfs` ftype option is set to 1. To format an xfs filesystem for `overlay`, use the flag `-n ftype=1` with `mkfs.xfs`. The provisioner backend can be specified through the agent flag `--imageprovisionerbackend`. If not set, Mesos will select the best backend automatically for the users/operators. The selection logic is as following: Use `overlay` backend if the overlayfs is available. Use `aufs` backend if the aufs is available and overlayfs is not supported. Use `copy` backend if none of above is selected. The Copy backend simply copies all the layers into a target root directory to create a root filesystem. This is a specialized backend that may be useful for deployments using large (multi-GB) single-layer images and where more recent kernel features such as overlayfs are not available. For small images (10's to 100's of MB) the copy backend may be sufficient. Bind backend is faster than Copy as it requires nearly zero

The bind backend currently has these two limitations: The bind backend supports only a single layer. Multi-layer images will fail to provision and the container will fail to launch! The filesystem is read-only because all containers using this image share the source. Select writable areas can be achieved by mounting read-write volumes to places like `/tmp`, `/var/tmp`, `/home`, etc. using the `ContainerInfo`. These can be relative to the executor work directory. Since the filesystem is read-only, `--sandbox_directory` and `/tmp` must already exist within the filesystem because the filesystem isolator is unable to create it (e.g., either the image writer needs to create the mount point in the image, or the operator needs to set agent flag `--sandbox_directory` properly). The reason overlay backend was introduced is because the copy backend will waste IO and space while the bind backend can only deal with one layer. The overlay backend allows containizer to utilize the filesystem to merge multiple filesystems into one efficiently. The overlay backend depends on support for multiple lower layers, which requires Linux kernel version 4.0 or later. For more information of overlayfs, please refer to . The reason AUFS is introduced is because overlayfs support hasn't been merged until kernel 3.18 and Docker's default storage backend for ubuntu 14.04 is AUFS. Like overlayfs, AUFS is also a unioned file system, which is very stable, has a lot of real-world deployments, and has strong community support. Some Linux distributions do not support AUFS. This is usually because AUFS is not included in the mainline (upstream) Linux kernel. For more information of AUFS, please refer to . Mesos has this concept of executors. All tasks are launched by an executor. For a general purpose executor (e.g., thermos) of a framework (e.g., Aurora), requiring it and all its dependencies to be present in all possible container images that a user might use is not trivial. In order to solve this issue, we propose a solution where we allow the executor to run on the host filesystem (without a container image). Instead, it can specify a `volume` whose source is an `Image`. Mesos containerizer will provision the `image` specified in the `volume`, and mount it under the sandbox directory. The executor can perform `pivot_root` or `chroot` itself to enter the container root filesystem. Experimental support of garbage-collecting unused container images was added at Mesos 1.5. This can be either configured automatically via a new agent flag `--imagegcconfig`, or manually invoked through agent's . This can be used to avoid unbounded disk space usage of image stores. This is implemented with a simple mark-and-sweep logic. When image GC happens, we check all layers and images referenced by active running containers and avoid removing them from the image store. As a pre-requisite, if there are active containers launched before Mesos 1.5.0, we cannot determine what images can be safely garbage collected, so agent will refuse to invoke image GC. To garbage collect container images, users are expected to drain all containers launched before Mesos 1.5.0. NOTE: currently, the image GC is only supported for docker store in Mesos Containerizer. To enable automatic image GC, use the new agent flag `--imagegcconfig`: --imagegcconfig=file:///home/vagrant/image-gc-config.json or as a JSON object, --imagegcconfig="{ \ \"imagediskheadroom\": 0.1, \ \"imagediskwatch_interval\": { \ \"nanoseconds\": 3600000000000 \ }, \ \"excluded_images\": \[ \] \ }" See `PRUNE_IMAGES` section in for manual image GC through the agent HTTP API. For more information on the Mesos containerizer filesystem, namespace, and isolator features, visit [Mesos Containerizer](mesos-containerizer.md). For more information on launching Docker containers through the Docker containerizer, visit .