Project: rkt (Runtime / Container Runtime)
File: inspect-containers.md

There's a variety of ways to inspect how containers work. Linux provides APIs that expose information about namespaces (proc filesystem) and cgroups (cgroup filesystem). We also have tools like strace that allow us to see what system calls are used in processes. This document explains how to use those APIs and tools to give details on what rkt does under the hood. Note that this is not a comprehensive analysis of the inner workings of rkt, but a starting point for people interested in learning how containers work. Let's use to find out what system calls rkt uses to set up containers. We'll only trace a handful of syscalls since, by default, strace traces every syscall resulting in a lot of output. Also, we'll redirect its output to a file to make the analysis easier. ```bash $ sudo strace -f -s 512 -e unshare,clone,mount,chroot,execve -o out.txt rkt run coreos.com/etcd:v2.0.10 ... ^]^]Container rkt-e6d92625-aa3f-4449-bf5d-43ffed440de4 terminated by signal KILL. ``` We now have our trace in `out.txt`, let's go through some of its relevant parts. First, we see the actual execution of the rkt command: ``` 5710 execve("/home/iaguis/work/go/src/github.com/rkt/rkt/build-rkt/target/bin/rkt", ["rkt", "run", "coreos.com/etcd:v2.0.10"], 0x7ffce2052be8 / 22 vars /) = 0 ``` Since the image was already fetched and we don't trace many system calls, nothing too exciting happens here except mounting the container filesystems. ``` 5710 mount("overlay", "/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/stage1/rootfs", "overlay", 0, "lowerdir=/var/lib/rkt/cas/tree/deps-sha512-cc076d6c508223cc3c13c24d09365d64b6d15e7915a165eab1d9e87f87be5015/rootfs,upperdir=/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/overlay/deps-sha512-cc076d6c508223cc3c13c24d09365d64b6d15e7915a165eab1d9e87f87be5015/upper,workdir=/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/overlay/deps-sha512-cc076d6c508223cc3c13c24d09365d64b6d15e7915a165eab1d9e87f87be5015/work") = 0 5710 mount("overlay", "/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/stage1/rootfs/opt/stage2/etcd/rootfs", "overlay", 0, "lowerdir=/var/lib/rkt/cas/tree/deps-sha512-c0de11e9d504069810da931c94aece3bcc5430dc20f9a5177044eaef62f93fcc/rootfs,upperdir=/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/overlay/deps-sha512-c0de11e9d504069810da931c94aece3bcc5430dc20f9a5177044eaef62f93fcc/upper/etcd,workdir=/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/overlay/deps-sha512-c0de11e9d504069810da931c94aece3bcc5430dc20f9a5177044eaef62f93fcc/work/etcd") = 0 ``` We can see that rkt mounts the stage1 and stage2 filesystems with the as `lowerdir` on the directory rkt expects them to be. Note that the stage2 tree is mounted within the stage1 tree via `/opt/stage2`. You can read more about the tree structure in . This means that, for a same tree store, everything will be shared in a copy-on-write (COW) manner, except the bits that each container modifies, which will be in the `upperdir` and will appear magically in the mount destination. You can read more about this filesystem in the . This is where most of the interesting things happen, the first being executing the stage1 , which is `/init` by default: ``` 5710 execve("/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/stage1/rootfs/init", ["/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/stage1/rootfs/init", "--net=default", "--local-config=/etc/rkt", "d5513d49-d14f-45d1-944b-39437798ddda"], 0xc42009b040 / 25 vars / <unfinished ...> ``` init does a bunch of stuff, including creating the container's network namespace and mounting a reference to it on the host filesystem: ``` 5723 unshare(CLONE_NEWNET) = 0 5723 mount("/proc/5710/task/5723/ns/net", "/var/run/netns/cni-eee014d2-8268-39cc-c176-432bbbc9e959", 0xc42017c6a8, MS_BIND, NULL) = 0 ``` After creating the network namespace, it will execute the relevant plugins from within that network namespace. The default network uses the with as IPAM: ``` 5725 execve("stage1/rootfs/usr/lib/rkt/plugins/net/ptp", ["stage1/rootfs/usr/lib/rkt/plugins/net/ptp"], 0xc4201ac000 / 32 vars / <unfinished ...> 5730 execve("stage1/rootfs/usr/lib/rkt/plugins/net/host-local", ["stage1/rootfs/usr/lib/rkt/plugins/net/host-local"], 0xc42008e240 / 32 vars / <unfinished

``` In this case, the CNI plugins use come from rkt's stage1, but . The plugins will do some iptables magic to configure the network: ``` 5739 execve("/usr/bin/iptables", ["iptables", "--version"], 0xc42013e000 / 32 vars / <unfinished ...> 5740 execve("/usr/bin/iptables", ["/usr/bin/iptables", "-t", "nat", "-N", "CNI-7a59ad232c32bcea94ee08d5", "--wait"], 0xc4200b0a20 / 32 vars / <unfinished ...> ... ``` After the network is configured, rkt mounts the container cgroups instead of letting systemd-nspawn do it because we want to have control on how they're mounted. We also mount the host cgroups if they're not already mounted in the way systemd-nspawn expects them, like in old distributions or distributions that don't use systemd (e.g. ). We do this in a new mount namespace to avoid polluting the host mounts and to get automatic cleanup when the container exits (`CLONE_NEWNS` is the flag for for historical reasons: it was the first namespace implemented on Linux): ``` 5710 unshare(CLONE_NEWNS) = 0 ``` Here we mount the container hierarchies read-write so the pod can modify its cgroups but we mount the controllers read-only so the pod doesn't modify other cgroups: ``` 5710 mount("stage1/rootfs/sys/fs/cgroup/freezer/machine.slice/machine-rkt\\x2dd5513d49\\x2dd14f\\x2d45d1\\x2d944b\\x2d39437798ddda.scope/system.slice", "stage1/rootfs/sys/fs/cgroup/freezer/machine.slice/machine-rkt\\x2dd5513d49\\x2dd14f\\x2d45d1\\x2d944b\\x2d39437798ddda.scope/system.slice", 0xc42027d2a8, MS_BIND, NULL) = 0 ... 5710 mount("stage1/rootfs/sys/fs/cgroup/freezer", "stage1/rootfs/sys/fs/cgroup/freezer", 0xc42027d2b8, MSRDONLY|MSNOSUID|MSNODEV|MSNOEXEC|MSREMOUNT|MSBIND, NULL) = 0 ``` Now is the time to start systemd-nspawn to create the pod itself: ``` 5710 execve("stage1/rootfs/usr/lib/ld-linux-x86-64.so.2", ["stage1/rootfs/usr/lib/ld-linux-x86-64.so.2", "stage1/rootfs/usr/bin/systemd-nspawn", "--boot", "--notify-ready=yes", "--register=true", "--link-journal=try-guest", "--quiet", "--uuid=d5513d49-d14f-45d1-944b-39437798ddda", "--machine=rkt-d5513d49-d14f-45d1-944b-39437798ddda", "--directory=stage1/rootfs", "--capability=CAPAUDITWRITE,CAPCHOWN,CAPDACOVERRIDE,CAPFSETID,CAPFOWNER,CAPKILL,CAPMKNOD,CAPNETRAW,CAPNETBINDSERVICE,CAPSETUID,CAPSETGID,CAPSETPCAP,CAPSETFCAP,CAPSYSCHROOT", "--", "--default-standard-output=tty", "--log-target=null", "--show-status=0"], 0xc4202bc0f0 / 29 vars / <unfinished ...> ``` Note we don't need to pass the `--private-network` option because rkt already created and configured the network using CNI. Some interesting things systemd-nspawn does are moving the container filesystem tree to `/`: ``` 5747 mount("/var/lib/rkt/pods/run/d5513d49-d14f-45d1-944b-39437798ddda/stage1/rootfs", "/", NULL, MS_MOVE, NULL) = 0 5747 chroot(".") = 0 ``` And creating all the other namespaces: mount, UTS, IPC, and PID. Check for more information. ``` 5747 clone(childstack=NULL, flags=CLONENEWNS|CLONENEWUTS|CLONENEWIPC|CLONE_NEWPID|SIGCHLD) = 5748 ``` Once it's done creating the container, it will execute the init process, which is systemd: ``` 5748 execve("/usr/lib/systemd/systemd", ["/usr/lib/systemd/systemd", "--default-standard-output=tty", "--log-target=null", "--show-status=0"], 0x7f904604f250 / 8 vars /) = 0 ``` Which then will execute systemd-journald to handle logging: ``` 5749 execve("/usr/lib/systemd/systemd-journald", ["/usr/lib/systemd/systemd-journald"], 0x5579c5d79d50 / 8 vars / <unfinished ...> ... ``` And at some point it will execute our application's service (in this example, etcd). But first, it needs to execute its companion `prepare-app` dependency: ``` 5751 execve("/prepare-app", ["/prepare-app", "/opt/stage2/etcd/rootfs"], 0x5579c5d7d580 / 7 vars /) = 0 ``` `prepare-app` bind-mounts a lot of files from stage1 to stage2, so our app has access to a : ``` 5751 mount("/dev/null", "/opt/stage2/etcd/rootfs/dev/null", 0x49006f, MS_BIND, NULL) = 0 5751 mount("/dev/zero", "/opt/stage2/etcd/rootfs/dev/zero", 0x49006f, MS_BIND, NULL) = 0

``` After it's finished, our etcd service is ready to start! Since we use some additional security directives in our service file (like or ), systemd will create an additional mount namespace per application in the pod and move the stage2 filesystem to `/`: ``` 5753 unshare(CLONE_NEWNS) = 0 ... 5753 mount("/opt/stage2/etcd/rootfs", "/", NULL, MS_MOVE, NULL) = 0 5753 chroot(".") = 0 ``` Now we're ready to execute the etcd binary. ``` 5753 execve("/etcd", ["/etcd"], 0x5579c5dbd660 / 9 vars /) = 0 ``` And that's it, etcd is running in a container! We'll now inspect a running container by using the . Let's start a new container limiting the CPU to 200 millicores and the memory to 100MB: ``` $ sudo rkt run --interactive kinvolk.io/aci/busybox --memory=100M --cpu=200m / # ``` First we'll need to find the PID of a process running inside the container. We can see the container PID by running `rkt status`: ``` $ sudo rkt status 567264dd state=running created=2018-01-03 17:17:39.653 +0100 CET started=2018-01-03 17:17:39.749 +0100 CET networks=default:ip4=172.16.28.37 pid=10985 exited=false ``` Now we need to find the sh process running inside the container: ``` $ ps auxf | grep [1]0985 -A 3 root 10985 0.0 0.0 54204 5040 pts/2 S+ 17:17 0:00 \ stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=567264dd-f28d-42fb-84a1-4714dde9e82c --machine=rkt-567264dd-f28d-42fb-84a1-4714dde9e82c --directory=stage1/rootfs --capability=CAPAUDITWRITE,CAPCHOWN,CAPDACOVERRIDE,CAPFSETID,CAPFOWNER,CAPKILL,CAPMKNOD,CAPNETRAW,CAPNETBINDSERVICE,CAPSETUID,CAPSETGID,CAPSETPCAP,CAPSETFCAP,CAPSYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0 root 11021 0.0 0.0 62280 7392 ? Ss 17:17 0:00 \_ /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 root 11022 0.0 0.0 66408 8812 ? Ss 17:17 0:00 \_ /usr/lib/systemd/systemd-journald root 11026 0.0

1212 4 pts/0 Ss+ 17:17 0:00 \_ /bin/sh ``` It's 11026! Let's start by having a look at its namespaces: ``` $ sudo ls -l /proc/11026/ns/ total 0 lrwxrwxrwx 1 root root 0 Jan 3 17:19 cgroup -> 'cgroup:[4026531835]' lrwxrwxrwx 1 root root 0 Jan 3 17:19 ipc -> 'ipc:[4026532764]' lrwxrwxrwx 1 root root 0 Jan 3 17:19 mnt -> 'mnt:[4026532761]' lrwxrwxrwx 1 root root 0 Jan 3 17:19 net -> 'net:[4026532702]' lrwxrwxrwx 1 root root 0 Jan 3 17:19 pid -> 'pid:[4026532765]' lrwxrwxrwx 1 root root 0 Jan 3 17:19 pidforchildren -> 'pid:[4026532765]' lrwxrwxrwx 1 root root 0 Jan 3 17:19 user -> 'user:[4026531837]' lrwxrwxrwx 1 root root 0 Jan 3 17:19 uts -> 'uts:[4026532763]' ``` We can compare it with the namespaces on the host ``` $ sudo ls -l /proc/1/ns/ total 0 lrwxrwxrwx 1 root root 0 Jan 3 17:20 cgroup -> 'cgroup:[4026531835]' lrwxrwxrwx 1 root root 0 Jan 3 17:20 ipc -> 'ipc:[4026531839]' lrwxrwxrwx 1 root root 0 Jan 3 17:20 mnt -> 'mnt:[4026531840]' lrwxrwxrwx 1 root root 0 Jan 3 17:20 net -> 'net:[4026532009]' lrwxrwxrwx 1 root root 0 Jan 3 17:20 pid -> 'pid:[4026531836]' lrwxrwxrwx 1 root root 0 Jan 3 17:20 pidforchildren -> 'pid:[4026531836]' lrwxrwxrwx 1 root root 0 Jan 3 17:20 user -> 'user:[4026531837]' lrwxrwxrwx 1 root root 0 Jan 3 17:20 uts -> 'uts:[4026531838]' ``` We can see that the cgroup and user namespace are the same, since rkt doesn't use cgroup namespaces and user namespaces weren't enabled for this execution. If, for example, we run rkt with `--net=host`, we'll see that the network namespace is the same as the host's. Running we can see this information too, along with the PID that created the namespace: ``` sudo lsns -p 11026 NS TYPE NPROCS PID USER COMMAND 4026531835 cgroup 231 1 root /sbin/init 4026531837 user 231 1 root /sbin/init 4026532702 net 4 10945 root stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal= 4026532761 mnt 1 11026 root /etcd 4026532763 uts 3 11021 root /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 4026532764 ipc 3 11021 root /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 4026532765 pid 3 11021 root /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 ``` We can also see some interesting data about the process: ``` $ sudo cat /proc/11026/status Name: sh Umask: 0022 State: S (sleeping) ... CapBnd: 00000000a80425fb ... NoNewPrivs: 0 Seccomp: 2 ... ``` This tells us the container is not using the `nonewprivs` feature, but it is using . We can also see what are in the bounding set of the process, let's decode them with `capsh`: ``` $ capsh --decode=00000000a80425fb 0x00000000a80425fb=capchown,capdacoverride,capfowner,capfsetid,capkill,capsetgid,capsetuid,capsetpcap,capnetbindservice,capnetraw,capsyschroot,capmknod,capauditwrite,capsetfcap ``` Another interesting thing is the environment variables of the process: ``` $ sudo cat /proc/11026/environ | tr '\0' '\n' PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root LOGNAME=root USER=root SHELL=/bin/sh INVOCATION_ID=d5d94569d482495c809c113fca55abd4 TERM=xterm ACAPPNAME=busybox ``` Finally, we can check in which cgroups the process is: ``` $ sudo cat /proc/11026/cgroup 11:freezer:/ 10:rdma:/ 9:cpu,cpuacct:/machine.slice/machine-rkt\x2d97910fdc\x2d13ec\x2d4025\x2d8f93\x2d5ddea0089eff.scope/system.slice/busybox.service 8:devices:/machine.slice/machine-rkt\x2d97910fdc\x2d13ec\x2d4025\x2d8f93\x2d5ddea0089eff.scope/system.slice/busybox.service 7:blkio:/machine.slice/machine-rkt\x2d97910fdc\x2d13ec\x2d4025\x2d8f93\x2d5ddea0089eff.scope/system.slice 6:memory:/machine.slice/machine-rkt\x2d97910fdc\x2d13ec\x2d4025\x2d8f93\x2d5ddea0089eff.scope/system.slice/busybox.service 5:perf_event:/ 4:pids:/machine.slice/machine-rkt\x2d97910fdc\x2d13ec\x2d4025\x2d8f93\x2d5ddea0089eff.scope/system.slice/busybox.service 3:netcls,netprio:/ 2:cpuset:/ 1:name=systemd:/machine.slice/machine-rkt\x2d97910fdc\x2d13ec\x2d4025\x2d8f93\x2d5ddea0089eff.scope/system.slice/busybox.service 0::/machine.slice/machine-rkt\x2d97910fdc\x2d13ec\x2d4025\x2d8f93\x2d5ddea0089eff.scope/system.slice/busybox.service ``` Let's explore the cgroups a bit more. systemd offers tools to easily inspect the cgroups of containers. We can use `systemd-cgls` to see the cgroup hierarchy of a container: ``` $ machinectl MACHINE CLASS SERVICE OS VERSION ADDRESSES rkt-97910fdc-13ec-4025-8f93-5ddea0089eff container rkt - - 172.16.28.25... 1 machines listed. $ systemd-cgls -M rkt-97910fdc-13ec-4025-8f93-5ddea0089eff Control group /machine.slice/machine-rkt\x2d97910fdc\x2d13ec\x2d4025\x2d8f93\x2d5ddea0089eff.scope: -.slice init.scope 12474 /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 system.slice busybox.service 12479 /bin/sh systemd-journald.service 12475 /usr/lib/systemd/systemd-journald ``` And we can use `systemd-cgtop` to see the resource consumption of the container. This is the output while running the `yes` command (which is basically an infinite loop that outputs the character `y`, so it takes all the CPU) in the container: ``` Control Group Tasks %CPU Memory Input/s Output/s