bpftraceで Docker の仕組み調べてみる

# 概要

Dockerはよく`docker run`とか`docker exec`のコマンドで Black Box みたいに使われて完結ですが、その中身をもっと詳しく調査してみたい。そのための有力ツールの一つは[bpftrace](https://github.com/iovisor/bpftrace)です。

# 作業環境

* Ubuntu 20.04

# Dockerの構成

Dockerはおよそ以下のパーツに構成されている

* `docker-cli`：`docker run`と`docker exec`などのコマンドを提供するDockerのクライアント側
* `dockcerd`：Docker のサーバー側、Docker Engine のデーモンプロセス、`docker-cli`などクライアントからのリクエストを処理する場所
* `containerd`：Container Runtime の一つ、いろんな Container の管理を機能しているデーモンプログラム
* `runc`：いろんなOS機能を利用して、単独のContainerの実行を成せているツール

# Dockerがやっていること

Containerは Namespace、Cgroup、OverlayFS、Virtual Network などのOS機能利用して可能になるのがよくご存知ですが、Docker はこちらの機能どう利用したますか？

まずは、Container が実行した結果を確認していきます。この一番簡単の Container 起動します

```sh
$ docker run -ti --rm --name test alpine sh
```

これからの利用のために pid と Container ID をに保存しておきます

```
$ export CPID=$(docker inspect test -f '{{ .State.Pid }}')
$ echo $CPID
230973
$ export CID=$(docker inspect test -f '{{ .Id }}')
$ echo $CID
b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
```

## `containerd`との親子関係

```sh
$ pstree -s $CPID -STUpau
systemd,1
  └─containerd,80405
      └─containerd-shim,230955 -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd ...
          └─sh,230973,ipc,mnt,net,pid,uts
```

`containerd`は`containerd-shim`を実行して、そしてまた Container プロセスを実行する。また、このContainer プロセスには`ipc`とか`mnt`などの namespace がついている。

## runcの設定

`containerd`が`runc`を利用してるので、その設定も確認します

```sh
$ jq -r '.process' < /var/run/containerd/io.containerd.runtime.v1.linux/moby/$CID/config.json
{
  "terminal": true,
  "user": {
    "uid": 0,
    "gid": 0,
　　...
  },
  "args": [
    "sh"
  ],
  "env": [
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "HOSTNAME=b6431f14cf40",
    "TERM=xterm"
  ],
  "cwd": "/",
  "capabilities": {
    "bounding": [
      "CAP_CHOWN",
      "CAP_DAC_OVERRIDE",
      "CAP_FSETID",
      "CAP_FOWNER",
      "CAP_MKNOD",
      "CAP_NET_RAW",
      "CAP_SETGID",
      "CAP_SETUID",
      "CAP_SETFCAP",
      "CAP_SETPCAP",
      "CAP_NET_BIND_SERVICE",
      "CAP_SYS_CHROOT",
      "CAP_KILL",
      "CAP_AUDIT_WRITE"
    ],
    ...
  },
  "apparmorProfile": "docker-default",
  "oomScoreAdj": 0
}
```

## Namespaces

Container プロセスについている namespace を具体的に見に行きましょう

```sh
$ lsns -p $CPID --output-all
        NS TYPE   PATH                NPROCS    PID   PPID COMMAND    UID USER NETNSID NSFS
4026531835 cgroup /proc/1/ns/cgroup      120      1      0 /sbin/init   0 root
4026531837 user   /proc/1/ns/user        120      1      0 /sbin/init   0 root
4026532186 mnt    /proc/230973/ns/mnt      1 230973 230955 sh           0 root
4026532187 uts    /proc/230973/ns/uts      1 230973 230955 sh           0 root
4026532188 ipc    /proc/230973/ns/ipc      1 230973 230955 sh           0 root
4026532189 pid    /proc/230973/ns/pid      1 230973 230955 sh           0 root
4026532191 net    /proc/230973/ns/net      1 230973 230955 sh           0 root       0 /run/docker/netns/904e1ae9696c
```

## Cgroups

所属している Cgroup はこちら

```sh
$ cat /proc/$CPID/cgroup
12:freezer:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
11:devices:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
10:blkio:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
9:rdma:/
8:pids:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
7:memory:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
6:hugetlb:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
5:net_cls,net_prio:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
4:cpu,cpuacct:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
3:cpuset:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
2:perf_event:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
1:name=systemd:/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0
0::/system.slice/containerd.service
```

そして Cgroup の内容もちょっと見に行きましよう

```sh
$ cgget -g pids:/docker/$CID
/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0:
pids.current: 1
pids.events: max 0
pids.max: max
$ cgget -g net_cls,net_prio:/docker/$CID
/docker/b6431f14cf40267d3eed22b34fc6e974be28f2e0f5b9b2bfbccdffaa5327a4a0:
net_cls.classid: 0
net_prio.prioidx: 3
net_prio.ifpriomap: lo 0
	enp0s3 0
	docker0 0
	veth18981d6 0
```

## Mounts

このContainer プロセスがマウントしたディレクトリはこちら

```sh
$ cat /proc/$CPID/mounts
overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/JVWDDYIF5YQFGNGL3PDNWJ4A4M:/var/lib/docker/overlay2/l/TQBVPLJN6SEMDRW7VD5FBARG4F,upperdir=/var/lib/docker/overlay2/ae3eab82db4efee497c2d69a4ad18a8cfc816ce0d61296fee7b6f611d7f6ebb3/diff,workdir=/var/lib/docker/overlay2/ae3eab82db4efee497c2d69a4ad18a8cfc816ce0d61296fee7b6f611d7f6ebb3/work,xino=off 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
cgroup /sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup ro,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/memory cgroup ro,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/pids cgroup ro,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/rdma cgroup ro,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
/dev/sda1 /etc/resolv.conf ext4 rw,relatime 0 0
/dev/sda1 /etc/hostname ext4 rw,relatime 0 0
/dev/sda1 /etc/hosts ext4 rw,relatime 0 0
devpts /dev/console devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
proc /proc/bus proc ro,relatime 0 0
proc /proc/fs proc ro,relatime 0 0
proc /proc/irq proc ro,relatime 0 0
proc /proc/sys proc ro,relatime 0 0
proc /proc/sysrq-trigger proc ro,relatime 0 0
tmpfs /proc/acpi tmpfs ro,relatime 0 0
tmpfs /proc/kcore tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/keys tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/timer_list tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/sched_debug tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/scsi tmpfs ro,relatime 0 0
tmpfs /sys/firmware tmpfs ro,relatime 0 0
```

ここで注目するのは overlay の情報

```
overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/JVWDDYIF5YQFGNGL3PDNWJ4A4M:/var/lib/docker/overlay2/l/TQBVPLJN6SEMDRW7VD5FBARG4F,upperdir=/var/lib/docker/overlay2/ae3eab82db4efee497c2d69a4ad18a8cfc816ce0d61296fee7b6f611d7f6ebb3/diff,workdir=/var/lib/docker/overlay2/ae3eab82db4efee497c2d69a4ad18a8cfc816ce0d61296fee7b6f611d7f6ebb3/work,xino=off 0 0
```

ここでマウントした`/var/lib/docker/overlay2/l/TQBVPLJN6SEMDRW7VD5FBARG4F`は実は alpine イメージの内容です

```sh
$ cat $(docker image inspect alpine -f '{{ .GraphDriver.Data.UpperDir }}')/../link
TQBVPLJN6SEMDRW7VD5FBARG4F
```

## ネットワーク

最後にネットワークの設定も確認しましよう。Container プロセスは独自の network namespace に区別しているから、host mahcineと通信のためには、[veth](https://man7.org/linux/man-pages/man4/veth.4.html)を利用します。

まずは、Container側のネットワーク設定

```sh
$ nsenter -t $CPID -n ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 17  bytes 1366 (1.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
$ nsenter -t $CPID -n ip link show type veth
280: eth0@if281: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
$ nsenter -t $CPID -n ip route show dev eth0
default via 172.17.0.1
172.17.0.0/16 proto kernel scope link src 172.17.0.2
```

host machine 側

```sh
$ ifconfig docker0
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:ffff:fe14:4bd6  prefixlen 64  scopeid 0x20<link>
        ether 02:42:ff:14:4b:d6  txqueuelen 0  (Ethernet)
        RX packets 8  bytes 433 (433.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 283  bytes 30567 (30.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
$ ip addr show type veth
281: veth18981d6@if280: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether fa:eb:12:ad:57:d1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::f8eb:12ff:fead:57d1/64 scope link
       valid_lft forever preferred_lft forever
$ ip route show dev docker0
172.17.0.0/16 proto kernel scope link src 172.17.0.1
```

Container のアドレスは`172.17.0.2`で、hostの方は`172.17.0.1`、`veth18981d6`という veth で２つの interface を繋がている。そしてそれぞれの route table に転送経路のルールが載せている。
 

# `bpftrace`でDocker実行順番を明白する

さて、いろいろの結果を確認してきたから、具体的に Docker はどうやってその結果のたどり着いたか、例えばいつにどんな関数をどんな変数で呼びましたか。ここでは[bpftrace](https://github.com/iovisor/bpftrace)が活躍するところです。

bpftrace っていうのは Linux Kenel の eBPF 機能を awk みたい記述で簡単に利用できるツールです。具体的な使用方法はここでは省きますが、[Doc](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md)に参照してください。

## `bpftrace`のセットアップ

```sh
# bcc
$ apt-get install -y linux-headers-$(uname -r) bison build-essential cmake flex g++ git libelf-dev zlib1g-dev libfl-dev systemtap-sdt-dev binutils-dev llvm-8-dev llvm-8-runtime libclang-8-dev clang-8 arping netperf iperf3 python3-distutils
$ git clone --recurse-submodules https://github.com/iovisor/bcc.git
$ mkdir bcc/build; cd bcc/build
$ cmake -DPYTHON_CMD=python3 ..
$ make -j8 && make install && ldconfig

$ cd ../..

# bpftrace
$ git clone https://github.com/iovisor/bpftrace.git
$ mkdir bpftrace/build; cd bpftrace/build
$ cmake -DHAVE_BCC_PROG_LOAD=ON -DHAVE_BCC_CREATE_MAP=ON -DBUILD_TESTING=OFF ..
$ make -j8 && make install
```

## Dockerをトレース

bpftraceで docker と OS のやり取りがトレースすることができる。まずはこちらの内容で`docker.bt`のファイル作成する

```awk:docker.bt
#!/usr/bin/env bpftrace

#include <linux/nsproxy.h>
#include <linux/ns_common.h>
#include <linux/utsname.h>
#include <linux/pid_namespace.h>
#include <linux/ipc_namespace.h>
#include <linux/cgroup.h>
#include <net/net_namespace.h>
#include <linux/netdevice.h>

BEGIN { 
    printf("%-12s %-15s %-8s %-10s %s\n", "TIME", "COMMAND", "PID", "ACTION", "CONTENT"); 
}

tracepoint:syscalls:sys_enter_execve { 
    printf("%-12ld %-15s %-8d %-10s ", elapsed , comm, pid, "execve");
    join(args->argv); 
}

tracepoint:syscalls:sys_enter_mount { 
    printf("%-12ld %-15s %-8d %-10s ", elapsed, comm, pid, "mount");
    printf("type=%s, dev=%s, dir=%s\n", str(args->type), str(args->dev_name), str(args->dir_name))
}

kretprobe:create_new_namespaces /comm=="runc:[1:CHILD]"/ { 
    printf("%-12ld %-15s %-8d %-10s ", elapsed, comm, pid, "namespace");
    $nsp = (struct nsproxy *)retval;
    printf("uts=%ld, ipc=%ld, cgroup=%ld, net=%ld, pid=%ld\n", 
        ((struct uts_namespace *)$nsp->uts_ns)->ns.inum,
        ((struct ipc_namespace *)$nsp->ipc_ns)->ns.inum,
        ((struct cgroup_namespace *)$nsp->cgroup_ns)->ns.inum,
        ((struct net *)$nsp->net_ns)->ns.inum,
        ((struct pid_namespace *)$nsp->pid_ns_for_children)->ns.inum);
}

kprobe:veth_newlink {
    printf("%-12ld %-15s %-8d %-10s ", elapsed, comm, pid, "veth");
    printf("name=%s, netns=%ld\n", ((struct net_device *)arg1)->name, ((struct net *)arg0)->ns.inum);
}
```

そして実行する

```sh
$ ./docker.bt
Attaching 5 probes...
TIME         COMMAND         PID      ACTION     CONTENT
```

もう一度 Container の起動する（別のターミナル）

```sh
$ docker run -ti --rm --name test alpine echo hi
```

`docker.bt`に戻せば、こちらの内容出力されます

```
TIME         COMMAND         PID      ACTION     CONTENT
1827725094   bash            235026   execve     docker run -ti --rm --name test alpine echo hi
1874368200   dockerd         81563    mount      type=overlay, dev=overlay, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
1892118900   dockerd         81563    mount      type=overlay, dev=overlay, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
1904728454   dockerd         81563    mount      type=overlay, dev=overlay, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
1905395722   dockerd         81563    veth       name=veth3d71f6e, netns=4026531992
1912608987   networkd-dispat 235038   execve     /usr/bin/networkctl list --no-pager --no-legend
1913900866   (spawn)         235036   execve     /lib/udev/bridge-network-interface
1917882282   (spawn)         235039   execve     /lib/open-iscsi/net-interface-handler start
1922126799   (spawn)         235040   execve     /lib/udev/bridge-network-interface
1923736757   (spawn)         235041   execve     /lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/veth3d71f6e --prefix=/net/ipv4/neigh/veth3d71f6e --prefix=/net/ipv6/conf/veth3d71f6e --prefix=/net/ipv6/neigh/veth3d71f6e
1925374081   (spawn)         235042   execve     /lib/open-iscsi/net-interface-handler start
1935733149   containerd      235044   execve     containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
1940074927   containerd-shim 235053   execve     runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb --pid-file /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb/init.pid --console-socket /tmp/pty046813817/pty.sock 6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb
1940243736   (spawn)         235043   execve     /lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/vethc30992d --prefix=/net/ipv4/neigh/vethc30992d --prefix=/net/ipv6/conf/vethc30992d --prefix=/net/ipv6/neigh/vethc30992d
1955997514   runc            235060   execve     runc init
1959669403   exe             235060   mount      type=, dev=/proc/self/exe, dir=/var/run/docker/runtime-runc/moby/6975e3344ad32b92a6a09066f3113
1962524451   exe             235060   mount      type=, dev=, dir=/var/run/docker/runtime-runc/moby/6975e3344ad32b92a6a09066f3113
1977110103   runc:[1:CHILD]  235061   namespace  uts=4026532187, ipc=4026532188, cgroup=4026531835, net=4026532191, pid=4026532189
2002233022   runc:[2:INIT]   235062   mount      type=, dev=, dir=/
2002647666   runc:[2:INIT]   235062   mount      type=bind, dev=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2002787106   runc:[2:INIT]   235062   mount      type=proc, dev=proc, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2002848509   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2003072500   runc:[2:INIT]   235062   mount      type=devpts, dev=devpts, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2006987497   runc:[2:INIT]   235062   mount      type=sysfs, dev=sysfs, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2007808775   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008001937   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/systemd/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008021032   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/systemd/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008102923   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/perf_event/docker/6975e3344ad32b92a6a09066f31132, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008116742   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/perf_event/docker/6975e3344ad32b92a6a09066f31132, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008189722   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/cpuset/docker/6975e3344ad32b92a6a09066f31132f9b4, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008203301   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/cpuset/docker/6975e3344ad32b92a6a09066f31132f9b4, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008284557   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/cpu,cpuacct/docker/6975e3344ad32b92a6a09066f3113, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008300251   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/cpu,cpuacct/docker/6975e3344ad32b92a6a09066f3113, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008365239   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/net_cls,net_prio/docker/6975e3344ad32b92a6a09066, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008397968   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/net_cls,net_prio/docker/6975e3344ad32b92a6a09066, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008471763   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/hugetlb/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008489627   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/hugetlb/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008575004   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/memory/docker/6975e3344ad32b92a6a09066f31132f9b4, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008588697   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/memory/docker/6975e3344ad32b92a6a09066f31132f9b4, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008655100   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/pids/docker/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008667811   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/pids/docker/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008731445   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/rdma, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008744138   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/rdma, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008792554   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/blkio/docker/6975e3344ad32b92a6a09066f31132f9b4e, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008805034   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/blkio/docker/6975e3344ad32b92a6a09066f31132f9b4e, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008855551   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/devices/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008868073   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/devices/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008919459   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/freezer/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008932321   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup/freezer/docker/6975e3344ad32b92a6a09066f31132f9b, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008958905   runc:[2:INIT]   235062   mount      type=bind, dev=/sys/fs/cgroup, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2008977832   runc:[2:INIT]   235062   mount      type=mqueue, dev=mqueue, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009006418   runc:[2:INIT]   235062   mount      type=tmpfs, dev=shm, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009102662   runc:[2:INIT]   235062   mount      type=bind, dev=/var/lib/docker/containers/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009116671   runc:[2:INIT]   235062   mount      type=, dev=, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009152544   runc:[2:INIT]   235062   mount      type=bind, dev=/var/lib/docker/containers/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009184117   runc:[2:INIT]   235062   mount      type=, dev=, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009256053   runc:[2:INIT]   235062   mount      type=bind, dev=/var/lib/docker/containers/6975e3344ad32b92a6a09066f31132f9b4e3, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2009268672   runc:[2:INIT]   235062   mount      type=, dev=, dir=/var/lib/docker/overlay2/ec1fbee3aa02f9596794e45afddaddf75028e0
2012566089   runc            235085   execve     libnetwork-setkey -exec-root=/var/run/docker 6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb ee4af79b8952
2055580696   dockerd         81563    mount      type=bind, dev=/proc/235062/ns/net, dir=/var/run/docker/netns/53666134ee2d
2058245033   dockerd         235093   execve     set-ipv6 /var/run/docker/netns/53666134ee2d all false
2117481953   (spawn)         235100   execve     /lib/open-iscsi/net-interface-handler stop
2135647000   runc:[2:INIT]   235062   mount      type=, dev=, dir=.
2135881831   runc:[2:INIT]   235062   mount      type=bind, dev=/dev/pts/0, dir=/dev/console
2135959898   runc:[2:INIT]   235062   mount      type=, dev=/proc/bus, dir=/proc/bus
2135974823   runc:[2:INIT]   235062   mount      type=, dev=/proc/bus, dir=/proc/bus
2135984627   runc:[2:INIT]   235062   mount      type=, dev=/proc/fs, dir=/proc/fs
2135995698   runc:[2:INIT]   235062   mount      type=, dev=/proc/fs, dir=/proc/fs
2136004311   runc:[2:INIT]   235062   mount      type=, dev=/proc/irq, dir=/proc/irq
2136017445   runc:[2:INIT]   235062   mount      type=, dev=/proc/irq, dir=/proc/irq
2136025919   runc:[2:INIT]   235062   mount      type=, dev=/proc/sys, dir=/proc/sys
2136036802   runc:[2:INIT]   235062   mount      type=, dev=/proc/sys, dir=/proc/sys
2136045583   runc:[2:INIT]   235062   mount      type=, dev=/proc/sysrq-trigger, dir=/proc/sysrq-trigger
2136057589   runc:[2:INIT]   235062   mount      type=, dev=/proc/sysrq-trigger, dir=/proc/sysrq-trigger
2136066994   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/asound
2136076692   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/acpi
2136145362   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/proc/acpi
2136198403   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/kcore
2136211095   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/keys
2136271650   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/latency_stats
2136281142   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/timer_list
2136292991   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/timer_stats
2136302265   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/sched_debug
2136312542   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/proc/scsi
2136411156   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/proc/scsi
2136458345   runc:[2:INIT]   235062   mount      type=, dev=/dev/null, dir=/sys/firmware
2136537743   runc:[2:INIT]   235062   mount      type=tmpfs, dev=tmpfs, dir=/sys/firmware
2325058622   containerd-shim 235102   execve     runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb/log.json --log-format json start 6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb
2334946680   runc:[2:INIT]   235062   execve     echo hi
2357752100   containerd-shim 235108   execve     /usr/bin/containerd --address /run/containerd/containerd.sock publish --topic /tasks/exit --namespace moby
2389193617   containerd-shim 235116   execve     runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb/log.json --log-format json delete 6975e3344ad32b92a6a09066f31132f9b4e3d5c65e1b827ed91486f1128c17eb
2438983541   (spawn)         235129   execve     /lib/udev/bridge-network-interface
2441125682   networkd-dispat 235130   execve     /usr/bin/networkctl list --no-pager --no-legend
2441917196   (spawn)         235131   execve     /lib/open-iscsi/net-interface-handler start
2445708579   (spawn)         235132   execve     /lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/vethc30992d --prefix=/net/ipv4/neigh/vethc30992d --prefix=/net/ipv6/conf/vethc30992d --prefix=/net/ipv6/neigh/vethc30992d
2460630127   networkd-dispat 235133   execve     /usr/bin/networkctl list --no-pager --no-legend
2469750858   (spawn)         235134   execve     /lib/open-iscsi/net-interface-handler stop
2470070134   (spawn)         235135   execve     /lib/open-iscsi/net-interface-handler stop
```

これは Docker がこの Container を実行するためにやったことは一目瞭然です。簡単にまとめると、順番的にはこちらです

* イメージの overlay をマウントする
* veth などのネットワーク設定
* `containerd-shim`を実行
* `runc init`
* namespaces 作り
* cgroup などにいろんなマウント
* `runc start`
* ユーザーコマンド `echo hi`
* `runc delete`
* ネットワークの cleanup

# 参考資料

* [First steps in system-wide Linux tracing · Martin Pitt](https://piware.de/post/2020-02-28-bpftrace/)