Project: rkt (Runtime / Container Runtime)
File: debugging.md

This guide gives an overview of the debugging techniques that can be used to pinpoint problems with rkt itself or with particular images. is the most straight-forward technique. It allows you to enter the namespaces of an app within a pod so you can explore its filesystem and see what's running. By default it tries to run `/bin/bash` if it's present in the app's filesystem but you can specify a command to run if the app doesn't provide a bash binary. ```bash $ rkt list UUID APP IMAGE NAME STATE CREATED STARTED NETWORKS 54ca5d22 busybox kinvolk.io/aci/busybox:1.24 running 1 hour ago 1 hour ago default:ip4=172.16.28.114 $ sudo rkt enter 54ca5d22 sh / # ls bin dev etc home proc root run sys tmp usr var / # ps PID USER TIME COMMAND 1 root 0:00 /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 3 root 0:00 /usr/lib/systemd/systemd-journald 6 root 0:00 /bin/sh 40 root 0:00 sh 44 root 0:00 ps ``` This works well for images that have a shell baked in but a common practice is creating minimal images with only a statically compiled binary or a binary plus its needed libraries, which limits the usefulness of `rkt enter`. We'll later see how to overcome this limitation. As explained in , execution with rkt is divided into several distinct stages. While with `rkt enter` you enter stage2, it is sometimes useful to see what happens in stage1. For example, you might want to check what options are passed to the unit file of an app, or interact with stage1 systemd. rkt doesn't include a subcommand you can use to enter stage1, but you can use for that purpose. Note that this only applies for container stage1 images: kvm pods run in an isolated VM and `nsenter` can't enter its stage1, and fly only allows one app per pod and it doesn't have long-running processes on stage1 so there's nothing to enter. Let's enter a pod's stage1. We'll need to find stage1 systemd's PID, which is the child of the PID shown in `rkt status`. Then we can just call nsenter with it: ```bash $ rkt status 86e6df38 state=running created=2017-12-19 14:49:00.376 +0100 CET started=2017-12-19 14:49:00.466 +0100 CET networks=default:ip4=172.16.28.27 pid=8469 exited=false $ ps auxf | grep -A1 [8]469 root 8469 0.1 0.0 54204 5076 pts/1 S+ 14:48 0:00 \ stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=86e6df38-0762-4261-af21-d2b265555179 --machine=rkt-86e6df38-0762-4261-af21-d2b265555179 --directory=stage1/rootfs --capability=CAPAUDITWRITE,CAPCHOWN,CAPDACOVERRIDE,CAPFSETID,CAPFOWNER,CAPKILL,CAPMKNOD,CAPNETRAW,CAPNETBINDSERVICE,CAPSETUID,CAPSETGID,CAPSETPCAP,CAPSETFCAP,CAPSYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0 root 8505 0.0 0.0 62276 7300 ? Ss 14:48 0:00 \_ /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 $ sudo nsenter -m -u -i -p -t 8505 -bash-4.3# ``` However, the stage1 environment doesn't include basic binaries necessary for debugging: ```bash -bash-4.3# ls -bash: ls: command not found -bash-4.3# cat -bash: cat: command not found ``` If you only need to inspect the stage1 filesystem, you can check `/proc/$PID/root/` from the host: ```bash appadd appstart attach dev enter etc gc iottymux lib64 prepare-app reaper.sh root stop systemd-version usr apprm appstop bin diagnostic enterexec flavor init lib opt proc rkt run sys tmp var ``` Luckily, rkt includes a helper script that injects a static busybox binary and makes entering stage1 easier. It can be found in `scripts/debug/stage1installbusybox.sh`. It takes a pod UUID as a parameter and it outputs the right nsenter command: ```bash $ rkt list --full UUID APP IMAGE NAME IMAGE ID STATE CREATED STARTED NETWORKS 86e6df38-0762-4261-af21-d2b265555179 busybox kinvolk.io/aci/busybox:1.24 sha512-140375b2a2bd running 2017-12-19 14:49:00.376 +0100 CET 2017-12-19 14:49:00.466 +0100 CET default:ip4=172.16.28.27 $ ./scripts/debug/stage1installbusybox.sh 86e6df38-0762-4261-af21-d2b265555179 Busybox

Use the following command to enter pod's stage1: sudo nsenter -m -u -i -p -t 8505 -bash-4.3# cat rkt/env/busybox SHELL=/bin/sh USER=root LOGNAME=root HOME=/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ACAPPNAME=busybox ``` By default, the script creates links for common busybox programs, but if you need to run more commands you can call busybox accordingly: ```bash -bash-4.3# busybox head -n2 /lib/systemd/system/busybox.service [Unit] After=systemd-journald.service ``` Run `busybox` without arguments to see a list of available commands. If you need custom tools you can add them by copying them to `/proc/$PID/root/bin`. It's easier if they're static binaries to avoid complications with dynamic libraries. As mentioned, `rkt enter` has limitations when the image to debug doesn't ship with tools to debug it or even a shell. One possible way to debug those images is to use the ACI dependencies mechanism to add a custom tools image to your container. Let's use to patch the manifest. ```bash $ rkt fetch debug-image.aci sha512-847f2ca60e473121311fac62d014a4f1 $ actool cat-manifest --pretty-print image-to-patch.aci > manifest.json ``` We'll now add the dependency to the manifest, right before the "app" section: ```json ], "dependencies": [ { "imageID": "sha512-847f2ca60e473121311fac62d014a4f1" } ], "app": { ``` Then we'll patch the image and run it: ```bash $ actool patch-manifest --manifest manifest.json --replace image-to-patch.aci $ sudo rkt --insecure-options=image run --interactive image-to-patch.aci (...) ``` You'll have your favorite debugging tools available. For examples of how to build images, check out . Creating images with statically built binaries is preferred to avoid complications with shared libraries. As an alternative to adding a dependency in the ACI, you can also add a volume after a container has been started using : ```bash $ sudo mkdir /proc/$PID/root/opt/stage2/$APP_NAME/rootfs/tools $ machinectl MACHINE CLASS SERVICE OS VERSION ADDRESSES rkt-b7458bf5-8e19-40c1-a7dc-7c5aa25e4970 container rkt - - 172.16.28.27... 1 machines listed. $ sudo machinectl bind $PWD/tools /opt/stage2/$APP_NAME/rootfs/tools ``` Where `$PID` is still the PID of systemd in the pod. is a very powerful tool to inspect what applications are doing. For container stage1 images or fly, you can just attach strace to processes running inside pods like any other process on the host: Let's trace the shell process running in our example busybox container. We'll also follow forks to trace new processes created by the shell: ```bash $ ps auxf | grep -A3 [8]469 root 8469 0.1 0.0 54204 5076 pts/1 S+ 14:48 0:00 \ stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=86e6df38-0762-4261-af21-d2b265555179 --machine=rkt-86e6df38-0762-4261-af21-d2b265555179 --directory=stage1/rootfs --capability=CAPAUDITWRITE,CAPCHOWN,CAPDACOVERRIDE,CAPFSETID,CAPFOWNER,CAPKILL,CAPMKNOD,CAPNETRAW,CAPNETBINDSERVICE,CAPSETUID,CAPSETGID,CAPSETPCAP,CAPSETFCAP,CAPSYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0 root 8505 0.0 0.0 62276 7300 ? Ss 14:48 0:00 \_ /usr/lib/systemd/systemd --default-standard-output=tty --log-target=null --show-status=0 root 8507 0.0 0.0 66408 8740 ? Ss 14:48 0:00 \_ /usr/lib/systemd/systemd-journald root 8510 0.0 0.0 1212 672 pts/3 Ss+ 14:48 0:00 \_ /bin/sh $ sudo strace -f -p 8510 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WSTOPPED, NULL) = 2646 SIGCHLD {sisigno=SIGCHLD, sicode=CLDEXITED, sipid=2646, siuid=0, sistatus=0, siutime=0, sistime=0} ioctl(10, TIOCSPGRP, [6]) = 0 wait4(-1, 0x7ffc0226b918, WNOHANG|WSTOPPED, NULL) = -1 ECHILD (No child processes) wait4(-1, 0x7ffc0226b7b8, WNOHANG|WSTOPPED, NULL) = -1 ECHILD (No child processes) write(1, "This is the current date: ", 26) = 26 clone(strace: Process 12807 attached (...) ``` To find out more about strace, check out Julia Evans' . Sometimes strace is not enough to figure out problems with your applications. From version 7.10, it is possible to use to debug processes running in containers from the host. Same as with strace, you need to find the PID of the process you want to debug and then just run: ``` $ sudo gdb -p $PID ```