Project: rkt (Runtime / Container Runtime)
File: troubleshooting.md

This document lists common rkt problems and how to fix or work around them. When checking the logs of a container, they might be missing with an error like this: ``` $ journalctl -M rkt-3f045be0-1632-42f1-ba15-df984a82636f Journal file /var/lib/rkt/pods/run/3f045be0-1632-42f1-ba15-df984a82636f/stage1/rootfs/var/log/journal/3f045be0163242f1ba15df984a82636f/system.journal uses an unsupported feature, ignoring file. -- No entries -- ``` This is because rkt's journald integration is only supported if systemd is compiled with `lz4` compression enabled. You can check if it is enabled by making sure you see `+LZ4` when running `systemctl --version`: ``` $ systemctl --version systemd 235 [...] +LZ4 [...] ``` During rkt execution, you might encounter the message `Bad system call` followed by rkt terminating. It's most likely a result of a too restrictive seccomp profile. As a workaround, you can disable seccomp with `--insecure-options=seccomp`. As a proper fix, you can . During rkt execution, you might encounter a `Operation not permitted` message followed by rkt exiting. Your image probably uses more capabilities than allowed in rkt's default list. As a workaround, you can disable capabilities enforcement with `--insecure-options=capabilities`. As a proper fix, you can . ``` prepare-app@opt-stage2-alpine\x2dsh-rootfs.service: Job prepare-app@opt-stage2-alpine\x2dsh-rootfs.service/start failed with result 'dependency'. systemd-journald.service: Unit entered failed state. systemd-journald.service: Failed with result 'signal'. systemd-journald.service: Service has no hold-off time, scheduling restart. ``` To solve this update to Linux 4.5.2 or newer (see ). You might se an error like this one when starting a rkt pod: ``` /usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Permission denied ``` The overlay filesystem doesn't work with SELinux in kernels older than 4.9 (see ). Please update your kernel to a newer version. You might see messages like these when running `rkt gc`: ``` Unable to remove pod "42e78965-c60b-4f4f-b412-484cd381fe90": remove /var/lib/rkt/pods/exited-garbage/42e78965-c60b-4f4f-b412-484cd381fe90/stage1/rootfs: device or resource busy ``` This might be due to using a kernel older than 3.18 (see and ). Please update your kernel to a newer version. Due to limitations in the Linux kernel, using rkt's overlay support on top of an overlay filesystem requires the upperdir and workdir to support the creation of trusted.* extended attributes and valid d_type in readdir responses (see ). The symptom is an error message like this: ``` stage0: error setting up stage1 error rendering overlay filesystem problem mounting overlay filesystem error mounting overlay with options 'lowerdir=/var/lib/rkt/cas/tree/deps-sha512-f3d5f69d7faba1be7067d610f33131c18ac59eb43b1495016ade65bd13912578/rootfs,upperdir=/var/lib/rkt/pods/run/307bd207-7eab-4028-8d12-2d525e5b8ed9/overlay/deps-sha512-f3d5f69d7faba1be7067d610f33131c18ac59eb43b1495016ade65bd13912578/upper,workdir=/var/lib/rkt/pods/run/307bd207-7eab-4028-8d12-2d525e5b8ed9/overlay/deps-sha512-f3d5f69d7faba1be7067d610f33131c18ac59eb43b1495016ade65bd13912578/work' and dest '/var/lib/rkt/pods/run/307bd207-7eab-4028-8d12-2d525e5b8ed9/stage1/rootfs' invalid argument ``` This problem typically happens when trying to run rkt inside rkt. To successfully run rkt inside rkt, use one of the following workarounds: set up `/var/lib/rkt` in the outer rkt as a host volume use `--no-overlay` for either the outer or the inner rkt