Navigating Containers in the Linux Kernel
By: Chris Arges
Tags: Linux Containers, Security, Kubernetes, Linux, Infrastructure

The Technologies Behind Containers

Penguins at Shedd Aquarium in Chicago got to go on a field trip to meet the Beluga whales after it closed due to the coronavirus pandemic. Shedd Aquarium

Containers are an amalgamation of various technologies to limit resources, provide isolation, and mimic various aspects of a full system. Here I’ll go through the smorgasbord of various Linux technologies.

Cgroups

Control groups or cgroups is a Linux feature that allows the limiting of resources for a particular group of processes. Resources such as CPU, memory, and I/O can be limited using cgroups.

An example of the power of cgroups is to use docker run flags such as memory or cpus to limit resources when running a new container.

Enforcing Cgroup limits is important to ensure that a container cannot Denial-Of-Service the host machine.

Namespaces

Namespaces isolate a global operating system resource and are the heart of containers. New namespaces can be minted by using clone system call flags or using the unshare system call. Below are a summary of the difference name spaces and a short description of what they accomplish.

PID — Process IDs

Your container will have its own set of PIDs. Generally the root of the container will appear as PID 1 within the pid namespace.

Net — Network Interface, Routing Tables

This allows your container to have its own IP address and routing.

IPC — Message Queues, Shared Memory

Posix IPC and message queues are unique within a namespace.

Mount — Mount Paths

The path ‘/’ inside your container looks different than ‘/’ on your root filesystem. This provides path isolation and makes it easy for containerized apps to read from regular paths without disturbing the host filesystem.

User — Use IDs and Group IDs

This requires a mapping of root user namespace IDs to namespace IDs, but can be very effective in ensuring that UID 0 inside a container is different from UID 0 on your root namespace.

UTS — Hostname

This allows your container to have its own hostname.

User namespaces are not enabled by default in Docker, and can be the source of many security problems. For example if a container bind-mounts a sensitive directory into a container, and the user is UID 0, then that container user could potentially attack the host system. Detecting these sensitive mounts is something that can easily be detected by inspecting docker container configurations.

Overlay Filesystems

Overlayfs and AUFS allow for overlaying an ‘upper’ filesystem onto a ‘lower’ filesystem. This forms the basis of Docker’s layering and means that instead of having to copy an entire filesystem on each new container we only track changes within an ‘upper’ overlay. The ‘lower’ filesystem can even be another overlayfs filesystem which allows for stacking.

Linux Security Modules

Linux has a few Security Modules with AppArmor and SELinux being the most prominent. These security modules hook into various points in the Linux kernel and can perform actions such as allowing, denying, or logging events. Security modules implement Mandatory Access Control which supplements traditional UNIX Discretionary Access Control.

Apparmor allows a program to ship with a profile that describes which capabilities a program can access. For example, one could limit a program to only access files within a certain path, only read files within another path, limit network connections to a process, or even limit executing other processes entirely.

SELinux allows one to apply contexts to files and processes, and use policy to describe which programs and users can access which contexts. It has a very powerful syntax.

In docker a default profile is applied to all container processes, but it is easy to override it with the following syntax:

docker run --rm -it --security-opt apparmor=docker-default ubuntu

LSMs form a strong perimeter of defense, but the default configurations may be more permissive to allow for ease of use. Ideally you should run containers with the least set of access as possible; however this requires tuning to your specific application.

Ulimits

Each process can have ulimits applied to ensure it does not exceed certain operating system resource limits. For example, if I have a process that needs to open many files I may increase the number of open file descriptors a process is allowed to have:

docker run --ulimit nofile=9999:9999 --rm debian sh -c "ulimit -n"

Limiting container processes to sane ulimit defaults is important, otherwise containers may deny other applications access to operating system resources.

Capabilities

Traditional UNIX permissions only distinguish between a privileged ID of 0 and everybody else. Linux capabilities allow for more fine grained control of what a process is allowed to do. For example if I want a normal process to be able to execute network operations (CAP_NET_ADMIN) or sending signals to processes (CAP_KILL), these capabilities can be enabled or disabled for any particular process. Docker will use the LSM profile (apparmor,selinux) to form a basis for capabilities defaults; however capabilities can be configured using the cap-add and cap-drop flags. An example of using NET_ADMIN to create a dummy interface:

docker run --cap-add=NET_ADMIN --rm debian sh -c 'ip link add dummy0 type dummy; ip link show dev dummy0'

Seccomp

Seccomp or Secure Computing Mode is a kernel feature that is like a firewall for syscalls. You can log, allow, deny various syscalls and even provide some programmable filters that can check syscall arguments. Docker provides a default seccomp profile which provides some sane defaults; however you can also provide your own profile with the following:

docker run --rm -it --security-opt seccomp=/path/to/seccomp/profile.json ubuntu

This is yet another subsystem where you can limit specific syscalls and parameters to provide another layer of security for your container.