Linux Containers

Related articles
/Using VPNs
Cgroups
Docker
Incus
LXD
Podman
systemd-nspawn
Linux Containers
(LXC) is a userspace interface for the Linux kernel containment features, providing a method for
OS-level virtualization
, using
namespaces
,
cgroups
and other Linux kernel
capabilities(7)
on the LXC host.
lxc(7)
is considered something in the middle between a chroot and a full-fledged virtual machine.
Incus
or
LXD
can be used as a manager for LXC. This page deals with using LXC directly.
Alternatives for using containers comprise
systemd-nspawn
,
Docker
and
Podman
.
Privileged or unprivileged
LXC supports two types of containers:
privileged
and
unprivileged
.
On
privileged
containers, the root UID—UID 0—within the container is mapped to the root UID on a host.
Note
LXC developers consider
privileged
containers as
not root-safe
—see aforementioned links for more information.
On
unprivileged
containers, the root UID within the container is mapped to an unprivileged UID on the host, which makes it more difficult for a hack inside the container to lead to consequences on the host system.  In other words, if an attacker manages to escape the container, they should find themselves with limited or no rights on the host.
The Arch
linux
,
linux-lts
and
linux-zen
kernel packages currently provide out-of-the-box support for
unprivileged
containers. With the
linux-hardened
package,
unprivileged
containers are only available for the system  administrator; hence additional kernel configuration changes are required to enable user namespaces for normal users there.
This article contains information for users to run either type of container, but
additional steps
may be required in order to use
unprivileged
containers.
To illustrate the power of UID mapping, consider the output below from a running,
unprivileged
container.  Therein, we see the containerized processes owned by the containerized root user in the output of
ps
:
[root@unprivileged_container /]# ps -ef | head -n 5
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 17:49 ?        00:00:00 /sbin/init
root        14     1  0 17:49 ?        00:00:00 /usr/lib/systemd/systemd-journald
dbus        25     1  0 17:49 ?        00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
systemd+    26     1  0 17:49 ?        00:00:00 /usr/lib/systemd/systemd-networkd
On the host, however, those containerized root processes are actually shown to be running as the mapped user (ID>99999), rather than the host's actual root user:
[root@host /]# lxc-info -Ssip --name sandbox
State:          RUNNING
PID:            26204
CPU use:        10.51 seconds
BlkIO use:      244.00 KiB
Memory use:     13.09 MiB
KMem use:       7.21 MiB
[root@host /]# ps -ef | grep 26204 | head -n 5
UID        PID  PPID  C STIME TTY          TIME CMD
100000   26204 26200  0 12:49 ?        00:00:00 /sbin/init
100000   26256 26204  0 12:49 ?        00:00:00 /usr/lib/systemd/systemd-journald
100081   26282 26204  0 12:49 ?        00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
100000   26284 26204  0 12:49 ?        00:00:00 /usr/lib/systemd/systemd-logind
Setup
Required software
Installing
lxc
and
arch-install-scripts
will allow the host system to run privileged lxcs.
Enable support to run unprivileged containers (optional)
Modify the configuration to contain the following lines:
/etc/lxc/default.conf
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
In other words, map a range of 65536 consecutive uids, starting from container-side uid 0, which shall be uid 100000 from the host’s point of view, up to and including container-side uid 65535, which the host will know as uid 165535. Apply that same mapping to gids.
Note
systemd
dynamically allocates several UID/GIDs in specific ranges for its own purposes (
systemd-homed
, the
DynamicUser=
option in systemd unit files,
systemd-nspawn
, etc). See
[1]
for a list of UID/GID ranges used by systemd.
Create or edit
both
subuid(5)
at
/etc/subuid
and
subgid(5)
at
/etc/subgid
to contain the mapping to the containerized uid/gid pairs for each user who shall be able to run the containers.  The example below is simply for the root user (and
systemd
system unit):
/etc/subuid
root:100000:65536
/etc/subgid
root:100000:65536
The factual accuracy of this article or section is disputed.
Reason:
Below instructions do not match with the provided reference
[2]
(Discuss in
Talk:Linux Containers#Unprivileged_containers_for_unprivileged_users
)
In addition, running unprivileged containers as an unprivileged user only works if you delegate a
cgroup
in advance (the cgroup2 delegation model enforces this restriction, not liblxc). Use the following
systemd
command to delegate the cgroup (per
LXC - Getting started: Creating unprivileged containers as a user
):
$ systemd-run --unit=
myshell
--user --scope -p "Delegate=yes" lxc-start
container_name
This works similarly for other lxc commands.
This article or section is a candidate for merging with
cgroups#User delegation
.
Notes:
Not specific to Linux Containers, avoid duplication. (Discuss in
Talk:Linux Containers
)
Alternatively, delegate
unprivileged
cgroups
by creating a
systemd
unit (per
Rootless Containers: Enabling CPU, CPUSET, and I/O delegation
):
/etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=cpu cpuset io memory pids
Unprivileged containers on linux-hardened and custom kernels
Users wishing to run
unprivileged
containers on
linux-hardened
or their custom kernel need to complete several additional setup steps.
Firstly, a kernel is required that has support for user namespaces (a kernel with
CONFIG_USER_NS
). All Arch Linux kernels have support for
CONFIG_USER_NS
. However, due to more general security concerns, the
linux-hardened
kernel does ship with user namespaces enabled only for the
root
user. There are two options to create
unprivileged
containers there:
Start the unprivileged containers only as
root
. Also give the
sysctl
setting
user.max_user_namespaces
a positive value to suit your environment if its current value is
0
(this fixes
Failed to clone process in new user namespace
errors seen in
lxc info --show-log
container_name
).
Under
linux-hardened
&
lxd 5.0.0
you may need to set
/etc/subuid
&
/etc/subgid
to use a range of
root:1000000:65536
. You may also need to start your
first
container as privileged. This fixes error
newuidmap failed to write mapping "newuidmap: uid range [0-1000000000) -> [1000000-1001000000) not allowed"
Enable the
sysctl
setting
kernel.unprivileged_userns_clone
to allow normal users to run unprivileged containers. This can be done for the current session by running
sysctl kernel.unprivileged_userns_clone=1
as root and can be made permanent with
sysctl.d(5)
.
Host network configuration
LXC supports different virtual network types and devices (see
lxc.container.conf(5) § NETWORK
). A bridge device on the host is required for most types of virtual networking, which is illustrated in this section.
There are several main setups to consider:
A host bridge
A NAT bridge
The host bridge requires the host's network manager to manage a shared bridge interface.  The host and any lxc will be assigned an IP address in the same network (for example 192.168.1.x).  This might be more simplistic in cases where the goal is to containerize some network-exposed service like a webserver, or VPN server.  The user can think of the lxc as just another PC on the physical LAN, and forward the needed ports in the router accordingly.  The added simplicity can also be thought of as an added threat vector, again, if WAN traffic is being forwarded to the lxc, having it running on a separate range presents a smaller threat surface.
The NAT bridge does not require the host's network manager to manage the bridge.
lxc
ships with
lxc-net
which creates a NAT bridge called
lxcbr0
. The NAT bridge is a standalone bridge with a private network that is not bridged to the host's ethernet device or to a physical network. It exists as a private subnet in the host.
Using lxc-net "Virtual NAT Router"
Note
By default,
lxc
is shipped with the
lxc-net
shell script, which requires the optional dependency
dnsmasq
to be
Install
to function.
lxc-net
acts like a NAT router, similar to OpenWRT, and creates a
Network bridge
named
lxcbr0
by default. It also manages
nftables
or
iptables
for traffic forwarding. Most configurations for this tool can be modified in its configuration file located at
/etc/default/lxc-net
.
Enable lxc-net.service
/etc/default/lxc-net
USE_LXC_BRIDGE="true"
Now you can
start
the
lxc-net.service
and among other things the default Bridge
lxcbr0
should appear in
$ ip a
Advanced Setup
One can override some
lxc-net
defaults by
creating
/etc/default/lxc-net
file or
editing
/etc/default/lxc
file using the following template:
/etc/default/lxc-net
# Leave USE_LXC_BRIDGE as "true" if you want to use lxcbr0 for your
# containers.  Set to "false" if you'll use virbr0 or another existing
# bridge, or mavlan to your host's NIC.
USE_LXC_BRIDGE="true"

# If you change the LXC_BRIDGE to something other than lxcbr0, then
# you will also need to update your /etc/lxc/default.conf as well as the
# configuration (/var/lib/lxc/<container>/config) for any containers
# already created using the default config to reflect the new bridge
# name.
# If you have the dnsmasq daemon installed, you'll also have to update
# /etc/dnsmasq.d/lxc and restart the system wide dnsmasq daemon.
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"
# Uncomment the next line if you'd like to use a conf-file for the lxcbr0
# dnsmasq.  For instance, you can use 'dhcp-host=mail1,10.0.3.100' to have
# container 'mail1' always get ip address 10.0.3.100.
#LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf

# Uncomment the next line if you want lxcbr0's dnsmasq to resolve the .lxc
# domain.  You can then add "server=/lxc/10.0.3.1' (or your actual $LXC_ADDR)
# to your system dnsmasq configuration file (normally /etc/dnsmasq.conf,
# or /etc/NetworkManager/dnsmasq.d/lxc.conf on systems that use NetworkManager).
# Once these changes are made, restart the lxc-net and network-manager services.
# 'container1.lxc' will then resolve on your host.
#LXC_DOMAIN="lxc"
Tip
Make sure the bridge's IP range does not interfere with the local network. One way to pick an available IP address is to use one of the addresses already assigned dynamically to the container. This can be inspected using
lxc-ls -f
command.
Optionally, create a configuration file to manually define the IP address of any containers:
/etc/lxc/dnsmasq.conf
dhcp-host=playtime,10.0.3.100
Firewall considerations
Depending on which firewall the host machine is running, it might be necessary to allow inbound packets from
lxcbr0
to the host, and outbound packets from
lxcbr0
to traverse through the host to other networks.  To test this, try to bring up a container configured to use DHCP for its IP assignment and see if
lxc-net
is able to assign an IP address to the container (check with
lxc-ls -f
.  If no IP is assigned, the host's policies will need to be adjusted.
Tip
Multiple programs may influence the
nftables
on a single system beyond dedicated firewalls, including VPNs and other virtualization software. Run
nft list ruleset
to show all currently active rules.
Users of
ufw
can simply run the
following two lines
to enable this:
# ufw allow in on lxcbr0
# ufw route allow in on lxcbr0
Alternatively, users of
nftables
can modify
/etc/nftables.conf
(and reload it with
nft -f /etc/nftables.conf
; check if the config syntax is correct with
nft -cf /etc/nftables.conf
) to allow the container to have internet access (replace
"eth0"
with the device on your system that has internet access; list existing devices with
ip link
):
/etc/nftables.conf
table inet filter {
  chain input {
    ...
    iifname "lxcbr0" accept comment "Allow lxc containers"

    pkttype host limit rate 5/second counter reject with icmpx type admin-prohibited
    counter
  }
  chain forward {
    ...
    iifname "lxcbr0" oifname "eth0" accept comment "Allow forwarding from lxcbr0 to eth0"
    iifname "eth0" oifname "lxcbr0" accept comment "Allow forwarding from eth0 to lxcbr0"
  }
}
Additionally, since the container is running on the 10.0.3.x subnet, external access to services such as ssh, httpd, etc. will need to be actively forwarded to the lxc.  In principle, the firewall on the host needs to forward incoming traffic on the expected port on the container.
Example iptables rule
The goal of this rule is to allow ssh traffic to the lxc:
# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 2221 -j DNAT --to-destination 10.0.3.100:22
This rule forwards tcp traffic originating on port 2221 to the IP address of the lxc on port 22.
Note
Make sure to allow traffic on 2221/tcp on the host and to allow 22/tcp traffic on the lxc.
To ssh into the container from another PC on the LAN, one needs to ssh on port 2221 to the host.  The host will then forward that traffic to the container.
$ ssh -p 2221 host.lan
Example ufw rule
If using
ufw
, append the following at the bottom of
/etc/ufw/before.rules
to make this persistent:
/etc/ufw/before.rules
*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -i eth0 -p tcp --dport 2221 -j DNAT --to-destination 10.0.3.100:22
COMMIT
Running containers as non-root user
To create and start containers as a non-root user, extra configuration must be applied.
According to
lxc-usernet(5)
, the entry per line is:
/etc/lxc/lxc-usernet
user type bridge number
Configure the file with the user needing to create containers. The bridge will be the same as defined in
/etc/default/lxc-net
.
A copy of the
/etc/lxc/default.conf
is needed in the non-root user's home directory, e.g.
~/.config/lxc/default.conf
(create the directory if needed).
Running containers as a non-root user requires
+x
permissions on
~/.local/share/
. Make that change with
chmod
before starting a container.
Container creation
Containers are built using
lxc-create(1)
command.  With the release of lxc-3.0.0-1, upstream has deprecated locally stored templates.
To create an Arch container:
# lxc-create --name playtime --template download -- --dist archlinux --release current --arch amd64
To create a container by interactively choosing from a list of supported distributions:
# lxc-create -n playtime -t download
To see a list of
download
template options:
# lxc-create -t download --help
Tip
Users of
Btrfs
can append
-B btrfs
to create a Btrfs subvolume for storing containerized rootfs. This comes in handy if cloning containers with the help of
lxc-clone
command.
ZFS
users may use
-B zfs
, correspondingly.
Note
Users wanting the legacy templates can find them in
lxc-templates
AUR
or alternatively, users can build their own templates with
distrobuilder
.
Container configuration
The examples below can be used with
privileged
and
unprivileged
containers alike.  Note that for unprivileged containers, additional lines will be present by default, which are not shown in the examples, including the
lxc.idmap = u 0 100000 65536
and the
lxc.idmap = g 0 100000 65536
values optionally defined in the
#Enable support to run unprivileged containers (optional)
section.
Basic configuration with networking
Note
With the release of lxc-1:2.1.0-1, many of the configuration options have changed. Existing containers need to be updated; users are directed to the table of these changes in the
v2.1 release notes
.
Configurations specific to a container, including system resources to be virtualized/isolated when a process is using the container, are defined in
/var/lib/lxc/CONTAINER_NAME/config
. Read
lxc.container.conf(5)
for the syntax and possible options of the configuration file.
A basic configuration file generated when creating containers using templates. Read
lxc.conf(5)
for more information.
By default, the creation process will make a minimum setup without networking support.  Below is an example configuration with networking supplied by
lxc-net.service
:
/var/lib/lxc/playtime/config
# Template used to create this container: /usr/share/lxc/templates/lxc-archlinux
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = x86_64

# Container specific configuration
lxc.rootfs.path = dir:/var/lib/lxc/playtime/rootfs
lxc.uts.name = playtime

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = ee:ec:fa:e9:56:7d
Mounts within the container
One can create a host volume outside the container's
rootfs
and then mount that volume inside the container. This can be advantageous, for example if the same architecture is being containerized and one wants to share pacman packages between the host and container. Another example could be shared directories. Read
lxc.container.conf(5) § MOUNT POINTS
for more information. The general syntax is:
lxc.mount.entry = /var/cache/pacman/pkg var/cache/pacman/pkg none bind 0 0
Note
This will not work without filesystem permission modifications on the host if using
unprivileged
containers.
Xorg program considerations (optional)
In order to run programs on the host's display, some bind mounts need to be defined so that the containerized programs can access the host's resources.
/var/lib/lxc/playtime/config
## for xorg
lxc.mount.entry = /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry = /dev/snd dev/snd none bind,optional,create=dir
lxc.mount.entry = /tmp/.X11-unix tmp/.X11-unix none bind,optional,create=dir,ro
lxc.mount.entry = /dev/video0 dev/video0 none bind,optional,create=file
The factual accuracy of this article or section is disputed.
Reason:
Setting
xhost +
is very unsafe, use
cookie-based authentication
instead. (Discuss in
Talk:Linux Containers
)
If still experiencing a
permission denied
error in the LXC guest, call
xhost +
in the host to allow the guest to connect to the host's display server. Take note of the security concerns of opening up the display server by doing this.
In addition, add the following line
before
the above bind mount lines.
/var/lib/lxc/playtime/config
lxc.mount.entry = tmpfs tmp tmpfs defaults
VPN considerations
To run a containerized
OpenVPN
or
WireGuard
, see
Linux Containers/Using VPNs
.
Managing containers
Basic usage
To list all installed LXC containers:
# lxc-ls -f
Systemd can be used to
start
and to
stop
LXCs via
lxc@CONTAINER_NAME.service
.
Enable
lxc@CONTAINER_NAME.service
to have it start when the host system boots.
Users can also start/stop LXCs without systemd.
Start a container:
# lxc-start -n
CONTAINER_NAME
Stop a container:
# lxc-stop -n
CONTAINER_NAME
To login into a container:
# lxc-console -n
CONTAINER_NAME
Once logged, treat the container like any other linux system, set the root password, create users, install packages, etc.
To attach to a container:
# lxc-attach -n
CONTAINER_NAME
--clear-env
That works nearly the same as lxc-console, but it causes starts with a root prompt inside the container, bypassing login. Without the
--clear-env
flag, the host will pass its own environment variables into the container (including
$PATH
, so some commands will not work when the containers are based on another distribution).
Advanced usage
LXC clones
Users with a need to run multiple containers can simplify administrative overhead (user management, system updates, etc.) by using snapshots.  The strategy is to setup and keep up-to-date a single base container, then, as needed, clone (snapshot) it.  The power in this strategy is that the disk space and system overhead are truly minimized since the snapshots use an overlayfs mount to only write out to disk, only the differences in data.  The base system is read-only but changes to it in the snapshots are allowed via the overlayfs.
This article or section needs expansion.
Reason:
The note needs a reference. (Discuss in
Talk:Linux Containers
)
Note
overlayfs for unprivileged containers is not supported in the current mainline Arch Linux kernel due to security considerations.
For example, setup a container as outlined above.  We will call it "base" for the purposes of this guide.  Now create 2 snapshots of "base" which we will call "snap1" and "snap2" with these commands:
# lxc-copy -n base -N snap1 -B overlayfs -s
# lxc-copy -n base -N snap2 -B overlayfs -s
Note
If a static IP was defined for the "base" lxc, that will need to manually changed in the configuration for "snap1" and for "snap2" before starting them. If the process is to be automated, a script using sed can do this automatically although this is beyond the scope of this wiki section.
The snapshots can be started/stopped like any other container.  Users can optionally destroy the snapshots and all new data therein with the following command.  Note that the underlying "base" lxc is untouched:
# lxc-destroy -n snap1 -f
Systemd units and wrapper scripts to manage snapshots for
pi-hole
and
OpenVPN
are available to automate the process in
lxc-service-snapshots
.
Converting a privileged container to an unprivileged container
Once the system has been configured to use unprivileged containers (see
#Enable support to run unprivileged containers (optional)
), it is possible to make an existing privileged container run as unprivileged by editing the configuration file.
/var/lib/lxc/playtime/config
# Copy these from /etc/lxc/default.conf.
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
# Make the rootfs mount automatically map the UIDs, so they appear the same as before from inside the container.
lxc.rootfs.options = idmap=container
As an alternative to the
idmap=container
option, there is also a utility
uidmapshift
in
nsexec-bzr
AUR
which changes UIDs and GIDs in the container's rootfs.
Warning
It is recommended to backup the existing image before using this utility!
This utility will not shift UIDs and GIDs in
ACL
, users will need to shift them manually!
Invoke the utility to convert over like so:
# uidmapshift -b /var/lib/lxc/foo 0 100000 65536
Additional options are available simply by calling
uidmapshift
without any arguments.
Running Xorg programs
Either attach to or
SSH
into the target container and prefix the call to the program with the DISPLAY ID of the host's X session.  For most simple setups, the display is always 0.
An example of running Firefox from the container in the host's display:
$ DISPLAY=:0 firefox
Alternatively, to avoid directly attaching to or connecting to the container, the following can be used on the host to automate the process:
# lxc-attach -n playtime --clear-env -- sudo -u YOURUSER env DISPLAY=:0 firefox
Tips and tricks
Ping not working in an unprivileged container
In unprivileged containers, ping is likely to not work without an extra config step. Example error:
$ ping www.google.com
ping: socktype: SOCK_RAW
ping: socket: Operation not permitted
ping: -> missing cap_net_raw+p capability or setuid?
To fix this in container foo, on the host:
# lxc-attach -n foo -- chmod u+s /usr/bin/ping
Troubleshooting
Root login fails
If presented with following error upon trying to login using lxc-console:
login: root
Login incorrect
And the container's
journal
shows:
pam_securetty(login:auth): access denied: tty 'pts/0' is not secure !
Delete
/etc/securetty
[3]
and
/usr/share/factory/etc/securetty
on the
container
file system. Optionally add them to
NoExtract
in
/etc/pacman.conf
to prevent them from getting reinstalled. See
FS#45903
for details.
Alternatively, create a new user in lxc-attach and use it for logging in to the system, then switch to root.
# lxc-attach -n playtime
[root@playtime]# useradd -m -Gwheel newuser
[root@playtime]# passwd newuser
[root@playtime]# passwd root
[root@playtime]# exit
# lxc-console -n playtime
[newuser@playtime]$ su
No network-connection with veth in container config
If you cannot access your LAN or WAN with a networking interface configured as
veth
and setup through
/etc/lxc/
containername
/config
.
If the virtual interface gets the ip assigned and should be connected to the network correctly.
$ ip addr show veth0
inet 192.168.1.111/24
You may disable all the relevant static ip formulas and try setting the ip through the booted container-os like you would normaly do.
Example
container
/config
...
lxc.net.0.type = veth
lxc.net.0.name = veth0
lxc.net.0.flags = up
lxc.net.0.link =
bridge
...
And then assign the IP through a preferred method
inside
the container, see also
Network configuration#Network management
.
Error: unknown command
The error may happen when a basic command (
ls
,
cat
, etc.) on an attached container is typed hen a different Linux distribution is containerized relative to the host system (e.g. Debian container in Arch Linux host system). Upon attaching, use the argument
--clear-env
:
# lxc-attach -n
container_name
--clear-env
Error: Failed at step KEYRING spawning
Services in an unprivileged container may fail with the following message
Failed to change ownership of session keyring: Permission denied
Failed to set up kernel keyring: Permission denied
Failed at step KEYRING spawning ....: Permission denied
Create the following seccomp ruleset:
/etc/lxc/unpriv.seccomp
2
blacklist
[all]
keyctl errno 38
Then add the following line to the container configuration
after
lxc.idmap
:
/etc/lxc/
container
/config
lxc.seccomp.profile = /etc/lxc/unpriv.seccomp
Known issues
lxc-execute fails due to missing lxc.init.static
lxc-execute
fails with the error message
Unable to open lxc.init.static
. See
FS#63814
for details.
Starting containers using
lxc-start
works fine.
See also
Official Website
Getting Started Guide
Documentation
Forum
Official GitHub Repository
Security Guide
Ubuntu LXD documentation