Chapter 23. Setting system resource limits for applications by using control groups Using the control groups ( cgroups ) kernel functionality, you can control resource usage of applications to use them more efficiently. You can use cgroups for the following tasks: Setting limits for system resource allocation. Prioritizing the allocation of hardware resources to specific processes. Isolating certain processes from obtaining hardware resources. 23.1. Introducing control groups Using the control groups Linux kernel feature, you can organize processes into hierarchically ordered groups - cgroups . You define the hierarchy (control groups tree) by providing structure to cgroups virtual file system, mounted by default on the /sys/fs/cgroup/ directory. The systemd service manager uses cgroups to organize all units and services that it governs. Manually, you can manage the hierarchies of cgroups by creating and removing sub-directories in the /sys/fs/cgroup/ directory. The resource controllers in the kernel then modify the behavior of processes in cgroups by limiting, prioritizing or allocating system resources, of those processes. These resources include the following: CPU time Memory Network bandwidth Combinations of these resources The primary use case of cgroups is aggregating system processes and dividing hardware resources among applications and users. This makes it possible to increase the efficiency, stability, and security of your environment. Control groups version 1 Control groups version 1 ( cgroups-v1 ) provide a per-resource controller hierarchy. Each resource, such as CPU, memory, or I/O, has its own control group hierarchy. You can combine different control group hierarchies in a way that one controller can coordinate with another in managing their respective resources. However, when the two controllers belong to different process hierarchies, the coordination is limited. The cgroups-v1 controllers were developed across a large time span, resulting in inconsistent behavior and naming of their control files. Control groups version 2 Control groups version 2 ( cgroups-v2 ) provide a single control group hierarchy against which all resource controllers are mounted. The control file behavior and naming is consistent among different controllers. Note cgroups-v2 is fully supported in RHEL 8.2 and later versions. For more information, see Control Group v2 is now fully supported in RHEL 8 . Additional resources Introducing kernel resource controllers The cgroups(7) manual page Role of systemd in control groups 23.2. Introducing kernel resource controllers Kernel resource controllers enable the functionality of control groups. RHEL 8 supports various controllers for control groups version 1 ( cgroups-v1 ) and control groups version 2 ( cgroups-v2 ). A resource controller, also called a control group subsystem, is a kernel subsystem that represents a single resource, such as CPU time, memory, network bandwidth or disk I/O. The Linux kernel provides a range of resource controllers that are mounted automatically by the systemd service manager. You can find a list of the currently mounted resource controllers in the /proc/cgroups file. Controllers available for cgroups-v1 : blkio Sets limits on input/output access to and from block devices. cpu Adjusts the parameters of the Completely Fair Scheduler (CFS) for a control group's tasks. The cpu controller is mounted together with the cpuacct controller on the same mount. cpuacct Creates automatic reports on CPU resources used by tasks in a control group. The cpuacct controller is mounted together with the cpu controller on the same mount. cpuset Restricts control group tasks to run only on a specified subset of CPUs and to direct the tasks to use memory only on specified memory nodes. devices Controls access to devices for tasks in a control group. freezer Suspends or resumes tasks in a control group. memory Sets limits on memory use by tasks in a control group and generates automatic reports on memory resources used by those tasks. net_cls Tags network packets with a class identifier ( classid ) that enables the Linux traffic controller (the tc command) to identify packets that originate from a particular control group task. A subsystem of net_cls , the net_filter (iptables), can also use this tag to perform actions on such packets. The net_filter tags network sockets with a firewall identifier ( fwid ) that allows the Linux firewall to identify packets that originate from a particular control group task (by using the iptables command). net_prio Sets the priority of network traffic. pids Sets limits for multiple processes and their children in a control group. perf_event Groups tasks for monitoring by the perf performance monitoring and reporting utility. rdma Sets limits on Remote Direct Memory Access/InfiniBand specific resources in a control group. hugetlb Limits the usage of large size virtual memory pages by tasks in a control group. Controllers available for cgroups-v2 : io Sets limits on input/output access to and from block devices. memory Sets limits on memory use by tasks in a control group and generates automatic reports on memory resources used by those tasks. pids Sets limits for multiple processes and their children in a control group. rdma Sets limits on Remote Direct Memory Access/InfiniBand specific resources in a control group. cpu Adjusts the parameters of the Completely Fair Scheduler (CFS) for a control group's tasks and creates automatic reports on CPU resources used by tasks in a control group. cpuset Restricts control group tasks to run only on a specified subset of CPUs and to direct the tasks to use memory only on specified memory nodes. Supports only the core functionality ( cpus{,.effective} , mems{,.effective} ) with a new partition feature. perf_event Groups tasks for monitoring by the perf performance monitoring and reporting utility. perf_event is enabled automatically on the v2 hierarchy. Important A resource controller can be used either in a cgroups-v1 hierarchy or a cgroups-v2 hierarchy, not simultaneously in both. Additional resources The cgroups(7) manual page Documentation in /usr/share/doc/kernel-doc-<kernel_version>/Documentation/cgroups-v1/ directory (after installing the kernel-doc package). 23.3. Introducing namespaces Namespaces create separate spaces for organizing and identifying software objects. This keeps them from affecting each other. As a result, each software object contains its own set of resources, for example, a mount point, a network device, or a a hostname, even though they are sharing the same system. One of the most common technologies that use namespaces are containers. Changes to a particular global resource are visible only to processes in that namespace and do not affect the rest of the system or other namespaces. To inspect which namespaces a process is a member of, you can check the symbolic links in the /proc/< PID >/ns/ directory. Table 23.1. Supported namespaces and resources which they isolate: Namespace Isolates Mount Mount points UTS Hostname and NIS domain name IPC System V IPC, POSIX message queues PID Process IDs Network Network devices, stacks, ports, etc User User and group IDs Control groups Control group root directory Additional resources The namespaces(7) and cgroup_namespaces(7) manual pages 23.4. Setting CPU limits to applications using cgroups-v1 To configure CPU limits to an application by using control groups version 1 ( cgroups-v1 ), use the /sys/fs/ virtual file system. Prerequisites You have root permissions. You have an application to restrict its CPU consumption installed on your system. You verified that the cgroups-v1 controllers are mounted: Procedure Identify the process ID (PID) of the application that you want to restrict in CPU consumption: The sha1sum example application with PID 6955 consumes a large amount of CPU resources. Create a sub-directory in the cpu resource controller directory: This directory represents a control group, where you can place specific processes and apply certain CPU limits to the processes. At the same time, a number of cgroups-v1 interface files and cpu controller-specific files will be created in the directory. Optional: Inspect the newly created control group: Files, such as cpuacct.usage , cpu.cfs._period_us represent specific configurations and/or limits, which can be set for processes in the Example control group. Note that the file names are prefixed with the name of the control group controller they belong to. By default, the newly created control group inherits access to the system's entire CPU resources without a limit. Configure CPU limits for the control group: The cpu.cfs_period_us file represents how frequently a control group's access to CPU resources must be reallocated. The time period is in microseconds (ms, "us"). The upper limit is 1 000 000 microseconds and the lower limit is 1000 microseconds. The cpu.cfs_quota_us file represents the total amount of time in microseconds for which all processes in a control group can collectively run during one period, as defined by cpu.cfs_period_us . When processes in a control group use up all the time specified by the quota during a single period, they are throttled for the remainder of the period and not allowed to run until the period. The lower limit is 1000 microseconds. The example commands above set the CPU time limits so that all processes collectively in the Example control group will be able to run only for 0.2 seconds (defined by cpu.cfs_quota_us ) out of every 1 second (defined by cpu.cfs_period_us ). Optional: Verify the limits: Add the application's PID to the Example control group: This command ensures that a specific application becomes a member of the Example control group and does not exceed the CPU limits configured for the Example control group. The PID must represent an existing process in the system. The PID 6955 here was assigned to the sha1sum /dev/zero & process, used to illustrate the use case of the cpu controller. Verification Verify that the application runs in the specified control group: The process of an application runs in the Example control group applying CPU limits to the application's process. Identify the current CPU consumption of your throttled application: Note that the CPU consumption of the PID 6955 has decreased from 99% to 20%. Note The cgroups-v2 counterpart for cpu.cfs_period_us and cpu.cfs_quota_us is the cpu.max file. The cpu.max file is available through the cpu controller. Additional resources Introducing kernel resource controllers cgroups(7) , sysfs(5) manual pages

Commands:
mount -l | grep cgroup tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
top top - 11:34:09 up 11 min, 1 user, load average: 0.51, 0.27, 0.22 Tasks: 267 total, 3 running, 264 sleeping, 0 stopped, 0 zombie %Cpu(s): 49.0 us, 3.3 sy, 0.0 ni, 47.5 id, 0.0 wa, 0.2 hi, 0.0 si, 0.0 st MiB Mem : 1826.8 total, 303.4 free, 1046.8 used, 476.5 buff/cache MiB Swap: 1536.0 total, 1396.0 free, 140.0 used. 616.4 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6955 root 20 0 228440 1752 1472 R 99.3 0.1 0:32.71 sha1sum 5760 jdoe 20 0 3603868 205188 64196 S 3.7 11.0 0:17.19 gnome-shell 6448 jdoe 20 0 743648 30640 19488 S 0.7 1.6 0:02.73 gnome-terminal- 1 root 20 0 245300 6568 4116 S 0.3 0.4 0:01.87 systemd 505 root 20 0 0 0 0 I 0.3 0.0 0:00.75 kworker/u4:4-events_unbound
mkdir /sys/fs/cgroup/cpu/Example/
ll /sys/fs/cgroup/cpu/Example/ -rw-r- r--. 1 root root 0 Mar 11 11:42 cgroup.clone_children -rw-r- r--. 1 root root 0 Mar 11 11:42 cgroup.procs -r- r- r--. 1 root root 0 Mar 11 11:42 cpuacct.stat -rw-r- r--. 1 root root 0 Mar 11 11:42 cpuacct.usage -r- r- r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_all -r- r- r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu -r- r- r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu_sys -r- r- r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu_user -r- r- r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_sys -r- r- r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_user -rw-r- r--. 1 root root 0 Mar 11 11:42 cpu.cfs_period_us -rw-r- r--. 1 root root 0 Mar 11 11:42 cpu.cfs_quota_us -rw-r- r--. 1 root root 0 Mar 11 11:42 cpu.rt_period_us -rw-r- r--. 1 root root 0 Mar 11 11:42 cpu.rt_runtime_us -rw-r- r--. 1 root root 0 Mar 11 11:42 cpu.shares -r- r- r--. 1 root root 0 Mar 11 11:42 cpu.stat -rw-r- r--. 1 root root 0 Mar 11 11:42 notify_on_release -rw-r- r--. 1 root root 0 Mar 11 11:42 tasks
echo "1000000" > /sys/fs/cgroup/cpu/Example/cpu.cfs_period_us echo "200000" > /sys/fs/cgroup/cpu/Example/cpu.cfs_quota_us
cat /sys/fs/cgroup/cpu/Example/cpu.cfs_period_us /sys/fs/cgroup/cpu/Example/cpu.cfs_quota_us 1000000 200000
echo "6955" > /sys/fs/cgroup/cpu/Example/cgroup.procs
cat /proc/6955/cgroup 12:cpuset:/ 11:hugetlb:/ 10:net_cls,net_prio:/ 9:memory:/user.slice/user-1000.slice/user@1000.service 8:devices:/user.slice 7:blkio:/ 6:freezer:/ 5:rdma:/ 4:pids:/user.slice/user-1000.slice/user@1000.service 3:perf_event:/ 2:cpu,cpuacct:/Example 1:name=systemd:/user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service
top top - 12:28:42 up 1:06, 1 user, load average: 1.02, 1.02, 1.00 Tasks: 266 total, 6 running, 260 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.0 us, 1.2 sy, 0.0 ni, 87.5 id, 0.0 wa, 0.2 hi, 0.0 si, 0.2 st MiB Mem : 1826.8 total, 287.1 free, 1054.4 used, 485.3 buff/cache MiB Swap: 1536.0 total, 1396.7 free, 139.2 used. 608.3 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6955 root 20 0 228440 1752 1472 R 20.6 0.1 47:11.43 sha1sum 5760 jdoe 20 0 3604956 208832 65316 R 2.3 11.2 0:43.50 gnome-shell 6448 jdoe 20 0 743836 31736 19488 S 0.7 1.7 0:08.25 gnome-terminal- 505 root 20 0 0 0 0 I 0.3 0.0 0:03.39 kworker/u4:4-events_unbound 4217 root 20 0 74192 1612 1320 S 0.3 0.1 0:01.19 spice-vdagentd