Repository: DIRACGrid/DIRAC
Pull request #8110: [8.0] Add cgroup2 limit support
Author: login: sfayer
Labels: name: alsoTargeting:integration
description: Cherry pick this PR to integration after merge
name: sweep:done
description: All sweeping actions have been done for this PR
name: sweep:failed
description: Sweeping failed and needs manual intervention
Created: 2025-03-28T16:30:22Z | Merged: 2025-04-15T14:51:10Z

Hi,

Here is an initial attempt at cgroup2 support via delegation... It supports `MemoryLimitMB` (int) and `MemoryNoSwap` (bool) options in ceParameters for both InProcessCE and SingularityCE (although I've not tested the latter). It's opportunistic, so should be transparent on sites that don't have cgroup delegation support (just running the payload unrestricted as before).

A hint for LHCb deployment: I suggest setting your limit a little shy of 4G, maybe around 3900 (as you won't get any benefit at sites with 4096 slots otherwise, as the pilot won't be protected without a small gap).

Closes #7853.

Regards,
Simon

BEGINRELEASENOTES
*WorkloadManagement
NEW: cgroup2 limit support
ENDRELEASENOTES


Linked issue #7853: Using cgroups to limit job resource usage
We recently failed to add support for cgroups in https://github.com/DIRACGrid/DIRAC/pull/7723 due to there being no session dbus instance started for the non-interactive jobs started by the batch system.

@fstagni pointed out to me that ALICE has made progress with this in:

https://indico.cern.ch/event/1338689/contributions/6010982/attachments/2952948/5191366/CHEP2024_Subdivision_GM.pdf

We should see if we can profit from it also.

cc @sfayer as you helped me understand this in the first place.
Linked issue comments: user: login: fstagni
created_at: 2024-10-23T15:03:04Z
body: Few additional minor bullets after talking with the author from ALICE:
- they patched HtCondor and SLURM 
- the WN needs to be Rhel9
- they create a "slot cgroup" and inside it sub-cgroups
- they create first the cgroup and inside it they launch apptainer (IIUC we were try to do the opposite) and inside it the job
- check the backup slides for some implementation (and caveat) details
author_association: CONTRIBUTOR
user: login: sfayer
created_at: 2024-10-23T15:28:12Z
body: Hi,

I had a quick look on our production HTCondor at IC and we do seem to have the patch already: A few of our cgroup control files for the per-slot cgroup tree are owned by the job user (rather than root), so it should be possible to create custom groups below that.

We're running the feature release of condor (to fix an unrelated cgroup bug!), so I'm not sure if it'll be available at most condor sites yet, but it certainly looks promising... I can't comment about the other batch systems, I felt they seemed less optimistic about getting the patches generally available for those in the talk.

Regards,
Simon

author_association: MEMBER
user: login: sfayer
created_at: 2025-03-12T17:30:02Z
body: Hi,

I wasn't really sure what you'd like me to do about this, but I made a guess:

I did a quick survey of UK sites and around 60% of them support the required cgroup2 delegation now (it's all the sites that use HTCondor batch, regardless of CE type, minus a few that do special things).

I've written a demo class/test program for doing the cgroup stuff, which I've tested on the UK HTCondor CEs successfully:
https://gist.github.com/sfayer/304b053c38d2f76a4ce4d2547160fd05
There are a number of caveats listed at the start.

The next steps (probably for someone who has worked on the pilot/CE code more recently than me), would likely be to:
- Integrate the cgroup class into the pilot and have it set-up the cgroups on start-up if they're available.
- Write a version of (or modify) systemCall / shellCall to be cgroup aware (avoiding any subprocess race conditions, which is the main challenge as described in the demo).
- Update the execution CEs (SingularityCE/InProcessCE/PoolCE) to create a slot cgroups and do the modified systemCall/shellCall with the cgroup at the correct point (and optionally use this for more precise accounting information at the end, keeping in mind that it's not always going to be available).

Regards,
Simon

author_association: MEMBER
user: login: fstagni
created_at: 2025-04-30T08:11:26Z
body: Fixed in https://github.com/DIRACGrid/DIRAC/pull/8110
author_association: CONTRIBUTOR
user: login: fstagni
created_at: 2025-05-27T13:21:04Z
body: Hi, I am looking into the development done for this issue, and I have a few questions:
- IIUC as implemented the functionality will work per CE (and if the WN allows it). Suppose that jobs of different "size" are submitted (e.g. Simulation jobs requesting 1 processor and 2 GBs, and user jobs requesting 8 processors and 8 GBs). At the moment it does not seem like there's a way to apply different limits for different jobs, am I right?
- A use case for using cGroups limits are partitioning of (possibly very) large WN slots (for DIRAC it means using Pool Computing Element, with InProcess or Singularity). Do I understand correctly that we are missing a connection with the knowledge of the slot limits?
author_association: CONTRIBUTOR
user: login: sfayer
created_at: 2025-06-05T16:41:36Z
body: Hi,

Yes, as it stands the source of the limits is fairly restrictive as they're only read from the ceParameters, based on the design from the original pull request (#7723).

I think the answers to both questions are fairly similar: As far as I'm aware, the dynamic "slot size" information isn't readily available at the levels it would need to be:
- At the pilot/JobAgent level, the overall size of the slot should be passed down to the execution CE (based on what was requested at the submission CE? or the ceParameters value?).
- The Pool Computing Element should then take the total slot size, subdivide it as it splits off the sub-slots and then pass that value down to the innerCE.

If that information was available, it'd be trivial to use it to set the cgroup limits.

I'm not sure the user job sizes specifically matter for this: I generally think of it from the slot perspective where the limits are set to the "slot size" (which should always be the same or bigger than the "job size" as long as the matching has worked correctly).

Regards,
Simon

author_association: MEMBER
user: login: fstagni
created_at: 2025-06-06T08:57:02Z
body: In general, DIRAC has a missing feature when it comes down to the size of the slot: the notion of RAM needed (from the job) and available (from the slot) is not fully there. Specifically, I think (I'd need to re-check the details) it's not taken into consideration at matching time.
The pilot at the same time knows the slot parameters, and so does the JobAgent. I will take note to look into this.
author_association: CONTRIBUTOR
user: login: fstagni
created_at: 2025-06-25T15:03:51Z
body: Contrary to what I wrote previously, DIRAC has the notion of RAM needed (as requirement from the job) and available (from the WN slot). As requirement, it is as a "Tag", e.g. "8GB". The RAM available in the slot is passed to the Matcher as "MaxRAM", and a matching is performed considering it. What was missing is the case of PoolComputingElement, for which I just made PR https://github.com/DIRACGrid/DIRAC/pull/8232
author_association: CONTRIBUTOR

Diff:
diff --git a/docs/source/AdministratorGuide/Resources/computingelements.rst b/docs/source/AdministratorGuide/Resources/computingelements.rst
index cd8ac1f3d8a..76d2874c901 100644
--- a/docs/source/AdministratorGuide/Resources/computingelements.rst
+++ b/docs/source/AdministratorGuide/Resources/computingelements.rst
@@ -180,3 +180,37 @@ section ::
       }
     }
   }
+
+Applying cgroup2 limits to computing resources
+@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+
+Both the :mod:`~DIRAC.Resources.Computing.InProcessComputingElement` and
+:mod:`~DIRAC.Resources.Computing.SingularityComputingElement` CEs support applying Linux cgroup2 CPU and memory limits to
+the slot. These will be applied if the site allows cgroup2 delegation, if this is not available execution will continue
+without the limits. The limit values can be specified using the following CE parameters (all settings are optional and can
+be left undefined if not needed):
+
+- CPULimit (float) - The number of cores that the job may use. Usage beyond this will be throttled.
+- MemoryLimitMB (int) - The memory limit for the job in MB. Usage beyond this will trigger the out-of-memory killer
+                        considering processes within the slot.
+- MemoryNoSwap (bool) - If yes or true, the job will not be allowed to use swap memory. Swap memory is not included
+                        in the main memory limit.
+
+Note that the memory limit should be lower than the amount requested with the submission CE in order to allow the main
+pilot processes to be protected. For example if you request 4096M (e.g. via XRSL) at submission, around 150M is needed
+for the pilot, so a limit of 3950M would be recommended.
+
+These can be specified in the CEDefaults section to apply a standardised slot size limit::
+
+  Resources
+  {
+    Computing
+    {
+      CEDefaults
+      {
+        CPULimit = 1.0
+        MemoryLimitMB = 3950
+        MemoryNoSwap = True
+      }
+    }
+  }
diff --git a/src/DIRAC/Core/Utilities/CGroups2.py b/src/DIRAC/Core/Utilities/CGroups2.py
new file mode 100644
index 00000000000..f874f85483f
--- /dev/null
+++ b/src/DIRAC/Core/Utilities/CGroups2.py
@@ -0,0 +1,328 @@
+#!/usr/bin/env python3
+"""cgroup2 support for DIRAC pilot."""
+
+import os
+import functools
+import subprocess
+from DIRAC import S_OK, S_ERROR, gLogger
+from DIRAC.Core.Utilities.DIRACSingleton import DIRACSingleton
+from DIRAC.Core.Utilities import Subprocess
+
+
+class CG2Manager(metaclass=DIRACSingleton):
+    """A class to manage cgroup2 hierachy for a typical pilot job use-case.
+
+    This creates a group for all of the pilot processes (anything in the
+    group at the start. This is a requirement for controlling the
+    sub-groups (no processes in non-leaf groups).
+
+    A group is then created on request for each "slot" under the pilot,
+    with the requested limits.
+    """
+
+    # Paths used to lookup cgroup info
+    FILE_MOUNTS = "/proc/mounts"
+    FILE_CUR_CGROUP = f"/proc/{os.getpid()}/cgroup"
+    # Control file names within the cgroup2 hierachy
+    CTRL_CONTROLLERS = "cgroup.controllers"
+    CTRL_PROCS = "cgroup.procs"
+    CTRL_SUBTREE = "cgroup.subtree_control"
+    CTRL_MEM_OOM_GROUP = "memory.oom.group"
+    CTRL_MEM_EVENTS = "memory.events"
+    CTRL_MEM_MAX = "memory.max"
+    CTRL_MEM_SWAP_MAX = "memory.swap.max"
+    CTRL_MEM_PEAK = "memory.peak"
+    CTRL_CPU_MAX = "cpu.max"
+    # CPU controller constants
+    # Weight is the max value for 1 CPU core
+    CPU_WEIGHT = 100000
+    # Period is the averaging time in us to apply the limit
+    # The default is 100k and I see no particularly reason this should change
+    CPU_PERIOD = 100000
+    # Name of the group for the existing pilot processes
+    PILOT_GROUP = f"dirac_pilot_{os.getpid()}"
+
+    def __init__(self):
+        """Set-up CGroup2 manager."""
+        # This boolean will be set to True if the cgroups are configured
+        # in the expected way
+        self._ready = False
+        # A counter of number of subgroups created
+        # Used to create unique group names
+        self._subproc_num = 0
+        # Physical path to the starting cgroup for this process
+        # (i.e. the base of our hierachy)
+        self._cgroup_path = None
+        # Logger
+        self.log = gLogger.getSubLogger("CG2Manager")
+
+    @staticmethod
+    def _filter_file(path, filterfcn):
+        """Opens a file and runs filterfcn for each line.
+        If filterfcn returns any value, that value will be returned
+        by this function.
+        Returns None if no line matches.
+        """
+        with open(path, encoding="ascii") as file_in:
+            for line in file_in.readlines():
+                line = line.strip()
+                if res := filterfcn(line):
+                    return res
+        return None
+
+    def _detect_root(self):
+        """Find the cgroup2 filesystem mountpoint on this system.
+        Returns the mountpoint path or None if it isn't found.
+        """
+
+        def filt(line):
+            """Filter function to find the first cgroup2 mount point
+            from a standard /proc/mounts layout file.
+            """
+            parts = line.split(" ")
+            if len(parts) < 3:
+                return None
+            if parts[2] == "cgroup2":
+                return parts[1]
+            return None
+
+        return self._filter_file(self.FILE_MOUNTS, filt)
+
+    def _detect_path(self):
+        """Finds the full physical path to the current cgroup control dir.
+        Sets self._cgroup_path on success.
+        Raises a RuntimeError if the path cannot be determined.
+        """
+
+        def filt(line):
+            """Filter to find the current cgroup2 name for the current
+            process, without the leading /.
+            """
+            if line.startswith("0::/"):
+                return line[4:]
+            return False
+
+        if not (root_path := self._detect_root()):
+            raise RuntimeError("Failed to find cgroup mount point")
+        if not (cur_group := self._filter_file(self.FILE_CUR_CGROUP, filt)):
+            raise RuntimeError("Failed to find current cgroup")
+        self._cgroup_path = os.path.join(root_path, cur_group)
+
+    def _create_group(self, group_name, isolate_oom=True):
+        """Creates a new group.
+        If "isolate_oom" is True, the new group will be decoupled
+        from the parent's OOM group.
+        Raises a RuntimeError if the group cannot be created.
+        """
+        try:
+            os.mkdir(os.path.join(self._cgroup_path, group_name))
+        except PermissionError as err:
+            raise RuntimeError(f"Permission denied creating sub-cgroup '{group_name}'") from err
+        if isolate_oom:
+            self._write_control(group_name, self.CTRL_MEM_OOM_GROUP, "0")
+
+    def _remove_group(self, group_name):
+        """Removes a group."""
+        os.rmdir(os.path.join(self._cgroup_path, group_name))
+
+    def _move_init_procs(self):
+        """Creates the pilot sub-group and moves all of the initial processes
+        from the top group into the new sub-group.
+        Will raise a RuntimeError if any cgroup configuration problem
+        prevents this from completing succesfully.
+        """
+        self._create_group(self.PILOT_GROUP, isolate_oom=False)
+        cur_pids = self._read_control("", self.CTRL_PROCS)
+        self._write_control(self.PILOT_GROUP, self.CTRL_PROCS, cur_pids)
+
+    def _read_control(self, group_name, ctrl_name):
+        """Reads a control value for the given group_name (relative to our base path).
+        The returned value varies depending on the value content:
+          - For a single token value, a string containing that token will be returned.
+          - For a single line value with space-seperated tokens, a list of tokens will be returned.
+          - For a multi-line value (where each line is a token), a list of tokens will be returned.
+        All tokens in the return values are strings.
+        A RuntimeError will be raised if the control cannot be read.
+        """
+        try:
+            with open(
+                os.path.join(self._cgroup_path, group_name, ctrl_name),
+                encoding="ascii",
+            ) as file_in:
+                values = [line.strip() for line in file_in.readlines()]
+                if " " in values and len(values) == 1:
+                    values = values[0].split(" ")
+                if len(values) == 1:
+                    values = values[0]
+                return values
+        except PermissionError as err:
+            raise RuntimeError(f"Access denied reading read control '{group_name}/{ctrl_name}'") from err
+
+    def _write_control(self, group_name, ctrl_name, value):
+        """Writes a control value for a given group_name (relative to our base path).
+        The value can be a string or an iterable of strings. The values should not
+        contain any whitespace characters.
+        A RuntimeError will be raised if the control cannot be set.
+        """
+        try:
+            ctrl_path = os.path.join(self._cgroup_path, group_name, ctrl_name)
+            with open(ctrl_path, "w", encoding="ascii") as file_out:
+                if isinstance(value, str):
+                    value = [value]
+                for arg in value:
+                    file_out.write(f"{arg}\n")
+                    # Flush is critical here as setting multiple values at the same time may fail
+                    file_out.flush()
+        except PermissionError as err:
+            raise RuntimeError(f"Access denied writing control '{group_name}/{ctrl_name}'") from err
+        except OSError as err:
+            # This generally happens if we're trying to set a value that is
+            # considered invalid, for example delegating a controller that isn't enabled
+            # in the first place.
+            raise RuntimeError(f"Error writing control '{group_name}/{ctrl_name}' = {value}") from err
+
+    def _get_oom_count(self, slot_name):
+        """Extracts the OOM counter as an int for the given slot.
+        Returns an int on success, can return a None if the memory.events
+        doesn't contain an oom counter or throws RuntimeError on failure.
+        """
+
+        def filt(line):
+            """Filter to find the oom counter from a memory.events file."""
+            if line.startswith("oom "):
+                return int(line[4:])
+            return False
+
+        mem_events = os.path.join(self._cgroup_path, slot_name, self.CTRL_MEM_EVENTS)
+        return self._filter_file(mem_events, filt)
+
+    def _set_limits(self, group_name, cores=None, memory=None, noswap=False):
+        """Sets the limits for an existing group.
+        See create_slot for a description of the other parameters.
+        This will raise a RuntimeError if appyling any of the limits fail to apply.
+        """
+        if cores:
+            proc_max = int(cores * self.CPU_WEIGHT)
+            self._write_control(group_name, self.CTRL_CPU_MAX, f"{proc_max} {self.CPU_PERIOD}")
+        if memory:
+            self._write_control(group_name, self.CTRL_MEM_MAX, f"{memory}")
+        if noswap:
+            self._write_control(group_name, self.CTRL_MEM_SWAP_MAX, "0")
+
+    def _prepare(self):
+        """Sets up the cgroup tree for the current process.
+        Should be called once, before using any of the other functions in this class.
+
+        Note that this function (specifcally the _move_init_procs call) assumes that
+        the list of processes is static. If the process list changes while this is running,
+        it is likely that this will fail to set things up properly.
+        """
+        self._detect_path()
+        controllers = self._read_control("", self.CTRL_CONTROLLERS)
+        if not controllers:
+            raise RuntimeError("No controllers enabled")
+        for ctrl in ["cpu", "memory"]:
+            if not ctrl in controllers:
+                raise RuntimeError(f"{ctrl} controller not enabled")
+        self._move_init_procs()
+        self._write_control("", self.CTRL_SUBTREE, ["+cpu", "+memory"])
+        self._ready = True
+
+    def _create_slot(self, slot_name, cores=None, memory=None, noswap=False):
+        """Creates a slot for a job with the given slot_name.
+        Cores is a float, number of CPU cores this group may use.
+        Memory is a string or int, either a number of bytes to limit the group RSS,
+        or a string limit with a unit suffix, e.g. "1G" as supported by the cgroup memory
+        controller.
+        If noswap is set to true, the swap memory limit will be set to 0; this is mostly
+        useful for testing (where the system may swap memory instead of triggering an
+        OOM, which may allow a process to use more than the memory limit).
+        This will raise a RuntimeError if setting up the slot fails.
+        """
+        if not self._ready:
+            return
+        self._create_group(slot_name)
+        self._set_limits(slot_name, cores, memory, noswap)
+
+    def _remove_slot(self, slot_name):
+        """Removes a slot with the given name.
+        Can raise usual filesystem OSError if the slot doesn't exist.
+        """
+        if not self._ready:
+            return
+        self._remove_group(slot_name)
+
+    def _setup_subproc(self, slot_name):
+        """A subprocess preexec function for setting up cgroups.
+        This will move te current process into the given cgroup slot.
+        On failure, no error will be reported.
+        """
+        # Threading danger!
+        # There are potential threading issues with preexec functions
+        # They must not hold any locks that the parent process might already
+        # be holding, including ones in standard library functions.
+        # This function should be kept as minimal as possible.
+        try:
+            self._write_control(slot_name, self.CTRL_PROCS, f"{os.getpid()}")
+        except Exception as err:
+            # We can't even really log here as we're in the set-up
+            # context of the new proces
+            pass
+
+    def setUp(self):
+        """Creates the base cgroup tree if possible. Should be called once
+        per process before using systemCall.
+        Returns S_OK/S_ERROR.
+        """
+        try:
+            self._prepare()
+        except Exception as err:
+            # The majority of CGroup failures will be RuntimeError
+            # However we don't want any unexpected failure to crash the upstream module,
+            # We just want to continue without cgroup support instead
+            return S_ERROR(str(err))
+        return S_OK()
+
+    def systemCall(self, *args, **kwargs):
+        """A proxy function for Subprocess.systemCall but will create a cgroup2 slot
+        if the functionality is available. An optional ceParameters dictionary
+        may be included, which will be searched for specific cgroup memory options.
+        Returns the usual S_OK/S_ERROR from Subprocess.systemCall.
+        """
+        preexec_fn = None
+        slot_name = f"subproc_{os.getpid()}_{self._subproc_num}"
+        self._subproc_num += 1
+        if self._ready:
+            self.log.info(f"Creating slot cgroup {slot_name}")
+            cores = None
+            memory = None
+            noswap = False
+            if "ceParameters" in kwargs:
+                if cpuLimit := kwargs["ceParameters"].get("CPULimit", None):
+                    cores = float(cpuLimit)
+                if memoryMB := int(kwargs["ceParameters"].get("MemoryLimitMB", 0)):
+                    memory = memoryMB * 1024 * 1024
+                if kwargs["ceParameters"].get("MemoryNoSwap", "no").lower() in ("yes", "true"):
+                    noswap = True
+            try:
+                self.log.info(f"CGroup Limits, CPU: {cores}, Mem: {memory}, NoSwap: {noswap}")
+                self._create_slot(slot_name, cores=cores, memory=memory, noswap=noswap)
+                preexec_fn = functools.partial(CG2Manager._setup_subproc, self, slot_name)
+            except Exception as err:
+                self.log.warn("Failed to create slot cgroup:", str(err))
+        kwargs["preexec_fn"] = preexec_fn
+        kwargs.pop("ceParameters", None)
+        res = Subprocess.systemCall(*args, **kwargs)
+        if self._ready:
+            self.log.info(f"Removing slot cgroup {slot_name}")
+            try:
+                oom_count = self._get_oom_count(slot_name)
+                if oom_count:
+                    # Child process triggered an OOM
+                    # We can't readily report this upstream (child process will probably
+                    # fail with an error code), so just log it and continue
+                    self.log.info(f"OOM detected from child process (slot {slot_name})")
+                self._remove_slot(slot_name)
+            except Exception as err:
+                self.log.warn(f"Failed to delete slot {slot_name} cgroup:", str(err))
+        return res
diff --git a/src/DIRAC/Core/Utilities/Subprocess.py b/src/DIRAC/Core/Utilities/Subprocess.py
index aad550ad1fe..eb52fa4ac55 100644
--- a/src/DIRAC/Core/Utilities/Subprocess.py
+++ b/src/DIRAC/Core/Utilities/Subprocess.py
@@ -424,7 +424,7 @@ def __readFromSystemCommandOutput(self, fd, bufferIndex):
             exitStatus = self.killChild()
             return self.__generateSystemCommandError(exitStatus, f"{retDict['Message']} for '{self.cmdSeq}' call")
 
-    def systemCall(self, cmdSeq, callbackFunction=None, shell=False, env=None):
+    def systemCall(self, cmdSeq, callbackFunction=None, shell=False, env=None, preexec_fn=None):
         """system call (no shell) - execute :cmdSeq:"""
 
         if shell:
@@ -444,6 +444,7 @@ def systemCall(self, cmdSeq, callbackFunction=None, shell=False, env=None):
                 close_fds=closefd,
                 env=env,
                 universal_newlines=True,
+                preexec_fn=preexec_fn,
             )
             self.childPID = self.child.pid
         except OSError as v:
@@ -545,7 +546,7 @@ def __callLineCallback(self, bufferIndex):
         return False
 
 
-def systemCall(timeout, cmdSeq, callbackFunction=None, env=None, bufferLimit=52428800):
+def systemCall(timeout, cmdSeq, callbackFunction=None, env=None, bufferLimit=52428800, preexec_fn=None):
     """
     Use SubprocessExecutor class to execute cmdSeq (it can be a string or a sequence)
     with a timeout wrapper, it is executed directly without calling a shell
@@ -555,13 +556,15 @@ def systemCall(timeout, cmdSeq, callbackFunction=None, env=None, bufferLimit=524
         sysCall = Watchdog(
             spObject.systemCall,
             args=(cmdSeq,),
-            kwargs={"callbackFunction": callbackFunction, "env": env, "shell": False},
+            kwargs={"callbackFunction": callbackFunction, "env": env, "shell": False, "preexec_fn": preexec_fn},
         )
         spObject.log.verbose("Subprocess Watchdog timeout set to %d" % timeout)
         result = sysCall(timeout + 1)
     else:
         spObject = Subprocess(timeout, bufferLimit=bufferLimit)
-        result = spObject.systemCall(cmdSeq, callbackFunction=callbackFunction, env=env, shell=False)
+        result = spObject.systemCall(
+            cmdSeq, callbackFunction=callbackFunction, env=env, shell=False, preexec_fn=preexec_fn
+        )
     return result
 
 
diff --git a/src/DIRAC/Resources/Computing/InProcessComputingElement.py b/src/DIRAC/Resources/Computing/InProcessComputingElement.py
index 5df0e77b0b7..4a5f91ab23f 100755
--- a/src/DIRAC/Resources/Computing/InProcessComputingElement.py
+++ b/src/DIRAC/Resources/Computing/InProcessComputingElement.py
@@ -7,8 +7,8 @@
 import stat
 
 from DIRAC import S_OK, S_ERROR
-from DIRAC.Core.Utilities.Subprocess import systemCall
 from DIRAC.Core.Utilities.ThreadScheduler import gThreadScheduler
+from DIRAC.Core.Utilities.CGroups2 import CG2Manager
 
 from DIRAC.Resources.Computing.ComputingElement import ComputingElement
 
@@ -61,7 +61,9 @@ def submitJob(self, executableFile, proxy=None, inputs=None, **kwargs):
             os.chmod(executableFile, stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH)
         cmd = os.path.abspath(executableFile)
         self.log.verbose("CE submission command:", cmd)
-        result = systemCall(0, cmd, callbackFunction=self.sendOutput, env=payloadEnv)
+        result = CG2Manager().systemCall(
+            0, cmd, callbackFunction=self.sendOutput, env=payloadEnv, ceParameters=self.ceParameters
+        )
         if payloadProxy:
             os.unlink(payloadProxy)
 
diff --git a/src/DIRAC/Resources/Computing/SingularityComputingElement.py b/src/DIRAC/Resources/Computing/SingularityComputingElement.py
index ee3264b4bdc..8bcbeaad5c4 100644
--- a/src/DIRAC/Resources/Computing/SingularityComputingElement.py
+++ b/src/DIRAC/Resources/Computing/SingularityComputingElement.py
@@ -21,7 +21,7 @@
 
 import DIRAC
 from DIRAC import S_OK, S_ERROR, gConfig, gLogger
-from DIRAC.Core.Utilities.Subprocess import systemCall
+from DIRAC.Core.Utilities.CGroups2 import CG2Manager
 from DIRAC.ConfigurationSystem.Client.Helpers import Operations
 from DIRAC.Core.Utilities.ThreadScheduler import gThreadScheduler
 from DIRAC.Resources.Computing.ComputingElement import ComputingElement
@@ -413,7 +413,9 @@ def submitJob(self, executableFile, proxy=None, **kwargs):
 
         self.log.debug(f"Execute singularity command: {cmd}")
         self.log.debug(f"Execute singularity env: {self.__getEnv()}")
-        result = systemCall(0, cmd, callbackFunction=self.sendOutput, env=self.__getEnv())
+        result = CG2Manager().systemCall(
+            0, cmd, callbackFunction=self.sendOutput, env=self.__getEnv(), ceParameteres=self.ceParameters
+        )
 
         self.__runningJobs -= 1
 
@@ -425,7 +427,9 @@ def submitJob(self, executableFile, proxy=None, **kwargs):
             self.log.error(f"Singularity env was: {self.__getEnv()}")
             debugCmd = [outerCmd[0], "--debug"] + outerCmd[1:] + ["echo", "All okay"]
             self.log.error("Running with debug output to facilitate debugging", debugCmd)
-            result = systemCall(0, debugCmd, callbackFunction=self.sendOutput, env=self.__getEnv())
+            result = CG2Manager().systemCall(
+                0, debugCmd, callbackFunction=self.sendOutput, env=self.__getEnv(), ceParameters=self.ceParameters
+            )
             if proxy and renewTask:
                 gThreadScheduler.removeTask(renewTask)
             self.__deleteWorkArea(baseDir)
diff --git a/src/DIRAC/WorkloadManagementSystem/Agent/JobAgent.py b/src/DIRAC/WorkloadManagementSystem/Agent/JobAgent.py
index aa21b2c604d..2180e134f35 100755
--- a/src/DIRAC/WorkloadManagementSystem/Agent/JobAgent.py
+++ b/src/DIRAC/WorkloadManagementSystem/Agent/JobAgent.py
@@ -23,6 +23,7 @@
 from DIRAC.Core.Security.ProxyInfo import getProxyInfo
 from DIRAC.Core.Security import Properties
 from DIRAC.Core.Utilities import DErrno
+from DIRAC.Core.Utilities.CGroups2 import CG2Manager
 from DIRAC.FrameworkSystem.Client.ProxyManagerClient import gProxyManager
 from DIRAC.Resources.Computing.BatchSystems.TimeLeft.TimeLeft import TimeLeft
 from DIRAC.Resources.Computing.ComputingElementFactory import ComputingElementFactory
@@ -139,6 +140,14 @@ def initialize(self):
 
         # Utilities
         self.timeLeftUtil = TimeLeft()
+
+        # Some innerCEs may want to make use of CGroup2 support, so we prepare it globally here
+        res = CG2Manager().setUp()
+        if res["OK"]:
+            self.log.info("CGroup2 support configured successfully.")
+        else:
+            self.log.info("CGroup2 support unavailable:", res["Message"])
+
         return S_OK()
 
     def _initializeComputingElement(self, localCE):


Reviews:
user: login: chrisburr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T11:50:34Z
user: login: chrisburr
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-15T13:21:02Z

Review comments:
diff_hunk: @@ -0,0 +1,326 @@
+#!/usr/bin/env python3
+"""cgroup2 support for DIRAC pilot."""
+
+import os
+import functools
+import subprocess
+from DIRAC import S_OK, S_ERROR, gLogger
+from DIRAC.Core.Utilities.DIRACSingleton import DIRACSingleton
+from DIRAC.Core.Utilities import Subprocess
+
+
+class CG2Manager(metaclass=DIRACSingleton):
+    """A class to manage cgroup2 hierachy for a typical pilot job use-case.
+
+    This creates a group for all of the pilot processes (anything in the
+    group at the start. This is a requirement for controlling the
+    sub-groups (no processes in non-leaf groups).
+
+    A group is then created on request for each "slot" under the pilot,
+    with the requested limits.
+    """
+
+    # Paths used to lookup cgroup info
+    FILE_MOUNTS = "/proc/mounts"
+    FILE_CUR_CGROUP = f"/proc/{os.getpid()}/cgroup"
+    # Control file names within the cgroup2 hierachy
+    CTRL_CONTROLLERS = "cgroup.controllers"
+    CTRL_PROCS = "cgroup.procs"
+    CTRL_SUBTREE = "cgroup.subtree_control"
+    CTRL_MEM_OOM_GROUP = "memory.oom.group"
+    CTRL_MEM_EVENTS = "memory.events"
+    CTRL_MEM_MAX = "memory.max"
+    CTRL_MEM_SWAP_MAX = "memory.swap.max"
+    CTRL_MEM_PEAK = "memory.peak"
+    CTRL_CPU_MAX = "cpu.max"
+    # CPU controller constants
+    # Weight is the max value for 1 CPU core
+    CPU_WEIGHT = 100000
+    # Period is the averaging time in us to apply the limit
+    # The default is 100k and I see no particularly reason this should change
+    CPU_PERIOD = 100000
+    # Name of the group for the existing pilot processes
+    PILOT_GROUP = f"dirac_pilot_{os.getpid()}"
+
+    def __init__(self):
+        """Set-up CGroup2 manager."""
+        # This boolean will be set to True if the cgroups are configured
+        # in the expected way
+        self._ready = False
+        # A counter of number of subgroups created
+        # Used to create unique group names
+        self._subproc_num = 0
+        # Physical path to the starting cgroup for this process
+        # (i.e. the base of our hierachy)
+        self._cgroup_path = None
+        # Logger
+        self.log = gLogger.getSubLogger("CG2Manager")
+
+    @staticmethod
+    def _filter_file(path, filterfcn):
+        """Opens a file and runs filterfcn for each line.
+        If filterfcn returns any value, that value will be returned
+        by this function.
+        Returns None if no line matches.
+        """
+        with open(path, encoding="ascii") as file_in:
+            for line in file_in.readlines():
+                line = line.strip()
+                if res := filterfcn(line):
+                    return res
+        return None
+
+    def _detect_root(self):
+        """Find the cgroup2 filesystem mountpoint on this system.
+        Returns the mountpoint path or None if it isn't found.
+        """
+
+        def filt(line):
+            """Filter function to find the first cgroup2 mount point
+            from a standard /proc/mounts layout file.
+            """
+            parts = line.split(" ")
+            if len(parts) < 3:
+                return None
+            if parts[2] == "cgroup2":
+                return parts[1]
+            return None
+
+        return self._filter_file(self.FILE_MOUNTS, filt)
+
+    def _detect_path(self):
+        """Finds the full physical path to the current cgroup control dir.
+        Sets self._cgroup_path on success.
+        Raises a RuntimeError if the path cannot be determined.
+        """
+
+        def filt(line):
+            """Filter to find the current cgroup2 name for the current
+            process, without the leading /.
+            """
+            if line.startswith("0::/"):
+                return line[4:]
+            return False
+
+        if not (root_path := self._detect_root()):
+            raise RuntimeError("Failed to find cgroup mount point")
+        if not (cur_group := self._filter_file(self.FILE_CUR_CGROUP, filt)):
+            raise RuntimeError("Failed to find current cgroup")
+        self._cgroup_path = os.path.join(root_path, cur_group)
+
+    def _create_group(self, group_name, isolate_oom=True):
+        """Creates a new group.
+        If "isolate_oom" is True, the new group will be decoupled
+        from the parent's OOM group.
+        Raises a RuntimeError if the group cannot be created.
+        """
+        try:
+            os.mkdir(os.path.join(self._cgroup_path, group_name))
+        except PermissionError as err:
+            raise RuntimeError(f"Permission denied creating sub-cgroup '{group_name}'") from err
+        if isolate_oom:
+            self._write_control(group_name, self.CTRL_MEM_OOM_GROUP, "0")
+
+    def _remove_group(self, group_name):
+        """Removes a group."""
+        os.rmdir(os.path.join(self._cgroup_path, group_name))
+
+    def _move_init_procs(self):
+        """Creates the pilot sub-group and moves all of the initial processes
+        from the top group into the new sub-group.
+        Will raise a RuntimeError if any cgroup configuration problem
+        prevents this from completing succesfully.
+        """
+        self._create_group(self.PILOT_GROUP, isolate_oom=False)
+        cur_pids = self._read_control("", self.CTRL_PROCS)
+        self._write_control(self.PILOT_GROUP, self.CTRL_PROCS, cur_pids)
+
+    def _read_control(self, group_name, ctrl_name):
+        """Reads a control value for the given group_name (relative to our base path).
+        The returned value varies depending on the value content:
+          - For a single token value, a string containing that token will be returned.
+          - For a single line value with space-seperated tokens, a list of tokens will be returned.
+          - For a multi-line value (where each line is a token), a list of tokens will be returned.
+        All tokens in the return values are strings.
+        A RuntimeError will be raised if the control cannot be read.
+        """
+        try:
+            with open(
+                os.path.join(self._cgroup_path, group_name, ctrl_name),
+                encoding="ascii",
+            ) as file_in:
+                values = [line.strip() for line in file_in.readlines()]
+                if " " in values and len(values) == 1:
+                    values = values[0].split(" ")
+                if len(values) == 1:
+                    values = values[0]
+                return values
+        except PermissionError as err:
+            raise RuntimeError(f"Access denied reading read control '{group_name}/{ctrl_name}'") from err
+
+    def _write_control(self, group_name, ctrl_name, value):
+        """Writes a control value for a given group_name (relative to our base path).
+        The value can be a string or an iterable of strings. The values should not
+        contain any whitespace characters.
+        A RuntimeError will be raised if the control cannot be set.
+        """
+        try:
+            ctrl_path = os.path.join(self._cgroup_path, group_name, ctrl_name)
+            with open(ctrl_path, "w", encoding="ascii") as file_out:
+                if isinstance(value, str):
+                    value = [value]
+                for arg in value:
+                    file_out.write(f"{arg}\n")
+                    # Flush is critical here as setting multiple values at the same time may fail
+                    file_out.flush()
+        except PermissionError as err:
+            raise RuntimeError(f"Access denied writing control '{group_name}/{ctrl_name}'") from err
+        except OSError as err:
+            # This generally happens if we're trying to set a value that is
+            # considered invalid, for example delegating a controller that isn't enabled
+            # in the first place.
+            raise RuntimeError(f"Error writing control '{group_name}/{ctrl_name}' = {value}") from err
+
+    def _get_oom_count(self, slot_name):
+        """Extracts the OOM counter as an int for the given slot.
+        Returns an int on success, can return a None if the memory.events
+        doesn't contain an oom counter or throws RuntimeError on failure.
+        """
+
+        def filt(line):
+            """Filter to find the oom counter from a memory.events file."""
+            if line.startswith("oom "):
+                return int(line[4:])
+            return False
+
+        mem_events = os.path.join(self._cgroup_path, slot_name, self.CTRL_MEM_EVENTS)
+        return self._filter_file(mem_events, filt)
+
+    def _set_limits(self, group_name, cores=None, memory=None, noswap=False):
+        """Sets the limits for an existing group.
+        See create_slot for a description of the other parameters.
+        This will raise a RuntimeError if appyling any of the limits fail to apply.
+        """
+        if cores:
+            proc_max = int(cores * self.CPU_WEIGHT)
+            self._write_control(group_name, self.CTRL_CPU_MAX, f"{proc_max} {self.CPU_PERIOD}")
+        if memory:
+            self._write_control(group_name, self.CTRL_MEM_MAX, f"{memory}")
+        if noswap:
+            self._write_control(group_name, self.CTRL_MEM_SWAP_MAX, "0")
+
+    def _prepare(self):
+        """Sets up the cgroup tree for the current process.
+        Should be called once, before using any of the other functions in this class.
+
+        Note that this function (specifcally the _move_init_procs call) assumes that
+        the list of processes is static. If the process list changes while this is running,
+        it is likely that this will fail to set things up properly.
+        """
+        self._detect_path()
+        controllers = self._read_control("", self.CTRL_CONTROLLERS)
+        if not controllers:
+            raise RuntimeError("No controllers enabled")
+        for ctrl in ["cpu", "memory"]:
+            if not ctrl in controllers:
+                raise RuntimeError(f"{ctrl} controller not enabled")
+        self._move_init_procs()
+        self._write_control("", self.CTRL_SUBTREE, ["+cpu", "+memory"])
+        self._ready = True
+
+    def _create_slot(self, slot_name, cores=None, memory=None, noswap=False):
+        """Creates a slot for a job with the given slot_name.
+        Cores is a float, number of CPU cores this group may use.
+        Memory is a string or int, either a number of bytes to limit the group RSS,
+        or a string limit with a unit suffix, e.g. "1G" as supported by the cgroup memory
+        controller.
+        If noswap is set to true, the swap memory limit will be set to 0; this is mostly
+        useful for testing (where the system may swap memory instead of triggering an
+        OOM, which may allow a process to use more than the memory limit).
+        This will raise a RuntimeError if setting up the slot fails.
+        """
+        if not self._ready:
+            return
+        self._create_group(slot_name)
+        self._set_limits(slot_name, cores, memory, noswap)
+
+    def _remove_slot(self, slot_name):
+        """Removes a slot with the given name.
+        Can raise usual filesystem OSError if the slot doesn't exist.
+        """
+        if not self._ready:
+            return
+        self._remove_group(slot_name)
+
+    def _setup_subproc(self, slot_name):
+        """A subprocess preexec function for setting up cgroups.
+        This will move te current process into the given cgroup slot.
+        On failure, no error will be reported.
+        """
+        # Threading danger!
+        # There are potential threading issues with preexec functions
+        # They must not hold any locks that the parent process might already
+        # be holding, including ones in standard library functions.
+        # This function should be kept as minimal as possible.
+        try:
+            self._write_control(slot_name, self.CTRL_PROCS, f"{os.getpid()}")
+        except Exception as err:
+            # We can't even really log here as we're in the set-up
+            # context of the new proces
+            pass
+
+    def setUp(self):
+        """Creates the base cgroup tree if possible. Should be called once
+        per process before using systemCall.
+        Returns S_OK/S_ERROR.
+        """
+        try:
+            self._prepare()
+        except Exception as err:
+            # The majority of CGroup failures will be RuntimeError
+            # However we don't want any unexpected failure to crash the upstream module,
+            # We just want to continue without cgroup support instead
+            return S_ERROR(str(err))
+        return S_OK()
+
+    def systemCall(self, *args, **kwargs):
+        """A proxy function for Subprocess.systemCall but will create a cgroup2 slot
+        if the functionality is available. An optional ceParameters dictionary
+        may be included, which will be searched for specific cgroup memory options.
+        Returns the usual S_OK/S_ERROR from Subprocess.systemCall.
+        """
+        preexec_fn = None
+        slot_name = f"subproc_{os.getpid()}_{self._subproc_num}"
+        self._subproc_num += 1
+        if self._ready:
+            self.log.info(f"Creating slot cgroup {slot_name}")
+            memory = None
+            noswap = False
+            if "ceParameters" in kwargs:
+                memoryMB = int(kwargs["ceParameters"].get("MemoryLimitMB", 0))
+                if memoryMB:
+                    memory = memoryMB * 1024 * 1024
+                if kwargs["ceParameters"].get("MemoryNoSwap", "no").lower() in ("yes", "true"):
+                    noswap = True
path: src/DIRAC/Core/Utilities/CGroups2.py
user: login: chrisburr
body: Can you document these new options in `docs/source/AdministratorGuide/Resources/computingelements.rst`?
created_at: 2025-04-04T11:48:43Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,326 @@
+#!/usr/bin/env python3
+"""cgroup2 support for DIRAC pilot."""
+
+import os
+import functools
+import subprocess
+from DIRAC import S_OK, S_ERROR, gLogger
+from DIRAC.Core.Utilities.DIRACSingleton import DIRACSingleton
+from DIRAC.Core.Utilities import Subprocess
+
+
+class CG2Manager(metaclass=DIRACSingleton):
+    """A class to manage cgroup2 hierachy for a typical pilot job use-case.
+
+    This creates a group for all of the pilot processes (anything in the
+    group at the start. This is a requirement for controlling the
+    sub-groups (no processes in non-leaf groups).
+
+    A group is then created on request for each "slot" under the pilot,
+    with the requested limits.
+    """
+
+    # Paths used to lookup cgroup info
+    FILE_MOUNTS = "/proc/mounts"
+    FILE_CUR_CGROUP = f"/proc/{os.getpid()}/cgroup"
+    # Control file names within the cgroup2 hierachy
+    CTRL_CONTROLLERS = "cgroup.controllers"
+    CTRL_PROCS = "cgroup.procs"
+    CTRL_SUBTREE = "cgroup.subtree_control"
+    CTRL_MEM_OOM_GROUP = "memory.oom.group"
+    CTRL_MEM_EVENTS = "memory.events"
+    CTRL_MEM_MAX = "memory.max"
+    CTRL_MEM_SWAP_MAX = "memory.swap.max"
+    CTRL_MEM_PEAK = "memory.peak"
+    CTRL_CPU_MAX = "cpu.max"
+    # CPU controller constants
+    # Weight is the max value for 1 CPU core
+    CPU_WEIGHT = 100000
+    # Period is the averaging time in us to apply the limit
+    # The default is 100k and I see no particularly reason this should change
+    CPU_PERIOD = 100000
+    # Name of the group for the existing pilot processes
+    PILOT_GROUP = f"dirac_pilot_{os.getpid()}"
+
+    def __init__(self):
+        """Set-up CGroup2 manager."""
+        # This boolean will be set to True if the cgroups are configured
+        # in the expected way
+        self._ready = False
+        # A counter of number of subgroups created
+        # Used to create unique group names
+        self._subproc_num = 0
+        # Physical path to the starting cgroup for this process
+        # (i.e. the base of our hierachy)
+        self._cgroup_path = None
+        # Logger
+        self.log = gLogger.getSubLogger("CG2Manager")
+
+    @staticmethod
+    def _filter_file(path, filterfcn):
+        """Opens a file and runs filterfcn for each line.
+        If filterfcn returns any value, that value will be returned
+        by this function.
+        Returns None if no line matches.
+        """
+        with open(path, encoding="ascii") as file_in:
+            for line in file_in.readlines():
+                line = line.strip()
+                if res := filterfcn(line):
+                    return res
+        return None
+
+    def _detect_root(self):
+        """Find the cgroup2 filesystem mountpoint on this system.
+        Returns the mountpoint path or None if it isn't found.
+        """
+
+        def filt(line):
+            """Filter function to find the first cgroup2 mount point
+            from a standard /proc/mounts layout file.
+            """
+            parts = line.split(" ")
+            if len(parts) < 3:
+                return None
+            if parts[2] == "cgroup2":
+                return parts[1]
+            return None
+
+        return self._filter_file(self.FILE_MOUNTS, filt)
+
+    def _detect_path(self):
+        """Finds the full physical path to the current cgroup control dir.
+        Sets self._cgroup_path on success.
+        Raises a RuntimeError if the path cannot be determined.
+        """
+
+        def filt(line):
+            """Filter to find the current cgroup2 name for the current
+            process, without the leading /.
+            """
+            if line.startswith("0::/"):
+                return line[4:]
+            return False
+
+        if not (root_path := self._detect_root()):
+            raise RuntimeError("Failed to find cgroup mount point")
+        if not (cur_group := self._filter_file(self.FILE_CUR_CGROUP, filt)):
+            raise RuntimeError("Failed to find current cgroup")
+        self._cgroup_path = os.path.join(root_path, cur_group)
+
+    def _create_group(self, group_name, isolate_oom=True):
+        """Creates a new group.
+        If "isolate_oom" is True, the new group will be decoupled
+        from the parent's OOM group.
+        Raises a RuntimeError if the group cannot be created.
+        """
+        try:
+            os.mkdir(os.path.join(self._cgroup_path, group_name))
+        except PermissionError as err:
+            raise RuntimeError(f"Permission denied creating sub-cgroup '{group_name}'") from err
+        if isolate_oom:
+            self._write_control(group_name, self.CTRL_MEM_OOM_GROUP, "0")
+
+    def _remove_group(self, group_name):
+        """Removes a group."""
+        os.rmdir(os.path.join(self._cgroup_path, group_name))
+
+    def _move_init_procs(self):
+        """Creates the pilot sub-group and moves all of the initial processes
+        from the top group into the new sub-group.
+        Will raise a RuntimeError if any cgroup configuration problem
+        prevents this from completing succesfully.
+        """
+        self._create_group(self.PILOT_GROUP, isolate_oom=False)
+        cur_pids = self._read_control("", self.CTRL_PROCS)
+        self._write_control(self.PILOT_GROUP, self.CTRL_PROCS, cur_pids)
+
+    def _read_control(self, group_name, ctrl_name):
+        """Reads a control value for the given group_name (relative to our base path).
+        The returned value varies depending on the value content:
+          - For a single token value, a string containing that token will be returned.
+          - For a single line value with space-seperated tokens, a list of tokens will be returned.
+          - For a multi-line value (where each line is a token), a list of tokens will be returned.
+        All tokens in the return values are strings.
+        A RuntimeError will be raised if the control cannot be read.
+        """
+        try:
+            with open(
+                os.path.join(self._cgroup_path, group_name, ctrl_name),
+                encoding="ascii",
+            ) as file_in:
+                values = [line.strip() for line in file_in.readlines()]
+                if " " in values and len(values) == 1:
+                    values = values[0].split(" ")
+                if len(values) == 1:
+                    values = values[0]
+                return values
+        except PermissionError as err:
+            raise RuntimeError(f"Access denied reading read control '{group_name}/{ctrl_name}'") from err
+
+    def _write_control(self, group_name, ctrl_name, value):
+        """Writes a control value for a given group_name (relative to our base path).
+        The value can be a string or an iterable of strings. The values should not
+        contain any whitespace characters.
+        A RuntimeError will be raised if the control cannot be set.
+        """
+        try:
+            ctrl_path = os.path.join(self._cgroup_path, group_name, ctrl_name)
+            with open(ctrl_path, "w", encoding="ascii") as file_out:
+                if isinstance(value, str):
+                    value = [value]
+                for arg in value:
+                    file_out.write(f"{arg}\n")
+                    # Flush is critical here as setting multiple values at the same time may fail
+                    file_out.flush()
+        except PermissionError as err:
+            raise RuntimeError(f"Access denied writing control '{group_name}/{ctrl_name}'") from err
+        except OSError as err:
+            # This generally happens if we're trying to set a value that is
+            # considered invalid, for example delegating a controller that isn't enabled
+            # in the first place.
+            raise RuntimeError(f"Error writing control '{group_name}/{ctrl_name}' = {value}") from err
+
+    def _get_oom_count(self, slot_name):
+        """Extracts the OOM counter as an int for the given slot.
+        Returns an int on success, can return a None if the memory.events
+        doesn't contain an oom counter or throws RuntimeError on failure.
+        """
+
+        def filt(line):
+            """Filter to find the oom counter from a memory.events file."""
+            if line.startswith("oom "):
+                return int(line[4:])
+            return False
+
+        mem_events = os.path.join(self._cgroup_path, slot_name, self.CTRL_MEM_EVENTS)
+        return self._filter_file(mem_events, filt)
+
+    def _set_limits(self, group_name, cores=None, memory=None, noswap=False):
+        """Sets the limits for an existing group.
+        See create_slot for a description of the other parameters.
+        This will raise a RuntimeError if appyling any of the limits fail to apply.
+        """
+        if cores:
+            proc_max = int(cores * self.CPU_WEIGHT)
+            self._write_control(group_name, self.CTRL_CPU_MAX, f"{proc_max} {self.CPU_PERIOD}")
+        if memory:
+            self._write_control(group_name, self.CTRL_MEM_MAX, f"{memory}")
+        if noswap:
+            self._write_control(group_name, self.CTRL_MEM_SWAP_MAX, "0")
+
+    def _prepare(self):
+        """Sets up the cgroup tree for the current process.
+        Should be called once, before using any of the other functions in this class.
+
+        Note that this function (specifcally the _move_init_procs call) assumes that
+        the list of processes is static. If the process list changes while this is running,
+        it is likely that this will fail to set things up properly.
+        """
+        self._detect_path()
+        controllers = self._read_control("", self.CTRL_CONTROLLERS)
+        if not controllers:
+            raise RuntimeError("No controllers enabled")
+        for ctrl in ["cpu", "memory"]:
+            if not ctrl in controllers:
+                raise RuntimeError(f"{ctrl} controller not enabled")
+        self._move_init_procs()
+        self._write_control("", self.CTRL_SUBTREE, ["+cpu", "+memory"])
+        self._ready = True
+
+    def _create_slot(self, slot_name, cores=None, memory=None, noswap=False):
+        """Creates a slot for a job with the given slot_name.
+        Cores is a float, number of CPU cores this group may use.
+        Memory is a string or int, either a number of bytes to limit the group RSS,
+        or a string limit with a unit suffix, e.g. "1G" as supported by the cgroup memory
+        controller.
+        If noswap is set to true, the swap memory limit will be set to 0; this is mostly
+        useful for testing (where the system may swap memory instead of triggering an
+        OOM, which may allow a process to use more than the memory limit).
+        This will raise a RuntimeError if setting up the slot fails.
+        """
+        if not self._ready:
+            return
+        self._create_group(slot_name)
+        self._set_limits(slot_name, cores, memory, noswap)
+
+    def _remove_slot(self, slot_name):
+        """Removes a slot with the given name.
+        Can raise usual filesystem OSError if the slot doesn't exist.
+        """
+        if not self._ready:
+            return
+        self._remove_group(slot_name)
+
+    def _setup_subproc(self, slot_name):
+        """A subprocess preexec function for setting up cgroups.
+        This will move te current process into the given cgroup slot.
+        On failure, no error will be reported.
+        """
+        # Threading danger!
+        # There are potential threading issues with preexec functions
+        # They must not hold any locks that the parent process might already
+        # be holding, including ones in standard library functions.
+        # This function should be kept as minimal as possible.
+        try:
+            self._write_control(slot_name, self.CTRL_PROCS, f"{os.getpid()}")
+        except Exception as err:
+            # We can't even really log here as we're in the set-up
+            # context of the new proces
+            pass
+
+    def setUp(self):
+        """Creates the base cgroup tree if possible. Should be called once
+        per process before using systemCall.
+        Returns S_OK/S_ERROR.
+        """
+        try:
+            self._prepare()
+        except Exception as err:
+            # The majority of CGroup failures will be RuntimeError
+            # However we don't want any unexpected failure to crash the upstream module,
+            # We just want to continue without cgroup support instead
+            return S_ERROR(str(err))
+        return S_OK()
+
+    def systemCall(self, *args, **kwargs):
+        """A proxy function for Subprocess.systemCall but will create a cgroup2 slot
+        if the functionality is available. An optional ceParameters dictionary
+        may be included, which will be searched for specific cgroup memory options.
+        Returns the usual S_OK/S_ERROR from Subprocess.systemCall.
+        """
+        preexec_fn = None
+        slot_name = f"subproc_{os.getpid()}_{self._subproc_num}"
+        self._subproc_num += 1
+        if self._ready:
+            self.log.info(f"Creating slot cgroup {slot_name}")
+            memory = None
+            noswap = False
+            if "ceParameters" in kwargs:
+                memoryMB = int(kwargs["ceParameters"].get("MemoryLimitMB", 0))
path: src/DIRAC/Core/Utilities/CGroups2.py
user: login: chrisburr
body: Can we also limit the amount of CPU used here?
created_at: 2025-04-04T11:49:35Z
author_association: MEMBER

PR comments:
user: login: sfayer
created_at: 2025-04-09T12:07:12Z
body: Thanks for the review (I didn't notice it for a while, the notification must have got lost in the flurry of recent diracx activity :-)

I've added support for setting a CPU limit and written the requested documentation.

Regards,
Simon
author_association: MEMBER
user: login: DIRACGridBot
created_at: 2025-04-15T14:56:01Z
body: **Sweep summary**

Sweep ran in https://github.com/DIRACGrid/DIRAC/actions/runs/14472511501

### Failed:
* **integration**
  cherry-pick 16a2dd1a3 into integration failed
  check merge conflicts on a local copy of this repository
  ```bash
  git fetch upstream
  git checkout upstream/integration -b cherry-pick-2-16a2dd1a3-integration
  git cherry-pick -x -m 1 16a2dd1a3
  # Fix the conflicts
  git cherry-pick --continue
  git commit --amend -m 'sweep: #8110 Add cgroup2 limit support' --author='Simon Fayer <sf105@ic.ac.uk>'
  git push -u origin cherry-pick-2-16a2dd1a3-integration
  
  # If you have the GitHub CLI installed the PR can be made with
  gh pr create \
       --label 'sweep:from rel-v8r0' \
       --base integration \
       --repo DIRACGrid/DIRAC \
       --title '[sweep:integration] Add cgroup2 limit support' \
       --body 'Sweep #8110 `Add cgroup2 limit support` to `integration`.
  
  Adding original author @sfayer as watcher.
  
  BEGINRELEASENOTES
  *WorkloadManagement
  NEW: cgroup2 limit support
  ENDRELEASENOTES
  Closes #8158'
  ```
author_association: NONE