# Coarse-grain Congestion Regulation Using Control Theory

Thomas Collignon, Kouds Halitim, Raphaël Bleuse, Sophie Cerf, Bogdan

Robu, Eric Rutten, Lionel Seinturier, Alexandre van Kempen

► To cite this version:

Thomas Collignon, Kouds Halitim, Raphaël Bleuse, Sophie Cerf, Bogdan Robu, et al. Coarse-grain Congestion Regulation Using Control Theory. COMPAS 2025 - Conférence francophone d'informatique en Parallélisme, Architecture et Système, Jun 2025, Bordeaux, France. pp.1-10. hal-05075480

HAL Id: hal-05075480

https://hal.science/hal-05075480v1

Submitted on 20 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Coarse-grain Congestion Regulation Using Control Theory

Thomas COLLIGNON*§, Kouds HALITIM‡, Raphaël BLEUSE‡, Sophie CERF§, Bogdan ROBU†, Éric RUTTEN‡, Lionel SEINTURIER§, Alexandre VAN KEMPEN*

*Qarnot Computing, F-92120 Montrouge, France

§Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 CRStAL, F-59000 Lille, France

‡Univ. Grenoble Alpes, Inria, CNRS, LIG, F-38000 Grenoble, France

†Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, F-38000 Grenoble, France

E-mail : * first.last@qarnot-computing.com

§, ‡, † first.last@inria.fr

## Résumé

Efficient data access in High-Performance Computing (HPC) systems is essential to the performance of intensive computing tasks. Traditional optimizations of the I/O stack aim to improve peak performance but are often workload specific and require deep expertise, making them difficult to generalize. In shared HPC environments, resource congestion can lead to unpredictable performance, causing slowdowns and timeouts. To address these challenges, a self-adaptive approach based on Control Theory is proposed to dynamically regulate client-side I/O rates. By using few run-time system load metrics, the approach aims to reduce congestion and improve performance stability. We implement a controller on a single node system as a preliminary work, and obtain encouraging results for addressing a multi-node cluster in future works.

## 1. Introduction

High-Performance Computing (HPC) systems play an increasingly crucial role in scientific discovery, complex simulations, fast prototyping, and other computationally intensive tasks. Performance in these systems heavily depends on efficient data access, making fast and scalable parallel I/O a critical requirement for many HPC applications. To meet this demand, the HPC I/O stack has grown highly complex, spanning multiple layers—including hardware, filesystems, and middleware—each with thousands of tunable parameters (see [5] for a comprehensive survey). Traditional optimization approaches focus on maximizing peak I/O performance through exhaustive parameter tuning, sophisticated benchmarking tools [13], and, more recently, machine learning techniques [3, 14]. While such tuning can yield significant performance gains, it is usually workload-specific, requires deep expertise, and does not generalize easily across diverse HPC environments.

In the context of shared HPC infrastructures (e.g., cloud-based HPC), multiple users and applications may compete for limited storage resources, leading to resource congestion that can significantly degrade system-wide performance [8]. Storage congestion results in highly unpredictable performance, causing random slowdowns and timeouts. In many cases, unpredictability can be more problematic than consistently lower but stable performance. In this context, we argue that preventing performance degradation due to resource congestion can sometimes be more important than maximizing peak I/O speed for individual applications. Several studies
<figure><img src="image_2.png" /><figcaption>FIGURE 1 – Overview of the control on the I/O path on a single node</figcaption></figure>

follow this approach by trying to mitigate I/O interferences [6, 15]. However, this requires an *a priori* knowledge about the I/O patterns [4] of the applications, which is hardly available.

To prevent congestion, we propose a coarse-grained, self-adaptive approach that relies on control theory. That is, instead of fine-tuning each layer of the I/O stack, we dynamically regulate client-side I/O rates based on real-time system load metrics, abstracting away much of the stack's complexity thanks to the feedback formulation. Using the control theory framework, we propose a generic architecture that employs an actuator-sensor pair to dynamically adjust load on shared storage, ensuring more stable and predictable performance.

This paper presents the first steps towards implementing a controller on a concrete instance of this architecture where we use Linux `cgroups` to enforce client-side I/O rate limits while leveraging the server dispatch queue as a system load metric. We detail the implementation of a feedback Proportional-Integral (PI) controller on a single node system, explaining how to properly calibrate it for effective regulation. Figure 1 is a representation of our defined control setup and it will be explained throughout this paper.

Preliminary results demonstrate that such a controller is able to maintain a desired level of disk activity with an acceptable error, paving the way to dynamically throttle client I/O rates in response to an observed congestion.

## 2. Challenges of dynamic I/O management

### 2.1. I/O congestion

The general studied system is a computing cluster that would form a typical site of a distributed HPC cloud. This cluster is made of a supervisor node, responsible of distributing incoming computing tasks to available computing nodes, handling the data transfers for the necessary data of the tasks and the upload of their results to higher levels of the infrastructure. The supervisor node is also handling the storage of the cluster with a shared filesystem (such as NFS, Lustre or CephFS). As a preliminary work, we will focus on this node alone in the following.

It has been observed that congestion can happen on the disk due to factors such as the storage hardware, the intensiveness of the concurrent workloads, data locality and the amount of active computing nodes [15]. The I/O congestion problem is inherently hard to predict because (i) it is caused by complex interference between layers of the I/O path, which is summarized on the right side of Figure 1, (ii) of the sophisticated treatments of I/O requests by those layers,
(iii) those mechanisms happen on very short time periods. Furthermore, we want to be able to support the widest range of hardware and architecture, so we cannot address this issue by fine tuning every layer of the stack. Also, we do not have an a priori information about the I/O profiles of the incoming tasks and their duration as they are submitted by users, which makes it impossible to tune our system parameters before running the task. We argue that to address these issues, we should be tuning the system dynamically at run-time, with a feedback loop.

## 2.2. The closed-loop approach

Feedback regulation has been widely used to dynamically monitor computing systems' performance and adjust their behavior to meet desired objectives [10]. Examples include the MAPE-K loop in autonomous computing [11] and control-theory-based feedback mechanisms [9, 7].

In our system, a closed-loop control mechanism is employed as shown in the dashed box in Figure 1, where we continuously monitor performance at run-time through appropriate sensors (metrics), detailed in Section 3.2. These measurements are compared against a predefined reference state, that generates an error signal, quantifying deviation from desired behavior.

To correct this deviation, the controller dynamically adjusts system parameters (control inputs) via an actuator—a component responsible for modifying system settings in response to the error signal. The actuator executes the control action, allowing the system to adapt at run-time. This feedback loop operates continuously to achieve the control objectives specified in Figure 3, ensuring that the system meets user-defined performance criteria : (i) *reference tracking*, which ensures that the system's output, represented by the black curve in the figure, closely follows the desired reference value with minimal *steady-state error*—the residual discrepancy at $t \to \infty$ between reference and actual output, (ii) the system must achieve a fast and stable response, meaning the output should reach the reference value with a *short settling time*, defined as the time required for the output to remain within 5% of its final value, with minimal oscillations and *overshoot*, i.e. avoid excessive deviations above the reference due to an input change.

By achieving these objectives, the system can autonomously optimize performance, maintain stability, and respond efficiently to workload variations and unexpected disturbances. The translation of those objectives in the I/O congestion avoidance problem is explained next.

# 3. Disk activity Controller

We present different steps in designing a controller of the disk activity. We detail the choices made for the different components of the control stack applied to the single host system.

## 3.1. Single node system

As a preliminary work, we focus on a simplified architecture, with the objective of showing that doing control to have an impact on the congestion is possible. In that context, we define a system with a single host that represents the supervisor of the computing cluster. The I/O intensive workload as well as the controller will run on this node, so there will not be any shared storage nor network communications involved.

## 3.2. Choosing sensor and actuator signals

To build a controller, it is required to identify a sensor, to quantify the congestion level of the shared storage, as well as an actuator, to dynamically restrict the I/O rates of the applications. Control Theory requires the chosen sensor to fit multiple criteria for the controller design, e.g. constant frequency, continuous values, ideal reference value, etc. Dealing with I/Os also adds some time granularity or delay requirements. The sensor should reflect as closely as possible
the state of the storage device at a given time. Looking at the complex I/O stack, we focus on the closest layers to the disk as they are the first to be affected by disk congestion. We opt for not using the disk utilization rate as a congestion indicator because a rate of 100% does not mean that the disk is congested, simply that it is treating requests at its maximum speed, which is good for performances. Instead, we use the size of the dispatch queue of the block layer to measure disk utilization [2], that fills up when I/O requests are waiting on the disk.

As well as for the sensor, there are requirements for the chosen actuator. It needs to affect the disk congestion level (i.e. the dispatch queue size), possibly with some delay or dynamics, and to preferably support continuous values as a command. We choose to use the control group (cgroup) feature of the Linux kernel, which allows to set limits on applications for their access to resources such as CPU cycles, memory, and I/O operations. We focus on the io controller and specifically on the wiops command, that defines a I/O request rate limit per cgroup. This allows to dynamically change the I/O rates of an application by writing the command in a specific control file. The control files are stored within a cgroup filesystem, which only involves kernel memory and does not lead to disk I/Os when writing cgroup commands. That way, the mechanism of changing the wiops limit will not be affected by the congestion level of the disk. Figure 1 shows how the selected sensors and actuators interact with the I/O stack. They are specifically chosen for the single host system that we are studying. We might need to select different ones when shared filesystems and network will be introduced to the controlled system.

### 3.3. Controller design

For the control strategy, we use a discrete-time Proportional-Integral (PI) controller [9], chosen for its balance between responsiveness and stability. Its simplicity and effectiveness make it a widely adopted solution, with most industrial systems relying on it. The PI controller computes the next action to be applied on the system based on the error between the measured value of the sensor and its reference value e[k] (See algorithm 1). The proportional term provides immediate correction based on deviations (current error signal e[k]), ensuring fast response (reducing settling time). The integral term accumulates past errors to eliminate steady-state error (performance drift) over time as illustrated in the following equation :

$$wiops[k] = K_p e[k] + K_i T_s \sum_{i=0}^{k} e[i], \quad T_s = 100 \text{ms} \quad (1)$$

Where Kp and Ki are the PI controller gains, e[k] is the error at discrete time step k, where k is an integer index corresponding to multiples of the sampling time Ts.

The sampling time value impacts the stability and performance of the system. A too small value would lead to instability due to large signal variations, while a larger one would cause measurement imprecision due to average on long periods. Furthermore, the chosen cgroup actuator is involving averages over time, using a small sampling time would lead to inefficiency. Setting the sampling time to Ts = 100ms has experimentally shown a good trade-off.

Tuning the controller gains Kp and Ki is crucial, affecting the performance objectives – reference tracking, stability, overshoot, and settling time explained in Section 2.2. Proper tuning also enhances robustness to disturbances, ensuring stable performance under varying workloads.

To properly select the parameters for the PI controller, we rely on a methodology that, given a dynamic model of the system, translates our specification (overshoot and settling time) in value of the gains Kp and Ki. The chosen model should accurately capture the essential dynamics while remaining simple enough for effective control design. First, to determine an appropriate model, we first examine the static behavior of the system by analyzing how the dispatch queue
size (output) changes with *wiops* (input). As shown in Figure 4, the response follows a decaying exponential shape, which can be approximated as a linear function within a certain range. Additionally, we observe a saturation effect at a queue size of 50 for *wiops* values exceeding 30k. Next, we analyze the dynamic behavior of the system, as shown in Figure 4b. The settling time is in the order of a few hundreds milliseconds, indicating a fast reaction to changes in *wiops*. The system then quickly settles to its final value, with some oscillations present. We assume that a portion of these oscillations is due to high-frequency noise, as they are significantly reduced when increasing the sampling time. Based on these observations, we approximate the system as a first-order linear model in discrete time, given by: $Q(k+1) = a.Q(k) + b.wiops(k)$, where $Q_k$ is the dispatch queue size at time k and $wiops_k$ is the *wiops* value at time k, and a, b are identified system parameters. Finally, we define the design specifications values, including settling time and overshoot, which are used to calculate the PI controller gains $K_p$, $K_i$ using Equation 2 in the appendix. The equations were derived from control theory pole-placement principles, and are detailed in [9]. The behavior of the dispatch queue varies across different hardware architectures, the system identification process—which involves determining the values of a and b—must be repeated for each new architecture.

# 4. Experimental results

## 4.1. Experimental setup

We deploy the I/O controller on a dedicated Linux machine. We use Python for the entire control system by defining multiple main classes for root components of the control (ControllerPI, Sensor, Actuator). They are used to define sub classes, specific to an experiment, depending on the chosen control metrics. Here, we develop a sensor of the length of the dispatch queue using the *time_in_queue* metric from the *sysfs* stat file (`/sys/block/<dev>/stats`), and a cgroup actuator that allows to write cgroup *wiops* limits in the right control file.

The controller computes the control parameters $K_p$ and $K_i$ from the previously established model of the system and the desired properties of the controller, including settling time and overshoot (i.e. the maximum peak value of the system response measured from the objective). It then starts the main loop, polling sensors and using actuators with the computed command.

## 4.2. Controller performance

The controller is run concurrently with a write-intensive FIO<fnref n="1" /> job using direct I/Os to bypass any cache and generate a more intense workload for the disk. It saturates the dispatch queue when no control action is applied. We change the control target over time according to a step function (the dashed line in Figure 2), and monitor how the system reacts to those changes using the collected dispatch queue sensor data (the orange signals in the figure) and the *wiops* limit decision of the controller (the blue signals in the figure).

Figure 2 shows the results of multiple executions with different control parameters. The left plot shows good reference tracking (the sensor meets its target value) with minor oscillations, a mean absolute tracking error (MAE) of 17%, a dispatch queue size that settles to the reference value with a short settling time of approximately 200ms, minimal overshoot, and no steady-state error, eliminated by the integral action of the controller, along with relatively stable Wiops values. The other two plots on the right, using different sets of controller parameters, show poorer controller performance due to improper tuning. For example, the upper-right plot displays large oscillations in both the dispatch queue and *wiops* values, almost leading to system instability. The lower-right plot, on the other hand, shows a much smoother response, but the

<footnote n="1">1. https://fio.readthedocs.io/en/latest/</footnote>
<figure><img src="image_3.png" /><figcaption>FIGURE 2 – Control results for multiple parameters combinations- Closed-loop dispatch queue response (orange signal in the top plots) for different decreasing steps of reference values (dashed black line), and the wiops control adjustments (blue signal in the lower plots) : The plots show that controller adjusts the number of wiops so that the dispatch queue size matches the reference value. The response differs when we change the controller parameters, the left plot shows the best performance, where the right plots show poor performance and instability.</figcaption></figure>

system reacts very slowly. This illustrates how the choice of controller parameters affects per-
formance, oscillation levels, and responsiveness. Some methods on how to tune the parameters
using control theory principles are explained in this tutorial [9].

5. Conclusion and future work

In this preliminary work, we built a simple controller that is able to maintain desired levels of disk activity, specified by dispatch queue size targets, on a single local node. From a control perspective, in ongoing work we tend to improve the controller efficiency by adding an event activator and an anti-windup mechanism. The event activator ensures the controller operates only when I/O contention is detected, while the anti-windup mechanism prevents excessive integration during control saturation (see Algorithm 2). We also aim to make the control more robust by taking into account the imperfections in our model (noise, non-linearity, varying parameters and operating conditions, etc.).

From a systems point of view, we did our experiments on a single node with direct I/Os so it greatly simplified the I/O path of a functional computing cluster, omitting the network, shared filesystem, caching mechanisms and concurrent access to resources, which are the sources of additional congestion issues. The kernel of the server node cannot handle those issues by itself as well as it can for congestion caused by local I/O requests. We will add the network and shared filesystem layers to our experiments on a multi-node setup, e.g. Grid5000[1] cluster. The next objective is to implement controllers on the client side of a shared filesystem.

Future works will aim to validate our established strategies for building a controller on such
complete system. We might need to explore new sensors and actuators on more complex archi-
tectures as our current control was designed for a single node. We could for example use eBPF<fnref n="2" />
to obtain fine-grained, low overhead and high efficiency action on the client nodes.

<footnote n="2">2. https://ebpf.io</footnote>
## Acknowledgement

Our work is done in the context of the Inria – Qarnot Pulse project : https://www.inria.fr/en/pulse.
Bibliographie

1. Balouek (D.), Carpen-Amarie (A.), Charrier (G.), Desprez (F.), Jeannot (E.), Jeanvoine (E.), Lèbre (A.), Margery (D.), Niclausse (N.), Nussbaum (L.), Richard (O.), Pérez (C.), Quesnel (F.), Rohr (C.) et Sarzyniec (L.). – Adding Virtualization Capabilities to the Grid’5000 Testbed. – In CLOSER, *Communications in Computer and Information Science*, volume 367, pp. 3–20. Springer, 2013.

2. Batsakis (A.), Burns (R.), Kanevsky (A.), Lentini (J.) et Talpey (T.). – Ca-nfs : A congestion-aware network file system. *ACM Transactions on Storage (TOS)*, vol. 5, n4, 2009, pp. 1–24.

3. Behzad (B.), Byna (S.), Prabhat et Snir (M.). – Optimizing i/o performance of hpc applications with autotuning. *ACM Trans. Parallel Comput.*, vol. 5, n4, mars 2019.

4. Bez (J. L.), Byna (S.) et Ibrahim (S.). – I/O access patterns in hpc applications : A 360-degree survey. *ACM Comput. Surv.*, vol. 56, n2, septembre 2023.

5. Boito (F. Z.), Inacio (E. C.), Bez (J. L.), Navaux (P. O. A.), Dantas (M. A. R.) et Denneulin (Y.). – A checkpoint of research on parallel i/o for high-performance computing. *ACM Comput. Surv.*, vol. 51, n2, mars 2018.

6. Dorier (M.), Antoniu (G.), Ross (R.), Kimpe (D.) et Ibrahim (S.). – Calciom : Mitigating i/o interference in hpc systems through cross-application coordination. – In *2014 IEEE 28th International Parallel and Distributed Processing Symposium*, pp. 155–164, 2014.

7. et al. (F.). – Software Engineering Meets Control Theory. – In *Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)*, pp. 71–82, Firenze, Italy, mai 2015.

8. Gainaru (A.), Aupy (G.), Benoit (A.), Cappello (F.), Robert (Y.) et Snir (M.). – Scheduling the i/o of hpc applications under congestion. – In *2015 IEEE International Parallel and Distributed Processing Symposium*, pp. 1013–1022, 2015.

9. Guilloteau (Q.), Cerf (S.), Bleuse (R.), Robu (B.) et Rutten (E.). – Under Control : A Control Theory Introduction for Computer Scientists. – In *ACSOS 2024 - 5th IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS 2024)*, pp. 1–10, Aahrus, Denmark, septembre 2024. IEEE.

10. Hellerstein (J. L.), Diao (Y.), Parekh (S.) et Tilbury (D. M.). – *Feedback Control of Computing Systems*. – Hoboken, NJ, USA, John Wiley & Sons, Inc., 2004.

11. Rutten (E.), Marchand (N.) et Simon (D.). – Feedback Control as MAPE-K loop in Autonomic Computing. In : *Software Engineering for Self-Adaptive Systems III. Assurances.*, pp. 349–373. – Springer, janvier 2018.

12. Shevtsov (S.), Berekmeri (M.), Weyns (D.) et Maggio (M.). – Control-theoretical software adaptation : A systematic literature review. *IEEE Transactions on Software Engineering*, vol. PP, 05 2017, pp. 1–1.

13. Tarasov (V.), Zadok (E.) et Shepler (S.). – Filebench : A flexible framework for file system benchmarking. *login Usenix Mag.*, vol. 41, 2016.

14. Tipu (A. J. S.), Conbhuí (P. O.) et Howley (E.). – Artificial neural networks based predictions towards the auto-tuning and optimization of parallel io bandwidth in hpc system. *Cluster Computing*, vol. 27, n1, décembre 2022, p. 71–90.

15. Yildiz (O.), Dorier (M.), Ibrahim (S.), Ross (R. B.) et Antoniu (G.). – On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems. – In *IPDPS*, pp. 750–759. IEEE, mai 2016.
# Appendix

<figure><img src="image_4.png" /><figcaption>FIGURE 3 – Graphical explanation of the different properties of a controlled system (stability, settling time, overshoot) [12]</figcaption></figure>

## Parameter tuning :

The proportional-integral (PI) controller gains $K_p$ and $K_i$ are determined using the system parameters $a$ and $b$, as well as the design specifications, which include the settling time $K_s$ and the overshoot $M_p$. These are given by :

$$ \begin{cases} K_p = \frac{a-r^2}{b} \\ K_i = \frac{1-2r \cos \theta + r^2}{b} \end{cases} \qquad (2) $$

where :

$$ r = \exp \left( -\frac{4}{K_s} \right) $$

$$ \theta = \pi \frac{\log r}{\log M_p} $$
<figure><img src="image_5.png" /><figcaption>(a) Identification experiments to model Static behavior of the dispatch queue size (y-axis) for fixed values wiops input (x-axis))</figcaption></figure>

$$y(i + 1) = a * y(i) + b * u(k), a = 0.76241, b = 0.00029$$

<figure><img src="image_6.png" /><figcaption>(b) Identification experiments to model dynamic behavior of the dispatch queue size (y-axis) : The red signal re-present the real dispatch queue size at run-time (red signal) for different values of the wiops inputs (dashed black line), and we compare this to our linear model (green signal). The model fits well for a certain operation region and poorly for others.</figcaption></figure>

<caption>FIGURE 4 – Open-Loop Experiments used for Identification– No Feedback or Control : The upper plot illustrates the system's static behavior under fixed values input, while the lower plot depicts its dynamic response to input varying during run-time.</caption>
Algorithm 1 illustrates the control logic :

**Algorithm 1** Proportional Integral (PI) - based Control

**Require:** wiops<sub>sat</sub> (saturation limit), k<sub>p</sub> (proportional gain), k<sub>i</sub> (integral gain), Q<sub>sat</sub> (queue sa-
turation limit), Ts = 100ms

1: Initialize wiops ← wiops<sub>sat</sub>

2: Initialize I ← 0

3: Set sampling time : Ts = 100ms

4: **while** System is running **do**

5: Measure queue size : Q

6: Compute error : e ← Q<sub>ref</sub> − Q

7: Compute proportional term : P ← k<sub>p</sub> · e

8: Update integral term : I ← I + k<sub>i</sub> · e · Ts

9: Compute control action : wiops ← P + I

10: Apply wiops

11: Wait for Ts

12: **end while**

**Algorithm 2** PI-based Control with Event Activator and Anti-Windup

**Require:** wiops<sub>sat</sub> (saturation limit), k<sub>p</sub> (proportional gain), k<sub>i</sub> (integral gain), Q<sub>sat</sub> (queue sa-
turation limit), Ts = 100ms

1: Initialize wiops ← wiops<sub>sat</sub>

2: Initialize I ← 0

3: Set sampling time : Ts = 100ms

4: **while** System is running **do**

5: Measure queue size : Q

6: Compute error : e ← Q<sub>ref</sub> − Q

7: **if** Event Activator is triggered **then**

8: Compute proportional term : P ← k<sub>p</sub> · e

9: Update integral term : I ← I + k<sub>i</sub> · e · Ts

10: Compute control action : wiops ← P + I

11: **if** wiops > wiops<sub>sat</sub> **then**

12: Limit control action : wiops ← wiops<sub>sat</sub>

13: Adjust integral term : I ← I − k<sub>i</sub> · e · Ts

14: **end if**

15: **else**

16: Set wiops ← wiops<sub>sat</sub>

17: **end if**

18: Apply wiops

19: Wait for Ts

20: **end while**