# Autonomic Resource Harvesting in HPC: Control Methods and their Reusability

Quentin Guilloteau, Raphaël Bleuse, Sophie Cerf, Bogdan Robu, Rosa Pagano, Éric Rutten

► To cite this version:

Quentin Guilloteau, Raphaël Bleuse, Sophie Cerf, Bogdan Robu, Rosa Pagano, et al. Autonomic Resource Harvesting in HPC: Control Methods and their Reusability. ACM Transactions on Autonomous and Adaptive Systems, 2025, pp.1-24. 10.1145/3764590. hal-05234898

HAL Id: hal-05234898

https://inria.hal.science/hal-05234898v1

Submitted on 2 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Autonomic Resource Harvesting in HPC: Control Methods and their Reusability

QUENTIN GUILLOTEAU, Univ. Grenoble Alpes, Inria, CNRS, LIG, France

RAPHAËL BLEUSE, Univ. Grenoble Alpes, Inria, CNRS, LIG, France

SOPHIE CERF, Univ. Lille, Inria, CNRS, Centrale Lille, CRISTAL, France

BOGDAN ROBU, Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, France

ROSA PAGANO, Univ. Grenoble Alpes, Inria, CNRS, LIG, France

ÉRIC RUTTEN, Univ. Grenoble Alpes, Inria, CNRS, LIG, France

High Performance Computing (HPC) systems are subject to dynamical variations occurring in e.g., jobs execution duration, I/O quantity, network consumption. Adapting to these unpredictable variations requires using autonomic management in an online feedback loop. The introduction of control theory methods allows for the design of well-founded autonomic managers. Choosing the relevant approach is daunting due to the variety of existing controllers. The criteria are of different natures, involving performance and efficiency, but also required expertise in control theory, and reusability or portability between sub-systems. Therefore, there is a need for comparative studies to assist designers choices.

We consider the problem of resource harvesting in HPC systems, where scheduling often leaves resources idle. Our approach controls – through a feedback loop – the injection of small jobs in order to maximize the resources’ usage. The control problem is to manage the trade-off between harvesting and performance, in a reusable manner. We study how reusability relates to the adaptivity and robustness properties in control. We illustrate our approach with the classic Proportional-Integral-Derivative (PID) control, its upgrade as adaptive control, and Model-Free Control (MFC). We target CiGri, a system harvesting idle resources in a computing grid. We perform experimental evaluation and compare performance and reusability. Trade-offs are found on different criteria: while adaptive control is largely portable, its design complexity is significant for non-experts; PID control has good nominal performance, yet its portability is limited; MFC requires few competences to be used, but cannot provide strong guarantees.

CCS Concepts: • Computer systems organization → Grid computing; • Software and its engineering → Scheduling; Extra-functional properties; • Computing methodologies → Modeling methodologies.

Additional Key Words and Phrases: control theory, autonomic computing, resource management

## ACM Reference Format:

Quentin Guilloteau, Raphaël Bleuse, Sophie Cerf, Bogdan Robu, Rosa Pagano, and Éric Rutten. 2025. Autonomic Resource Harvesting in HPC: Control Methods and their Reusability. *ACM Trans. Autonom. Adapt. Syst.* 1, 1, Article 1 (January 2025), 25 pages. https://doi.org/10.1145/3764590

Authors' Contact Information: Quentin Guilloteau, Univ. Grenoble Alpes, Inria, CNRS, LIG, Grenoble, France, quentin.guilloteau@univ-grenoble-alpes.fr; Raphaël Bleuse, Univ. Grenoble Alpes, Inria, CNRS, LIG, Grenoble, France, raphaël.bleuse@inria.fr; Sophie Cerf, Univ. Lille, Inria, CNRS, Centrale Lille, CRISTAL, Lille, France, sophie.cerf@inria.fr; Bogdan Robu, Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France, bodgan.robu@univ-grenoble-alpes.fr; Rosa Pagano, Univ. Grenoble Alpes, Inria, CNRS, LIG, Grenoble, France, rosapagano0976@gmail.com; Éric Rutten, Univ. Grenoble Alpes, Inria, CNRS, LIG, Grenoble, France, eric.rutten@inria.fr.

<figure><img src="image_2.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.

© 2025 Copyright held by the owner/author(s).

ACM 1556-4703/2025/1-ART1

https://doi.org/10.1145/3764590
# 1 Introduction

Computation- and data-heavy workloads have become essential tools for studying complex phenomena. Such workloads are commonly used in physics, climate science, biology, and others. Personal computers are no longer up to the task due to the requirements in computation power, memory, or bandwidth. High Performance Computing (HPC) systems, regrouping powerful machines and high-bandwidth storage interconnected with a fast network, allow for parallel and efficient use of resources. Such systems are however costly to build and operate. Ergo, HPC systems are shared by multiple users.

The Resource and Job Management System (RJMS) is a key component to operate a HPC system [18, 44]. Users submit jobs: a description of their computation, data, and resource requirements. With respect to the resources status reported by the resource manager, the scheduler decides which resources to allocate to a job and assigns a time slot for the job's execution. A job uses computing power and requires input/output (I/O) operations during its execution. The performance of I/O operations is paramount as slow I/O can result in the job exceeding its allocated time slot: in such a case the RJMS kills the job and the computed results are lost. Figure 1 summarizes the job submission process.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Overview of a job submission on a HPC system [18].</figcaption></figure>

## 1.1 The Need of Runtime Management for High Performance Computing

The quest to squeeze the performance out of the available hardware and software stacks leads to complex systems. HPC systems are difficult to model, and classical optimization techniques struggle to efficiently manage resources. The performance of the resource management is inherently limited by the quality of the models. Yet obtaining a high-quality model requires enormous quantity of data that are difficult if not impossible to collect: the monitoring overhead would collapse the performance of the system. Furthermore, the dynamics and variability in both hardware and application behaviors have an impact on performance and energy consumption. For example, processors of identical specification can vary in speed or thermal behavior [6, 51]. The architecture of clusters is not set in stone: network topologies, file systems do evolve. Furthermore, it is difficult to cope with variability and emergent unpredictable phenomena at runtime. Applications have phases with varying resources acting as bottlenecks: compute-, memory- or I/O-bound phases. The speed of the processors is impacted by the temperature and the instruction set in use [28, Table 2-12].
Hence, HPC systems require runtime management. We focus in this work on runtime management achieved via feedback loops. More particularly, we study the need to adapt to variability over time, but also the need to cope with variations of systems/plants. The latter point is related to the need for managers to be reusable across a distributed architecture.

## 1.2 Runtime Autonomic Management for Self-Adaptive HPC

We therefore claim that HPC systems can benefit from the autonomic computing philosophy [31]. Rather than relying on complex models, runtime management is performed by continuously monitoring the system and adapting its configuration accordingly. Such an approach allows for simple yet reactive and efficient management. The RJMS is the component where management decisions are being taken at runtime, according to measurements made on the infrastructure, and transformed into actions upon the system: therefore it is a natural candidate to host the composition with an autonomic manager.

Control theory is a natural tool from the autonomic computing toolset for our use case. One of the main advantage of control theory is that it provides explainable and mathematically proven tools that provides reliable and stable performance for the user.

## 1.3 Contributions of the Paper

The design of such controllers is the object of this paper, where we target more particularly CiGri, a system harvesting idle resources in a computing grid. We give special attention to the study of the relationship between reusability and the properties of adaptivity and robustness in control.

We recall in Section 2 the basic notions of autonomic control, the need for reusability, and the need of regulation in HPC.

The main contribution of this work is the discussion in Section 3 around the notion of reusability of control-based managers from the point of view of HPC system administrators. The comparison criteria (performance, robustness, ease of use) are classic: the novelty lies in their interpretation and formalization from the perspective of control theory. More precisely, we translate the criteria into measurable metrics, and we enhance the criteria usually considered by control engineers. Such a formalization is unconventional when studying self-adaptive systems. In particular, the formalism of control theory brings sound guarantees.

Given the almost impossible aspect of providing general theoretical tradeoffs for all HPC systems, we focus on a meaningful real world example. The experiments rely on a real-world use case to assess realism and relevance of the approach on a real HPC system. This allows to extract empirical, yet useful, practical tradeoffs. We validate in Section 6 our approach with an experimental campaign conducted in a reproducible manner on the target system for our study: the CiGri middleware and its resource harvesting loop. CiGri is described in Section 4. We study three types of controller relevant for the target system: a classical PID control, its upgrade as adaptive control, and model-free control. Section 5 describes the controllers. The results are discussed in the light of the criteria previously defined.

# 2 Background

## 2.1 Resource Management and Harvesting in High Performance Computing

We consider in this work the use case of resource management in HPC, and more specifically the harvesting of unused resources in clusters. The presence of unused resources results from the limits of the scheduling of jobs. The allocation of resources to jobs, while respecting all constraints, leaves resources idle. Such inefficiency is sometimes referred as fragmentation. The loss of computing power resulting from the fragmentation represents an exploitable pool of resource.
Previous work has proposed to submit a new type of jobs to be scheduled on this pool. This new type of jobs consists of jobs that are more flexible than classic jobs, and a lower quality of service has to be guaranteed. These flexible jobs require less resources, are interruptible, and are given a lower priority than classic jobs submitted by premium users. This strategy has been used in various contexts: personal machines [2, 17], HPC clusters for Big-Data [37], or serverless computing [43]. The scheduler considers these flexible jobs as second class citizens, and can preempt them should a higher-priority jobs require the resources. Yet, these jobs still impact the performance of the system: they have an impact on shared resources that are not taken into account or modeled at the scheduling level. For example, the load induced on the communication network, the file system, or the caches, does hinder the performance of the jobs of premium users.

Harvesting the pool of unused resources calls for a runtime management in order to handle the above-mentioned phenomena. This can be done using the tools of control theory. However, the challenge then lies in the ability to design a controller for a given system, while ensuring it can be deployed with acceptable performance on other systems of different characteristics, as a form of reusability. Characterizing the performance can be done on multiple dimensions. Hence there is a need for comparative studies to support the design choices.

## 2.2 Autonomic Control and the Need for Reusability

Autonomic computing relies on feedback loops to regulate systems [31]. This regulation can take many forms based on the systems and its objectives: self-configuration, self-healing, self-optimization or self-protection. An autonomic system is governed by the MAPE-K loop, composed of five phases: (i) Monitor: when the autonomic controller will sense the regulated system, (ii) Analyze: based on the value of the sensors decided whether to take a decision, (iii) Plan: if there is the need for a decision then chose the one to satisfy the guarantees of the system, (iv) Execute: apply the action of the plan onto the system, and (v) Knowledge: contains information about the system or the requirements.

The architectural concept of the *autonomic controller* is simple, and does not dictate implementation details. Autonomic computing gives a framework for implementing feedback loops, and the interpretation of these loops can take several forms. One can find in the literature autonomic controller implemented using rules or artificial intelligence [26, 41]. Rule-based implementations are simple to understand (for a small number of rules) but fail to provide guarantees on the system. Machine learning can provide some guarantees (e.g., training accuracy) on the closed-loop system but it lacks explainability and its inner workings remain a black box to the user. To the best of our knowledge, control theory tools provide a unique trade-off, as they remain simple to understand and provide proven guarantees on the closed loop system. The amount of data required by the different approaches is another important dimension. Rule-based systems rely on small datasets. On the contrary, machine learning need to ingest gigantic amount of data during training, and relies on huge model for inference. Here again, control theory provides a unique trade-off.

Software cybernetics can be also seen as a more general notion of self-adaptive software [55]. The main concepts overlap with classic notion of control theory. Control theory is notably mentioned in [55] as one of the approaches falling under the category of model-based decision and control mechanisms in software systems.

*Strengths and Limits of Approaches Based on Control Theory.* Control theory has been identified as a powerful tool to design autonomic managers. It provides a strong basis to design clear, well-founded and well-understood components [36, 47]. In particular, control theory offers tools to design feedback loop with proven performance guarantees. Consequently, this approach has been
applied to manage various components of computing systems [13, 14, 25, 30, 41, 48]. Recent works
apply such techniques to the management of resources in the context of HPC [21, 23, 54].

Despite its promising results and guarantees, a wider application of control theory is limited
by two factors: it has a steep learning curve, and computer scientists are not acquainted with the
concepts [14, 34]. There exists a great variety of control approaches: from the simplest one such as
PID control (proportional, integral, derivative) to more elaborate auto-tuning techniques including
non-linear control, model-predictive control, optimal control among others. These approaches offer
efficient and safe solutions to classes of problems. Elaborate techniques require precise and complex
models of the systems under control, as well as deep expertise in control theory. It is difficult for
HPC system designers and administrators to select a control approach for a given project. The
criteria are numerous: performance and efficiency are natural, but reusability or portability between
sub-systems do matter. Some controllers suffer from a too tight coupling to the particular system
they are designed and tuned for. Yet, it is unreasonable to expect HPC system administrators to
have the skills required to design/tune controllers. Hence we focus in the remaining of this paper
on control approaches relying on simple linear models where an advanced control theory expertise
is not required.

The feedback loop is first designed to manage the trade-off between the efficiency of the harvesting and the performance degradation induced on the priority jobs. One also expects that a controller be as reusable as possible. The controller should be deployable in different contexts with varying amount of processors, operating systems, applications characteristics, while keeping satisfactory performance. This should be possible with little or no modification of the controller. This is a difficult design goal as the control theory methodology requires going through an *identification* phase of the system: this phase characterizes (models) the behavior of the system w.r.t. the actuators and sensors. The identification induces a tight coupling between the controller and the system, effectively limiting the reusability of the former on other systems.

*Reusability.* Reusability and modularity are welcome and crucial computer science properties [42]. For example, to ease the maintenance, an administrator will prefer to deploy the *exact same* controller on all the clusters of a HPC center. While a specific control design for each cluster could allow for improved performance, it significantly complexify the implementation and maintenance of the whole system, and offer little flexibility when new clusters are added. Such properties are not usually considered by control theory, where the focus is on adaptation on the time dimension rather than other dimensions. Hence, the adoption of control-based tools in HPC centers could be limited by their lack of reusability, as it would require administrators to maintain *several versions* of the controllers for the different clusters of the center. To the best of our knowledge, comparative studies exploring potential approaches and addressing reusability of controllers are rare, and focus on a single dimension of the problem such as performance [45].

There exist similar approaches to ours, using control theory as an inspiration of as a formal
technique [27, 50, 53, 55]. In summary, our work departs from these existing works in several ways:
(i) these works do not consider the resource harvesting problem, (ii) we conduct a *comparative study* backed by an *extensive experimental campaign* of several controllers on a practical example,
and (iii) we consider *several criteria*.

3 Reusability of Control-Based Managers

Several aspects should be considered when comparing controllers reusability: from the design
phase and setup, to usage and possibly portability. When reusing a controller, one may consider
reuse without modification, reuse with different parameter values, or a more-involved re-design of
all or part of the controller. In this section, we present five complementary criteria to characterize reusability, along with metrics to evaluate them. We further discuss their interdependence and known trade-offs, from a theoretical point of view.

## 3.1 Criteria for Comparison

The controllers are compared based on five criteria, each answering a specific design or evaluation problematic:

**Nominal Performance** To what extent and how does the controlled system meet its objectives, in the experimental conditions in which the controller has been designed?

**Portability** How will the controlled system behave if the system varies from the nominal version considered for design? What set of systems can the controller manage?

**Setup complexity** To what extent is the controller plug and play? How easy is it to set up the controller?

**Support Guarantees** Is the controller design backed by a methodology bringing behavioral guarantees? Under which hypothesis do the guarantees hold?

**Competence required** How knowledgeable one needs to be to soundly design the controller?

The next sections define in details each criterion, discuss their relevance and propose metrics to characterize them.

### 3.1.1 Nominal Performance.

**Definition.** Nominal performance characterizes the ability of the system to perform its adaptation task. Data- and model-based controllers are inherently dependent on the configuration and experimental conditions in which the design was done. Usually, performance of an autonomous system is evaluated in similar conditions, what we call *nominal* system. Nominal performance covers both aspects of transient, dynamical behavior, and converged, stabilized behavior.

Quantitative metrics of nominal performance are mostly used when controllers (or controllers' tuning) need to be compared. Note that some thresholds can exist, beyond which the adaptation is no longer considered valid. This is the case, for example, during a reconfiguration phase.

**Metrics.** Classical control theory provide three metrics to characterize the dynamical behavior of a controlled system. Note that stability is eluded here, as we suppose all controllers are functional. **Precision** reflects the ability to reach the desired target. It is measured using the *static error* metric, giving the sum of the absolute distances between the measured performance and its reference value, once the system converged. 0 is the best value: the smaller, the better. **Rapidity** characterizes the speed of convergence of the controlled system. It is measured using the *response time* metric, computed as the time needed from the performance signal to reach a given percentage of its final converged value. The smaller, the better. Note that the noisiness of the signals should be taken into account. When facing a very noisy signal, one should lower the percentage and consider, for example, the 85 % response time instead of the usual 95 % response time. **Quality** of the signal is also considered in cases there are oscillations in the performance signal. It can be measured via the *overshoot* metric, the relative value of the highest peak of performance signal. 0 is the best value: the smaller, the better.

### 3.1.2 Portability.

**Definition.** When used in real conditions, the controller may have to manage a system that does not correspond to the nominal situation. Portability captures the ability of a controller to be applied, without or with limited modification, to a system different to the nominal design one, and to keep adequate performance. Two cases can be distinguished for the portability conditions. First, the
variation occurs at runtime, i.e., the system deviates from the nominal version, possibly due to
some disturbances or external events. The ability of a controller to keep good performance in case
of runtime variations of the system is to be linked with the well studied property in control theory
of robustness. Robust controllers are able to reject disturbances: they can ensure that the effect
of the disturbance on the system remains limited and mastered. Second, the controller may be
applied to a system with different configurations: for example variations may occur offline, or on
different runs. Such software or hardware configuration changes can be significant, for instance
when different clusters are considered. This property is rarely considered in control theory for
physical system, where the notion of different *runs* does not exist: controllers continuously monitor
the same physical system.

Note that the nominal system may be different from the expected system on which the adaptation is to be performed. Indeed, it is not always possible or desirable to precisely characterize or model the real systems' conditions. For instance, the analysis on the duration of best-effort jobs of a campaign is a difficult task [22], it is not realistic to suppose that it should be done for each campaign.

*Metrics.* Performance metrics of precision, rapidity, and quality can be used to analyze the controlled system. Different systems can be tested, by varying the configurations and injecting disturbances, when possible. The distance between the current system and the nominal system can be characterized, to create a set of systems that the controller can handle. Portability is then the performance achieved by a controller for several configurations and disturbances.

### 3.1.3 *Setup Complexity.*

*Definition.* Autonomous controllers are implemented and used by system experts, who are not a priori knowledgeable in control theory. A controller with low design complexity is more likely to be understood, correctly implemented, and tuned. If performances are acceptable, the simplest controller may be the best option. Note that design complexity is different from the mathematical complexity of the controller algorithm, whereas rather addresses the user experience aspect. The equations underlying the controller algorithms usually are simple with a straightforward implementation. The implementation has a minimal footprint on computing resources (e.g., a small constant number of arithmetic operations on a small constant number of variables at each time step for a PID controller). The complexity however resides in understanding the mathematical framework used to derive the control equations and tune the controller.

*Metrics.* There is no commonly agreed and general metric for design complexity. We propose to use as a metric the number of input parameters of the controller model plus the number of initial conditions that have to be tuned in the controller algorithm. While this metric only covers part of the difficulties a user can experience in developing a controller, we believe that it simply gives a meaningful quantitative metric that can be applied to any algorithm. Note that the nature of the parameters can differ w.r.t. the control type. Some parameters may represent information about the model of the system, others may be a user-tunable value of the controller algorithm. Some parameters can either be set based on theoretical results or extracted from past data following a well defined identification methodology, while other parameters are set based on trial-and-error experiments.

### 3.1.4 *Support Guarantees.*

*Definition.* One advantage of control theory-based adaptation is regarding the mathematical guarantees one can have that the controlled system will perform as desired. Guarantees are not a binary property: there are nuances beyond having or not having guarantees. First, a methodology
is needed for the controller tuning. Then, the performance properties that are ensured come with some conditions, depending on the system’s variations or on appropriate system excitation. There exists a gradation of guarantees: from safety-critical system requiring strong guarantees, system subject to SLAs, to best-effort system without the need of guarantees.

*Metrics.* Measuring guarantees level to compare controllers is not an easy task, there is no agreed-upon metric to characterize this criterion. We opt out for a qualitative evaluation of guarantees. To do so, we look at the availability of a mathematically-backed methodology for controller tuning, and at the hypothesis and conditions that need to be fulfilled to have those guarantees.

### 3.1.5 *Competence Required.*

*Definition.* Designing and tuning a controller is not straightforward. PID controllers are known for being fairly accessible, resulting in their predominance in the industrial applications. However, the proper design of a PID controller may require control expertise: as it is the case to perform sound identification experiments and tune accordingly the PID parameters. Advanced techniques such as optimal control, model predictive control, or robust control require even more domain-specific knowledge. The need for a control expert leads to additional costs and delays, that may orient systems' experts toward simpler solutions.

*Metrics.* Evaluating the competence required for designing and using a controller will be qualitative. Such metric reflects the complexity of the design process, where the need for complementary actions such as feedforward, anti-windup, preventing bursting, etc., have to be investigated. The metric also covers the domain-specific knowledge needed for fine-tuning and to properly understand the achieved guarantees.

## 3.2 Theoretical Trade-Offs Between Criteria

No controller is expected to outperform the others on all criteria. Control theory highlights the inherent discrepancies between nominal performances and reusability. Design complexity and guarantees are also hardly compatible. Only a trade-off can be achieved. Then, strategies for concealing multi-criteria have to be applied, such as setting priorities, weighting the criteria, or finding Pareto-optimal sets. Such choice of priorities or weights is not straightforward, and can largely differ depending on the application context and objectives. This work focuses on this challenge by evaluating various controllers on complementary criteria. The objective it to highlight which controller to choose, given one's priorities.

In the following, we discuss two important trade-offs between pairs of criteria.

### 3.2.1 *Trading Nominal Performance for Reusability*
Concealing optimality and robustness is a primary concern in the control field. Having highly performing controllers on the nominal system may come at the cost of limited performances, or even instability, in case the actual system differs significantly. Relaxing constraints on precision, rapidity, and quality can allow for the reusability of the controller on a large class of systems.

### 3.2.2 *Trading Support Guarantees for Low Setup Complexity and Limited Control Competences*
Strong guarantees on the behavior of the controlled system over a wide variety of condition can be achieved, however it often requires using advanced control techniques. Properly designing such controllers assumed significant control engineering knowledge, and its implementation and usage are consequently complex. This discussion on the simplicity of the controller usage, even at the cost of loss of performance guarantees, is very common in the control field. However, it reflects the reality of systems' experts when using controllers in practice.
# 4 Resource Harvesting with CiGri

We use in this work CiGri to conduct experiments and illustrate the comparison framework defined in Section 3. CiGri aims at harvesting idle computing resources on HPC systems by submitting small and interruptible jobs to the RJMS [7, 19]. CiGri stands out from other state-of-the-art solutions as the submitted jobs are regular jobs with a lower priority.

This section describes in further details how CiGri and the operated HPC system interact. The various implemented controllers are described and discussed in Section 5.

## 4.1 Resources Harvesting

When using CiGri, jobs can be submitted in two ways: directly to the job manager of the RJMS, or to CiGri. In the latter case, CiGri is then responsible to submit its jobs to the RJMS. Jobs directly submitted to the RJMS are called *priority jobs*, while jobs submitted through CiGri are called *best-effort jobs*. Unlike priority jobs, best-effort jobs have the lowest priority and can be killed to re-allocate their resources to priority jobs. Typical best-effort jobs consist of the submission of *bag-of-tasks* applications, a.k.a. *campaigns*: a large number of small independent parametric jobs. These jobs are referred as *embarrassingly parallel jobs* and are perfect candidates to exploit the idle resources of a cluster. In this work, we consider that all the jobs of a campaign share a common behavior in execution duration and I/O patterns [22].

## 4.2 Jobs Submission

*Challenges.* One must be careful when designing the submission policy of CiGri: the rate at which best-effort jobs are forwarded to the RJMS should be high enough to fill the idle gaps, but low enough not to hinder the HPC system nominal performance. In other words, the absence or presence of best-effort jobs should be transparent to priority jobs. For example, one cannot submit all the jobs of a campaign at once, as submitting too many jobs at once will drastically increase the time to compute the schedule.

Furthermore, many resources, with the exception of computation resources, are shared between jobs. In particular, the file system and network are potential bottlenecks. Every running job places a load on the file server: it either reads, writes, or does both. Too many simultaneous read/write requests may overload the file server, and impede its responsiveness [11, 51]. In extreme cases, too high a load may need a restart of the servers hosting the file system.

*Original Algorithm.* CiGri relies on a simplistic tap-like heuristic to determine the number of jobs to submit to the RJMS [19]. Every 30 s, CiGri checks if previously submitted jobs are still running. If no jobs remain, CiGri opens the tap, submits a few new jobs, and closes the tap. The number of submitted jobs is determined, per campaign, by the initial number of submitted jobs and follows a geometric progression.

This algorithm is not satisfactory. It fails to harvest an arbitrary number of idle resources should a single slow job prevent CiGri from opening the tap. It also fails to avoid overloading shared resources should it submits too many I/O-intensive jobs.

## 4.3 Dealing with Variability

The original behavior of CiGri is unsatisfactory as it depends on the dynamic behavior of the HPC system but fails to capture the relevant information: abrupt changes in the number of idle resources due to the dynamics of priority jobs, best-effort jobs behaviors, or file server evolving load. Despite its shortcomings, the original algorithm lays the foundations of a feedback loop. As depicted on Fig. 2, one may observe that the decision to open/close the tap relies on the retrieval of some information about the system state: namely, the state of the waiting queue for best-effort jobs.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Architecture of the CiGri system. CiGri takes jobs from *bag-of-tasks* applications, also called *campaigns*. It submits them to the RJMS that will schedule them onto the physical resources. During their execution, jobs can do some I/O operations on the file system. CiGri collects and feeds the state of both the file server and the RJMS to the controller to set the next quantity of jobs to submit. The duration between each submission is 30 s.</figcaption></figure>

Designing a submission algorithm for CiGri so that it is reusable requires to take into account different kinds of variability. Apart from the above-mentioned variability arising at runtime, an acceptable solution should smoothly deploy on various production environments. Hence, the solution should cope with varying number of nodes or system configurations.

# 5 Control Approaches for CiGri

This section presents the design of a controller following the control theory method, that will regulate the system by submitting jobs from a *bag-of-tasks* application in a controlled fashion.

The methodology for designing controllers [9, 13] consists first in defining the objectives, knobs and sensors; followed by, in case of model-based approaches, devising a mathematical representation of the system's behavior; then the controller in itself is designed and tuned; eventually, controllers are experimentally validated.

In this section, we present first the control problem formulation. Then, three controllers are presented:

(1) A basic control algorithm, namely a Proportional-Integral (PI) control, consolidating the one constructed in [52]. The PI control computes the action based on the error between the sensor measure and its desired value. The control action has two parts: a part proportional to the current error, and a part proportional to the time integral of the error.

(2) A novel auto-adaptative control algorithm, following a common approach in control theory to provide reusability. Various techniques exist: for the sake of a fair comparison, we choose to build the adaptive part on top of the existing PI control. This controller is able to self-adapt to various configurations of the system, as the behavior of our system depends on jobs characteristics that we are not able to efficiently predict ahead of their actual execution.

(3) A control algorithm which does not need a model of the system. Several techniques fall under this category: for the sake of fair comparison we use a Model-Free Controller [23], as it can be tuned to have similar performances as the PI.

The various control approaches are described in pedagogic detail for a Computer Science readership. We do so in order to explicitly show the design steps and the work involved in their setup. In
particular, we focus on the parameter tuning step, as we rely on the existing body of control theory literature to define the parameters. This allows us to thereby exhibit the level of expertise needed.

## 5.1 Control Problem Definition

The system we work with is made up of three main components: the hardware (cluster and file server), CiGri and the RJMS. The structure of our feedback loop is recalled in Fig. 2.

As a sensor, we use the UNIX load average /proc/loadavg sensor [12]. This metric is well known by system administrators for monitoring the load of the file server. We denote $y(k)$ the output of this sensor at step $k$. A value of loadavg greater than the number of threads in the system indicates an overload. This metric has the particularity of having inertia, a property for which control theory provide suitable tools. By controlling the load of the file server at a constant value, we could avoid CiGri-induced overload of the file server. Hence, we are controlling the impact of the resource harvesting on the high priority applications.

As an actuator – also called knob by control theory practitioners – we will be adjusting the number of jobs that CiGri submits to the RJMS. This number is our control action. We denote $u(k)$ the control input of the system at step $k$.

The number of jobs submitted by the premium users of the cluster plays the role of an external signal. We do not have any control on those jobs and consider them as the perturbation.

In this context, our control objective is to aim at **minimizing the under-usage of the cluster** while **avoiding impacting nominal performances**, w.r.t. file server or scheduler overload. In other terms, we aim at tracking a reference value for the file server load, and reject the disturbances caused by premium users. We denote the reference file server load as $y^*$.

## 5.2 Proportional-Integral Controller

The Proportional-Integral (PI) control computes the action based on the error between the sensor measure and its desired value. The control action has two parts, one proportional to the error, and one based on the integral of this error. Some parameters weight the impact of those two actions.

The first step in the design of the PI controller is to get an approximate model of the system, so as to soundly tune the controller parameters. Note that PI controllers can be used without models, however, without providing the same guarantees.

*Modeling.* We suppose that we have an approximate dynamical model of the form:

$$y(k + 1) = a \cdot y(k) + b \cdot u(k) \quad (1)$$

that is a first order system (i.e., the next sensor value at time $k + 1$ only depends on the signals current values at time $k$). Here $a$ and $b$ are scalar parameters of the model, where $a$ represents the vanishing effect of the sensor, and $b$ the impact of the action on the sensor.

By definition, the load average metric [12] updates itself every 5 s following an exponential smoothing of weight $W = \exp(-\frac{5}{60})$. Given that our sampling period of CiGri is $\Delta t$, one has:

$$a = W^{\frac{\Delta t}{5}} = \left( \exp\left(-\frac{5}{60}\right) \right)^{\frac{\Delta t}{5}} = \exp\left(-\frac{\Delta t}{60}\right) \quad (2)$$

In order to find the value of $b$, we can look at the behavior of our system in steady state, when the system is stabilized. This happens when $y(k + 1) = y(k) \triangleq y_{ss}$ and $u(k + 1) = u(k) \triangleq u_{ss}$:

$$y_{ss} = a \cdot y_{ss} + b \cdot u_{ss} \iff b = \frac{(1 - a) y_{ss}}{u_{ss}} \quad (3)$$

To find the stabilized ratio $\frac{y_{ss}}{u_{ss}}$, we can perform so-called open-loop experiments, that is, we set the input and observe the output stabilize, without any feedback from the system.
<figure><img src="image_5.png" /><figcaption>Fileserver Load for different Submissions (number of jobs and filesize)</figcaption></figure>

Fig. 3. File server load for different submissions in number of jobs and I/O loads. The graphs represent experiments conducted for the identification phase, with varying quantity of I/O (columns). The load of the file server (y axis) is shown w.r.t. the number of simultaneous write requests/jobs (x-axis). We observe that the loadavg sensor captures the (over)load of the file server.

Therefore, we perform experiments in which we submit different numbers of jobs and see their impact on the load of the file server. The number of jobs will be constant on intervals in a staircase-like manner: this pattern is known in control as a step. We know that the impact of the number of jobs on the load of the file server depends on the size of the jobs. Then, we considered four sizes of files: 25, 50, 75 and 100 MB. For each file size, the number of jobs is varying stepwise, with the following amplitudes: 1, 10, 20, 30, 40 and 50 jobs. For each step, we execute 120 consecutive identical submissions of the step. Results of these experiments are in Fig. 3. We observe that, for a given file size, the relation between the file server load and the number of jobs is fairly linear. By fitting a bi-linear regression on this open-loop data, we get the following relation:

$$y_{ss} = \alpha_1 + \beta_1 \cdot f + \beta_2 \cdot u_{ss} + \gamma \cdot f \cdot u_{ss} \quad (4)$$

linking the output in steady state $y_{ss}$ to the input in steady state $u_{ss}$ and the file size of the current campaign $f$. The numerical values of all these parameters can be found in Section 6.

By combining Eqs. (3) and (4) we get:

$$ \begin{aligned} \lim_{u \to +\infty} b &= \lim_{u \to +\infty} \frac{(1-a)(\alpha_1 + \beta_1 \cdot f + \beta_2 \cdot u + \gamma \cdot f \cdot u)}{u} \\ &= (1-a)(\beta_2 + \gamma \cdot f) \end{aligned} \quad (5) $$

which shows that the value of $b$ depends also on the file size of the current campaign $f$.

Note that the parameters of the linear regression are valid for the desired operating region of the system. For instance, the estimation of the load $y_{ss}$ as expressed in Eq. (4) yields absurd values for near-zero $u_{ss}$ values. Such an edge-case shows the limit of the model, as the system has a strong nonlinear behavior with very few jobs. Let us emphasize that these parameters are those of a crude model giving a coarse view of the system behavior. This crude model is used to tune the controller. We do not aim nor need to perfectly model the system under control: the innate robustness of the PI control is able to cope with these variations. Furthermore, in the case of the adaptive control (see Section 5.3), the parameters are continuously updated.

**PI Formulation & Tuning.** Once we have the model of the system and its parameters, we can dive into the controller's design. Two parts constitute the PI Controller: the proportional part reacts
<figure><img src="image_6.png" /><figcaption>Fig. 4. Impact of the $k_s$ and $M_p$ parameters on the closed-loop behavior of the system. The smaller $k_s$, the fastest the closed-loop system will converge to the reference value. But too small values of $k_s$ can lead to some overshooting. $M_p$ controls the allowed overshoot: the greater $M_p$, the greater the allowed overshoot.</figcaption></figure>

proportionally to the error while the integral part adds past knowledge to correct the input to reach the desired value.

The following equation governs the controller:

$$u(k) = K_p \cdot e(k) + K_i \cdot \sum_{n \le k} e(n) \quad (6)$$

where $e(k)$ is the control error, and $K_p, K_i \in \mathbb{R}$ are resp. the proportional and integral gains. The control error is defined as the difference between the desired value of the output $y^*$ and the actual output value $y(k): e(k) \triangleq y^* - y(k)$. The values $K_p$ and $K_i$ are tuned based on the model parameters $a$ and $b$, and on the desired behavior of the controlled system.

To represent the desired behavior, we define the two following quantities: $k_s$ represents the maximum time for the closed-loop system to get to the steady state, and $M_p$ represents the maximum overshoot (as a percentage of the reference) allowed. Figure 4 illustrates those two quantities on our model, and the possibilities offered by their tuning on the dynamic behavior of the system. The $K_p$ and $K_i$ gains of the PI controller are then computed in terms of $k_s$ and $M_p$ with the following relations:

$$K_p = \frac{1}{b} \left[ a - \exp\left(-\frac{4}{k_s}\right) \right]^2, \quad (7)$$

$$K_i = \frac{1}{b} \left[ 1 - 2 \exp\left(-\frac{4}{k_s}\right) \cos\left(-\frac{4\pi}{k_s \log_e M_p}\right) + \exp\left(-\frac{4}{k_s}\right) \right]^2 \quad (8)$$
## 5.3 Adaptive PI Controller

In this section, we present an adaptive controller, able to self-adapt to various configurations of the system. Indeed, the behavior of our system depends on jobs characteristics that we are not able to efficiently predict long before it is actually executing itself. An example is the dependency of the parameter b on the file size f, see Fig. 3 and Eq. (3). This motivates the use of adaptive control.

Adaptive control encompasses a set of techniques that dynamically adjust controller parameters in real-time to maintain a desired level of control system performance [33]. We build adaptation on top of the PI controller. In the case of a simplified linear model and a Single Input Single Output (SISO) system, an adaptive Proportional-Integral (PI) controller is frequently employed. The PI controller is initially designed based on the model of Eq. (1), and the adaptive algorithm adjusts the controller parameters to account for the unmodeled dynamics beyond the scope of the linear model [49]. Within the framework of adaptive control, two primary approaches exist: direct and indirect. In the direct approach, the controller parameters are updated directly. Conversely, the indirect approach involves estimating the model parameters and subsequently updating the controller. In our specific case, the indirect adaptive control approach appears more suitable due to the reliance of the PI controller on the system parameter b. Thus, by employing an identification algorithm, we estimate the true value of b(k) as $\hat{b}(k)$ at each time step, to adapt to different jobs, and update accordingly the PI parameters.

### 5.3.1 Controller Design
Our objective is to estimate $\hat{b}(k)$ using an online algorithm. Prior to delving into the core of the algorithm, we define the prediction error $\epsilon$:

$$ \epsilon(k) = y(k) - a \cdot y(k-1) - \hat{b}(k-1) \cdot u(k-1), \quad (9) $$

representing how far our estimated model is from the measure.

Due to the presence of significant noise in the measured data, see for instance Fig. 3, we need an algorithm capable of mitigating noise. Therefore, we improved the widely known Recursive Least Square (RLS) [4] algorithm to bolster its robustness against noise. That is, we substitute $\epsilon$ with its filtered version $\Phi(\epsilon(k))$, given by:

$$ \Phi(\epsilon(k)) \triangleq \frac{\epsilon(k)}{1 + \phi \cdot |\epsilon(k)|} \quad (10) $$

where $\phi \ge 0$ represents a smoothing parameter to be chosen. By doing so, we preserve a linear structure when the error is small, while introducing a slower-than-linear behavior for larger errors. Eventually, we define our robust estimator as:

$$ \hat{b}(k+1) = \hat{b}(k) + V(k+1) \cdot u(k) \cdot \Phi(\epsilon(k+1)) \quad (11) $$

With $V(k+1)$ being:

$$ V(k+1) = \frac{1}{\mu} \cdot V(k) - \frac{1}{\mu^2} \cdot \frac{V(k) \cdot u(k)^2 \cdot V(k)}{1 + u(k) \cdot V(k) \cdot u(k)} \quad (12) $$

where $\mu$ represents the forgetting factor. The presence of changing parameters over time, such as b varying with different jobs, necessitates the use of forgetting effect of $\mu \in (0, 1]$. However, the $\mu$ parameter introduces a significant drawback known as the “bursting phenomena”. As elaborated in [1, 16], when the system lacks sufficient excitation, the algorithm may approach a singularity, resulting in large spikes in the estimated parameter. To mitigate this behavior and to adapt to different changing rates of b, we employ the parallel estimation technique [4, 32], that consists in using a vector of forgetting factors $\mu$, rather than a unique value. Thus, the algorithm can estimate multiple distinct values of $\hat{b}$. The estimation errors (see Eq. (9)) are evaluated for each prediction, and the value of $\hat{b}$ achieving the smallest error is selected. Then, at each time step, once $\hat{b}$ is selected,
we use Eqs. (7) and (8) to compute the new controllers parameters and Eq. (6) to choose the control action.

### 5.3.2 Tuning.
The algorithm relies on different parameters: the forgetting factor $\mu$, the smoothing parameter $\phi$, and the initial conditions $\hat{b}(0)$ and $V(0)$. Note also that the adaptive PI is built on top of the PI, which requires the computation of the model parameter $a$ (see Eq. (2)) and the definition of $k_s$ and $M_p$, two quantities that allow choosing for the closed-loop behavior (see Fig. 4).

*Forgetting Factor.* The parameter $\mu$ is needed for estimating a time-varying parameter. A small $\mu$ results in a fast algorithm, that can track the time-varying parameter swiftly, but it also becomes more sensitive to noise. Conversely, a value closer to 1 leads to a slower algorithm in tracking the parameter; however, it diminishes the impact of noise. By employing the Parallel Estimation technique, we no longer need to use a single value for the forgetting factor. Instead, we can select a vector of possible values. We choose $\mu = [0.5, 0.6, 0.7, 0.8, 0.9, 1]$: the number of values in the $mu$ vector is a trade-off between precision of the estimations and computing overhead. We deliberately avoided values below 0.5 due to the high level of noise present in the system.

*Smoothing Parameter.* We know from theory that increasing $\phi$ reduces overshoot and oscillation. However, larger values lead to a slowdown in the algorithm's performance. To choose the filter parameter $\phi$, we adopted an experimental approach. Through our analysis, omitted for the sake of concision, we observed satisfactory results with $\phi \ge 1.5$. While larger values yield similar outcomes, we selected $\phi = 2$, as it experimentally appeared to be the optimal choice.

*Initial Conditions.* There are two distinct initial conditions: $\hat{b}(0)$ and $V(0)$. For $\hat{b}(0)$, a value of 0.5 was chosen. Indeed, for a smaller initial condition the estimation $\hat{b}$ approaches zero due to the starting delay of the system. Consequently, the control action has an overshoot due to the relationship between $u$ and $b$. Surprisingly, even though 0.5 may seem like a significant overestimation, it yielded satisfactory results, because it leads to a soft initial control action. The significance of $V(0)$ lies in its impact on the algorithm's initial speed. Several values were tested, and ultimately, $10^4$ was determined to be the most appropriate choice. Please note that the choice of these values is not crucial for the convergence of the algorithm but impacts the closed-loop performance.

## 5.4 Model-Free Controller

The third type of controller we want to compare with is one that does not rely on a model. Many choices exist in this case such as Active Disturbance Rejection Control [24], Model-Free Control [15] or the ones using learning techniques (see [40] among others), each of them with several variants and improvements. Model-Free Control (MFC) is an approach that has the advantage of not necessitating the possibly tedious phase of modeling. It has been successfully applied to the problem of resource harvesting [23], the main ideas and formulations are summarized in the remainder of this section. MFC formulation relies on an *ultra-local model*, but does not require building a reliable global model. In that sense, it justifies its name of Model-Free Control. The considered ultra-local model is of the form:

$$ \dot{y}(k) \triangleq \frac{y(k) - y(k-1)}{\Delta t} = F(k) + \alpha \cdot u(k) \quad (13) $$

with $\dot{y}(k)$ the discrete derivative of $y(k)$, $\Delta t$ the sampling period of $CiGri$, $\alpha \in \mathbb{R}$ a constant, and $F(k)$ a term to be estimated (see Eq. (15)), reflecting the unknown structure of the system and its disturbances.
The Model-Free approach allows plugging PID controllers, defining the notion of *intelligent* PIDs. For an intelligent Proportional (iP), the control action computation is:

$$u(k) = -\frac{1}{\alpha} \cdot (\hat{F}(k) - \dot{y}^*(k) + K_p \cdot e(k)) \quad (14)$$

where $\hat{F}(k)$ is the estimated value of $F(k)$, and $K_p \in \mathbb{R}$ the proportional gain. Fliess and Join present different techniques to compute $\hat{F}$ in [15]. Yet, as a first approach, we use past values of $y$ and $u$ to estimate it from (13):

$$\hat{F}(k) = \dot{y}(k) - \alpha \cdot u(k-1) \quad (15)$$

*Tuning*. In [15], the authors recommend taking $\alpha$ such that $\dot{y}$ and $\alpha \cdot u$ have the same order of magnitude. Thus, as in [23], we rely on experiments to set the parameter $\alpha$.

The value of the gain $K_p$ accounts for the closed-loop behavior of the system. Small values of $K_p$ yield conservative and slow controllers, whereas greater values yield more aggressive controllers prone to overshooting and oscillations. Therefore, the $K_p$ value should be chosen by trial and error methods or by realizing a Pareto-type analysis.

# 6 Evaluation and Comparison

## 6.1 Experimental Setup

The experiments were carried out on the nodes from the grisou Cluster of Grid'5000 [5], which is a shared French testbed for experimental research in distributed and parallel computing. Each node of this cluster has two Intel Xeon E5-2630 v3 CPU with eight cores per CPU and 128 GiB of memory. Each server of our system is being deployed onto a single Grid'5000 node from a Kameleon system image [46]. We used four nodes for the deployment setup: one for the CiGri server, one for the RJMS (OAR version 3 in our case [8, 39]), one for the file server (implemented with NFSv3), and one node emulating a cluster of 100 OAR resources. The job model used in this work (job = sleep + I/O) allows us to efficiently and realistically emulate several computing resources on a single physical node, and thus reduce the number of machines required to perform experiments at full scale. The impact on the realism of such a folded deployment is studied in [20].

## 6.2 Controller Configurations

The PI is tuned based on the model of Eq. (4). Based on identification from experimental data (Fig. 3), we have the following values: $\alpha_1 = -0.5071484$, $\beta_1 = 0.0086335$, $\beta_2 = 0.0451394$, and $\gamma = 0.001633$. The gains of the PI controller are computed with Eqs. (7) and (8) and the design parameters set to $k_s = 12$ (fast response) and $M_p = 0$ (no overshoot): $K_p = 1.3505409$ and $K_i = 0.9795209$.

The adaptive PI relies on the PI tuning, with additional parameters set as follows: $\phi = 2$, $\hat{b}(0) = 0.5$, $V(0) = 10^4$, and $\mu = [0.5, 0.6, 0.7, 0.8, 0.9, 1]$.

For the MFC, we reuse the configuration of [23]: $\alpha = 0.008$ and $K_p = 0.078$ with an exponential smoothing filter on the sensor output of weight 0.22.

## 6.3 Experimental Protocol

For the three controllers considered in this study, we will evaluate their performance by varying the characteristics of the CiGri jobs from the identified system. Controllers were designed for jobs lasting 30 s and writing 100 MB to the file server. This configuration represents our nominal system. For testing, we consider variations on two dimensions of the workloads: the amount of I/O written by the jobs, and the execution time of the jobs. We consider that in the studied system, these two dimensions are the most sensitive to uncertainties/variations, and are thus the most interesting to study. One could also consider variations along the hardware, or cluster configuration dimensions.
<figure><img src="image_7.png" /><figcaption>Fig. 5. Nominal performance of the different controllers through time. The solid line represents the aggregated behavior from five different experiments (light dots). The bottom plots compare the controllers on the performance metrics defined in Section 3.1.1, evaluated on the first 2000 s.</figcaption></figure>

The impact of these configuration choices boils down to variations in the execution time of the CPU-bound and I/O-bound phases of the jobs: slower hardware will slow down the CPU-bound phase, a file server with more resources will reduce the I/O-bound phase execution time. Hence, we directly consider these variations from the point-of-view of the workload where sleep time represents the CPU-bound phase, and I/O load represents the I/O-bound phase. The repetitions of the experiments exposes the controller to several stochastic behaviors, including scheduler decision or I/O-phase synchronization. For the I/O, we consider the following file sizes: 50, 100, 200 and 400 MB and for the execution times, we consider: 10, 30, 60 and 120 s.

For each triplet (I/O load, execution time, controller), we repeat five times the following scenario: (i) we start with an empty cluster, (ii) we define the reference value for the controller at 3, (iii) we submit a campaign of jobs to CiGri with the desired characteristics (I/O load, execution time), (iv) we start the controller, (v) at t = 2000 s, we introduce a disturbance in I/O which represents a step of 2 for the load of the file server, and (vi) at t = 4100 s, we stop the disturbance.

## 6.4 Performance-Related Comparison

6.4.1 Nominal Performance. We first compare the performance of controllers on the nominal system, i.e., the system’s configuration that was used for design and tuning. The three controllers are executed, and the measure of the file server load through time is reported in Fig. 5. The top plots represent the first 2000 s of the scenario (before the introduction of the disturbance) for each controller. The solid lines aggregate the behavior of five repetitions of each experiment (light dots), and the dashed line represents the reference value that the controllers follow.
All controllers are successful in tracking the reference, however with varying performances. A qualitative analysis of the aggregated behaviors (top plots) shows that the PI outperforms the adaptive PI (aPI) and MFC. The MFC has a large overshoot, while the PI seems more stable and precise. The aPI is slower, with oscillation around the reference value.

In order to compare quantitatively the controllers, we use the performance metrics defined in Section 3.1.1. Note first that the measures of the file server load is *very noisy* (gray clouds of points). It makes it difficult to properly compute the quantitative analysis metrics, especially the rapidity and precision, as they are dependent on the convergence in steady state. Thus, instead of considering a convergence when reaching 95 % of the reference value, we will consider that the system has converged when its variations are within the noise of the signal. We identified that the amplitude of the noise is around ±0.75, and this value will thus be our threshold value to define convergence.

Overshoot, response time, and precision are presented in the bottom plots of Fig. 5. Note that those metrics are computed for each run (gray curves), results can slightly differ from the previous aggregated analysis. All controllers present overshoots. The MFC can present large overshoots and has a large variability, from 15 % up to 60 % of the reference value. The PI and aPI have in average a similar overshoot, the latter showing little variability across runs. The precision of the three controllers is fairly similar, with the adaptive PI being less precise, as could be observed in the plots through time. Note that despite using an integral action, that theoretically ensures a perfect reference tracking, the uncertainties and noise in the real system lead to a precision different from 0. The PI shows faster response time than the aPI and MFC. The rapidity of the worst runs is, however, similar for all controllers.

In conclusion, all controllers successfully track the reference value, with overall similar performances. The PI is slightly better in terms of precision and rapidity. The MFC can lead to high overshoot. The aPI is less precise than other controllers.

### 6.4.2 *Portability.*
To evaluate the portability of controllers, we look at their performance when used on a different system than the nominal one. Two variations are considered: one on the *I/O* load, where the size of the file that the jobs write varies, and one where the duration of jobs' execution time varies.

Figure 6 depicts the dynamic behavior of the different controllers for the different scenarios: variations in *I/O* and in execution times (Figs. 6a and 6b respectively). Note that the nominal performance (i.e., Fig. 5) are repeated and shown in their entirety until 6000 s (second line of the plots) to allow for comparison. Similarly to the previous analysis, aggregated metrics of performance on the first 2000 s are computed in Fig. 7.

*Variations of I/O Load.* All controllers manage to track the reference value, however some configurations lead to large oscillations in the file server load. The disturbance does not seem to have a detrimental impact, it even reduces the oscillations in some cases.

In Fig. 6a, the trend of the MFC controller is similar than in nominal conditions: it can lead to large overshoots. The PI has fewer oscillations when the system is highly different from the nominal one. The adaptive PI seems to have worse performance on the nominal system (100 MB) in case of disturbance than the other two controllers. The aPI is, however, less impacted by the variations in *I/O* load.

The quantitative analysis of Fig. 7 clarifies those analyses over the first 2000 s: (i) overshoot soars for the PI and MFC with 400 MB (when we are far away from the nominal system), while it stays at acceptable values with the aPI; closer to the nominal conditions, the aPI has slightly higher overshoot, (ii) precision far from the nominal system (50 and 400 MB) is significantly better with the aPI, while closer to the nominal system, the PI has the advantage, and (iii) rapidity is
Comparison with variations in the I/O impact of jobs

<figure><img src="image_8.png" /><figcaption>(a) Performance with varying I/O loads of the jobs.</figcaption></figure>

Comparison with variations in the execution time of jobs

<figure><img src="image_9.png" /><figcaption>(b) Performance with varying jobs execution time.</figcaption></figure>

Fig. 6. Global comparison of the controllers’ performance for variations in I/O loads (Fig. 6a) and jobs execution times (Fig. 6b). The solid lines represent the aggregated behavior from five different experiments (light dots).

<figure><img src="image_10.png" /><figcaption>(a) Performance with varying I/O loads of the jobs.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(b) Performance with varying jobs execution time.</figcaption></figure>

Fig. 7. Comparison of controllers’ portability. The plots represent the performance metrics computed on various systems between 0 s to 2000 s. The solid lines link the means over varying configurations for each controller.
fairly similar for all controllers on non-nominal systems (note that for the experiments with large
oscillatory behavior, the rapidity criteria should be mitigated as convergence is not reached).

In conclusion, the MFC and the PI perform well close to the nominal system, but have difficulties not overshooting and oscillating when the system is too far from the nominal system – i.e., 400 MB in our scenario. On the other hand, the adaptive PI has a fairly similar behavior whatever the system configuration.

*Variations of Jobs Execution Time.* When variation in the jobs’ execution time is introduced, controllers successfully track the reference, with a better behavior than in the case of I/O variations, as shown on Figs. 6b and 7b.

For the PI and MFC controllers, reducing the job duration to 10 s even improves the performances by reducing the overshoot, precision and rapidity; with an advantage for the MFC, while the aPI’s behavior is worse than in nominal conditions. When increasing the duration, the PI controls the system with a better precision, while the aPI is the most imprecise. Relative performances of the three controllers on overshoot and rapidity vary, while the adaptive PI always seems slightly the worst.

The relative poor performance of the adaptive PI for job duration variations can be theoretically explained. The increase of jobs duration leads to the addition of a delay in the system. The adaptive PI was built to adapt the model parameter *b* in Eq. (1), not to identify a delay. Thus, this adaptive PI is not designed to deal with such type of system’s variation.

Overall, concerning the variations in execution times, the controllers appear less sensible than with I/O variations. The adaptive PI does not perform as good. The MFC shows slight performance advantage over the PI.

## 6.5 Methodology and Implementation-Related Comparison

This section complements the comparison of controllers on their performance and probability, by focusing on aspects linked to their design, tuning, theoretic soundness and implementation. Such considerations are not classical for the point of view of control-theory oriented evaluation, that focuses mainly on performance and robustness to unpredicted external events. On the contrary, from the practical engineering point of view, simplicity in design and usage can be a valuable asset. Some performance can even be traded to allow lower complexity, so that system administrators can master solutions without the need for help from an external control expert, or then only in a limited and temporary fashion.

In the following, we evaluate our three controllers on the methodology- and implementation-related criteria of Section 3: set-up complexity, guarantees, and control-theoretical competence required. A discussion of the trade-off between those criteria are provided in the next section.

6.5.1 *Set-up Complexity.* In a first attempt to evaluate the setup complexity, we count the number of parameters to be tuned for each controller. The MFC requires two parameters: $K_p$ and $\alpha$. $\alpha$ is a non-physical constant parameter. It is chosen by the practitioner such that $\dot{y}$ and $\alpha \cdot u$ are of the same magnitude. It should be therefore clear that its numerical value, which is obtained by trials and errors, is not a priori precisely defined [15]. The PI also requires two parameters: $K_p$ and $K_i$. They are computed as in Eqs. (7) and (8) based on the model of the system (two parameters in the case of the linear first order model: *a* and *b*), that can be derived from the identification experiments. Moreover, the tuning allows translating the problem of finding $K_p$ and $K_i$ into the choice of the design quantities $k_s$ and $M_p$. Thus, two parameters are needed for setting up the PI. Finally, the adaptive PI is based on the PI: thus it requires the same two parameters. It additionally requires parameters specific to the adaptive part: $\mu$, $\phi$, $\hat{b}(0)$, $V(0)$, leading to a total of six parameters.
Besides the raw count of parameters, we can point out that some parameters are more sensitive than others in terms of precision needed. We therefore see three cases: (i) the very sensitive parameters for which an ill-tuned value can even make the system unstable (e.g., $K_p$), (ii) the parameters for which an ill-tuned value makes the system slower (e.g., $\phi$), and (iii) the parameters which are continuously updated and therefore an approximate initial value has little impact (e.g., $\hat{b}(0)$ or $V(0)$).

### 6.5.2 Guarantees.
The controllers studied in this paper have different guarantees from the point of view of their behavior.

As seen in Section 5.2, the design of a PI controller requires following a well-defined methodology. This has the benefits of allowing users to choose the closed-loop behavior (choice of $k_s$ and $M_p$ as illustrated in Fig. 4), and to have guarantees that the controller will behave as intended.

Designing and using a Model-Free controller requires less knowledge of control theory than a classic PI. It removes the need for complex identification and modeling of the system. It also has an online estimation of the underlying dynamics of the system. Yet, this simplicity comes at a cost: the MFC does not have a fixed methodology to choose all the parameters. It hence does not provide the same guarantees as the classical PI w.r.t. convergence or robustness. The MFC does not have a methodology to choose the parameter $K_p$, and as a result, there is no guarantee that the system won't overshoot or oscillate.

The adaptive PI provides additional guarantees to the PI, by broadening the conditions in which we have our controller working. That is to say that the guarantees that one has on the nominal system, also extends to systems close to the nominal one, as seen in Section 6.4.2. However, knowing in advance if a system falls into the region of stability of a controller is not straightforward for a computing system, especially without performing a preliminary modeling identification phase. Additionally, the adaptive PI with a single value for $\mu$ different from 1 has the drawback of requiring specific input signals – known as persistent excitation – to avoid its divergence, namely the bursting phenomena.

### 6.5.3 Competence in Control Required.
On most computer science curriculums, there are no classes on control theory. Most cloud/grid platforms usually do not hire control scientists or engineers in their system administration team. Therefore, the choice of a control method should be weighted by its difficulty to be mastered by the administrators.

In this perspective, MFC has the advantage of very low requirements, as it does not require specific engineering education. On the other hand, it is a novel, emerging method, with only little literature and experience from which to draw practical guidelines.

In the case of the classical PI, there is a need to perform open-loop (identification) experiments. It has a practical cost, but there is only relative low mathematical difficulty in the equations at play. Therefore, it is accessible and taught to undergraduate students in control engineering curriculums. As it is very widely used in industry (around 90 % of the industrial processes use a form of PI/PID controller) there is a huge amount of available literature ([3, 10, 35] among many others).

The adaptive PI builds up on PI: it hence requires a greater set of skills to use [33]. Moreover, over the years, many upgrading elements and variants have been proposed in order to correct and improve certain aspects in its design ([4, 29, 38] among others). In terms of skills, this usually is taught in Master curriculums dedicated to control engineering or even at the PhD level for the more advanced variants.
<table><caption>Table 1. Comparison of controllers on all criteria. The “+” symbol indicates a positive evaluation of the criteria – e.g., high portability, low complexity. The “-” symbol indicates the criteria is poorly fulfilled – e.g., few guarantees, high competence required.</caption><thead><tr><th>Criteria</th><th>PI</th><th>Adaptive PI</th><th>MFC</th></tr></thead><tbody><tr><td>Nominal Performance (close to identified sys.)</td><td>+</td><td>+</td><td>+</td></tr><tr><td>Portability</td><td>-</td><td>+</td><td>-</td></tr><tr><td>Design Complexity</td><td>+</td><td>-</td><td>+</td></tr><tr><td>Guarantees</td><td>+</td><td>+</td><td>-</td></tr><tr><td>Competence required</td><td>+</td><td>-</td><td>+</td></tr></tbody></table>

## 6.6 Discussion

Table 1 summarizes the comparison of the different controllers on the criteria based on our observations in the framework of our project. More general comparative results are still an open question due to the great variety of cases according to the control problems. As expected, there is no clear “best controller” from a computer engineer point of view, but the table exhibits several tradeoffs. The recommendations and guidelines that can be given to system administrators depend on different contexts: whether a control engineer is available or not, whether guarantees on the behavior are crucial or not, etc. For example, if the system on which the controller will be deployed only varies “close” to the nominal system, then both the PI and the MFC are good solutions from the reusability point of view. However, if the system can vary a lot, then an adaptive PI appears to be a more judicious choice from the point of view of portability. Note that it will be adaptive only with respect to the variables for which it has been designed.

In the case where one wants to give away some design complexity and required competence but still have acceptable nominal performance, the MFC appears as an excellent candidate as it has a low setup complexity.

In the case where a well documented and explainable solution is required, then the PI controller has the advantage of being a very well-known control algorithm.

## 7 Conclusion & Perspectives

The work presented in this paper addresses the need for reusability of autonomic controllers, with a focus on the use case of resource harvesting in for High Performance Computing. The controllers designed following the methodology of control theory are tuned for a target nominal context. The problem is that such controllers can be too tightly dependent on the exact values of the design parameters tuned for the nominal context. As a result, the performance of the controlled systems can drastically decrease when controllers are used in non-nominal contexts. The variations inducing such performance loss may be temporal (e.g., campaigns with different types of best-effort jobs) or spatial (e.g., clusters with different configurations). A summary of our contributions in this work is as follows.

(1) We define criteria relevant to both the HPC and control communities, aiming at evaluating the reusability of autonomic controllers.

(2) We conduct a reproducible experimental evaluation on the concrete use case that is resource harvesting for HPC.

(3) We identify the potential of adaptive control techniques for reusability of autonomic controllers. This potential spans across the classic temporal dimension, but also extends to other spatial dimensions including the characteristics of resources.
As for future works, regarding the system under control, one may want to explore other dimensions of variations. One could enrich the nature of the jobs running on the system, by considering more than two classes of jobs with different interaction models. One dimension of variations could be the hardware or the configuration supporting the system: processor architecture, operating system, file system (block vs. object). Some parts of this variation dimension are encompassed in this work for the performance objective, as described in Section 6.3. Other objectives might interact differently.

Some additional perspectives lie in the design of controllers. For example adaptive control could be considered w.r.t. other variables, as mentioned in Section 6.4.2. One could also design autonomic managers with other methods than control theory, such as reinforcement learning.

While Section 3 defines various criteria to evaluate the reusability, most of them allow for a qualitative comparison of the controllers. The definition of quantitative metrics remains open. For example, control theory defines quantitative metrics such as response time for a given experiment. How would one define the distance to a nominal system? Once such a distance is defined, how may one quantify the performance impact w.r.t. to the distance to a nominal system? What are the interesting trade-offs? To what extent are those metrics sensitive to noise in the measures? Exploring reusability of autonomic managers with control theory methodology opens novel research questions.

## Acknowledgments

Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).

## References

[1] B. D. O. Anderson. 1985. Adaptive Systems, Lack of Persistency of Excitation and Bursting Phenomena. Automatica, 21, 3, (May 1985), 247-258. doi:10.1016/0005-1098(85)90058-5.

[2] D. P. Anderson. 2004. BOINC: A System for Public-Resource Computing and Storage. In GRID. IEEE, (Nov. 2004), 4-10. doi:10.1109/GRID.2004.14.

[3] K. J. Åström and T. Hägglund. 2006. Advanced PID Control. International Society of Automation.

[4] K. J. Åström and B. Wittenmark. 2008. Adaptive Control. (2nd ed.). Dover Publications.

[5] D. Balouek et al. 2013. Adding Virtualization Capabilities to the Grid'5000 Testbed. In CLOSER (Communications in Computer and Information Science). Vol. 367. Springer, 3-20. doi:10.1007/978-3-319-04519-1_1.

[6] A. Bhatele, K. M. Mohror, S. H. Langer, and K. E. Isaacs. 2013. There Goes the Neighborhood: Performance Degradation due to Nearby Jobs. In SC. ACM, 41:1-41:12. doi:10.1145/2503210.2503247.

[7] [SW] B. Bzeznik and O. Richard, CiGri. URL: http://cigri.imag.fr/, vcs: https://github.com/oar-team/cigri.git, swhid: `swh:1:dir:f8b6df45300534e9807addc7ec125cd72b92c139;origin=https://github.com/oar-team/cigri`.

[8] N. Capit, G. Da Costa, Y. Georgiou, G. Huard, C. Martin, G. Mounié, P. Neyron, and O. Richard. 2005. A batch scheduler with high level components. In CCGRID. IEEE, 776-783. doi:10.1109/CCGRID.2005.1558641.

[9] S. Cerf, R. Bleuse, V. Reis, S. Perarnau, and É. Rutten. 2021. Sustaining Performance While Reducing Energy Consumption: A Control Theory Approach. In Euro-Par (Lecture Notes in Computer Science). Vol. 12820. Springer, (Sept. 2021), 334-349. doi:10.1007/978-3-030-85665-6_21.

[10] R. C. Dorf and R. H. Bishop. 2021. Modern Control Systems. (14th ed.). Pearson.

[11] M. Dorier, G. Antoniu, R. B. Ross, D. Kimpe, and S. Ibrahim. 2014. CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination. In IPDPS. IEEE, (May 2014), 155-164. doi:10.1109/IPDPS.2014.27.

[12] D. Ferrari and S. Zhou. 1987. An Empirical Investigation of Load Indices for Load Balancing Applications. In Performance. North-Holland, 515-528. Retrieved Oct. 9, 2023 from https://www2.eecs.berkeley.edu/Pubs/TechRpts/1987/CSD-87-353.pdf.

[13] A. Filieri et al. 2017. Control Strategies for Self-Adaptive Software Systems. ACM Trans. Auton. Adapt. Syst., 11, 4, (Feb. 2017), 24:1-24:31. doi:10.1145/3024188.

[14] A. Filieri et al. 2015. Software Engineering Meets Control Theory. In SEAMS@ICSE. IEEE, (May 2015), 71-82. doi:10.1109/SEAMS.2015.12.
[15] M. Fliess and C. Join. 2013. Model-free control. *Int. J. Control*, 86, 12, 2228-2252. doi:10.1080/00207179.2013.810345.

[16] T. R. Fortescue, L. S. Kershenbaum, and B. E. Ydstie. 1981. Implementation of Self-tuning Regulators with Variable Forgetting Factors. *Automatica*, 17, 6, (Nov. 1981), 831-835. doi:10.1016/0005-1098(81)90070-4.

[17] J. Frey, T. Tannenbaum, M. Livny, I. T. Foster, and S. Tuecke. 2001. Condor-G: A Computation Management Agent for Multi-Institutional Grids. In *HPDC. IEEE*, (Aug. 2001), 55. doi:10.1109/HPDC.2001.945176.

[18] Y. Georgiou. 2010. *Contributions for Resource and Job Management in High Performance Computing*. Ph.D. Dissertation. LIG, Univ. Grenoble Alpes, France, (Nov. 2010). Retrieved Oct. 11, 2023 from https://tel.archives-ouvertes.fr/tel-01499598.

[19] Y. Georgiou, O. Richard, and N. Capit. 2007. Evaluations of the Lightweight Grid CIGRI upon the Grid5000 Platform. In *eScience*. IEEE, 279-286. doi:10.1109/E-SCIENCE.2007.32.

[20] Q. Guilloteau, O. Richard, R. Bleuse, and É. Rutten. Folding a Cluster containing a Distributed File-System. (Mar. 2023). Retrieved Oct. 13, 2023 from https://hal.science/hal-04038000v1.

[21] Q. Guilloteau, O. Richard, B. Robu, and É. Rutten. 2021. Controlling the Injection of Best-Effort Tasks to Harvest Idle Computing Grid Resources. In *ICSTCC. IEEE*, (Oct. 2021), 334-339. doi:10.1109/ICSTCC52150.2021.9607292.

[22] Q. Guilloteau, O. Richard, and É. Rutten. 2022. Étude des applications Bag-of-Tasks du méso-centre Gricad. In *COMPAS*. (July 2022). Retrieved Oct. 11, 2023 from https://2022.compas-conference.fr/programme/papiers/ComPAS2022_paper_19-2.pdf.

[23] Q. Guilloteau, B. Robu, C. Join, M. Fliess, É. Rutten, and O. Richard. 2022. Model-Free Control for Resource Harvesting in Computing Grids. In *CCTA. IEEE*, (Aug. 2022), 384-390. doi:10.1109/CCTA49430.2022.9966035.

[24] J. Han. 2009. From PID to Active Disturbance Rejection Control. *IEEE Trans. Ind. Electron.*, 56, 3, (Mar. 2009), 900-906. doi:10.1109/TIE.2008.2011621.

[25] J. L. Hellerstein, Y. Diao, S. S. Parekh, and D. M. Tilbury. 2004. *Feedback Control of Computing Systems*. Wiley. doi:10.1002/047166880X.

[26] M. C. Huebscher and J. A. McCann. 2008. A survey of Autonomic Computing—Degrees, Models, and Applications. *ACM Comput. Surv.*, 40, 3, (Aug. 2008), 7:1-7:28. doi:10.1145/1380584.1380585.

[27] C. Imes, D. H. K. Kim, M. Maggio, and H. Hoffmann. 2015. POET: A Portable Approach to Minimizing Energy Under Soft Real-time Constraints. In *RTAS. IEEE*, (Apr. 2015), 75-86. doi:10.1109/RTAS.2015.7108419.

[28] 2024. *Intel® 64 and IA-32 Architectures Optimization Reference Manual. Volume 1*. Version 050. (Apr. 2024). Retrieved Nov. 20, 2024 from https://www.intel.com/content/www/us/en/content-details/821612/.

[29] P. A. Ioannou and J. Sun. 1996. *Robust adaptive control*. Prentice-Hall.

[30] P. K. Janert. 2013. *Feedback Control for Computer Systems*. O'Reilly, (Oct. 2013).

[31] J. O. Kephart and D. M. Chess. 2003. The Vision of Autonomic Computing. *IEEE Computer*, 36, 1, (Jan. 2003), 41-50. doi:10.1109/MC.2003.1160055.

[32] Z. Kowalczuk. 1992. Competitive Identification for Self-tuning Control: Robust Estimation Design and Simulation Experiments. *Automatica*, 28, 1, (Jan. 1992), 193-201. doi:10.1016/0005-1098(92)90021-7.

[33] I. D. Landau, R. Lozano, M. M'Saad, and A. Karimi. 2011. *Adaptive Control. Algorithms, Analysis and Applications*. (2nd ed.). *Communications and Control Engineering*. Springer. doi:10.1007/978-0-85729-664-1.

[34] A. Leva. 2019. PID control education for computer engineering students: A step to bridge a cultural gap. *IFAC Journal of Systems and Control*, 8, (June 2019), 100051. doi:10.1016/j.ifacsc.2019.100051.

[35] Y. Li, K. H. Ang, and G. C. Y. Chong. 2006. Patents, Software, and Hardware for PID Control. An Overview and Analysis of the Current Art. *IEEE Control Systems Magazine*, 26, 1, (Feb. 2006), 42-54. doi:10.1109/MCS.2006.1580153.

[36] M. Litoiu, M. Shaw, G. Tamura, N. M. Villegas, H. A. Müller, H. Giese, R. Rouvoy, and É. Rutten. 2013. What Can Control Theory Teach Us About Assurances in Self-Adaptive Software Systems? In *Software Engineering for Self-Adaptive Systems* (Lecture Notes in Computer Science). Vol. 9640. Springer, (Dec. 2013), 90-134. doi:10.1007/978-3-319-74183-3 4.

[37] M. Mercier, D. Glesser, Y. Georgiou, and O. Richard. 2017. Big Data and HPC collocation: Using HPC idle resources for Big Data Analytics. In *IEEE BigData*. IEEE, (Dec. 2017), 347-352. doi:10.1109/BigData.2017.8257944.

[38] R. H. Middleton, G. C. Goodwin, D. J. Hill, and D. Q. Mayne. 1988. Design Issues in Adaptive Control. *IEEE Trans. Autom. Control.*, 33, 1, (Jan. 1988), 50-58. doi:10.1109/9.360.

[39] [SW] OAR team, OAR version 3. url: https://oar.imag.fr/, vcs: https://github.com/oar-team/oar3.git, swhid: (swh:1:dir:ab502da78090e4d77ee363ce7ccc2f6ea65c2d92;origin=https://github.com/oar-team/oar3).

[40] V. Pong, S. Gu, M. Dalal, and S. Levine. 2018. Temporal Difference Models: Model-Free Deep RL for Model-Based Control. arXiv: 1802.09081 [cs.LG].

[41] B. Porter, R. Rodrigues Filho, and P. Dean. 2020. A Survey of Methodology in Self-Adaptive Systems Research. In *ACOS. IEEE*, (Aug. 2020), 168-177. doi:10.1109/ACOS49614.2020.00039.

[42] R. Prieto Díaz. 1993. Status Report: Software Reusability. *IEEE Softw.*, 10, 3, (May 1993), 61-66. doi:10.1109/52.210605.
[43] B. Przybylski, M. Pawlik, P. Żuk, B. Lagosz, M. Malawski, and K. Rzadca. 2022. Using Unused: Non-Invasive Dynamic FaaS Infrastructure with HPC-Whisk. In *SC. IEEE*, (Nov. 2022), 40:1–40:15. doi:10.1109/SC41404.2022.00045.

[44] A. Reuther et al. 2018. Scalable system scheduling for HPC and big data. *Journal of Parallel and Distributed Computing*, 111, (Jan. 2018), 76–92. doi:10.1016/j.jpdc.2017.06.009.

[45] N. S. Rosa and D. J. M. Cavalcanti. 2024. Exploiting Controllers to Adapt Message-Oriented Middleware. In *ACSOS. IEEE*, (Sept. 2024), 91–100. doi:10.1109/ACSOS61780.2024.00027.

[46] C. Ruiz, S. Harrache, M. Mercier, and O. Richard. 2015. Reconstructable Software Appliances with Kameleon. *ACM SIGOPS Oper. Syst. Rev.*, 49, 1, (Jan. 2015), 80–89. doi:10.1145/2723872.2723883.

[47] É. Rutten, N. Marchand, and D. Simon. 2013. Feedback Control as MAPE-K Loop in Autonomic Computing. In *Software Engineering for Self-Adaptive Systems* (Lecture Notes in Computer Science). Vol. 9640. Springer, 349–373. doi:10.1007/978-3-319-74183-3_12.

[48] R. Sanz and K. Årzén. 2003. Trends in Software and Control. *IEEE Control Systems Magazine*, 23, 3, (June 2003), 12–15. doi:10.1109/MCS.2003.1200238.

[49] S. Shevtsov, M. Berekmeri, D. Weyns, and M. Maggio. 2018. Control-Theoretical Software Adaptation: A Systematic Literature Review. *IEEE Trans. Software Eng.*, 44, 8, 784–810. doi:10.1109/TSE.2017.2704579.

[50] S. Silva, P. Pelliccione, and A. Bertolino. 2024. Self-Adaptive Testing in the Field. *ACM Trans. Auton. Adapt. Syst.*, 19, 1, (Feb. 2024), 4:1–4:37. doi:10.1145/3627163.

[51] D. Skinner and W. Kramer. 2005. Understanding the Causes of Performance Variability in HPC Workloads. In *IISWC. IEEE*, (Oct. 2005), 137–149. doi:10.1109/IISWC.2005.1526010.

[52] E. Stahl, A. G. Yabo, O. Richard, B. Bzeznik, B. Robu, and É. Rutten. 2018. Towards a control-theory approach for minimizing unused grid resources. In *AI-Science@HPDC*. ACM, (June 2018), 4:1–4:8. doi:10.1145/3217197.3217201.

[53] S. Wang, C. Li, H. Hoffmann, S. Lu, W. Sentosa, and A. I. Kistijantoro. 2018. Understanding and Auto-Adjusting Performance-Sensitive Configurations. In *ASPLOS*. ACM, 154–168. doi:10.1145/3173162.3173206.

[54] A. G. Yabo, B. Robu, O. Richard, B. Bzeznik, and É. Rutten. 2019. A control-theory approach for cluster autonomic management: maximizing usage while avoiding overload. In *CCTA. IEEE*, (Aug. 2019), 189–195. doi:10.1109/CCTA.2019.8920473.

[55] H. Yang, F. Chen, and S. O. Aliyu. 2017. Modern software cybernetics: New trends. *Journal of Systems and Software*, 124, (Feb. 2017), 169–186. doi:10.1016/J.JSS.2016.08.095.

Received 21 November 2023; revised 24 November 2024; revised 4 April 2025; accepted 16 June 2025