STREAMLINE: Dynamic and Resource-Efficient
Auto-Tuning of Stream Processing Data Pipeline
Ensembles

Stefan Pedratscher, Zahra Najafabadi Samani, Juan Aznar Poveda, Thomas

Fahringer, Marlon Etheredge, Abolfazl Younesi, Juan Jose Durillo

Barrionuevo, Peter Thoman

► To cite this version:

Stefan Pedratscher, Zahra Najafabadi Samani, Juan Aznar Poveda, Thomas Fahringer, Marlon Etheredge, et al.: STREAMLINE: Dynamic and Resource-Efficient Auto-Tuning of Stream Processing Data Pipeline Ensembles. Internet of Things, 2025, 34, 10.1016/j.iot.2025.101731. hal-05271540

HAL Id: hal-05271540

https://hal.science/hal-05271540v1

Submitted on 22 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Research article

# STREAMLINE: Dynamic and Resource-Efficient Auto-Tuning of Stream Processing Data Pipeline Ensembles

<figure><img src="image_2.png" /></figure>

Stefan Pedratscher<sup>a ID,*</sup>, Zahra Najafabadi Samani<sup>a</sup>, Juan Aznar Poveda<sup>a ID</sup>, Thomas Fahringer<sup>a ID</sup>, Marlon Etheredge<sup>a</sup>, Abolfazl Younesi<sup>a</sup>, Juan Jose Durillo Barrionuevo<sup>b</sup>, Peter Thoman<sup>a ID</sup>

## ARTICLE INFO

Dataset link: https://github.com/UIBK-DPS-DC/Streamline

**Keywords:**

Internet of Things
Stream processing
Data pipelines
Apache Flink
Auto-tuning
Optimization

## ABSTRACT

With the growing volume of data generated by IoT devices and user-driven services, stream processing has become essential for handling continuous, real-time data. However, fluctuating workloads and the dynamic nature of data streams make it difficult to maintain consistent performance over time, requiring adaptive resource allocation and frequent configuration tuning. Running multiple data stream processing pipelines on shared resources further exacerbates the problem by increasing contention, leading to higher end-to-end latency and reduced performance stability. Most existing approaches focus on tuning individual configuration parameters in isolation and overlook interactions between concurrently running data pipelines. To address these limitations, we present STREAMLINE, a dynamic multi-layer auto-tuning framework designed for stream processing environments. STREAMLINE uses transformers to predict future workloads and an evolutionary algorithm to automatically tune configuration parameters. It also includes a resource-efficient scheduler that efficiently assigns operators to resources across a compute cluster. Our dynamic update mechanism minimizes downtime and preserves state during configuration parameter and scheduling changes. We evaluate STREAMLINE on the Grid'5000 testbed using real-time IoT and streaming benchmarks. Results show that STREAMLINE outperforms state-of-the-art methods, improving throughput, end-to-end latency, and CPU utilization by up to 4x, 10x, and 9x, respectively, while reducing costs by up to 10x.

## 1. Introduction

The rapid advancement of digital technologies has led to an exponential increase in data generation, fueled by the proliferation of IoT devices and data-intensive applications. Real-time analysis of this data is essential for deriving actionable insights and enabling automated decisions in domains such as social media trend analysis, stock trading, fraud detection, alarm systems, V2X systems, healthcare IoT, and traffic monitoring [1,2]. Stream processing enables low end-to-end latency and high throughput for such tasks, typically using a cluster of computational resources [3]. However, the unbounded and fluctuating nature of data streams, especially in IoT scenarios such as vehicle-to-everything (V2X) communication, healthcare IoT, industrial automation, smart cities, and traffic monitoring, presents serious difficulties for long-running data pipelines. Heavy workloads increase resource demands and degrade

<footnote><sup>a</sup> Distributed and Parallel Systems group, University of Innsbruck, Technikerstrasse 21a, Innsbruck, 6020, Austria</footnote>

<footnote><sup>b</sup> Leibniz Supercomputing Centre, Boltzmannstrasse 1, Munich, 85748, Germany</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail address: Stefan.Pedratscher@uibk.ac.at (S. Pedratscher).</footnote>
performance, while light workloads waste resources and inflate costs. Without dynamic adaptation, data pipelines suffer from poor throughput, increased end-to-end latency, and inefficient resource usage.

In stream processing, configuration parameters govern performance at both the data pipeline and operator levels. One of the most widely studied parameters is the degree of parallelism, which determines how many parallel operator instances process data [1,4–8]. Higher parallelism improves throughput, reduces end-to-end latency, and mitigates backpressure by distributing workload across more compute resources. However, prior work often overlooks parameter interdependencies. For example, increasing parallelism typically requires adjusting buffer sizes, i.e., the amount of data temporarily stored between operators, to ensure smooth data flow and avoid processing bottlenecks [3,9–11]. Similarly, scheduling operators across compute nodes affects load balancing and CPU utilization [3,5], but existing methods [12–16] rarely account for workload fluctuations or configuration parameter interactions, both critical for stable performance over time. These challenges intensify in data pipeline ensembles, where multiple data pipelines run concurrently and compete for shared resources. Here, naive optimization leads to resource contention and imbalance. Optimizing such ensembles requires balancing competing demands, anticipating interference, and dynamically tuning configuration parameters and schedules. A holistic strategy that considers resource constraints and data pipeline trade-offs is therefore essential for efficient and reliable ensemble performance [17].

To this end, we present STREAMLINE, a novel multi-layer dynamic auto-tuning framework for data pipeline ensembles. STREAMLINE optimizes throughput, end-to-end latency, CPU utilization, and cost, collectively referred to as optimization objectives, through four key components. First, it uses time series transformers [18,19] to predict future workload patterns, enabling proactive tuning of data pipelines. Second, STREAMLINE automatically tunes parallelism and buffer sizes based on predicted workloads using a multi-objective evolutionary algorithm [20]. It balances trade-offs such as increased concurrency versus buffer demands by considering user-defined and data pipeline-specific weights. To reduce overhead, a surrogate model accelerates convergence while preserving fidelity to real-world performance. Third, to minimize cost and maximize CPU utilization, STREAMLINE performs workload-aware scheduling to efficiently map operators to shared compute resources. Finally, it applies configuration parameters and schedule updates at runtime with minimal downtime while preserving state and ensuring correctness. In contrast to systems like Apache Flink, Spark, or Storm [21–23], which often require costly restarts via savepoints, STREAMLINE enables seamless, state-preserving dynamic updates.

The contributions of this paper are summarized as follows:

* We introduce STREAMLINE, a publicly available,<fnref n="1" /> dynamic, multi-layer auto-tuning framework for data pipeline ensembles that jointly optimizes throughput, end-to-end latency, CPU utilization, and cost.

* STREAMLINE incorporates workload prediction and proactive adaptation of configuration parameters and operator schedules.

* We present a mechanism to update configuration parameters and schedules dynamically, ensuring continuous processing.

* We evaluate STREAMLINE on the Grid'5000 testbed using IoT and streaming benchmarks across five data pipelines and compare it to three state-of-the-art baselines. Key findings include:

- Up to 10× improvement in end-to-end latency, 9× in CPU utilization, 10× in cost savings, and 4× in throughput.

- STREAMLINE leverages higher parallelism to reduce end-to-end latency under moderate to high workloads by speeding up data processing and minimizing waiting time. It also accounts for low-workload scenarios, where synchronization overhead reduces the benefits of increased parallelism, and accordingly avoids unnecessary scaling.

- Our approach avoids frequent changes in parallelism, which can harm throughput and increase end-to-end latency, by taking the parallelism of predecessor operators into account.

- STREAMLINE selects buffer sizes depending on the workload: low workloads benefit from small buffers, which fill quickly and reduce end-to-end latency, while higher workloads require larger buffers to sustain throughput. A lower buffer size usually requires higher parallelism and more resources to handle the workload efficiently.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the system model and architecture of STREAMLINE, whereas the proposed method is detailed in Section 4. Section 5 evaluates STREAMLINE's performance compared to three competitors using IoT and streaming benchmarks on heterogeneous resources. Finally, Section 6 concludes the paper.

## 2. Related work

This section reviews state-of-the-art optimization methods for data pipelines, including approaches that handle workload fluctuations and data pipeline ensembles.

<footnote n="1">¹ https://github.com/UIBK-DPS-DC/Streamline</footnote>
## 2.1. Data pipeline optimization

One of the most common methods to optimize data stream pipelines is adjusting operator parallelism [1,4-8]. For example, Kalavri et al. [6] tune the degree of parallelism by combining a performance model with historical data to estimate operator processing and output rates. While this work reduces backpressure, their linear model may be inaccurate for stateful operators [17]. ZeroTune [8] is a novel cost model that accurately sets initial parallelism degrees for streaming queries using zero-shot learning techniques. It leverages a graph neural network architecture to generalize across unseen workloads and hardware configurations. Although these approaches optimize throughput and end-to-end latency, solely adjusting parallelism can lead to resource imbalances if operators are not efficiently scheduled across computing nodes [3,5,7]. Huasong Shan et al. [3] address this issue by proposing a guided machine-learning approach to find the best configuration parameters, including parallelism and buffer size [11], of an Apache Flink data pipeline by leveraging generative adversarial networks. Their approach tunes the data pipeline statically, which limits its adaptability to dynamic workload fluctuations. Lohrmann et al. [7] propose Nephele-Streaming as an extension of their batch processing framework, Nephele [10]. By adjusting the buffer size and task chaining, Nephele-Streaming achieves a trade-off between higher throughput and lower end-to-end latency but disregards fluctuations in data stream workloads.

Scheduling operators on compute nodes is another common optimization strategy [12-16,24-29]. For instance, Sun et al. [14] propose a latency- and resource-aware scheduling framework for stream-oriented big data applications that optimizes for throughput and end-to-end latency. However, many works overlook the monetary cost of their scheduling approaches, which is a critical factor in stream processing. Hongjian et al. [26] propose a cost-efficient scheduling algorithm for deadline-constrained stream processing applications for the cloud. Even though they prioritize resources to reduce the total cost of the stream processing cluster, they lack the flexibility to optimize other objectives.

Dynamic adaptation to fluctuating workloads is essential for stable performance and efficient resource use [12,24,27]. Cheng et al. [12] introduce A-Scheduler, an adaptive scheduling approach for Spark Streaming that enhances performance and resource efficiency. It employs reinforcement learning to schedule parallel micro-batch jobs based on data dependencies and adjusts job parallelism and resource allocation based on workload properties. In particular, the throughput of each data pipeline is maximized using a maximum throughput threshold that needs to be measured under ideal conditions. A similar adaptive approach is presented in [27], where the authors introduce a dynamic scheduling technique for managing cloud resources, addressing the challenges posed by heterogeneous data sources, multiple outputs, and fluctuating datastream rates. Unlike traditional streaming approaches, this technique dynamically adjusts resource allocation over time to ensure that real-time data analysis requirements are consistently met. By proactively adapting to changes in data flow, it optimizes computational efficiency while balancing performance and cost.

A common limitation of the aforementioned works is their focus on single data pipelines, which are often tailored to specific workloads and lack dynamic adjustment to ensure efficient processing across different data pipelines that may run simultaneously. Additionally, these approaches either rely on static mechanisms without adapting configuration parameters over time, prioritize different objectives, or optimize at the data pipeline level rather than the operator level.

## 2.2. Ensembles of data pipelines

To ensure efficient processing of different data pipelines that potentially run simultaneously, more dynamic and holistic approaches that consider underlying resources are required. However, most existing approaches focus on optimizing isolated data pipelines, and only limited works in the literature have explicitly addressed data pipeline ensembles [17,30-32]. Maroulis et al. [30] propose a framework-agnostic scheduler for stream and batch applications that employs voltage scaling to dynamically adapt CPU frequencies, reducing energy consumption while ensuring that the performance requirements of the applications are still met. Nevertheless, the authors assume a fixed deadline for each application, which is only suitable for bounded data streams. Additionally, they focus on different objectives and scheduling entire applications rather than individual data pipeline operators, which may not be the most efficient approach for fine-grained optimization. Wang et al. [31] introduce an elastic scheduling algorithm for microservices, integrating auto-scaling and task scheduling in cloud environments. To manage multiple data pipelines with deadline constraints, the authors introduce an operator that serves as the predecessor to all entry operators and another as the successor to all exit operators within the data pipelines. Although this paper improves the success ratio of meeting deadlines and reduces the cost of well-known applications, it also limits the execution of data pipelines with varying start or end times. Barika et al. [32] propose a dynamic scheduling technique that enables users to define algorithms for handling runtime changes in stream workflows. Additionally, three algorithms for workflow autoscaling in a multi-cloud environment are presented. While this approach takes into account various workflows with different start and end times, it focuses more on enhancing usability for the user rather than optimizing the performance of these applications. Lian et al. [17] present ContTune, a continuous tuning system for elastic stream processing applications. The authors present a novel big-small algorithm to tune the levels of parallelism for stream processing applications, in which the big phase first gives sufficient levels of parallelism, and the small phase focuses on rapidly identifying minimal parallelism levels to enhance resource utilization. This approach optimizes throughput and end-to-end latency while reducing costs by selecting only the minimum required parallelism. However, ContTune lacks a proactive decision-making mechanism to anticipate future workload fluctuations.

## 3. System model

This section presents the system model underlying STREAMLINE, detailing the structure of data streams and data pipelines, along with key performance metrics used for the evaluation.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Model of a data pipeline comprising two data sources ($v_0$ and $v_1$), one data sink ($v_8$), and seven tasks. Data sources and sinks are connected to external services, and operators $v_6$, $v_7$, and $v_8$ consist of several instances.</figcaption></figure>

## 3.1. Data model

A data stream $D = \{(d_t, t) | t \in T\}$ is a potentially infinite sequence of discrete data elements $d_t$, each tagged with a timestamp $t$ representing its generation time. Data elements originate from sources like IoT devices, flow through ingestion systems such as Apache Kafka, and are delivered to sinks like databases or storage systems.

## 3.2. Data pipeline model

We model a data pipeline $P_k$ (see Fig. 1), where $k$ denotes a specific data pipeline instance, as a directed acyclic graph $(O, E)$. Here, $O = \{o_0, ..., o_n\}$ is a finite set of operators comprising data sources $O_{source}$, tasks $O_{task}$, and data sinks $O_{sink}$.

$$O := O_{source} \cup O_{task} \cup O_{sink} \qquad (1)$$

The set $E$ represents directed data flow connections between operators. Specifically, each edge $e_{ij} \in E$ denotes a data stream from upstream operator $o_i$ to downstream operator $o_j$. Operators with no incoming edges from other operators are source operators, while those without outgoing edges represent sink operators. Each operator $o_i$ applies a processing function $f_i(d_t)$ to every incoming data element $d_t$. These functions range from simple tasks like parsing or filtering IoT data to complex operations such as stateful event-time processing, pattern detection, or machine learning inference. Additionally, each operator $o_i$ is characterized by a workload $W_i^t$, a degree of parallelism $x_i$, and a buffer size $s_i$. The workload $W_i$ denotes the number of input data elements received per time unit. The degree of parallelism $x_i$ indicates how many parallel instances process data independently, improving data pipeline capacity by workload distribution. The buffer size $s_i$ defines the amount of data elements temporarily stored before processing.

Multiple data pipelines together form an ensemble $ENS = \{P_0, ..., P_q\}$. It is important to note that the data pipelines within an ensemble do not necessarily start or end at the same time, i.e., their arrival and termination times may vary.

## 3.3. Performance model

The performance model defines the key metrics used to evaluate data pipelines over a time interval $\Delta t = [t, t + \Delta t]$. We consider a set of heterogeneous resources $R = \{r_0, ..., r_m\}$, where each resource $r_j \in R$ represents a computing entity such as a virtual machine, a server, or a containerized environment on which operators can be executed. Every resource $r_j$ is characterized by its number of CPU cores $CPU_j$, and associated cost $C_j$.

### 3.3.1. Throughput

The throughput of an operator $o_i$ is the number of data elements it emits during the time interval $\Delta t$.

$$TP_{\Delta t}(o_i) := \int_{t}^{t+\Delta t} d(o_i, t)dt \qquad (2)$$

where $d(o_i, t)$ is the instantaneous number of data elements produced by operator $o_i$ at time $t$. The throughput of a data pipeline $P_k$ is the cumulative sum of throughputs of all sink operators, representing the total number of data elements produced by the data pipeline at the time interval $\Delta t$.

$$TP_{\Delta t}(P_k) := \sum_{o_i \in O_{sink}} TP_{\Delta t}(o_i) \qquad (3)$$

### 3.3.2. End-to-end latency

The end-to-end latency of an operator $o_i$ is the average time it takes for data elements to travel from their source operators to $o_i$ during the interval $\Delta t = [t, t + \Delta t]$.

$$L_{\Delta t}(o_i) := avg \int_{t}^{t+\Delta t} (t(o_i, d_i) - t(o_s, d_i)) dt \qquad (4)$$
<figure><img src="image_4.png" /><figcaption>Fig. 2. Architecture of STREAMLINE consisting of four components: (A) Workload prediction, (B) Configuration auto-tuning, (C) Operator scheduling, and (D) Stateful seamless update.</figcaption></figure>

where $t(o_i, d_i)$ is the time when the data element $d_i$ is emitted by operator $o_i$, and $o_s \in O_{source}$ is the source operator that originates $d_i$. The end-to-end latency of a data pipeline $P_k$ is the average time required for data elements to travel from source operators to sink operators within $\Delta t$.

$$L_{\Delta t}(P_k) := \operatorname*{avg}_{o_i \in O_{\text{sink}}} \{L_{\Delta t}(o_i)\} \qquad (5)$$

### 3.3.3. CPU utilization

The CPU time used by an operator $o_i$ during the time interval $\Delta t$ is defined as:

$$C P U_{\Delta t}^{T}(o_{i}) := \int_{t}^{t+\Delta t} c(o_{i}, t) d t \qquad (6)$$

where $c(o_i, t)$ is the instantaneous CPU time consumed by operator $o_i$ at time $t$. The CPU utilization for a set of resources $R$ is the average fraction of time the CPUs are actively processing data across all resources during $\Delta t$.

$$C P U_{\Delta t}^{U}(R) := \operatorname*{avg}_{r_j \in R} \left\{ \frac{\sum_{o_i \in O_j} C P U_{\Delta t}^{T}(o_i)}{C P U_j \times \Delta t} \right\} \qquad (7)$$

where $O_j$ is the set of operators running on resources $r_j \in R$.

### 3.3.4. Cost

The cost of a set of resources $R$ over a time interval $\Delta t$ is the total sum of the individual costs of all resources used during that period.

$$C_{\Delta t}(R) := \sum_{r_j \in R} C_j \times \Delta t \qquad (8)$$

## 4. STREAMLINE

This section presents STREAMLINE, a multi-layer dynamic auto-tuning framework for managing data pipeline ensembles on resource clusters, for example, edge nodes processing IoT data streams with low end-to-end latency. Unlike existing state-of-the-art methods, STREAMLINE accounts for interdependent configuration parameters, cluster resource constraints, and trade-offs across concurrently running data pipelines. It automatically tunes operator parallelism and buffer sizes to optimize throughput, end-to-end latency, and CPU usage while employing a resource-efficient scheduling strategy to further optimize CPU utilization and monetary costs. As illustrated in Fig. 2, STREAMLINE consists of four key components: (A) Workload prediction, forecasting future workloads; (B) Configuration auto-tuning, adjusting configuration parameters based on predicted workloads and user-defined objective weights; (C) Operator scheduling, assigning operators to resources; and (D) Stateful seamless update, applying configuration parameter and operator scheduling updates. Each component is detailed in the following subsections.

Algorithm 1 outlines the operation of STREAMLINE, which takes as input the time interval $\Delta t$, the rescheduling threshold $\alpha$, and the set of available resources $R$. STREAMLINE is triggered either when a new data pipeline $P_k$ is submitted (line 3) or at fixed intervals (line 7). When a new data pipeline is submitted, it is added to the active ensemble ENS (line 4). Then, STREAMLINE predicts the workload $W_i$ and automatically tunes configuration parameters $CF_i$ for each operator $o_i \in P_k$ using the predConf(·) function (see Sections 4.1 and 4.2). Once suitable configuration parameters have been found for each operator in $P_k$ (line 5), STREAMLINE schedules operators on available resources via schedule(·) (line 6; see Section 4.3). The scheduler checks for underutilization (line 18), i.e., if the average CPU utilization falls below the threshold $\alpha$, a global rescheduling is triggered to consolidate operators onto fewer resources, improving efficiency and reducing cost. In other words, instead of keeping operators
<table><caption>**Algorithm 1: STREAMLINE**</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input:</td><td>Δt, α, R;</td></tr><tr><td>1</td><td>ENS ← θ;</td></tr><tr><td>2</td><td>while running do</td></tr><tr><td>3</td><td>if P<sub>k</sub> &notin; ENS arrives then</td></tr><tr><td>4</td><td>ENS ← ENS &cup; P<sub>k</sub>;</td></tr><tr><td>5</td><td>CF &leftarrow; predConf(P<sub>k</sub>, Δt, β<sub>k</sub>);</td></tr><tr><td>6</td><td>schedule(CF, R);</td></tr><tr><td>7</td><td>if Δt elapsed then</td></tr><tr><td>8</td><td>CF &leftarrow; θ;</td></tr><tr><td>9</td><td>foreach P<sub>k</sub> &isin; ENS do</td></tr><tr><td>10</td><td>CF &leftarrow; CF &cup; predConf(P<sub>k</sub>, Δt, β<sub>k</sub>);</td></tr><tr><td>11</td><td>schedule(CF, R)</td></tr><tr><td>12</td><td>Function predConf(P<sub>k</sub>, Δt, β<sub>k</sub>):</td></tr><tr><td>13</td><td>W'<sub>i</sub> &forall; o<sub>i</sub> &isin; P<sub>k</sub> &leftarrow; using Alg. 2;</td></tr><tr><td>14</td><td>CF<sub>i</sub> &forall; o<sub>i</sub> &isin; P<sub>k</sub> &leftarrow; using Alg. 3;</td></tr><tr><td>15</td><td>return {CF<sub>i</sub> | if CF<sub>i</sub> has changed};</td></tr><tr><td>16</td><td>Function schedule(CF, R):</td></tr><tr><td>17</td><td>schedule all o<sub>i</sub> &isin; CF using Alg. 4;</td></tr><tr><td>18</td><td>if CPU<sub>Δt</sub><sup>U</sup>({r &isin; R | ∃o<sub>j</sub> &isin; &cup;<sub>P<sub>k</sub>&isin;ENS</sub> s. t. o<sub>j</sub> uses r}) &lt; α then</td></tr><tr><td>19</td><td>schedule all o<sub>i</sub> &isin; &cup;<sub>P<sub>k</sub>&isin;ENS</sub> using Alg. 4;</td></tr><tr><td>20</td><td>update configuration parameters and operator schedules using Alg. 5;</td></tr></tbody></table>

<table><caption>**Algorithm 2: Workload prediction.**</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input:</td><td>o<sub>i</sub> &isin; O, h<sub>i</sub> &forall; o<sub>i</sub> &isin; O, Δt;</td></tr><tr><td>Output:</td><td>predicted workload W'<sub>i</sub> &forall; o<sub>i</sub> &isin; O over Δt;</td></tr><tr><td>1</td><td>for each o<sub>i</sub> &isin; O do</td></tr><tr><td>2</td><td>if o<sub>i</sub> &isin; O<sub>source</sub> then</td></tr><tr><td>3</td><td>W'<sub>i</sub> &leftarrow; Transformer(h<sub>i</sub><sup>W</sup>, Δt);</td></tr><tr><td>4</td><td>else</td></tr><tr><td>5</td><td>v<sub>i</sub> &leftarrow; h<sub>i</sub><sup>W</sup>/h<sub>i</sub><sup>out</sup>;</td></tr><tr><td>6</td><td>W'<sub>i</sub> &leftarrow; avg({W'<sub>p</sub> &times; v<sub>i</sub> | o<sub>p</sub> &isin; pred(o<sub>i</sub>)});</td></tr></tbody></table>

running on multiple underutilized resources, the algorithm reschedules them onto fewer resources to reduce costs and make more efficient use of available resources. Updated configuration parameters and schedules are applied to the stream processing framework in line 20 (see Section 4.4). To adapt to dynamic and fluctuating workloads, STREAMLINE periodically tunes all active data pipelines in ENS (line 7). It iterates through each data pipeline (line 9), identifies operators requiring updates (line 10), and reschedules them based on CPU requirements (line 11). Although STREAMLINE does not directly respond to transient network latency fluctuations, their effects appear in processing metrics. By continuously monitoring these metrics, STREAMLINE implicitly accounts for such delays and adapts operator configurations and scheduling in subsequent cycles.

## 4.1. Workload prediction

STREAMLINE uses prediction to enable proactive, rather than reactive, adaptation by forecasting future operator workloads within a given time interval. Without prediction, STREAMLINE would rely solely on current observations, which may be sufficient in stable environments but inadequate for dynamic scenarios. In such cases, delayed updates lead to higher end-to-end latency, inefficient resource use, and degraded performance. Workload forecasting methods range from lightweight statistical techniques (e.g., moving averages, autoregressive models) to advanced machine learning approaches like recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) [33,34]. While statistical models are efficient, they struggle with non-linear or abrupt workload changes. RNNs and LSTMs better capture temporal patterns but have limitations: RNNs suffer from vanishing gradients and poor long-term memory, while LSTMs improve this with gating mechanisms but remain sequential, limiting responsiveness to sudden shifts [35–37]. Transformer-based models overcome these issues using self-attention mechanisms that capture long-range dependencies and complex temporal correlations [18,38]. They handle non-linear, irregular workload patterns common in real-time IoT streams, support parallel training for scalability, and allow multivariate input modeling. This enables the integration of contextual features, such as time-of-day, operator type, or historical metrics, to further improve prediction accuracy [18,19].

Algorithm 2 outlines the procedure for predicting operator workloads $W_i$ for all $o_i \in O$ over a given time interval $\Delta t$. Operators are processed in topological order (line 1) to ensure that all predecessors of an operator $o_i$ are predicted before $o_i$ itself. For source operators (line 2), whose workloads cannot be derived from upstream activity, STREAMLINE employs a pre-trained transformer model (line 3). This model processes recent workload histories $h_i^W$, incorporating positional encodings and self-attention layers to
<table><caption>**Algorithm 3: Configuration auto-tuning.**</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input:</td><td>o<sub>i</sub> ∈ O, W'<sub>p</sub> ∀ o<sub>i</sub> ∈ O over Δt, β;</td></tr><tr><td>Output:</td><td>configuration CF<sub>i</sub> ∀ o<sub>i</sub> ∈ O.</td></tr><tr><td>1</td><td>for each o<sub>i</sub> ∈ O do</td></tr><tr><td>2</td><td>&nbsp;&nbsp;&nbsp;&nbsp;x'<sub>p</sub> ← avg({x<sub>p</sub> | o<sub>p</sub> ∈ pred(o<sub>i</sub>)});</td></tr><tr><td>3</td><td>&nbsp;&nbsp;&nbsp;&nbsp;for each CF<sub>k</sub> proposed by NSGA-II do</td></tr><tr><td>4</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| TP<sub>k</sub>, L<sub>k</sub>, CPU<sup>T</sup><sub>k</sub> ← evaluate(o<sub>i</sub>, W<sub>i</sub>, x<sub>k</sub>, s<sub>k</sub>, x'<sub>p</sub>);</td></tr><tr><td>5</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CF<sub>i</sub> ← {CF<sub>k</sub> | min(β<sub>L</sub> × L<sub>k</sub> + β<sub>C</sub> × CPU<sup>T</sup><sub>k</sub> − β<sub>T</sub> × TP<sub>k</sub>)};</td></tr><tr><td>6</td><td>Function evaluate(o<sub>i</sub>, W'<sub>i</sub>, s<sub>i</sub>, x<sub>i</sub>, x'<sub>p</sub>):</td></tr><tr><td>7</td><td>&nbsp;&nbsp;&nbsp;&nbsp;for each w'<sub>j</sub> ∈ W'<sub>i</sub> do</td></tr><tr><td>8</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;PA(o<sub>i</sub>, x<sub>i</sub>) ← PolynomialRegression(o<sub>i</sub>, x<sub>i</sub>);</td></tr><tr><td>9</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;w'<sub>j</sub> ← min(w'<sub>j</sub>, PA(o<sub>i</sub>, x<sub>i</sub>));</td></tr><tr><td>10</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;L'(o<sub>i</sub>), CPU<sup>T</sup>(o<sub>i</sub>) ← NN(o<sub>i</sub>, w'<sub>j</sub>, s<sub>i</sub>, x<sub>i</sub>, x'<sub>p</sub>);</td></tr><tr><td>11</td><td>&nbsp;&nbsp;&nbsp;&nbsp;return avg(w'<sub>j</sub>, L'(o<sub>i</sub>), CPU<sup>T</sup>(o<sub>i</sub>));</td></tr></tbody></table>

capture temporal dependencies. The predicted workload for the next interval $\Delta t$ is generated via a final linear layer. The transformer-based model is trained offline on highly variable workload traces to ensure it generalizes well across diverse scenarios. While real-world workloads may shift in characteristics such as variance or periodicity, we assume the trained model remains valid for all workloads considered in this paper. Retraining strategies to address distribution shifts are left for future work.

For non-source operators, whose workloads are highly correlated with their predecessors, STREAMLINE avoids model-based inference and instead uses an empirical ratio $v_i$ (line 6). This ratio, computed from historical input/output data (line 5), represents the average number of output elements produced per input. For example, $v_i = 0.5$ means two output elements per input; $v_i = 2.0$ means one output every two inputs. By limiting transformer-based prediction to source operators, STREAMLINE achieves accurate workload estimation while maintaining efficiency across the data pipeline.

## 4.2. Configuration auto-tuning

After predicting operator workloads, STREAMLINE tunes each operator's configuration parameters $CF_i = (x_i, s_i)$, where $x_i$ is the parallelism and $s_i$ the buffer size. Tuning these parameters involves complex trade-offs between throughput, end-to-end latency, and resource usage while avoiding costly runtime profiling.

Algorithm 3 details the tuning of configuration parameters for each operator $o_i \in O$. The process begins by iterating over all operators (line 1). To account for synchronization overhead between operators with different degrees of parallelism, the average parallelism of each operator's predecessors $x'_p$ is computed (line 2). STREAMLINE uses the NSGA-II algorithm [20] to auto-tune each operator's parallelism $x_i$ and buffer size $s_i$ (line 3). Lightweight heuristics incur low overhead but often fail to generalize [39], whereas exhaustive searches yield better results at the cost of high computation time [3]. We selected NSGA-II for its efficient non-dominated sorting and crowding-distance mechanisms, which outperform alternatives like SPEA2 or MOEA/D in maintaining solution diversity, a key factor for generating a well-distributed Pareto front. The auto-tuning aims to maximize throughput while minimizing CPU time and end-to-end latency (line 5), using a weighted objective function parameterized by $\beta$. We estimate CPU time per time unit ($CPU_k^T$) rather than absolute time to remain hardware-agnostic and flexible to core availability.

To avoid costly evaluations on the real system, we employ a pre-trained neural network as a surrogate model that approximates the behavior of the stream processing framework (lines 6–11). Trained on a diverse set of workloads and configuration parameters, the model predicts performance metrics based on the operator $o_i$, estimated workload $w'_j$, proposed buffer size $s_i$, proposed parallelism $x_i$, and average predecessor parallelism $x_p$ (line 10). This surrogate model replaces direct interactions with the stream processing framework, significantly accelerating optimization while maintaining accuracy. To account for backpressure, we cap the estimated workload using the operator's maximum processing ability ($PA$), which is modeled via polynomial regression [17] as a function of parallelism (line 8):

$$PA(o_i, x_i) := TP_{\Delta t}(o_i)/PT_{\Delta t}(o_i) \qquad (9)$$

where $TP_{\Delta t}(o_i)$ is the number of processed data elements per time unit and $PT_{\Delta t}(o_i)$ is the operator's active processing time. The expected throughput $TP'(o_i)$ is the minimum of the predicted workload and $PA$, ensuring throughput estimates remain realistic.

This combination of the surrogate model and evolutionary algorithm enables high-quality configuration parameter tuning under fluctuating workloads without incurring prohibitive runtime costs. Static heuristics, by contrast, would yield suboptimal configuration parameters, while direct system interaction would be too slow for practical use. Adapting the surrogate model to unseen configurations and hardware will be explored in future work.
**Algorithm 4: Operator scheduler**

Input: $o_i \in O, r_j \in R, CF_i$ and $CPU^{T'}(o_i) \forall o_i \in O, \alpha$;

Output: schedule of operators $o_i \in O$ on $r_j \in R$.

1. $G \leftarrow \{O_b | O_b = \{o_i \in O | s_i = s_b\}\}$;

2. **for each** $O_b \in G$ **do**

3. &nbsp;&nbsp;&nbsp;&nbsp;$$CPU^T(O_b) \leftarrow \sum_{o_i \in O_b} CPU^{T'}(o_i);$$

4. &nbsp;&nbsp;&nbsp;&nbsp;sort($G$, by $CPU^T(O_b)$);

5. **for each** $O_b \in G$ **do**

6. &nbsp;&nbsp;&nbsp;&nbsp;$$R_{U} \leftarrow \{r_j \in R | \exists o_i \in O \text{ s.t. } o_i \text{ uses } r_j\};$$

7. &nbsp;&nbsp;&nbsp;&nbsp;$$CPU^A(R_U) \leftarrow CPU_J \times (1 - CPU^U(R_U));$$

8. &nbsp;&nbsp;&nbsp;&nbsp;$$R_M \leftarrow \min(R | CPU^T(O_b) < CPU^A(R_U) + CPU^A(R));$$

9. &nbsp;&nbsp;&nbsp;&nbsp;sort($O_b$, by $CPU^{T'}(o_i)$);

10. **for each** $o_i \in O_b$ **do**

11. &nbsp;&nbsp;&nbsp;&nbsp;sort($R_M$, by $CPU^A(r_j)$);

12. &nbsp;&nbsp;&nbsp;&nbsp;**for each** $r_j \in R_M$ **do**

13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** $CPU^A(r_j) > CPU^{T'}(o_i)$ **then**

14. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;schedule $o_i$ on $r_j$;

**Algorithm 5: Stateful seamless update.**

Input: $P_k$;

Output: reconfigured $P'_k$.

1. $s \leftarrow \text{savepoint}(P_k)$;

2. $P'_k \leftarrow \text{start}(s, \text{deactivate}(o_i) \forall o_i \in O_{\text{sink}}^{P'_k})$;

3. **if** $\forall d \in A_p, d \in A_{p'}$ **then**

4. &nbsp;&nbsp;&nbsp;&nbsp;$$\text{stop}(P_k);$$

5. &nbsp;&nbsp;&nbsp;&nbsp;$$A_L \leftarrow \{d | d \in A_p \text{ and } d = \text{last}(o_i) \forall o_i \in P_k\};$$

6. &nbsp;&nbsp;&nbsp;&nbsp;$$\text{activate}(o_i) \forall o_i \in O_{\text{sink}}^{P'_k}, \text{ starting from } A_L;$$

### 4.3. Operator scheduler

Efficient operator scheduling in stream processing systems requires balancing multiple objectives, such as minimizing resource usage, preventing over-utilization, and sustaining high throughput. Traditional static heuristics [16,28] lack adaptability to workload changes, leading to resource waste during low demand or performance drops during spikes. On the other hand, reactive methods [17,24] may cause temporary performance degradation, as they react after performance issues have arisen. STREAMLINE overcomes these limitations by dynamically scheduling operators based on predicted CPU requirements, enabling precise resource control and avoiding inefficiencies inherent in static or reactive approaches.

Algorithm 4 schedules operators based on predicted CPU requirements from Algorithm 3, along with configuration parameters and available resources. It starts by grouping operators with the same buffer size (line 1), as streaming frameworks typically require consistent buffer sizes across operators within a data pipeline. For each group, it sums CPU demands of operators (line 3) and sorts groups by total CPU demand (line 4). The algorithm then iterates over these groups (line 5) and calculates the available CPU capacity of already used resources (lines 6–7). This prevents resource over-utilization and enables scheduling that considers previously allocated resources. In line 8, the algorithm selects the minimal set of resources to meet the group’s total CPU demand by combining used and new resources. In line 9, operators are sorted by CPU requirements, prioritizing higher demands to minimize allocation gaps and ensure all operators are scheduled. Finally, each operator instance (lines 10–12) is assigned to available resources sorted by CPU capacity, allocating them only where their demand fits (lines 13–14).

### 4.4. Stateful seamless update

When a configuration parameter or schedule update is required, STREAMLINE considers two scenarios: (i) the data pipeline has not yet started, allowing the update to be applied seamlessly; or (ii) the data pipeline is already running, requiring the update to be applied at runtime. For runtime updates, common stream processing frameworks [21–23] use two approaches: **A** **Stateful stop-and-restart** and **B** **Stateless parallel update**. **A** In most stream processing frameworks, the default way to update a data pipeline involves stopping it with a savepoint and restarting it with new settings. This preserves application state but causes downtime, which can disrupt time-sensitive IoT applications. **B** Stateless parallel update starts a new data pipeline alongside the running one, terminating the old only when the new data pipeline is ready. This avoids downtime but risks processing duplicate data, causing state inconsistencies.

To reduce downtime while preserving application state consistency, we propose a novel approach, **C**, for applying configuration parameter updates or reassigning operators in a running data pipeline. Fig. 3 illustrates this method alongside existing solutions. In line 1 of Algorithm 5, a savepoint of the running data pipeline is created without interruption, capturing its state and metadata
<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Approaches for updating configurations and schedule to a running data pipeline: (A) Stateful stop-and-restart, (B) Stateless parallel update, and (C) Stateful seamless update.</figcaption></figure>

(e.g., Kafka offsets). In line 2, it launches a new data pipeline $P'_k$ with updated configuration parameters and schedules, using the previously generated savepoint. The new data pipeline processes the same data elements as the original one, starting from the point at which the savepoint was created. To prevent duplicate outputs, the new data pipeline initially discards all sink data. Assuming deterministic processing, the state remains consistent as both data pipelines process identical data. In line 3, the algorithm waits for $P'_k$ to catch up with all data processed by the original data pipeline $P_k$, synchronizing their progress. Once synchronized, it stops the original data pipeline (line 4) and retrieves its last processed data elements, $A_L$ (line 5). Finally, it activates the sinks of $P'_k$ from $A_L$ onward (line 6), ensuring exactly-once processing and full data consistency. STREAMLINE assumes reliable data delivery, as provided by frameworks such as Apache Flink or Apache Kafka. It does not handle packet loss at the network level directly, but ensures correct processing as long as upstream components provide exactly-once semantics.

## 4.5. Complexity analysis

This section analyzes the worst-case time complexity of STREAMLINE. We examine the complexity of workload prediction, configuration auto-tuning, operator scheduling, and stateful seamless update separately, before summarizing the overall complexity of STREAMLINE. Let $N$ denote the total number of operators across all data pipelines in the ensemble, and $R$ the number of available computing resources. We assume that all model-related hyperparameters (e.g., transformer depth, neural network size, number of generations) are fixed and independent of $N$.

**Workload Prediction.** The worst-case time complexity of Algorithm 2 during inference is $O(N \cdot T^2 + N^2)$, where $T^2$ denotes the transformer inference cost, depending on the sequence length and model dimensions [38]. The $O(N \cdot T^2)$ component accounts for transformer inference applied to all $N$ source operators, while $O(N^2)$ arises from aggregating empirical ratios at non-source operators, each of which may depend on up to $N-1$ predecessors in the worst case.

**Configuration Auto-Tuning.** For each operator, the NSGA-II algorithm [20] proposes a set of configuration candidates (parallelism and buffer size), which are evaluated using surrogate models. The complexity of this step is $O(N^2 + N \cdot C \cdot W)$, where $N^2$ captures averaging across predecessors, and $N \cdot C \cdot W$ accounts for evaluating $C$ configurations per operator over $W$ timesteps. Since $C$ and $W$ are fixed, the overall complexity simplifies to $O(N^2)$.

**Operator Scheduling.** Operators are first grouped by buffer size and then greedily assigned to resources based on available CPU capacity. The scheduling algorithm involves sorting and resource-aware assignment, resulting in a worst-case time complexity of $O(N \log N + N \cdot R \log R)$. Since $R$ is typically small and constant in practical deployments, the effective complexity simplifies to $O(N \log N)$.

**Stateful Seamless Update.** The time complexity of the stateful seamless update involves iterating over $N$ operators for activating and deactivating sinks, as well as tracking $A$ active data elements, resulting in a complexity of $O(N + A)$. Since $A$ is typically small and bounded, the overall complexity simplifies to $O(N)$.

**Overall Complexity.** Combining the above, the worst-case time complexity of STREAMLINE as implemented in Algorithm 1 is dominated by the workload prediction with complexity up to $O(N^2 + N \cdot T^2)$. We emphasize that this complexity analysis pertains exclusively to the online phase, while model training is conducted offline and may involve higher computational costs.

## 5. Evaluation

This section describes the experimental evaluation of STREAMLINE. We tested STREAMLINE with varying ensemble sizes using IoT and streaming benchmarks and compared it against three state-of-the-art competitors.

**Infrastructure.** All experiments were conducted using Apache Flink 1.19.1 on a Kubernetes cluster with 30 nodes, deployed on the Gros cluster at Grid’5000’s Nancy site [41]. Resources were virtualized as shown in Table 1. Two type-3 nodes hosted the Apache Flink JobManager, Kafka (v3.7.0), and storage services, while the remaining nodes acted as worker nodes running TaskManagers.

**Data Pipelines.** We evaluated five data pipelines: four from the RIoTBench suite [2], which features 27 commonly used IoT operators across various functional categories, and the Yahoo Streaming Benchmark (YSB) [42], a widely used streaming
<figure><img src="image_6.png" /><figcaption>**Fig. 4.** High-level interactions between RIoTBench data pipelines. ETL produces input for STATS, PRED, and TRAIN, while TRAIN produces additional input for PRED.</figcaption></figure>

<table><caption>**Table 1**
Grid'5000 resources used in our evaluation, including cost from comparable resources of AWS [40].</caption><thead><tr><td>Type</td><td>Quantity</td><td>CPU cores</td><td>Memory</td><td>$/h</td></tr></thead><tbody><tr><td>1</td><td>10</td><td>4</td><td>8 GB</td><td>0.192</td></tr><tr><td>2</td><td>10</td><td>8</td><td>16 GB</td><td>0.384</td></tr><tr><td>3</td><td>10</td><td>16</td><td>24 GB</td><td>0.768</td></tr></tbody></table>

<table><caption>**Table 2**
Hyperparameter configuration.</caption><thead><tr><th>Hyperparameter</th><th>Value</th><th>Hyperparameter</th><th>Value</th></tr></thead><tbody><tr><td colspan="4">NSGA-II</td></tr><tr><td>n_var</td><td>2 (x<sub>i</sub>, s<sub>i</sub>)</td><td>n_obj</td><td>3 (TP<sub>k</sub>, L<sub>k</sub>, CPU<sub>k</sub><sup>T</sup>)</td></tr><tr><td>xl, xu</td><td>[1, 16k] to [max x<sub>i</sub>, 64k]</td><td>pop_size</td><td>20</td></tr><tr><td>n_gen</td><td>20</td><td>initialPopulation</td><td>99</td></tr><tr><td>parallelism_options</td><td>1 to max x<sub>i</sub></td><td>segment_size_options</td><td>[16k, 32k, 64k]</td></tr><tr><td>default β</td><td>[0.5, 0.25, 0.25]</td><td>Loss function</td><td>MSELoss</td></tr><tr><td>optimizer</td><td>Adam</td><td></td><td></td></tr><tr><td colspan="4">Transformer architecture</td></tr><tr><td>d_model</td><td>128</td><td>nhead</td><td>4</td></tr><tr><td>num_layers</td><td>4</td><td>input_size</td><td>100</td></tr><tr><td>output_size</td><td>30</td><td>max_len</td><td>500</td></tr><tr><td>epochs</td><td>500</td><td>batch_size</td><td>128</td></tr><tr><td>lr</td><td>0.0002</td><td></td><td></td></tr><tr><td colspan="4">Surrogate neural network</td></tr><tr><td>batch_size</td><td>32</td><td>lr</td><td>0.001</td></tr><tr><td>epochs</td><td>200</td><td>train/test split</td><td>80/20</td></tr><tr><td>hidden_layers</td><td>[256, 128]</td><td>Activation</td><td>ReLU</td></tr></tbody></table>

benchmark. The RIoTBench pipelines are designed to reflect realistic IoT scenarios in smart cities and fitness, such as traffic monitoring and personal activity tracking, making STREAMLINE applicable to those domains. Additionally, the system's low-latency and adaptive tuning capabilities are particularly well-suited for V2X (Vehicle-to-Everything) and healthcare IoT scenarios, where dynamic workload shifts and quick responsiveness are critical.

* **YSB**: An advertising analytics data pipeline that detects and stores relevant events.

* **ETL**: Performs preprocessing and cleaning on IoT sensor data streams.

* **STATS**: Conducts three types of statistical analysis, grouped by IoT sensor, and visualizes the results.

* **TRAIN**: Periodically trains forecasting models using preprocessed data.

* **PRED**: Generates predictions and classifications for domain-specific decision-making.

While the YSB operates independently, ETL, STATS, TRAIN, and PRED interact with each other, as illustrated in Fig. 4, and represent a scenario that is generalizable to multiple domains such as smart power grids, transportation systems, and fitness tracking [2].

**Workload.** For YSB, we varied the rate of incoming ad events (i.e., data elements) to represent low (50,000 el/s), medium (250,000 el/s), and high (500,000 el/s) workloads. **TRAIN** periodically triggered the training process every two hours. For the remaining RIoTBench data pipelines, we used two publicly available datasets, **FIT** and **TAXI**, with scaling applied as recommended in [2]. The **FIT** dataset, containing body motion and vital signs, produced a steady workload of approximately 7000 el/s. The **TAXI** dataset, based on GPS traces from New York City taxis, exhibited fluctuating workloads ranging from 1000 to 10,000 el/s.

**Training Setup.** STREAMLINE used two pre-trained models: one for workload prediction and a surrogate model of the underlying stream processing framework. For this evaluation, both models were trained on the real-world TAXI dataset, which captures typical IoT workload patterns. Hyperparameters were selected through empirical validation (see Table 2 for details). Training was conducted on hardware with NVIDIA T600 GPUs (4 GB memory) using CUDA 12.6.
<figure><img src="image_7.png" /><figcaption>Fig. 5. Throughput, end-to-end latency, CPU utilization, and monetary cost for the ETL data pipeline across two datasets.</figcaption></figure>

**Competitors.** We compared STREAMLINE against three state-of-the-art baselines: FD, FMAX, and ContTune (CT). FD uses Apache Flink's default configuration ($x_i = 1$, $s_i = 32k$) and scheduling strategy, filling up available slots in one TaskManager [21] before moving to the next one. It serves as a baseline to evaluate how our approach compares to default stream processing framework configuration parameters and scheduling strategies. FMAX maximizes operator parallelism ($x_i = \max x_i$) up to the point of diminishing returns, using Apache Flink's scheduler, aiming to boost throughput through parallelism. CT [17] is a continuous tuning framework that minimizes operator parallelism to reduce CPU usage and monetary cost while improving throughput and end-to-end latency. We include CT as it represents the most relevant adaptive tuning approach, unlike alternatives that focus on batch processing, single data pipelines, or static optimization. All baselines follow Apache Flink's best practices [43], configuring TaskManagers with slots equal to CPU cores, enabling stronger nodes to host more operators. STREAMLINE, in contrast, uses a fine-grained scheduler that can assign more operators per TaskManager, improving resource efficiency. Unless stated otherwise, STREAMLINE used $\beta = [0.5, 0.25, 0.25]$ as the default objective weights.

**Evaluation Metrics.** We evaluated STREAMLINE and its competitors using four key metrics: throughput, end-to-end latency, CPU utilization, and monetary cost (see Section 3). Throughput was measured every 5 s via the Apache Flink REST API, while latency markers tracked end-to-end latency at the same interval. As Apache Flink lacks native operator-level CPU metrics, we used ThreadMXBean [44] to monitor CPU time per operator. Monetary cost was estimated using AWS EC2 instance pricing, based on machines with equivalent CPU and memory configurations. Each experiment was repeated three times, and results were averaged for consistency. CT applied updates using a stateful stop-and-restart method, while STREAMLINE used its stateful seamless update mechanism.

## 5.1. Single data pipeline

In our first experiment, we run each data pipeline independently under the workloads described in Section 3 to analyze their behavior. To evaluate runtime adaptability, we re-trigger both STREAMLINE and CT midway through execution, allowing them to adjust operator configuration parameters and scheduling in response to workload fluctuations.

### 5.1.1. ETL

As shown in Fig. 5, STREAMLINE and FMAX achieved the highest throughput, differing by only 1%, as both maintained sufficient parallelism to handle the workload. Compared to CT, STREAMLINE increased throughput by 19% for TAXI and 8% for FIT. CT underperformed on TAXI due to its lack of proactive adjustment, reacting too slowly to workload fluctuations. On FIT, it was unable to make full use of operator capacity, as its focus on minimizing parallelism led to synchronization delays and uneven data distribution. In contrast, STREAMLINE considered the parallelism of predecessor operators and selected higher degrees to accelerate processing. FD, using Apache Flink's default parallelism of one, handled low TAXI workloads but struggled at higher rates, resulting in 14% lower average throughput. On FIT, the sustained workload further exposed FD's limitations, yielding 34% lower throughput than STREAMLINE. The fluctuations seen in TAXI reflected its variable workload, while the steady pattern in FIT led to stable results. Lower initial throughput on FIT was due to data pipeline warm-up before reaching peak performance.

End-to-end latency on FIT differed by less than 4% across STREAMLINE, FMAX, and CT. However, STREAMLINE reduced end-to-end latency by 76% compared to FD, which failed to process data in time under the sustained workload. While CT also failed to keep up with the full workload, its end-to-end latency impact was smaller, as it nearly processed all incoming data in time. On TAXI, STREAMLINE reduced end-to-end latency by 46% compared to CT, thanks to its ability to manage workload spikes. Average end-to-end latencies for FD and FMAX differed by under 4%. FD performed well under light workload but suffered high end-to-end latencies as the workload increased due to limited parallelism. In contrast, STREAMLINE and FMAX consistently maintained low end-to-end latencies by processing data on time.

STREAMLINE utilized resources on average 136% more efficiently than state-of-the-art methods on the TAXI dataset by scheduling operators on shared resources. FD achieved better CPU utilization than FMAX and CT due to its low parallelism, which required fewer operators. In contrast, FMAX scheduled the most operators, resulting in the poorest CPU efficiency. On FIT, STREAMLINE achieved 211% higher CPU utilization, again due to its efficient scheduling. CPU utilization on TAXI showed greater variance due to workload fluctuations, while FIT exhibited stable usage under its steady workload. In terms of cost, STREAMLINE reduced expenses by 60%, 51%, and 67% compared to CT, FD, and FMAX on TAXI, respectively, by maximizing resource efficiency. Unlike the other methods, which relied on multiple resources, STREAMLINE ran the data pipeline on a single type-2 instance. For FIT, it reduced costs by an average of 63% across all baselines by placing operators on shared resources.
<figure><img src="image_8.png" /><figcaption>Fig. 6. Throughput, end-to-end latency, CPU utilization, and monetary cost for the STATS data pipeline across two datasets.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Throughput, end-to-end latency, CPU utilization, and monetary cost for the PRED data pipeline across two datasets.</figcaption></figure>

### 5.1.2. STATS

Fig. 6 shows that STREAMLINE and FMAX achieved similar throughput on the TAXI dataset by successfully processing the full workload. In contrast, CT and FD failed to scale parallelism proactively, resulting in an average 9% lower throughput than STREAMLINE. On FIT, all methods reached peak throughput, as its steady workload was manageable across configuration parameters and operator schedules. For end-to-end latency, STREAMLINE outperformed CT and FD on TAXI by 90% and 79%, respectively, and matched FMAX. The higher end-to-end latency in CT and FD stemmed from their inability to keep up with the data pipeline's workload. On FIT, STREAMLINE reduced end-to-end latency by 13%, 21%, and 2% compared to CT, FD, and FMAX, respectively. These gains resulted from STREAMLINE's assignment of operators on shared resources to minimize communication overhead, increased parallelism for faster processing, and well-selected buffer sizes for efficient data transfer. STREAMLINE also utilized resources 409% and 331% more efficiently than competitors on TAXI and FIT, respectively, thanks to its ability to schedule operators on shared resources, maintaining a consistent 45% average utilization. Competitors required significantly more resources due to the large number of operators in STATS. In terms of cost, STREAMLINE reduced expenses by 76% on TAXI and 79% on FIT by using fewer, more efficiently utilized resources.

### 5.1.3. PRED

As shown in Fig. 7, throughput remained consistent across all methods and datasets, as each configuration successfully handled the full workload over time. However, in terms of end-to-end latency, STREAMLINE outperformed competitors by 14% on TAXI and 20% on FIT by scheduling operators on shared resources and minimizing parallelism. Lower parallelism reduced coordination overhead, enabling faster data processing. In contrast, FMAX exhibited 35% higher end-to-end latency on FIT due to its excessive parallelism, which proved inefficient for this workload. Although high parallelism may benefit heavier workloads or other data pipelines, it was suboptimal in this setting. STREAMLINE utilized resources 556% more efficiently by scheduling all operators on the weakest viable resource (type 1), whereas competitors required additional resources. Despite this, STREAMLINE maintained a 19% CPU utilization, indicating effective use without overloading the resource. In terms of cost, STREAMLINE reduced expenses by 86% compared to all competitors through superior resource efficiency. Costs were similar across datasets, as each method relied on comparable infrastructure.

### 5.1.4. TRAIN

As shown in Fig. 8, throughput remained consistent across all methods, as only a single message was processed while the system mostly idled, waiting for new data. In terms of end-to-end latency, STREAMLINE achieved 10% lower end-to-end latency than competitors by automatically tuning buffer sizes and efficiently scheduling operators on shared resources. Although overall CPU utilization was low due to idle periods, STREAMLINE utilized resources 344% more efficiently. This efficiency resulted in an 81% cost reduction compared to competitors, as STREAMLINE ran the data pipeline on a single type-1 resource.

### 5.1.5. YSB

Fig. 9 compares YSB performance across competitors under low, medium, and high workloads. Under a low workload, all methods achieved similar throughput, fully handling the input. Under medium workload, STREAMLINE, CT, and FMAX maintained performance, while FD suffered a 91% throughput drop due to insufficient parallelism. Under high workload, only STREAMLINE and FMAX reached full throughput. CT trailed by 12%, and FD fell behind by 306%, both constrained by insufficient parallelism.
<figure><img src="image_10.png" /><figcaption>Fig. 8. Throughput, end-to-end latency, CPU utilization, and monetary cost for TRAIN.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 9. Throughput, end-to-end latency, CPU utilization, and monetary cost for the YSB data pipeline across three datasets.</figcaption></figure>

STREAMLINE reduced end-to-end latency by 8% compared to CT and FD under low workload, benefiting from tuned operator scheduling and buffer sizes. FMAX had 74% higher end-to-end latency due to increased synchronization overhead from greater parallelism. For medium and high workloads, STREAMLINE and FMAX showed similar end-to-end latency, as higher workloads improved parallelism efficiency. CT experienced 11% and 13% higher end-to-end latency at medium and high workloads due to parallelism fluctuations causing data rebalancing. FD had the lowest end-to-end latency (31% below STREAMLINE) at high workload due to the processing of fewer data elements under backpressure. CPU utilization and cost rose with workload in STREAMLINE. It used a single type-1 resource for low workloads and added a type-2 resource for higher workloads. Competitors incurred similar costs across workloads due to comparable resources used. CT improved efficiency at high workload, as the nine-operator YSB data pipeline fully utilized resources despite increased parallelism. On average, STREAMLINE achieved 225% higher CPU utilization and 67% lower cost than competitors.

## 5.2. Ensembles of concurrent data pipelines

This experiment evaluated STREAMLINE's performance under resource contention and analyzed the impact of data pipeline ensembles. The number of concurrently running data pipelines was incrementally increased from 5 to 30 in steps of 5. The experiment began with five distinct data pipelines, *ETL*, *STATS*, *PRED*, *TRAIN*, and *YSB*, and added more in each step. *ETL*, *STATS*, and *PRED* used the *TAXI* dataset to capture a broad spectrum of workload fluctuations, while *YSB* maintained a medium workload to reflect realistic conditions. Fig. 10 presents the total throughput and cost across all data pipelines in each ensemble, along with average end-to-end latency and CPU utilization. Fig. 11 shows the normalized throughput and end-to-end latency for an ensemble of 25 data pipelines. *CT* and *STREAMLINE* were re-evaluated midway through execution, potentially triggering configuration parameter or scheduling updates.

With 5 and 10 data pipelines running simultaneously, STREAMLINE, *CT*, and *FMAX* achieved comparable throughput by efficiently handling the workload. In contrast, *FD* underperformed due to limited parallelism, particularly for more demanding data pipelines like *YSB*, *ETL*, and *STATS*, resulting in 46% lower throughput. *FMAX* maximized parallelism per operator without considering global resource constraints. As a result, it exhausted all available task manager slots after deploying only 8 data pipelines. *CT* and *FD* faced similar limitations, each managing a maximum of 14 data pipelines. In contrast, STREAMLINE scaled efficiently and successfully ran all 25 data pipelines. At 25 data pipelines, STREAMLINE achieved 50%, 174%, and 126% higher throughput
<figure><img src="image_12.png" /><figcaption>Fig. 10. Total throughput and monetary cost, and average end-to-end latency and CPU utilization, for 5 to 30 concurrently running data pipelines.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 11. Normalized throughput and end-to-end latency values for an ensemble of 25 data pipelines.</figcaption></figure>

than CT, FD, and FMAX, respectively. Fig. 11 further shows that STREAMLINE consistently reached the highest throughput across all data pipelines. FMAX often recorded the lowest throughput, as it executed only 8 data pipelines. For YSB, FD struggled most due to insufficient parallelism. Throughput for TRAIN remained similar across all methods, as it processed only a single data element.

STREAMLINE consistently achieved the lowest end-to-end latency, followed by FMAX and FD, while CT showed the highest values. This was primarily due to the STATS data pipeline, where CT failed to proactively adjust parallelism. At 10 data pipelines, CT's end-to-end latency peaked due to the inclusion of two STATS instances, which significantly impacted the overall average. Fig. 11 further showed that STREAMLINE outperformed all competitors in ETL, STATS, PRED, and TRAIN. For YSB, however, FMAX achieved the lowest end-to-end latency, followed by FD, as both executed fewer data pipelines, reducing contention. In contrast, STREAMLINE ran 25 data pipelines concurrently, leading to higher resource sharing and occasional CPU contention, which slightly increased processing time per data element.

STREAMLINE also achieved 335% improved average CPU utilization compared to competitors by efficiently scheduling operators on shared resources. FMAX showed the lowest utilization due to excessive parallelism, which led to wasted resources. CT achieved the highest utilization among the baselines by selecting only necessary parallelism, avoiding over-provisioning. As the number of data pipelines increased, costs rose for all methods. FMAX reached peak cost at 8 data pipelines after exhausting all available slots. CT and FD peaked at 14 data pipelines, with CT slightly reducing costs through runtime updates. In contrast, STREAMLINE scaled cost gradually with each data pipeline set. On average, it reduced costs by 64% compared to competitors.

### 5.3. Ensembles of data pipelines with varying start times

This experiment evaluated an ensemble of 25 data pipelines with staggered start times across five data pipeline types over 30 min. Data pipeline start delays followed a Poisson distribution, commonly used to model inter-arrival times in networks and scheduling systems [45]. CT and STREAMLINE were re-evaluated periodically every five minutes. Fig. 12 shows total throughput, average end-to-end latency, accumulated cost, and average CPU utilization over time.

Throughput increased notably at the start of YSB data pipelines (around minutes 1, 7, 19, 20, and 24), which delivered the highest throughput rates. FD's limited parallelism could not handle YSB workloads, leading to persistent throughput degradation. Up to 20 min, STREAMLINE, CT, and FMAX showed similar throughput. However, CT saturated the cluster after starting 12 data pipelines, FMAX managed only 7, and FD completed 13. STREAMLINE was the only system to execute all 25 data pipelines, achieving 45%, 50%, and 139% higher average throughput than CT, FMAX, and FD, respectively. STREAMLINE consistently achieved the lowest end-to-end latency, improving over CT, FMAX, and FD by 34%, 7%, and 82%, respectively. FD suffered the worst end-to-end latency due to queuing delays caused by low parallelization. CT experienced rising end-to-end latency after 17 min as the STATS data pipeline failed to keep pace with incoming data, but recovered after an update at minute 20 increased operator parallelism.
<figure><img src="image_14.png" /><figcaption>Fig. 12. Ensemble of 25 data pipelines with varying start times (dashed lines) and periodic update intervals (solid lines).</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Fig. 13. Evaluation of several combinations of objective weights, representing (from left to right) throughput, end-to-end latency, and CPU time.</figcaption></figure>

STREAMLINE achieved 248% higher average CPU utilization than competitors by efficiently locating operators on shared resources. At minute 15, utilization dropped below $\alpha$ = 20%, triggering a full update. STREAMLINE then restarted all data pipelines with updated configurations, restoring utilization to around 60%. Fluctuations in CPU usage reflected varying workloads and resource demands over time. FMAX incurred the highest cost early on due to aggressive parallelism exhausting cluster capacity. CT showed a gradual cost rise, with brief reductions during updates. FD displayed a steady cost increase. In contrast, STREAMLINE maintained lower overall costs through efficient resource use. Temporary cost spikes occurred during STREAMLINE's update phases, where data pipelines were restarted using seamless, state-preserving mechanisms.

## 5.4. Analysis of different objective weights

This experiment evaluated the impact of different objective weights $\beta$ in STREAMLINE's operator configuration parameter auto-tuning (Section 4.2). As shown in Fig. 13, prioritizing throughput or end-to-end latency ($\beta$ = [0.5, 0.25, 0.25], [1, 0, 0], or [0, 1, 0]) yielded the highest throughput across the 25-data-pipeline ensemble. End-to-end latency-focused optimization ($\beta$ = [0, 1, 0]) also improved throughput by increasing parallelism for concurrent processing but raised cost by roughly 15%. In contrast, cost-focused optimization ($\beta$ = [0, 0, 1]) minimized spending by reducing parallelism and increasing buffer sizes, which lowered CPU usage but significantly degraded throughput and end-to-end latency. The configuration $\beta$ = [0.5, 0.25, 0.25] offered the best overall tradeoff. While throughput was prioritized, this setting incurred only a 4% end-to-end latency increase compared to end-to-end latency-optimized setups and reduced cost by 7%. By comparison, the balanced setting $\beta$ = [0.33, 0.33, 0.33] achieved slightly lower cost but at the expense of throughput, end-to-end latency, and CPU utilization.

## 5.5. Analysis of data pipeline updates

This experiment evaluated the impact of updating the configuration parameters and operator schedules of a running data pipeline. We used a custom data pipeline, *StatefulCounter*, which incremented a stateful counter for each incoming data element, allowing us to verify state preservation, i.e., the counter should reflect the total number of processed elements. Data elements were streamed continuously at a rate of 10 el/s, with a configuration update applied approximately every 1000 elements. As shown in Fig. 14, both the stateless parallel and seamless stateful update approaches exhibited similar maximum processing delays, defined as the longest time taken to process any element. However, the seamless stateful method introduced 7% more delay due to the synchronization overhead needed to guarantee exactly-once processing. In contrast, it reduced delay by 93% compared to the stateful stop-and-restart method, which suffered from downtime during updates, delaying messages that arrived shortly after each restart. Regarding state accuracy, both stateful approaches preserved the counter's state without error. In contrast, the stateless parallel approach failed to maintain state, causing cumulative errors with each update. Without savepoints, each new instance resets the counter. After five updates, the counter read 997 instead of the expected 5000, resulting in an 80% error. Overall, the seamless stateful update achieved the best trade-off. It preserved application state and minimized processing delay, albeit at the cost of a 13% increase in CPU utilization when running data pipelines in parallel.
<figure><img src="image_16.png" /><figcaption>Fig. 14. Comparison of maximum processing delays and state errors (x) across three scenarios after several updates.</figcaption></figure>

# 6. Conclusions

In this work, we presented STREAMLINE, a dynamic, multi-layer auto-tuning framework that enables efficient execution of data pipeline ensembles under fluctuating workloads. STREAMLINE uses a transformer-based workload predictor to anticipate upcoming workload shifts, enabling proactive auto-tuning of configuration parameters and operator schedules. STREAMLINE continuously optimizes throughput, end-to-end latency, CPU utilization, and monetary cost by dynamically adapting to predicted workload changes. Unlike prior work that tunes configuration parameters in isolation, STREAMLINE automatically tunes parallelism and buffer size using a multi-objective evolutionary algorithm (NSGA-II) in combination with a surrogate model to identify Pareto-optimal trade-offs. User-defined weights guide the selection of final configuration parameters. To further improve efficiency, a resource-aware scheduler assigns operators to compute nodes, reducing cost and improving CPU utilization. At runtime, STREAMLINE applies new configuration parameters and schedules with minimal downtime, preserving data pipeline state and data correctness. We evaluated STREAMLINE on the Grid'5000 testbed using five real-world data pipelines. Compared to state-of-the-art approaches, STREAMLINE achieved up to 10x lower end-to-end latency, 9x better CPU utilization, 10x lower cost, and 4x higher throughput. These improvements largely stem from STREAMLINE's proactive tuning of configuration parameters to fluctuating workloads. It increases parallelism under moderate to high workloads to reduce end-to-end latency by accelerating data processing and minimizing wait times. Conversely, it avoids unnecessary scaling during low workloads, where synchronization overhead outweighs parallelism benefits. To maintain stable performance, STREAMLINE limits frequent parallelism changes, which can otherwise degrade throughput and increase end-to-end latency, especially for operators with low parallelism. Additionally, buffer sizes are dynamically adjusted based on workload. Smaller buffers reduce end-to-end latency under light workload, while larger buffers sustain throughput under heavy workload. These characteristics make STREAMLINE well-suited for dynamic IoT domains where workload variability demands efficient and adaptive tuning, such as smart cities, V2X systems, and healthcare IoT. Future work will focus on enhancing the surrogate model to handle unseen configurations and adapt to changing resource constraints dynamically.

## CRediT authorship contribution statement

**Stefan Pedratscher:** Writing – review & editing, Writing – original draft, Visualization, Validation, Software, Resources, Project administration, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. **Zahra Najafabadi Samani:** Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Methodology, Conceptualization.

**Thomas Fahringer:** Writing – review & editing, Writing – original draft, Validation, Supervision, Resources, Project administration, Funding acquisition, Conceptualization. **Marlon Etheredge:** Writing – review & editing, Visualization, Validation. **Abolfazl Younesi:** Writing – review & editing, Visualization, Validation.

**Juan Jose Durillo Barrionuevo:** Data curation, Conceptualization. **Peter Thoman:** Writing – review & editing, Visualization, Validation, Supervision.

## Declaration of Generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the author(s) used ChatGPT in order to improve the readability and language of the manuscript. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the published article.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
## Acknowledgments

This article is an output of a project supported by the Recovery and Resilience Plan of the Slovak Republic under the call 'Transformation and Innovation Consortia' (project code: 09I02-03-V01-00012). The authors contribute to this research as part of a consortium coordinated by InterWay, a. s.

Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).

## Data availability

https://github.com/UIBK-DPS-DC/Streamline.

## References

[1] H. Röger, R. Mayer, A comprehensive survey on parallelization and elasticity in stream processing, ACM Comput. Surv. 52 (2) (2019) 1–37.

[2] A.S.S.C.Y. Simmhan, RIoTBench: An IoT benchmark for distributed stream processing systems.

[3] Y. Guo, H. Shan, S. Huang, K. Hwang, J. Fan, Z. Yu, Gml: efficiently auto-tuning flink's configurations via guided machine learning, IEEE Trans. Parallel Distrib. Syst. 32 (12) (2021) 2921–2935.

[4] M.I. Gordon, W. Thies, S. Amarasinghe, Exploiting coarse-grained task, data, and pipeline parallelism in stream programs, ACM SIGPLAN Not. 41 (11) (2006) 151–162.

[5] V. Cardellini, F. Lo Presti, M. Nardelli, G.R. Russo, Runtime adaptation of data stream processing systems: The state of the art, ACM Comput. Surv. 54 (11s) (2022) 1–36.

[6] V. Kalavri, J. Liagouris, M. Hoffmann, D. Dimitrova, M. Forshaw, T. Roscoe, Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows, in: 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), 2018, pp. 783–798.

[7] B. Lohrmann, D. Warneke, O. Kao, Nephele streaming: stream processing under QoS constraints at scale, Clust. Comput. 17 (2014) 61–78.

[8] P. Agnihotri, B. Koldehofe, P. Stiegele, R. Heinrich, C. Binnig, M. Luthra, ZeroTune: Learned zero-shot cost models for parallelism tuning in stream processing, in: ICDE 2024, 2024.

[9] L. Amini, N. Jain, A. Sehgal, J. Silber, O. Verscheure, Adaptive control of extreme-scale stream processing systems, in: 26th IEEE International Conference on Distributed Computing Systems, ICDCS'06, IEEE, 2006, p. 71.

[10] D. Warneke, O. Kao, Exploiting dynamic resource allocation for efficient parallel data processing in the cloud, IEEE Trans. Parallel Distrib. Syst. 22 (6) (2011) 985–997.

[11] P.S. Wilmanns, S.J. Geuns, J.P. Hausmans, M.J. Bekooij, Buffer sizing to reduce interference and increase throughput of real-time stream processing applications, in: 2015 IEEE 18th International Symposium on Real-Time Distributed Computing, 2015, pp. 9–18, http://dx.doi.org/10.1109/ISORC.2015.14.

[12] D. Cheng, X. Zhou, Y. Wang, C. Jiang, Adaptive scheduling parallel jobs with dynamic batching in spark streaming, IEEE Trans. Parallel Distrib. Syst. 29 (12) (2018) 2672–2685.

[13] O. Farhat, K. Daudjee, L. Querzoni, Klink: Progress-aware scheduling for streaming data systems, in: Proceedings of the 2021 International Conference on Management of Data, 2021, pp. 485–498.

[14] D. Sun, H. He, H. Yan, S. Gao, X. Liu, X. Zheng, Lr-Stream: Using latency and resource aware scheduling to improve latency and throughput for streaming applications, Future Gener. Comput. Syst. 114 (2021) 243–258.

[15] H. Ji, G. Wu, Y. Zhao, Y. Yuan, G. Wang, Multi-job merging framework and scheduling optimization for apache flink, in: Database Systems for Advanced Applications: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 11–14, 2021, Proceedings, Part I 26, Springer, 2021, pp. 20–36.

[16] G. Janßen, I. Verbitskiy, T. Renner, L. Thamsen, Scheduling stream processing tasks on geo-distributed heterogeneous resources, in: 2018 IEEE International Conference on Big Data (Big Data), IEEE, 2018, pp. 5159–5164.

[17] J. Lian, X. Zhang, Y. Shao, Z. Pu, Q. Xiang, Y. Li, B. Cui, ContTune: Continuous tuning by conservative Bayesian optimization for distributed stream data processing systems, Proc. VLDB Endow. 16 (2023) 4282–4295.

[18] Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, L. Sun, Transformers in time series: A survey, 2022, arXiv preprint arXiv:2202.07125.

[19] A. Zeng, M. Chen, L. Zhang, Q. Xu, Are transformers effective for time series forecasting? in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 11121–11128.

[20] K. Deb, A. Pratap, S. Agarwal, T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Trans. Evol. Comput. 6 (2) (2002) 182–197, http://dx.doi.org/10.1109/4235.996017.

[21] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, K. Tzoumas, Apache flink: Stream and batch processing in a single engine, Bull. Tech. Comm. Data Eng. 38 (4) (2015).

[22] M. Zaharia, R.S. Xin, P. Wendell, T. Das, M. Armbrust, A. Dave, X. Meng, J. Rosen, S. Venkataraman, M.J. Franklin, et al., Apache spark: a unified engine for big data processing, Commun. ACM 59 (11) (2016) 56–65.

[23] A. Toshniwal, S. Taneja, A. Shukla, K. Ramasamy, J.M. Patel, S. Kulkarni, J. Jackson, K. Gade, M. Fu, J. Donham, et al., Storm@ twitter, in: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, 2014, pp. 147–156.

[24] Y. Liu, H. Xu, W.C. Lau, Online resource optimization for elastic stream processing with regret guarantee, in: Proceedings of the 51st International Conference on Parallel Processing, ICPP '22, Association for Computing Machinery, New York, NY, USA, ISBN: 9781450397339, 2023, http://dx.doi.org/10.1145/3545008.3545063.

[25] L. Eskandari, J. Mair, Z. Huang, D. Eyers, I-Scheduler: Iterative scheduling for distributed stream processing systems, Future Gener. Comput. Syst. 117 (2021) 219–233.

[26] H. Li, H. Fang, H. Dai, T. Zhou, W. Shi, J. Wang, C. Xu, A cost-efficient scheduling algorithm for streaming processing applications on cloud, Clust. Comput. (2022) 1–23.

[27] M. Barika, S. Garg, A.Y. Zomaya, R. Ranjan, Online scheduling technique to handle data velocity changes in stream workflows, IEEE Trans. Parallel Distrib. Syst. 32 (8) (2021) 2115–2130.

[28] X. Tang, W. Cao, H. Tang, T. Deng, J. Mei, Y. Liu, C. Shi, M. Xia, Z. Zeng, Cost-efficient workflow scheduling algorithm for applications with deadline constraint on heterogeneous clouds, IEEE Trans. Parallel Distrib. Syst. 33 (9) (2021) 2079–2092.

[29] A. Brown, S. Garg, J. Montgomery, K. Ujjwal, Resource scheduling and provisioning for processing of dynamic stream workflows under latency constraints, Future Gener. Comput. Syst. 131 (2022) 166–182.

[30] S. Maroulis, N. Zacheilas, V. Kalogeraki, A holistic energy-efficient real-time scheduler for mixed stream and batch processing workloads, IEEE Trans. Parallel Distrib. Syst. 30 (12) (2019) 2624–2635.
[31] S. Wang, Z. Ding, C. Jiang, Elastic scheduling for microservice applications in clouds, IEEE Trans. Parallel Distrib. Syst. 32 (1) (2020) 98-115.

[32] M. Barika, S. Garg, R. Ranjan, Cost effective stream workflow scheduling to handle application structural changes, Future Gener. Comput. Syst. 112 (2020) 348-361.

[33] M. Masdari, A. Khoshnevis, A survey and classification of the workload forecasting methods in cloud computing, Clust. Comput. 23 (4) (2020) 2399-2424.

[34] D. Saxena, J. Kumar, A.K. Singh, S. Schmid, Performance analysis of machine learning centered workload prediction models for cloud, IEEE Trans. Parallel Distrib. Syst. 34 (4) (2023) 1313-1330, http://dx.doi.org/10.1109/TPDS.2023.3240567.

[35] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput. 9 (8) (1997) 1735-1780.

[36] R. Pascanu, T. Mikolov, Y. Bengio, On the difficulty of training recurrent neural networks, in: International Conference on Machine Learning, Pmlr, 2013, pp. 1310-1318.

[37] S. Siami-Namini, N. Tavakoli, A.S. Namin, A comparison of ARIMA and LSTM in forecasting time series, in: 2018 17th IEEE International Conference on Machine Learning and Applications, ICMLA, Ieee, 2018, pp. 1394-1401.

[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Adv. Neural Inf. Process. Syst. 30 (2017).

[39] R.J. Hyndman, G. Athanasopoulos, Forecasting: Principles and Practice, OTexts, 2018.

[40] S. Pedratscher, T. Fahringer, J. Aznar-Poveda, Dynamic workflow scheduling in the edge-cloud continuum: Optimizing runtimes under budget constraints, in: 2024 IEEE 17th International Conference on Cloud Computing, CLOUD, IEEE, 2024, pp. 69-80.

[41] D. Balouek, A. Carpen Amarie, G. Charrier, F. Desprez, E. Jeannot, E. Jeanvoine, A. Lèbre, D. Margery, N. Niclausse, L. Nussbaum, O. Richard, C. Pérez, F. Quesnel, C. Rohr, L. Sarzyniec, Adding virtualization capabilities to the Grid'5000 testbed, in: I.I. Ivanov, M. van Sinderen, F. Leymann, T. Shan (Eds.), Cloud Computing and Services Science, in: Communications in Computer and Information Science, Vol. 367, Springer International Publishing, ISBN: 978-3-319-04518-4, 2013, pp. 3-20, http://dx.doi.org/10.1007/978-3-319-04519-1_1.

[42] S. Chintapalli, D. Dagit, B. Evans, R. Farivar, T. Graves, M. Holderbaugh, Z. Liu, K. Nusbaum, K. Patil, B.J. Peng, et al., Benchmarking streaming computation engines: Storm, flink and spark streaming, in: 2016 IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW, IEEE, 2016, pp. 1789-1792.

[43] A. Flink, Apache flink - configuration common options, 2025, https://nightlies.apache.org/flink/flink-docs-release-1.9/ops/config.html. (Accessed 2025).

[44] Oracle Corporation, ThreadMXBean API Documentation, 2025, URL https://docs.oracle.com/en/java/javase/. (Accessed 26 February 2025).

[45] H.C. Tijms, A First Course in Stochastic Models, John Wiley and sons, 2003.