Crucible: Quantifying the Potential of Control Algorithms through LLM Agents
Authors: Lianchen Jia
Chaoyang Li
Qian Houde
Tianchi Huang
Jiangchuan Liu
Lifeng Sun
Venue: NeurIPS 2025 poster ()
Keywords: Control Algorithms
Large Language Models
Agent
Potential
Primary area: applications
TL;DR: Quantifying the Potential of Control Algorithms through LLM Agents

Abstract: Control algorithms in production environments typically require domain experts to tune their parameters and logic for specific scenarios. However, existing research predominantly focuses on algorithmic performance under ideal or default configurations, overlooking the critical aspect of Tuning Potential. To bridge this gap, we introduce \texttt{Crucible}, an agent that employs an LLM-driven, multi-level expert simulation to turn algorithms and defines a formalized metric to quantitatively evaluate their Tuning Potential. We demonstrate \texttt{Crucible}'s effectiveness across a wide spectrum of case studies, from classic control tasks to complex computer systems, and validate its findings in a real-world deployment. Our experimental results reveal that \texttt{Crucible} systematically quantifies the tunable space across different algorithms. Furthermore, \texttt{Crucible} provides a new dimension for algorithm analysis and design, which ultimately leads to performance improvements. Our code is available at https://github.com/thu-media/Crucible.

Crucible: Quantifying the Potential of Control Algorithms through LLM Agents
Lianchen Jia', Chaoyang Li Houde Qian', Tianchi Huang', Jiangchuan Liu?, Lifeng Sun S.n 1.3+
DDepartment of Computer Science and Technology, Tsinghua University, 2Sion Fraser University, 3BNRist
j1c21 @mails. taingguu edu.cn, cn, sunlf@tsinghua edu.
Abstract
Control algorithms in production environments typically require domain experts to tune their parameters and logic for specific scenarios. However, existing research predominantly focuses on algorithmic performance under ideal or default config- urations, overlooking the critical aspect of Tuning Potential. To bridge this gap, we introduce Crucible, an agent that employs an L.M-rriven, multi-level expert simulation turn algorithms and defines a formalized metric quantitatively evaluate their Tuning Potential. We demonstrate Crucible's effectiveness across wide spectrum of case studies, from classic control tasks complex computer systems, and validate its findings in a real-world deployment. Our experimental results reveal that Crucible systematically quantifies the tunable space across different algorithms. Furthermore, Crucible provides a new dimension for algo- rithm analysis and design, which ultimately leads performance improvements. Our code is available at httpss//gitubhcco/mmmmmiiaaaaai
Introduction
Control algorithms are the core mechanisms that dynamically regulate system behavior to achieve specific objectives, with widespread applications from industrial automation to complex computer systems [ 2) These algorithms play pivotal roles various contexts, including gait control in
robotic systems 3 motion control for autonomous vehicles (4) adaptive bitrate (ABR) control [5|8
and congestion control
? in network applications, and scheduling control within data centers [10]
Such control algorithms are essential for ensuring system stability while simultancously optimizing overall performance.
However, current research predominantly evaluates algorithms based on their performance under ideal conditions or with default parameters [T] [2]. This paradigm overlooks a critical reality of production environments: algorithms always tuned by domain experts adapt to specific scenarios. The performance an algorithm in the wild therefore not just a function its default design, but also of its inherent adaptability- property define its Tuning Potential. The lack of a systematic way to measure Tuning Potential makes difficult to compare the practical adaptability of different algorithms and hinders its establishment
explicit design goal.
Evaluating Tuning Potential presents significant challenge that extends beyond conventional pa- rameter sensitivity analysis [13.. The assessment must encompass deeper, logic-level modifica- tions, including the incorporation of additional control branches or the integration of novel com- ponents interventions commonly employed by domain experts. The efficacy of such structural adjustments is linked to expert subjective understanding of the underlying algorithmic logic [14)
Cooresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

This complex interplay between objective performance metrics and subjective understanding factors impedes the establishment of Tuning Potential evaluation framework.
To address this gap, we introduce Crucible, the first framework designed to quantitatively evaluate the Tuning Potential of control algorithms. Crucible operates two key principles. First, it employs a Large Language Model (LLM)-driven. multi-level expert simulation agent that is defined with varying capabilities, such as the ability utilize optimization tools or perform multi-step reflection, thereby emulating how developers with different expertise levels approach algorithm tuning. This approach cireumvents the prohibitive costs associated with large-scale subjective studies [5]. Second, it establishes a formalized metric to normalize potential scores across diverse environments. This metric characterizes each task environment by analyzing the performance profile that emerges when a set of probe algorithms is applied to it, thereby enabling consistent comparisons across different domains.
To validate the effectiveness and generalizability of the Crucible framework, our evaluation en- compasses a wide spectrum of case studies, from classic control tasks (Cart-Pole (16)) to complex computer systems (ABR and scheduling control), and we validate our findings in real-world de- ployment. Our experiments demonstrate that Crucible consistently identifies larger optimization space than traditional Bayesian methods. By analyzing the results, we identify that an algorithm's 's representational capacity and comprehensibility are two primary factors influencing its potential. Finally, we show that these insights can guide targeted algorithm redesign, leading to significant performance improvements.
Our key contributions are summarized as follows:
We identify and formalize Tuning Potential as a critical, yet overlooked, dimension in algorithm evaluation. We argue that neglecting this dimension limits the practical impact of algorithms and hinders its adoption core design objective (Section 2
We propose Crucible ble, the first system designed to quantify the Tuning Potential of control algorithms. Through an LLM-based multi-level expert simulation agent and a formalized en- vironmental metric, provides systematic, quantifiable standard for measuring algorithmic adaptability (Section
We validate Crucible's ''s effectiveness and generalizability across a diverse range of scenar- ios, from classic control tasks to complex computer systems, including weal-world validation. Our results show that Crucible quantitative analysis offers clear guidance for algorithm design, ultimately enhancing both performance limits and practical value (Section
2
Motivation
2.1 LLM-Based Human Behavior Simulation
LLMs, trained on internet-scale corpora, have demonstrated exceptional performance not only in traditional natural language processing tasks such as translation, summarization, and question answer- ing [17] 18), but also exhibited emergent general knowledge and analytical reasoning capabilities in accordance with scaling laws [19 20
LLM advancements have prompted researchers to explore the application of LLMS in human behavior simulation. In social sciences, numerous studies (21)23 have investigated LLM-based user behavior simulation and subjective perception assessment, successfully generating credible individual behavioral patterns and their resulting emergent social behaviors within groups. In the field of recommendation systems, researchers (24 have effectively utilized LLMs to simulate diverse user behaviors, including browsing, searching, and content consumption activities. In our research, we apply LLMs to simulate developers' understanding and adjustment processes of algorithms, circumventing the high costs and time expenditures associated with personnel training in traditional large-scale subjective studies, thereby providing an efficient and scalable new method for algorithm evaluation.

---

2.0
1.0
Bitrate (mbps)
1.8
Better
0.8
1.6
0.6
better
1.4
1.2
BBA MPC
HYB Pitree
Hra(B)
0.2
Pensieve: -0.23
0.85
Fitre:: 0.22
0.0
HYB: -2 38
HYB(CI:: 0.91
15.0
12.5
10.0
7.5
5.0
2.5
0.0
-1.0
0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Time Spent on Stall (%6)
QoE
(a) ABR: Bitrate and Stall in Puffer
(b) ABR: CDF of QoE in Puffer
1.0
1.0
Exenuttors
Better
26,th
28.4%
27.3%
29.2%
29.2%
29.0%
0.8
Executer Usage Ratio
0.8
0.6
0.4
72.3%
Dynamic Partition, avgi6
Round Robin, avg.6.67
0.2
avg
avgré FIROIC), aug 6.53
0.2
Multi feed, avg
0.0
0.0
5jF
RFFIC)
Boand Robin
Dyr.micc Portidoon
FiFo
Mulli Feed
10
15
20
25
30
Cumulative Waiting Time(S)
(c) Scheduling Control: Executor utilization Ratio in (d) Scheduling Control: Cumulative Waiting Time in
2G Input Sizes
2G Input Sizes
Figure The Importance Algorithm Potential in the Real World
2.2 Control Algorithms in the Real World
We selected two representative examples from the field of computer systems ABR control in network transmission and scheduling control in distributed systems-t demonstrate the effectiveness of Crucible real-world control algorithms.
Adaptive Bitrate Control Adaptive bitrate algorithms enhance the quality of experience (QoE) by dynamically selecting the bitrate for the next playback chunk, aiming to improve playback quality while preventing stalling events [5]. Common control approaches include buffer-based algorithms
(BBA 27 BOLA
(28 hybrid methods combining bandwidth and buffer information (MPC 22
HYB 30 decision tree-based approaches (Pitree (31), and reinforcement learning (RL)-based
solutions such as Pensieve 32
Scheduling Control Distributed directed acyclic graph (DAG) task scheduling algorithms are re- sponsible for efficiently allocating and executing interdependent tasks in distributed computing environments, where dependencies are represented through DAGs [10]. These scheduling algorithms optimize task allocation mechanisms while considering resource utilization and task dependencies to enhance overall computational efficiency and reduce completion time. Common scheduling strategies include Shortest Job First (SJF), Shortest Remaining Time First (SRTF), Fair Scheduling. First- Seaa-In--Frrt-OO- First-Out (FIFO), Round Robin. Dynamic Partitioning 33 Multi-feed methods [34), and
Tetris
35
2.3 From Algorithm Design to Production Deployment
During the algorithm design phase, researchers typically focus theoretical performance and cross-scenario robustness [12] pursuing universal performance across wide range of application scenarios. However, when algorithms are actually deployed in production environments, we can obtain stable feature information specific to the application scenario, which enables targeted optimization. For instance, user bandwidth demands in network content provider services (36) and system loads in task scheduling scenarios 37 often exhibit stable and predictable patterns change. This stability and predictability of scenario characteristics allow developers to customize optimization strategies

---

Feedback
Domain
Integrated
Knowledge
LLM
Output
Potential
Optimization
Tools
Alg Log
Optimization
After LLM optimization
Congestion Control
Gait Control
Deployment, Application
Design
Auto Driving
Adaptive video streaming
Scheduling
Ideal env
Control Algorithm
Real World
Figure 2: Crucible System
according to specific application environments, thereby achieving performance that surpasses general- purpose algorithms in practical deployments.
2.4 Experimental Validation of the Importance of Potential
Figure demonstrates experimental results in the domains of ABR and scheduling (detailed experimental setup in Appendix [A). validating the importance of tunability, Our findings reveal: I) Algorithms based on simple logic, after appropriate tuning. significantly outperform complex designs. the ABR task, as shown in Figure T(a) HYB(B) optimized through Bayesian opti- mization 38 and HYB(C) adjusted via Crucible reduced video playback stalling time by 92%. Figure [(b) further indicates that these optimizations elevated the overall QoE ranking from the lowest position fourth and first place, respectively. Similarly, in the scheduling problem, the simple FIFO algorithm tuned by Crucibi ble achieves sub-optimal executor utilization as Figure [(c) and the optimal cumulative waiting time Figure [(d] Effective tuning encompasses optimization but also improvements in logical structure. In the ABR task, while parameter- parameter-optimized HYB(B) still could not surpass the QoE performance of BOLA and RobustMPC, HYB(C) with logical adjustments through Crucible ultimately achieves the best QoE performance, thoroughly demonstrating the importance of algorithm logic adjustment.
only parameter
Design
The overall design of Crucible can seen in Figure 2 We introduce the design of Crucible from three aspects: Section 3.1 describes the workflow of the Crucible agent, Section 3.2 provides a formalized definition of potential, and Section 3.3 explains the interaction between Crucible and the control algorithms being evaluated.
3.1 Agent Workflow
Domain Knowledge Acquisition We inject domain knowledge into LLMs through system prompts across three dimensions: task description, optimization objectives, and environment overview. The task description defines the basic information of the control task, including the input states and the scope of optional output behaviors; the optimization objectives specify the improvement direction and evaluation criteria for the control algorithm; the environment overview provides key characteristics and constraints the testing scenarios. These three dimensions construct comprehensive knowlede ledge framework, enabling LLMs to thoroughly understand the task context, optimization direction, and operational environmental characteristics.
Tool Utilization The potential of algorithms is primarily manifested in two key dimensions: first, representational capacity of the algorithm, referring to the breadth and granularity of its control space-al algorithsss with higher control dimensions and greater precision exhibit richer performance variability the hyperparameter space, thus possessing greater optimization potential [39 40 sec- ond, the comprchensibility 41 42 of the algorihm--algrritme with higher structural transparency

---

enable developers to implement targeted functional enhancements through logical reconstruction. The former can be systematically improved through mature automatic optimization techniques, while the latter relies on the abstract understanding capabilities of L.Mss. Therefore, we encapsulate opti- mization tools (such as Bayesian optimization)as standardized function interfaces to quantitatively evaluate the performance improvement of the current algorithmic logic within its hyperparameter space.
Action and Feedback Loop Drawing inspiration from optimization iteration patterns in industrial practice, we utilize historical adjustment records as an experiential foundation for subsequent opti- mization. Each algorithm modification is structurally preserved as triplet including modification rationale, specific action, and observed results. Before new round of optimization, these historical experiences are comprehensively presented to the LLM, enabling it to learn from previous attempts, avoid repeating mistakes, and simultaneously identify and replicate successful
Differential Developer Capability Simulation To model developers with varying expertise and available resources, we simulate different capability levels by adjusting the computational budget of the agent, rather than by crafting different prompts. We primarily restrict agent capabilities along two dimensions: first, limiting the number of Bayesian optimization calls available for fine- grained parameter tuning; second, constraining the number of reflection iterations for breaking through algorithmic logical boundaries through systematic trial and error. This differential simulation, grounded in resource consumption, enables us to more realistically evaluate the practical value of algorithms under various tuning conditions.
3.2 Formalization of Potential
We formalize an algorithm's 's potential, P, as its performance gain, weighted by unified environmental distance metric that is derived from the performance profiles of set of probe algorithms.
Performance Characteristic Vector: To quantify an environment's characteristics, we first define an evaluation set of environments T. We then select a small set of representative probe algorithms. For any environment F& E T, we the n probe algorithms obtain raw performance score vector [81 (Fx)...... 8 [. To eliminate dimensional effects, we normalize each score component across all environments in
norm(s((E)) (E{ - sjmin
sjmmax 8j,miu
where ma, = E (E)) and 8j.min minE,ET(B (Ex))) This process yields an n- dimensional normalized performance characteristic vector V(Ea) for each environment Ex. which serves as its quantitative fingerprint. The vector explicitly defined as:
V (Ex)= = [norm(s (Ex)), norm (s2((s)))) (.
(2)
Unified Environment Distance and Similarity. Using these characteristic vectors, the distance between two environments E, and E, is defined as the root mean square error (RMSE):
dis( E, Ee) =
(V(E)),-VEL
(3)
Here, V(Fa), denotes the j-th component of the vector V(F..). The corresponding environment similarity then defined as: sim Ei, Ex) = max(0,1 1 - dis(E,, E.)).
(4)
Tuning Potential Definition. With the environmental similarity metric now formally defined, we can present the complete definition of an algorithm's potential, P. It is the similarity average performance gain across all test environments. For a given algorithm, let Sr, and S., be its original and Crucible-tuned performance in test environment E,, and let E, be its ideal environment. The potential is calculated as:
Σ E.ET
Σ KS - x sim (in,E, Ex]]
P
(5)
This formulation ensures that performance gains in environments highly dissimilar to the ideal one weigtwe, yielding more robust and fair measure of an algorithm's 's intrinsic tunability.
5

---

3.3 Interaction Between Crucible and Control Algorithms
The Crucible framework supports multiple control tasks, thus designing standardized interaction interface. this interface, control algorithms provide two types of information the L.MM: the algorithm code itself and execution logs. The execution logs consist of series of triplets containing states, actions, and results. Based on this information, the LLM exclusively modifies the control algorithm and obtains new test results by invoking the original execution file. This unified interface effectively standardizes the interaction logic, enabling Crucible to flexibly support any type of control algorithm.
The overall interaction logic of the system is as follows: First, the system traverses all preset test environments and executes the LLM optimization cycle in each environment. In this cycle, the system compares the performance of the current algorithm with reference algorithms (which can be overfit learning-based algorithms or theoretically optimal algorithms) and collects cases with significant performance differences. Subsequently, the LLM provides algorithm optimization suggestions based on these difference information. The system implements these suggestions and optionally applies Bayesian optimization methods to further adjust algorithm parameters. When all environments have been traversed, the system enters the evaluation phase. For each algorithm under test, it first determines ideal environment (e.g., the one yielding the best performance), Then, using the unified whw.eeceeccaaatetitiiii--------- based metric defined in Section 3.2, it calculates the similarity between each test environment and the ideal environment. Finally, it computes the algorithm's 's potential by aggregating the monilarity-weigheee performance gains across all test environments. Appendix B provides a detailed pseudocode of the complete Crucible workflow.
4 Evaluation
We begin by outlining our experimental setup in Section 4.1 with complete configuration details deferred to Appendix A Our evaluation then unfolds in three progressive stages to systematically validate the Crucible framework.
First, in Section 4.2 we establish the framework's effectiveness and generalizability. We demon- traditional tuning, validate its performance in real-world deployment, and
strate its superiority confirm its robustness across different LLMs. Building on this, Section 4.3 shows how Crucible moves beyond mere performance enhancement to provide quantitative insights into the abstract concept of algorithmic potential, revealing the key factors that govern it. Finally, in Section 4.4] we illustrate the practical impact of these insights, showing how they guide targeted algorithm design and establish potential as valuable optimization target.
4.1 Experimental Setup
4.1.1 Crucible
This research mainly employs API calls to the Claude 3.7 Sonnet [44,, with Bayesian optimization serving a hyperparameter tuning tool. For simulating developers with varying optimization capabilities, we configure the Bayesian iteration count at three distinct levels (0, 10, and iterations) while setting the reflection iteration steps to 2, and 3, respectively. Our evaluation spans diverse range of control tasks to demonstrate Crucible's generalizability.
4.1.2
Case Studies
We select testbeds from three distinct domains to demonstrate Crucible's generalizability.
Classic Control. We use the "CartPole-vl" environment from Gym 45 a standard benchmark in control task. Our evaluation focuses on classic controllers such as (PID) 46 and Linear- Quaarrtte Regulator (IQQR) 47 comparing their optimized performance against based methods DQN [48].
Deriivatvee

---

Table I: Performance of classic control algorithms on Cart-Pole, tuned by Crucible.
Algorithm
Initial
After Bayes
After 1 LLM
After 2 Bayes
After 2 LLM
Bang bang
34
56
500
PID
34
77
110
271
500
LQR
161
500
DQN (Reference)
500
0.8
0.8-
0.6
0.8
0.6-
0
Oboe lava: 0.92%)
Oboe (ava: 1.03%)
Oboe (ava: 1.61%)
3G lavg: 5.34%)
0.4
Puffer (avg: 2.54%)
0.4
FCC (avg: (88%)
FCC (avg: 5899%)
3G (avg: 0.03%)
3G (aug: 5.09%)
Putfer 12vg: 44 10%)
FCC tavg: 23. 64%)
Puffer (avg: 20.18%)
10
40
20 Improvement Bayes
0.0
20 Improvement Bayes
Improverment over Dayes
(a) CDF When Bayes=0
(b) CDF When Bayes=10
(c) CDF When Bayes=20
Figure 3: Optimization Space between Crucible and Bayes
Computer Systems (ABR). We utilize a widely adopted adaptive bitrate (ABR) simulator 32 29
with four public network datasets: Oboe [30]. FCC 49], 3G (50), and Puffer (5T). The experiments evaluate prominent algorithms, including BBA, MPC, HYB, BOLA, and Pitree. using the Envivio
video trace [32] as standardized content. In addition simulated environments, we conduct validation
in real-workd setting using Dash.js (52]
Computer Systems (Scheduling). We employ Spark simulator 53 with TPC-H query tasks [54]
to evaluate classical and modern scheduling algorithms, including Shortest Job First (SJF), First-In- First-Out (FIFO), Multi-level feedback (MF), and Tetris (35].
4.2
Crucible's Effectiveness and Generalizability
4.2.1
Effectiveness: Expanding the Optimization Space
Unlike traditional optimization tools confined to hyperparameter tuning, Crucible implements modifications at the algorithmic logic level, expanding the optimization space. An illustration of this is found in the classic Cart-Pole control problem. As shown in Table simple heuristic algorithms like Bang-bang [66] and PID, which initially perform limited, can achieve the optimal score of 500, matching the performance of complex black-box algorithm like DQN. This improvement, particularly the jump from a score of 56 to 500 for the Bang-bang controller after a single LLM- driven logic modification, is direct result of altering the core control logic-a performance leap unattainable through mere parameter tuning.
This principle of logic-level enhancement is not an isolated phenomenon but general advantage that holds true more complex domains. Across our computer system benchmarks, Figure illustrates the relative improvement ratio achieved by Crucible's enhanced results S, compared to Bayesian optimization results Sp, expressed as (S. S,)/S,. The results demonstrate that Crucib sercible'' logic- level improvements consistently yield performance enhancements, achieving gains of up to 44. % on the Puffer dataset.
However, these logic-level modifications also introduce variability. Our experiments indicate that without the fine-tuning provided by Bayesian optimization, pure LLM--riven adjustments can be unreliable; approximately 60% of test scenarios show no significant performance gains, primarily at- tributable the LLM's limitations in fine- grained numerical operations (FigureBja), This highlights the critical synergy within our framework. As Bayesian optimization is incorporated, it not only improves the baseline but also effectively explores the new solution space opened up by the LLM's logic changes. Consequently, the proportion of ineffective test scenarios decreases from 60% to 20% (Figure confirming the powerful combination of logic-level exploration and leeel exploitation.
7

---

4.2.2 Real World Evaluation
We validate Crucible e in real-world ABR scenario using Dash.js over public WiFi network. We test five heuristic algorithms and the RL-based Pensieve as a reference. The results, presented in T demonstrate that Crucible successfully enhances the performance of heuristic algorithms this noisy, unpredictable environment. For example, the tuned HYB and BBA algorithms achieve QoE score of 1.72, outperforming their original versions and even surpassing the RL-based Pensieve (1.66). Conversely, the complex and less interpretable Pitree algorithm exhibits no improvement, reinforcing that Crucible' effectiveness correlates with algorithm's 's comprchensibility. These findings provide evidence that the benefits of Crucible-guided tuning transfer directly to practical, real-world deployments.
Table 2: QoE. of ABR in real-world Dash.js deployment before and after Crucible tuning.
QoE State
HYB
BBA
BOLA
Pitree
MPC
Pensieve (RL)
Original Tunee
1,40
1.56
1.20
1.73
1.72
1.66
1.72
1.72
1.54
1.73
1.79
4.2.3 Robustness Across Different LLMs
We evaluate the framework using three different models: Claude 3.7 Sonnet (our primary model), the previous generation Claude 3.5 Sonnet, and GPT-44-mini. As detailed in Table while the absolute final performance varies slightly across models, the overall conclusions remain consistent. All LLMs effectively tune the ABR algorithms, and the relative performance ranking among them remains largely stable. Claude 3.7 Sonnet achieves higher final score on the HYB algorithm, suggesting that more powerful models can unlock greater potential in certain cases, but other models also deliver significant improvements.
Table Final ABR performance after tuning with different LLMs. All runs were configured with iterations and 20 Bayesian iterations.
Algorithm
Initial
Claude 3.5 Sonnet
GPT-4o-mini
Claude 3.7 Sonnet
BBA
0.75
1.13
1.10
1.11
BOLA
1.02
1.07
1.08
1.06
HYB
0.92
1.03
1.04
1.12
Pitree
0.31
0.35
0.37
0.36
MPC
1.07
1.09
1.10
1.09
4.3 Potential Analysis
In Figure 4 we compare the percentage performance improvements of ABR and scheduling al- gorithms under different Crucible capability settings against their original scores. Regarding optimization tool usage constraints, Figures (aa through 4(c) demmonstrtteee a positive correlation between increased Bayesian optimization iterations and significant algorithm performance enhance- ments. Specifically, when the number of iterations increased from one two, notable performance lepp occurred- in ABR, the improvement rate with zero Bayesian iterations rose from 9.54% to 29.91 %, while in scheduling algorithms, the proportion of scenarios failing to achieve performance improvements decreased from 80% to 60%. Between the two dimensions examined, the ability of optimization tools to unlock algorithmic logic potential has more pronounced impact per- formance enhancement. Furthermore. comparing ABR and scheduling algorithms reveals that due to differences in input state complexity (ABR only involves buffer size and bandwidth, whereas scheduling encompasses complex DAG graph information and node states), LLMs face varying degrees of comprehension challenges, resulting in significantly lower improvement magnitudes for scheduling algorithms compared ABR algorithms.
We illustrate the detailed potential analysis results with ABR algorithms as an example in Table 4 Through our analysis, we derive two key findings: First, even among algorithms representing simple logic, the HYB algorithm utilizing broader state space demonstrates significantly greater potential

---

1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0
Dayes(0), 9.54%
0.4
Bayes(0), 29.91%
0.4
Bayes(O),
Bayes(10), avg: 45, 45.40%
Bayes(10), avg: 52.61%
Bayes|101, avg: 59: 21%
Bayes(20), 59,78%
Bayes(20), 60.37%
Bayes/20), awg: 67.85%
0.0
20
40
60
0.0
20
40
60
0.0g
20
40
60
80
Improvement over Init 196)
improvament over Init (%)
Improvement over Init (86
ABR, Iterate Times
(b) ABR, Iterate Times
ABR, Iterate Times
1.00
1.00
1.00
18
0.75
0.75
Modn=1.37
0.75
L.70
0 0.50
0 0.50
0.50
0.25
0.25
0.25
0.00
5
10
15
20
25
0.00
5
10
15
0.00
0.0
2.5
5.0
7.5
10.0
12.5
Improvement over (%)
Improvement over Init (50)
Improvement over Init 196
(d) Scheduling, Iterate 1 Times
(e) Scheduling, Iterate 2 Times
(f) Scheduling. Iterate Times
Figure 4: Improvement Different Ability
than the BBA algorithm using only a single state space, indicating that algorithmic representational capacity substantially influences potential enhancement: Second, as a decision tree algorithm distilled from deep reinforcement learning, Pitree exhibits lower optimization potential despite its initial limited performance, suggesting that the complex logical structure of decision trees may negatively impact potential enhancement. Therefore, researchers aspire to design ABR algorithms that improve comprchensibility, such as ComTree []4.
Table 4: Potential of ABR Algorithms
Alg
init
1 Iter
2 Iter
3 Iter
Impro
Potential
Ideal
HYB
0.92 + 0.64
0.99 0.61
1.03
0.57
1.02 + 0.60 0.10 + 0.18 0.068 + 0.117
FCC
Pitree
0.31 0.10
0.33 + 0.10
0.44 + 0.27
0.38 + 0.09 0.07 + 0.12 0.033 + 0.022
Puffer
BOLA
1.01 1
1.05
= 0.46
1.04 0.45
1.05
0.45 0.03 + 0.06 0.025 0.032
3G
BBA
0.75
1.04
= 0.59
0.98
4: 0.56
1.01
= 0.57 0.26
+ 0.38 0.018
0.008
Oboe
MPC
1.07 0.52 1.10 + 0.59
1.08
+ 0.53
1.09
+ 0.54
0.02 + 0.04
0.017 + 0.014
3G
4.4
From Potential Assessment to Algorithm Optimization
This section explores how to transform Crucible's potential assessments into effective algorithm optimization strategies to enhance ultimate performance.
4.4.1 Enhancing Algorithmic Representational Capacity
In the case of ABR algorithms, the results show that the BBA algorithm exhibits both lower opti- mization potential and inferior final performance compared to the HYB algorithm. This indicates inherent representational limitations in BBA's buffer-only control approach. Based on this insight, we improve BBA to create the BBA_ CC algorithm (detailed implementation in Appendix C Specific enhancements include: incorporating current bandwidth a bandwidth-basee control branch to select the highest bitrate that would not cause video stalling.
additional control input and introducing
As shown in Figure 5(a) the enhanced BBA C's initial performance closely resembles the original BBA algorithm, diftering by only 0.5%. From traditional evaluation perspective, these algorithms appear nearly equivalent. However, after optimization adjustments via Crucible, BBA C's enhanced representational capacity advantage becomes evident. During each optimization iteration, BBA_C consistently outperforms BBA, ultimately achieving 4% performance improvement. This case demonstrates how enhancing algorithmic representational capacity can increase optimization potential and ultimately improve performance.

---

1.0
1.0
0.8
0.8
Better
Better
0.6
0
MF11 avg:145.6
0.4
2480
BBA,C:
MF(2) avg:148.2
0.2
BBA(11): 1.57
BBA CILL): 1.93
SF12). 149:445.4
8BML2): 1,87
BBA_C112): (112) 1.95
MF13) 3vg:148.3
0.0
[BA_ 1.97
S/113),
avg:342.1
1.5
2.5
3.0
3.5
4.0
140
160
180
200
220
QoE
Final completion Lime(S)
(a) CDF of ABR QoE, Oboe, means iters
(b) CDF of Scheduling FCT, 10*2G input Sizes
Figure 5: CDF After Optimization
4.4.2 Simplifying Algorithm Logic
Another important dimension involves improving developers' comprehension capabilities and opti- mization space by simplifying algorithmic logic. Figure ((b) presents a typical case in task scheduling: initially, the Multilevel Feedback algorithm outperforms the SJF algorithm; however, after optimiza- tion adjustments through Crucible, the SJF algorithm achieves shorter task completion times than the supposedly more advanced Multilevel Feedback algorithm.
The fundamental cause of this phenomenon lies in the high-dimensional complexity of input states in DAG scheduling problems. This complexity makes it difficult for LLMs to comprehensively understand the internal logic of complex algorithms, thereby limiting their ability to add appropriate optimization logic complex algorithms. In contrast, LLMs can more effectively understand and optimize foundational algorithms with simple, clear logic. This finding emphasizes the important value of "simplicity" in algorithm design, particularly in scenarios leveraging Al-assisted optimization.
5 Limitations and Broader Impacts
Limitations As the first work exploring the potential of control algorithms, this research has two main limitations. First, the stability issue: since we use an LLM our foundation, different capabilities and versions of LLMS may influence the results. However, we believe this does not diminish the value of assessing algorithmic potential, as different I.MMs essentially simulate developers with varying skill levels, making the evaluation, selection, and design of algorithms still practically meaningful. Second, we currently cannot directly modify the internal logic of black-box algorithms; therefore, in this paper, we discuss and analyze decision trees distilled from black-box algorithms. Effectively understanding and adjusting the internal logic of black-box algorithms remains an open challenge, providing direction for future research.
Broader Impacts We hope our work can inspire rethinking of algorithm design, positioning potential as a new optimization direction or even as an optimization metric.
Conclusion
We introduced Crucible, framework that addresses the gap between algorithm design and practical deployment by quantitatively evaluating Tuning Potential. Crucible leverages an LLM agent to simulate developer behavior and introduces formalized potential metric quantify this untapped optimization space. Our extensive evaluations--spnnnnng spanning classic control tasks, complex computer systems, and real-world deployment- demonstrate that Crucible not only enhances algorithm per- formance beyond traditional methods but also establishes tuning potential as a valuable, quantifiable metric. This work advocates for shift algorithm design, treating Tuning Potential as a property to be evaluated from the outset, rather than as a post-deployment afterthought. This empowers designers build adaptable algorithms that maintain long-term value in evolving real-world environments.
10

---

Acknowledgments We thank the anonymous NeurIPS reviewers for their constructive feedback, which has significantly improved this work exploring new optimization directions. This research was supported by the Beijing National Research Center for Information Science and Technology under Grant BNR2023TTD03005-2 the NSERC Discovery Grant, and the Beijing Key Laboratory of Networked Multimedia.
References
[1] C. Doyle, A. Francis, and R. Tannenbaum, Feedback control theory. tion, 2013.
Courier Corpora-
[2] E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, "A survey of autonomous driving: Common practices and emerging technologies," IEEE access, vol. 8, pp. 58 443-54-5 469, 2020.
[3] G. Tzafestas, "Mobile robot control and navigation: global overview," Journal of Intelligent & Robotic Systems, vol. 91, no. 1, pp. 35-58, 2018.
[4] B. Paden, M. Čáp. S. Z. Yong, D. Yershov, and E. Frazzoli, "A survey of motion planning and control techniques for self-driving urban vehicles," IEEE Transactions on intelligent vehicles, vol. I, no. 1, pp. 33-55,22 2016.
[5] Y. Sani, A. Mauthe, and C. Edwards, "Adaptive bitrate selection: A survey," IEEE Communica- fions Surveys Tutorials, vol. 19, no. 4, pp. 2985-3014, 2017.
[6] L. Jia, C. Zhou, T. Huang, C. Li, and L. Sun, "Rdladder: Resolution-duration ladder for vbr-encoded videos via imitation learning," in JEEE INFOCOM 2023-IEEE Conference on Computer Communications. IEEE, 2023, 1-10.
[7] L. Jia, T. Huang, and I.. Sun, "Zixia: A reinforcement learning approach via adjusted rank- ing reward for internet congestion control," in ICC 2022-IEEE International Conference on Communications. IEEE, 2022, pp. 365-370.
[8] L. Jia, C. Zhou, C. Li, J. Liu, and L. Sun, "Towards user-level qoe: Large-scale practice in personalized optimization of adaptive video streaming," in Proceedings of the ACM SIGCOMM 2025 Conference, 2025, pp. 1154-1166.
[9] L. Jia, C. Zhou, T. Huang, Li, a.d Sun, "Meet challenges of rtt jitter, a hybrid internet congestion control algorithm," in Proceedings of the ACM Web Conference 2024, 2024, pp. 2768-2776.
[10] M. Kumar, S. Sharma, A. Goel, and P. Singh, "A comprehensive survey for scheduling techniques in cloud computing," Journal of Network and Computer Applications, vol. 143, pp. 1-33, 2019.
[11] D. E. Kirk, Optimal control theory: introduction.
Courier Corporation, 2004.
[12] U. Mackenroth, Robust control systems: theory and case studies. Media, 2013.
Springer Science & Business
[13] D. M. Hamby, "A review techniques for parameter sensitivity analysis of environmental models," Environmental monitoring and assessment, vol. 32, pp. 135-154, 1994.
[14] L. Jia, C. Li, Z. Yuan, J. Chen, T. Huang, J. Liu, and L. Sun, "Beyond interpretability: Exploring the comprehensibility of adaptive video streaming through large language models," arXiv preprint arXiv:2508. 164488 2025.
[15] A. Berenzweig, B. Logan, D. Ellis, and B. Whitman, "A large-scale evaluation of acoustic and subjective music-similarity measures," Computer Music Journal, pp. 63-76, 2004.
[16] R. S. Sutton, A. G. Barto et al., Reinforcement learning: introduction. MIT press Cambridge, 1998, vol. 1, no. 1.

---

[17] R. Nakano, J. Hilton, S. Balaji. J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders et al., "Webgpt: Browser-assisted weession-answeriin with human feedback," " arXiv preprint 122.9321 2021.
[18] Achiam, Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al,, "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023.
[19] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei, "Scaling laws for neural language models," arXiv preprint arXiv:2001.088 000301,00006111111 2020.
[20] W. Zhao, K. Zhou, J. Li, Tang, X. Wang, Hou, Min, B. Zhang. Zhang. 7. Dong et al., "A survey of large language models," arXiv preprint arXiv:2303. 18223, 2023.
[21] S. Park, J. O'Brien, C. J. Cai, M. Morris, P. Liang, and M. S. Bernstein, "Generative agents: Interactive simulacra of human behavior," in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023, pp. 1-22.
[22] G. V. Aher, R. I. Arriaga, and A. T. Kalai, "Using large language models to simulate multiple humans and replicate human subject studies," " in International Conference on Machine Learning. PMLR, 2023, pp. 337-371.
[23] Z. Hussain, M. Binz, R. Mata, and U. Wulff, "A tutorial on open-source large language models for behavioral science," Behavior Research Methods, vol. 56, no. 8, pp. 8214 8237, 2024.
[24] R. Ren, P. Qiu, Y. Qu, J. Liu, W. X. Zhao, H. Wu, J.-R. Wen, and H. Wang, "Bases: Large- scale web search user simulation with large language model based agents," arXiv preprint arXiv:2402. /7505, 2024.
[25] Z. Zhao, W. Fan, Li, Y. Liu, X. Mei, Y. Wang. Wen, F. Wang. X. Zhao, Tang et al. "Rec- ommender systems in the era of large language models (Ilms), arXiv preprint arXiv:2307.02046, 2023.
[26] X. Huang, J. Lian, Y. Lei, J. Yao, D. Lian, and X. Xie, "Recommender ai agent: Integrating large language models for interactive recommendations, arXiv preprint arXiv:2308.16500 2023.
[27] T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, and M. Watson, "A based approach to rate adaptation: Evidence from a large video streaming service," in Proceedings of the 2014 ACM conference on SIGCOMM, 2014, pp. 187-198.
[28] K. Spiteri, R. Urgaonkar, and R. K. Sitaraman, "Bola: Near-optimal bitrate adaptation for online videos." IEEE/ACM transactions on networking, vol. 28, no. 4, pp. 1698-1711, 2020.
[29] X. Yin, A. Jindal, V. Sekar, and B. Sinopoli, "A control-theoretic approach for dynamic adaptive video streaming over http," in Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communicution, 2015, pp. 325-338.
[30] Z. Akhtar, Y. S. Nam, R. Govindan, Rao, Chen, E. Katz-Bassett B. Ribeiro, Zhan, and H. Zhang, "Oboe: Auto-tuning video abr algorithms to network conditions." in Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, 2018, 8, pp. 44-58.
[31] Z. Meng, J. Chen, Y. Guo, C. Sun, H. Hu, and M. Xu, "Pitree: Practical implementation of abr algorithms using decision trees," in Proceedings of the 27th ACM International Conference on Multimedia, 2019, pp. 2431-2439.
[32] H. Mao, R. Netravali, and M. Alizadeh, "Neural adaptive video streaming with pensieve," in Proceedings of the Conference of the ACM Special Interest Group on Data Communication, 2017, pp. 197-210.
12

---

[33] M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay scheduling: simple technique for achieving locality and faimess cluster scheduling," in Proceedings of the 5th European conference on Computer systems, 2010, pp. 265-278.
[34] M. Isard, V. Prabhakaran, Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quiney: fair scheduling for distributed computing clusters," in Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, 2009, pp. 261-276.
[35] R. Grandl, G. Ananthanarayanan, S. Kandula, S. Rao, and A. Akella, "Multi-resource packing for cluster schedulers," ACM SIGCOMM Computer Communication Review, vol. 44, no. 4, pp. 455-466, 2014.
[36] R. Zhang, C. Yang, X. Wang. T. Huang, C. Wu, J. Liu, and L. Sun, "Practical cloud-edge scheduling for large-scale crowdsourced live streaming." IEEE Transactions on Parallel and Distributed Systems, vol. 34, no. 7, pp. 2055-2071, 2023.
[37] S. Shen, V. Van Beek, and A. Iosup, "Statistical characterization of business-criticch workloads hosted in cloud datacenters," 2015 15th IEEE/ACM international symposium on cluster, cloud and grid computing. IEEE, 2015, pp. 465-474
[38] J. Snoek, H. Larochelle, and R. P. Adams, "Practical bayesian optimization of machine learning algorithms," Advances in neural information processing systems, vol. 25, 2012.
[39] T. Yu and H. Zhu, "Hyper-paranee optimization: A review of algorithms and applications." arXiv preprint arXiv:2003.05689 2020.
[40] M. Črepinšek, S.-H. Liu, and M. Mernik, "Exploration and exploitation in evolutionary algo- rithms: A survey." ACM computing surveys (CSUR), vol. 45, no. 3, pp. 1-33, 2013.
[41] A. A. Freitas, "Comprehensible classification models: position paper," ACM SIGKDD explo- rations newsletter, vol. 15, no. 1, pp. 1-10, 2014.
[42] D. Martens, J. Vanthienen, W. Verbeke, and B. Baesens, "Performance of classification models from a user perspective," Decision Support Systems, vol. 51, no. 4, pp. 782-793, 2011.
[43] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou al., "Chain-of- thought prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 24 824-24 2022.
[44] Anthropic, "Claude 3.7 sonnet," httpss//www.aattrrppicoooodddaas 2025, accessed May 2025.
[45] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, "Openai gym," arXiv preprint 160iv: 2016.
[46] M. J. Willis, "Poppotiinall-itteaaaleeeiiviii control," Dept. of Chemical and Process Engi- neering University of Neweastle, vol. 6, 1999.
[47] A. Bemporad, M. Morari, V. Dua, and E. N. Pistikopoulos, "The explicit linear quadratic regulator for constrained systems," Automatica, vol. 38, no. 1, pp. 3-20, 2002.
[48] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., "Human-level control through deep rein- forcement learning," nature, vol. 518, no. 7540, pp. 529-533, 2015.
[49] M. F. B. Report, "Raw data measuring broadband america 2016," https://www.fc. wtwp//cc....o//eoor.....ooo www.aa/waaanaaeerraaaaaaaaaaaaaa 2016, 2016, [Online; accessed 19-July-2066.
[50] H. Riiser, P. Vigmostad, C. Griwodz, and P. Halvorsen, "Commute path bandwidth traces from 3g networks: Analysis and applications," in Proceedings of the 4th ACM Multimedia Systems Conference, 2013, pp. 114-118.
13

---

[51] F. Y. Yan, H. Ayers, C. Zhu, S. Fouladi, J. Hong, K. Zhang, P. A. Levis, and K. Winstein, "Learning in situ: a randomized experiment in video streaming." NSDI, vol. 20, 2020, pp. 495-511.
[52] DASH Industry Forum, "dash.js." https///dashjsorrg 2024, [Online; accessed 15-February- 2025]. [Online]. Available: https://dashjs.org
[53] H. Mao, M. Schwarzkopf, S. B. Venkatakrishnan, Z. Meng, and M. Alizadeh, "Learning scheduling algorithms for data processing clusters," in Proceedings of the ACM special interest group on data communication, 2019, pp. 270-288.
[54] Transaction Processing Performance Council, "Tpc-h: A decision support benchmark," https /www.tpc.org//pch/ 2025, accessed May 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
As shown in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations as Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our code is available in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
Our code is available in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com / wwwssnnnnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We introduce our experimental setup at Section 4.1 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: As shown in Table 4 displayed using CDFs.
report standard error, and the main results
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
As described in Section 4.1 we introduce the token API.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research fully complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Assets used in this paper comply with Licenses and Terms of Use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our code is available in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not perform crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We introduce the usage of LLMs as Section 3
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neuriie httppp/....//oooeeeeeee///////// for what should or should not be described.
21

---

Dataset
Traces
Avg. BW (Mbit/s)
BW Range (Mbit/s)
3G
142
1.52 + 0.72
[0.60. 4.59]
Oboe
428
2.77 + 1.32
[0.34, 5.70]
FCC
264
1.33 + 0.55
[0.19, 3.43]
Puffer
500
1.60 E 0.88
[0.30, 3.60]
Table 5: Details of Network Trace Datasets
Appendix
A Details Experimental Setup
A.1
ABR Experimental Setup
A.1.1
Network Traces
We summarize the characteristics of the four public network traces utilized in this study in Table including the number of traces, average bandwidth per trace, and bandwidth range.
A.1.2 Video Samples
We use the "EnvivoDash3" video from the "MPEG-DASH reference videos," consistent with previous works [32 22)]. The video is 193 seconds in length and segmented into 4-second chunks with bitrates of (300, 750, 1200, 1850, 2850, 4300) khps.
A.1.3 Player Configuration
We employ a classical VoD scenario player simulator as used in 32 29 quality of 750 kbps.
with an initial default video
A.1.4 Quality Experience (QoE) Metrics
For the video-on-demand scenario, we adopt the classical Qo as our QoE metric 32 model is defined as follows:
29 This
N
N-1 Σ ==1
QoEnin Eg(R.
Tx
((Rn41)-4he - ((R))
(6)
n=1
where represents the total number of video segments, R, denotes the bitrate of the n-th segment, and q( (R,,) is function mapping bitrate perceived user quality. T, represents the buffering time incurred while downloading the n-th segment, and is the weight coefficient for buffering time. The final term penalizes variations in video quality to ensure playback smoothness. Specifically, in implementation, we set y to 4.3.
A.2
Scheduling Experimental Setup
A.2.1
Spark Simulator
to evaluate the performance of our scheduling policies. This simulator is designed to faithfully emulate the behavior of a real Spark cluster by capturing several key real-world phenomena:
We utilize high-fidelity simulator 53
Initial Task Wave Effects: The first wave tasks in partieular stage often exhibits slower execution compared subsequent tasks. This slowdown arises from factors such as Spark's s pipelined task execution, just-in-time (JIT) compilation of task code, and initial overheads like establishing TCP connections between executors. The simulator accounts for this by
22

---

drawing the runtime of first-wave tasks from separate distribution distinct from that of
later waves.
2. Executor Startup Delays: Adding executor Spark job involves starting
JVM process, which typically incurs a delay of 2-3 seconds. To accurately reflect this behavior, the simulator imposes startup delay whenever executor reallocated across jobs, mirroring the real-world costs associated with executor initialization.
3. Impact of High Parallelism:
High degrees of parallelism can negatively impact the per- formance of individual tasks. Wider shuffle operations require more TCP connections and introduce additional computational overhead when merging data from a large number of shards. The simulator captures these effects by sampling task durations from distributions corresponding to different levels of parallelism, provided such data is available.
Through these mechanisms, the simulator effectively replicates the dynamic behavior of a real- world Spark cluster. This enables rigorous testing and validation of scheduling policies under realistic conditions, ensuring that the results are representative of practical deployment scenarios.
A.2.2 Workload Setup
For the input workload, TPC-H is a benchmark suite widely used to evaluate decision support systems. It consists of complex analytical queries that reflect real-world workloads in distributed data processing environments. Each query involves a combination of computations, such as joins, aggregations, and data filtering. which represented Directed Acyclic Graphs (DAGss of tasks in the Spark environment. The DAG structure introduces dependencies between tasks, making it ideal for testing the efficacy of advanced scheduling strategies.
2GB of data consisting of 10 TPC-H standard query tasks [54]
B Crucible Pseudocode
The detailed interaction logic between Crucible and the control algorithm is illustrated in Algo-
rithm R
C Optimization for BBA
We modify BBA to incorporate two control logics based on bandwidth and buffer occupancy. The bandwidth- based control selects the maximum bitrate that does not cause stalling, while the original buffer- baster---deddddee logic selects its own bitrate. The algorithm then chooses the smaller of these two bitrates. The detailed pseudocode can be found in [
23

---

Algorithm 1: Crucible Algorithm
Input:
Algorithm Algeur. Reference algorithm Algres, Number of LLM adjustments Num Number of Bayesian optimizations NBO. Test environments Envs
Output: Potentials
Function Apply
htpplylaaysssmmmpppiimaaattttto. NBO. E)
if NBo > ( then
params, ranges = LLM
Alg, score BO(Alg params, ranges, Nno, E)
else
ScoTe = E(Alg)
return Alg, score
for E' E Envs do
Algour Algrest, score mnt = EAAlgi score score inst
10
Algour: score oap..//ayessanngggiiitttttooooo (Algeur NBO, E
BadCases' = !}]
12
for j = Num do
13
1 Compare with reference algorithm and collect bad cases */
14
new BedCaeasess CompareAAgs (Alg'ss Algres E')
15
BadCases' BadCases' U new BadCases'
16
1* Get optimization suggestions from LLM */
17
suggestions' reasons'
BadCases')
1 Apply optimization suggestions */
19
Alghew Abwp:/Suggeettonsssisaassssseeee
20
g if score new score then
score abp.gaayerrannppsmmmataataaa...r NBO, E*)
22
/0 Score comparison and update */
23
Alg Alg
24
scoret our = score
1 Find ideal environment and calculate potentials */
26 Polentials = O
Frest null
28 Potentialmas =
29 for E E Enus do
E, GetldealEnci (Algour Envs)
D GetDistance( E E.)
Potential' score' score!
32
D
33
Potentials Potentials U U (Potential')
return Potentials
24

---

Algorithm BBA_C Algorithm
Input:
Current buffer size buf fer, chunk length chunk_len, download speed speed, video bitrate array bitrates, dimension dim
Output: Selected bitrate index rale
cushion + reservoir + predefined value bela +- 0.95
It Calculate bandwidth-based bitrate */
3 bw rate + for dim 1, dim - .......0 0 do
if bitrates[i] X chunk_len < beta x speed x buf fer then
L bw hhareate + break
Calculate buffer-based bitrate */ buf rate 0 if fer reservoir then buterate+ +
else
10
bur 2 reservoir + cushion then buf_rate dim -1
12
else
buf_rate + [dim - 1) x ba rer-rrooooo
13
cushion
14 1* Choose the minimum of both bitrates */
rate +- min(bw_rd tatturrteeeeeee buf __ate)
16 rate + max(0, min(rate, dim - 1))
return rate
25


Reviews:
[{"review_id": "2SX0kwkVZR", "reviewer": "Reviewer_4H7x", "summary": "The paper introduces the Crucible framework, a novel approach to quantify the \"tuning potential\" of control algorithms, defined as the performance improvement achievable through parameter and logic adjustments by domain experts in production environments. Crucible leverages large language models to simulate developers of varying expertise levels, using multi-level expert simulation and environment distance metrics (Wasserstein distance for ABR, initial performance ratio for scheduling). The framework evaluates adaptive bitrate (ABR) algorithms and distributed scheduling algorithms on standard datasets, using domain-specific metrics (QoE for ABR, flow completion time (FCT) for scheduling). Results demonstrate Crucible’s ability to quantify tuning potential, revealing the impact of representation capacity and logic complexity on optimization. The paper discusses LLM stability, limitations in improving black-box algorithms, and societal impacts.", "questions": "1. The tuning potential metric (Section 3.2) uses task-specific environment distance metrics (Wasserstein for ABR, ratio-based for scheduling), lacking a unified mathematical definition. Can you provide a formal derivation or a generalized metric that ensures cross-task comparability? For example, could a single distance metric (e.g., based on performance distributions) be applied across domains?\n2. The multi-level expert simulation uses Bayesian iterations and reflection steps to model varying expertise levels, but the basis for capability delineation is unclear. Can you specify the quantitative criteria for these levels and validate their impact on results (e.g., through ablation studies)?\n3. Experiments rely on simulated environments without real-world production system validation. Can you provide results from real-world or near-real-world settings? How do these results compare to simulations?\n4. Comparisons with existing optimization methods (e.g., hyperparameter tuning, reinforcement learning) are limited, weakening originality claims. Can you provide a detailed comparison with methods like Bayesian optimization or RL-based tuning?", "limitations": "The paper demonstrates Crucible’s applicability to ABR and scheduling but does not discuss the limitations of extending the framework to other domains (e.g., robotics, real-time systems) where control algorithms may have different constraints or data distributions. The task-specific environment distance metrics (Wasserstein for ABR, ratio-based for scheduling) may not generalize well.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- This paper introduces the concept of “tuning potential” for the first time and quantifies it through expert simulation based on large language models (LLMs), environmental distance regularization (ABR uses Wasserstein distance, scheduling uses ratio benchmarks), and a standardized interaction interface (Section 3.3), providing a pioneering method for the interdisciplinary field of control algorithms and machine learning.\n- The application of Crucible in ABR and scheduling demonstrates cross-domain generalizability. Results (e.g., Figure 5(b), showing that optimized SJF outperforms multi-level feedback) provide new insights for algorithm design, emphasizing the importance of logical simplification and representational capability.\n- The study encompasses multiple algorithms (ABR: BBA, MPC, HYB, BOLA, Prince; scheduling: SJF, FIFO, Round Robin, Tetris) and standard datasets (Obee, FCC, Puffer, Spark), using domain-specific metrics (QoE, FCT) and validated through Bayesian optimization (0, 10, 20 iterations). Figures 3 and 5 validate the framework's effectiveness.\n\n**Weaknesses:**\n\n- The definition of optimization potential (Section 3.2) relies on environment distance normalization, and the definition of environment distance varies depending on the task, lacking a unified mathematical definition, which limits its theoretical persuasiveness.\n- Multi-level expert simulation employs Bayesian iteration (0, 10, 20) and reflection steps (1, 2, 3), but the basis for defining capabilities (e.g., prompt design) is not explicitly stated, potentially affecting reproducibility.\n- Experiments rely on simulated environments (ABR simulator, Spark simulator) without validation on real-world production systems, limiting their credibility.\n- Comparisons with existing optimization methods (e.g., hyperparameter tuning, reinforcement learning) are incomplete, weakening the claim of originality.", "was_revised": true, "final_justification": "The authors' rebuttal provides a thoughtful and well-organized response that clarifies several key aspects of the paper. The effort to unify the environment distance metric through performance-based characterization is a meaningful step toward improving theoretical soundness, though a more formal treatment would be beneficial in the future. The inclusion of preliminary real-world validation using Dash.js adds practical value and supports the applicability of Crucible beyond simulations. The explanation of the multi-level expert simulation mechanism—grounded in computational resource usage rather than prompt design—clarifies the modeling choices and aligns with the workflow. Additionally, the expanded comparison with traditional optimization and RL-based methods helps better situate Crucible within the broader literature. While some concerns remain partially open, particularly regarding formalization and generalizability, the rebuttal has strengthened the overall presentation and reinforced the contribution's relevance. I maintain my original positive score.", "rebuttal": "Thank you very much for your valuable and constructive feedback. Your comments have helped us examine and refine this work from a deeper perspective. Our response addresses four main aspects: unified environment distance definition, real-world environment validation, multi-level expert simulation mechanism explanation, and comparison with existing optimization methods. We will revise the manuscript and add the newly introduced code to the supplementary materials. If you have any further questions, please feel free to continue the discussion with us.\n\n## **1. Unified Environment Distance Metric**\n\nYour observation regarding the lack of a unified mathematical definition for environment distance is very insightful and prompted us to re-examine the theoretical foundation of \"tuning potential.\" We believe that a unified, cross-task metric would significantly enhance both the theoretical value and practical applicability of our framework. Therefore, we have extended the score-based ratio method used as environmental distance for scheduling algorithms to form a unified performance characteristic-based distance measurement approach.\n\n\n### **Theoretical Derivation and Formal Definition**\n\n**Essential Characterization of Environment**\n\nFor control algorithms, the essence of an \"environment\" is not its physical parameters (such as bandwidth or queue length), but the **performance impact patterns** that the environment produces on different algorithms. Based on this insight, we propose a unified environment distance measurement method.\n\n**Step 1: Constructing Performance Characteristic Vector V(E)**\n\nTo capture the performance impact pattern of environment E, we define an evaluation environment set $T$, including the ideal environment $E_i$ and a collection of representative test environments $\\{E_t\\}$. We select $n$ representative \"probe algorithms\" following principles of diversity and classicality:\n\n- **ABR tasks**: Select model-based MPC and heuristic-based BBA\n- **Scheduling tasks**: Select classic FIFO and SJF\n\nFor environment $E_k \\in T$, we run probe algorithms to obtain raw performance scores $s_1(E_k), s_2(E_k), ..., s_n(E_k)$. To eliminate dimensional effects, we perform normalization:\n\n$$s_{j,max} = \\max_{E_k \\in T}(s_j(E_k)), \\quad s_{j,min} = \\min_{E_k \\in T}(s_j(E_k))$$\n\n$$\\text{norm}(s_j(E_k)) = \\frac{s_j(E_k) - s_{j,min}}{s_{j,max} - s_{j,min}}$$\n\nThus, environment $E_k$ is represented as an $n$-dimensional normalized **performance characteristic vector** $V(E_k)$.\n\n**Step 2: Defining Unified Environment Distance**\n\nThe distance between two environments $E_i$ and $E_t$ uses root mean square error:\n\n$$\\text{dis}(E_i, E_t) = \\sqrt{\\frac{1}{n} \\sum_{j=1}^{n} (V(E_i)_j - V(E_t)_j)^2}$$\n\nThe corresponding environment similarity is:\n$$\\text{sim}(E_i, E_t) = 1 - \\text{dis}(E_i, E_t)$$\n\n**Improved Tuning Potential Definition**\n\nAlgorithm potential $P$ is defined as similarity-weighted average performance gain:\n\n$$P = \\frac{1}{|T|} \\sum_{t \\in T} [(S_{tc} - S_{to}) \\times \\text{sim}(E_i, E_t)]$$\n\nwhere $S_{tc}$ and $S_{to}$ are the post-tuning and pre-tuning performance, respectively.\n\n### **Recalculated Results**\n\nResults in Table 1 recalculated based on the unified metric (sorted by \"Performance+Potential\"):\n\n| Algorithm | Potential | Performance+Potential |\n|-----------|-----------|----------------------|\n| MPC       | 0.0170    | 1.0862              |\n| BOLA      | 0.0250    | 1.0400              |\n| HYB       | 0.0679    | 0.9859              |\n| BBA       | 0.0181    | 0.7724              |\n| Pitree    | 0.0329    | 0.3407              |\n\nWe can still observe that: 1. The HYB algorithm based on both buffer and bandwidth has higher potential than BOLA and BBA which are solely buffer-based; 2. The HYB algorithm with simple single-step logic has higher potential than the multi-step decision-making MPC.\n\n## **2. Real-World Validation with Dash.js**\n\nAs a solution to address the pain points of algorithm deployment in real environments, we agree that adding real environment experiments would make Crucible more convincing. To address the limitations of simulated environments, we conducted real network environment validation using Dash.js. We deployed Dash.js on the client side, connected to cloud servers via public WiFi, and tested five heuristic algorithms (MPC, BOLA, BBA, Pitree, HYB) as well as the reinforcement learning-based Pensieve as a control. Each algorithm controlled 3 minutes of real playback of Envivio video. Crucible performed optimization based on real playback logs, configured for 3 iterations and a maximum of 10 Bayesian iterations. The results are shown in the following table:\n\n| Algorithm    | Original | Crucible-Tuned |\n|--------------|----------|----------------|\n| HYB          | 1.40     | 1.72           |\n| BBA          | 1.56     | 1.72           |\n| BOLA         | 1.20     | 1.54           |\n| Pitree       | 1.73     | 1.73           |\n| MPC          | 1.72     | 1.79           |\n| Pensieve(RL) | 1.66     | -              |\n\nThe real-world results validate two findings: (1) tuned simple heuristic algorithms (HYB, BBA) can outperform black-box algorithms; (2) algorithm potential correlates with its expressiveness and interpretability, with MPC achieving the highest post-tuning performance while complex Pitree showed no performance change after tuning.\n\n## **3. Multi-Level Expert Simulation**\n\nMulti-level expert simulation is primarily based on Crucible's workflow. The capability stratification is based on computational resources consumed during the tuning process, rather than prompt design. Specifically, this includes:\n\n- **Reflection iterations**: Number of iterative optimization rounds by the LLM agent\n- **Bayesian optimization calls**: Frequency of traditional optimization tool invocations\n\nWe only define interaction interfaces and basic knowledge in prompts (input/output formats, optimization objectives, etc. The specific prompt design can be found in Crucible/abr/server.py and Crucible/scheduling/server.py in the supplementary materials), and analyze performance improvements under different resource consumption levels (different iteration counts, different Bayesian optimization frequencies) through Figures 3 and 4 in sections 4.2 and 4.3.\n\n## **4. Comparison with Existing Optimization Methods**\n\n**Comparison with Tuning Methods**\n\nCrucible treats other optimization methods (such as Bayesian optimization) as tool calls. The relative improvement results from section 4.2 and Figure 3 comparing with Bayesian optimization demonstrate: (1) Crucible transcends the limitations of traditional parameter optimization and achieves higher post-adjustment results through logic adjustment; (2) traditional optimization tools can enhance Crucible's tuning effectiveness. Comparing Figure 3(c) with Figure 3(a), when Bayesian optimization becomes available, the effectiveness rate of control logic adjustments made by LLM improved from approximately 40% to 60% compared to the same number of Bayesian optimization iterations. This indicates that in about 20% of scenarios, the general direction of control logic adjustments made by LLM was correct, but the specific parameter selection was problematic. Therefore, with Bayesian optimization, this portion of adjustments also became effective.\n\n**Comparison with Direct Optimization by Reinforcement Learning Methods**\n\nFor black-box algorithms like RL, we employ decision tree conversion for potential analysis (such as Pitree in Table 1). Across the four datasets(Oboe, FCC, Puffer, 3G) in Table 1, Pensieve's average performance was 0.87, outperforming simple heuristic algorithms (BBA). However, after tuning, all four heuristic algorithms surpassed the RL algorithm, demonstrating the advantage of interpretable algorithms under tuning."}, {"review_id": "3hvTIFIrfF", "reviewer": "Reviewer_Cp8B", "summary": "The paper proposes the Crucible framework, which is the first to use LLM to simulate multi-level experts to tune the control algorithm, thereby calculating the tuning potential of the control algorithm. Through tests in two classic fields, ABR and scheduling algorithms, the effectiveness of the framework was verified, and the importance of the algorithm's tunability as a new optimization goal was explored, which can be customized according to real application scenarios.", "questions": "1. The article mentioned that different LLMs will affect the final results. Can you add corresponding comparative tests? I think this is very helpful for the effectiveness of the algorithm.\n2. The scheduling algorithm uses a score-based ratio as the environmental distance, which is tricky. Can more analysis or more detailed environmental distance modeling be added?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. Using \"tunability\" as a new dimension for control algorithm design and evaluation hits the pain points in the industrial deployment process.\n2. Using LLM as an expert agent to simulate actual developers to adjust the algorithm at the logic and parameter levels is novel and practical.\n3. The detailed pseudo-code of Crucible is provided in the appendix.\n\nCons:\n1. Currently, Crucible does not support optimization of the internal logic of black-box algorithms (it can only adjust the distillation version of the decision tree), which is a common situation for many industrial algorithms in reality.\n2. This article mainly focuses on two tasks: ABR and scheduling. The experimental results are not comprehensive enough.", "was_revised": true, "final_justification": "I sincerely apologize for being offline for the past two weeks due to work commitments. Prior to going offline, I had thoroughly reviewed the authors' rebuttal. At that time, my concerns were adequately addressed, and I deemed no additional experiments necessary to prove generalization or complex control algorithms. Therefore, I submitted an ​​expedited review response​​ directly through the button.\n\nTonight, upon carefully examining the authors' replies to other reviewers, I observed ​​consistent concerns​​ across the reviews. Notably, the supplementary experiments added by the authors in response to different reviewers appear ​​largely the same​​.\n\nNonetheless, given that my primary concerns have been resolved, I am willing to revise my score from ​​Borderline Reject​​ to ​​Borderline Accept​​.", "rebuttal": "Thank you very much for your valuable and constructive feedback. Your comments have helped us examine and refine this work from a deeper perspective. Regarding the optimization of internal logic for black-box algorithms, this can indeed be challenging for production systems because their black-box nature makes their logic incomprehensible to developers. As the first work to explore the new dimension of control algorithm potential, we have chosen to focus primarily on heuristic algorithms, and for black-box algorithms, we opt to distill them into decision trees for analysis. Our response mainly addresses three aspects: addition of classic control scenarios and real-world system validation, unified detailed environment distance modeling analysis, and horizontal comparison of mainstream LLMs with different capabilities. We will revise the manuscript and add the newly introduced code to the supplementary materials. If you have any further questions, please feel free to continue the discussion with us.\n\n## **1. Regarding the Focus on ABR and Scheduling with Insufficient Comprehensive Experimental Results**\n\n1. **We have added a classic control task, the cart-pole problem as a representative example.**\n\n     We selected 2 common computer system control tasks that have rich baselines and are widely applied in production systems to demonstrate our practical value. We understand the reviewer's concerns about task generalizability. Therefore, we chose the classic control task of cart-pole problem, so that we cover a range from simple classic tasks to complex real-world systems. We used \"CartPole-v1\" from Gym as simulated environment and selected bang-bang, PID, and LQR as baseline algorithms while using DQN algorithm as a reference. The experimental results after Crucible adjustment are as follows:\n\n   |           | **Initial** | **After Bayesian** | **After 1st LLM Optimization** | **After Bayesian** | **After 2nd LLM Optimization** | **After Bayesian** |\n   |--|-|-|-|--|-|-|\n   | Bang_bang | 34          | 56                 | 500                             |            -             |                  -                   |          -                |\n   | PID       | 34          | 77                 | 110                             | 271                | 500                            |            -              |\n   | LQR       | 161         | 500                |                    -                  |               -          |                   -                  |          -                |\n   | DQN       | 500         |       -                  |             -                         |           -              |              -                       |        -          |\nThe cart-pole problem, as a simple classic task, demonstrates two insights: 1. The importance of potential: simple heuristic algorithms can achieve performance comparable to black-box algorithms (reaching a score of 500); 2. LLM expands the optimization space of traditional adjustment methods by modifying control branches (both PID and bang_bang reached a score of 500 after LLM adjusted their control branches).\n\n2. **We have added real-world ABR experiments based on Dash.**\n\n   To demonstrate our effectiveness in complex tasks of real systems, we conducted real network environment validation using Dash.js. We deployed Dash.js on the client side, connected to cloud servers via public WiFi, and tested five heuristic algorithms (MPC, BOLA, BBA, Pitree, HYB) as well as the reinforcement learning-based Pensieve as a control. Each algorithm controlled 3 minutes of real playback of Envivio video. Crucible performed optimization based on real playback logs, configured for 3 iterations and a maximum of 10 Bayesian iterations. The results are shown in the following table:\n\n   | Algorithm    | Original | Crucible-Tuned |\n   |-|-|-|\n   | HYB          | 1.40     | 1.72           |\n   | BBA          | 1.56     | 1.72           |\n   | BOLA         | 1.20     | 1.54           |\n   | Pitree       | 1.73     | 1.73           |\n   | MPC          | 1.72     | 1.79           |\n   | Pensieve(RL) | 1.66     | -              |\n\n   The real-world results validate two findings: (1) tuned simple heuristic algorithms (HYB, BBA) can outperform black-box algorithms; (2) algorithm potential correlates with its expressiveness and interpretability, with MPC achieving the highest post-tuning performance while complex Pitree showed no performance change after tuning.\n\n## **2. Unified Detailed Environment Distance Modeling Analysis**\n\nThank you for appreciating the environment distance method for scheduling algorithms. This inspired us to extend the score-based ratio method used as environmental distance for scheduling algorithms to form a unified performance characteristic-based distance measurement approach.\n\n### **Theoretical Derivation and Formal Definition**\n\n**Essential Characterization of Environment**\n\nFor control algorithms, the essence of an \"environment\" is not its physical parameters (such as bandwidth or queue length), but the **performance impact patterns** that the environment produces on different algorithms. Based on this insight, we propose a unified environment distance measurement method.\n\n**Step 1: Constructing Performance Characteristic Vector V(E)**\n\nTo capture the performance impact pattern of environment E, we define an evaluation environment set $T$, including the ideal environment $E_i$ and a collection of representative test environments $\\{E_t\\}$. We select $n$ representative \"probe algorithms\" following principles of diversity and classicality:\n\n- **ABR tasks**: Select model-based MPC and heuristic-based BBA\n- **Scheduling tasks**: Select classic FIFO and SJF\n\nFor environment $E_k \\in T$, we run probe algorithms to obtain raw performance scores $s_1(E_k), s_2(E_k), ..., s_n(E_k)$. To eliminate dimensional effects, we perform normalization:\n\n$$s_{j,max} = \\max_{E_k \\in T}(s_j(E_k)), \\quad s_{j,min} = \\min_{E_k \\in T}(s_j(E_k))$$\n\n$$\\text{norm}(s_j(E_k)) = \\frac{s_j(E_k) - s_{j,min}}{s_{j,max} - s_{j,min}}$$\n\nThus, environment $E_k$ is represented as an $n$-dimensional normalized **performance characteristic vector** $V(E_k)$.\n\n**Step 2: Defining Unified Environment Distance**\n\nThe distance between two environments $E_i$ and $E_t$ uses root mean square error:\n\n$$\\text{dis}(E_i, E_t) = \\sqrt{\\frac{1}{n} \\sum_{j=1}^{n} (V(E_i)_j - V(E_t)_j)^2}$$\n\nThe corresponding environment similarity is:\n$$\\text{sim}(E_i, E_t) = 1 - \\text{dis}(E_i, E_t)$$\n\n**Improved Tuning Potential Definition**\n\nAlgorithm potential $P$ is defined as similarity-weighted average performance gain:\n\n$$P = \\frac{1}{|T|} \\sum_{t \\in T} [(S_{tc} - S_{to}) \\times \\text{sim}(E_i, E_t)]$$\n\nwhere $S_{tc}$ and $S_{to}$ are the post-tuning and pre-tuning performance, respectively.\n\n### **Recalculated Results**\n\nResults for Table 1 in the manuscript recalculated based on the unified metric (sorted by \"Performance+Potential\"):\n\n| Alg | Potential | Performance+Potential |\n|-|-|-|\n| MPC       | 0.0170    | 1.0862              |\n| BOLA      | 0.0250    | 1.0400              |\n| HYB       | 0.0679    | 0.9859              |\n| BBA       | 0.0181    | 0.7724              |\n| Pitree    | 0.0329    | 0.3407              |\n\nWe can still observe that: 1. The HYB algorithm based on both buffer and bandwidth has higher potential than BOLA and BBA which are solely buffer-based; 2. The HYB algorithm with simple single-step logic has higher potential than the multi-step decision-making MPC.\n\n## **3. Horizontal Comparison of Mainstream LLMs with Different Capabilities**\n\nIn the paper, we used the Claude-Sonnet-3.7 interface. LLMs provide us with the opportunity to mimic developers in understanding and adjusting algorithm logic, and we believe this capability is independent of specific LLM models because we primarily leverage the general knowledge of LLMs. Therefore, we did not discuss different LLMs in the manuscript.\n\nWe understand the reviewer's concerns about LLM capabilities, so we added Claude-Sonnet-3.5 as a previous generation model and GPT-4o-mini as another mainstream model. We set the iteration count to 3 and Bayesian iterations to 20. The experimental results are as follows (due to rebuttal time constraints, we did not complete the MPC algorithm results):\n\n| Alg | Initial Performance | Claude-Sonnet-3.5 Final Performance | GPT-4o-mini Final Performance | Claude-Sonnet-3.7 Final Performance |\n|-|-|-|-|-|\n| bb        | 0.7543              | 1.1283                               | 1.1024                         | 1.1083                               |\n| bola      | 1.015               | 1.0661                               | 1.0811                         | 1.0642                               |\n| hyb       | 0.918               | 1.0316                               | 1.0377                         | 1.1242                               |\n| pitree    | 0.3079              | 0.3524                               | 0.3671                         | 0.3616                               |\n\nLLMs do indeed affect the final post-adjustment performance, but the impact magnitude is modest. Claude-Sonnet-3.7's post-adjustment performance is overall superior to Sonnet-3.5 and 4o-mini, averaging 0.02 and 0.018 higher respectively. While we believe algorithm adjustment scenarios could be an interesting new LLM evaluation metric, the focus of this work remains on new optimization and evaluation metrics for control algorithms. Therefore, we believe that analyzing the potential dimension of control algorithms based on any current mainstream model is valuable. An interesting point lies in the HYB algorithm, which is the only one where relative ranking changed. After adjustment by Sonnet-3.5 and 4o-mini, its performance still falls short of BOLA and BBA, but after adjustment by Sonnet-3.7, it achieved optimal performance. This suggests that the logic of the HYB algorithm requires stronger models to understand and adjust."}, {"review_id": "A9fC6UFKQ2", "reviewer": "Reviewer_BJf8", "summary": "This manuscript proposes Crucible, a framework that uses large language models (LLMs) to evaluate the tuning potential of control algorithms, where an algorithm's tuning potential is defined as the performance improvement it could achieve when adjusted by domain experts, relative to the performance of its preset settings. Specifically, Crucible consists of a multi-level expert simulation system whereby an LLM (e.g. Claude 3.7 Sonnet) is prompted with domain knowledge including task description, optimization objectives, and environment overview, and acts as a proxy for human experts to iteratively refine an algorithm’s hyperparameters and logic with variable number of reflective iterations and access to optimization tools. To ensure a fair potential score across different scenarios, an environment distance metric is proposed to normalize performance gains  by how different the test environment is from the preset environment. The proposed framework is applied to two domains from computer systems, namely, adaptive bitrate control and distributed task scheduling, demonstrating that it can uncover a larger performance optimization space than Bayesian optimization alone by suggesting additions and removal of control logic. The analysis indicates that there are two key factors influencing an algorithm’s tunability: its representational capacity and comprehensibility.", "questions": "1. What exactly were the prompts and LLM outputs that drove Crucible’s algorithm improvements? The paper should include concrete examples of the LLM’s input and output (e.g. a sample prompt and the suggestions it gave for improving an algorithm).\n\n2. How sensitive are Crucible’s results to the choice of LLM (e.g., Claude vs. GPT-4)? Would different LLMs yield significantly different tuning outcomes? It would be beneficial to include a sensitivity analysis across multiple LLMs to assess robustness.\n\n3. The Crucible agent simulates different expert skill levels by limiting tool iterations and reflection steps, but how do we know these settings realistically correspond to expert abilities? Can the authors justify or validate this design? \n\n4. The manuscript uses Wasserstein distance for ABR and a proportion-based metric for scheduling. Can you include the mathematical equations for these metrics? How these metrics were chosen, and how sensitive are the results to these choices?\n\n5. How well do the LLMs simulate real expert behavior? Are there any benchmarks or validations comparing LLM-generated tuning strategies with those from human experts? Have the authors considered comparing against real expert adjustments or other algorithm-tuning techniques? For example, for one of the simpler scenarios, one could ask a human expert to manually fine-tune the algorithm and see how much improvement they get, to check if Crucible is on par or discovering similar enhancements.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "* Strengths\n\n1. The manuscript addresses a practical yet under-explored issue: algorithms in production are often tuned by experts, but research usually ignores this tunability aspect. Through two case studies from computer systems, it is shown that tuning potential is crucial for real-world performance and that a well-tuned simple algorithm can rival or exceed more sophisticated ones. \n\n2. The Crucible framework is innovative in its use of an LLM as a proxy for human developers, which provides a way to simulate expert tuning and human-in-the-loop evaluations at scale without the cost of human trials. \n\n3. It is interesting to systematically show that combining LLM-based logic updates with automatic hyperparameter tuning using bayesian optimization yields the best results.\n\n* Weaknesses\n\n1. The manuscript suffers from a lack of clarity and detail in its presentation. Specifically, the Crucible involves complex interactions (prompts, iterative loops), but the description in the paper is often high-level and omits key specifics. For example, the framework relies on system prompts to inject domain knowledge into the LLM, yet the actual prompt contents and examples of advice given by the LLM are neither provided nor discussed. Similarly, the environment distance metrics are not provided, motivated, or well-explained. While the algorithm pseudo-code for Crucible (Algorithm 1) is included, it is not discussed in depth beyond a formal outline, which makes it harder to appreciate the technicalities of the design.\n\n2. The manuscript doesn’t discuss how results might vary with a different or weaker LLM, which is important given that it's the core technology in the proposed framework. Furthermore, it is not clear how well Crucible generalizes to other domains like robotics or financial systems.\n\n3. The manuscript lacks an ablation study to assess how different design decision in the crucible framework impact the overall performance.\n\n4. The significance of the potential metric would be stronger if validated against human expert tuning in practice. Currently, it’s an interesting proxy, but we don’t know if algorithms that Crucible deems high potential indeed are really easier for real engineers to tune.", "was_revised": true, "final_justification": "I have carefully reviewed the author response and all subsequent discussions. The authors have addressed the main concerns raised in my initial reviews, including prompt specification, sensitivity to LLM choice, and distance metrics used. Therefore, I am increasing my overall score to reflect the improved clarity and completeness of the submission.", "rebuttal": "Thank you very much for your valuable and constructive feedback. Your comments have helped us examine and refine this work from a deeper perspective. We will respond from five aspects: Crucible's prompt and interaction details, analysis of results from different LLMs, ablation study explanations, unified environment distance metrics, and comparison between Crucible and human experts. We will revise the manuscript and add the newly introduced code to the supplementary materials. If you have any further questions, please feel free to continue the discussion with us.\n\n## 1. Exact Prompts and LLM Outputs Driving Crucible's Algorithm Improvements\n\nWe understand concerns about clarity, but since our article emphasizes LLMs providing opportunities to examine new optimization directions, we focused on high-level concepts rather than specific prompts. Our multi-level capability aspects use workflow definitions like tool call frequency and LLM iteration counts. Prompts mainly introduce basic task knowledge and interaction interfaces. We believe our work should be decoupled from specific prompt designs - readers can easily design equivalent or better prompts. \n\nBelow we provide the general prompt structure using ABR algorithms as an example. The specific implementation details can be found in the supplementary materials (Crucible/abr/server.py for ABR tasks and Crucible/scheduling/server.py for scheduling tasks).\n\n**Tool calling prompt structure:** \n\n```python\nPlease analyze the following ABR algorithm code and identify parameters for Bayesian optimization. Return results in JSON format with parameter name, type, range, current value, and description.\n\n```\n**Background and optimization prompt structure:**\n```python \nI need you to optimize a bitrate adaptive algorithm with 6 bitrate levels [300,750,1200,1850,2850,4300]kbps.  Current code:{code}. Current reward: {current_reward}. Optimize history is {history}. Please analyze the code and suggest improvements. First explain improvement ideas, then provide complete modified abr function code.\n\n```\n## 2. Analysis of Results from Different LLMs\n\nIn the paper, we used Claude-Sonnet-3.7 interface. LLMs provide us with the opportunity to mimic developers in understanding and adjusting algorithm logic, and we believe this capability is independent of specific LLM models because we primarily leverage the general knowledge of LLMs. \n\nWe understand the reviewer's concerns about LLM capabilities, so we added Claude-Sonnet-3.5 as a previous generation model and GPT-4o-mini as another mainstream model. We set the iteration count to 3 and Bayesian iterations to 20. The experimental results are as follows (due to rebuttal time constraints, we did not complete the MPC algorithm results):\n\n| Alg| Initial | Sonnet-3.5 | GPT-4o-mini | Sonnet-3.7 |\n|-|-|-|-|-|\n| bb        | 0.7543  | 1.1283     | 1.1024      | 1.1083     |\n| bola      | 1.015   | 1.0661     | 1.0811      | 1.0642     |\n| hyb       | 0.918   | 1.0316     | 1.0377      | 1.1242     |\n| pitree    | 0.3079  | 0.3524     | 0.3671      | 0.3616     |\n\nLLMs do indeed affect the final post-adjustment performance, but the impact magnitude is modest. Sonnet-3.7's post-adjustment performance is overall superior to Sonnet-3.5 and 4o-mini, averaging 0.02 and 0.018 higher respectively. While we believe algorithm adjustment scenarios could be an interesting new LLM evaluation metric, the focus of this work remains on new optimization and evaluation metrics for control algorithms. Therefore, we believe that analyzing the potential dimension of control algorithms based on any current mainstream model is valuable. An interesting point lies in the HYB algorithm, which is the only one where the relative ranking changed. After adjustment by Sonnet-3.5 and 4o-mini, its performance still falls short of BOLA and BBA, but after adjustment by Sonnet-3.7, it achieved optimal performance. This suggests that the logic of the HYB algorithm requires stronger models to understand and adjust.\n\n## 3. Ablation Study\n\nCrucible improves algorithms through LLM understanding of algorithm modifications to control loops and calling Bayesian optimization methods. We compared improvements over Bayesian optimization in section 4.2 to show that Crucible transcends the limitations of traditional parameter optimization and achieves higher post-adjustment results through logic adjustments. Additionally, Figure 3 compares the different benefits brought by using and not using Bayesian methods to demonstrate the value of tool calling. Comparing Figure 3(c) with Figure 3(a), when Bayesian optimization becomes available, the effectiveness rate of control logic adjustments made by LLM improved from approximately 40% to 60% compared to the same number of Bayesian optimization iterations. This indicates that in about 20% of scenarios, the general direction of control logic adjustments made by LLM was correct, but the specific parameter selection was problematic. Therefore, with Bayesian optimization, this portion of adjustments also became effective.\n\n## 4. Unified Environment Distance Metrics\n\nWe extended the score-based ratio method used as environmental distance for scheduling algorithms to form a unified performance characteristic-based distance measurement approach to improve the applicability of our work.\n\n### **Theoretical Derivation and Formal Definition**\n\n**Essential Characterization of Environment**\n\nFor control algorithms, the essence of an \"environment\" is not its physical parameters (such as bandwidth or queue length), but the **performance impact patterns** that the environment produces on different algorithms. Based on this insight, we propose a unified environment distance measurement method.\n\n**Step 1: Constructing Performance Characteristic Vector V(E)**\n\nTo capture the performance impact pattern of environment E, we define an evaluation environment set $T$, including the ideal environment $E_i$ and a collection of representative test environments $\\{E_t\\}$. We select $n$ representative \"probe algorithms\" following principles of diversity and classicality:\n\n- **ABR tasks**: Select model-based MPC and heuristic-based BBA\n- **Scheduling tasks**: Select classic FIFO and SJF\n\nFor environment $E_k \\in T$, we run probe algorithms to obtain raw performance scores $s_1(E_k), s_2(E_k), ..., s_n(E_k)$. To eliminate dimensional effects, we perform normalization:\n\n$$s_{j,max} = \\max_{E_k \\in T}(s_j(E_k)), \\quad s_{j,min} = \\min_{E_k \\in T}(s_j(E_k))$$\n\n$$\\text{norm}(s_j(E_k)) = \\frac{s_j(E_k) - s_{j,min}}{s_{j,max} - s_{j,min}}$$\n\nThus, environment $E_k$ is represented as an $n$-dimensional normalized **performance characteristic vector** $V(E_k)$.\n\n**Step 2: Defining Unified Environment Distance**\n\nThe distance between two environments $E_i$ and $E_t$ uses root mean square error:\n\n$$\\text{dis}(E_i, E_t) = \\sqrt{\\frac{1}{n} \\sum_{j=1}^{n} (V(E_i)_j - V(E_t)_j)^2}$$\n\nThe corresponding environment similarity is:\n$$\\text{sim}(E_i, E_t) = 1 - \\text{dis}(E_i, E_t)$$\n\n**Improved Tuning Potential Definition**\n\nAlgorithm potential $P$ is defined as similarity-weighted average performance gain:\n\n$$P = \\frac{1}{|T|} \\sum_{t \\in T} [(S_{tc} - S_{to}) \\times \\text{sim}(E_i, E_t)]$$\n\nwhere $S_{tc}$ and $S_{to}$ are the post-tuning and pre-tuning performance, respectively.\n\n### **Recalculated Results**\n\nWe recalculated Table 1 in the manuscript based on the unified metric (sorted by \"Performance+Potential\"):\n\n| Alg| Potential | Performance+Potential |\n|-|-|-|\n| MPC       | 0.0170    | 1.0862              |\n| BOLA      | 0.0250    | 1.0400              |\n| HYB       | 0.0679    | 0.9859              |\n| BBA       | 0.0181    | 0.7724              |\n| Pitree    | 0.0329    | 0.3407              |\n\nWe can still observe that: 1. The HYB algorithm based on both buffer and bandwidth has higher potential than BOLA and BBA which are solely buffer-based; 2. The HYB algorithm with simple single-step logic has higher potential than the multi-step decision-making MPC.\n\n## 5. Comparison Between Crucible and Human Experts\nThanks for your suggestion regarding real experts, but if real experts were introduced, how to eliminate the subjective uncertainty of the real experts themselves would become a new problem. Therefore, this work demonstrates the role of Crucible through two parts: first, by comparing with optimization tools and existing algorithms that have more complete algorithmic logic to show Crucible's adjustment ability; and second, by showing how the insights brought by Crucible can be used to redesign algorithms.\n\nFigure 1(b) shows that we compared four algorithms: HYB, HYB(B), HYB(C), and MPC. HYB(B) is the result after Bayesian optimization. Both MPC and HYB algorithms use buffer size and bandwidth for decision-making, but MPC uses multi-step model prediction while HYB uses single-step prediction. Therefore, we can view HYB(B) and MPC as human experts' tuning strategies through parameter optimization and control logic modification respectively. The final performance is HYB(C)>MPC>HYB(B)>HYB. This demonstrates two things: 1. Algorithm adjustment is not merely parameter adjustment - HYB after Bayesian optimization still falls short of logic-adjusted MPC; 2. Crucible-adjusted HYB(C) achieved the highest performance through both control logic and parameter adjustments, demonstrating the importance of exploring algorithm potential and the comprehensiveness of Crucible's tuning strategy.\n\nFurthermore, we discussed in section 4.4 how to use the knowledge analyzed by LLM in the potential quantification process to guide algorithm optimization. For example, by expanding algorithm expression space (BBA_C) and simplifying algorithm control logic (from MF to SJF). This part represents the discussion of combining Crucible's potential quantification capability with human expert knowledge, where we obtained insights on how to design algorithms with higher potential from Crucible's adjustment process."}, {"review_id": "CFc1AFJjJl", "reviewer": "Reviewer_V5Ex", "summary": "This paper proposes an evaluation framework called `Crucible`, which is used to quantify the upper limit of the performance that a control algorithm can achieve through expert tuning in actual deployment, i.e., the \"potential\" of the algorithm. `Crucible` uses large language models (LLMs) as expert agents and systematically evaluates the tunable space of control algorithms in different scenarios through multi-level ability simulations (such as parameter tuning and logical structure optimization) and environment distance normalization methods. The paper selects adaptive bit rate (ABR) control and distributed scheduling as two representative application scenarios, demonstrating that `Crucible` has a wider optimization space compared with traditional parameter tuning methods. It proves the importance of algorithm potential for actual deployment value.", "questions": "1. The success rate/failure rate of `Crucible` or the statistics on the effectiveness of the suggestions are not provided. The lack of systematic analysis of the failure rate and error types of the suggestions generated by the LLM makes it difficult to judge whether the successful optimization of `Crucible` is stable and reproducible.\n2. All experiments are based on Claude 3.7 Sonnet, and there is no horizontal comparison of the output behavior of `Crucible` on different mainstream LLMs (such as GPT - 4, Gemini).\n3. It is recommended that the authors include at least one traditional control scenario (such as trajectory planning, PID control) in future work or extended experiments to verify the generality and depth of the `Crucible` method.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**1. Strengths**\n- **Quality.**\nThis paper introduces `Crucible`, an innovative framework for evaluating control algorithm tuning potential. The framework uniquely integrates large language models (LLMs) with Bayesian optimization, creating expert agents that simulate developer optimization behaviors in a closed-loop \"evaluation-tuning-feedback\" system. Through comprehensive experiments with multi-dimensional metrics and reproducible mechanisms, the results demonstrate performance improvements across various environments, empirically validating the framework's effectiveness.\n- **Clarity.**\nThe structure of the paper is reasonable, the discussion is fluent, the objectives are clear, and the logic is complete.\n- **Significance.**\nThe quantitative indicator of \"performance improvement space after expert tuning\" provided by `Crucible` helps incorporate \"tunability\" into the algorithm design goals and provides a new evaluation basis for directions such as system optimization, and operator search.\n- **Originality.**\nIn task definition, the \"potential\" of the control algorithm is taken as a new modeling objective and converted into observable indicators by the LLM expert agent. In method design, a hierarchical expert ability simulation system is constructed, and an environmental distance normalization strategy is introduced. Overall, it demonstrates strong originality and insight.\n\n**2. Weaknesses**\n- The absence of failure case analysis for LLM-generated suggestions limits the reader’s ability to evaluate `Crucible` ’s robustness and practical deployability.\n- `Crucible` heavily depends on commercial LLM APIs for generating tuning strategies, making it difficult to isolate the contribution of the framework from the behavior of specific models.\n- `Crucible` 's key components are built from existing tools, suggesting that its novelty lies more in integration and workflow design than in foundational methodological innovation.", "was_revised": true, "final_justification": "The Crucible framework proposed in this paper combines large language models and Bayesian optimization to quantitatively control the upper limit of algorithm tuning from the perspective of \"algorithmic potential\", filling the gap in the evaluation of tuning space systems in existing methods. The thesis is reasonably designed and has a clear structure. The multi-level expert capability simulation and environmental distance normalization introduced demonstrate innovation. The author supplemented a wealth of experimental data, including multi-model comparisons, classical control tasks, and real system verifications, enhancing the universality and practical value of the method.", "rebuttal": "Thank you very much for your valuable and constructive feedback. Your comments have helped us examine and refine this work from a deeper perspective. We will respond from three aspects: Crucible's effectiveness rate, horizontal analysis of results from different LLMs, and validation of traditional control scenarios and real-world experiments. We will revise the manuscript and add the newly introduced code to the supplementary materials. If you have any further questions, please feel free to continue the discussion with us.\n\n## 1. Crucible's Effectiveness Rate\n\nThe analysis of Crucible's effectiveness rate can be found in Figures 3 and 4 and the analysis in sections 4.2 and 4.3. We discuss this from two dimensions: 1. The effectiveness rate of control logic adjustments made by LLM 2. The overall effectiveness rate of Crucible (retaining LLM control logic adjustment and Bayesian parameter optimization)\n\n1. **Effectiveness rate of LLM control logic adjustments.** We compare Crucible with Bayesian optimization results in Figure 3, where Figure 3(a) shows pure LLM control logic adjustment without Bayesian optimization. We can see that the proportion of improvements close to 0 accounts for about 60%, which means the effectiveness rate of pure LLM logic adjustments is only 40%. However, we can see another dimension: when Bayesian optimization becomes available, compared to the same number of Bayesian optimization iterations, the effectiveness rate of LLM control logic adjustments improved to approximately 60%. This indicates that in 20% of scenarios, the general direction of LLM control logic adjustments was correct, but specific parameter selection was problematic. Therefore, with Bayesian optimization, this portion of adjustments also became effective.\n\n2. **Overall effectiveness rate.** Comparing with initial results in Figure 4, we can see that in ABR scenarios, there are still approximately 20% with improvement rates close to 0. This indicates that in 20% of scenarios, neither logic adjustments nor Bayesian optimization were effective. Scheduling algorithms mainly rely on LLM control logic modifications due to fewer involved parameters. Figure 4(e) shows that because scheduling algorithms often involve high-dimensional inputs, making LLM understanding of inputs more difficult, only about 40% of adjustments are effective.\n\n## 2. Horizontal Analysis of Results from Different LLMs\n\nIn the paper, we used Claude-Sonnet-3.7 interface. LLMs provide us with the opportunity to mimic developers in understanding and adjusting algorithm logic, and we believe this capability is independent of specific LLM models because we primarily leverage the general knowledge of LLMs. Therefore, we did not discuss different LLMs in the manuscript.\n\nWe understand the reviewer's concerns about LLM capabilities, so we added Claude-Sonnet-3.5 as a previous generation model and GPT-4o-mini as another mainstream model. We set the iteration count to 3 and Bayesian iterations to 20. The experimental results are as follows (due to rebuttal time constraints, we did not complete the MPC algorithm results):\n\n| Algorithm | Initial | Sonnet-3.5 | GPT-4o-mini | Sonnet-3.7 |\n|-----------|---------|------------|-------------|------------|\n| bb        | 0.7543  | 1.1283     | 1.1024      | 1.1083     |\n| bola      | 1.015   | 1.0661     | 1.0811      | 1.0642     |\n| hyb       | 0.918   | 1.0316     | 1.0377      | 1.1242     |\n| pitree    | 0.3079  | 0.3524     | 0.3671      | 0.3616     |\n\nLLMs do indeed affect the final post-adjustment performance, but the impact magnitude is modest. Claude-Sonnet-3.7's post-adjustment performance is overall superior to Sonnet-3.5 and 4o-mini, averaging 0.02 and 0.018 higher respectively. While we believe algorithm adjustment scenarios could be an interesting new LLM evaluation metric, the focus of this work remains on new optimization and evaluation metrics for control algorithms. Therefore, we believe that analyzing the potential dimension of control algorithms based on any current mainstream model is valuable. An interesting point lies in the HYB algorithm, which is the only one where relative ranking changed. After adjustment by Sonnet-3.5 and 4o-mini, its performance still falls short of BOLA and BBA, but after adjustment by Sonnet-3.7, it achieved optimal performance. This suggests that the logic of the HYB algorithm requires stronger models to understand and adjust.\n\n## 3. Validation of Traditional Control Scenarios and Real-World Experiments\n\n\n1. **We have added a classic control task, the cart-pole problem as a representative example.**\n\n     We selected 2 common computer system control tasks that have rich baselines and are widely applied in production systems to demonstrate our practical value. We understand the reviewer's concerns about task generalizability. Therefore, we chose the classic control task of cart-pole problem, so that we cover a range from simple classic tasks to complex real-world systems. We used \"CartPole-v1\" from Gym as simulated environment and selected bang-bang, PID, and LQR as baseline algorithms while using DQN algorithm as a reference. The experimental results after Crucible adjustment are as follows:\n\n   |           | **Initial** | **After Bayesian** | **After 1st LLM Optimization** | **After Bayesian** | **After 2nd LLM Optimization** | **After Bayesian** |\n   |--|-|-|-|--|-|-|\n   | Bang_bang | 34          | 56                 | 500                             |            -             |                  -                   |          -                |\n   | PID       | 34          | 77                 | 110                             | 271                | 500                            |            -              |\n   | LQR       | 161         | 500                |                    -                  |               -          |                   -                  |          -                |\n   | DQN       | 500         |       -                  |             -                         |           -              |              -                       |                -          |\nThe cart-pole problem, as a simple classic task, demonstrates two insights: 1. The importance of potential: simple heuristic algorithms can achieve performance comparable to black-box algorithms (reaching a score of 500); 2. LLM expands the optimization space of traditional adjustment methods by modifying control branches (both PID and bang_bang reached a score of 500 after LLM adjusted their control branches).\n\n2. **We added real-world ABR experiments based on Dash.**\n\n   To demonstrate our effectiveness in complex tasks of real systems, we conducted real network environment validation based on Dash.js. We deployed Dash.js on the client side, connected to cloud servers via public WiFi, and tested five heuristic algorithms (MPC, BOLA, BBA, Pitree, HYB) as well as the reinforcement learning-based Pensieve as a control. Each algorithm controlled 3 minutes of real playback of Envivio video. Crucible optimized based on real playback logs, configured with 3 iterations and maximum 10 Bayesian iterations. The results are as follows:\n\n| Algorithm    | Original | Crucible-Tuned |\n|--------------|----------|----------------|\n| HYB          | 1.40     | 1.72           |\n| BBA          | 1.56     | 1.72           |\n| BOLA         | 1.20     | 1.54           |\n| Pitree       | 1.73     | 1.73           |\n| MPC          | 1.72     | 1.79           |\n| Pensieve(RL) | 1.66     | -              |\n\n   Real environment results validated two findings: (1) Tuned simple heuristic algorithms (HYB, BBA) can potentially surpass black-box algorithms; (2) Algorithm potential correlates with its expressiveness and interpretability - MPC achieved the highest post-tuning performance, while complex Pitree showed no performance change after tuning."}]

Author rebuttal:

Meta-review: The paper presents a framework for tuning the parameters of a control algorithm using an LLM, who plays the role of a domain expert. Using adaptive bitrate/scheduling algorithms as applications, the paper demonstrates improved performance over several reference scheduling algorithms.

During the rebuttal phase, the added a comparative analysis across different mainstream LLMs (e.g., GPT-4o-mini, Claude series) and expanded to a classic control task (Cart-Pole) and a preliminary real-world validation using Dash.js. It would be good to include both in the paper.

That said, more broadly, the paper would be strengthened by the application of the proposed algorithm to more domains beyond communications. As one reviewer noted, the paper demonstrates Crucible’s applicability to ABR and scheduling but does not discuss the limitations of extending the framework to other domains (e.g., robotics, real-time systems) where control algorithms may have different constraints or data distributions. A concerned remained that the task-specific environment distance metrics (Wasserstein for ABR, ratio-based for scheduling) may not generalize well.

Reviewers generally wanted to see more details regarding the implementation and experimental setting. These included a description of prompts used, but also formal/precise definitions of distance metrics employed. The authors have addressed these concerns in the rebuttal, and these edits should also make it to the paper.
Decision: Accept (poster)