Title: Crucible: Quantifying the Potential of Control Algorithms through LLM Agents
Venue: NeurIPS 2025
Accepted: yes
Keywords: Control Algorithms

Large Language Models

Agent

Potential
Abstract: Control algorithms in production environments typically require domain experts to tune their parameters and logic for specific scenarios. However, existing research predominantly focuses on algorithmic performance under ideal or default configurations, overlooking the critical aspect of Tuning Potential. To bridge this gap, we introduce \texttt{Crucible}, an agent that employs an LLM-driven, multi-level expert simulation to turn algorithms and defines a formalized metric to quantitatively evaluate their Tuning Potential. We demonstrate \texttt{Crucible}'s effectiveness across a wide spectrum of case studies, from classic control tasks to complex computer systems, and validate its findings in a real-world deployment. Our experimental results reveal that \texttt{Crucible} systematically quantifies the tunable space across different algorithms. Furthermore, \texttt{Crucible} provides a new dimension for algorithm analysis and design, which ultimately leads to performance improvements. Our code is available at https://github.com/thu-media/Crucible.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an evaluation framework called `Crucible`, which is used to quantify the upper limit of the performance that a control algorithm can achieve through expert tuning in actual deployment, i.e., the \"potential\" of the algorithm. `Crucible` uses large language models (LLMs) as expert agents and systematically evaluates the tunable space of control algorithms in different scenarios through multi-level ability simulations (such as parameter tuning and logical structure optimization) and environment distance normalization methods. The paper selects adaptive bit rate (ABR) control and distributed scheduling as two representative application scenarios, demonstrating that `Crucible` has a wider optimization space compared with traditional parameter tuning methods. It proves the importance of algorithm potential for actual deployment value.\n\nSTRENGTHS AND WEAKNESSES:\n**1. Strengths**\n- **Quality.**\nThis paper introduces `Crucible`, an innovative framework for evaluating control algorithm tuning potential. The framework uniquely integrates large language models (LLMs) with Bayesian optimization, creating expert agents that simulate developer optimization behaviors in a closed-loop \"evaluation-tuning-feedback\" system. Through comprehensive experiments with multi-dimensional metrics and reproducible mechanisms, the results demonstrate performance improvements across various environments, empirically validating the framework's effectiveness.\n- **Clarity.**\nThe structure of the paper is reasonable, the discussion is fluent, the objectives are clear, and the logic is complete.\n- **Significance.**\nThe quantitative indicator of \"performance improvement space after expert tuning\" provided by `Crucible` helps incorporate \"tunability\" into the algorithm design goals and provides a new evaluation basis for directions such as system optimization, and operator search.\n- **Originality.**\nIn task definition, the \"potential\" of the control algorithm is taken as a new modeling objective and converted into observable indicators by the LLM expert agent. In method design, a hierarchical expert ability simulation system is constructed, and an environmental distance normalization strategy is introduced. Overall, it demonstrates strong originality and insight.\n\n**2. Weaknesses**\n- The absence of failure case analysis for LLM-generated suggestions limits the reader’s ability to evaluate `Crucible` ’s robustness and practical deployability.\n- `Crucible` heavily depends on commercial LLM APIs for generating tuning strategies, making it difficult to isolate the contribution of the framework from the behavior of specific models.\n- `Crucible` 's key components are built from existing tools, suggesting that its novelty lies more in integration and workflow design than in foundational methodological innovation.\n\nQUESTIONS:\n1. The success rate/failure rate of `Crucible` or the statistics on the effectiveness of the suggestions are not provided. The lack of systematic analysis of the failure rate and error types of the suggestions generated by the LLM makes it difficult to judge whether the successful optimization of `Crucible` is stable and reproducible.\n2. All experiments are based on Claude 3.7 Sonnet, and there is no horizontal comparison of the output behavior of `Crucible` on different mainstream LLMs (such as GPT - 4, Gemini).\n3. It is recommended that the authors include at least one traditional control scenario (such as trajectory planning, PID control) in future work or extended experiments to verify the generality and depth of the `Crucible` method.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes Crucible, a framework that uses large language models (LLMs) to evaluate the tuning potential of control algorithms, where an algorithm's tuning potential is defined as the performance improvement it could achieve when adjusted by domain experts, relative to the performance of its preset settings. Specifically, Crucible consists of a multi-level expert simulation system whereby an LLM (e.g. Claude 3.7 Sonnet) is prompted with domain knowledge including task description, optimization objectives, and environment overview, and acts as a proxy for human experts to iteratively refine an algorithm’s hyperparameters and logic with variable number of reflective iterations and access to optimization tools. To ensure a fair potential score across different scenarios, an environment distance metric is proposed to normalize performance gains  by how different the test environment is from the preset environment. The proposed framework is applied to two domains from computer systems, namely, adaptive bitrate control and distributed task scheduling, demonstrating that it can uncover a larger performance optimization space than Bayesian optimization alone by suggesting additions and removal of control logic. The analysis indicates that there are two key factors influencing an algorithm’s tunability: its representational capacity and comprehensibility.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n\n1. The manuscript addresses a practical yet under-explored issue: algorithms in production are often tuned by experts, but research usually ignores this tunability aspect. Through two case studies from computer systems, it is shown that tuning potential is crucial for real-world performance and that a well-tuned simple algorithm can rival or exceed more sophisticated ones. \n\n2. The Crucible framework is innovative in its use of an LLM as a proxy for human developers, which provides a way to simulate expert tuning and human-in-the-loop evaluations at scale without the cost of human trials. \n\n3. It is interesting to systematically show that combining LLM-based logic updates with automatic hyperparameter tuning using bayesian optimization yields the best results.\n\n* Weaknesses\n\n1. The manuscript suffers from a lack of clarity and detail in its presentation. Specifically, the Crucible involves complex interactions (prompts, iterative loops), but the description in the paper is often high-level and omits key specifics. For example, the framework relies on system prompts to inject domain knowledge into the LLM, yet the actual prompt contents and examples of advice given by the LLM are neither provided nor discussed. Similarly, the environment distance metrics are not provided, motivated, or well-explained. While the algorithm pseudo-code for Crucible (Algorithm 1) is included, it is not discussed in depth beyond a formal outline, which makes it harder to appreciate the technicalities of the design.\n\n2. The manuscript doesn’t discuss how results might vary with a different or weaker LLM, which is important given that it's the core technology in the proposed framework. Furthermore, it is not clear how well Crucible generalizes to other domains like robotics or financial systems.\n\n3. The manuscript lacks an ablation study to assess how different design decision in the crucible framework impact the overall performance.\n\n4. The significance of the potential metric would be stronger if validated against human expert tuning in practice. Currently, it’s an interesting proxy, but we don’t know if algorithms that Crucible deems high potential indeed are really easier for real engineers to tune.\n\nQUESTIONS:\n1. What exactly were the prompts and LLM outputs that drove Crucible’s algorithm improvements? The paper should include concrete examples of the LLM’s input and output (e.g. a sample prompt and the suggestions it gave for improving an algorithm).\n\n2. How sensitive are Crucible’s results to the choice of LLM (e.g., Claude vs. GPT-4)? Would different LLMs yield significantly different tuning outcomes? It would be beneficial to include a sensitivity analysis across multiple LLMs to assess robustness.\n\n3. The Crucible agent simulates different expert skill levels by limiting tool iterations and reflection steps, but how do we know these settings realistically correspond to expert abilities? Can the authors justify or validate this design? \n\n4. The manuscript uses Wasserstein distance for ABR and a proportion-based metric for scheduling. Can you include the mathematical equations for these metrics? How these metrics were chosen, and how sensitive are the results to these choices?\n\n5. How well do the LLMs simulate real expert behavior? Are there any benchmarks or validations comparing LLM-generated tuning strategies with those from human experts? Have the authors considered comparing against real expert adjustments or other algorithm-tuning techniques? For example, for one of the simpler scenarios, one could ask a human expert to manually fine-tune the algorithm and see how much improvement they get, to check if Crucible is on par or discovering similar enhancements.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

comments: SUMMARY:
The paper proposes the Crucible framework, which is the first to use LLM to simulate multi-level experts to tune the control algorithm, thereby calculating the tuning potential of the control algorithm. Through tests in two classic fields, ABR and scheduling algorithms, the effectiveness of the framework was verified, and the importance of the algorithm's tunability as a new optimization goal was explored, which can be customized according to real application scenarios.

STRENGTHS AND WEAKNESSES:
Pros:
1. Using "tunability" as a new dimension for control algorithm design and evaluation hits the pain points in the industrial deployment process.
2. Using LLM as an expert agent to simulate actual developers to adjust the algorithm at the logic and parameter levels is novel and practical.
3. The detailed pseudo-code of Crucible is provided in the appendix.

Cons:
1. Currently, Crucible does not support optimization of the internal logic of black-box algorithms (it can only adjust the distillation version of the decision tree), which is a common situation for many industrial algorithms in reality.
2. This article mainly focuses on two tasks: ABR and scheduling. The experimental results are not comprehensive enough.

QUESTIONS:
1. The article mentioned that different LLMs will affect the final results. Can you add corresponding comparative tests? I think this is very helpful for the effectiveness of the algorithm.
2. The scheduling algorithm uses a score-based ratio as the environmental distance, which is tricky. Can more analysis or more detailed environmental distance modeling be added?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Crucible framework, a novel approach to quantify the \"tuning potential\" of control algorithms, defined as the performance improvement achievable through parameter and logic adjustments by domain experts in production environments. Crucible leverages large language models to simulate developers of varying expertise levels, using multi-level expert simulation and environment distance metrics (Wasserstein distance for ABR, initial performance ratio for scheduling). The framework evaluates adaptive bitrate (ABR) algorithms and distributed scheduling algorithms on standard datasets, using domain-specific metrics (QoE for ABR, flow completion time (FCT) for scheduling). Results demonstrate Crucible’s ability to quantify tuning potential, revealing the impact of representation capacity and logic complexity on optimization. The paper discusses LLM stability, limitations in improving black-box algorithms, and societal impacts.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- This paper introduces the concept of “tuning potential” for the first time and quantifies it through expert simulation based on large language models (LLMs), environmental distance regularization (ABR uses Wasserstein distance, scheduling uses ratio benchmarks), and a standardized interaction interface (Section 3.3), providing a pioneering method for the interdisciplinary field of control algorithms and machine learning.\n- The application of Crucible in ABR and scheduling demonstrates cross-domain generalizability. Results (e.g., Figure 5(b), showing that optimized SJF outperforms multi-level feedback) provide new insights for algorithm design, emphasizing the importance of logical simplification and representational capability.\n- The study encompasses multiple algorithms (ABR: BBA, MPC, HYB, BOLA, Prince; scheduling: SJF, FIFO, Round Robin, Tetris) and standard datasets (Obee, FCC, Puffer, Spark), using domain-specific metrics (QoE, FCT) and validated through Bayesian optimization (0, 10, 20 iterations). Figures 3 and 5 validate the framework's effectiveness.\n\n**Weaknesses:**\n\n- The definition of optimization potential (Section 3.2) relies on environment distance normalization, and the definition of environment distance varies depending on the task, lacking a unified mathematical definition, which limits its theoretical persuasiveness.\n- Multi-level expert simulation employs Bayesian iteration (0, 10, 20) and reflection steps (1, 2, 3), but the basis for defining capabilities (e.g., prompt design) is not explicitly stated, potentially affecting reproducibility.\n- Experiments rely on simulated environments (ABR simulator, Spark simulator) without validation on real-world production systems, limiting their credibility.\n- Comparisons with existing optimization methods (e.g., hyperparameter tuning, reinforcement learning) are incomplete, weakening the claim of originality.\n\nQUESTIONS:\n1. The tuning potential metric (Section 3.2) uses task-specific environment distance metrics (Wasserstein for ABR, ratio-based for scheduling), lacking a unified mathematical definition. Can you provide a formal derivation or a generalized metric that ensures cross-task comparability? For example, could a single distance metric (e.g., based on performance distributions) be applied across domains?\n2. The multi-level expert simulation uses Bayesian iterations and reflection steps to model varying expertise levels, but the basis for capability delineation is unclear. Can you specify the quantitative criteria for these levels and validate their impact on results (e.g., through ablation studies)?\n3. Experiments rely on simulated environments without real-world production system validation. Can you provide results from real-world or near-real-world settings? How do these results compare to simulations?\n4. Comparisons with existing optimization methods (e.g., hyperparameter tuning, reinforcement learning) are limited, weakening originality claims. Can you provide a detailed comparison with methods like Bayesian optimization or RL-based tuning?\n\nLIMITATIONS:\nThe paper demonstrates Crucible’s applicability to ABR and scheduling but does not discuss the limitations of extending the framework to other domains (e.g., robotics, real-time systems) where control algorithms may have different constraints or data distributions. The task-specific environment distance metrics (Wasserstein for ABR, ratio-based for scheduling) may not generalize well.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}