Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Authors: Xu Wan
Wenyue Xu
Chao Yang
Mingyang Sun
Venue: ICML 2025 poster (2025)
Keywords: Power Grid Control
Large Language Models-enhanced Reinforcement Learning
Primary area: applications->energy
TL;DR: We present a co-evolution framework where large language models and reinforcement learning agents enhance each other's capabilities to achieve efficient learning and rapid execution in large-scale numerical decision-making.

Abstract: Recent advancements in Large Language Models (LLMs) and Reinforcement Learning (RL) have shown significant promise in decision-making tasks. Nevertheless, for large-scale industrial decision problems, both approaches face distinct challenges: LLMs lack real-time long-sequence decision-making capabilities, while RL struggles with sample efficiency in vast action spaces. To bridge this gap, we propose **A**gents **C**o-**E**volution (ACE), a synergistic framework between LLMs and RL agent for large-scale decision-making scenarios. ACE introduces a dual-role trajectory refinement mechanism where LLMs act as both Policy Actor and Value Critic during RL's training: the Actor refines suboptimal actions via multi-step reasoning and environment validation, while the Critic performs temporal credit assignment through trajectory-level reward shaping. Concurrently, RL agent enhance LLMs' task-specific decision-making via prioritized experience replay.

Through extensive experiments across multiple power grid operation challenges with action spaces exceeding 60K discrete actions, ACE demonstrates superior performance over existing RL methods and LLM-based methods.

Think Twice, Act Once:
A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Xu Wan Wenyue Xu
Chao Yang²
Mingyang Sun
Abstract
et al., 2019; Kalashnikov al., 2022), require sophisticated reasoning capabilities and rapid response mechanisms. Re- inforcement Learning (RL) has been extensively studied as a promising approach to tackle these challenges for a long time. Through iterative interaction with the environ- ment, RL agents learn optimal control policies by max- imizing cumulative rewards. However, the paradigm of learning from scratch without prior knowledge (Sutton & Barto, 1999) making RL agents inherently suffer from sam- ple inefficiency. Moreover, the time-varying and stochastic characteristics of large-scale industrial scenarios result in a substantial gap between the converged solutions obtained through RL models and true optimality (Nian et al., 2020).
Recent advancements in Large Language Models (LLMS) and Reinforcement Learning (RL. have shown significant promise in decision-making tasks. Nevertheless, for large-scale industrial de- cision problems, both approaches face distinct challenges: LLMs lack real-time long-sequence decision-making capabilities, while RL struggles with sample efficiency in vast action spaces. To bridge this gap, we propose Agents Co-Evolution (ACE), a synergistic framework between LLMs and RL agents for large-scale decision-making scenarios. ACE introduces dual-role trajec- tory refinement mechanism where LLMs act as both Policy Actor and Value Critic during RL's training: the Actor refines suboptimal actions via multi-step reasoning and environment vali- dation, while the Critic performs temporal credit assignment through trajectory-level reward shap- ing. Concurrently, RL. agent enhances I.LMs' task-specific ppcisiinnmmkingg with high-quality fine-tuning datasets generated via prioritized ex- perience replay. Through extensive experiments across multiple power grid operation challenges with action spaces exceeding 60K discrete actions, ACE demonstrates superior performance over ex- isting RL methods and L.LM-based methods.
Various approaches leverage expert knowledge alleviate the inefficiency and sub-optimality intrinsic in RL training. In terms of policy guidance, straightforward yet effective approach is Learning from Demonstration (LFD) (Argall et al., 2009). LFD-based methods directly mimics expert demonstrations through behavior cloning or tries to optimize the reward function through Inverse Reinforcement Learn- ing (IRL) (Ng al., 2000; Nair et al., 2018; Torabi et al., 2018; Adams et al., 2022). While theoretically promising, LFDD-based methods heavily rely on the quality of expert demonstrations and struggle to generalize to unseen scenar- ios, often suffering from distribution shift problems when encountering states not covered in the demonstration data. Meanwhile, Muman-in-the-Love (HITL) approaches (Abel et al., 2017) has emerged as promising paradigm that leverages expert feedback adaptive guide policy learn- ing. These methods incorporate human guidance either through real-time interaction (Knox & Stone, 2009; Mac- Glashan et al., 2017) or offline preference alignment (Bai et al., 2022: Dai et al., 2023), enabling human trainers to pro- vide evaluative feedback or corrective advice during policy learning. However, due to the unpredictable and subjective nature of human feedback, HITL--aasd approaches face key challenges including intensive human time requirements, feedback inconsistency, and the balance between human intervention and autonomous learning (Kumar et al., 2024).
1. Introduction
Making effective control in large-scale physical systems has been a long-standing goal in artificial intelligence research (Stoica et al., 2017). Such decision-making tasks, including traffic control (Zhang et al., 2022; Du et al., 2023a), power system operating (Yoon al., 2021; Dorfer al., 2022; Chauhan al., 2023), and multi-robot coordination (Mahler
'Zhejiang University TTongii University 3 AAlibaba DAMO Academy "Peking University, Correspondence Mingyang Sun <smy @ @ww.eeuuu...c...
Recent advances Large Language Models (LLMs) have opened new possibilities for enhancing RL frameworks. Through training on massive text data, LLMs have acquired
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
rich world knowledge and reasoning capabilities, making them promising candidates for decision-making tasks (Lu et al., 2024; Wang et al., 2023). Researchers have explored multiple LLM-RL integration paradigms: 1) semantic action space compression through L.L.M-uided abstraction (Zhou et al., 2023). 2) reward shaping via natural language instruc- tion following (Tan et al., 2024), and 3) code generation for policy implementation (Du et al., 2023b). Recent work like Thought Cloning (Liu al., 2024) further demonstrates how LLMs can generate chain-of-thought trajectories to guide policy learning through structured reasoning traces. However, these approaches face significant challenges in industrial control scenarios. While LLMs excel at high- level strategic planning, they struggle with long-sequence gecision-making required for fine-grained control. Indus- trial tasks exhibit inherent time delays and temporal cou- pling, making it challenging for LLMs generate coherent sequences of decisions over extended time horizons. Ad- ditionally, the autoregressive nature of transformer-based LLMs introduces significant latency (Fu et al., 2024), mak- ing them impractical for real-time control loops requiring sub-1000ms responses. Our analysis suggests that direct LLM-based action generation fundamentally conflicts with the precision and timing requirements of industrial con- trol systems, necessitating new paradigm that combines LLMs' strategic reasoning with traditional RL's numerical optimization strengths.
Critic, addressing sample inefficiency and solution sub- optimality, respectively.
(3) For enabling continuous improvement of RL and LLMs, we establish an automated high-quality dataset generation workflow through reward-based prioritization and weighted policy update strategies in experience gathering.
(4) We demonstrate state-of-the-ar performance on L2RPN competitions, outperforming existing expert-guided RL methods and LLM methods in industrial environments with over 60K action space.
2. Related Work
Expert-Guided Reinforcement Learning Expert guid- ance has been widely explored address the sample ef- ficiency and exploration challenges inherent in RL. Imi- tation learning-based methods, like DAgger (Ross et al., 2011), Hg-DAgger (Le et al., 2018), Soft DAgger (Nazeer et al., 2023) iteratively collect expert feedback on states visited by the learning policy to address distribution shift. Instead of direct action imitation, IRL-based methods se- lect reward function from the set of possible solutions that best explains expert behavior. AIRL (Fu al., 2017), MaxEnt-IRL (Zeng et al., 2022), and Offline IRL (Zeng et al., 2023) leveraging different optimization principles to recover more robust reward functions. For industrial sce- narios with large action spaces, (Yoon et al., 2021) reduces the action space based on sample frequency of offline ex- pert data, while (Chauhan et al., 2023) employs predefined rules to simplify RI. decision-making tasks. However, these methods often sacrifice the exploration of optimal solutions. Alternativeyy, (Dorfer et al., 2022) utilizes monte-carlo tree search to guide policy exploration through look-ahead plan- ning, but it incurs significant computational overhead from extensive simulations.
Motivated by these observations, we develop a more suitable LLM-RL collaborative framework for industrial decision- making. Unlike previous works that integrate LLMSs during inference, we restrict LLM an offline guider in the train- ing phase, preserving real-time performance during online interaction and deployment. Specifically, in the "think twice" phase, LLMs guide RL. policy updates through two key mechanisms: (1) rectifying interaction trajectories as a Pol- icy Actor to refine suboptimal decisions, and (2) performing trajectory- level reward shaping as a Value Critic enable better credit assignment. In the "act once" phase, RL agent interacts with the environment to generate RL trajectories, which are then combined with LLM-refined trajectories to create a mixed experience buffer. This buffer serves both as a training buffer for RL and as a fine-tuning dataset to enhance L.M''s task-specife guidance capabilities. To sum- marize, our main contributions are:
Unlike previous approaches that rely on extensive offline expert data look-ahead simulation, the ACE framework leverages LLM's superior reasoning and in-context learning capabilities to perform trajectory refinement and reward shaping from RL's demonstrations.
Language Models in Decision-Making Focusing on com- bining LLMs capabilities in control tasks, several works adapt LLMs directly generate executable actions. For instance, CALM (Yao et al., 2020), TWOSOME (Tan et al., 2024) and POAD (Wen et al., 2024) constrain the action space to a restricted subset using LLMs and employ RL to align agents' knowledge with specific environments. Sim- ilarly, various approaches decompose complex tasks into manageable subgoals. SayCan (Ahn et al., 2022), LgTs (Shukla et al., 2023) and DART-LLM (Wang et al., 2024) break down high-level instructions into executable skills using affordance functions and validity scores, while ReAct
(1) We propose Agents Co-Evolution (ACE), a co-evolution framework that separates LLM reasoning and RI. execution into offline training and online deployment, enabling both effective learning and real-time decision-mann -aainng in large- scale industrial scenarios.
(2) For guiding efficient exploration in large-scale decision- making, we develop dual-role trajectory refinement mech- anism where LLMs serve both Policy Actor and Value
2

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Agents Co-Evolution
Hard for Large Numerical Action Space
Response Sequential Decision
LLM as a decison-maker
Large Action space: R Ph
vector to language
language
reasoning....
to
Up 60K discrete actions
ff 100ma)
4 10a)
goppl Semo
Low Learning Efficiency
High Gap Optimal
Environment
High Gap
Specde ted)
Low Quality
Higher learning efficiency convergence rewards
More diverse samples Higher Quality dataset
RL as a decison-maker
Figure 1. The motivation and challenges of integrating LLM with RL for large-scale industrial decision making.
(Yao et al., 2022) utilizes chain-of-thought prompting generate task-specific actions with reasoning traces.
is the transition probability, R: A IR is the reward function, and y (0, 1] is the discount factor.
Furthermore, recent studies have explored integrating LLMs as policy experts to guide RL agents' interactions (Nam et al., 2023). LLM4Teach (Zhou et al., 2023) incorporates LLM guidance by introducing policy regularization terms in the RL optimization process. ELLM (Du al., 2023b) guides RL policy pre-training through LL-M-suggested goals, while LINVIT (Zhang al., 2024) incorporates LLM guidance as regularization factor for value-based RL. Most closely aligned with our motivation is RL-GPT (Liu et al., 2024), which unifies coding and learning optimization in the RL training pipeline to help RL systems learn better decision decision-making strategies. However, RL-GPT relies on continuous LLM interaction during game tasks, resulting in training costs and real-time requirements that exceed the
Optimization Objective. Efficient policy exploration and experience collection are crucial for handling large-scale numerical action spaces. Therefore, we adopt the Soft Actor- Critic (SAC) (Haarnoja et al., 2018) algorithm as our RL module through off-policy training for better experience utilization. The SAC agent aims to learn policy that maximizes both the expected return and policy entropy. Our objective function is:
J(T) = ETAT
Σ 1=0
(R(s,, az) (+H(T((()))) (I)
where H(x(-18=)) = - (aa(st) log (aa(s)) represents the entropy of the policy state st, is the temperature parameter controlling the trade-off between exploration and exploitation.
constraints of industrial applications.
3. Method
Learning from Interaction. During standard environ- ment interaction, transition tuples т (st. at, Tt, S1+1, d) are stored into replay buffer DRL for each timestep. Each tuple consists of the current state &t, the action a, sampled from RL's policy π (ar[s:), the received reward r,, and the next state St+1- To learn from experiences, mini-batch of transitions is sampled from DRL optimize both the Q-function Qe and policy me. Specifically, the Q-function is updated by minimizing:
In this section, we describe the ACE framework, consisting of I) direct policy learning through RL's environmental interaction, 2) trajectory refinement through LLM's s dual roles as Actor and Critic, and 3) experience gathering that enables effective co-evolution between the two agents. The pseudo-code of ACE is shown in Algorithm 1.
3.1. First Think: Direct Policy Learning through Environment Interaction
(x, a) - - (o( = ( [(Q
We first formulate the sequential decision-making problem as a Markov Decision Process (MDP) (Puterman, 1990) defined by tuple (S, P, 7), where and A denote the state and action spaces, respectively. P: S + [0,
= r (Qx (s', a log Te(a')s')
where o' denotes the target network parameters and a' is sampled from the current policy æo. The policy then
3

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
where P. is the task description prefix that supplies nec- essary domain knowledge, (-(-) and (-).) denote the text conversion functions that transform states and actions into natural language descriptions, respectively. The refined ac- tion 0e is then selected as the action from the L.M'' output and subsequently validated through environment simulation to obtain new transitions (st, ae, (x( oz), $141, d), which are stored separate LLM buffer PLLM-
LLM as Policy Actor
Value Critic
Sampled
RL Buffer
Sampled
Batch
Episode |P:
Bad Case
Tulrl, Falllewn))ss
Key Case
Observation Language Bad Language
1: Trajectories Language
Juz
Key Language
BLAM
Re-Decision
3: Re-score
LLMs as Value Critic In later training stages, we lever- age LLMs to perform trajectory-level counterfactual reason- ing and re-evaluate the long-term impact of key decisions:
Pizse
Re-decision Fra case
Re-score case
LLM Buffer
Experience Prioritization
A = 1IMMM (T) ae)llr(se at)l > r})) (5)
M--F)= eapl (5
Pai
Pas(s.o)
(t, at, Ft, 8141, d)) represents the com- plete episode trajectory, denotes the threshold for identify- ing potentially critical trajectories. This mechanism can be viewed extension of implicit multi-step TD(A) (Sut- ton, 2018): LLMs approximate eligibility traces through trajectory-level reasoning, enabling credit assignment over broader temporal horizons. Compared to TD(A)'s exponen- tial decay assumption, LLM's causal attribution provides non-parametric modeling capability for long-term depen- dencies.
where r = {(8,,
Weighted Policy Update (ss()) E ( qulnaa
RL Buffer
Mix Buffer
Experience Gathering
Figure The architecture of ACE framework with dual-role LLM.
improved by minimizing:
C (0) = Dm. [aaw log (a(ss) - (s)) (3)
To maintain stability while allowing meaningful adjust- ments, we discretize the reward modifications into four levels: {-2K, -K, +K, and limit the number of modifications per episode, where K is a predefined adjust- ment scale.
As shown in Eq. (2) and Eq. (3), the learning efficiency heavily depends on the target value computed using both Q-function and policy. However, in early training stages with large action spaces, both the Q-function estimation and policy exploration are unreliable. It directly motivates our second think mechanism, which leverages LLM's 's reasoning capabilities to enhance the quality of experiences in DRI.-
3.3. Co-evolution through Experience Gathering
To enable effective co-evolution between RL and LLM, we further construct a mixed buffer Danx that serves both RL policy training and LLM online fine-tuning through the following sampling strategy.
3.2. Second Think: Trajectory Refinement through LLMs
In this section, we introduce a dual-role refinement frame- work in which LLMs function as both an Actor for policy refinement and Critic for value re-assessment, as shown in Figure 2. It is driven by two key insights: (1) LLMs' strong in-context reasoning capabilities in inferring better alternative policy from failure cases; (2) LLMs' counter- factual reasoning abilities facilitate better credit assignment over long-term dependencies.
Let 3 denote the ratio of samples drawn from the LLM buffer and denote the sampled transitions. The base mix- ing distribution is:
Pmix (T) = (1 - 3) (T) + (PLLM(T)
(6)
where (T) and (T represent the sampling distribu- tions for DRL and DLLM respectively.
To ensure the quality of L.LM-refined experiences, we em- ploy two key mechanisms:
LLMs as Policy Actor When activated, LLMs serve as a policy actor to refine suboptimal decisions. For batches sampled from the replay buffer DRL, we identify transitions where the reward < I, marking them as inappropriate decisions that require refinement. These states are converted into natural language descriptions and provided to the LLM along with essential context:
Reward-based Experience Prioritization First, we eval- uate the quality of LLM refinements based on the immediate outcomes and termination signal:
Te(r)) = T[f(,, a) 2 AΛ (s')
where d(3²) E {0, 1] indicates whether the next state á' after
à = fiLM Ty(ss) Ta(a)))) <r)
(4)
L.L refine leads to episode termination (1) or continuation

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
(0), and f(s, a) 2 0 refers to the non-negative reward after LLM's refinement.
Algorithm Agents Co-Evolution (ACE) Framework
Require: Initial RL policy тя. Q-function Qs. model fLLM
and 9LLM, buffers DRL. PLLM
To prioritize high-reward experiences during sampling, we introduce a reward- based importance weight:
1: // Act Once
2: for each environment step do
(xp(r(s,a)/B) 1o (7) = cxp(r(s,a)/3)
3: Sample action d, =T((ss))
(8)
4: Execute observe $1+1. done marker de
5: Store T : (8t, Ot, ft, $111, de) in DRI.
where wy (T) serves as a prior for experience replay, ensur- ing that the refined transitions by fLLM or 9LLM leading to higher rewards than RI. agent are more likely to be sampled.
end for
I/ Think Twice
for each LLM active step do
Sample batch DRL
Therefore, the complete mixed sampling distribution is re- formulated as:
10:
if fILM is active and < in then
11: Convert state-action to text: T, (s:),, (a:)
f
if E DRL if E DLIM
12:
Get refined action å by Eq. (4)
Pmixx (T)
Dac. (-I(()).u.r)) DUM (T)))
(9)
13:
end
14:
9LLM active and [rx] > F then
15: Convert trajectories to text: T7
Reware-weighted Policy Learning To prioritize learning
16:
Get refined rewards Ó. by Eq. (5)
from valuable LLM refinements, we introduce reward- based weighting mechanism for RL agent:
17:
end if
18: Simulate by Grid2Op and store refined transitions
(st, âr, fr, 8141, de) in DLIM
19: end for
(0)
EE 7ropmis
(T)
E lag Qs(s, a)ll
20: for each RL update step do
21: Sample mixed batch according to Pmix in Eq. (9)
(10)
22: Compute importance weights w-(() by Eq. (8)
(T) naturally emphasizes LLM refinements that lead to higher rewards, allowing the RL agent to focus on learning from more valuable demonstrations.
23: Update Q-function by (2)
24: Update RL policy with weighted loss by Eq. (10)
25: end for
For the LLM module, we apply low-rank adaptation fine- tuning on Dmix with reward signals as labels to enhance the task-speeific capabilities.
26: for each LLM fine-tuning step do
27: Generate buffer Dmix according to pmix in Eq. (9)
28: Fine-tuning fLLM and 9LLM using Dmix
29: end for
4. Experiments
4.1. Environmental Setup
10 different episodic scenarios, each spanning 3 days (864
steps) with varying difficulty levels.
We evaluate ACE in three complex real-world power system operation cases. The data is sourced from the Grid2Op open- source platform (Donnot, 2020) provided by RTE France, Europe's 's largest grid operator. Specifically, use datasets follows:
(2) L2RPN NeurIPS 2020 Challenge (Marot et al., 202 This challenge extends the WCCI 2020 grid environment by introducing adversarial setting. The key innovation is the addition of an "opponent" that follows heuristic strategy of randomly disconnecting heavily loaded power lines. The test dataset is conducted on 24 weekly episodes (2016 steps), with control decisions made at 5-minute intervals.
(1) L2RPN WCCI 2020 Challenge (Marot et al., 2020) This challenge presents medium-sized power grid chal- lenge that simulates one-third the US Midwest grid. The environment consists of 36 substations, 59 lines, and 22 generators, with an extensive topology assignment action space exceeding 000 possible actions (Yoon et al., 2021).
(3) L2RPN WCCI 2022 Challenge (Marot et al., 2022): This competition utilizes the industry standard synthetic IEEE-118 power grid environment, featuring 118 substa- tions, 186 power lines, 91 loads, and 62 generators. It introduces more renewable generators and storage systems than previous challenges, focusing on electricity production uncertainty and Al agent robustness. The action space is significantly larger, with over 70,000 discrete actions for substation switching alone (Dorfer et al., 2022). The train-
The objective is twofold: (1) developing strategies to over- come operational obstacles like grid congestion, and (2) optimizing various operational costs including power line losses, dispatch costs, and outage costs. The challenge in- corporates realistic scenarios such as load fluctuations and line maintenance. For evaluation, the test dataset comprises
5

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
ing dataset spans 32 years of grid data at S-minute intervals (2016 steps), totaling approximately 1.7 GB.
generated samples, while ACE (GPT-4) operates without fine-tuning. Besides, we activate the LLM Value Critic mod- ule 9LLM only in the later stages of training (PLLM is full), where the modified rewards directly replace the original rewards without additional storage.
4.2. Baseline Methods
We benchmark the ACE framework against state-of-the-art approaches across three categories:
4.4. Main Results
(1) Expert-guided RL: We compare with the winning solu- tions from previous challenges. Building upon the WCCI 2020 winning solution (Yoon et al., 2021), which leverages hierarchical policy and after-state representation, we adopt its architecture as our backbone but enhance it with our dual-role LLM refinement mechanism.
We evaluate ACE against baseline methods across three L2RPN competition environments. As shown in Figure 3 and Table I, ACE consistently outperforms all baselines across different metrics.
From the perspective of episode rewards, ACE demon- strates substantial improvements across all three challenges, including the champion solutions from WCCI 2020 and Neurips 2020. In the WCCI 2020 challenge, ACE achieves the highest episode reward, surpassing the pure RL approach by 22.2% and pure LLM approaches by over 130%. Similar results are observed the NeurIPS 2020 challenge, where ACE achieves 145.3, outperforming other methods by at least 10.5%. Most notably, in the more complex WCCI 2022 environment, ACE achieves remarkable 145% im- provement over the Expert-guided RL baseline.
(2) LLM only: We evaluate the direct decision-making performance of different pre-trained LLM models: GPT- 4o-0806 (Achiam et al., 2023) and Owen2-7B-Instrue (Bai et al., 2023) using the same prompts as ACE on the test set.
(3) LLM-guided RL: Given that existing LLM-RL meth- ods typically operate during inference, which impracti- cal for L2RPN's large-scale tasks, we implement a mod- ified version of LLM4Teach (Zhou et al., 2023): using KL-divergence constraints to regularize the RL policy To 's updates by minimizing the deviation from the LLM guider's policy distribution TLLM-
From the decision-making efficiency perspective, ACE maintains competitive real-time performance. In the WCCI 2020 challenge, ACE (GPT-4) achieves a test time of 38.7s, which comparable to the expert-guidee RL baseline and significantly faster than pure LLM approaches. This phe- nomenon further demonstrates that for large-scale sequential decision decision-making problems with long time series, the running time of pure LLMs does not meet industrial requirements.
(0) = CRL(() + AE (LLLLMM (8) (s)ss))()))) (11)
As illustrated in Eq. (11). the modified LLM4Teach adapts the mode similar to ACE, where alignment is only enabled during the training phase with Qwen2-7B-Instruct as the aligned model.
In terms of sample efficiency, we find that by injecting refinements for less than 700 selected samples, the con- vergence speed of RL can be significantly improved. For the WCCI 2020 challenge, ACE requires only 287 LLM refinements to achieve state-of-the-ar art performance, com- pared to 100K samples needed by traditional approaches. This efficiency is also evident in the WCCI 2022 challenge, where ACE achieves superior performance with just 50K samples plus refinements, while baseline methods require 200K samples yet achieve lower performance.
4.3. Implementation Details
We utilize Qwen2-7B instruct and GPT-4o-0806 as the ACE framework's 's LLM component, named ACE (Qwen2-7B) and ACE (GPT-4), respectively.
For trajectory refinement, we introduce two effective tricks: bad case reasoning and multi-round reasoning mechanism. The bad case reasoning mechanism integrates RL's bad ac- tions of the current state into fLLM'S 's prompt and formulates instructions that prompt LLMs to reason and avoid making similar suboptimal decisions. Moreover, multi-round rea- soning leverages the simulation functionality provided by Grid2Op to estimate the rewards following LLMs' refined actions. If the estimated rewards are inferior to RL agent, the system initiates multi-round decision-making, with a maximum of five rounds set in our experiments.
4.5. Ablation Studies
To comprehensively assess the efficacy of each component within the ACE framework, we conduct series of abla- tion studies of two pivotal modules, fuLM and 9LLM, as well as two reasoning strategies used for fLLM interaction: multi-round reasoning and bad-case reasoning. The detailed results are summarized in Table 2 and Table 3.
For efficient memory management, we limit the LLM buffer's ' maximum size to 256. When updating policy, we sample batches 7 according to importance weights (.) from Dmix for RL training. In ACE (Qwen2-7B), we per- form online fine-tuning of the L.M model for every 100
First, we examine the impact of removing the core mod- ules. As demonstrated in Table 2, eliminating the Actor component fLLM leads to the most pronounced performance
6

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
WCCI 2020 Challenge
NeurIPS 2020 Challenge
WCCI 2022 Challenge
81
192
170
Episode Reward
52
124
110
23
57
50
5
-10
-9
1
3
2
0
2
3
5
100
100
100
Survival Rate
66
66
66
33
33
33
3
4
5
Episodes
Episodes
Episodes
x10
x103
x10
RL
LLM4Teach
ACE(GPT-4)
ACE(Qwen2-7B)
Figure Training episode rewards comparison between ACE and baselines across L2RPN competitions.
decline, with the episode rewards plummeting to 48.3 and the survival rate dropping to 71.4%. Similarly, the removal
80
80
Episode Reward
Survival Rate (%)
60
60
of the Critic component 9LLM also results in a notable de- crease performance, with episode rewards reaching 61.5 and survival rate of 84.7%.
20
20
Additionally, we analyze the impact of the reasoning strate- gies. The results in Figure. 4 show that removing the bad case reasoning strategy reduces the rewards to 60.2 and the survival rate to 76.3%, while the removal of multi-round reasoning leads rewards 65.7 and a survival rate of 88.5%. This demonstrates that using bad actions neg- ative examples effectively leverages the LLM's reasoning capabilities to generate superior decisions.
0
1000
2000
1000
2000
Episodes
Episodes
RL
w/o multi round
Full ACE
w/o bad case
Figure 4. Ablation study results comparing ACE variants over training episodes on NeurIPS 2020 dataset.
Furthermore, we conducted an in-depth analysis of key hy- perparameters within fLLM and ILLM in Table 3.
impracticality using complete trajectories for LLM-based reward shaping, we explore three key trajectories selection criteria to reduce 9LLM input tokens: reward-base selec- tion filter trajectories by reward absolute values to capture high- and low-reward cases, state-based selection identify trajectories where maximum line flow change exceeds a threshold [Prax| > p. and ration-based selection designate trajectories with action changes * 0 as key trajectories; and (3) to prevent policy oscillation in Q-value estimation, test shaping parameters K {0.2, 0.3, 0.4) with 9LLM fixed at a 32-step query interval.
For fILM, we analyze three key aspects: (1) we test fLIM query intervals of 128, 256, and 512 training steps with 9LIM query interval fixed at 32 to assess the impact of fuLM' 's activation frequency on convergence and efficiency; (2) we explore bad-case thresholds (-0.3, O, 0.3) to regulate LLM- refined sample volume; and (3) we evaluating "No SFT", "One-time SFT" at RL's 2000th epoch, and "three-time SFT" per 100 refined samples quantify the fine-tuning effects on performance.
For 9LLM- also analyze three key aspects: (1) test OLLM query intervals of 32, 128, and 512 training steps to as- sess the impact of 9LLM'S 's activation frequency; (2) given the
The experimental results highlight several critical insights:
(1) Higher activation frequency leads to faster initial RL learning. 256-step interval of fUL and 32-step interval of

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Table Test Performance comparison across 3 1.2RPN competitions. Results show + standard deviation 5
Best results
are in bold.
Dataset
Method
Type
Episode Rewards
Survival Rate (%)
Sample Requirements
Test Time(s)
Expert-guided RL
RL
57.1 3.9
78.3 3.9
100
46.
Qwen2-7B
LLM
21.9 1.9
30.5 = 0.8
N/A
1480.8
WCCI 2020
GPT-4
LLM
29.5 19.5:22 2.1
41.7 = 1.5
N/A
3415.3
LLM4Teach
LLM+RL
64.4 + 4.2
78.5 + 2.4
100 K
46.2
ACE (Qwen2-7B SFT)*
LLM+RL
69.8 + 1.9
92.9 + 0.0
40 K + 287
54.8
ACE (GPT-4)
LLM+RL
67.7 + 1.8
92.7 + 0.0
40 K + 364
38.7
Expert-guided RI.
RL
128.5 + 5.1
5.5
100 K
217.2
Qwen2-7B
LLM
37.5 1.8
21.8 = 1.0
N/A
10729.7
NeurlPS 2020
GPT-4
LLM
37.8 1.88100 1.9
42.3 1 1.3
N/A
12949.8
LLM4Teach
LLM+RL
131.5 0.3
74.8 = 0.2
100 K
212.9
ACE (Qwen2-7B SFT)
LLM+RL
145.3 - 6.4
84.8 + 3.7
40 K + 400
218.5
ACE (GPT-4)
LLM+RL
143.5 4.3
84.1 + 4.9
40 K + 307
219.1
Expert-guided RL
RL
31.0 2.7
3.0
200 K
1440.0
Qwen2-7B
LLM
16.7 + 0.9
19.6 + 2.6
N/A
22436.0
WCCI 2022
GPT-4
LLM
18.3 0.8
22.2 1.4
N/A
22188.6
LLM4Teach
LLM+RL
23.6 3.4
29.5
200
1116.8
ACE (Qwen2-7B SFT)
LLM+RL
75.9 2.7
54.3 + 2.0
50 K + 564
3627.2
ACE (GPT-4)
LLM+RL
75.4 + 6.3
49.5 + 4.0
50 K + 682
7478.9
"SFT" refers to the process of fine-tuning LLM components using LoRA parameters with the mixed buffer D ACE (GPT-4) does not include SFT, only using GPT for enhancing RL with our ACE framework.
Table Ablation study of ACE components on WCCI 2020
Table 3. Ablation study of fuLM and gum Variants on WCCI 2020
dataset.
dataset
ACE Variant
Episode Rewards
Survival Rate (%)
FULM Variants
GLAM Variants
Full ACE
69.8
92.9
Params
Reward
Survival
Params
Reward
Survival
Active Frequency
Active Frequency
w/o fUL
48.3
71.4
128
66.2
89.1
68.1
94.0
w/o 9LLM
61.5
84.7
256
68.1
94.0
128
64,3
87.4
w/o Bad Case
60.2
76.3
512
63.4
85.3
512
51.5
72.7
w/o Multi-round
65.7
88.5
Bad Threshold
Trajectory Selection
Note: We use ACE as baseline and limit w/o Multi-round Reasoning to process the same samples of RL trajectories.
=
0.3
61.8
86.2
[|| >
68.1
94.0
2
68.1
94.0
Padd P
68.1
94.0
0.3
64.1
88.5
51.5
72.7
SFT Frequency
Reward Shaping
SFT
58.9
77.5
K 0.2
68.0
92.6
ØLLM shows significant improvements 512 steps. Inter- estingly, as the number of training scenarios increased from 288 to 576 cases, the performance gap between different frequencies narrowed, suggesting that we can reduce acti- vation frequency for environments with sufficiently diverse scenarios while maintaining effectiveness.
One-time
64.3
80.4
0.3
64.3
87.5
Three-time
68.1
94.0
K 0.4
68.2
94,0
degradation compared to the standard settings.
(3) Both reward-based and state-based key trujectory se- lection criteria in gLLM improve performance, while action- based selection shows significantly limited performance. This suggests that LLMs are more effective at extracting meaningful patterns from explicit information, such as per- formance indicators or state changes, compared to abstract topological changes.
(2) Extreme thresholds for filtering bad cases are subopti- included 275 0.3 allowed only 83 samples for
mal. r == 0.3 introduced 510 samples, samples, and finement. In our experiments, constrained by the reasoning capability of the base model, introducing too many samples results minimal refinement benefits while increasing fuLM reference time by 46%. Conversely, too few samples led slower convergence and approximately 6% performance
(4) SFT frequency directly impacts ACE performance. No
8

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
SFT showed limited improvement after about 2000 epochs,
ACS1Owen-81
AGE (Quamn-78)
with the survival rate stabilizing at 77.5%. Applying one- time SFT led to immediate improvement, reaching 78.5% in the first post-SFT evaluation step. Moreover, multiple SFT iterations demonstrated further enhancement over single SFT, ultimately achieving a survival rate of 84.8%.
70
0.5
Episoda fewwrds
60
Step Reward
RL
0.0
40
-0.5
Ceenn
-1.0
30
Quwer
GPI-4
4.6. Computational Overheads
0
400
800 Episode
1200
20
10a
10'
Test Time
We conducted experiments on the NeurlPS 2020 competi- tion environment to evaluate the computational efficiency of ACE. For expert-guided RL, we trained for 100K timesteps with a total duration of 6h 4m 14s. Table 4 presents the detailed computational costs and memory requirements of ACE using Qwen2-7B with SFT.
Figure 5. Left: Step-wise reward analysis in refining RL trajecto- ries. Right: Comparison between episode performance and test time for different RI. and L.M configurations on WCCI 2020.
and correction capabilities. As shown in Figure 5 (a), where ACE (Qwen2-7B+SFTT) refined step reward grad- ually increases after SFT while GPT4's refined reward re- mains constant. Additionally, with Dmix fine-tuning, Qwen2- 7B+SFT's independent decision-making performance im- proves by 10.9% compared to the version without SFT, surpassing that of GPT4.
Table Computational overhead breakdown for ACE training
Module
Counts
Samples
Time
ACE-RL
400
3h 4m 41s
ACE-LLM Inference
508
264
1h 48m Os
ACE-LLM Sampling
4981
32
59m 12s
ACE-LLM Training
2
200
26m 10s
5.2. Why using LLMs for Trajectories Refinements Better Than Policy Regularization?
ACE-Total
440K 6h 18m 3s
ACE's additional computation primarily stems from three LLM modules: selective inference, sampling, and training. With fILM and fTLM query intervals set to 256 and 32 respec- tively, only 264 samples and 508 inferences were required during ACE training. The SFT module executed only twice during the entire co-training process. Unlike tradi- tional LLM-RL collaboration that requires continuous LLM interaction, ACE maintains learning efficiency by sampling from the constructed LLM buffer even when fLLM and 9LLM are inactive, significantly improving sample efficiency.
LLM4Teach uses KL-divergence constraints to enforce static alignment between LLM priors and RL policies. How- ever, as demonstrated in Figure 5, pure L.LM policies are not well-suited for sequential Decision-making in industrial see- narios. This policy-level alignment inevitably leads to over- conservative prcision-making as it forces RL policies to match LLM behaviors that are suboptimal for long-horizon control. In contrast, the ACE framework, by reprocessing trajectories offline, performs targeted reasoning key cases to identify critical decision points. The test performance indicates that this selective refinement approach is more suitable for large-scale industrial scenarios.
5. Discussion
5.1. Why LLMs Can Guide RL but Struggle in Sequential Industrial Decision-Making?
6. Conclusion
This paper introduces ACE, framework that synergizes LLMs and RL for large-scale industrial control through a "Think Twice, Act Once" mechanism. We highlight our con- tributions from two perspectives: (1) we develop a more suitable LLM-RL collaborative framework for industrial decision-making leveraging offline LLM refinement for both action refine and reward reshaping: (2) we establish an effective workflow that combines RL interaction and LLM refinement to generate high-quality mixed datasets, provid- ing new insights for LLM applications in industrial tasks. Through extensive experiments on three large-scale power grid competitions, we demonstrate ACE's superior perfor- mance in terms of control effectivvnness, decision-making making efficiency, and sample efficiency.
As illustrated in Figure (a), LLMs exhibit strong single- step reasoning and error-correctoon capabilities, especially in RL's early training stages. However, as shown in Figure 5 (b), L.MMs struggle to independently complete all decisions in an entire episode (approximately 800-2000 consecutive decision steps). This is because industrial data often in- volves control delays and high temporal coupling, while LLMs have difficulty modeling long-horizon dependencies due to their token-based memory bottleneck. Extending con- text windows to retain trajectory history increases inference latency quadratically, violating real-time constraints.
Furthermore, utilizing the mixed buffer constructed from RL and L.M can significantly enhance L.LM's decision-making

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Impact Statement
from human feedback. arXiv preprint arXiv:2310.12775 2023.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Donnot, B. Grid2op-a testbed platform to model sequential
decision making in power systems. GitHub repository, 2020.
Dorfer, M., Fuxjäger, A. R., Kozak, K., Blies, P. M., and Wasserer, M. Power grid congestion management via topology optimization with alphazero. arXiv preprint 1.056222 2022.
Acknowledgements
This work was supported in part by the Natural Sci- ence Foundation of Zhejiang Province under Grant LZ23F030009, and the National Natural Science Founda- tion of China under Grants 22161135201.
Du, W., Ye, J., Gu, Li, J., Wei, H., and Wang, G. Safelight: A reinforcement learning method toward collision-free traffic signal control. In Proceedings of the AAAI con- ference on artificial intelligence, volume 37, pp. 14801- 14810, 2023a.
References
Abel, D., Salvatier, J., Stuhlmüiller, A., and Evans, O. Agent-agnostic tomaa--n--thelooy reinforcement learn- ing. arXiv preprint /tdi::/700.00000 2017.
Du Y. Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, pp. 8657- 8677. PMLR, 2023b.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F, L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint 1:Xiv:110..122: 2017.
Adams, S., Cody, T., and Beling, A. A survey of inverse reinforcement learning. Artificial Intelligence Review, 55 (6):4307-4346, 2022.
Fu, Y., Xue, L.. Huang. Y., Brabete, A.-O., Ustiugov, D., Pa- tel, Y., and Mai, L. Serverlessllm: Low-latency serverless inference for large language models. In 18th USENIX Symposium on Operating Systems Design and Implemen- tation, pp. 135-153. USENIX Association, 2024.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C.,, Fu, C.,, Gopalakrishnan, K., Hausman, K., et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018
Argall, B. D., Chernova, S., Veloso, M., and Browning, B. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint 23Xi:23099.16099 2023.
Kalashnikov, D., Varley, J., Chebotar, Y., Swanson, B., Jon- schkowski, R., Finn, C., Levine, S., and Hausman, K. Scaling up
robotic reinforcement learning. In Conference on Robot Learning, pp. 557-575. PMLR, 2022.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022.
Knox, W. and Stone, Interactively shaping agents via human reinforcement: The tamer framework. In Proceed- ings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
Chauhan, A., Baranwal, M., and Basumatary, A. Powrl: A reinforcement learning framework for robust manage- ment of power networks. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 37, pp. 14757- 14764, 2023.
Kumar, S., Datta, S., Singh, V., Datta, D., Singh, S. K., and Sharma, R. Applications, challenges, and future directions of human-in-the-loop learning. IEEE Access, 2024.
Dai, J., Pan, X., Sun, R., J.,
J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe rlhf: Safe reinforcement learning
Le, H., Jiang, N., Agarwal, A., Dudík, M., Yue, Y., and Daumé III, H. Hierarchical imitation and reinforcement
10

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
learning. In International conference on machine learn- ing. pp. 2917-2926. PMLR, 2018.
Nian, R., Liu, and Huang, B. review on reinforcement learning: Introduction and applications in industrial pro- cess control. Computers & Chemical Engineering, 139: 106886, 2020.
Liu, S., Yuan, H., Hu, M., Li, Y., Chen, Y., Liu, S., Lu, Z., and Jia, J. Rl-gpt: Integrating reinforcement learning and code-as- as-plcyy.. arXiv preprint arXiv:2402.1929) 2024.
Puterman, M. L. Markov decision processes. Handbooks in operations research and management science, 2:331 2:331-434, 1990.
Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J. Chameleon: Plug-and- play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita- tion learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D. L.,, Taylor, M. E., and Littman, M. L. Interactive learning from policy-dependent human feed- back. In International conference on machine learning, pp. 2285 2294.. PMLR, 2017.
Shukla. Y., Gao. W., Sarathy, V., Velasquez, A., Wright, R., and Sinapov, J. Lgts: Dynamic task sampling using Ilm- generated sub-goals for reinforcement learning agents. arXiv preprint arXiv:23 10.09454, 2023.
Mahler, J., Matl, M., Satish, V., Danielczuk, M., DeRose, B., McKinley, S., and Goldberg, K. Learning ambidex- trous robot grasping policies. Science Robotics, 4(26): eaau4984, 2019.
Stoica, I., Song, D., Popa, R. A., Patterson, D., Mahoney, M. W., Katz, R., Joseph, A. D., Jordan, M., Hellerstein, J. M., Gonzalez, J. E., al. A berkeley view of systems challenges for ai. arXiv preprint arXiv:1712.05855, 2017.
Marot, A., Donnot, B., Romero, C., Donon, B., Lerousseau, M., Veyrin-Forrer, L., and Guyon, Learning to run a power network challenge for training topology controllers. Electric Power Systems Research, 189: 189:106635, 2020.
Sutton, R. S. Reinforcement learning: An introduction. A
Bradford Book, 2018.
Marot, A., Donnot, B., Dulac-Arnold, G., Kelly, A., O' S'Slllia,,, A., Viebahn, J., Awad, M., Guyon, I., Pan- ciatici, P., and Romero, C. Learning to run a power network challenge: retrospective analysis. In NeurlPS 2020 Competition and Demonstration Track, pp. 112- PMLR, 2021.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. Robotica, 17(2):229-22) 1999.
Tan, W., Zhang, W., Liu, S., Zheng, I.., Wang, X., and An, True knowledge comes from practice: Aligning Ilms with embodied environments via reinforcement learning. arXiv preprint 2rXiv:2401.141555 2024.
Marot, A., Donnot, B., Chaouache, K., Kelly, A., Huang, Q., Hossain, R.-R., and Cremer, J. L.. Learning a power network with trust. Electric Power Systems Research, 212: 108487, 2022.
Torabi, F,, Warnell, G., and Stone, P. Behavioral cloning from observation. arXiv preprint arXiv: /005.0095.. 2018.
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Overcoming exploration in reinforcement learning with demonstrations. In 20/8 IEEE international conference robotics and automation (ICRA), pp. 6292- 6299. IEEE, 2018
Wang. H., Wang, H., Wang. L., Hu, M., Wang, R., Xue, B., Lu, H., Mi, F., and Wong, K.-F. Tpe: Towards better compositional reasoning over conceptual tools with multi- persona collaboration. arXiv preprint arXiv:2309, 16090, 2023.
Nam, T., Lee, J., Zhang, J., Hwang, J., Lim, J. J., and Pertsch, K. Lift: Unsupervised reinforcement learn- ing with foundation models as teachers. arXiv preprint arXiv:2312.08958, 2023.
Wang. Y., Xiao, R., Kasahara. J. Y. L., Yajima, R., Na- gatani, K., Yamashita, A., and Asama, H. Dart-llm: Dependency-aware multi-robot task decomposition and execution using large language models. arXiv preprint arXiv:2411.0000 2024.
Nazeer, M. S., Laschi, C., and Falotico, Soft dagger: Sample-efficient imitation learning for control of soft robots. Sensors, 23(19):8278, 2023.
Wen, M., Wan. Z., Wang, J., Zhang, W., and Wen, Y. Re- inforcing I1m agents via policy optimization with action decomposition. In The eighth Annual Conference on Neural Information Processing Systems, 2024.
Ng, A. Y., Russell, S., et al. Algorithms for inverse rein- forcement learning. In leml, volume pp. 2, 2000.

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Yao, S., Rao, R., Hausknecht, M., and Narasimhan, K. Keep calm and explore: Language models for action generation in text-based games, arXiv preprint arXiv:2010.02900, 2020.
Yao, S., Zhao, J.. Yu, D., Du, N., Shafran, Narasimhan, K. and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03622 2022.
Yoon, D., Hong, S., Lee, B.-J. and Kim, K.-E. Winning the 12rpn challenge: Power grid management via semi- markov afterstate actor-critic. In International Confer- ence on Learning Representations, 2021.
Zeng. S.. Li. C., Garcia, A., and Hong, M. Maximum- likelihood inverse reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 35:10122-10135, 2022.
Zeng. S., Li, C., Garcia, A., and Hong, M. Understanding expertise through demonstrations: A maximum likeli- hood framework for offline inverse reinforeement learn- ing. arXiv preprint
2023.
Zhang, L., Wu, Q., Shen, J., Lü, L., Du, B., and Wu, Expression might be enough: representing pressure and demand for reinforeement learning based traffic signal control. In International Conference Machine Learn- ing. pp. 116645-26654 PMLR, 2022.
Zhang. S., Zheng. S., Ke. S., Liu, Z., Jin, W., Yuan, J., Yang. Y., Yang, H., and Wang, Z. How can Ilm guide rl? value-based approach. arXiv preprint 161111100..11 2024.
Zhou, Z. Hu, B., Zhao, C., Zhang, P., and Liu, B. Large language model as policy teacher for training reinforce- ment learning agents, arXiv preprint arXiv:2311 133733 2023.
12

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
A. Appendix/ / Experiment Setting
A.1. Environment deseription
We evaluate the ACE framework on the Grid2Op platform (Donnot, 2020), which provides an ceuuustry--tandard power grid simulation environment, as shown Figure 6.
4B
20
23
25
18
28
08/08 00:00
powerline
31
substation
58.71 %
load
generator
57.70
no bus
16
bus 1
bus
Figure The test environment of L2RPN WCCI 2020 and NeurIPS 2020 challenge, consisting of 36 substations (blue circles), 59 transmission lines (blue lines), 22 generators (green marks) and several loads (ycllow marks). Red boxes highlight critical load areas with percentage indicators showing the operational status of power lines.
Below, we describe the key components of the MDP formulation for ACE setting.
The power grid state is characterized by six carefully selected features from the 12 available observation variables. The detailed state shown in Table A. These features selected based on the prior work (Yoon al., 2021) provide compact yet comprehensive representation of the grid's operational state.
State Space
Action Space A
The control actions available to the agent fall into two categories:
Bus assignment: modifying element connections within substations
Line switch: controlling the connection status of transmission lines
Given Nime transmission lines, Nom substations, and Sub(i) elements in the i-th substation, the action space cardinality is:
N
Nine + x Nine
Sub(E)
(12)
==0
For WCCI 2020 and NeurIPS 2020 challenges, the cardinality exceeds 60,000 possible actions, while for the WCCI 2022 challenge with the JEFE-11 188 bus system, it grows further to over 70,000 actions (Yoon et al., 2021; Dorfer al., 2022).
13

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Table Details of Grid2Op environment observation space
Name
Type
Size
Description
Date
The current year, month, day, hour day, minute of hour, day of week.
Active power
float
N
Non
Niad + 2 X Nime
Active power magnitude for all elements.
Reactive power
float
N
Niad + 2 X Nine
Reactive power magnitude for all elements.
pen
Voltage
float
N gen
Nuad + Nine
Voltage magnitude each bus.
Rho
Hlaat
N lnne
The capacity ratio between current flow and thermal limit for each line.
Topology
int
N gon
Noad + x Nune
Bus connectivity status for each element in its corresponding substation.
Line status
bool
Nine
Binary indicator for line connection status.
Time step overflow
int
Vine
Duration counter line overflow events.
Time before cooldown line
int
Nine
Remaining cooldown time hefore lines become operable again. Remaining cooldown time hefore substations become operable again.
Time before cooldown sub
int
N
Time next maintenance
Vine
Time until next scheduled maintenance for each line.
Duration next maintenance
int
Nine
Expected duration of next maintenance period.
Reward Function R Inspired by the setup of the WCCI 2020 winning solution (Yoon et al., 2021; Dorfer et al., 2022), we design a reward mechanism that integrates dual objectives of operational efficiency and system security:
load,
not de clse
prod,
(13)
Afail
load, where prod, represents the instantaneous grid efficiency at time step However, if demand exceeds production it triggers an immediate episode termination with substantial penalty Asil-
loade prod,
> 1),
A.2. Hyper-parameters
We show the hyperparameters used in our experiments, detailed in Table A.2.
Table 6. RL-LLM Collahorative Decision Parameters
Parameter
Value
Parameter
Value
RL Parameters
Replay buffer size
50,000
Training episodes
100,000
State embedding dim
128
Attention heads
8
History window
Batch size
64
Discount (Y)
0.995
Learning rate
5e-5
Target update interval
Soft update (r)
le-3
Max episode length
864 / 2016
Samples drawn ratio
0.5
LLM Parameters
LLM guidance buffer
256
LLM response budget
512 1024
Actor query interval
256
Critic query interval
32
SFT buffer size
10,00
SFT batch size
8
Adjustment scale
0.5
Reward threshold (F&or)
0.1 0.5
Max Token fLLM
5120
Max Token of 9LLM
4096
14

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
B. Appendix More Detail of ACE
B.1. Agent Prompt Detail
Input State-Action Parsing for fLLM
You expert power grid operator. Now, Let's analyze the current situation (2012-1-15 00:30) step by step:
Grid Overview:
Total elements: 177
Operable substations: [16, 23, 26, 21, 9, 29, 33, 35, 1, 4, 7]
Total lines: 59
Overload Lines:
Line id 21 (Usage: 96.30%) connects Substation 15 and Substation 16
Line id 23 (Usage: 98.43%) connects Substation 16 and Substation 18
Line id 24 (Usage: 118.57%) connects Substation 18 and Substation 19
-Line id 25 (Usage: 163.69%) connects Substation 19 and Substation 20
Line id 26 (Usage: 95.56%) connects Substation 20 and Substation 21
LLie id 38 (Usage: 185,42% connects Substation 23 and Substation 26
Line id 39 (Usage: 101.19%) connects Substation 22 and Substation 26
disconnected line!
Crucial Substations:
Substation id 16 current topology:
LLines connected in Bus O: 17 (Usage: 32.23%). 18 (Usage: 54.52%), 19 (Usage: 0.00%). 20 (Usage: 42.02%),
21 (Usage: 96.30%) 22 (Usage: 67.58%), 23 (Usage: 98.43%% 27 (Usage: 34.86%), 28 (Usage: 34.99%), 48 (Usage: 23.52%), 49 (Usage: 23.52%), 54 (Usage: 40.54%)
Lines connected Bus 1: {
Lines disconnected: 0 Substation id current topology:
Linees connected in Bus O: 26 (Usage: 95.56%), 27 (Usage: 34.86%), 28 (Usage: 34.99%), 29 (Usage: 116.08%).
30 (Usage: 44.63%), 36 (Usage: 71.57%)
Lines connected in Bus 1: 0
Lines disconnected: ! Substation id 23 current topology:
Liness connected in Bus O: 30 (Usage: 44.63%), 31 (Usage: 175.90%), 32 (Usage: 56.19%), 34 (Usage: 60.74%),
37 (Usage: 53.60 53,, 38 (Usage: 185.42%)
-Lines connected in Bus 1:
Lines disconnected: 0 Substation id 26 current topology:
Lines connected in Bus O: 36 (Usage: 71.57%), 37 (Usage: 53.60%), 38 (Usage: 185.42%) 40 (Usage:
119.37%), 41 (Usage: 194.56%) (,66)..555. 56 (Usage: 29.90%)
Lines connected Bus I: 39 (Usage: 101.19%)
Lines disconnected: 0
Bad Line Change Examples:
Please AVOID the Line change: () as it is BAD action because it results in reward of -1.0.
Operational Constraints
No lines in cooldown
15

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Task Description Prefix of fLLM
Important Notes:
Limit your changes no more than 5 lines id.
Consider adjusting the topology of their shared (connected) lines. This indirect approach may help redistribute
the load and reduce stress overloaded lines.
3. Reason from the example line changes, and avoid outputting the BAD line change.
Response Format:
Please analyze the situation and provide your response in the following format:
Analysis of critical issues.
Reason and analysis why the provided line change examples are BAD if provided.
3. Propose your response to target line changes (Use ONLY values or 1 in bus id.)] proposed line changes:
{line id: new new_bus_ie id, line id: new .ew_ids
Remember:
1. Use exactly the format shown above. Do not add any bold formatting, asterisks, other special characters.
3. For proposed line changes, only include the chosen lines and the target topology.
Use ONLY values 0 or new bus id.
Consider line cooldown constraints.
Input State-Action Parsing for 9LLM
Episode Overview:
Total steps: 14
Initial time step: 2012-4-23-6-55
Final time step: 2012-8-23-5-15
-Cumulative reward: 34.46
Key Timestep Analysis:
TTime step 201 2012-4-23-6-55:
Action: [73: } Reward: -0.48
Highest line usage: 90.68% (Line 39) Overloaded lines:
Key reason: First/Last step, Significant change in highest usage
Time step 2:15-4-11-5-5:
Action: [58: 0} Reward: 0.90
Highest line usage: 424.50% (Line 0) Overloaded lines:
Key reason: Significant reward change, Significant change in highest usage, Change in number of overloaded
lines, Topology change
Time step 2012-4-21-5-20:
Action: {6: 0} Reward: 0.58
Highest line usage: 128.16% (Line 13) Overloaded lines: 2
Key reason: Significant change in highest usage, Topology change
Time step 2012-4-21-0-30 Action: (15: 17: 0, 20: 1) Reward: 4.58
Highest line usage: 90.55% (Line 15) Overloaded lines: 0
Key reason: Significant reward change, Significant change in highest usage, Change in number of overloaded
lines, Topology change
Time step 2012-4-20-23-55:
Action: {56: 0} Reward: 0.56
Highest line usage: 71.97% (Line 41) Overloaded lines:
Key reason: Significant reward change, Topology change
Time step 2012-4-21-6-5;
Action: {30: }} Reward: -0.73
Highest line usage: 125.50% (Line 41) Overloaded lines:
Key reason: Significant reward change, Significant change in highest usage, Change number of overloaded lines, Topology change
16

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Current State
Original Action
Original Next State
substation
Danger
Assign bus 50 1iee
Ascign bus 1122
14 37
36
P8
More Danger
Assign 1ine
Assign 1ine
1ine
34850
Assign bus
Aseign bus 1ine
10 41
Assign 1ine
gen
Assigo load
Current State
Refined Action
Refined Next State
substar tation 21]
Danger
igu
1ice
Normal
Assign bus :0
1ine
27
Assign bus 11ne
14 28
Assign
1ine
Assign :0 1i3e
1ine
Assign bus
Assign bus
id
Assign
1ead
Figure 7. The demonstration of ACE action refinement in power grid operation. Upper row shows the state transition under original RI. action, where dangerous state (left) hecomes more severe (right) after applying the topology change substation 26 (middle). Lower row demonstrates ACE's improvement, where the refined action on substation 21 successfully transitions the system from dangerous to normal operation.
Task Description Prefix of 9LLM
Remember:
1. Key decision point indices are the time step indices in the episode (starting from 0), select up to most important decision points
Reward adjustments can only be one of +0.4, +0.2, -0.2, -0.4
Both lists must be of the same length and correspond in order
Response Format:
Please analyze the below information and select decision points where the reward estimation might be erroneous. Provide your analysis results in the following format:
Key Decision Point Indices: [X, Y, A, B]
Reward Adjustments: [W, V, T.
Index X (Adjustment W): [Explain why this decision point is important and why this adjustment value was chosen]
Index Y (Adjustment V): [Explain why this decision point important and why this adjustment value was chosen]
3. Index A (Adjustment T): [Explain why this decision point is important and why this adjustment value chosen]
4. Index B (Adjustment S): [Explain why this decision point is important and why this adjustment value chosen]
While film and JLIM serve different roles ACE, their prompts share a structured design pattern consisting of two essential components:
Task Description Prefix: A comprehensive context header that provides necessary domain knowledge and operational guidelines. For fIIM, this includes power grid topology rules and operational constraints, while for guM, it focuses on reward assessment criteria and safety standards.
Input State-Action Parsing: Dynamic information extracted from current sample's key transitions. Specifically, for fuIM: Detailed state-action parsing focusing on critical line loadings and substation configurations. For ILLM: Trajectory-level analysis highlighting key decision points and their consequences.
17

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
WCCI 2020 Challenge (Episode Reward)
73.3
73.35.00
74.14
76. 76.42..80
75.45.19
74.3 74.35 35.55
75.5/
37.29.99
71.42
69.02
68.84
65.48
61.96
52.06
Rewards
40
Method
5.64
ACE
0
RL
mnni 016
august, 001
december 015
february 019
january 031
June 013
march 038
may 023
november, 005
october 004
Test
Figure 8. Performance comparison on WCCI 2020 test scenarios. We evaluate the best models from ACE (trained with 40K episodes) and baseline RL (trained with 100K episodes).
C. Appendix More Result of ACE
C.1. Case Studies and Visualization
We present a detailed case study demonstrating ACE's decision refinement process in Figure 7. In this scenario, the power grid faces a critical situation with a severe line overload of 192.75% between substations 21 and 26. The original RI agent's action, focusing on substation 26, fails to resolve the overload and even leads to a more precarious state with multiple stress points. contrast, when ACE intervenes, the LLM analyzes the topology and suggests concentrating reconfiguration on substation 21. This refined action successfully redistributes power flows, reducing the critical line loading 59.65% while maintaining stable operation across the network.
C.2. Test Result
We evaluate ACE's performance on 3 challenge test scenarios. shown Figure 8,9, 10, ACE consistently outperforms the baseline RI. approach across different test cases, achieving around 10-15% higher episode rewards while maintaining more stable performance in the 2020 challenge. In the more complex 2022 environment with the IEEE- 118 bus system, ACE demonstrates an even more significant advantage to 30% improvement in episode rewards for challenging scenarios, highlighting its effectiveness in handling larger-scale control problems.
18

---

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making
Neurios Challange
06332
Juto
mart
Figure 9. Performance comparison on NeurIPS 2020 test scenarios. We evaluate the best models from ACE (trained with 40K opisodes) and baseline RL (trained with 100K episodes).
WCCI 2022 Challenge (Episode Reward)
156:54
148.40
113.206.20
126326.70
Rewands
33.57
31.24
Method
RL
5.62
1.66
0050-01-333
2050-01-10.5
2050-01-17.2
2050-01-24.5
2050-02-07.4 .4
2050-02-14.5
2050-04-04.6 6
2050-04-18.11
2050-05-23,6 6
2050-05-30.
2050-06-06.6 6
2050-06-13.12
Test
Figure 10. Performance comparison on WCCI 2022 test scenarios (We select for show). We evaluate the best models from ACE
(trained with 50K episodes) and baseline RL (trained with 200K episodes).
19


Reviews:
[{"review_id": "ZTdizGQahV", "reviewer": "Reviewer_MxZp", "summary": "The paper introduces **Agents Co-Evolution (ACE), a framework that leverages Large Language Models (LLMs) to enhance the sample efficiency of large-scale Reinforcement Learning (RL) decision-making systems**. The core principle of ACE involves using the reasoning capabilities of LLMs to guide the RL training phase mainly by refining the replay buffer, a memory of state transitions, with transitions deemed more valuable for updating the policy.\n\nThe ACE framework was evaluated on three power grid operation challenges (L2RPN competitions) with large action spaces. **ACE outperformed existing RL and LLM-based methods** on the chosen benchmarks, demonstrating significant improvements in sample efficiency compared to RL-based methods and faster decision-making speed compared to LLM-based methods. Furthermore, the study presents ablation studies to validate the effectiveness of the different LLM-based refinement strategies used in the pipeline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Baselines\n\nWe sincerely appreciate your insightful suggestion. In response, we explain our baseline selection criteria and add new baselines for comparison:\n\n- **Original baseline**: We choose SMAAC because it is the winning solution in WCCI 2020 and relies less on predefined rules than other methods, offering better scalability across different environments. \n\n- **Additional baselines**: Given the extensive training scenarios in L2RPN competitions (about 32 years of grid data at 5-minute intervals), creating comprehensive human feedback is challenging. Therefore, we leverage Grid2Op's official rule-based solutions as expert knowledge and add two baselines:\n  - ExpertAgent [1] as the official rule-based agent.\n  - CurriculumAgent [2] as the imitation learning-based RL baseline.\n\n- **Result**: For the NeurIPS 2020 task, ExpertAgent and CurriculumAgent achieved survival rates of 46.2% and 80.7%, respectively, **both lower than ACE's performance**. Notably, although CurriculumAgent performs better than SMAAC, its expert experience comes from extensive environment simulations, resulting in significantly higher time costs. Moreover, since the core components of the ACE framework (the f_LLM and g_LLM) operate independently of RL training, interacting solely with the RL buffer, we will further explore the integration of ACE during CurriculumAgent's RL fine-tuning phase.\n\n[1] Marot et al., Expert system for topological remedial action discovery in smart grids.\n\n[2] Lehna et al., Managing through topology actions: A comparative study between advanced rule-based and RL agents.\n\n> Ablations\n\nTo further investigate the details of ACE co-training, we conducted an in-depth analysis of three key hyperparameters as follows: \n\n-  **LLM Activation Frequency**: We examine how different query intervals of f_LLM affect convergence and efficiency. We test intervals of 128, 256, and 512 steps while keeping the g_LLM query interval fixed at 32 steps. \n-  **LLM Activation Conditions**: We explored different bad case thresholds {-0.3, 0, 0.3} to control the volume of samples refined by LLM. \n- **LLM Training Frequency**: We investigated three LLM training modes for ACE (Qwen2-7B): No SFT, One-time SFT (at RL's 2000th epoch), and K-time SFT (once per 100 refined LLM samples).\n\nBased on the NeurIPS 2020 environment, we find:\n\n- **Higher activation frequency leads to faster initial RL learning**. A 256-step interval showed significant improvements over 512 steps. Interestingly, as training scenarios increased from 288 to 576 cases, the performance gap between different frequencies narrowed, suggesting we can reduce activation frequency for environments having enough diverse scenarios while maintaining effectiveness.\n\n- **Extreme thresholds for filtering bad cases are suboptimal.** A threshold of 0.3 introduced 510 samples, 0 included 275 samples, and -0.3 allowed only 83 samples for refinement. In our experiments, constrained by the capability of the base model, introducing too many samples results in minimal refinement benefits while increasing LLM reference time by 46%. Conversely, too few samples led to slower convergence and approximately 6% performance degradation compared to the standard settings. Additionally, incorporating a planning perspective by considering cumulative multi-step rewards thresholds for consecutive substation operations outperformed single-step thresholds.\n\n- **SFT directly impacts ACE performance.** Using f_LLM without SFT showed limited improvement after about 2000 epochs, with the survival rate stabilizing at 77.5%. Applying a single SFT at the 2000th epoch led to immediate improvement, reaching 78.5% in the first post-SFT evaluation step. Moreover, multiple SFT iterations demonstrated further enhancement over single SFT, ultimately achieving a survival rate of 84.8%.\n\nThank you for providing these valuable references. We commit to including these ablation studies in the revised manuscript.\n\n> Prompting strategies\n\nOur approach combines theoretical principles with empirical evidence as follows:\n\n- Theoretical Foundation: \n\t- f_LLM leverages multi-step reasoning and counterexample analysis to refine suboptimal actions. The core mechanism relies on LLM's semantic understanding of state-action pairs and environment feedback validation (Eq. 4). \n\t- Inspired by TD(λ)’s credit assignment principle, g_LLM replaces fixed temporal decay with trajectory-level causal reasoning. LLMs analyze delayed impacts and adjust rewards non-parametrically (Eq. 5), addressing non-uniform dependencies inherent in industrial control.\n- Empirical Validation:\n\t- As shown in Table 2, removing f_LLM reduces the WCCI 2020 task reward from 69.8 to 48.3, and survival rate from 92.9% to 71.4%; removing g_LLM decreases reward to 61.5.\n\t- Without multi-round reasoning validation and bad case reasoning, reward drops to 65.7 and 60.2, respectively.\n\nThank you again for your time and expertise. Hope my responses address your concerns.", "claims_and_evidence": "The study claims that training an RL agent using trajectories from a replay buffer modified by an LLM enhances both performance and sample efficiency. However, the baseline models employed to support this claim appear insufficient—especially those based solely on an RL approach.\n\nMoreover, RL decision-making systems are known to be highly sensitive to hyperparameter choices [1]. The proposed method relies on numerous hyperparameters, as shown in Table 4 (e.g., the fine-tuning interval for the LLM and the number of transitions used during fine-tuning). Yet, there is little to no analysis of how these hyperparameters affect the system’s overall performance.\n\n[1] Adkins et al., A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning, 2024", "methods_and_evaluation_criteria": "The proposed method makes sense, and the benchmarks derived from a series of competitions aimed at training an intelligent system to operate a power network align with the goal of improving RL systems for large-scale decision-making environments.", "theoretical_claims": "The study does not include traditional mathematical proofs for theoretical claims. Instead, the paper offers empirical evidence and ablation studies to support its claims.", "experimental_designs_or_analyses": "The main concerns regarding the soundness of the analyses are as follows:\n- The limited range of RL-based approaches evaluated.\n- The low number of seeded runs used for benchmarking each method in the study.\n- The unexpectedly high uncertainty observed in the performance plots (Figure 3 and Figure 4) across different episodes.", "relation_to_broader_scientific_literature": "The study addresses challenges encountered in decision-making systems with large action spaces, a long-standing issue in reinforcement learning (RL). One of the main difficulties is the sample inefficiency that arises when tackling such problems [1]. Industrial decision-making systems are often characterized by the need to make multiple decisions involving different components in a relatively short period of time, which exacerbates the problem in these scenarios [2]. As a result, these problems are known to involve high-dimensional action spaces, leading to high sample inefficiency when addressed with RL.\n\nSeveral techniques have been developed to tackle the problem of high-dimensional action spaces, ranging from value-based approaches like AIN-DQN [3] to actor-critic frameworks such as Wolpertinger [4]. However, this paper focuses on approaches that leverage guidance from external sources to improve exploration, such as those used in imitation learning [5], [6], [7] and reinforcement learning with expert feedback [8], [9].\n\nFurthermore, the study focuses on works that use Large Language Models (LLMs) to influence the RL system's training regime without involving them in decision-making. As a result, it examines sample efficiency from a novel perspective, exploring approaches that use LLMs to make the RL training pipeline more sample efficient, such as LLM4Teach [10].\n\\\n\\\n\\\n[1] Sutton et al., Reinforcement learning: An introduction (2018)\\\n[2] Dulac-Arnold et al., Challenges of Real-World Reinforcement Learning (2021)\\\n[3] Zahavy et al., Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning (2019)\\\n[4] Dulac-Arnold et al., Deep Reinforcement Learning in Large Discrete Action Spaces (2016)\\\n[5] Argall et al., A survey of robot learning from demonstration (2009)\\\n[6] Nair et al., Overcoming exploration in reinforcement learning with demonstrations (2017)\\\n[7] Torabi et al., Behavioral Cloning from Observation (2018)\\\n[8] Ouyang et al., Training language models to follow instructions with human feedback (2022)\\\n[9] Christiano et al., Deep reinforcement learning from human preferences (2017)\\\n[10] Zhou et al., Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents (2023)", "essential_references_not_discussed": "No, I believe that most key contributions related to the study have been cited, including:\n\n- Fully RL-based approaches to improve sample efficiency\n    - Behavior cloning\n    - Inverse reinforcement learning\n    - Reinforcement learning with Human Feedback [1]\n- LM-RL integrations for decision-making systems\n\nHowever, one key contribution relevant to the Related Works discussion is missing: the research on replacing human feedback in RLHF with AI feedback [2]. Additionally, the omission of LiFT [3], which leverages foundation models to guide RL training through unsupervised learning, further limits the discussion.\n\\\n\\\n\\\n[1] Christiano et al., Deep reinforcement learning from human preferences (2017)\\\n[2] Bai et al., Constitutional AI: Harmlessness from AI feedback, (2022)\\\n[3] Nam et al., LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers, (2023)", "other_strengths_and_weaknesses": "### Strengths\n- The proposed contribution is well understood and devoid of any ambiguity.\n- The experimental setup is well-organized and easy to follow.\n\n### Weaknesses\n- **Major:**\n  - The baseline methods used to represent RL-based approaches do not seem competitive enough given the breadth of work done to leverage expert knowledge in RL.\n  - More time should be devoted to explaining how both the RL agent and LLM agent co-evolve/train and how the training frequency impacts the system's overall performance.\n  - Some hyperparameters are introduced by the system (see Table 4), but the study does not present any analysis of how these parameters affect the overall training regime.\n- **Minor:**\n  - Algorithm 1 outlines the overall framework and is a critical component of the study; therefore, it should be moved from the appendix to Section 3, which discusses the proposed approach.\n  - The term “co-evolution” is ambiguous, as it may lead readers to assume that the study employs evolutionary strategies for large-scale decision making. \n  - Additionally, while Table 1 provides a comparative analysis of ACE against various baselines—demonstrating its sample efficiency—it would be valuable to examine how its performance changes with different numbers of refined samples injected into the replay buffer.", "other_comments_or_suggestions": "- Figure 2, which illustrates the architecture of the proposed approach, appears to be overloaded with information. It would be beneficial to split it into two separate figures: one that outlines the overall pipeline, and another that focuses on the different prompting strategies used for the LLMs.\n- I’d suggest using terms like “dual training framework” or “multi-agent framework” instead of “co-evolution” to highlight the fact that the parameters of different agents are being updated through gradient-based strategies.", "questions_for_authors": "- I would like to understand why the experimental analysis does not include additional expert-guided RL methods, despite their thorough discussion in the related work section. \n- Moreover, the ablation studies in Section 4.5 appear insufficient given the numerous hyperparameters introduced, as detailed in Table 4 of the appendix. \n- Lastly, while Section 3.2 outlines the various prompting strategies used to update the RL pipeline’s replay buffer, it remains unclear whether these strategies were designed based on theoretical principles, empirical evidence, or a combination of both. Can you provide more information about that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U5Dlp3sn94", "reviewer": "Reviewer_idsB", "summary": "The paper proposes Agents Co-Evolution (ACE), a synergistic framework that integrates Large Language Models (LLMs) and Reinforcement Learning (RL) agents to address challenges in large-scale decision-making problems. While LLMs struggle with long-sequence, real-time decision-making, and RL faces inefficiency in vast action spaces, ACE combines the strengths of both. The framework employs a dual-role trajectory refinement mechanism, where LLMs act as both Policy Actors and Value Critics during RL training. The Actor refines suboptimal actions through multi-step reasoning, while the Critic handles temporal credit assignment and trajectory-level reward shaping. Meanwhile, RL agents improve LLMs' task-specific decision-making through prioritized experience replay. ACE is shown to outperform existing RL and LLM-based methods in experiments involving complex power grid operation tasks with large action spaces (over 60,000 discrete actions).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Computational and Memory Overheads\n\nThank you for raising the important question regarding computational costs. We conduct experiments in the NeurIPS 2020 competition environment. For expert-guided RL, we trained for 100K timesteps with a total duration of 6h 4m14s. Below are the computational costs and memory requirements of ACE (Qwen2-7B + SFT):\n\n|    Module     | Count | Samples |  Time   |\n| :-: | :-: | :-: | :-: |\n|   ACE-RL    |   -   |~40K | 3h 4m 41s |\n| ACE-LLM Inference |  508  |   264   | 1h 48m 0s |\n| ACE-LLM Sampling  | 4981  |   32   | 59m12s  |\n| ACE-LLM Training  |   2   |   200   | 26m10s  |\n|    ACE-Total     |   -   |  ~40K   | 6h18m 3s |\n\n- **Computational Overhead**: ACE's additional computation primarily comes from three modules: selective LLM inference, sampling, and training. With f_LLM and g_LLM query intervals set to 256 and 32, respectively, only 264 samples and 508 inferences were required (due to multi-round reasoning) for ACE training. The SFT module is performed just twice during co-training. Unlike traditional LLM-RL interactions, ACE can continue learning through sampling from the constructed LLM buffer even during update steps when f_LLM and g_LLM are not activated, significantly improving sample efficiency.\n\n- **Memory Overhead**: The primary memory overhead comes from the base model (approximately 14GB for ACE(Qwen2-7B)). The memory required for the generated LLM buffer and SFT datasets is substantially smaller than the RL buffer. Moreover, model memory requirements become negligible when using the GPT4 API instead of hosting the model locally.\n\n> Presentation\n\nWe appreciate the reviewer's feedback regarding the visual presentation. We have addressed this concern by splitting Figure 2 into two separate figures to enhance clarity and readability: Figure 2a now presents the algorithmic framework, while Figure 2b demonstrates the prompt examples.\n\n> Experimental Expand\n\nWe appreciate the reviewer's suggestion regarding model diversity. We agree that testing additional models would help validate ACE's generalizability. We are exploring additional experiments with DeepSeek-R1-Distill-Qwen-7B as the base model, and we commit to including these results in the revised manuscript.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. The proposed method combine LLM agent and RL agent with their Policy and Value model to benifit from them both. The theoretical claims is mainly about how to mix up and they seem to be right.", "experimental_designs_or_analyses": "Yes. The experiments are on L2RPN task which are standard power grid challenges. This paper shows the learning effect of proposed ACE method with different kind of LLM.", "relation_to_broader_scientific_literature": "This article proposes a method that simultaneously utilizes both reinforcement learning (RL) agents and large language model (LLM) agents in parallel to balance the training iteration speed of RL and the prediction accuracy of LLM. There are many methods for joint training of the two, usually using a cascaded hierarchical reinforcement learning approach to divide the tasks at different granularities. The method proposed in this article introduces an innovation in the learning framework, presenting a new way to combine the two.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1.This paper proposes a new framework for training RL agents combined with LLM agents, which improves the iteration speed of training.\n2.The proposed method can significantly enhance learning performance with a smaller sample size.\n3.The proposed method is applicable to different language models and has a certain degree of scalability.\nWeaknesses\n1.There is too much text in the images; the aesthetic could be optimized.\n2.More types of LLMs, such as DeepSeek and Qwen, could be tried in the experiment.", "other_comments_or_suggestions": "None", "questions_for_authors": "The experiment shows Expert-guided RL needs more sample and achieve worse survival rate, I wonder the time cost comparision between Expert-guided RL and ACE method because LLM spend more time while sample generation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WP2HOWq46Y", "reviewer": "Reviewer_PbNn", "summary": "This paper proposes the ACE framework, which co-evolves LLMs and RL agents for industrial-scale decision-making. ACE decouples the high-level reasoning and fine-grained control by employing a \"Think Twice, Act Once\" strategy. The framework is evaluated on multiple power grid operation challenges from the L2RPN competitions, where it consistently outperforms both RL-only approachs and LLM-only approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Cross-Domain Applicability of ACE\n\nWe appreciate the reviewer's question about ACE's generalizability. While our study focuses on power grid control, the core modules of ACE are domain-agnostic and not constrained by specific environment characteristics. We believe ACE's potential applications in other domains depend on two core mechanisms:\n\n(1) The **trajectory refinement module** effectively compresses and navigates large action spaces through LLM-based multi-round semantic reasoning, validated through environment interaction. The combination of **semantic abstraction with RL optimization** could benefit other discrete decision-making problems with large action spaces.\n\n(2) The **reward reshaping module** uses LLMs' causal reasoning to adjust reward signals based on trajectory-level analysis. This is particularly relevant for environments with **sparse rewards** or where reward functions are **difficult to specify explicitly**. The LLM's semantic understanding enables dynamic reward adjustment, offering an alternative to fixed rule-based reward designs.\n\n> Computational and Memory Overheads\n\nThank you for raising the important question regarding computational costs. We conducted experiments in the NeurIPS 2020 competition environment. For expert-guided RL, we trained for 100K timesteps with a total duration of 6h 4m14s. Below are the computational costs and memory requirements of ACE (Qwen2-7B + SFT):\n\n|    Module     | Count | Samples |  Time   |\n| :-: | :-: | :-: | :-: |\n|   ACE-RL    |   -   |~40K | 3h 4m 41s |\n| ACE-LLM Inference |  508  |   264   | 1h 48m 0s |\n| ACE-LLM Sampling  | 4981  |   32   | 59m12s  |\n| ACE-LLM Training  |   2   |   200   | 26m10s  |\n|    ACE-Total     |   -   |  ~40K   | 6h18m 3s |\n\n- **Computational Overhead**: ACE's additional computation primarily comes from three modules: selective LLM inference, sampling, and training. With f_LLM and g_LLM query intervals set to 256 and 32, respectively, only 264 samples and 508 inferences were required (due to multi-round reasoning) for ACE training. The SFT module is performed just twice during co-training. Unlike traditional LLM-RL interactions, ACE can continue learning through sampling from the constructed LLM buffer even during update steps when f_LLM and g_LLM are not activated, significantly improving sample efficiency.\n\n- **Memory Overhead**: The primary memory overhead comes from the base model (approximately 14GB for ACE(Qwen2-7B)). The memory required for the generated LLM buffer and SFT datasets is substantially smaller than the RL buffer. Moreover, model memory requirements become negligible when using the GPT4 API instead of hosting the model locally.", "claims_and_evidence": "The claims in the work are supported by experiments on three different L2RPN challenge environments, where ACE shows improvements in key metrics.", "methods_and_evaluation_criteria": "The evaluation is carried out on power grid operation benchmarks with several metrics. Comparisons against multiple baselines (expert-guided RL, LLM-only, and LLM-guided RL) are made, ensuring that the evaluation criteria are well matched to the problem.", "theoretical_claims": "There are no theoretical claims that need to be checked in this work.", "experimental_designs_or_analyses": "I check the soundness of experimental design, including benchmarks, baselines, and ablation studies. The experimental design is robust and comprehensive", "relation_to_broader_scientific_literature": "ACE builds upon and extends prior work in RL and LLM-guided decision-making. It relates to earlier approaches like LLM4Teach and RL-GPT. This work situates its contributions within the context of industrial control challenges and large-scale decision-making, addressing critical limitations of both RL (sample inefficiency and large action spaces) and LLMs (inability to maintain long-horizon consistency).", "essential_references_not_discussed": "The key related works are well discussed in this work.", "other_strengths_and_weaknesses": "Strengths:\n\n- Novel co-evolution framework that leverages the strengths of LLMs for offline reasoning and RL for online control.\n\n- Comprehensive experimental evaluations and detailed ablation studies.\n\nWeaknesses:\n\n- Experiments are confined to power grid control scenarios, leaving open questions about generalizability to other industrial domains.\n\n- Potential computational overhead due to the dual-system (LLM and RL) integration is not fully discussed.", "other_comments_or_suggestions": "1. Figure 2 does not clearly depict the specific process, consider providing additional examples to illustrate how the dual-role refinement leads to improved credit assignment and decision making.\n\n2. Discuss potential limitations in extending the framework to other domains and the associated computational trade-offs.", "questions_for_authors": "1. Have you considered applying ACE to other industrial control tasks beyond power grid management? How do you anticipate the framework will perform in domains with different dynamics and action space structures?\n\n2. What are the specific computational and memory overheads introduced by the dual-system architecture of ACE compared to standard RL methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g3zp38fBur", "reviewer": "Reviewer_Q3yH", "summary": "This paper introduces a novel framework, termed Agents Co-Evolution (ACE), which combines Large Language Models (LLMs) and Reinforcement Learning (RL) for large-scale decision-making in the context of power grid operations. In this framework, the Soft Actor-Critic (SAC) algorithm (Haarnoja et al., 2018) is first employed to learn an initial policy that maximizes both the expected return and policy entropy. The states from low-reward mini-batch samples in the replay buffer are then converted into natural language descriptions, which are provided to a pretrained LLM along with the relevant context. The refined actions are derived from the output of the LLM, which acts as the policy actor. Subsequently, the LLM is used as a value critic to re-evaluate the long-term impact of key decisions at the trajectory level. Additionally, the framework integrates reward-based prioritization and weighted policy update strategies to generate a high-quality replay buffer that combines experiences from both direct environmental interactions and LLM refinement. The performance of ACE is demonstrated via experiments on three power grid competitions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> LLM4Teach\n\nThank you very much for your insightful question. You raise a valid point regarding the original version of LLM4Teach requiring environmental interaction after policy-level alignment. However, due to the significant time cost associated with LLM-environment interactions, we adopted the mode similar to ACE, where alignment is only enabled during the training phase with Qwen2-7B-Instruct as the aligned model. We will provide a more detailed explanation of this modification in the revised version to ensure complete clarity.\n\n> Computational and Memory Evaluation Metrics\n\nThank you for raising the important question regarding computational costs. We conduct experiments in the NeurIPS 2020 competition environment. For expert-guided RL, we trained for 100K timesteps with a total duration of 6h 4m14s. Below are the computational costs and memory requirements of ACE (Qwen2-7B + SFT):\n\n|    Module     | Count | Samples |  Time   |\n| :-: | :-: | :-: | :-: |\n|   ACE-RL    |   -   |~40K | 3h 4m 41s |\n| ACE-LLM Inference |  508  |   264   | 1h 48m 0s |\n| ACE-LLM Sampling  | 4981  |   32   | 59m12s  |\n| ACE-LLM Training  |   2   |   200   | 26m10s  |\n|    ACE-Total     |   -   |  ~40K   | 6h18m 3s |\n\n- **Computational Overhead**: ACE's additional computation primarily comes from three modules: selective LLM inference, sampling, and training. With f_LLM and g_LLM query intervals set to 256 and 32, respectively, only 264 samples and 508 inferences were required (due to multi-round reasoning) for ACE training. The SFT module is performed just twice during co-training. Unlike traditional LLM-RL interactions, ACE can continue learning through sampling from the constructed LLM buffer even during update steps when f_LLM and g_LLM are not activated, significantly improving sample efficiency.\n\n- **Memory Overhead**: The primary memory overhead comes from the base model (approximately 14GB for ACE(Qwen2-7B)). The memory required for the generated LLM buffer and SFT datasets is substantially smaller than the RL buffer. Moreover, model memory requirements become negligible when using the GPT4 API instead of hosting the model locally.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "Yes, I've checked the soundness and validity of experimental designs and related analyses.", "relation_to_broader_scientific_literature": "This paper introduces a novel algorithmic framework that enables collaboration between Large Language Models (LLMs) and Reinforcement Learning (RL) for large-scale decision-making. While several collaboration frameworks have been proposed in the literature, to the best of my knowledge, this is the first paper to employ an LLM as both a policy actor and a value critic simultaneously.", "essential_references_not_discussed": "I think the reference part of the paper is ok.", "other_strengths_and_weaknesses": "The paper conducts an ablation study to assess the effectiveness of each component in the ACE framework. Additionally, Section 5 offers a detailed discussion, providing a deeper analysis. Both of these efforts contribute to a clearer understanding of the work for the reader.\n\nA weakness of this paper is the lack of a theoretical proof or at least an analysis for the convergence of the proposed method.", "other_comments_or_suggestions": "1. In the experimental section, the method in this paper is compared with LLM4Teach, but it seems that the way LLM4Teach is used here differs from the original LLM4Teach paper. In the original paper, policy-level alignment is only the first step in training the RL model. The second step is to allow the agent to update and optimize the policy through further interactions with the environment. It is suggested to clarify this.\n2. It is recommended to include additional evaluation metrics, such as the communication cost introduced by interacting with the LLM and the model sizes corresponding to different comparison methods. This would help readers gain a more comprehensive understanding of the practical significance of the proposed method.", "questions_for_authors": "For ACE in the experiment, Qwen2-7B instruct and GPT-4o-0806 are used. Which LLM is used for the other comparsion methods, say LLM4Teach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary:** This work explores the use of large language models (LLMs) to enhance reinforcement learning by employing LLMs as both the actor and the critic. This represents a notable departure from prior approaches that typically use LLMs as auxiliary components or regularizers for either the actor or the critic. The authors demonstrate the effectiveness of their framework through real-world experiments in power system operations, showing improved performance.

**Justification:** All reviewers acknowledge the novelty of directly approximating both the actor and critic with LLMs, marking a significant contribution to the field. However, the current experimental validation is limited to power system scenarios. As several reviewers have pointed out, the lack of evaluation on more widely-used and diverse benchmarks restricts the perceived generality and impact of the method. It is recommended that the authors extend their experiments to include standard RL benchmarks to strengthen the empirical support for their framework.
Decision: Accept (poster)