Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learning
Authors: Zhihao Liu
Xianliang Yang
Zichuan Liu
Yifan Xia
Wei Jiang
Yuanyu Zhang
Lijuan Li
Guoliang Fan
Lei Song
Jiang Bian
Venue: Accepted by TMLR ()

Abstract: Multi-agent reinforcement learning (MARL) is employed to develop autonomous agents that can learn to adopt cooperative or competitive strategies within complex environments. However, the linear increase in the number of agents leads to a combinatorial explosion of the action space, which always results in algorithmic instability, difficulty in convergence, or entrapment in local optima. While researchers have designed a variety of effective algorithms to compress the action space, these methods also introduce new challenges, such as the need for manually designed prior knowledge or reliance on the structure of the problem, which diminishes the applicability of these techniques. In this paper, we introduce \textbf{E}volutionary action \textbf{SPA}ce \textbf{R}eduction with \textbf{K}nowledge (eSpark), an exploration function generation framework driven by large language models (LLMs) to boost exploration and prune unnecessary actions in MARL. Using just a basic prompt that outlines the overall task and setting, eSpark is capable of generating exploration functions in a zero-shot manner, identifying and pruning redundant or irrelevant state-action pairs, and then achieving autonomous improvement from policy feedback. In reinforcement learning tasks involving inventory management and traffic light control encompassing a total of 15 scenarios, eSpark consistently outperforms the combined MARL algorithm in all scenarios, achieving an average performance gain of 34.4% and 9.9% in the two types of tasks respectively. Additionally, eSpark has proven to be capable of managing situations with a large number of agents, securing a 29.7% improvement in scalability challenges that featured over 500 agents. The code can be found in https://github.com/LiuZhihao2022/eSpark.

Published in Transactions on Machine Learning Research (05/2025)
Knowing What Not to Do: Leverage Language Model In- sights for Action Space Pruning in Multi-agent Reinforcement Learning
Zhihao Liu Xianliang Yang?, Zichuan Liu², Yifan Xia², Wei Jiang", Yuanyu Zhange, Lijuan Li", Guoliang Fant, Lei Song?, Jiang Bian
The Key Laboratory of Cognition and Decision Intelligence for Complez Systems, Institute of Automation, Chinese Academy of Sciences Microsoft Research Asia Nanjing University
University of Chinese Academy of Sciences
University of Illinois Prbana-Champaign
Guizhoo University
*Work done during internship at. Microsoft. Research Asia. |Corresponding author.
Reviewed on OpenReview: http:://ppeeeieee..eee///m////sp
Abstract
Multi-agent reinforcement learning (MARL) is employed to develop autonomous agents that can learn adopt cooperative or competitive strategies within complex environments, However, the linear increase in the number of agents leads a combinatorial explosion of the action space, which may result in algorithmic instability, difficulty in convergence, or entrapment in local optima. While researchers have designed a variely of effective algorithms to compress the action space, these methods also introduce new challenges, such as the need for manually designed prior knowledge or reliance on the structure of the problem, which diminishes the applicability of these techniques. In this paper, we introduce Evolutionary action SPAce Reduction with Knowledge (eSpark), an exploration function generation framework driven by large language models (LLMs) to boost exploration and prune unneccssary actions in MARL. Using just basic prompt that outlines the overall task and setting, eSpark is capable of generating exploration functions in zero-shot manner, identifying and pruning redundant or irrelevant state-action pairs, and then achieving autonomous improvement from policy feedback. In reinforcement learning tasks involving inventory management and traffic light control encompassing a total of scenarios, eSpark consistently outperforms the combined MARL algorithm in all scenarios, achieving an average performance gain of 34.4% and 9.9% in the two types of tasks respectively. Additionally, eSpark has proven to be capable of managing situations with large number of agents, securing 79.7% improvement in scalability challenges that featured over 500 agents. The code can be found in https: //githbb http:////o////aaa2222/2////rrrr
1 Introduction
Multi-agent reinforcement learning (MARL) has emerged as a powerful paradigm for solving complex and dynamic problems that involve multiple picrision-makens Zhang et al. (2021): Wang et al. (2021). However, the intricacics of agent interplay and the exponential expansion of state and action spaces render the solution of
MARL problems difficult. Researchers have proposed the Centralized Training with Decentralized Execution (CTDE) framework Oliehoek al. (2008) and parameter sharing methods, decomposing the value policy

---

Published in Transactions on Machine Learning Research (05/2025)
functions of multi-agent system into individual agents and sharing model parameters among all agents. As experimentally verified by many of the most prominent MARL, algorithms such as Multi-agent PPO (MAPPO) Yu et al. (2022), QMIX Rashid et al. (2020), QPLEX [Wang et al. (2021) or QTRAN [Son et al. (2019). these methodologies have been demonstrated to be robust strategies for surmounting the challenges posed by MARL. MARL methods based on parameter sharing and CTDE have achieved notable success in a varioty of well-established tasks, including StarCraft Multi-Agent Challenge (SMAC) Li ct al. (2023a): Wang eet al. (020), the Multi-Agent Particle Environment (MPE) Lowe et al. (2017). and Simulation of Urban
MObility (SUMO) Wei et al. (2019): Lu et al. (023)..
Despite the great success of paramctor-sharing CTDE methods, their practicality dwindles in real-world tasks involving large number of agents, such as large-scale traffic signal control Mousavi et al. (2017), wireless
communication networks Zocca 2019), and humanitarian assistance and disaster response Meier 2015 where centralized training becomes impractical due to large problem scale Munir et al. (2021). Fully Decentralized Training and Execution (DTDE) methods, such as Independent PPO (IPPO) de Witt et al. (2020), offer a scalable solution where resource consumption does not escalate drastically with an increase in the number of agents. However, due to the lack of consideration for agent interactions, they often struggle to find optimal solutions and fall into local optima. Current strategies for addressing large-scale MARL tasks involve introducing task-specific structures to model agent interactions or dividing agents into smaller, independently trained groups Ying et al. (2023 : Chen et al. (2020). These methods, however, are constrained by their dependence on task-related structuring, limiting their applicability to a narrow range of problems.
Additionally, the 'curse of dimensionality'' presents a significant challenge in multi-agent systems Hao et al. (2022b/a) Ib) where agents are required to navigate through an expansive action space saturated with numerous actions that are either irrelevant or markedly suboptimal (relative to states). While humans can deftly employ contextual cues and prior knowledge to sidestep such challenges, MARL algorithms typically engage in the exploration of superfluous and extraneous suboptimal actions Zahavy et al. (2018). Besides, prevailing parameter sharing can exacerbate this exploratory dilemma, as will be clucidated in Proposition 2 The issue occurs primarily because agents with shared parameters often prefer suboptimal policies that present short-term advantages, rather than exploring policies that may potentially deliver higher long-term returns.
Exploration is crucial for overcoming local optima, as it encourages agents to discover potentially better states thus refining their policies. While single-agent exploration techniques like the Upper Confidence Bound (UCB) Auer (2002), entropy regularization Haarnoja et al. (2018), and muriosity-based exploration Groth let al. (2021); | Pathak et al. (2017) have shown promising results, they struggle with the escalated complexity
in MARL scenarios, compounded by issues like deceptive rewards and the "Noisy-TV' problem Burda et al.
(2018). Integrating domain knowledge into exploration could significantly enhance exploration efficiency, by helping identify critical elements and problem structures, thereby aiding in the selection of optimal actions Simon (1956). However, the integration of knowledge into data-driven framework poses significant challenges, particularly when manual input from domain experts is required, thus reducing its practicality.
Recently, Large Language Models (LLMs) such as GPT-4 Achiam et al. 2023 have shown formidable skills in language comprehension, strategic planning, and logical reasoning across various lasks Yao et al. (2023);
Zhu et al. (2023). Although not always directly solving complex, dynamic problems, their inferential and orror-learnig abilitics facilitate progrossively better solutions through itcrative feedback Ma et al. (2024).
The integration of LLMs with MARL presents a promising new avenue by facilitating exploration through the pruning of redundant. actions. In this paper, we introduce Evolutionary action SPAce Reduction with Knowledge (eSpark), novel approach that utilizes LLMs to improve MARL training via optimized exploration functions, which are used to prune the action space. eSpark begins by using LLMs to generate exploration functions from task descriptions and environmental rules in a zero-shot fashion. It then applies evolutionary search within MARL to pinpoint the best performing policy, Finally, by analyzing the feedback on the performance of this policy, eSpark reflects and proposes a set of new exploration functions, and iteratively optimizes them according to the aforementioned steps. This process enhances the MARL policy by continuously adapting and refining exploration. To summarize, our contributions are follows:
1. We introduce the eSpark framework, which harnesses the intrinsic prior knowledge and encoding capability of LLMs to design exploration functions for action space pruning, thus guiding the explo-
2

---

Published in Transactions on Machine Learning Research (05/2025)
ration and learning process of MARL algorithms. eSpark requires no complex prompt engineering and can be easily combined with MARL algorithms.
We validate the performance of eSpark across 15 different environments within the inventory management lask MABIM Yang al. (2023) and the traffic signal control task SUMO Behrrisch et al. (2011). Combined with IPPO, eSpark outperforms IPPO in all scenarios, realizing average profit increase of 34.4% in the MABIM and improving multiple metrics in SUMO by an average of 9.9%.
Even in the face of scalability challenges where the DTDE methods typically encounter limitations, eSpark elevates the performance of TPPO by 29.7%.
3. We conduct controlled experiments and ablation studies to analyze the effectiveness of cach compo- nent within the eSpark framework. We first validate the advantages of kuowledge- pruning. Subsequently, we conduct ablation studies to demonstrate that both retention training and LLM pruning techniques contribute to the performance of eSpark. These effects are even more pronounced in the more complex MABIM environment. Finally, we confirme the significance of the LLM checker and comprehensive feedback in enhancing code generation and exploration function improvement.
2 Related works
LLMs for code generation. LLMs have demonstrated remarkable advancements in automated code generation, enabling natural-language-tooon
(NI.2Code) translation with unprecedented accuracy and efficiency Chen et al. (2021). Recent research has explored various enhancements to LLM-based code generation, including instruction tuning Wei et al. (2021). reinforcement Icarning with feedback Le et al. (2022). retrieval-augmnented approaches Gou el al. (002)) and repository-level code generation Liu et al.
(2023 Morcover, specialized code LLMs such as StarCoder Li et al. (2023b), CodeLlama Roziere et al. (2023)
and DedpSeek-Coder Zhuu et al.] (2024) have been developed to improve performance on coding benchmarks like HumanEval and MBPP Chen et al. (2021): Austin al. (0021).
LLMs for RL and MARL. The integration of LLMs into RL and MARL has sparked considerable research interest Sharma et al. (2022): Kwon et al. (2023): Li et al. (2024). Some works Hill et al. (2020); ; Chan
et al. 2019)) incorporate the goal descriptions of language models that help in enhancing the generalization capabilities of agents designed to follow instructions. Another work serves LLM as reward designer for robot control Ma et al. (2024). Further studics have extended this approach to complex tasks involving reasoning and planning Huang et al. (2023 2022b). Moreover, LLMs have been employed to guide exploration and boost. RL efficiency Du et al. (2023 : Chang et al. (2023); Hu Sadigh (2023). However, scaling to high-complexity, real-time, multi-agent settings remains challenge. Our method mitigates this by generating exploration functions to navigate the policy space, thus facilitating the application to complex MARL scenarios without direct L.M-ageett decision-making interaction.
Action space pruning in RL and MARL. Pruning the action space has been shown to be effective in guiding agent behaviors in complex environments Lipton et al. (2016); Fulda et al. (2017). Techniques include learning an climination signal to discard unnecessary actions Zahavy et al. (2018), and employing transfer learning that pre-trains agents to isolate useful experiences for later action refinement Shirali et al.
2023
Lan et al. 22022; ; Ammanabrolu & Ried] (2018). Some works transform action space pruning into
state-dependent action selection problem, making decisions within the reduced space Sun & Wang (2022): Huang et al. (2022a). Further works use manually designed data structures based on prior knowledge to filter actions Dulac-Arnold el al. (2015); Padullaparthi et al.] (2022); Nagarathinam el al. (2020). However, training pruning signals or applying transfer learning is inherently difficult with many agents, and the need for expert knowledge in manual pruning rules hampers their transferability, limiting the applicability of current methods. We harness the abundant knowledge embodded within LLMs for action space pruning, demonstrating universal applicability across multitude of scenarios.
3

---

Published in Transactions on Machine Learning Research (05/2025)
3 Preliminaries
3.1 Problem formulation and notations
Markov game framework. In our study, we explore a Markov game framework, formally defined by the tuple (N,S,O,A P.P.R.R R 7). Here N represents the total number of participating agents, S denotes a well-defined state space, and O = II O2 constitutes the combined observation space, A= Ag is the joint action space for all agents involved. The transition dynamics are captured by the probability function P A x S + [0, 1], the reward function R S x R maps stale-action pairs to real-valued rewards. The discount factor denoted by y E [0, 1]. All the notations used are summarized Appendix A
At each discrete time step t, the environment is in state s S. Each agent A [1, 21.2..... N] reccives an observation of Ok and draws an action from d TT(- ot), where Tą OO x AA + [0, denotes the policy of agent k, and EateA. (a (o)) 1. The joint actions of all agents a (oi,ah....,,a is drawn from the joint policy x(. [+) = Tx( og)) Subsoquently, reward r' R(s a²) is given based on the current state and joint action. The state transition is determined by s*#1 P(- s a²).
In this paper, we focus on a fully cooperative scenario where all agents share a common reward signal. The collective objective is to maximize the expected cumulative reward, starting from an initial state distribution p.. This collaborative approach emphasizes the alignment of individual agent strategies towards maximizing unified reward J(T)::
J(T) E 2 ooow
Σ 7
Policy with exploration function. In the configuration of our study, we introduce the exploration lunction E O₂ x A + (0, 13, indicating whether an action selectable by agent For a given policy me of agent & and an exploration function E, we define new policy mh as follows:
Tn( of) E(ok.-)
E (-1 of Entea. Th (a o%) Eloj af)
if EatEA, (a of Blok, at) > 0; otherwise, T of) = Tu(- of)). Consequently, the joint policy for all agents under the guidance of E is defined as:
N
π (.1s =
II
TW F
We define the set of all joint policies as {π) and the set of all exploration functions as (E). Tet. (Λ²) denote the set of joint policies when subjected to an exploration function E E (E). An exploration function E is non-trivial if it assigns zero probability to at least one obscrvation-action pair. The following proposition
naturally arises from the definition: Proposition 1.
1. For any E (E), 5 If E is non-trivial, then (T²) (Tr..
2. For any {T], there crists non-trivial E {E} such that J( 2 J(T).
An intelligent choice of exploration functions does not diminish our ability to discover optimal policies; instead, it allows us to refine the policy space, thereby enhancing the efficiency of the learning process. The proof of this proposition can be found in Appendix B
3.2 Challenges and motivations
The intricate relationships among multiple agents make extremely difficult to scarch for the optimal solution in MARL. Without powerful exploration methods, it is nearly impossible to avoid suboptimal outcomes. We will elaborate this with example from the following proposition:
4

---

Published in Transactions on Machine Learning Research (05/2025)
Let's consider fully cooperative game with N agents, one state, and the joint action space {0, 10%1 where the reward is given by r(0", 1M) = rı and r(0N-", ==)== for all m & N, r1, 12 are positive real numbers. We suppose the initial policy is randomly and uniformly initialized, and the policy is optimized in the form of gradient descent. Let be the probability that the shared policy converges to the best policy, then:
Proposition 2.
72 r1 + Nr²
p=
N-1
Detailed proof is provided in Appendix B In the above example, we show that the increase in the number of agents makes it more difficult for MARL algorithms to reach the optimal solution. However, based the problem context, humans can understand problems from high-level semantic perspective, and quickly find optimal solutions. As LLMs have demonstrated surprising abilities in semantic understanding, reasoning, and planning across various lasks Yuan et al. (2023 : Wang et. al. (2023) we conduct a simple experiment to test.
GPT-4's capability for the issue in Proposition and here is GPT-4's response:
In fully cooperative game, all agents work together to maximize the total reward. There tuo distinct revard conditions:
1. When all agents choose action the reward is 71, positive real number.
2. When there is any number of agents m (vhere < m < N) choosing action 1, the reward mr2, where
r2 i8 a positive real number.
way avoids negative anytime there mix of 0's and in the action space, which
agents should
scenario. negative reward scenario happens agents choosing and others choosing Therefore, the optimal joint action for all agents is all choose 1.
GPT-4 exhibits reasoning abilitics on par with those of humans and directly solves the problem in Proposition 2 Propositions [ has already shown that an intelligent exploration function can not only reduce the searching space but also improve the final performance. This makes us think about the probability of applying the powerful LLMs to prune the redundant action space and thereby guide the exploration MARL. In Appendix C example which demonstrates GPT-4 can ef- fectively leverage its reasoning capacity to generate code for pruning redundant action spaces. In the following sections, propose the eSpark framework, which integrates the prior knowledge and inferential capability LLMs boost the exploration in MARL.
we designed a more complex sequential Docision-making
Algorithm eSpark
Input: Tnitial prompt prom, LLM checker LLM,, LLM code generator LLMy, the evolution iteration number N, and sample batch size K
2: Initialize: policies $1 ol........
3: for == to N do
4 Method
4:
// Exploration Function Generation Fi,..., ., Ex ~ (pMom LLMy (prom))
In this section, introduce novel frame- work, eSpark, which integrates robust prior knowledge encapsulated in LLMs. It improves iteratively through a cycle of trial and error, leveraging the capability of LLMs. Figure Dil il- lustrates the overall training procodure. eSpark
// Retention training
if # then
8:
Φ1. oh. 1i + -bb
9:
end if
10:
// Evolutionary search
11:
G1, Fi= (EE)),)))..)) GK Tx FK ((Ek)
is composed of three components: (i) zero-shot
12:
// Reflection and Feedback
generation of exploration functions, (ii) evo- lutionary search for best performing MARL policy, and (iii) detailed feedback of the policy to improve the generation of exploration func- tions. We denote the performance policy i as G, and the policy feodback as F; (defined in Section 4.3), and the pseudocode is shown in A Algorithm
13:
best arg max (G1, G2,.... Gk)
14: prom prom Reflection(E) (Errsssssnssss......... Fhost)
15: end for
16: Output: &est
5

---

Published in Transactions on Machine Learning Research (05/2025)
4.1 Exploration function generation
LLMs have been demonstrated to possess exceplional capabilities in both code comprehension and generation. To this end, we employ a LLM as LLM code generator, denoted as LLMg, whose task is to understand the objectives of the current environment, and output an exploration function:
1. Exploration function generation
Reflection
Exploration
function
Exploration
Generate
LLM
function
Exploration Function
LLM,
2. Evolutionary search
Exploration
Performance
functions
Feedback 1
state
Binary action mask
In|.
Performance (best)
Feedback
Feedback
state
Performance K
action
Environment
RL policy
Feedback
Figure 1: eSpark firstly generates exploration functions via zero-shot creation. Each exploration function is then used to guide independent policy, and the evolutionary search is performed find the best-performing policy. Finally, eSpark reflects on the feedback from the best performance policy, refines and regenerates the exploration functions for the next iteration.
E EK ~ LLM, (prom).
(1)
where prom the prompt for LLMy, and the generation of K exploration functions is to circumvent the suboptimality that may arise from single-sample generation. The initial prom includes an RL formulation describing the reward system, state items, transitions, and the action space, alongside task description that specifics the task objectives, expected outputs, and formatting rules. Details the initial prom are provided in Appendix K We use code for the RL formulation as it effectively captures the physical transition dynamics crucial to RT problems, which are always difficult to express precisely through the text alone, especially when environmental complexity increases. Code contexts also improve code generation and clarify environmental semantics and variable roles Ma et al. (2024). In Appendix H we discuss the impact of different forms RL formulation on the final performance of eSpark when the environment code is unavailable.
During the code generation, however, LLM, may incorrectly interpret variables and produce logically flawed code. This kind of flawed logic could persist if it is added to the prompt context for the next generation. As research has shown that collaboration among multiple LLMs can enhance the quality and efficacy of the generated contents Chen et al. (2023): Zhang et al. (2023), we introduce the LLM checker denoted as LLM,, which reviews LLM, 's output to pursue an enhanced generation. LLM, uses the same prompt as LLM, but is prompted to focus on verifying the accuracy of code relative to environmental transitions and variable specifications. If inconsistencies are found, LLM, signals the error, prompting LLM, to regenerate the code. The reasons for introducing LLM, are further discussed in Section 5.5 Finally, exploration functions are generated by:

---

Published in Transactions on Machine Learning Research (05/2025)
E,,..., EK ~ LLM, (prom, LLM,(prom)
(2)
Exploration functions applied only during the training phase guide the exploration of MARL. During the execulion phase, all exploration functions are removed.
4.2 Evolutionary search
During the generation, however, it should be noted that the initially generated exploration function may not always guarantee executability and effectiveness. To address this, eSpark performs an evolutionary search paradigm that selects the best-performing policy in one iteration and uses its feedback for subsequent generation Ma et al. (2024). Specifically, eSpark samples a batch of K exploration functions each generation ensure there are enough candidates for successful code execution. Performance assessed at regular checkpoints within an ileration, with the final evaluation based the average of the last. few checkpoints. The policy achieving the highest performance is selected, and the feedback obtained from this policy is integrated to optimize the exploration functions in the following steps.
Due to the dynamic nature of exploration, the exploration function generated based on feedback from the best-performing policy may not be applicable to other policies. As the proof of Proposition [ demonstrates, when an exploration function incapable of intelligently pruning, it may even impair the performance of the policy. To this end, we utilize retention training to maintain continuity of exploration. Let o represent best-performing policy from the 1)-th iteration. For the i-th iteration except for the first, at the start of the iteration, we set:
(3)
This allows to match exploration functions with their corresponding policies, subsequently refining performance incrementally, We will verify the impact of retention training in Section 5.4
4.3 Reflection and feedback
Feedback from the environment can significantly enhance the quality of the generated output by LLMs Nasci- mento et al. (0023 Du et al. [2023]. In eSpark, we leverage policy feedback, which contains the evaluation of policy performance from varions aspects, to enhance the gencration of LLMs. This policy feedback may either come from experts or be automatically constructed from the environment, as long as it encompasses insights into the aspects where the current algorithm performs well and areas where requires improve- ment. As illustrated in Equation by correlating the best-performing policy feedback lbbsst and the most effective exploration function Ebest, LLMs introspect, update the prompt prom, and gear up for the ensuing evolutionary cycle.
prom + prom Reflection(Ebet.. Fbest).
(4)
In our experiments, generate automated policy feedback from environmental reward signals, as domain experts in relevant fields are not available. We acknowledge that obtaining feedback from human experts can be expensive. Nevertheless, it is important to note that within our framework, the number of rounds for feedback collection is specified by a predefined hyperparameter, which is typically kept low (in our experiments, is set to 10). Therefore, in scenarios where human experts are accessible, incorporating their insights feasible and can potentially enhance performance.
5 Experiments
5.1
Experiment settings
For comprehensive evaluation of eSpark's capabilities, we perform detailed validations within two distinct industrial environments: the inventory management environment MABIM and the traffic signal control environment SUMO.

---

Published in Transactions on Machine Learning Research (05/2025)
MABIM setting: MABIM simulates multi-echelon inventory management by modeling each stock-keeping unit. (SKU) as an agent, mirroring real-world operations and profits within the MARL framework. The total reward is composed of multiple reward components. We utilize the total reward to identify the best-performing policy, while those components evaluate the policy's multifaceted performance lo generale policy leedback. We locus on three key challenges within inventory management: multiple ochelons, capacity constraints and scalability, selecting corresponding scenarios for experiments.
SUMO setting: SUMO is a traffic signal control environment in which each intersection is represented as an agent. It offers a variety reward functions, and we use "the number of stopped vehicles' as the reward for evolutionary search, while other rewards are for policy leedback. The Average Delay, Average Trip Time, and Average Waiting Time metrics are chosen for evaluation Lu al. (2023) We employ GESA Jiang et al. (2024) to standardize intersections into 4-arm configurations. Each simulation spans 3600 seconds, with decisions at 15-second intervals.
Model setting: We use IPPO as the base MARL framework for eSpark due to its DTDE structure, which suitable for large-scale challenges. But note that our approach can also be applied as a plugin in other MARL methods. We select GPT-4 for the LLM, and LLM, due its superior comprehension and generation abilities. The performance of different. LLMs scenario, we conduct three rums with batch size of K = 16. eSpark has the same number of training steps as the compared MARL basclines, with 10 iterations evenly selected throughout the training process for feedback, reflection, and exploration function editing.
be found in Appendix G For each
All training jobs are executed with Intel(R) Xeo Gold 6348 CPU and 4 NVIDIA RTX A6000 GPUs. In Appendix D we provide detailed introduction and setting for the environments and model. In Appendix we give hyperparamcter configurations and descriptions of cach bascline method.
5.2 Experiment results
In this section, present the key findings for eSpark in the MABIM and SUMO, highlighting the best and second best results in bold and underline. More detailed results and computational costs presented in Appendix
5.2.1 Performance on MABIM
Figure 2 illustrates the performance of eSpark and the MARL baselines throughout the training process, with the detailed final results presented in Table [ It can be observed that eSpark continuously improves its performance and stabilizes within 10 iterations. With IPPO as the base MARL algorithm, eSpark not only outperforms IPPO in all scenarios but also exceeds the performance of all compared baselines in 4 out of 5 scenarios. For an in-depth analysis, we discuss the policy differences between IPPO and eSpark in Appendix I along with eSpark's reflective mochanism and exploration function adjustments in Appendix L While IPPO struggles to learn the intricate interplay among SKUs, eSpark excels particularly navigating cooperation among SKUs and refining its search in a broad space, leading marked improvements in managing capacity constraints and multi-echelon coordination.
eSpark
MAPPO
OTRAN
IPPO
OPLEX
x10
Standard
X10
2 echelons
X10
3 echelons
x10²
Lower
x100
Lowest
Profit
Profit
Profit
Profit
20000 Episade
20000
20000
40000
20000
40000
Episade
Episade
Episede
Episode
Figure 2: The performance of eSpark and MARL basclincs in the MABIM 100 SKUs scenarios. indicates eSpark collects feedback here and regenerates the exploration function. The solid line represents the median, while the shaded region indicates the range between the maximum and minimum values.
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Performance in MABIM, a higher profit indicates a better performance. The "Standard" scenario features a single echelon with sufficient capacity. The "2/3 echelons' involves challenges of multi-echelon cooperation. The 'Lower/Lowest" scenarios are the challenges where SKUs compete for insufficient capacity, while '500 SKUs scenarios' assess scalability. The '_' indicates CTDE algorithms are not researched in the scalability challenges.
Avg. profits (K)
Method
100 SKUs scenarios
500 SKUs scenarios
Standard
2 echelons
3 echelons
Lower
Lowest
Standard
echelons
3 echelons
Lower
Lowest
IPPO
690.6
1412.5
1502.9
431.1
287.6
3021.2
7052.0
7945.7
3535.9
2347.4
QTRAN
529.6
1595.3
2012.2
70.1
19.5
QPLEX
358.9
1580.7
704.2
379.8
259.3
MAPPO
719.8
1513.8
1905.4
478.3
265.8
BS static
563.7
1666.6
2338.9
390.7
-1757.5
3818.5
8151.2
11926.3
3115.1
-9063.8
BS dynamic
684.2
1354.2
2378.2
660.6
-97.1
4015.7
8399.3
11611.1
3957.5
2008.6
(S)s)
737.8
1660.8
1725.2
556.9
203.7
4439.4
9952.1
10935.7
3769.3
2212.4
eSpark 823.7
1811.4
2598.7
579.5
405.0
4468.6
9437.3
12134.2
3775.7
2519.5
In Table I we also present the performance outcomes of the eSpark algorithm in the scaling-up 500 SKUs scenarios. Due to the centralized nature of the CTDF methods, they struggle to scale to large-scale problems and therefore are not presented in the table. Despite IPPO's markedly inferior performance on scenarios when problems scale up, eSpark exhibits significant enhancements and consistently achieves optimal results across multiple scenarios. We attribute this improvement to eSpark's s action space pruning strategy, which effectively addressos the heightened exploration nceds in scenarios with many agents, providing a clear advantage such complex environments.
5.2.2 Performance on SUMO
To further assess eSpark's capabilities across different tasks, we have compiled a summary of results in Table 2 based the SUMO enviromment. Similar to the outcomes in MABIM, eSpark consistently enhances the performance of the IPPO in all scenarios, and it has outperformed the CTDE baselines domain-specific MARL baselines to achieve the best performance. Notably, even when IPPO alone is capable of good results (as seen in scenarios such as Grid 4x4 and Cologne8), the pruning method designed in eSpark does not compromise the effectiveness of TPPO. We will delve further into the analysis of exploration functions produced by eSpark in Section 5.3
well
5.3 eSpark learns intelligent pruning methods
Given that eSpark employs the prior knowledge of the LLMs to craft its exploration function, our study aimed investigate two critical aspects: (1) the validity of action space pruning via prior knowledge, and (2) the potential advantages of this method over rule-based heuristic pruning.
To address the questions raised, we devise two pruning strategics. First, we implement a random pruning method, wherein agents randomly exclude a portion of actions during decision-making to test the validity of knowledge-basee pruning. Secondly, we utilize delain- OR algorithms to implement heuristic pruning methods. For MABIM, actions are pruned using the (S, 8) policy and unbound limit, while for SUMO, pruning relics on MaxPressure to keep only a few actions with the highest pressure. The details of these methods are presented in Appendix
D.3 Just like eSpark, these pruning strategies are integrated with
IPPO during training but not execution. We conducted experiments under the same setting in Section 5.1 with results presented in Tables 3 and Table
As shown in the tables, random pruning marginally affects performance by merely altering exploration rales without providing new insights. Heuristic pruning's impact varies with its design and context. In MABIM, (S, s) pruning less effective in the 100 SKUs scenario, as it restricts the already effective IPPO's exploration in smaller scales. However, it proves beneficial in the 500 SKUs scenario, where it guides the exploration and leads to better results. Upbound pruning consistently underperforns due to its overly simplistic heuristic.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Performance in SUMO, including the mean and standard deviation. A lower time indicates a better performance.
Method
Metric
Grid 4x4
Arterial 4x4
Grid 5x5
CologneS
Ingolstadt21
Delay
161.14+3.77
1229.68+166 16.79
820.88+24.33
85.27+1.2 +121
224.96+ 11.91
FTC
Trip time
291.48+4.445
676.77+13.70
584.54+24.17
145.93+0084
352.06 +9.29
Wait time
155.6623.44
521.86:13.33
441.633:21.13
58.92:00.6
161.22:177 7888
Delay
63.39++1.34
995.23-t77.09 72.02
242.85++:1600
31.63++0.61
228.64+115.88
MaxPressure
Trip time
174.68:12.08
702.09:1:3.66
269.35:+9962
95.67400.62 0.62
352.30+t 15.06
Wait time
37.37:11.06
511.06:1:22.55
114.36:16.48
11.03:10.28
159.44+ 13.34
Delay
48.40=00445
884.73+138.8
228. 188.78411.55
27.60:11.70
342.97:+43.61
IPPO
Trip time
160.12-:.60
506.18410.39
238.03++7.10
91.41+1.60
164.50+443.38
Wait time
22.69:10.38
435.44:177.65
918416631
7.70:10.82
267. 4-0.03
Delay
51.25+00.58
958. 188.772
221.62--00.75
32.55++4.66
347.59++44.59
MАPPO
Trip time
160.01++.554
757.40+2442.00
247.56+33.71
94.31+1777
180.66-+449.04
Wait time
25.41+00.54
609.80+-255.22
97.10++5.22
9.39++.553
283.59++-3..0
Delay
63.51:10.4
1142.98-79.
12,
37.93:10.45
196.74-9.88
MPLight
Trip time
172.47=20.86 0.99
483.211455.8 84
255.49-16.20
110.56
331.42=11.77
Wait time
40.32:+0.96
366.27+:58.00
126.42+-5.31
12.98+00.57
126.09 13.60
Delay
53.40=1.889
820.67
$39.664448.55
27. 17.40
296. 17:447-6668
CoLight
Trip time
165.77=1.89
470.33+12.34
305.41+44.413
1..66:1..9
410.59+
Wait time
27.25=1.64
312.47416.66
157.65:+35.69
9.35+1.09
215.98++90.62
Delay
48.36:10.32
854.22:168 21
209.49-13.98
25.39=1.27
243.92:115.88
cSpark
Trip time
159.7410.44
484.871:58.2
285.20-6..80
89.50=1.36
367.57-15.00
Wait time
22.58:+0.29
88.38-4.41
6.94+0.38
180.09+ 13.84
Table 3: Average performance changes
Table 4: Average performance changes on SUMO. All changes are relative to IPPO.
MABIM. All changes are relative to IPPO.
Avg. profits change ratio (%)
Avg. time change ratio (%)
Method
Method
SKUs
500 SKUs
Delay
Trip time
Wait
Random pruning (S,a) pruming
2.1
-0.5
Random pruming
-0.1
2.2
-2.5
-25.9
15.5
MaxPressure pruning
-0.5
1.5
-0.1
Uphound pruming
-23.2
-32.7
eSpark
-9.7
+5.7
-14.3
epark 39.1
29.7
For SUMO, pressure- based pruning does not offer signíficant benefits. Nevertheless, eSpark demonstrates remarkable adaptability across all testing tasks, adeptly selecting pruning methods that substantially enhance results. Its knowledge-basee generative technique and evolution capability enable it to master intelligent. pruning strategies.
IPPO
Frequency
Methods
Random pruning (5.s) pruning Upbound pruning eSpark
1e.1
demand)
Actions
Figure 3: Action selection frequency for IPPO and various pruning methods on the 100 SKUs Lowest scenario. Actions" represents the replenishment quantity is a multiple of the mean demand within the sliding window. eparr learns not only to minimize restocking but also to diversify with small purchases below the mean demand, balancing demand fulfillment and overflow prevention.
Figure g presents a frequency heatmap of action selection for IPPO and various pruning methods in the 100 SKUs Lowest scenario. IPPO learns a minimally restocking strategy, risking unmet demand. Random pruning chooses actions more uniformly yet mirrors IPPO's pattern. (S, pruning excessively exceeds mean demand, ignoring no-restock actions and leading to significant overflow. Upbound pruning typically avoids
10

---

Published in Transactions on Machine Learning Research (05/2025)
restocking, but prefers to purchase near the mean demand, which could result in overflow costs. In contrast, eSpark adopts a balanced policy, avoiding overstocking while diversifying its minor restocking strategies to meet demand without causing overflow.
5.4 eSpark benefits from retention training and action space reduction
Extensive research has underscored the importance of refloction in LLM-driven content generation Ma et al. (2024); ; Nascimento el al. (2023) Herein, we focus the effects of retention training and action pruning on eSpark's ' performance.
We first design an ablation experiment, which we refer to as the eSpark w/o retention. The model parameters are initialized when an iteration is finished, and the newly generated exploration functions are equippod, after which the training starts from scratch. Given that the initialized model needs a more extensive number of steps to converge, we accordingly triple the training steps per iteration in comparison to the standard eSpark. Another ablation retains the retention training, while the only difference that the LLMs and reflection are removed. We name this experiment eSpark w/o LLM. The comparative analysis of these two ablations is delineated in Table and Table 6 The detailed results are shown in Appendix F.3
Table 5: Average performance change across 100 SKU scenarios in the MABIM environment. All changes are relative to IPPO.
Table 6: Average performance change in the SUMO environment. All changes are relative to IPPO.
Method
Avg. profits change ratio (%)
Avg. time change ratio (%)
Method
eSpark
39.1
Delay
Trip time
Wait time
eSpark w/o retention oSpark w/o LLM
24.0
eSpark
-9.7
-5.7
-14.3
<park w/o retention
-9.6
-4.6
-11.2
eSpark w/o LLM
-9.1
-5.0
-12.8
The removal of retention training and LLMs both result in a decline in the performance of the eSpark. In the SUMO scenario, the performance gap between the two ablations and the complete eSpark relatively small, whereas it more pronounced in the MABIM scenarios. This can be attributed the fact that MABIM involves a greater number of agents and a more complex observation space action space, where superior pruning significantly enhance the performance of MARL methods. Additionally, observe that the lack of LLMs leads to significant decrease in performance on MABIM, emphasizing the central role of knowledge-basee action space pruning within the eSpark.
5.5 LLM checker and detailed reward feedback promotes the performance of eSpark
We go deeper to investigate the impact of feedback prompt design and the LLM checker on the performance of eSpark. eSpark utilizes detailed reward factors to evaluate policy performance comprehensively. To elaborate the significance of this design, perform an ablation experiment, eSpark w/o factors, in which the reward factors are removed, leaving only the total reward.
The introduction of LLM checker comes from the following observations: LLM code generator occasionally produces flawed exploration functions (e.g., variable misuse, misaligned task logic). Although these are usually eliminated during evolutionary search, when the pool of executable exploration functions is small, flawed yet exccutable functions may still be selected as oditing templates, thus hindering further refinement. We conduct the second ablation eSpark w/o checker by removing the LLM checker. Both ablations are performed in the MABIM (100 SKUs) and SUMO environments, with the performance averaged in respective environments and presented in Table and Table The detailed results are shown in Appendix F.3
After removing detailed reward information, eSpark showed a significant performance decline in both the MABIM and SUMO environments, particularly in MABIM, where complex transition logic and variable usage exacerbated the impact. Without dotailed reward factors, eSpark struggled to analyze policy performance and propose targeted improvements, leading to diminished ability refine the action space. We provide examples of eSpark's responses with and without reward factors in Appendix to offer more insights into
11

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7: Average performance change across 100 SKU scenarios in the MABIM environment. All changes are relative to IPPO.
Table Average performance change in the SUMO environment. All changes are relative to IPPO.
Method
Avg. profits change ratio (%)
Avg. Delay
time change ratio (%)
Method
Trip time
Wait time
eSpark
39.1
eSpark w/o factors eSpark w/o checker
17.8
eSpark
-9.7
-5.7
-14.3
26.2
eSpark w/o factors
-9.4
-13.5
eSpark w/o checker
-8.1
-3.1
-10.8
these results. Additionally, the ablation experiment on the LLM checker revealed its critical role in preventing flawed exploration functions from being selected as editing templates, further demonstrating the importance of both detailed reward feedback and the LLM checker maintaining eSpark's overall performance.
6 Conclusions, limitations and future work
We present eSpark, novel framework for generating exploration functions, leveraging the advanced capa- bilities of LLMs to integrate prior knowledge, generate code and reflect, thereby refining the exploration in MARL. eSpark has surpassed its base MARL algorithm across all scenarios in both MABIM and SUMO environments. In terms of pruning strategies, pruning based on the prior knowledge from LLMs outshines both random and heuristic approaches. Ablation experiments demonstrate the indispensable role retention training in accurately improving exploration functions based on policy flaws and enhancing sample efficiency. The LLM checker and detailed policy feedback prompt design together ensure the superior performance of eSpark.
Nevertheless, eSpark also has certain limitations. First, currently eSpark is only applicable to tasks involving homogencous agents. For heterogencous agents, a potential method could be to generate distinct exploration functions for each agent; however, this approach becomes impractical when the number of agents is too large. Moreover, eSpark benefits from policy feedback to refine the exploration functions. When feedback is not informative regarding how to modify the exploration (c.g., tasks with sparse rewards, end-of-episode feedback alone is too limited to develop automated feedback), eSpark may struggle to improve and need extra expert. input for effective reflection.
Future work encompasses numerous potential directions. Existing rescarch advocates for assigning different roles or categories to agents Christianos et al. (2021): Wang et al. (200)), which could offer a compromise for the application of eSpark in heterogeneous multi-agent systems. Furthermore, state-specific feedback for more granular improvement represents an intriguing avenue Subramanian et al. (0016). Our future endeavors will investigate these questions, striving develop algorithms that are robust and exhibit strong generalizability.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyanal Anadkal, et al. Gpt-4 technical report. ar preprin! SSf::2303.08774,, 2023.
Prithviraj Ammanabrolu and Mark O Riedl. Playing text-adventure games with graph-based deep reinforce- ment learning. arXiv preprint arXiv: 11111111.10122 2018.
Kenneth Arrow, Theodore Harris, and Jacob Marschak. Optimal inventory policy. Econometrica: Journal of the Econometric Society, pp. 250-272, 1951.
Peter Aucr. Using confidence bounds for exploitatiion-colloaaiin tradc-offs. Journal of Machine Learning Research, pp. 397 422, 2002.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models, orXiv preprint ar Xix:22108. 07732, 2021.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Michael Behrisch, Laura Bicker, Jakob Erdmann, and Daniel Krajzewiez. simulattn of urban mobility: an overview. In SIMUL, 2011.
Alan S Blinder. Inventory theory and consumer behavior. Harvester Wheatsheaf, 1990.
Yuri Burda, Harrison Edwards, Amos Storkey, and Olog Klimov. Exploration by random network distillation.
In ICLR, pp. 1 17, 2018.
Harris Chan, Yuhuai Wu, Jamic Kiros, Sanja Fidler, and Jimmy Ba. Actree: Augmenting experience via teacher's advice for multi-gont multi-goal reinforcement learning. arXiv preprint MXKi::992.005444 2019.
Jonathan Chang, Kiante Brantley, Rajkumar Ramamurthy, Dipendra Misra, and Wen Sun. Learning to generate better than your Ilm. arXiv preprint #XXv:2306.118166 2023.
Chacha Chen, Hua Wei, Nan Xu, Guanjic Zhong, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li. Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic sigual control. In AAAL, pp. 3414 3421, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. Xáv preprint arXiv:2107. 03374, 2021.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, al. Agentversc: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. Xii preprinl urXiv:2308. 110008...888 10848, 2023.
Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrocht. Scaling multi-agent reinforcement learning with selective parameter sharing. In JCML, pp. 1989-1998, 2021.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arx:::001 1.09533, 2020.
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In JCML, pp. 8657-8677, 2023.
Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunchag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXee preprint XXXv:1512.076799 2015.
Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. What can you do with rock? affordance extraction via word embeddings. Xrxiv preprint arXin:1 703.03349, 2017.
Qianwen Gou, Yunwei Dong, Yujiao Wu, and Qiao Ke. Rrgeode: Deep hicrarchical cearch-based code generation. Journal of Systems and Software, 211:111982, 2024.
Oliver Groth, Markus Wulfmeier, Giulia Vezzani, Vibhavari Dasagi, Tim Hertweck, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Is curiosity all you need? on the utility of emergent behaviours from curious exploration. Xi preprint (xXiv:210.08660) 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, pp. 1861 1870, 2018
Jianye Hao, Xiaotian Hao, Hangyu Mao, Weixun Wang, Yaodong Yang, Dong Li, Yan Zheng, and Zhen Wang. Boosting multiagent reinforcement learning via permutation invariant and permutation equivariant networks. In ICLR, 2022a.
Xiaotian Hao, Hangyu Mao, Weixun Wang, Yaodong Yang, Dong Li, Yan Zheng, Zhen Wang, and Jianye Hao. Breaking the curse of dimensionality in multiagent state space: unified agent permutation framework. aix preprint ar arxi::2003.0522 2022b.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human Instruction-followig with deep reinforcement learning via transfer-learning from text. orXiv preprinl (rXii:2005-00382 2020.
Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordination. In
ICML, pp. 13584-13596 2023.
Biwei Huang, Chaochao Lu, Liu Leqi, José Miguel Hernández-Lobato, Clark Glymour, Bernhard Schölkopf, and Kun Zhang. Action-sufficien state representation learning for control with structural constraints. In International Conference on Machine Learning, pp- 9260 9279. PMLR, 2022a.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, pp. 9118 118899147, 2022b.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In CoRL, pp. 1769 1782, 2023.
Haoyuan Jiang, Ziyue Li, Zhishuai Li, Lei Bai, Hangyu Mao, Wolfgang Ketter, and Rui Zhao. A general scenario-agnostic reinforcement learning for traflic signal control. IEEE Transaclions on Intelligent Transportation Systems, 2024.
Anastasios Kouvelas, Jennie Lioris, Alireza Fayazi, and Pravin Varaiya. Maximum pressure controller for stabilizing queues in signalized arterial networks. Transportation Research Record, 2421(1):133 141, 2014.
Minac Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In ICLR, 2023.
Yixing Lan, Xin Xu, Qiang Fang, Yujum Zeng, Xinwang Liu, and Xianjian Zhang. Transfer reinforcement learning via meta-kme knowledde extraction using auto-pruned decision trees. Knowledge 108221, 2022.
Baaeed Bosed Systems, 242:
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforccment learning. Advances in Neural Information Processing Systems, 35:21314 21328, 2022.
Dapeng Li, Zhiwei Xu, Bin Zhang, and Guoliang Fan. From explicit communicttion to tacit cooperation: A novel paradigm for cooperative marl. arXiv preprint ar Xix:2904.14000 14656, 2023a.
Dapeng Li, Hang Dong, Lu Wang, Bo Qiao, Si Qin, Qingwei Lin, Dongmei Zhang, Qi Zhang, Zhiwei Xu, Bin Zhang, et al. Verco: Learning coordinatod verbal communication for multi-agent reinforeement learning. arXiv preprinl 2xx::2404.17788 2024.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint (tt::230500111. 2023b.
Zachary C Lipton, Kamyar Azizzadnnshhlli, Abhishek Kumar, Lihong Li, Jianfeng Gao, and Li Deng. Combating reinforcement learning's sisyphean curse with intrinsic fear. arXiv preprint arXiv: 112111, 2016.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto- completion systems. arXiv preprint arXi::2306.0300) 2023.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Tgor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. In NewrIPS, 2017.
Jiaming Lu, Jingqing Ruan, Haoyuan Jiang, Ziyuc Li, Hangyu Mao, and Rui Zhao. Dualight: Enhanc- ing traffic signal control by leveraging scenario-specific and scenario-shared knowledge. ar preprint 2r 24.553 14532, 2023.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Fureka: Human-level reward design via coding large language models. In ICLR, pp. 1 39, 2024.
Patrick Meier. Digital humanitorians: how big data is changing the face of hamanitarian response. Crc Press, 2015.
Seyed Sajad Mousavi, Michael Schukat, and Enda Howley. Traflic light control using deep policy-gradient and nalue-function-basee reinforcement learning. IET Intelligent Transport Systems, 11(7):417 1241112444,,,- 2017.
Md Shirajum Munir, Nguyen H Tran, Walid Saad, and Choong Scon Hong. Multi-agent meta-reinforrement learning for sewerpo and sustainable edge computing systems. IEEE Transactions on Network and Service Management, 18(3):3353 33333-377, 2021.
Srinarayana Nagarathinam, Vishmu Menon, Arunchandar Vasan, and Anand Sivasubramaniam. Marco multi-agent reinforcement learning based control of building hvac systems. In ACM e-Energy, 2020.
Nathalia Nascimento, Paulo Alencar, and Donald Cowan. Gpt-in-the-loop: Adaptive decision-making for multiagent systems. arxiv preprint or Xii:23308. 10435, 2023.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32:289-355 2008.
Venkata Ramakrishna Padullaparthi, Srinarayana Nagarathinam, Arunchandar Vasan, Vishmu Menon, and Depak Sudarsanan. Falcon-farm level control for wind turbines using multi-agent deep reinforcement learning. Renewable Energy, 181:445-45 45655-556641116 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, pp.
2017.
Tabish Rashid, Mikayel Sanvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21 (1):73234 -2844
7284, 2020.
Roger P Roess, Elena S Prassas, and William R McShane. Traffic engincering. Pearson/Prentice PPrentice Hall, 2004.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint 2308 0888.12911 12950, 2023.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. ACL, pp. 1713 1726, 2022.
Ali Shirali, Alexander Schubert, and Ahmed Alaa. Pruning the way to reliable policies: A multi-objocctive deep l-lannning approach to critical care. orXiv preprint arXiv:2306. 08044, 2023.
Herbert A Simon. Rational choice and the structure of the environment. Psychological review, (3(2))129, 1956.
Kyunghwan Son, Dacwoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agen reinforcement learning. In ICML, pp. 5887-5896, 2019.
Kaushik Subramanian, Charles L Isbell Jr, and Andrea Thomaz. Exploration from demonstration for interactive reinforcement learning. In AAMAS, pp. 447-456, 2016.
Hao Sun and Taiyi Wang. Toward causal-aware rl: State-wise action-refined temporal difference. arXiv preprint ar xrXi:22011.000 00354, 2022.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arv preprinl arXiv:2305.16291 2023.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex dueling multi-agent. Q-lcarning. In ICLR, pp. 1-27, 2021.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles, arXiv preprint Xrv1:22033 000009 2020.
Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. Colight: Learning nelwork-level cooperation for traflic signal control. In CIKM, pp. 1913-1922, 2019.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint Xrx:::20090000 2021.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. 2rx(2reward: Aulomated dense reward function generation for reinforcement learning, arXic preprinl arXiv:2309.110 11489, 2023.
Xianliang Yang, Zhihao Liu, Wei Jiang, Chuheng Zhang, Li Zhao, Lei Song, and Jiang Bian. A versatile multi-agent reinforcement learning benchmark for inventory management. arXiv preprint ar arXiv:2306.075 07542, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, pp. 1-33, 2023.
Donghao Ying, Yunkai Zhang, Yuhao Ding, Alec Koppel, and Javad Lavaei. Scalable primal-dual actor-critic method for safe multi-agent RL with general utilities. In NeurIPS, 2023.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising 2022.
effectiveness of ppo in cooperative multi-agent games. In NewrIPS, pp.
Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4me: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint ar 2r3 16563, 2023.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Learn what not to learn: Action climination with deep reinforcement learning. In NewrIPS, 2018
Bin Zhang, Hangyu Mao, Jingqing Ruan, Ying Wen, Yang Li, Shao Zhang, Zhiwei Xu, Dapeng Li, Ziyue Li, Rui Zhao, et al. Controlling large language model-based agents for large-scale decision-making: An actor-critic approach. arXie preprint X11:22311..1888, 2023.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pp. 321 181-384,27 2021.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Rumxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepscck-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprinl ar Xic:2400 11931, 2024.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, al. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. ar preprint or 2rXi:23355.11144 2023.
Alessandro Zocca. Temporal starvation in multi-channel csma networks: an analytical framework. ACM SIGMETRICS Performance Evaluation Review, 46(3):52 53, 2019.
16

---

Published in Transactions on Machine Learning Research (05/2025)
A Notations
All the notations used are summarized in Table 9
Notation
Definition
Combined policy of all agents
Policy of agent k
Ground truth state
Combined action of all agents
Action of agent k
Combined observation of all agents
Partial observation of agent
Discount factor
Reward obtained from environment
P(-|s,a
Transition function of environment
E
Exploration function generated by LLM
Tg
F
Policy of agent k under exploration function E
PE
Policy feedback from policy k Performance evaluation of policy k
Table 9: Notations and Definitions
B Proofs
Proposition 1.
1. For any E. E (F), (m S (T). If E is non-trivial, then (x²) (xr).
2. For any E {T), there cxists non-trivial F. E (F} such that J( B) 2 J(T).
Proof. We begin by offering the proof the first statement in Proposition 1 We denote { as the set of possible policies for agent k with each Ti satisfying the following two conditions:
T : O2 x A [0 Eares F (ak Okk =1
5
Let {T() } be the set of policies under exploration function E. For every element in क. E (=f),, of the two situations exists:
If E is trivial, then of ox) = Tx ox), hence o E {mK}.
2. If E is non-trivial, then 7. on) Emecas
[((oo))-E(oo)) mi
It is clear that 0 05 =f( |0)) S 1 and
Emes, (a oc) = 1, thus oh {mK).
Therefore, we know that:
{mh) C {TT).
(6)
Given that for Vi(: | 84) = I TT(
oz), we have æ E (r), where each mi belongs to {Tx)- According
Formula 6 it is known that for F. s²) = II xf( ox). it belongs to {T). Then:
( C (T).
(7)
17

---

Published in Transactions on Machine Learning Research (05/2025)
When E is non-trivial, TH { still holds, but Th { may not be true (i.e., when TT(a |oa) 0 for Vou E A, Th $ (m²)). Hence we can gel:
(mk C {Tk).
In similar manner, we can deduce that if there exists a [1, ..,,, M such that me {mh) then & { {T²), which means:
{ c {T}.
Therefore, we finish the proof of the first statement.
To proof the second statement, it is necessary to introduce a series of variables. We define the value function and state-action function for π as follows: Vs)(s) =
Ei=0 7'r² S = s| and Qm(s, a) = a]. The advantage function is defined as (s a) (s, - (s). The joint exploration function is introduced as E(s,-) = -IR= E(ox.)) The relationship between V and Qx (s, a) can be formulated as:
Bat [E==o 7 | = s. aa]
(i( = Σ T(a ())=(s,
(10)
For non-trivial E, the value function of π can be written as:
(s) = Σ a(saa)
aEAL Daca, B(s,a) (x,
E(8, ()Qx(s.a)
EacAe ((s,a)T(a acAL
(11)
Vi(() Σ (1 E(s a() s) (
EneA, Av a(s,a)T(a))
aEA:
Thus, we have:
V*E(s) (s( (s) EnEA. (1 E(s, a)) T(a | ()) Vs(s)
ExEA. ((ssa))
(1 - (1-Eae EaEA. a(s.a) 5)) (s(s) EaSA. (1-E(s,ay)) T(a ss)
EAEA. E(s, a) TT
(12)
EaEs. (1 E(s,a) π V EaEA. E-E((s,)) a(a (s)
EaEA. E(s,a)
EacA (1 (1-E(s, E(s,a) π (a
8)Ar (s,
E.EA. E (..))m a) 8)
When - -EEEA (1 a)) (aa s)A a) 2 0. which means the expectation of the advantage value for the pruned actions is less than or equal to 0, then (8) > (s). Because for every E S, E.EA, A (s, Eacsi Q (s, a) V (s) 0, there always exist actions for which the advantage function values are less than equal to zero.
As J( - J(T)= = Egerp (80) - V=())) if an exploration function E can satisfy the condition that for all states & E S, the incquality E.SA. (1 E(s, T(a s)A 2 0 holds, then it can be guaranteod that B 2 J(r).
Therefore, we finish the proof of the second statement.
Proposition 2. Let's consider fully cooperative game with agents, one state, and the joint action space 1 where the reward is given by r(0", 1N) rı and r(oN-m 11 =-mr for all m # N, Tı, T2 are posilive real numbers. We suppose the initial policy is randomly and uniformly inilialized, and the policy is
18

---

Published in Transactions on Machine Learning Research (05/2025)
optimized in the form of gradient descent. Let be the probability that the shared policy conterges to the best policy, then:
T2
N-1
(13)
Nr2
Proof. Clearly, the best policy is the deterministic policy with joint action (0", )
Now, let the shared policy be (1 - 0),, where 0 is the probability that an agent takes action 1. The expected reward can be written as:
N-1
((0) Pr (a 2NN = -00, (00,111 ri
Σ
Pr (a = -00N~111, - 12
N-1
,N .1
Σ
t. 0 -8)
(14)
t=
N
=
t 02 (1 - - + N 0V r2
t=
where Oy is combinatorial number. We need to simplify Σ t. (1 0)N for further analysis.
Notice the structural similarity between the results and the binomial theorem:
(1- -)0
-1-
(15)
t=0
We take the derivative of 8 on both sides of Formula Then:
15
Because the left side is constant, its derivative is 0.
d C/0(11 0-
d0
Clt 0--1 (1- +C - -t) (-1) (1 - N- 0²
(16)
Σ
Cl (1 0) 0)N N- 0--1 ((1-0)- 0)- (N 1)0)
Σ 1=
C (1 - 0) N NO)..
Thus, we have:
N
N
Ctt c (1 0)N N-t-1t
Σ
10 - --
t=
1=0
(17)
N
N
NO C1 - N =
Σ 0=
Σ
IC (1 N'88
Notice that the left side of the equation is the expansion form of Formula 15 and the right side of the equation is the desired Formula, we can get:
Σ A
No.
(18)
Bring Formula 18 back to Formula 14 we get:
J(O) ,N .r Nor2 N. ON r2z
(19)
19

---

Published in Transactions on Machine Learning Research (05/2025)
In order to maximise J(),, we must maximise 8% (r1 + Nr2) - -N8 Since the policy optimization usually
adopts a gradient manner, we calculate the derivative of Formula 19 with respect to 0 as:
dJ(()) Ng (r1 Nr2) Nr2-
(20)
d0
the point === N-1 PrtNr2 is the only zero of dJ(0) de
When 0 S 8", 00,
dJ(0) 0 00 02 0", aJ(0) 0.
de
Remember are trying to maximize J(f) through gradient way, and then the policy improves the parameters in the direction of the gradient. As the imitial policy is randomly and uniformly initialized, the 0 is uniformly distributed in the interval [0.1], then the probability that the shared policy converges to the bost policy is:
p p=1-
N-1
72
(21)
r1 + Nr2
Therefore, we finish the proof of Proposition
2
C An example of GPT performing action space pruning
In this section, we present a challenging sequential decision-making example to show GPT's capability of reasoning and code generation for pruning redundant action spaces.
GPT's input:
Consider a resource allocation game with agents. Each agent has discrete action space 10,1, representing the of resources allocated each step. goal maximize team's cumulative utility while satisfying dynamic constraints that evolves over time.
Game Rules:
1. Utility Function: The team's cumulative utility is the sun of all agents' contributions, defined as: = Σi= log(1 + a.)
U(a1, (2..., an)
2. Constraints: At each time step t, global constraint limita the total allocation Σ as S Ct allocation exceeds Ct, penalty proportional subtracted from the utility: P(excesa) k (excess)), where k positive constant.
Penalty Violations:
is
Please write an exploration function that masks invalid bids and bids likely to lead to inefficient budget usage (e.g., overbidding vhen the revard is low) for each agent. The function signature is:
def compute_mer mask (current allocations, actions_space constraints): return total_mask
sezrent_lllocatinns Current resource allocation agents.
actions_edt ssaions_pppce: Available actions for each agent.
constraints: The dynamic constraint Cr.
GPT's output:
In this game, the exploration function must account for the following:
1. Dynamic Constraints: Predict whether an action will exceed C, given the current state other agents' possible actions.
2. Utility Contribution: Estimate the marginal utility of each action and prioritize those with higher
expected contributions.
3. Penalty Avoidance: Mask actions likely to trigger penalties due constraint violations.
import numpy as np
def compute_mast mask Cuureent_alocatiinn
actions_space, constraints):
total_mast [] for action in actions_ space::
20

---

Published in Transactions on Machine Learning Research (05/2025)
rredicted_allocttion sum sumuuurrett_llloctionn. action
# Check if the action violates the constraint
if predicted_allocation constraints:
total_masti append (0) Mask the action
else:
Estimate marginal utility and penalize low contribution marginal_utility np. 1p + action) penalty -0.1
allocatiooo constraints) ** 2 if > constraints else 0 # Mask actions with low marginal utility or high penalties
if narginal_ttility penalty < 0.5: Threshold chosen heuristically
total_mask oopaenn append (0)
else:
appeed total_man _asal append (1)
return _ask
Detailed settings
D.1 MABIM details
MABIM is a simulation environment. dedicated to leveraging MARL. to tackle the challenges inherent inventory management problems. Within MABIM, each stock SKU every echelon is represented as an aulonomous agent. The decision-making process of each agent reflects the procurement requirements for the specific SKU at its corresponding echelon.
Rep Replenishment
Rep leppeenshment
Product
Product
Factory
Warehouse
Figure 4: MABIM inventory model.
Each time step involves the agent making decisions regarding replenishment quantities for SKUs and subsequently transitioning the environment to new state. Let M E Z be the total warehouses, with the first one being closest to customers, and N E Z the total SKUs. Given a variable X E (D, L... X represents its value for the j-th SKU in the i-th echelon at step t, with < M and 0 S j < N. Given the above notations, the main progression of a step can be described as follows:
D = R
(Replenish)
5 min Di I1,))
(Sell)
1-1
A
Σ k=O
I(k: + L = 1) S
(Arrive)
W; W.--E, E, Σ, (,, Alj
7. = min
Big = LAi 7
(Receive)
Th-1
Ii S% + B
(Update)
Here, D, D, = and I(condition) is an indicator function that returns the condition true, and O otherwise. For the topmost echelon, orders channeled to super vendor capable of fulfilling all
21

---

Published in Transactions on Machine Learning Research (05/2025)
order demands at that level. Orders from other echelons are directed to their immediate upstream echelons, where the demands are satisfied based on the inventory levels of the upper echelons. The demand at the bottom echelon is derived from actual customer orders captured within real-world data sets. The reward function within MABIM is meticulously calibrated based on the economic realities of inventory management, integrating five fundamental elements: sales profit, order cost, holding cost, backlog cost, and excess cost. The summation of these elements constitutes the reward value, thereby incentivizing agents to optimize inventory control for enhanced profitability and operational efficacy.
MABIM incorporates challenges across five key categories: Scaling up, Cooperation, Competition, General- ization, and Robustness. We concentrate on the challenges associated with Scaling up, Cooperation, and Competition, as these challenges not only manifest in inventory management problems but also exist in a broad range of MARL tasks. We catalog the number of agents, challenges and degrees difficulty within all the experimental scenarios Table T0 The specific setting of each scenario is given in Table
Table 10: Tasks and corresponding challenges. denotes the extent of the challenges.
Challenge Scaling up Cooperation Competition
Task name
Agents number
Standard (100 SKUs)
100
2 echelons (100 SKUs)
200
3 echelons (100 SKUs)
300
Lower capacity(100 (100 SKUs)
100
Lowest capacity (100 SKUs)
100
Standard (500 SKUs)
500
2 echelons (500 SKUs)
1000
3 echelons (500 SKUs)
1500
++
Lower capacity (500 SKUs)
500
Lowest capacity (500 SKUs)
500
Table 11: Experiments settings.
"#SKU No indicates N times the mumber of SKUs.
Task name
Ecchelon
#SKU
Capacity per echelon
Standard (100 SKUs)
1
100
#SKU*100
echelons (100 SKUs)
2
100
#SKU*100
echelons (100 SKUs)
3
100
#SKU*100
Lower capacity(100 SKUs)
1
100
#SKU*50
Lowest capacity (100 SKUs)
1
100
#SKU*25
Standard (500 SKUs)
1
500
#SKU*100 100
2 echelons (500 SKUs)
2
500
#SKU*100
echelons (500 SKUs)
3
500
#SKU*100
Lower capacity (500 SKUs)
1
500
#SKU*50
Lowest capacity (500 SKUs)
1
500
#SKU*25
For each scenario, we carry out three independent runs. Performance is reported as average test set profits from the top model in each run.
D.2 SUMO details
SUMO is an open-source, highly portable, microscopic and continuous road traffic simulation package designed handle large road networks. In the SUMO simulation environment, each intersection conceptualized autonomous agent equipped with array of predefined traffic signal phases. These phases orchestrate
22

---

Published in Transactions on Machine Learning Research (05/2025)
the traffic flow across the intersection's multiple approaches. The selection of these phases, driven by the assessment of live traffic conditions, is aimed at attenuating road congestion and enhancing the fluidity of vehicular movement through the network, thus contributing to the overall efficiency of urban traffic management.
conduct thorough evaluation of each algorithm, we select a total of five scenarios from both synthetic and real-world datasets. These datasets encompass diverse array of intersections, varying number and type. The intersections classified according to their configuration into three categories: bi-directional (2-arm), tri-directional (3-arm), and quadri-directionll (4-arm), indicating the number of exit points at each junction. We summarize the type of cach datasct, the number of intersections included, and the classification of these intersections in Table
Table 12: The categories of each SUMO dataset, along with the number and types of intersections included.
Dataset
Category
Intersections number
2-arm
3-arm
4-arm
Grid 4x4
synthesis syuthesis
16
0
16
Arterial 4x4
16
16
Grid 5x5 Lu al. (0233
synthesis
25
25
Cologne8
real-world
8
Ingolstadt21
real-world
21
17
4
To facilitate a homogeneous observation and action space conducive to the deployment various MARL algorithms, we employ the GEneral Ecenario-Agnostic (GESA) framework parse each intersection into a standardized 4-arm intersection with eight potential actions.
D.3 Model details
We employ IPPO as the base MARL algorithm for eSpark due to its ability to scale to large-scale MARL challenges. We select GPT-4 our LLM, and LLMg. specifically opting for the 1033009-01-preiia version. The temperature of GPT-4 is set to 0.7, with no frequency penalty and presence penalty. For each scenario, we conduct three runs, setting the batch size for each generation of exploration functions to K 16. This batch size is chosen because guarantees that the initial generation contains at least one executable exploration function for our environment. We limit the number of training iterations to 10, as we observe that the performance for most scenarios tends to converge within this number of iterations.
E Baseline details
In our experiments, employed three categories of baselines: OR baselines, MARL baselines and pruning baselines. In Table D we list the characteristics and environment of all the baselines utilized in our study, In the rest of this section, we will elucidate the underlying principles of cach OR bascline, articulate the design of the pruning baselines, and present the hyperparameter for the MARL baselines.
E.1
OR baselines
E.1.1 Base stock algorithm
The base stock algorithm constitutes a streamlined and ellicacious approach for inventory control, whereby replenishment orders are initiated upon inventory below a predefined threshold level. This policy is traditionally acknowledgod as a fundamental benchmark, favored for its straightforwarddess and rapid implementation. The computation of the base stock level facilitated through programmatic methodology, as explicated in Equation 22
23

---

Published in Transactions on Machine Learning Research (05/2025)
Table 13: All the baselines used the experiments
MARL baseline CTDE
Used environment
Algorithm name
OR baseline
Pruming baseline
DTDE
MABIM SUMO
Base stock (BS)
Arrow et al.
1951
(S, s)
Blinder 1990
FTC
Roess et al.
2004
MaxPressure Kouvelas al. 2014
IPPO
QTRAN QPLEX
MAPPO
MPLight Chen et al.
2020
CoLight Wei et al.
2019
Random pruning
(S pruning
Upbound pruming
MaxPressure pruning
o1 = 5/. C.j S/ his 1111 Cag T ēij 12
max
s.t /-11 1i + Sitis Shh
T Th, Str + $:-1.
S min(T (Ris)
(22)
The Σ Si+1. (=-
Zi -1t1 Si+1. + Ti.
Zij E RR..
In the above equations, i, j, and t are indexes for the warehouse, SKU, and discrete time, respectively, The indicators p, c, h and L represent the average selling price, cost of procurement, cost of holding, and lead time. The variables S, R, and denote the quantities associated with sales, orders for replenishment, inventory in stock and inventory in transit. of, describes the profit. objective, while z.j is indicative of the base stock level.
We utilize two approaches for computing stock levels. The first approach, named BS static, involves calculating all base stock levels with historical data from the training set, which are then applied consistently to the test set. The levels remain unchanged during the test period. The second approach, termed as BS dynamic, compules slock levels directly on the lest set relying on historical data and updates on regular basis
E.1.2 (S,s) algorithm
The (S,s) inventory policy serves as robust framework for managing stock levels. Under this policy, a restocking order is triggered once the inventory count falls below a predefined threshold, identified as s. The objective of this replenishment to elevate the stock to its upper limit, designated as S. Empirical analyses have substantiated the efficacy of this protocol in optimizing inventory control processes. As a result, is adopted as benchmark heuristic, with the aim of algorithmically ascertaining the most efficacious (S. parameter for each discrete SKU in the given inventory dataset. Tn implementation, conduct search
24

---

Published in Transactions on Machine Learning Research (05/2025)
the training set to identify the optimal values of and S, after which we apply these values consistently to the test set.
E.1.3 Fixed-time control algorithm
The Fixed-Time Control (FTC) algorithm is a traditional traffic signal control strategy predicated on predefined signal plans. These plans are typically designed based on historical traffic flow patterns and do not adapt to rcal-time traffic conditions. The FTC operates on a static schedule where the signal phases at intersections change at fixed intervals. This approach straightforward and easy to implement but may not. be optimal under variable traffic conditions due its lack of responsiveness dynamic traffic demands.
In implementation, the FTC follows fixed sequence of signal phases: 'WT-ET', 'NT-ST, 'WL-EL', 'NT-SL', "WI-WW', 'EI-ET' 'SI-ST' 'NI-NT: Here, 'W', 'E', 'N', and 'S' denote westbound, eastbound, northbound, and southbound traffic, respectively, while "T" indicates through movement, and 'L' signifies a left turn. Each phase has a duration of 30 seconds
E.1.4 MaxPressure algorithm
The MaxPrcssure algorithm represents a more advanced traffic signal control strategy that dynamically adjusts signal phases in response real-time traflic conditions. It calculates the 'pressure' each intersection, defined as the difference between the number of vehicles on the incoming and outgoing lanes. The algorithm aims optimize traffic flow by selecting signal phases that reduce the maximum pressure across the network, thus alleviating congestion and enhancing network throughput. Unlike FTC, MaxPressure is adaptive and can continuously optimize signal timing based on the current traffic state, making it more suitable for managing fluctuating traffic volumes.
E.2 Hyperparameters settings for MARL baselines
The following table enumerates the hyperparameters employed during the training process for all MARL baselines. For all test scenarios, training is performed with a uniform suite of hyperparameters that have not. undergone specializod fine-tuning.
Table 14:
Hyperparameters of MARL Algorithms Used in MABIM and SUMO Environments. < indicates that the algorithm is not set or does not contain this hyperparameter.
MABIM environment
SUMO environment
Hyperparameter
IPPO
QTRAN
QPLEX
MAPPO
IPPO
CoLight
MPLight
Training steps
5020000
5020000
5020000
5020000
2400000
2400000
2400000
Discount rate
0.985
0.985
0.985
0.985
0.985
0.9
0.9
Optimizer
Adam
Adam
Adam
Adam
Adam
RMSProp
RMSProp
Optimizer alpha
0.99
0.99
0.99
0.99
0.95
Optimizer eps
lo-5
lc-5
1c-5
1c-5
1c-5
1c-7
1c-7
Learning rate
5o-4
5c-4
5c-4
5c-4
5c-4
1c-3
1e-3
Grad norm clip
10
10
Eps clip
0.2
0.2
Critic coef
0.5
0.5
0.5
Entropy coef
0
Accumulated episodes
8
10
Number neighbors
E.3
Pruning baselines
E.3.1
Random pruning
Random pruning implemented by randomly masking certain percentage of the available actions. During the action selection process, each agent will have p percent of its available actions randomly masked. To
25

---

Published in Transactions on Machine Learning Research (05/2025)
balance the observability of the pruning's impact with the preservation of the algorithm's capacity to utilize prior experience, we set p 0.3.
E.3.2 (S) pruning
According the (S,s) algorithm, for given SKU, a replenishment quantity of A - is ordered when the current inventory lovel falls below the threshold otherwise, order is placed. We extend the reference replenishment quantity A to a range (ri x A, x A], where T1:72 are both real nunbers and 0 S r1 S 1 and r2 2 1. Actions within this interval are deemed available, while those outside of this range are masked. In our implementation, we select r = 0.5 and T2 = 2.
E.3.3 MaxPressure pruning
The MaxPressure pruning method utilizes the hcuristic concept of 'pressure' at an intersection to prune actions. We calculate the pressure associated with each action, and these pressures are then ranked. The actions with the top-k highest pressures are rendered available for selection. Actions not meeting this threshold are subsequently masked.
A standardized intersection warped through the GESA modeled as a four-arm intersection comprising eight. potential actions. We empirically set k to ensure effective pruning while maintaining sufficient number of available actions.
F
Additional results
F.1
Computational costs
We present the token assumption of eSpark in Table 15 scenario tasks within the same environment, we calculate the average token consumption for all scenarios with each environment.
Since we do not design specific prompts for different.
Table 15: Average token assumption for MABIM and SUMO.
Environment
Token assumption (M)
MABIM
3.0
SUMO
2.6
The training time and GPU memory usage for eSpark and the baselines across different scenarios are presented
in Table 16 and Table
Table 16: GPU memory usage of eSpark and MARL baselines.
Table 17: Running time eSpark and MARL.
baselines.
GPU memory usage (G)
Running time (h)
Method
Method
Standard 2 echelons
3 echelons
Standard
2 echelons echelons
eSpark
27.2
33.9
42.4
eSpark
18
25
30
IPPO
2.3
3.4
4.4
IPPO
QTRAN
4.2
6.5
8.7
QTRAN
9 15
21
F.2 Detailed results of the pruning methods
In this section, we provide the detailed results for multiple pruning basclines as discussed in Section along with results of eSpark for comparison.
5.3
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table 18: Detailed performance of various pruning methods in MABIM.
Avg. profits (K)
Method
100 SKUs socmarios ochelons
500 UK secmarion
Standard
echelons
Lower
Standard
echelons
echelons
Lower
Random pruning
T33.0
1407.6
1426.6
511.6
262.0
2718.0
8667.4
9161.1
2535.5
22012.1
(S. a) pruning
394.4
832.3
9333.1
441.1
258.3
38813
9248.3
10282.1
3517.9
20S5.6
Upbound pruming
745.0
630.2
-2.2
637.7
29-17
3261.3
2473.6
1637.6
2833.0
2167.7
eSpark
823.7
18114
2598.7
579.5
105.0
1168.6
9137.3
12131.2
3175.7
2519.3
Table 19: Detailed performance of various pruning methods in SUMO, includes the mean and standard deviation.
Method
Metric
Grid 4x4
Arterial 4x4
Grid 5x5
Cologne8
Ingolstad121
Delay
49.07.10.36
858.33.148 48.20
238.57 238.5719.44 39.45
25.89.11.34
353.38 1:2.33
Ramdom pruning
Trip time
160.13+0.58
548808866.88
241.92=9.60
1.27
478.53=22.54
Wait. time
22.66+0.144
387.25+-433
93.49-+8.09
7.03++0.37
281.97+21.90
Delay
08.78
890.041121 121.50
231.27:::.22 2.224
26. 0.:2003
337 02.
MaxPressure pruming
Trip time
160.72+:0.1
633.36:78... 78.08
253.68::17 188:17.68 .8
90.362 .888
168.11=65.32
Wait time
23.03.10.56
391.96.178.30
7.33:10.12
257 17.98166..1 11.91
Delay
48.36=+0.32
684.22+68.00 21
209.49++13.99
25.39++1.22
243.92+155. 15.81
eSpark
Trip time
159.74++0444
484.87+58.2.
235.20+6.880
89.50+ 1.36
367.57 15.03
Wait time
22.58:0.229
128.82416..70
88.3814441
6.9420.38
180.09=13.84
F.3 Detailed results of the ablations
In this section, we provide the detailed results for ablations in Table 20 and Table 21 eSpark for comparison.
along with results of
Table 20: Detailed performance of ablations in MABIM.
Avg. profits (K)
Method
Standard
echelons
echeelons
Lower
Lowest
Spaar w/o retention
719.0
1806.1
2388.6
547.7
204.1
eSpark w/o LLM
754.7
1538.9
1109.9
536.7
198.5
eSpark w/o checker
780.7
1741.6
2037.6
494
295.3
eSpark w w/o factors
758.2
1688,1
2375.1
498.6
368.9
eSpark
823.7
1811.4
2598.7
579.5
405.0
G eSpark's performance with different LLMs
We evaluated the performance of eSpark under different LLMs. Specifically, we selected the advanced GPT-4o, the relatively less capable GPT-3.5, and the state-of-the-art open-source model DeepSeck-V? V3 (671B). Experiments were conducted on two inventory management scenarios, and the results are summarized
Table 22 The results indicate that eSpark with GPT-4o achieves performance comparable that of GPT-4. When using GPT-3.5, performance degrades slightly. Although DeepSeek- V3 can effectively support eSpark, performance gap remains compared to GPT-4.
We further asscssed compact versions of statc-of-tho-art open-source LLMs, including DecpSeck-R1-70B and Qwen2.5-72B. Unfortunately these models performed poorly in task comprehension and failed to generate executable exploration functions. While these compact models show promise, their current capabilities do not yet match those of full-scale models required by cSpark. As shown Figure 5 as example, the exploration functions generated by Qwen2.5-72B exhibit issues such variable misuse and logical inconsistencies, highlighting their limited understanding of task requirements.
27

---

Published in Transactions on Machine Learning Research (05/2025)
Table 21: Detailed performance of ablations SUMO, includes the mean and standard deviation.
Method
Metric
Grid 4x4
Arterial 4x4
Grid 3x5
Cologne8
Ingobstadt121
Delay
48.75=0.49
851.56-37.99
211.07+22.00 01
25. 25.1840.51
246.03=14.88
eSpark w/o retention
Trip time
160.42:10.44
487 65.66
248.961115.0 04
80.23:10.49
363 20:228-144 14.83
Walt time
22.97-0.39
338 (4.6661660000 07.
97.77=10.25
7.1420.15
175.79-12.47
Delay
48.67:10.48
354.10=63.36
212.25:1:6..1
24.70:10..6
257.14-40.50
uSpark w/o LLM
Trip time
159.90110.56
101.49_67.0
238.33.19
88.57.10.60
376.46 7776446-40.0
Wait time
22.99=0.43
342.62 342.62=73.88
90.89+6.60
6.69=0.23
188.14=37.03
Delay
48.29
872.6194.88
215.58111..0 188111:29 11.29
25 25..22.0.69
258 14-449
eSpark w/o checker
Trip time
159.62:40.48
511.45=79.67
246 9.349.88
89.20:10.68
383.33=17.007
Wait time
22.59:00.48
347.43 68.39
95.95-1-6.33
6.92 0 28
192.96 15.37
Delay
48.5411044
849.82
116.27-1220.. 01
25. 20.20..0.1
19.69
eSpark w/o factors
Trip time
159.89++0.63
479.29 57.12
8.30
89.25++0.22
373.32-21.11
Wait time
22.85-1036
74.999
90.14416.95
6.8811019
150.90-13.42
Delay
48.36=0.32
864.22-68.22 21
109.49-11..0 98
25 1.2397:1.27
243 243.92-15.81
eSpark
Trip time
159.74.10.44
484.87 -8.25581 58.21
136.20166.88
80.50.1
367 57 15.03
Wait time 22.58.10299
328 82 61.70
88.38.14.41
6.94
180.09 13.84
Table 22: eSpark's performance with different LLMs in MABIM.
Avg. profits (K) Standard
Method
Lowest
GPT-4
823.7
405.0
GPT-4o
829.2
397.5
GPT-335
807.8
381.3
DeepSeck-V3
778.0
370.1
def exp lllrtttionfunctio (aanaaaatatttteeeeeeeeeee supply_chain):
Initialize all actions as alllwwd
mask coech (number_of_actiae
for sku in agent_staten inventory'];
current_inventee L [sku]
agent_states "inventory"' [sku]
in_ transit agent_ (ineessi____teie transtrnntttt...
Lead_time agent_stated lela_ttamee [sku]
# Calculate expected stock after lead time
future_stock current_inventoryy in_transit (weadteeeadaaaeea time]
action in range(number actioos):
replenishment action values [action] # Assuming actionvvalue are defined
Check capacity constraint if (future_ston stockssscckk-- + replenishment) > ceopllychain.ccpaccii
mask(action]
Check demand constraint
avg_demand agent_states states "hisstry'') [sku] [' ['mean']
if replenishment avg_demand: # Arbitrary threshold
mask[action]
Prevent negative inventory
if (current ieteentt_iveetrryyooo avg_demand 0:
mask(action 8
return mask, O
Figure Exploration function generated by Qwen2.5-72B. The generated exploration functions exhibit. incorrect function signatures and invalid parameter. The function bodies fail incorporate the required variables, and misuse both the data structures and the expected usage patterns of those variables.
28

---

Published in Transactions on Machine Learning Research (05/2025)
Table 23 presents the token consumption across different LLMs. Token usage is relatively consistent across tasks, enabling users to select an appropriate LLM based on their computational budget and performance requirements.
Table 23: Average token assumption of eSpark with different LLMs.
Environment
Token assumption (M)
GPT-4
3.0
GPT-4o
3.1
GPT-3.5
2.7
DeepSeek-VS V3
2.8
H eSpark's performance with different RL formulation
When the environment code is unavailable, we consider manually adding descriptions for transitions and rewards. We first introduce the ablation eSpark w/ lang, which replaces the environment codes with natural language descriptions. We conduct experiments results are shown in Table 24 and Table 25 While the SUMO tasks remain largely unaffected, the MABIM tasks experience noticeable performance declines. This is because the transition logic and variable usage in SUMO are relatively simple, whereas MABIM involves more complex environment transitions and variable interactions. While environment codes provide precise transition dynamics and variable meanings, natural language often lacks this level of detail.
selected scenarios in MABIM with 100 SKUs, and the
To further explore effective representation for improving performance in complex environments, we also adopt strategy similar to Text2reward Xie et al. (2023), where experts create simplified Python-style representation, referred to as eSpark w/ pyrep. As shown in Table 24 the kind of representation obtain performance on par with eSpark, which indicates that as long accurate details and sufficient information are included, the language model can effectively understand the environment and eSpark can work well.
Table 24: Performance of eSpark and its ablations in the 100 SKUs setting of the MABIM environment.
Profits (K)
Method
Standard
3 echelous
Lowest
eSpark
823.7
2598.7 7
405.0
eSpark w/ lang
777.5
752.7
347.0
eSpark w/ pyrep
817.4
2522.2
409.7
Table 25: Performance of eSpark and its ablations in SUMO, includes the mean and standard deviation.
Time usage (s)
Method
Metric
Arterial 4x4
Ingolstadt21
Delay
851.56137.00
246.05 14..88
eSpark
Trip time
484.84155..2
367.57115..0 03
Wait time
328.82++6..70
180.09+
Delay
830.05
243.32 19.43
eSpark w/ lang
Trip time
1954188118... 60
365. (G0.64421.44
Wait time
351.32.127.00 .07
178.31119.31
Policy performance analysis
To gain deeper understanding of the policy difference between eSpark and IPPO, we seloct the capacity limit and multiple echelons challenges within the 100 SKUs scenario as representative cases, presenting in Figure the daily profit. of eSpark and IPPO the test. dataset. challenged with capacity limit and multiple
29

---

Published in Transactions on Machine Learning Research (05/2025)
echelons. In the capacity limit challenges, high daily overflow ratio and low fulfillment ratio suggest that IPPO falls short in mastering the adjustment of restocking quantities for individual agents when capacity is limited, leading to overstocking and substantial overflow. Concurrently, this prevents SKUs required by consumers from being accommodated, culminating in an exceedingly low fulfillment ratio. In multiple echelon challenges, the fulfillment ratio al each echelon gradually decreases over lime, indicating that IPPO struggles to comprehend and learn the intricate interplay required for cooperation among various echelons, thereby inadequately fulfilling the demands of each echelon. Such shortcomings not only diminish potential profits but also subject the system to considerable backlog expenses. However, through action space pruning, evolutionary search, and reflection, eSpark manages to reduce the search within the vast space, selecting the most effective exploration functions and continuously improving. This approach significantly reduces overflow in the capacity limil scenario and successfully learns suitable cooperation methods for multiple echelons.
Lower capacity
2 echelons
Fullilmment
Overflow
Down stream fulfiliment eSpark
Upstream fulfilmmnnt IPPO
eSpark
IPPO
100
100
100
100
(atio(%)
80-
80
80
60-
60
60-
20
Lowest capacity
3 echelons
Fultilient
Liverloow
Down stream fulfillment
Midstream fulfiliment
Upstream fulfillment
100
Ratio%%)
40
40
40
20
20
20
20
40
60
0
20
40
60
20
40
60
20
40
Days
Days
Days
Days
Figure 6: The performance comparison between eSaark and IPPO 100 SKUs scenarios. capacity-lmmtted
scenarios, eSpark strives meet the demands while minimizing overflow costs, boasting a lower overflow ratio and higher fulfillment ratio. In the multiplo-cchelon challenge, eSpark achieves muanced collaboration across different echelons, ensuring high fulfillment ratios.
30

---

Published in Transactions on Machine Learning Research (05/2025)
J eSpark's response with and without reward factors
We select the policy feodback from one iteration in the Lower scenario of the MABIM environment with 100 SKUs as an example. The complete policy feedback is shown in Figure 7 In the ablation experiment, all reward factors (highlighted in red) are removed. Figure g and Figure illustrate the differences in GPT-4's reflections with and without the reward components. After removing the reward feedback, GPT-4 struggles to identify the reasons for poor performance and to thoroughly analyze ways to improve it.
We trained a RL policy using the provided exploration function code and tracked the values of the individual components of the reward function as well as global policy metrics. Me also compute the maximun, mean in the early training stage, mean in the late training stage, mean In all the training stage, mininum values for reward and its components after every iepoch_frest freg) epochs. Each elenent representing the value of that component on different warehouses:
one-dinensional array of length n_warehouse,
metric_nane, _amee: reward, Max: [435592. Min: (-1331893.
Mean in the early training stage: (-212101.5832), Mean late training stage: (338683.3766] Mean atl the training stage: (71512.5884375)
metric_name __naee profit, Nax: [587080. J, Min: [438897.1, Mean in the early training stage: [493648.]] 1, Mean in the late training
stage: [564756.2], Mean in all the training stage: [527295.125]
metric nane: excess cost, Max: [1648683.5),
(17213.5),
the earlly training stage: [596130.9), Nean in the
training stage: [123962. 4], Hean in all the training stage: [349431.25]
metric_nand __nnee order_ddooooooo coste__cst,,oee Nax: [68580.], Min: [56498.], 1, Mean the early training stage: [58864.], Mean in the late training
stage: [58758.],
all the training stage: [58650.625] 625]
metric_ metric_nane: name: holdingg_oosttteeen cost, Max: 1351, Min: 1272.157], Hean in the early training stage: 3390), Mean in the late
training stage: 6234], Mean in all the training stage: [394. 1803125]
metric nane: backlag__osst cost, Max: [56796.1], Min: [48408.7] 7], Mean the early training stage: [50364.04 04], Mean in
training stage: [43012.8], Mean in all the training stage: [47386.48125]
Please carefully analyze the policy feedback and provide new, improved exploration function that can better solve the task. Some helpful tips analyzing the policy feedback:
(1) You can start with "let's think step by step", , and then look at each reward individually and think about how can you improve
(2) If the total reward maintains in the sane level or even reduce, then you must rewrite the entire exploration function
neans is not (a) Changing the temperature scale or value the related mask component so nore action
the values for certain reward component are near identical throughout,
to optimize this component as it is written. You may consider:
explored
(b) Re-writing the related mask companent
(c) Discarding mask component add new component
Figure 7: Policy feedback from last iteration in MABIM 100 SKUs Lower scenario.
From the policy feedback, we can identity a few key issues:
The total reward is negative
early training stage and only becones positive
training stages.
This suggests that the agent is not learning effective strategies early on, potentially due to lack of exploration.
2. The excess cost is quite high, especially in the early training stages. This suggests that the agent is overstocking the warehouse.
3. The holding cost remains high throughout the training, indicating that the agent is holding onto inventory for Long.
Given these issues, we need to revise the exploration function to encourage the agent to explore actions that would result less overstocking. Here's an improved exploration function:
Figure 8: eSpark's reflection with reward factors.
the policy feedback, can identify key issues:
The policy feedback indicates that the reward function has wide range, with the minimum values being significantly negative and the maximum being positive. The nean reward in the early stage is negative, while it becomes positive the late stage. This suggests in outcomes.
the policy improves over time but has considerable variability
2. The high variance in reward components suggests that the exploration may be too aggressive, leading to overstocking (incurring high holding costs and overflow costs) or understocking (incurring backlog costs).
3. The exploration function may be too restrictive lenient, preventing the RL agent from effectively learning the optinal policy.
We will modify the mask to include a temperature parameter, which will initially be set high to allow for broader exploration and gradually decrease it the training progresses. This change will help the agent explore a wider range of actions early on and focus nore on promising actions as it learns.
Figure 9: eSpark's reflection without reward factors.
31

---

Published in Transactions on Machine Learning Research (05/2025)
K
Full prompts
We reference the prompt design outlined in the Fureka Ma et al. (2024) and adapt specifically for exploration function generation. Our prompt provides general guidance on the design of exploration functions, specific code formatting suggestions, leedback, and ioss for improvement. We present our prompls for MABIM below.
You are an expert in both inventory management learning and reinforcement learning. You will get some exploration functions, which help reinforcement learning tasks as effective as possible.
some actions
logically unlikely
selected, help exploration
Your goal is to evaluate whether the given exploration function natches the task description and whether contains illogical errors in the code content, and evaluate whether it's possible to avoid some unreasonable actions, help exploration reinfarcement learning. and the logic of the task, making sure to detect incorrect use variables code (2) All the parts that don't follow logic.
pay special attention meaning each
The exploration function signature be: (task_expa expporatiio_ssgnaturee
Your advice can be text or snippets of code, but should not be the full explorat ion function code. Most importantly, remember that your response must begin with either "Cade passes check. or "Code fails to pass check.". Under no circumstance can you begin your answer with other content.
Figure 10: System prompt for LLM
You are expert both inventory nanagement learning reinforcement learning. are trying urite sone exploration functions, which helps nask some actions that are logically unlikely to be selected, to help exploration in reinforcenent learning tasks effective possible.
Your goal is to write a explaration functian for the agent that will nask the actions that's alrost inpossible to be chesen in the task described in text.
The exploration function signature can be: htk.ke/pplrrtttoooiiinnuuueerrre
exploration function should only Please just give only the exploration function and don't put it class. Please nake sure that the code conpatible with nunpy (e.g., use numpy array instead of torch tensor).
variables
argunent list.
Figure 11: System prompt for LLMy.
Write exploration function for the following task (task_introduction!
The definition environment transition are (transition_dee initioon
The agent and state definition is
istate_definition?
The reward definition is
[reward_des inition). inttino
Figure 12: Initial prompt for LLM,-
32

---

Published in Transactions on Machine Learning Research (05/2025)
Please carefully the exploration function following task (task_introductinn
The definition of environment and transition are cerrasstion_definnite
The agent and state definition
(state_de "iaateddefniitio)
The reward definition is dreward_definttion
The requirements code
[code_outpus) wwtppuuutuppppppp tip)
Figure 13: Initial prompt for LLM
Here is the latest exploration function and it's description: (gpt_response)
Please carefully review this code, check whether it natches the task description ion and whether contains any illogical errors in the code content, and evaluate whether it's possible to improve the exploration and the perfornance.
Figure 14: LLM, 's feedback to LLM
this experts, are as follows:
not approved by experts,
experts on
{checker_ {checker_feedback) feedback)
Please refer to expert advice and conbine your own knowledge, fix the problens and provide new, inproved exploration function!
Figure 15: LLM,'s 's feedback to LLMg.
def compute_maae mask(agent baaklaggntsstttes: AgentStates, supply. cchiin Supp lyChain action_ space: list): Here are some code you can refer when you generate your exploration function.
return toaak_nask, {
Figure 16: Signature of exploration function.
Please carefully analyze the policy feedback and provide new, improved exploration function that better solve the task. Some helpful tips for analyzing the policy feedback:
(1) You
start with "let's think step by step", and then look each reward individually and think about how reduce, then you must rewrite the entire exploration
can you improve it
(2) If the total reward maintains in the level
function
(3) If the values
identical throughout, then this neans not able component so that action can (c) Discarding the mask component or add new mask component
certain reward component
to optimize this component as it is written. You may consider:
(a) Changing temperature scale
explored
(b) Re-writing the related component
Figure 17: Output and improvement tips for LLM,,
33

---

Published in Transactions on Machine Learning Research (05/2025)
The output the exploration function should total mask (1 denote action not masked otherwisc). code output should formatted python code string: "" "python
Some helpful tips for writing the exploration function code:
(1) Your total mask and component should 3-D numpy array in (warehouse nane, sku type, action mask]. The masks of the actions that are available are set to 1, otherwise 8. No other elements are allowed to appear in total nask.
(2) you choose transform component nask, then you nust also introduce temperature paraneter inside the frrmatoon function; this paraneter must be named variable in the mask function and it must not be an input variable. Each transformed nask component should have its tenperature variable
(3) Make sure the type each input variable is correctly specified; For example, float input variable should not specified
torch. Tensor
(4) Most importantly, the exploration code's can only use variables defined in its argunents. Under no circunstance you introduce input variables. You only need give the definition of exploration function and no other function or class should be defined.
Figure 18: Output format for LLMg-
Your output should contain two parts:
(1) Your response nust begin with either "Code passes check." or "Code fails to pass check.", "Code passes check." " you believe that there with the description of the task. "Code fails to pass check." indicates the provided exploration function contains logical errors, you think the exploration.
logical errors the code the variables taken in accordance
obviously flawed you can point
facilitate
effective
(2) you begin with "Code fails pass check.", you give your advice on fixing the problens; If you begin with "Code passes check.", you also have to state why each part is logical and reasonable.
explain why code fails pass check and
logical errors include:
(1) Misunderstanding the meaning of the state items, including syntax errors when using variables
(2) Illogically handling the state items
(3) Using multiple unrelated state items to calculate mask component
(4) The way combining mask components into total mask is unreasonable
Figure 19: Output format for LLMc-
We trained policy using the provided exploration function code and tracked the va values of the individual components of the reward function as well as global policy metrics. We also compute the maxinun, mean in the early training stage, mean in the late training stage, nean in all the training stage, minimum values for reward and its one-dimensional array length n_warehouse,
conponents after representing the value of that component on different warehouses:
(epoch_frea) epochs. Each elenent
<Reward Feedback Here
Figure 20: Reward feedback and action feedback.
L eSpark's exploration function editing
In this section, we demonstrate the reward oditing capabilities of eSpark. eSpark is capable reflecting upon feedback to optimize the exploration for subsequent iterations.
34

---

Published in Transactions on Machine Learning Research (05/2025)
trained policy using provided exploration function components of the reward function as well as global policy netrics. We also conpute the naxinun, nean in the early training stage,
tracked values of individual
all training stage, mininum values for reward and its components after every (epoch_freas epochs. Each elenent is a one-dinensional array length n_warehouse, representing the value of that component on different warehouses:
the training stage,
netric_name reward, in the late training stage: (338683.3766), Mean in all the training stage: 58843755
(435592.
(-1331893.
in early training stage: [-212101.5832),
netric_name: profit, Max: [587880.], Min: [438897.], Mean the early training stage: (493648.), Mean the late training stage: [564756.2], Mean in all the training stage: [527295.125]
vetric_name: excess_cost, Max: [1648683.5], Min: [17213.5], Mean in the early training stage: 1596130.91, Mean in training stage: [349431.25]
late training stage: [123902.4], Mean
netric_name: order cost, Max: 160580.1, Min: [56498. Mean in the early training stage: [58864.1, Mean in the late
training stage: [58758.], Mean in all the training stage: [58650.625]
netric_name: voldinggcostt Max: (356.
[272.157),
early training stage: (390.6432) Mean in
late training stage: [399.6234], Mean in all the training stage: [394.1866 18031251
netric_nane: nane: backlog_ Max: [56796.1], Min: [48408.7], Mean the early training stage: [50364.04], Mean in the late training stage: [43012.81, Mean in all the training stage: [47306.48125]
provide new, inproved exploration function that better solve the task, Some helptut tips for analyzing the policy Teedback:
Please carefully analyze policy feedback
(1) You start with "let's think step step", and then look each reward individually and think about how can you inprove it
(2) If the total reward maintains in the same level or even reduce, then you must rewrite the entire exploration function
(3) If the values for a certain reward component are near identical throughout, then this means RL is not able optimize this component as is written. You may consider:
(a) Changing the tenperature scale or value of the related mask component so that nore action can be
exp sord
(b) Re-writing related mask component
(c) Discarding the mask component or add a new mask component
Figure 21: Policy feedback from the last itcration.
From the policy feedback, we can identify a few key issues:
1. The total reward is negative in the early training stage and only becones positive in the later training stages.
This suggests that the agent is not learning effective strategies early on, patentially due to lack of explorat
cost quite high, especially in the early training stages. This suggests that agent is overstocking the warehouse.
The
3. The holding cost remains high throughout the training, indicating that the agent is holding onto inventory for too long.
Given these issues, need revise the exploration function encourage the agent explore actions that would result in less overstocking. Here'' an improved exploration function:
Figure 22: eSpark reflects upon the policy feedback and proposes the modification.
def compute_nan wasklagent__steee _ttettes:: AgentStates,, aupply_chain: SupelyChein, actior_space: Uist) adanrerrey:
varehouse, list supply, supply_chainn.. httppyycchs..ge..areeeeee htt.rggettaehhoe_eeeeseeee List() mun len[agent_ [ss_tttttss total_rask on-ones [ ('en warehouse listl, nun_sku, tenfection_speet
for warehouse nare in enunerate (warehouse_ list): capecity supply_chain perrehossenaaee "capecity"] Get volume all sku sku_voluue agent_states [warehouse_nang, "volune", "today", "all_skus"]
ir_stack all
in_stock agent_atates [xorehouse_ nare, "in_stock", "today", "all_skus"]
Get rean demand for all skus past lookback period
history_denend, denennd_reen np. (agraggegggntsstteee (varehouse_naree "Jerand", "lookback", "aUl_kkus],, axis=1)
enuverote(actiae [ttpccettoo_sspceee _pace)) Calcc Late the patentisl rep lenishment for all SKUs replenishment history deranddadddddeee mean action
If the reptenisheent will nake the total volure atl SKUs exceeds the capacity of the warehouse, wask this action if np.sun(sku_ sunnskku_volure x (in_stock replenishment)) capacity: tota l_rask[1,
an additionnal condition to rask actions bring replenishment
if action 2:
ir action 1.5: Laakk
exploration training stages
if agent_statee states, current step statesstatesecm
stt.ssstttedddationnn. durations 6.3 and action 8.5:
total_ [1, i,
return totalrrsk, resk, O
Figure 23: Comparison of exploration functions before and after editing.
35


Reviews:
[{"review_id": "prEJXjahtJ", "reviewer": "Reviewer_B3N4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n1.  Introducing LLMs to MARL for action space pruning. \n2. Extensive experiments and good ablation studies.\n\n**Weakness:**\n1. The effectiveness of the eSpark framework relies heavily on the quality of LLM outputs. Errors in exploration function generation or feedback handling could degrade performance. Could the authors provide justification for this reliance?\n\n2. The complexity of the proposed algorithm is not explicitly discussed. A detailed analysis would help assess its feasibility in practical applications.\n\n3. The paper does not compare eSpark with other LLM-based approaches, making it difficult to evaluate its relative advantages. A comparative analysis would strengthen the contribution.\n\n4. Are there alternative choices for prompts? Exploring different prompt designs could provide insights into the framework's robustness.\n\n5. While tested on industrial tasks, broader applicability (e.g., robotics) needs exploration.\n\n6. The presentation is generally good and easy to follow, but there are some places that can be improved. Just to name a few examples: is that correct to denote policy \\pi as \\pi: O\\times A\\rightarrow [0,1]? What is the purpose of defining exploration function E? What is the significance of proposition 1?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel framework, ​eSpark, to automate action space pruning using Large Language Models, aiming to enhance exploration efficiency and policy performance in MARL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the above weakness part.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CB6HCOJrT1", "reviewer": "Reviewer_ZPNz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. Innovative Use of LLMs for Action Space Pruning.\n2. Improved Learning Efficiency and Performance.\n3. The paper demonstrates how LLMs can serve as external knowledge sources.\n4. Good experimental results.\n5. Paper well written and easy to follow. I enjoyed reading it.\n\n\n**Weaknesses:**\n1. Although experimental results are great, they are demonstrated on limited number (two) of tasks.\n2. Running multiple LLM queries per iteration may introduce high computational cost and response delays, which are not deeply analyzed in the paper.\n3. eSpark relies on a proprietary LLM (GPT-4) to generate exploration functions, which may limit reproducibility.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces eSpark, a novel framework that integrates LLMs with Multi-Agent Reinforcement Learning (MARL), using prior knowledge of the LLMs to enhance exploration efficiency via action space pruning. The key contributions of the paper are:\n1. Proposing a novel use of LLM to generate code with the aim of pruning redundant action space in MARK environments.\n2. The eSpark framework iteratively refines exploration functions using a feedback loop of Self-Improving Exploration via Evolutionary Search.\n3. Experimental Evaluation of the method in Multi-Agent Domains of Inventory management and traffic signal control.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Questions**\n1. While Proposition 1 guarantees that pruning does not eliminate optimal solutions, could excessive pruning still limit exploration if the LLM incorrectly removes useful actions?\n2. Did the authors perform comparative analysis with open source LLMs such as Llama or Mistral? I believe it would've highlighted both limitations and feasibility of such widely available models for use with this technique.\n3. What happens when the LLM generates flawed or inconsistent rules?\n\n**Recommendations**\n- I beileve the paper would benefit from an analysis/experiments of how the LLM usage affects the computational cost when generating useful outcome with this technique. This will allow the readers to gauge the efficacy of this method for their use cases.\n\nThe following recent paper on multi-task RL and pruning might be worth citing due to its relevance:\n1. Arnob et al. \"Efficient Reinforcement Learning by Discovering Neural Pathways\", NeurIPS 2024.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rYF0tRI7eF", "reviewer": "Reviewer_GmtW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The overall idea is clear and well presented in the paper. The authors conducted comprehensive experiments to verify the proposed method, the results look promising.\nFor potential improvements, please refer to the requested changes section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the use of LLMs to conduct action space pruning, which is implemented through an exploration function generation. In the designed algorithm, LLMs propose, reflect, and improve various exploration functions, such that the learning efficiency and performance of MARL can be improved. Experimental results are comprehensive and supportive to the paper's claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The current organization of the related work section seems a bit unnatural to me.\nI would recommend the authors to have a revision such that the LLM for code generation and the LLM for RL/MARL to be better separated. For example, Eureka's paper should be discussed in the section of LLMs for RL / reward design rather than for code generation, in my opinion.\n\nThe following work on action space pruning may be worth a cite: \n\n[1] Sun, Hao, and Taiyi Wang. \"Toward causal-aware RL: State-wise action-refined temporal difference.\" arXiv preprint arXiv:2201.00354 (2022).\n\nDiscussion on state-action abstraction may further enhance the comprehensiveness of the related work discussion section of this paper --- action abstraction may be considered to be an implicit approach of action pruning.\ne.g., \n\n[2] Huang, Biwei, et al. \"Action-sufficient state representation learning for control with structural constraints.\" International Conference on Machine Learning. PMLR, 2022.\n\n\nRegarding the method and formalism part, I would suggest using subscripts for the k-th agent all the time. Moreover, a table summarizing definitions of different variables could be helpful for the readers to follow the notation. And this can alleviate the missing definition problem sometimes, e.g., the definition of G in algorithm 1 is missing, and following the conventional definition of G it's reasonable to interpret G as the cumulative return.\n\n\n\nOther Questions:\n\n- How challenging would it be in scaling up the proposed method? \n- How hard would it be to run the method using local/open-sourced LLMs? \n- How sensitive are the results to the LLMs' abilities (this may include both reflection and proposing exploration functions)", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TD8qpL1OtP", "reviewer": "Reviewer_oyHM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n• Addresses action space explosion: The paper tackles the problem of combinatorial explosion of the action space in MARL, which leads to algorithmic instability and difficulty in convergence.\n• eSpark, in a novel zero-shot approach, utilizes LLMs to generate exploration functions from task descriptions and environmental rules. This innovative method streamlines Multi-Agent Reinforcement Learning (MARL) by pruning unnecessary actions.\n• Comprehensive methodology: The eSpark framework includes three components: zero-shot generation of exploration functions, evolutionary search for the best-performing MARL policy, and detailed feedback on the policy's performance to improve the exploration functions.\n• Demonstrated effectiveness: eSpark consistently outperforms the combined MARL algorithm in inventory management and traffic light control tasks, achieving an average performance gain of 34.4% and 9.9% respectively. It also shows a 29.7% improvement in scalability challenges with over 500 agents.\n• LLM checker and detailed reward feedback: The LLM checker reviews the LLM code generator's output to verify the accuracy of the code. This collaboration enhances the quality and efficacy of the generated content.\n\nWeakness:\n• Homogeneous agents: eSpark is currently only applicable to tasks involving homogeneous agents.\n• Informative policy feedback: eSpark benefits from policy feedback to refine the exploration functions. When feedback is not informative regarding how to modify the exploration, eSpark may struggle to improve and need extra expert input for effective reflection.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "eSpark, a new framework, uses Large Language Models (LLMs) to create exploration functions that enhance Multi-Agent Reinforcement Learning (MARL) algorithms. This integration allows the framework to leverage prior knowledge and improve iteratively through trial and error. The LLM-generated exploration functions are specifically designed for action space pruning, which guides the exploration and learning process of MARL algorithms. Notably, eSpark bypasses the need for complex prompt engineering and seamlessly integrates with existing MARL algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weakness", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers found the use of LLMs for action space pruning in MARL, its methodology, and demonstrated effectiveness as novel advancement to MARL. They confirmed the claims were supported and the paper relevant to the TMLR audience. Initial concerns included the reliance on LLM output quality and handling errors, the lack of detailed computational cost analysis of LLM usage, the need for comparison with other LLMs, exploring prompt sensitivity, applicability to heterogeneous agents and other domains like robotics, and the need for informative policy feedback, especially in sparse reward settings.

The revision addressed these points by explaining error handling mechanisms, providing detailed computational cost analysis and experiments with different LLMs (including open-source), clarifying notation, and discussing limitations and the importance of detailed feedback, 

Reviewers "B3N4" and "GmtW" explicitly stated that their initial concerns were addressed by the authors' detailed responses and revisions, appreciating the added analysis and experiments with open-source LLMs and computational cost. All reviewers ultimately recommended acceptance for publication in TMLR. For example: oyHM: "The paper's innovative framework for exploration in MARL, using LLMs to generate exploration functions, is a significant contribution. Its scalability, comprehensive methodology and proven effectiveness make it a valuable tool for MARL research."
Decision: Accept (poster)