YOLO-MARL: You Only LLM Once for Multi-agent Reinforcement Learning

## Abstract

Abstract Advancements in deep multi-agent reinforcement learning (MARL) have positioned it as a promising approach for decision-making in cooperative games. However, it still remains challenging for MARL agents to learn cooperative strategies for some game environments. Recently, large language models (LLMs) have demonstrated emergent reasoning capabilities, making them promising candidates for enhancing coordination among the agents. However, due to the model size of LLMs, it can be expensive to frequently infer LLMs for actions that agents can take. In this work, we propose You Only LLM Once for MARL (YOLO-MARL), a novel framework that leverages the high-level task planning capabilities of LLMs to improve the policy learning process of multi-agents in cooperative games. Notably, for each game environment, YOLO-MARL only requires one time interaction with LLMs in the proposed strategy generation, state interpretation and planning function generation modules, before the MARL policy training process. This avoids the ongoing costs and computational time associated with frequent LLMs API calls during training. Moreover, the trained decentralized normal-sized neural network-based policies operate independently of the LLM. We evaluate our method across three different environments and demonstrate that YOLO-MARL outperforms traditional MARL algorithms.

## 1 Introduction

Multi-agent reinforcement learning (MARL) algorithms have proven to be a powerful framework for addressing complex decision-making problems in multi-agent systems. With the rising applications of multi-agent systems, such as mobile robots in warehouses and games requiring complex reasoning and strategy, it is increasingly crucial for individual agents to learn, cooperate, or compete in dynamic environments without a centralized decision-maker (Papoudakis & Schäfer, ). In cooperative Markov games, agents are trained to coordinate their actions to maximize the joint rewards. However, existing MARL algorithms face challenges in learning distributed policies for cooperative games. Moreover, they struggle with tasks characterized by sparse rewards, dynamic environment, and large action spaces, which can hinder efficient learning and agent collaboration.

LLMs have excelled as high-level semantic planners due to its in-context learning abilities and prior knowledge (Ahn et al., ). Zhang et al. () and Kannan et al. () directly use LLMs as embodied agents, which demonstrate LLMs’ planning ability in multi-robot system. There are also works concentrating on utilizing the LLMs to guide the reinforcement learning (RL) training to reach better performances. ELLM (Du et al., ) leverage LLMs to suggest a goal to assist RL training whereas Kwon et al. () focusing on the alignment between the action provided by LLM and the RL policy. While these approaches show exciting potential for integrating LLM within policy training, they have yet to extend their methods on multi-agent scenarios. More importantly, utilizing LLMs as agents or integrating them into the RL training loop presents certain challenges. Repeated interactions with LLMs in long-episode tasks or complex environments—especially when using advanced LLMs like Claude-3.5 or GPT-o1 can be time-consuming and costly; it becomes intractable for tasks requiring training over tens of millions of steps. Additionally, there is a risk of intermittent disconnections with the LLM, which could disrupt the training process and affect the system’s stability.

Built on the identified insights and challenges, we introduce YOLO-MARL, as shown in Fig. 1, an innovative approach that leverages the planning capabilities of LLMs to enhance MARL policy training. In particular, the major strength of our framework is that it requires only a one-time interaction with the LLM for each game environment. After the strategy generation, state interpretation and planning function generation modules, there is no need for further LLMs interaction during the MARL training process, which significantly reduces the communication and computational overhead of LLM inferences. Moreover, YOLO-MARL demonstrates its strong generalization capabilities and simplicity for application: with the proposed strategy generation and state interpretation modules, our approach is compatible with various MARL algorithms such as Yu et al. (), Rashid et al. (), Lowe et al. (), and requires only basic background understanding of a new game environment from the users. We also evaluate our framework in a sparser reward multi-agent environment: Level-Based Foraging environment (Papoudakis & Schäfer, ), and a highly strategic task environment: the StarCraft Multi-Agent Challenge environment (Samvelyan et al., ), together with the MPE environment (Lowe et al., ), and show that YOLO-MARL outperforms several MARL baselines. We also provide several ablation study results to demonstrate the function of each module in the proposed framework. To the best of our knowledge, YOLO-MARL is among one of the first trials that incorporates the high-level reasoning and planning abilities of LLMs with MARL, since very limited literature of LLM for MARL has been introduced so far (Sun et al., ).

In summary, our proposed method YOLO-MARL has the following advantages:

- •
This framework synergizes the planning capabilities of LLMs with MARL to enhance the policy learning performance in challenging cooperative game environments. In particular, our approach exploits the LLM’s wide-ranging reasoning ability to generate high-level assignment planning functions to facilitate agents in coordination.
- •
YOLO-MARL requires minimal LLMs involvement, which significantly reduces computational overhead and mitigates communication connection instability concerns when invoking LLMs during the training process.
- •
Our approach leverages zero-shot prompting and can be easily adapted to various game environments, with only basic prior knowledge required from users.

An overview of YOLO-MARL is presented in Figure 1. All prompts, environments, and generated planning functions can be found in Appendix.

Figure: Figure 1: Depiction of our framework YOLO-MARL. (a). Strategy Generation: We pass basic environment and task description into the LLM to get generated strategies for this specific environment. (b). State Interpretation: We process the global states so that the format of global states will be more structured and organized for better comprehension by the LLM. (c). Planning Function Generation: We chain together the environment and task description, LLM generated strategies and state interpretation function. These prompts are then fed into the LLM to generate a planning function for this environment. (d). MARL Training: The state interpretation function and the generated planning function are integrated into the MARL training process. The LLM is no longer required for further interaction after the Planning Function Generation.
Refer to caption: extracted/5902951/framework.png

## 2 Related Work

### 2.1 Multi-Agent Reinforcement Learning

MARL has gained increasing attention due to its potential in solving complex, decentralized problems. Centralized training with decentralized execution has become a popular framework for overcoming the limitations of independent learning. Methods like QMIX (Rashid et al., ) and MADDPG (Lowe et al., ) use centralized critics or value functions during training to coordinate agents, while allowing them to execute independently during testing. In cooperative environments, algorithms like COMA (Foerster et al., ) and VDN (Sunehag et al., ) enable agents to share rewards and act in a coordinated fashion to maximize joint rewards.Wang et al. () introduce a new approach using language constraint prediction to tackle the challenge of safe MARL in the context of natural language. However, the existing MARL algorithms may not perform well in sparse reward environments and still struggle in learning fully cooperative policy in some environments. So far, only very limited literature of using LLM for MARL has been proposed (Sun et al., ), and it remains unclear whether and how can LLM be leveraged for MARL-based decision-making.

### 2.2 Large Language Models for Single-Agent RL and Decision-Making

Many existing works utilize LLMs as parts of RL training process. Du et al. () enhance agents’ exploration by computing the similarity between suggested goals from LLMs and agents’ demonstrated behaviors. Carta et al. () leveraging language-based goals from LLMs by generating actions conditioned on prompts during online RL. Kwon et al. () provides scalar rewards based on suggestions from LLMs to guide RL training. However, most of these approaches haven’t explored their works in the context of Markov games and require extensive interactions with LLMs during training.

Gupta et al. () utilize CLIP’s visual embedding to an agent exploring of environment. Fan et al. () studies a multi-task RL problem, where an agent is tasked with completing MINEDOJO tasks. Ahn et al. () proposes SayCan which grounds LLMs via value functions of pretrained skills to execute abstract commands on robots. Liang et al. () finds that code-writing LLMs can be re-purposed to write robot policy code. Huang et al. () shows that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan. Other research such as Ma et al. () and Xie et al. () use LLMs prior knowledge and code generation capability to generate reward functions, whereas we utilize code generation for planning functions. Lin et al. () highlights the limitations of LLMs in handling complex low-level tasks. On the other hand, we harness the high-level reasoning capabilities of LLMs to enhance low-level action performance within RL model training.

### 2.3 Large Language Models for Multi-Agent Systems

LLM-based Multi-Agent (LLM-MA) systems focus on diverse agent profiles, interactions, and collective decision-making. While this allows agents to collaborate on dynamic, complex tasks, it also increases computational overhead due to the communication between LLMs (Guo et al., ), (Sun et al., ). Camel Li et al. () and MetaGPT Hong et al. () employ multiple LLM agents to accomplish tasks like brainstorming and software development. Nascimento et al. () enhance communication and agent autonomy by integrating GPT-based technologies. In multi-robot contexts, Chen et al. () compare task success rates and token efficiencies of four multi-agent communication frameworks. SMART-LLM (Kannan et al., ) decompose multi-robot task plans into subgoals for LLM to enable efficient execution, while Co-NavGPT (Yu et al., ) integrates LLMs as global planners for cooperative navigation. Focusing on multi-agent pathfinding (MAPF), Chen et al. () studies the performance of solving MAPF with LLMs. Agashe et al. () proposes the LLM-Coordination (LLM-Co) to enable LLMs to play coordination games. Li et al. () explore the use of LLMs in cooperative games within a text-based environment, and Ma et al. () explores LLMs in the StarCraft II environment. In contrast, our method leverages the planning abilities of the LLM to train better MARL policies instead of using them directly as agents.

## 3 Problem Formulation

Markov game (MG) is defined as a multi-agent decision-making problem when the interaction between multiple agents affect the state dynamics of the entire system and the reward of each agent under certain conditions (Littman, ). In this work, we consider a Markov game, or a stochastic game Owen () defined as a tuple $G:=(\mathcal{N},S,A,\{r^{i}\}_{i\in\mathcal{N}},p,\gamma)$, where $\mathcal{N}$ is a set of $N$ agents, $S=S^{1}\times\cdots\times S^{N}$ is the joint state space, $A=A^{1}\times\cdots\times A^{N}$ is the joint action space, with $(S^{i},A^{i})$ as the state space and action space of agent $i$, respectively, $\gamma\in[0,1)$ is the discounting factor (Littman, ; Owen, ). The state transition $p:S\times A\rightarrow\Delta(S)$ is controlled by the current state and joint action, where $\Delta(S)$ represents the set of all probability distributions over the joint state space $S$. Each agent has a reward function, $r^{i}:S\times A\rightarrow\mathbb{R}$. At time $t$, agent $i$ chooses its action $a^{i}_{t}$ according to a policy $\pi^{i}:S\rightarrow\Delta(A^{i})$.
For each agent $i$, it attempts to maximize its expected sum of discounted rewards, i.e. its objective function $J^{i}(s,\pi)=\mathbb{E}\left[\sum_{t=1}^{\infty}\gamma^{t-1}r_{t}^{i}(s_{t},a_
{t})|s_{1}=s,a_{t}\sim\pi(\cdot|{s}_{t})\right]$. In the literature, deep MARL algorithms (Lowe et al., ; Yu et al., ; Rashid et al., ) have been designed to train neural network-based policies $\pi_{i}(\theta_{i})$. For a cooperative game, one shared reward function for all the agents is widely used during the training process, which is also considered in this work.

## 4 Methodology

In this section, we introduce our method, YOLO-MARL, which leverages LLMs to enhance MARL. Specifically, during training, we utilize the high-level task planning capabilities of LLMs to guide the MARL process. Our approach consists of four key components: Strategy Generation, State Interpretation, Planning Function Generation, and MARL training process with the LLM generated Planning Function incorporated throughout.

Figure: Algorithm 1 YOLO-MARL Training Process

### 4.1 Strategy Generation

To create a generalizable framework applicable to various environments—especially when users may have limited prior knowledge—we incorporate a Strategy Generation Module into our methodology. This module enables the LLM to autonomously generate strategies for different environments without requiring extensive human input or expertise.

As shown inside the blue box of Figure 1(a), the LLM is provided with the basic information about the environment, including task descriptions, relevant rules, and constraints of how to interact with the environment. Additionally, we supply a general guideline within the prompt to assist the LLM in generating effective strategies. Gathering all the information, the LLM will output detailed strategies to accomplish the tasks or achieve the goal, following the specified format.

By aggregating all this information, the LLM outputs detailed strategies to accomplish the tasks or achieve the goals, following a specified format. The Strategy Generation is crucial for several reasons:

- •
Reducing User Burden: It alleviates the need for users to comprehensively understand new environments, saving time and effort.
- •
Enhancing Generalization: It enables the framework to adapt to different environments with minimal prompt modifications.
- •
Facilitating Planning Function Generation: The strategies serve as vital components in the prompts used for the Planning Function Generation Module. The results of using YOLO-MARL but without Strategy Generation Module are shown in ablation study  6.1 .

The LLM-generated strategies are incorporated into the prompt alongside other necessary information to facilitate the subsequent planning function generation. Further details about the strategy prompts and their formats can be found in Appendix C.1.

### 4.2 State Interpretation

In many simulation environments, observations or states are typically provided as vectors, with each component constructed using various encoding methods. While the vector form of observation is easy to handle when training deep reinforcement learning models, it is difficult for LLMs to directly parse their semantic meaning due to the lack of explicit context for each component.

We propose the State Interpretation Module to assist the LLM in interpreting the environment state. By providing a semantically meaningful representation of the state, the LLM can successfully generate executable planning functions for training. Formally, given the current environment state in vector form $S_{v}$, we define an interpretation function $F_{S}$ such that $F_{S}(S_{v})\to S_{I}$, where $S_{I}$ provides more explicit and meaningful information about each state component. Specifically, we implement a Python function that processes the vector observation and restructures each component. For example, it extracts position information from the raw observation and organizes it into a variable name ”position”. This transformation enables the LLM to understand the context and significance of each part of the state.

Recent works like Ma et al. () and Xie et al. () have demonstrated the success of enhancing LLMs performance by providing relevant environment code. In the same manner, we include the interpretation function $F_{S}$ in the prompting pipeline, formatted as Pythonic environment code as shown in the purple box in Figure 1(b). The State Interpretation Module significantly reduces the risk of the LLM generating erroneous functions with outputs incompatible with the training procedures. An ablation study on the effectiveness of this module can be found in Sec 6.2, while more details about the interpretation function are provided in Appendix C.2.

### 4.3 Planning Function Generation

A crucial component of our method is leveraging the LLM to perform high-level planning instead of handling low-level actions. We combine all the prompts from the previous modules and input them into the LLM. The LLM then generates a reasonable and executable planning function that can be directly utilized in the subsequent training process.

To be more concise, given any processed state $S_{I}$, we define an assignment planning function as $\mathcal{F_{\mathcal{T}}}(S_{I})\to\mathcal{T}_{i}\in\mathcal{T}$, where $\mathcal{T}=\{\mathcal{T}_{1},...,\mathcal{T}_{n}\}$ is a set of target assignments that each agent can take. We define the assignment set $\mathcal{T}$ over the action space such that an action can belong to multiple assignments and vice versa. For example, if the assignment space is defined as $\mathcal{T}=\{Landmark_0,Landmark_1\}$, and landmark 0 and landmark 1 are located at the top right and top left positions relative to the agent respectively, then taking the action ”UP” can be associated with both assignments. Conversely, we can have multiple actions correspond to an assignment. For instance, moving towards ”Landmark 0” may involve actions like ”UP” and ”RIGHT”. This is referred to the red module in Fig. 1(c) and more information of generated function refer to Appendix D.

### 4.4 MARL training with Planning function incorporation

To incorporate the planning function into MARL training, we add an extra reward term to the original reward provided by environments. Specifically, we define the final reward $R$ used by the critic as: $R=r+\sum_{i}\Delta r_{i}$, where $r$ is the original reward from the environment and $\Delta r_{i}$ is an additional reward signal for each agent $i$. The term $\Delta r_{i}$ is based on whether the action taken by the agent’s actor aligns with the assigned task from the planning function. If the agent’s action belongs to the assigned task, it receives a reward $r^{\prime}$ as its $\Delta r_{i}$ and will receives a penalty $p^{\prime}$ as its $\Delta r_{i}$ if not.

Notably, we do not need to interact with the LLM during the entire training process, nor do we need to call the planning function after the policy has been trained. The training process $MARL_{alg}(R)$ takes $R$ as the reward function, uses the same state and action space definitions and follows the standard MARL algorithms and evaluation metrics within the literature, such as Yu et al. (), Rashid et al. (), and Lowe et al. (). This makes our method highly efficient compared to approaches that interact with LLMs throughout the training process or use LLMs as agents as shown in the greed box in Fig.1(d). In practice, using the LLM’s API to generate the planning function incurs minimal cost—less than a dollar per environment—even when using the most advanced LLM APIs.

## 5 Experiments

In this section, we evaluate our method across three different environments: MPE, LBF, and SMAC. We use claude-3-5-sonnet-20240620 for the experiments.(^*^**We mainly use the Claude 3.5 Sonnet model for the LLM in our work: [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet))

### 5.1 Setup

Baselines. In our experiments, we compare the MARL algorithm MADDPG (Lowe et al., ), MAPPO (Yu et al., ) and QMIX (Rashid et al., ) and set default hyper-parameters according to the well-tuned performance of human-written reward, and fix that in all experiments on this task to do MARL training. Experiment hyper parameters are listed in Appendix.

Metrics. To assess the performance of our method, we use win rate as the evaluation metric on the SMAC environment, and the mean return in evaluation for all other environments. During evaluation, we rely solely on the default return values provided by the environments for both the baseline and our method, ensuring a fair comparison.

### 5.2 Results

**Table 1: Comparison between YOLO-MARL and MARL in the LBF environment across three seeds. The highest evaluation return means during training are highlighted in bold. The corresponding results can be found in Figure 2. The M means one million training steps. We run all the experiments on the same machine.**
|  | Mean Return after 0.2M / 0.4M / 1.5M / 2M Steps |  |  |
| --- | --- | --- | --- |
|  | QMIX | MADDPG | MAPPO |
| MARL | 0.00/ 0.01/ 0.25/ 0.38 | 0.09/ 0.33/ 0.26/ 0.32 | 0.31/ 0.72/ 0.99/ 0.99 |
| YOLO-MARL | 0.01/ 0.02/ 0.60/ 0.78 | 0.13/ 0.38/ 0.39/ 0.44 | 0.93/ 0.98/ 0.99/ 0.99 |

Level-Based Foraging. Level-Based Foraging (LBF) (Papoudakis & Schäfer, ) is a challenging sparse reward environment designed for MARL training. In this environment, agents must learn to navigate a path and successfully collect food, with rewards only being given upon task completion. To evaluate our framework in a cooperative setting, we selected the 2-player, 2-food fully cooperative scenario. In this setting, all agents must work together and coordinate their actions to collect the food simultaneously. The environment offers an action space consisting of [NONE, NORTH, SOUTH, WEST, EAST, LOAD], and we define the task set as [NONE, Food i, …, LOAD]. Using the relative positions of agents and food items, we map assigned tasks to the corresponding actions in the action space and calculate the reward based on this alignment. We evaluated our framework over 3 different seeds, with the results shown in Figure 2 and Table 1. LLM assist the MARL algorithm by providing reward signals, our framework significantly outperformed the baseline, achieving a maximum improvement of 105 % in mean return and a 2x faster convergence rate among all tested MARL algorithms. According to the results, our framework is effective across all the baseline algorithms, with particularly large improvements observed in QMIX and MADDPG, and a faster convergence rate for MAPPO. To assess the variability in the quality of our generated functions, we present the results of three different generated functions in Figure 8 and Table 3 in Appendix B.1. The results demonstrate that our framework consistently generates high-quality functions, with each achieving similar improvements across all baseline algorithms.

Figure: (a) MADDPG
Refer to caption: extracted/5902951/figure/LBF_MADDPG_3seeds.png

Multi-Agent Particle Environment. We evaluate our framework in Multi-Agent Particle Environment (MPE) (Lowe et al., ) simple spread environment which is a fully cooperative game. This environment has N agents, N landmarks. At a high level, agents must learn to cover all the landmarks while avoiding collisions. It’s action space is consist of [no_action, move_left, move_right, move_down, move_up]. We define the assignment for each agent to take to be [Landmark_i,…,No action]. During training, based on the global observation, we obtain the relative position of each agent with respect to the landmarks. Similar to LBF, we map each assignment of agent back to the corresponding action space and then reward the action of policy in action space level. We evaluate our approach on 3-agent and 4-agent scenarios using QMIX and MADDPG as baselines. As shown in Figure 3, our framework(colored line) outperform the baseline(black line) algorithm in mean returns by 7.66% and 8.8% for 3-agent scenario, and 2.4% and 18.09% for 4-agent scenario with QMIX and MADDPG respectively. These improvements demonstrate the effectiveness of our framework in enhancing coordination among agents to cover up all the landmarks.

Figure: (a) MADDPG
Refer to caption: extracted/5902951/figure/MPE_3_MADDPG.png

Figure: (a) MADDPG
Refer to caption: extracted/5902951/figure/MPE_4_MADDPG.png

StarCraft Multi-Agent Challenge environment. The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., ) simulates battle scenarios where a team of controlled agents must destroy an enemy team using fixed policies within a limited number of steps. We tested our method on three different maps: 3M, 2s vs 1sc, and 2c vs 64zg. The action space in the environment consists of [none, stop, move north, move south, move west, move east, attack enemy 1,…attack enemy n], where n is the total number of enemies on the map. This action space becomes increasingly complex depending on the number of enemies the agent has to engage, particularly in the 2c vs 64zg map, which contains 64 enemies and offers 70 possible actions.

In our experiments, we define the assignment space simply as [Move, Attack, Stop, None (for dead agents)]. We tested the performance of MAPPO, and the results for SMAC are shown in Figure 5. As indicated by the figure, even though we provide simple assignments that may be far from optimal instructions, our framework still achieves comparable results on certain maps. This demonstrates that our framework remains competitive, even in environments requiring strategic movements. We also explore the sparse reward case for this environment where the win rate of baseline algorithms is always closed to 0 while we generate a planning reward function pairs that outperform baseline. We suggest this pair generation as a potential future work and leave this discussion to the Sec 7.

Figure: (a) 3m
Refer to caption: extracted/5902951/figure/3m_MAPPO_3seed.png

## 6 Ablation Study

In this section, we conduct the ablation studies mainly in LBF 2 players 2 food fully cooperative environment since rewards in LBF are sparser compared to MPE and SMAC (Papoudakis & Schäfer, ). We refer to 1 for more information about the environment. Due to page limitation, we also leave some discussions and figures in Appendix B.

### 6.1 Comparison between YOLO-MARL with and without Strategy Generation

In this section, we examine the impact of the Strategy Generation Module on the performance of the YOLO-MARL framework. Specifically, we compare the standard YOLO-MARL with a variant that excludes the Strategy Generation Module to assess its significance.

According to our tests, the Strategy Generation Module plays an important role in the YOLO-MARL method. As shown in Figure 6
, without the LLM generated strategy, we obtain a worse-performing planning function. Interestingly, the mean returns of evaluations for the functions without the LLM generated strategy are not always close to zero, indicating that the generated planning functions are not entirely incorrect. Based on this, we could confirm that the Strategy Generation Module would help Planning Function Generation Module provides better solutions to this game. Moreover, giving the strategy also helps stabilize the quality of the generated code. We observe a higher risk of obtaining erroneous functions without supplying the strategy.

Figure: (a) MADDPG
Refer to caption: extracted/5902951/figure/LBF_MADDPG_without_LLMstrategy.png

### 6.2 Comparison between YOLO-MARL with and without State Interpretation

To demonstrate how the State Interpretation Module enhances our framework, we present two failure case snippets:

- •
Without the Interpretation Function: The interpretation function is omitted entirely from the prompting pipeline.
- •
Providing Raw Environment Code Directly: Raw environment source code is fed directly to the LLM.

As shown in Figure 10, the LLM is unable to infer the type of state and attempts to fetch environment information via a non-existent key if no preprocessing code provided. And if environment code is provided without dimensional context for each component, the LLM is likely to make random guesses. In both scenarios, the absence of explicit state interpretation hinders the LLM’s ability to generate accurate and executable planning functions. These failures underscore the importance of the State Interpretation Module in bridging the gap between vectorized observations and the LLM’s requirement for semantically meaningful input.

By incorporating the State Interpretation Module, we enable the LLM to understand the environment’s state representation effectively. This results in the generation of reliable planning functions that significantly enhance the performance of our YOLO-MARL framework.

### 6.3 Comparison between YOLO-MARL and reward generation

In this section, we compare our YOLO-MARL method with approaches that utilize the LLM for reward generation without reward function template. We explore two scenarios: reward generation without feedback and reward generation with feedback.

Our experiments show that relying solely on the LLM-generated reward function leads to poor performance. As shown in Figure 7, the mean return for the LLM-generated reward function pair consistently falls below the performance of all three MARL algorithms. This indicates that agents are not learning effectively under the LLM-generated reward function. However, we do observe a slight positive return. This suggest the potential of using this framework for reward shaping tasks, particularly in situations where standard MARL algorithms struggle to learn in sparse reward scenarios.

Figure: (a) MADDPG
Refer to caption: extracted/5902951/figure/LBF_MADDPG_only_reward.png

To investigate whether iterative refinement could improve the LLM generated reward function, we supply the LLM with the generated reward function from the prior iteration and feedback on its performance. Despite this iterative process, the LLM still fails to output a suitable reward function for the LBF environment. The mean return of evaluations remains close to zero, as shown in figure 9. The generated reward functions for each iteration are provided in Appendix E.

## 7 Limitation and Future Work

We acknowledge that the performance of YOLO-MARL may be highly correlated with the LLM’s ability and we haven’t tested YOLO-MARL with other LLMs like GPT-o1 due to the tier5 user requirement, and there might be a gap of YOLO-MARL’s performance between the Claude-3.5 and GPT-o1.

For future work, we are enthusiastic about the potential for LLMs to further enhance MARL, particularly as their planning capabilities improve. Specifically, we envision combining reward generation with planning functions to boost the performance of existing MARL algorithms in fully sparse environments. In this approach, we prompt the LLM to generate both a planning function and a reward function that replaces the environment-provided reward, following the pipeline described in Section 4. The function-pair method may require further refinement, and we will explore it as a future direction. A preliminary test of this framework is provided in Appendix B.4.

## 8 Conclusion

We propose YOLO-MARL, a novel framework that leverages the high-level planning capabilities of LLMs to enhance MARL policy training for cooperative games. By requiring only a one-time interaction with the LLM for each environment, YOLO-MARL significantly reduces computational overhead and mitigates instability issues associated with frequent LLM interactions during training. This approach not only outperforms traditional MARL algorithms but also operates independently of the LLM during execution, demonstrating strong generalization capabilities across various environments.

We evaluate YOLO-MARL across three different environments: the MPE environment, the LBF environment, and the SMAC environment. Our experiments showed that YOLO-MARL outperforms or achieve competitive results compared to baseline MARL methods. The integration of LLM-generated high-level assignment planning functions facilitated improved policy learning in challenging cooperative tasks, even in environments characterized by sparser rewards and large action spaces. Finally, we mention a possible way to incorporate reward generation to our framework and we will step further.

## APPENDIX

## Appendix A Hyperparameter Details

The detail hyper-parameter for the baseline algorithm can be found in Yu et al. () and Papoudakis & Schäfer (). We provide the full hyper-parameters for the reward and penalty value given to the RL training throughout the experiments in 2.

**Table 2: Hyperparameter**
|  | LBF |  |  |
| --- | --- | --- | --- |
|  | QMIX | MADDPG | MAPPO |
| r’ | 0.02 $\pm$ 0.01 | 0.002 $\pm$ 0.001 | 0.005 $\pm$ 0.004 |
| p’ | 0.02 $\pm$ 0.01 | 0.002 $\pm$ 0.001 | 0.005 $\pm$ 0.004 |

## Appendix B Additional result

Given the page constraints, we present some additional experiments and ablation study results and figures in this section.

### B.1 Comparison for different generated funcions

Considering the variation on the output of LLMs, we evaluate the quality of generated functions and compare the results on 3 baseline methods and those using our framework. We conduct the experiments in LBF environment introduce in Sec 1

Figure: (a) MADDPG
Refer to caption: extracted/5902951/figure/LBF_MADDPG_3F.png

**Table 3: Comparison between YOLO-MARL and MARL in the LBF environment across three different generated planning functions. The highest evaluation return means during training are highlighted in bold. The corresponding results can be found in figure 8. The M means one million training steps. We use two different machines to generate planning functions and run MARL and YOLO-MARL on the same machines where the planning functions are generated.**
|  | Mean Return after 0.2M / 0.4M / 1.5M / 2M Steps |  |  |
| --- | --- | --- | --- |
|  | QMIX | MADDPG | MAPPO |
| MARL | 0.00/ 0.01/ 0.25/ 0.36 | 0.08/ 0.28/ 0.24/ 0.29 | 0.38/ 0.74/ 0.99/ 0.99 |
| YOLO-MARL | 0.00/ 0.03/ 0.69/ 0.95 | 0.18/ 0.40/ 0.42/ 0.47 | 0.94/ 0.97/ 0.99/ 0.99 |

### B.2 additional results for reward feedback

Figure: (a) Iteration one
Refer to caption: extracted/5902951/figure/LBF_feedback_V1.png

### B.3 additional results for state interpretation ablation study

Figure: (a) Failure Case: Without providing interpretation code
Refer to caption: extracted/5902951/figure/without_interp.png

### B.4 additional result on future work

We tested this new approach that utilizing YOLO-MARL to generate planning and reward function pair in the SMAC environment with a fully sparse reward setting. The baselines tested on the three SMAC maps performed poorly, with evaluation win rates consistently near zero. However, as demonstrated in Figure 11, incorporating the planning function into reward generation significantly improved performance.

Figure: (a) results of 3m map
Refer to caption: extracted/5902951/figure/sparse_3m.png

## Appendix C Prompt Detail

In this section, we provide a comprehensive overview of the prompts used throughout the research/application to facilitate various tasks. The prompts play a critical role in guiding the behavior of language models or agents by providing them with specific instructions and constraints. This section details the exact wording, format, and context of the prompts that were used to achieve the results described in the main body of the paper.

### C.1 Strategy

The prompt for the strategy generation is consisted of Environment Description, Assignment Class and Instruction. Environment Description is about the environment information, we only provide some necessary description on what is this environment look like, what’s the goal for the tasks. We also add the rules for some additional information or constraint for the game that should be followed and they can be found on the official website. Assignment Class can be viewed as splitting up the action space or sub goals that LLM could assigned to agent during the task, the formal definition can be found on Section 4.3. The Instruction is basically to tell what llm should output for the strategy. Below we provide the sample prompt for the most simple scenario in each environment, but prompt for rest of all scenarios is in the similar format following the these prompts.
Level-Based Foraging
{mdframed}[backgroundcolor=gray!10, linecolor=black]
Environment Description:
This Level-Based Foraging (LBF) multi-agent reinforcement learning environment has 2 agents and 2 food items. Your goal is to make the agents collaborate and pick up all the food present in the environment.

Game Rules:

- 1.
The Pickup action is successful if all the agents pick up the same target together.
- 2.
The Pickup action is only successful if the sum of the levels of the agents is equal to or higher than the level of the food.
- 3.
The Pickup action is only allowed if the agents are within a distance of 1 relative to the food.
- 4.
Success Condition: All food must be picked up before {time_steps} steps.

Tasks Assignment:
Available tasks for each agent:

- 1.
Target food 0
- 2.
Target food 1
- 3.
Pickup

Instruction Format:
Here is a general guideline for generating strategies:

- 1.
Goal or Purpose: Clearly state the overall objective of the task.
- 2.
Problem or Need: Consider different scenarios and identify the key problem or need that the task plan addresses.
- 3.
Approach / Methodology: Describe the overall approach or methodology step-by-step that will be followed.
- 4.
Scenario Analysis: Consider different scenarios that agents could encounter during task execution and how they will coordinate to adapt.
- 5.
Task Breakdown: Break down tasks, detailing the roles and responsibilities of each agent and how they will coordinate to achieve the overall objective.

Multi-Agent Particle Environment
{mdframed}[backgroundcolor=gray!10, linecolor=black]
Environment Description:
This Multi-Agent Particle Environment (MPE) multi-agent reinforcement learning environment has 3 agents and 3 landmarks. Your goal is to make agents collaborate and cover all the landmarks.
Game Rules:

- 1.
Agents must cover all landmarks by minimizing the distances between each landmark, with each agent going to a unique landmark.
- 2.
Agents cannot collide with another agent. The collision threshold is 0.3.

Tasks Assignment:
Available tasks for each agent:

- 1.
Landmark 0
- 2.
Landmark 1
- 3.
Landmark 2
- 4.
No op

Instruction Format:
Here is a general guideline for generating strategies:

- 1.
Goal or Purpose: Clearly state the overall objective of the task.
- 2.
Problem or Need: Consider different scenarios and identify the key problem or need that the task plan addresses.
- 3.
Approach / Methodology: Describe the overall approach or methodology step-by-step that will be followed.
- 4.
Scenario Analysis: Consider different scenarios that agents could encounter during task execution and how they will coordinate to adapt.
- 5.
Task Breakdown: Break down tasks, detailing the roles and responsibilities of each agent and how they will coordinate to achieve the overall objective.

StarCraft Multi-Agent Challenge Environment
{mdframed}[backgroundcolor=gray!10, linecolor=black]
Environment Description:
This SMAC 3m map has 3 Terran Marines agents and 3 Terran Marines enemies. The Agent unit is Marines, and its feature is that Marines are ranged units that can attack ground and air units. They are the basic combat unit for Terran and are versatile in combat. Your task is to utilize the unit information to win the battle scenario within 60 steps.

Game Rules:

- 1.
Shooting range is 6 and sight range is 9 for both agent and enemy.
- 2.
Success condition: Eliminate all enemy units before the episode ends.
- 3.
Failure condition: If agents aren’t aggressive enough to kill all the enemies to win within 60 steps, or if all agents die.

Tasks Assignment:
Available tasks for each agent:

- 1.
Move
- 2.
Attack
- 3.
Stop
- 4.
None (only for dead agents)

Instruction Format:
Here is a general guideline for generating strategies:

- 1.
Goal or Purpose: Clearly state the overall objective of the task.
- 2.
Problem or Need: Consider different scenarios and identify the key problem or need that the task plan addresses.
- 3.
Approach / Methodology: Describe the overall approach or methodology step-by-step that will be followed.
- 4.
Scenario Analysis: Consider different scenarios that agents could encounter during task execution and how they will coordinate to adapt.
- 5.
Task Breakdown: Break down tasks, detailing the roles and responsibilities of each agent and how they will coordinate to achieve the overall objective.

### C.2 Interpretation Function

Here we list the Interpretation Function for each scenerios that process the raw vector observation.
LBF 2 player 2 food scenerio

MPE 3 agents scenerio

MPE 4 agents scenerio

SMAC 3m map

SMAC 2s vs 1sc map

SMAC 2c vs 64zg map

Sample output
We provide some sample strategy generated by LLM for the simple scenerios.
{mdframed}[backgroundcolor=gray!10, linecolor=black]
Based on the environment description and game rules, I’ll provide a strategy focused on an aggressive approach to eliminate all enemy units within the 60-step limit. Here’s a concise, clear, and actionable set of instructions:

1. Goal: Eliminate all 3 enemy Marine units within 60 steps.

2. Problem: We need to overcome the symmetrical nature of the battle (3v3 Marines) and exploit any positional advantages to win quickly.

3. Approach: Adopt an aggressive strategy with coordinated attacks to focus fire and overwhelm enemies one by one.

4. Observations and Task Breakdown:

a) Initial Positioning (Steps 1-5):
- All agents: Move to form a tight triangle formation, reducing exposure and enabling focused fire.
- Agent 1: Move forward slightly to act as bait.
- Agents 2 and 3: Position slightly behind Agent 1 on either side.

b) Engagement (Steps 6-20):
- Agent 1: Move forward to draw enemy fire.
- Agents 2 and 3: Attack the nearest enemy unit, focusing fire on the same target.
- If Agent 1 takes heavy damage, it should retreat behind Agents 2 and 3.

c) Focus Fire (Steps 21-40):
- All agents: Continuously attack the same enemy unit until it’s eliminated.
- Prioritize the enemy unit with the lowest health.
- If an enemy unit tries to retreat, pursue it while maintaining formation.

d) Adapt and Eliminate (Steps 41-60):
- After eliminating the first enemy, quickly shift focus to the next nearest enemy.
- If any agent’s health drops below 30 percent, it should retreat and use the other agents as cover.
- Maintain aggressive pursuit of enemy units, don’t allow them to regroup or heal.

Throughout the engagement:
- Utilize the maximum shooting range of 6 to attack enemies while minimizing damage taken.
- Use the sight range of 9 to spot enemies early and plan movements accordingly.
- Constantly reassess the battlefield and adjust tactics if the initial approach isn’t working.
- If victory isn’t achieved by step 50, take calculated risks to ensure all enemies are eliminated before the 60-step limit.

This strategy emphasizes aggressive play, coordinated attacks, and adaptability to overcome the symmetrical nature of the battle and achieve victory within the time limit.

### C.3 More prompts for planning function generation

To ensure that the function generation integrates effectively with the reinforcement learning (RL) training, we provide a detailed prompt that contains the goals and expected formats of the functions. This prompt is designed to guide the creation of the functions, ensuring they enhance agent cooperation and coordination. The prompt is as follows:
{mdframed}[backgroundcolor=gray!10, linecolor=black]Your task is to create a planning function and a reward function that work together to improve agent cooperation. The planning function should help each agent reach its goal, and the reward function should encourage smooth collaboration. Both functions should follow the guide from tips and focus on ensuring that the agents coordinate their movements to reach their goals simultaneously.

The environment code information is provided as follows:

def process_global_state(global_state, n=3, m=3):
…
return processed_global_state

The format for function generation is as follows:

The planning function should look like:

def planning_function(processed_global_state, available_actions):
”””
Determines optimal tasks for each agent based on the current battle state.
Args:
processed_global_state: A tuple containing (available_move_actions, enemy_info, ally_info, own_info)
available_actions: A dict of available action indices for each agent

Returns:
llm_tasks: Dict containing optimal tasks for each agent (Assignment Class)
”””
…

return llm_tasks

The returned ‘ll_tasks‘ should be in the ¡tasks assignment class¿ as specified. Use ‘processed_global_state‘ to inform decision-making.

The reward function should look like:

def compute_reward(processed_global_state, llm_tasks, tasks):
”””
Calculate rewards based on the tasks assigned and their outcomes.

Args:
processed_global_state: Returned from the function process_global_state(global_state, n, m)
llm_tasks (dict): Dictionary containing tasks assigned to each agent.
tasks (dict): Dictionary of tasks actually performed by each agent, e.g., ’agent_0’: …

Returns:
reward: Dict containing rewards for each agent. For example: ’agent_0’: reward1, ’agent_1’: reward2, …
”””
…

return reward
You should adjust the reward value for each component based on the importance as suggested in the tips.
You may use or import any necessary APIs for code generation, but do not write into a class object.
The generated functions should only include ‘planning_function‘ and ‘compute_reward‘. Do not create new variables or subfunctions.
Strictly follow the size, shape, and format of the action space and ‘processed_global_state‘.
Think step-by-step before generating the two functions based on the information provided. First, consider the information available in ‘processed_global_state‘ and how to use it in the functions. Second, analyze the environment description and determine the appropriate strategies and task assignments for each agent in this scenario.
Ensure the functions not only work correctly but also maximize agent coordination based on the instructions.

By supplying this prompt, we aim to generate functions that not only operate correctly within the RL framework but also maximize agent coordination based on the provided instructions. This approach ensures that the agents learn to work together effectively, ultimately enhancing the overall performance of the multi-agent system.

## Appendix D Examples of generated planning functions

## Appendix E Reward Generation with feedback

The functions provided here are the generated reward function including: iteration 1, iteration 2, iteration 3, iteration 4.

Iteration 1:

Iteration 2:

Iteration 3:

Iteration 4:

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: - Could you clarify if the inquiry occurs only once at the beginning of the game?
- Could you provide experimental results on more challenging tasks?
soundness: 1
strengths: I believe that using LLMs to enhance collaboration in MARL is a highly promising direction.
summary: This paper proposes a framework that utilizes Large Language Models (LLMs) to facilitate MARL collaboration. By leveraging LLMs for task planning and assigning different tasks to underlying agents, the framework influences policy learning through additional rewards, aiming to accelerate MARL algorithm learning.
weaknesses: - The methodology throughout the paper doesn’t feel novel to me; I’ve seen several similar studies, and the quality of this paper doesn’t seem satisfactory.
- None of the result figures appear to be processed; it looks as though they were directly downloaded from WANDB. Some figures show results from only one run, lacking statistical significance. I believe at least five runs should be conducted for each result.
- Certain detailed introductions seem unnecessary, such as STATE INTERPRETATION, while others, like PLANNING FUNCTION GENERATION, are not sufficiently elaborated.
- The improvements in experimental results are somewhat expected. However, for slightly more challenging tasks, such as those in SMAC (although the tasks provided are relatively easy), there is no significant performance improvement observed.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: (1) Can the authors clarify why MARL is specifically required for the YOLO-MARL framework? Would similar benefits not be realized in single-agent settings?

(2) It is recommended to include and discuss additional related works. How does YOLO-MARL compare with these approaches, especially concerning reward assignment and policy generation in multi-agent contexts?

(3) Could the authors extend their experiments to show long-term training results? It seems that the YOLO-MARL help faster convergence rather than higher performance, especifically in SMAC. Could the authors show longer training results in MPE and LBF environments?

(4) Additionally, testing in environments with higher strategic depth might validate the framework's utility in more complex scenarios.
How does the dependence on pre-trained LLMs affect the generalizability and reliability of the planning functions, especially in domain-specific environments, e.g., chess? The current experimental environments rely on the general pre-training of the LLM to generate a reasonably effective strategy, which while facilitating MARL convergence, might limit the method's upper potential.

(5) **In my view, using human experience-based "State Interpretation" to prompt the LLM, which then generates a base strategy "Planning Function" for RL, seems to be a straightforward application of LLMs in MARL.** This approach does leverage human-like reasoning for initial strategy formulation but might not fully exploit the potential complexities and capabilities of LLMs in dynamic or highly strategic game settings.
soundness: 3
strengths: (1) The use of LLMs in MARL to reduce the frequency of model calls is relatively novel. The one-time interaction strategy for each environment, minimizing computational overhead while leveraging LLM's capability, is an innovative approach.
Addressing the computational inefficiency in MARL through LLMs is of high relevance, especially given the increasing complexity and scale of multi-agent systems. The potential application across different domains enhances the paper's impact.

(2) The experimental setup is robust, employing several environments to validate the effectiveness of YOLO-MARL against established MARL benchmarks.

(3) The paper is well-organized, with clear descriptions of the methodology, experiments, and results. The inclusion of figures, tables, and appendices aids in understanding the proposed framework.
summary: The paper introduces "You Only LLM Once for MARL" (YOLO-MARL), a novel framework that integrates large language models (LLMs) with multi-agent reinforcement learning (MARL) to enhance policy learning in cooperative game environments. The core innovation of YOLO-MARL lies in its one-time interaction with LLMs per game environment, which significantly reduces computational costs and avoids the inefficiencies associated with frequent LLM invocations. The framework utilizes LLMs to generate strategy plans, interpret states, and create planning functions that guide the MARL agents. The approach is tested across various game environments, showing improved performance over traditional MARL algorithms.
weaknesses: (1) **The necessity of using MARL settings as opposed to single-agent RL isn't convincingly justified. The claim of innovation specifically in the MARL domain seems overstretched without substantial differentiation from potential single-agent applications.** 

(2) The paper fails to discuss several relevant studies that also integrate LLMs with multi-agent systems, which could question the novelty and depth of the literature review. Notably, it omits significant recent works, which could provide crucial context and comparison.

For example:

[1] Hu, Sihao, et al. "A survey on large language model-based game agents." arXiv preprint arXiv:2404.02039 (2024).
multi-agent Cooperation and Communication games:

[2] Gong, Ran, et al. "Mindagent: Emergent gaming interaction." arXiv preprint arXiv:2309.09971 (2023).

[3] Wu, Shuang, et al. "Enhance reasoning for large language models in the game werewolf." arXiv preprint arXiv:2402.02330 (2024).

[4] Zhang, Hongxin, et al. "Building cooperative embodied agents modularly with large language models." arXiv preprint arXiv:2307.02485 (2023).

(3) While the experiments demonstrate faster convergence with YOLO-MARL, the longevity and stability of training are not addressed. More extensive training or delayed results could provide deeper insights into the effectiveness and robustness of the proposed method.

(4) The reliance on human expert and LLM-generated planning functions could limit the applicability in strategic or domain-specific scenarios. The adaptability and accuracy of these functions in complex environments like Go or Dota2 are not discussed.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Does table 1 refer to the same results in Figure 2? Can you mention the standard deviations in the table too? I see a high overlap in them for some results, which raises a concern about the quantifiable impact that using LLMs is making in the proposed approach. 
- Can you consider training Qmix on LBF for longer (about 10M)? This would help observe/compare the performances till/after convergences. Both approaches still seem to not have converged yet. 
- How many independent trials were done for experiments in Section 6? 
- About experiments on smac, can you give more details on the planning function generation using LLMs in this environment, as the actions here are complicated when compared to MPE. Also, the results on smac are almost overlapping. So, does this suggest that we may not need YOLO-MARL on smac maps? Did you also run qmix and YOLO-MARL+Qmix on smac maps? Also, did you consider smacv2? Any comment on why or why not? 
- For experiments on reward generation in your approach, does the LLM generate a reward function? When does the LLM generate it? Before training (just like the strategy and planning function generation)? I am also unsure of more details on how this reward was generated. 
- In Figure 2, any comment on the dip in performance of MADDPG (and YOLO-MARL + MADDPG) around 450K episodes? 
- Referring to Figures 3 and 4, do you think the scalability of the proposed approach is a concern? The results almost start overlapping on increasing the number of agents in MPE. 
- Minor: Can you label the axes in figures.
soundness: 2
strengths: - The proposed idea is new, interesting, and well-motivated. 
- The paper is easy to read and follow. 
- Ample supplementary material containing examples and code helps in better understanding the proposed approach.
summary: The authors propose a novel framework (YOLO-MARL) to enhance MARL by integrating LLMs for high-level task planning in cooperative game environments. The motivation is to minimize computational demands by using only a single LLM interaction per game. YOLO-MARL includes four modules - Strategy Generation, State Interpretation, Planning Function Generation, and MARL Training. Experiments were conducted in useful benchmarks like Level-Based Foraging, MPE, and SMAC, with ablation studies to identify the role of some of the components of the proposed approach.
weaknesses: - Some more experiments and analysis/explanations may be required. 
- See questions.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: no specific question.
soundness: 3
strengths: - Contemporary paper proposing to use LLMs to better design the MDP description.
- LLM-guided multiagent RL is a novel subject
- Authors were considerate about the explosive computational cots of using the LLM at every step the agents would take and thus engineered functions in a way that the LLM would need to be queried only once for the environment design.
summary: The paper proposes to use LLMs to assist in the training of multiagent Reinforcement Learning tasks, by letting the LLM generate Strategy and Planning functions, those functions are used to modify the MDP for better training.
weaknesses: - The main issue I see is that, although the LLM is used to assist in the design of the MDP description, a great deal of domain knowledge is still required of the designer. Put together, the work to format the environment description for the strategy generation and state interpretation steps is on par or even harder than more classical approaches of performing feature engineering such as OO-MDPs or simply  dedicating a good effort into modeling the task well. 

- While the experimental results show significantly better results for the proposed approach, I feel it's an unfair comparison due to the significant effort required to describe well the environment to the LLM. The main argument in favor of the approach is that it required less coding when compared to classical feature engineering, but overall the same mental exercise would be needed not to mention the additional costs associated to using the LLM.

- My suggestion to make the approach more general and impactful is to instead use a multi-modal foundation model to handcraft some fixed prompts (that would be general for any environment) to analyze the environment autonomously through raw observations (such as images). That way, the approach would require significantly less effort from the designer and would make the approach more impactful.

- For the reasons above, in my opinion the paper does not reach the impact level of a main track ICLR paper.

- minor: I would name the approach in another way given there is already a widely famous YOLO model

Meta-review: The paper presents a framework that integrates LLMs with MARL through one-time LLM interaction per environment. The approach uses LLMs for strategy generation, state interpretation, and planning function generation to enhance MARL training. Despite interesting ideas and potential, the paper needs substantial improvements in experimental validation, technical details, and addressing scalability concerns before meeting conference standards.

Decision: Reject