M³HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Authors: Ziyan Wang
Zhicheng Zhang
Fei Fang
Yali Du
Venue: ICML 2025 poster (2025)
Keywords: Multi-agent Reinforcement Learning
Human Feedback
Primary area: reinforcement_learning->multiagent

Abstract: Designing effective reward functions in multi-agent reinforcement learning (MARL) is a significant challenge, often leading to suboptimal or misaligned behaviors in complex, coordinated environments. We introduce Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality ($\text{M}^3\text{HF}$), a novel framework that integrates multi-phase human feedback of mixed quality into the MARL training process. By involving humans with diverse expertise levels to provide iterative guidance, $\text{M}^3\text{HF}$ leverages both expert and non-expert feedback to continuously refine agents' policies. During training, we strategically pause agent learning for human evaluation, parse feedback using large language models to assign it appropriately and update reward functions through predefined templates and adaptive weights by using weight decay and performance-based adjustments. Our approach enables the integration of nuanced human insights across various levels of quality, enhancing the interpretability and robustness of multi-agent cooperation. Empirical results in challenging environments demonstrate that $\text{M}^3\text{HF}$ significantly outperforms state-of-the-art methods, effectively addressing the complexities of reward design in MARL and enabling broader human participation in the training process.

M
Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Ziyan Wang"
Zhicheng Zhang?
Fei Fang? Yali Du
Abstract
1. Introduction
Designing effective reward functions for reinforcement learning (RL) agents is well-known challenge, particu- larly in complex environments where the desired behaviors are intricate or the rewards are sparse (Singh et al., 2009; Ng et al., 2000). This difficulty is magnified multi-agent reinforcement learning (MARL) settings, where agents must not only learn optimal individual behaviors but also coordi- nate with others, leading to exponential increase in task complexity (Zhang et al., 2021; Oroojlooy & Hajinezhad, 2023; Du et al., 2023). Sparse or hard-to-learn rewards can severely hinder the learning process, causing agents to con- verge slowly or settle on suboptimal policies (Andrychowicz et al., 2017; Pathak et al., 2017). In such scenarios, relying solely on environmental rewards may be insufficient for effective learning. Incorporating human feedback has thus emerged as promising approach (Christiano et al., 2017: Knox & Stone, 2009; Ho & Ermon, 2016), since human guidance can provide additional, informative signals that help agents navigate complex tasks more efficiently when intrinsic rewards are inadequate.
Designing effective reward functions in multi- agent reinforcement learning (MARL) is sig- nificant challenge, often leading to suboptimal misaligned behaviors in complex, coordinated en- vironments. We introduce Multi-agent Reinforce- ment Learning from Multi-phase Human Feed- back of Mixed Quality (M'HF), novel frame- work that integrates multi -hase human feedback of mixed quality into the MARL training pro- cess. By involving humans with diverse exper- tise levels to provide iterative guidance, M'HF leverages both expert and non-expert feedback to continuously refine agents' policies. Dur- ing training, we strategically pause agent Iearn- ing for human evaluation, parse feedback us- ing large language models to assign it appropri- ately and update reward functions through pre- defined templates and adaptive weights by us- ing weight decay and performance-based adjust- ments. Our approach enables the integration of nuanced human insights across various levels of quality, enhancing the interpretability and robust- ness of multi-agent cooperation. Empirical re- sults in challenging environments demonstrate that H significantly outperforms state-of-the- art methods, effectively addressing the complex- ities of reward design in MARL and enabling broader human participation in the training pro- cess. Code is available at https: / /htb. htt/cooprraiiee...
To leverage human expertise in accelerating the learning process of MARL agents, we propose the Multi-phi phase Hu- man Feedback Markov Game (MHF-MG), an extension of the Markov Game that incorporates human feedback across multiple generations of learning. At each generation, agents gather experiences using their current policies but may still struggle under the original reward function. Humans then observe the agents' behaviors, offering feedback that re- flects the discrepancy between their (potentially more expert) policy and the agents' policies. Building the MHF-MG, we develop Multi-agent Reinforcement Learn- ing from Multi-phase Human Feedback of Mixed Quality (M'HF), which operationalizes the MHF-MG by directly integrating feedback into the agents' reward functions. This framework utilizes large language models (LLMs) parse human feedback of various quality levels, employs prede- fined templates for structured reward shaping, and applies adaptive weight adjustments to accommodate mixed-qualis signals.
'Work done during a visit to CMU. 'Deparmment of In- formatics, King's 's College London, London, United Kingdom "Softaree and Societal Systems Department, Camegie Mellon University, Pittsburgh, PA, USA. Correspondence to: Ziyan Wang Slayan.wang@e @k..gkk...c..k....
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In summary, we make the following contributions: (1) We propose the MHF-MG address reward sparsity and com-

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Multi- auent
Rollout Video Fk
Reward Templates F
New Reward Functions for Each Agent
Act Ton- based
State- based
Blatance- based Ri
Prox bassd
Success based
bistance
Time based
Actiion- based
Human Feedback Uk
LLM
Lombos ibs, alsa 81 a2s12)
"The rose chef should first, then the green cheff can Lake it and cut it ws quickly as possible. The blue chef can take plate put
the torato
else
-hased
R ())
Larbda eko, set: of arc
Reward Functions Update
wlie if ets and cbs(3] else B)
(+,1 E, Why R)(2,a) NRJ P.
Funct Lin Weights Adjustment
Reward Function Pools
Figure I. Workflow of the H' method. Each generation & E (O, K 1) begins with Multi-agent RL training. Agents generate rollout vidcos Tu for human evaluation. Human feedback 2g is parsed by a Large Language Model (LLM) into agent-speciff instructions. The LLM then selects appropriate reward function templates f E F and parameterizes them based on the parsed feedback. New reward functions R2,s,, (s, a) are added to each agent's 's reward function pool P., with weights wi,, adjusted using verformance-basee criteria. The updated reward functions R2 (s, a) guide the next generation of agents' training, creating a loop of agents learning from feedback.
plexity in MARL through iterative human guidance: (2) We develop the M M'HF framework, which leverages LLMs to parse diverse human feedback and dynamically incorporates it into agents' reward functions; (3) We provide theoretical analysis justifying the use of rollout-based performance esti- mates and offering a weight decay mechanism that mitigates the impact of low-quality feedback. Extensive experiments in Overcooked demonstrate that HO consistently out- performs strong baselines, ultimately providing a robust and flexible method for enhancing multi-agent cooperation under challenging reward structures.
into contributions from self and nearby agents (Zhang et al.. 2020), combining dense individual rewards with sparse team rewards (Wang et al., 2022).
Reinforcement Learning from Human Feedback (RLHF) has emerged as a promising avenue to address the limitations of handcrafted reward functions. Christiano et al. (2017) introduced methods for training agents using human pref- erences to shape the reward function, demonstrating that human feedback can significantly enhance policy learning. Building on this, Lee et al. (2021) proposed PEBBLE, lever- aging unsupervised pre-training and experience relabeling to improve feedback efficiency in interactive RL settings.
2. Related Work
While RLHF has been successfully applied to train Large Language Models (LLMs) (Ouyang et al., 2022; Shani et al., 2024; Wang et al., 2025; Hu et al., 2025; Liu al., 2024), these approaches primarily focus on aligning LLM outputs with human preferences through single-turn interactions and scalar reward signals. Recently, PЫMARL (Zhang et al., 2024) applied RLHF to MARL by utilizing offline pairwise preference comparisons extracted from pre-collected simu- lated policy data, but remained limited to an offline single- phase feedback scenario without dynamic, iterative human interactions. In contrast, work incorporates multi-phase phase, mixed-quality human feedback directly into the reinforce- ment learning loop during online training in a multi-agent environment, enabling more flexible and scalable reward shaping.
Multi-Agent Reinforcement Learning (MARL) has been extensively studied to enable agents to learn coordinated behaviors in shared environments (Du al., 2023; Yu et al., 2022). Traditional MARI. approaches often rely on pre- detined reward functions and suffer from scalability and stability issues arising from the non-stationarity introduced by multiple learning agents (Busoniu et al., 2008; Canese et al., 2021). However, designing appropriate reward func- tions in MARL remains significant challenge due to the complexity of agent interactions and the potential for con- flicting objectives. To address this, researchers have ex- plored various techniques for reward design. These include credit assignment methods (Nguyen et al., 2018; Zhou et al., 2020), reward shaping (Mannion et al., 2018), and the use of intrinsic rewards (Du et al., 2019). Furthermore, reward decomposition approaches have been proposed to balance individual and team objectives, such separating rewards
Language Models in Reward Design and Policy Learn- ing. Recent advancements in LLMs have created oppor-
2

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
tunities for incorporating natural language guidance into reinforcement learning (RL) (Liu al., 2022; Chen al., 2024). For instance, Ma et al. (2024) introduced EUREKA, method employing code- gode-generating LLMs to craft sophis- ticated reward functions at a human-expert level. However, extending methods such as EUREKA to multi-agent settings remains non-trivial, as independently performing rollouts to optimize rewards for each agent can become prohibitively expensive and computationally intensive. Similarly, Liang et al. (2023) proposed Code as Policies, where language model programs are used for embodied control, allowing agents to interpret and execute high-level instructions.
3. Preliminaries
We consider a Markov Game (Littman, 1994), defined by the tuple (W.S, R, 7), in multi-agent reinforcement learning (MARL). Here, . = 21.2...... ] represents the set of agents. The state space S encompasses all possible configurations of the environment, while the action space A denotes the set of actions available to each agent. At each time step t, the environment is in a state 8t E S. Each agent i E V selects an action a; E A according to its policy x (aj]ss) The joint action a, = (at. ; aaa) leads state transition 814-1 according to the transition function P(s++ 82, a²). The agents receive shared reward re = Rs,, a²), where R: S A + R is the reward function, and y E [0, 1) is the discount factor. The objective for each agent is to learn a policy n' that maximizes the expected cumulative discounted reward:
Other recent works have further leveraged LLMs to trans- late linguistic instructions into reward functions and policies. Yu et al. (2023) utilized language instructions shape re- wards for robotic skills, effectively aligning robot behaviors with human guidance. Kwon et al. (2023) highlighted the capacity of language models to capture nuanced human preferences during reward design. In human-Al collabora- tion (Zou et al., 2025), Hu & Sadigh (2023) demonstrated how language-instruckt RL. could foster improved coordi- nation between humans and agents, Wang et al. (2024) uses LLMS to convert language constraints into cost signals for agents, Additionally, Liang et al. (2024) proposed leverag- ing language model predictive control to iteratively refine RL policy learning from human feedback more quickly.
((TT)
=E
E
Σ
T
(I)
where - denotes the policies of all agents other than agent i, and the expectation is over the trajectories induced by the policies and the environment dynamics.
In our setting, although the reward function R known (denoted as original reward function Roa), it challenging for agents to learn optimal policies due to its sparsity or complexity. This difficulty can lead to slow convergence or suboptimal performance for traditional reinforcement learning algorithms.
Relatedly, Klissarov et al. (2023) presented MOTIF, adopt- ing -eLe-egee intrinsic rewards from textual task spec- ifications to promote effective exploration in single-agent environments. Although MOTIF effectively leverages lan- guage grounding, it is inherently limited to single-agent contexts and does not incorporate iterative human inter- actions or complex multi-agent cooperation scenarios. In contrast, our approach specifically employs LLMs to parse multi-phase, mixed-quality human feedback dynamically to iteratively refine reward functions in complex multi-agent coordination problems.
4. Method
To address the challenges posed by sparse or complex re- ward functions in multi-agent environments, we introduce the Multi-phase Human Feedback Markoy Game (MHF- MG) as a tuple, (N,S, A, P. R. R, T²)). Compared to a standard Markov Game, the added U denotes the set of possible human utterances or feedback messages, n² rep- resents the human's 's policy. In this framework, the agents interact with both the environment and a human over dis- crete generations indexed by k: 1, K - 1. At each generation k. agents collect experiences by interacting with the environment using their current policies _
Multi-phase Human Feedback. Prior works have consid- ered the role of iterative and multi-phase human feedback in reinforcement learning. Yuan et al. (2022) and Sumers et al. (2022) explored multi-phase bidirectional interactions between humans and agents through predefined communica- tion protocols, which, while structured, limit the flexibility of feedback. Zhi-Xun Thi-Xuan et al. (2024) examined the use of human demonstrations via trajectories to convey intentions, requiring humans to perform the task themselves, which can be resource-intensive. Early attempts by Chen et al. (2021) and Zhang et al. (2023) delved into language for task generalization and policy explanation but were constrained to single-agent domains.
Each generation k consists of hundreds of iterations; each iteration comprises tens of episodes, and each episode spans on the order of hundreds of environment time steps t, de- pending on the specific environment. This setup allows agents gain substantial experience within a generation before receiving human feedback. The human possesses a policy T², which may be sub-optimal but is assumed to be initially superior to the agents' policies. This human policy provides valuable guidance that can accelerate the agents' learning. The human observes the agents' behaviors and
3

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
(a) Overcooked-A
(b) Overcooked-B
(c) Overcooked-C
(d) Pettuce-Tomalee salad recipe
(e)
Pettuce-Onions
Tomato salad recipe
Figure The Overcooked Environment. (a)-(c) The three different kitchen layouts with increasing difficulty: (a) Overcooked-A A offers ample movable space; (b) Overcooked-B B forces agents divided on both sides to cooperate due to the partitioned kitchen; (c) Overcooked C has less movable space compared
(d)-(e) The two salad recipes: In both recipes, the corresponding chopped foods must be combined on single plate and delivered. To facilitate training, we use macro-actions based on (Xiao et al., 2022), where the agents' actions are simplified. More details refer to Section
generates utterances VA E l at each generation k, offering feedback based on the discrepancy between their own policy and the agents' current policies.
trajectories induced by the policies and the environment dy- namics. This iterative process continues across generations, with agents repeatedly interacting with environment, receiving human feedback, and updating their reward func- tions and policies accordingly. The inclusion of human feedback helps agents navigate the challenges of sparse or complex reward functions by providing additional signals that highlight desirable behaviors and strategies. In this pa- per, to minimize human involvement, we limit agent-hummn interactions to at most five times throughout the entire train- ing process, enabling efficient learning with minimal guid- ance.
We model the human's utterances as mapping from the human's 's policy and the agents' policies to the set of possible utterances:
Uk = f (ITh Rk
(2)
where captures how the human generates feedback by comparing their policy with those of the agents. The utter- ances may include specific action recommendations, strate- gic advice, or corrections aimed at guiding the agents toward better performance. The agents parse the human's 's utterance ue extract actionable information. This process may involve natural language understanding techniques, poten- tially leveraging large language models (LLMs) to interpret the feedback accurately. Based on the parsed feedback, each agent adjusts its reward function to incorporate the human's guidance. For agent i, the updated reward function generation k + becomes:
4.1. Agent to Human Interaction
In the MHF-MG, agents periodically interact with the hu- receive feedback that guides their learning process. This interaction is initiated by the agents, who decide when to seek human input based on specific criteria. The primary mechanism for this interaction is through the generation of rollout trajectories, which approximate the agents' current policy performance.
R (8, a) R(s,a) + Pari (8, Ux),
(3)
Rollouts Generation and Communication. During each generation k. agents interact with the environment using their current policies oं. collecting substantial experience over multiple iterations. However, due to the complexity or sparsity of the original reward function Rt agents may still face difficulties in identifying efficient strategies or converging quickly to optimal policies.
where Rack represents the reward adjustment derived from the human's feedback ug at generation k. This adjustment modifies the reward signal to encourage behaviors aligned with the human's guidance, effectively reshaping the agents' learning objectives.
The agents then update their policies for the next generation by optimizing the expected cumulative reward under the new reward function ++++1 Formally, the policy update for agent is given by:
To address these challenges, agents periodically seek human feedback based predefined criteria. Specifically, after every fixed number of training episodes, agents temporarily pause their training and generate evaluation rollouts. These rollouts consist of X independent trajectories collected un- der the fixed joint policy v², defined formally as:
Th41 arg max E
Σ 7 Rit1 (8t, T') (4)
where Tp. denotes the policies of all other agents at genera- tion and the expectation is taken over the distribution of
Tk = (0(0
a (x)
#-1 x= I,..,,, X, (5)
1=0

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
where H is the time horizon, 8) denotes the state at time t, ar is the joint action chosen by all agents in trajectory a, and ri is the resulting environmental reward. These rollouts allow human observers assess current agent be- haviors and provide structured, actionable feedback for sub- sequent policy improvements.
4.2. Human to Agents
(2)
Feedback Parsing.
Our method employs Large Language Model (LLM), denoted as M, to parse the human feedback ug received at generation & and assign it either to specific agents or to all agents collectively. This parsing process is mathematically represented as u 4k
للل - M(ux, N), where
N is the number of agents, uj is the feedback assigned to agent i, and uz is the feedback applicable to all agents. This approach ensures that each agent receives relevant instructions corrections based on the human input. The detailed prompts used for guiding the LLM in this parsing process are provided in the Appendix D.4.
Approximation of Policy Performance via Rollouts Con- sider a stochastic game in which all agents follow stationary joint policies. At generation k the joint policy nh com- bined with the environment dynamics induces a distribution over state-action-reward trajectories. Our goal is formally justify that empirical estimates obtained from multiple col- lected rollouts reliably approximate the true performance J (**)) Given the empirical trajectories Th (mm) defined previ- ously, we define each rollout's 's discounted return as:
The new reward func- tion Rg for the current generation involves selecting and parameterizing predefined function templates based on hu- man feedback. For each agent i, the new reward function for the current generation i is generated as follows:
Generating New Reward Function
H-1 Σ
G H (1
x x= ==1..... X,
all (s, - (((s,))_M[Fru u "t e),
(10)
and consider the empirical performance estimator:
where F is set of predefined function templates, uf is the parsed feedback for agent at generation k, and e the entities based on the environment states.
X
Jx (x =
G H
==1
Under the Strong Law of Large Numbers (Kolmogorov, 1933; Durrett, 2010), as M + co, this estimator converges almost surely to the finite-horizon expected discounted re- turn:
Predefined Function Templates. In our framework, we define a set of predefined reward function templates F that can be parameterized based on human feedback and the specific entities within the environment, as shown in Fig- ure 1. These templates enable the system to systematically generate reward functions aligned with human intentions, facilitating efficient policy updates in response to feedback. The templates capture common interaction patterns such as distance- -baed rewards that encourage agents to minimize their distance to target entities, action-based rewards that incentivize specific actions, and status-based rewards that reward agents for achieving certain states of the environ- ment. For instance, given the human feedback "I think the red chef needs to be responsible for getting the onion" and the LLM will select the distance-based reward template and parameterize it as:
(x,x(=²)
a.a. [=(
X-000
H-1
(8)
where ==(=²)==11 (1 7) Σ
7 B(rr).
Furthermore, when horizon length H grows towards infinity, the finite-horizon discounted return Jn (=*) converges to the true discounted return J (T*). Thus, we have: Proposition 4.1 (Performance Estimation). Under Assump- tion A and Assumption A.4, assuming bounded rewards and given X independent rollouss of length H collected under policy x² the empirical estimator converges almost surely to the true discounted expected return:
R (s. a) = -1ss ||s||geetl 1 .pos] - s[Onnionpos]]l2
(11)
(x*)
J((*) (a.s.)
Here, the s[Agentl pos] and s [Onion.pos] are the relevant entities of the observation vector, which encourages Agent (The red chef in the rollout video) to minimize its distance to the onion, thus aligning its behavior with the desired ob- jective. This structured approach allows agents interpret and act upon multi- multi-fidelity human feedback effectively. De- tailed formulations of these reward templates and additional examples are provided in the Appendix D.2.
Leveraging Proposition 4.1, the multiple rollout trajectories serve as reliable empirical estimates of the policy perfor- mance J (m*). ). Human evaluators use these rollouts to effec- tively assess agent behavior and provide targeted feedback M4 = -[TT; "). This assumption that empirical rollout data accurately reflects policy- induued distributions is analogous to common practices in offline reinforcement learning and policy evaluation contexts (Levine et al., 2020; Kumar et al.,
Reward Function for the next generation. At the end of the processing of the human feedback in generation k we conclude the final reward function for each agent to a
2019).
5

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
150
250
KAPPO
Episode Raturn Mean
Gen
Epizodd Mean
Gen
Gen
Gen
100
Gen
slsoed Macelien
Gen
Gan
Gen
Gen
-50
Gon
Gen
Gen
Con
Iteration
Iteration
Iteration
(a) Overcooked-A: Lettuce- Tomato Salad
(b) Overcooked-B Lettuce- Tomato Salad
(c) Overcooked C: LettLee--Toneee Salad
IPPO
Gen
125
IPPO (Bareline)
Gen
IPPO 18 saseline KAPPO
Gen
MAPPO
Gen
MAPPO
Gen
Gen
Epirode Raiurra Mean
125
Episode Return Mean
230
Gen
Gen
Gen
Gen
200
Baselie
Mac Ouand Desetime
Return
75
30
12
20
-25
205
420
620
820
1050
200
400
500
503
1000
200
400
Iteration
6000
300
2000
(d) Overcooked-A Lettuce-Onion Tomato (e) Overcooked-B
Onion-Tomato (f) Overcooked-C Lettucc-Oninii Onion-Tomatoo
HF against baseline methods across different Overcooked environments and recipes. The plots show the mean episode return 1000 training iterations (approximately 25k episodes) for (a-c) Lettuce-Tomare salad recipe and (d-f) Lettuce-Onion- Onioo-Tonntoooo salad recipe in Overcooked layouts A, B, and C. respectively. ' consistently outperforms the baseline methods (Mac-based Baseline, IPPO, MAPPO) across all scenarios, with performance improvements becoming more pronounced in more complex environments and recipes. Vertical lines indicate the start of each generation where human feedback is incorporated. All experiments are with three random seeds, and the shaded areas represent the standard deviation.
Figure 3. Performance comparison S
weighted combination of the base reward and the consis- tency weight:
functions:
Wimm Wi,m C.M.M M- Vm I,...,,
M M
(13)
E, wis RR((s))) e P; 1 =)
(12)
where E (0, 1) is a constant decay factor. We then nor- malize all weights by using
Here, 1/ (s, a) denotes the final reward function for agent i after processing the k-th generation of human feedback, and it will be used for the next generation k + for the policy training and await the subsequent rollout generation and human interaction, as outlined in Algorithm
Wi.m
Wi.m Σ Wijm
Vm E 1, I,...,,4 M.
(14)
Additionally, we introduce a performance-based adjustment rule that compares the agent's performance under the origi- nal reward function R across consecutive generations. We calculate Fon -Tonk- where Fonk+41 is the performance of the policy trained using the new reward function fij (s, a) (after processing human feedback in generation k) when evaluated on R and Foni is the performance of the policy trained using the previous reward function R2 (s, a) (before processing human feedback in generation k)) when evaluated Rig If this difference is positive, it indicates that the new reward function leads to improved performance on the original task. Otherwise, it suggests that the new reward function may be detrimental to the agent's performance on the original task. We then adjust the weight of the newest
One main challenge is how to set the weights wi, which can effectively balance different reward components and adapt to changing human feedback. To address this challenge, we employ weight decay and performance-base based adjustment to optimize the weights Wi.z.
4.3. Weight Decay and Performance-basee Adjustment
The straightforward way to adjust weights is based on a simple weight decay mechanism and performance feedback. When generating new reward function, we add to the pool P., then set an initial weight, Wi,m म Then, we apply a decay to existing weights of the former reward
6

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
reward function component Wi,m as follows:
where ê is a bounded, positive constant independent of K, and each ij denotes a generation index at which the received human feedback reward is beneficial (i.e. yields a positive increment Ari," > 0).
Wi,m +3,,
if ronk+1 Fort > O,
Us.m max(0, Wi,m 3), otherwise,
(15) where 33 is small adjustment factor. This approach allows for the dynamic adjustment of the reward function pool, emphasizing recent human feedback while maintaining a di- verse set of reward components and adapting to performance changes.
Intuitively, Proposition 4.2 formalizes the notion that our algorithm accumulates the positive contributions of helpful feedback over multiple rounds of interaction. In contrast, its performance can suffer at most a single bounded degra- dation, reflecting the limited and transient influence of the most recent faulty feedback. Importantly, this robustness property ensures stable long-term learning dynamics even when human inputs are imperfect or mixed quality. We provide a detailed proof of this proposition in Appendix B.
4.4. Analysis of the Low-Quality Feedback
We now examine the robustness of our proposed M'HF framework under the scenario where human feedback of mixed quality-inehuding noisy, irrelevant, or erroneous instructions- is provided to the agents. Such situations naturally arise due to confusion, limited human domain expertise, misinterpretation of agent behaviors, or misun- derstanding of the task objectives. Under these conditions, it is crucial for a robust algorithm to elearly mitigate the negative impact from low-quality feedback signals, while consistently leveraging high-quality feedback to enhance learning. Formally, each generation k; we integrate re- wards by forming weighted combination of the original task-based reward and the accumulated human- feed- back rewards:
5. Experiment
In our experiment, we aim to address three key questions: Q1. What is the overall performance of M compared to current state-of-the-ae methods? Q2. To what extent does multi-quality human feedback impact the performance of M within the same environment? Q3. Can Vision- Language Models (VLMs) serve as a scalable and effective alternative to human feedback in M HFF In all experiments involving language-driven feedback parsing, we use the LLM gpt- 40 2024 1111--111111 11-20 (OpenAI, 2024).
Environment Macro-Action-Bastt Overcooked, as shown in Figure 2. In our experiments, we utilize a chal- lenging multi-agent environment based on the Overcooked game (Wu et al., 2021; Xiao et al., 2022), where three agents must learn to cooperatively prepare a correct salad and de- liver it to a designated delivery cell. We followed the work of Xiao al. (2022), where agents operate using macro- actions derived from primitive actions. These macro-actions facilitate effective navigation and interaction within the envi- ronment but also introduce complexities in learning optimal policies due to the increased action space. Each agent ob- serves only the positions and statuses of entities within a 5 5 square centered on itself, introducing partial observ- ability and heightening the coordination challenge. The agents will only receive significant reward for delivering the correct salad (+200) and punishment if they deliver the wrong salad or food (-50). During training, each generation consists of 200 iterations; each iteration runs 25 episodes of up to 200 timesteps. For more details about the environment setting, please refer to the Appendix C.
Rk
Wm Ran.
(16)
m==0
where we define Ro = Rors as the original reward func- tion and Rm>0 = Rhaman as human-gent -ennaa-eee feedback rewards. Correspondingly, the weights wh are dynamically adjusted as described in Equation 15, explicitly accounting for observable agent performance improvements or degra- dations.
Due to the design of our weighting mechanism, negative or unhelpful feedback rapidly loses influence, their corre- sponding weights decrease after any observed dip or stag- nation of performance improvements. Conversely, helpful feedback continuously guides performance upwards, main- taining substantial influence through increased weighting in the combined reward function. We formalize this intuition into the following robustness result:
Proposition 4.2 (Robustness to Low- Now-Quality Human Feed- back). Under Assumption A.3 (Performance Estimation Ac- curacy) and Assumption A.2 (Learning Algorithm Conver- gence), for any given integer K 1 and any arbitrary sequence of human feedback rewards (Rx))=11...... the performance improvement satisfies:
Baselines. We evaluate against three strong multi-agent reinforcement learning approaches: The MAPPO (Yu et al., 2022), IPPO (De Witt et al., 2020), and Macro-Action- Based Baseline from Xiao et al. (2022). Our own frame- work adopts IPPO as the backbone algorithm, while the macro- aatton baseline is the average performance of the two best-performing methods in Xiao et al. (2022), namely Mac-
n(K)
JOn(TT)= - -oo) 2
Σ 1=1
Ari, (8 -),
(17)

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
meeds tryaee the vegraaia
Muu suddy plsane the
Human Feedback
"The blue and green chefs should not serve the tonato and onzon salad, which should After receiving before serving if- The red should not touch the vegetables after placing then on the middle table.
125
IPPO (Baseline)
wrono
MAPPO
100
M3HF (wl Lew Quality Fendback)
Episode Return Mean
M3HF VLM Feedback)
M3HF (wl local Human Foodback)
g Low Quality Feedback
Mac-Based Baseline
"The agents succesafully complete the task or creating two lettuce, onion, and tonato salads.. Nothing to isprave"
50
VLM Feedback
agent, you knite the ingredients and choppang near their source to agents, you can isprove coordination. Mhile one agent chops, the other could alreacy be retrieving a plate or preparing the next ingredient.
consider bringing the
time.
Impertinn: like Agart onan loneoo bubi needs iroue
Enegane: bis ayem edvennd Enooed as the wila
Gen
Gen 1
Gen 2
Gen
Gen 4
200
400
600
800
1000
Iteration
(a) Example rollout in Generation 3 (b) Feedback example from different source
(c) Overcooked-BB B Lettuce-Onion Caaee--OOno----Taata- Salad
Figure 4. Impact Mixed-Quality Feedback on Agent Performance. (a) An example is rollout Generation 3, where agents exhibit suboptimal behavior due to poor coordination and inefficient task exccution. (b) Low-quality feodback provided to the agents, inaccurately stating that they successfully completed the task and offering no constructive guidance for improvement. (c) Performance comparison on salad recipe under mixed quality feedback conditions.
Overcooked-BB B with the Cattuce-Onnin--e
IAICC and Mac-CAC, over 25,000 training episodes. Fur- ther details on these baselines can be found in Appendix D.
Experiment Results for Question Impact of Mixed- Quality Human Feedback We evaluated method when facing low-quality feedback by simulating such feedback at each generation. For example, in the rollout shown in Figure 4a, agents exhibited suboptimal behavior due to poor coordination. Despite this, the low-quality feedback inaccu- rately stated, "The agents successfully complete the task of creating two lettuce, onion, and tomato salads. Nothing to improve," depicted in Figure 4b. When training with this irrelevant erroneous feedback, the agents' performance, illustrated in Figure 4c, remained only slightly below that of the baseline IPPO algorithm and did not degrade signifi- cantly. This outcome supports Proposition 4.2, demonstrat- ing that M "F effcetively mitigates the impact of unhelpful feedback through weight adjustment mechanisms. Even with mixed-quality human input, the framework maintains performance close to the backbone algorithm, showcasing its resilience to low-quality guidance.
Experiment Results for Question 1: Overall Perfor- mance of M HF Figure 3 demonstrates the superior per- formance of M compared to SOTA baselines across various Overcooked environments and recipes. Our method consistently outperforms Mac-based Baseline, IPPO, and MAPPO in all scenarios, maintaining& a substantial perfor- mance advantage across different levels of task complexity. The method exhibits accelerated learning, particularly in early training stages, and achieves higher asymptotic per- formance levels. Notably, in the simpler recipe setting, Figure 3a, 3b and 3c, H converges to the optimal per- formance less than five rounds of interaction, showcasing the method's exceptional efficiency in more straightforward settings. The method's robustness is evident as we move to more complex environments. In the challenging Layout C (Figure 3e and 3f), 'DHF maintains its effective perfor- mance advantage. particularly outperforming its backbone algorithm IPPO. This consistent superiority across varying complexity levels underscores M M'HF's effectiveness and adaptability in diverse multi-agent scenarios.
Experiment Results for Question 3: VLM-based Feed- back Generation We explore the potential of VLMs as an alternative to human feedback. The VLM is given the same video rollouts that humans would observe, sampled at a rate of frame per second. Using all sampled frames and prompt asking for feedback (detailed in Appendix D.4), the VLM generates feedback to the training agents. In our imple- mentation, we leverage Cemini-l- 1-..o---00022222 (Reid et al., 2024), which is chosen for its multimodal under- standing capability across a long context. We showcase an example of VLM feedback in Figure 4b alongside the human feedback. Here, the feedback provided by the VLM resembles human-like style but lacks specificity on critical issues, which, in this case, are "wrong variety", "cut it and
In addition, it is important note that the baseline MAPPO employs shared policy among agents alongside a cen- tralized value function, while IPPO utilizes independent policies. We have observed that IPPO often achieves better results in highly coordination intennsiveeeeeeeeeeeee scenarios such as Overcooked, potentially due to reduced interference among agents during policy training. Prior studies (De Witt al., 2020; Yu et al., 2022) similarly report that IPPO can match or outperform MAPPO even without centralized critics.
8

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Table Ablation results comparing feedhack parsing and weight adjustments. Overcooked-B B: Lettuce-Onion Tomato Salad sce- nario.
Table 3. Comparison of intrinsic reward-basee methods with our
M'HF HF on Overcooked-2 B: Settuce-Onion Tomato Salad task. Eval- uation training iterations 400, 600, 800, and 1000 (corresponding to Generations 1-4 in M3HF). Mean and standard deviation re- ported over three seeds.
Method
Average Return (Mean # Std)
Raw Feedback (wlo parsing)
45.3 5.2
LLM Parsing Only (no weight adj.)
68.7 = 4.1
Algorithm
Iter. 400
Iter. 600
Iter. 800
Iter. 1000
Full H'HF (LLM parsing weight adj.)
102.7 10.8
IPPO (basc)
19.2 + 4.5
23.1 2.7
23.2
27.4 + 4,9
Table 2. Performance comparison between single-phase and multi- phase feedback methods. Overcooked-Bi B: Lettuce-Onion- Tomato scenarro.
IRAT-rX.1
68.9 10.1
52.5+11.3
78.2 14,5
94,9 10.7
IRAT-rH.2
1.1
93=1114
16.0
34,5 14.0
IRAT-rH.3
99.1
17.3 10.6
33.8 8.73
M'HF (Ours)
164.8 1.2
Method
Average Return (Mean Std)
Single-phase feedback (initial only) Multi-phase feedback HMF
43.1 + 10.3
102.7 10.8
as rw_1, rw-2, zw_3) reflecting progressively stronger co- ordination requirements:
mix it before serving it". Instead offers vague suggestions like "improve coordination", which is hard to translate into reward design. This limitation indicative of the VLM's current inability to perform complex reasoning across im- ages. As a result, when plugged into our M HF framework, the VLM feedback method does not yield much benefit, as shown in Figure 4c. Nonetheless, we expect improved performance with future advancements in VLM.
rw-1: Rewards an agent whenever it picks up or chops any ingredient.
rw 2: Rewards the agent upon successfully reaching the knife after picking up an ingredient.
Ablation 1: Impact of Feedback Parsing and Weight Adjustment. We compare our complete proposed feedback integration pipeline ("Full M'HF") against two variants: Raw Feedback, which directly translates human instructions into rewards without parsing or structured adjustment, and LLM Parsing Only, which parses human feedback into struc- tured reward functions without performance-basee weight adjustment. The results are summarized in Table 1. The results indicate that structured parsing of feedback impacts considerably performance, increasing average returns sig- nificantly compared raw human feedback. Furthermore, adding weight adjustment mechanisms based on agent per- formance improvements further amplifies policy learning efficiency, underscoring the importance of combining struc- tured parsing and dynamic reward weighting.
rw_3: Rewards the agent only upon successfully chop- ping an ingredient after reaching the knife, approximat- ing an optimal coordination strategy.
The results summarized in Table 3 show that while intrinsic reward methods (IRAT variants) typically enhance perfor- mance over vanilla IPPO, they still remain significantly inferior to MFHFF especially in the early stages of train- ing. This is largely due IRAT's reliance on predefined reward structures generated prior to observing actual pol- icy behaviors, resulting in suboptimal coordination patterns (e.g., multiple agents crowding the same object). In con- trast, M utilizes human feedback derived from observed rollouts, precisely identifying and targeting coordination failures, enabling agents to quickly improve policy behav- iors and achieve superior cooperative results.
Ablation 2: Single-phase vs. Multi-phase Feedback. We compared our proposed multi-phase feedback collection methodology against single-phase scenario (feedback pro- vided only once at the start of training). Results are summa- rized in Table 2. Table 2 demonstrates superior performance for our multi-phase framework over single-phase feedback, highlighting the effectiveness of iterative, incremental feed- back enabling agents' policy refinement across multiple training stages.
6. Conclusion
In this paper, we introduced M' novel framework for MARL that incorporates multi-phase human feedback of mixed quality to address the challenges of sparse or complex reward signals. By extending the Markov Game to include human input and leveraging LLMs to parse and integrate human feedback, our approach enables agents to learn more effectively. Empirically. MFHF outperforms strong base- lines, particularly in scenarios with increasing complexity. Our findings highlight the potential of integrating diverse human insights to enhance multi-agen policy learning in a more accessible way.
Ablation 3: Comparison to Intrinsic Reward Methods. We further evaluated the effectiveness of M/HF relative to intrinsic soward-based methods, specifically comparing against the IRAT method (Wang et al., 2022). As Over- cooked lacks built-in intrinsic rewards, we manually con- structed three variants of intrinsic reward functions (denoted

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Impact Statement
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, 2017.
This research introduces HFHF, a framework that enables multi-agent reinforcement learning systems learn effec- tively from human feedback of varying quality. By allowing non-expert humans to provide meaningful feedback Al systems, M M democratizes the development of multi- agent systems while making them more robust to real-world situations where perfect expert guidance may not be avail- able. This could accelerate the deployment collaborative Al systems in areas such as healthcare, manufacturing, and emergency response, where multiple agents need coordi- nate while incorporating human domain knowledge. While this increased accessibility could lead to broader adoption, we acknowledge the importance of appropriate oversight and encourage future work to explore necessary safeguards for such systems.
De Witt, C. S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P. H., Sun, M., and Whiteson, S. Is indepen- dent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv: 1. 1.09533, 2020.
Du, Y., Han, L.,, Fang, M., Liu, J., Dai, T., and Tao, D. Liir: Learning individual intrinsic reward in multi-agent rein- forcement learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
Du, Y., Leibo, J. Z., Islam, U., Willis, R., and Sunehag, P. A review of cooperation in multi-agent learning. arXiv preprint arXiv:231 2023.
Acknowledgment
Durrett, R. Probubility Theory and Examples. Cambridge
University Press, 4th edition, 2010.
This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/Y003 18711, UKRI849). This work was also supported in part by NSF grant IIS-2046640 (CAREER).
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. In Advances in Neural Information Processing Sys- tems, volume 29, 2016.
Hu, H. and Sadigh, D. Language instructed reinforcement learning for human-ai coordination. In International Con- ference on Machine Learning, pp. 13584 13598. PMLR, 2023.
References
Andrychowiez, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems, vol- ume 30, 2017.
Hu, Y., Goel, K., Killiakov, V., and Yang, Y. Eigenspectrum analysis of neural networks without aspect ratio bias. In International Conference on Machine Learning. 2025.
Bertsekas, D. P. and Tsitsiklis, J. N. Neuro-Dynamic Pro- gramming. Athena Scientific, 1996.
Klissarov, M., D'Oro, P., Sodhani, S., Raileanu, R., Bacon, P.-.,, Vincent, P., Zhang, A., and Henaff, M. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:23 10.0010 2023.
Busoniu, L.., Babuska, R., and De Schutter, B. A comprehen- sive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cyberneties, Part C (Applications and Reviews), 38(2):156-172 2008
Knox, W. B. and Stone, P. Interactively shaping agents via human reinforcement: The tamer framework. In Proceed- ings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
Canese, L.,, Cardarilli, G. C.,, Di Nunzio, L., Fazzolari, R., Giardino, D., Re, M., and Spanò, S. Multi-agent reinforce- ment learning: A review of challenges and applications. In Applied Sciences, volume 11, 2021.
Kolmogorov, A. N. Sulla determinazione empirica di una legge di distribuzione. Giornale dell' IelltIstittto Italiano degli Attari, 4:83 91, 1933.
Chen, V., Gupta, A., and Marino, K. Ask your humans: Using human instructions to improve generalization in reinforcement learning. In International Conference on Learning Representattonss, 2021.
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, 2019.
Chen, X.-H., Wang, Z., Du, Y., Jiang. S., Fang, M., Yu, Y., and Wang, J. Policy learning from tutorial books via understanding, rehearsing and introspecting. In Advances in Neural Information Processing Systems, volume 37, 2024.
Kurach, K., Raichuk, A., Stańczyk, P., Zajac, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., et al. Google research football: A novel reinforcement learning environment. In Proceedings of
10

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
the AAAI conference on artificial intelligence, volume 34, pp. 4501-4510, 2020.
Nguyen, D. T., Kumar, A., and Lau, H. C. Credit assign- ment for collective multiagent rl with global rewards. In Advances in Neural Information Processing Systems, volume 31, 2018.
Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models. In The Eleventh Interna- tional Conference on Learning Reprexentations, 2023.
OpenAL. GPT-4o //platf i com/models/
(gpt-40-0024111220;
https:
Lee, K. Smith, L.,, and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Interna- tional Conference on Machine Learning, 2021.
gpt- 2024-11221111 11:44-112200 2024.
Large language
model.
Oroojlooy, A. and Hajinezhad, D. review of coopera- tive multi-agent deep reinforcement learning. Applied Intelligence, 53(11):13677-11322 2023.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint ar Xiv:20055006643 2020.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, 2022.
Liang. J., Huang. W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation, 2023.
Pathak, D., Agrawal, P., Efros, A., and Darrell, T. Curiosity-driver exploration by self-supervised predic- tion. In International Conference on Machine Learning, 2017.
Liang, J., Xia, F., Yu, W., Zeng, A., Arenas, M. G., Attarian, M., Bauza, M., Bennice, M., Bewley, A., Dostmohamed, A., al. Learning to learn faster from human feedback with language model predictive control. arXiv preprint 2rXiv:2402.11450 2024.
Puterman, M. L. Markov decision processes. Handbooks in Operations Research and Management Science, 2:331- 434, 1990.
Littman, M. I.. Markov games as framework for multi- agent reinforcement learning. In Machine learning pro- ceedings 1994, pp. 157-163. Elsevier, 1994.
Reid, M., Savinov, N., Teplyashin, D.,, Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Liu, R., Bai, F., Du, Y., and Yang, Y. Meta-reward-nett Implicitly differentiable reward learning for preference- based reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, 2022.
Shani, L.,, Rosenberg, A., Cassel, A., Lang, O., Calandriello,
D., Zipori, A., Noga, H., Keller, O., Piot, B., Szpektor, I., Hassidim, A., Matias, Y., and Munos, R. Multi-turn reinforcement learning from preference human feedback. In Advances in Neural Information Processing Systems, volume 37, 2024.
Liu, Z. Hu, Y., Pang, T., Zhou, Y.. Ren, P., and Yang, Y. Model balancing helps low-data training and fine- tuning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.N.. (eds.), Empirical Methods in Natural Language Process- ing, 2024.
Singh, S., Lewis, R. L., and Barto, A. G. Where do re- wards come from. In Conference of the Cognitive Science Society, 2009.
Ma. Y. Liang, W., Wang. G., Huang. D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. Eureka: Human-level reward design via coding large lan- guage models. In The Twelfth International Conference on Learning Representations, 2024.
Song, Y., Jiang, H., Tian, Z., Zhang, H., Zhang, Y., Zhu, J., Dai, Z., Zhang, W., and Wang, An empirical study on google research football multi-agent scenarios. Machine Intelligence Research, 21(3):549- 570, 2024.
Mannion, P., Devlin, S., Duggan, J., and Howley, E. Reward shaping for knowledge-basse based multi-objective multi-agent reinforcement learning. In The Knowledge Engineering Review, volume 33, 2018
Sumers, T., Hawkins, R., Ho, M. K., Griffiths, T., and Hadfeld-Memell D. How to talk so ai will learn: Instruc- tions, descriptions, and autonomy. In Advances in Neural Information Processing Systems, volume 35, 2022.
Ng, A. Y., Russell, S., et al. Algorithms for inverse reinforce- ment learning. In International Conference on Machine Learning, 2000.
Sutton, R. S. Reinforcement learning: An introduction. Bradford Book, 2018

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Wang, L.., Zhang, Y., Hu, Y., Wang, W., Zhang, C.,, Gao, Y., Hao, J., Lv, T., and Fan, C. Individual reward assisted multi-agent reinforcement learning. In International Con- ference on Machine Learning, 2022.
Zhi-Xuan, T., Ying, L., Mansinghka, V., and Tenenbaum, J. B. Pragmatic instruction following and goal assis- tance via cooperative language-guided inverse planning. In International Conference on Autonomous Agents and Multiagent Systems, 2024.
Wang. Y., Krotov, D., Hu, Y., Gao, Y., Zhou, W., McAuley, J.. Gutfreund, D.. Feris, R., and He, Z. M+: Extend- ing memoryllm with scalable long-term memory. In The Twelfth International Conference on Learning Represen- tations, 2025.
Zhou, M., Liu, Z.,, Sui, P., Li, Y., and Chung, Y. Y. Learning implicit credit assignment for cooperative multi-agent re- inforcement learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
Wang, Z., Fang, M., Tomilin, T., Fang, F., and Du, Y. Safe multi-agent reinforcement learning with natural language constraints. arXiv preprint #xXiv:2405220115 2024.
Zou, H. P. Huang, W.-C., Wu, Y., Chen, Y., Miao, C., Nguyen, H., Zhou, Y., Zhang, W., Fang, L., He, L., et al. A survey on large language model based human-agent systems. arXiv preprint 250v:25555.005:: 2025.
Wu, S. A., Wang, R. E., Evans, J. A., Tenenbaum, B., Parkes, D. C., and Kleiman- Nleiman-Weiner, M. Too many cooks: Bayesian inference for coordinating multi-agent collabo- ration. In Topics in Cognitive Science, volume 13, 2021.
Xiao, Y., Tan, W., and Amato, C. Asynchronous actor- critic for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, 2022.
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of ppo in cooperative multi-agent games. In Advances in Neural Information Processing Systems, volume 35, 2022.
Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Are- nas, M. G., Chiang, H.-T. L., Erez, T., Hasenclever, L., Humplik, J., et al. Language to rewards for robotic skill synthesis. In Conference on Robot Learning, 2023.
Yuan. L., Gao, X., Zheng, Z., Edmonds, M., Wu, Y. N., Rossano, F,, Lu, H., Zhu, Y., and Zhu, S.-C. In situ bidirectional human-robot value alignment. In Science robotics, volume 7, 2022.
Zhang, K., Yang. Z., and Başar, T. Multi-agent reinforce- ment learning: A selective overview of theories and al- gorithms. Handbook of Reinforcement Learning and Control, pp. 321-384, Springer, 2021.
Zhang, N., Wang, X., Cui, Q., Zhou, R., Kakade, S. M., and Du, S. S. Multi-agent reinforcement learning from hu- man feedback: Data coverage and algorithmic techniques. arXiv preprint arXiv:2409.00717 2024.
Zhang, T., Xu, H., Wang, X., Wu, Y., Keutzer, K., Gon- zalez, J. E., and Tian, Y. Multi-agent collaboration via reward attribution decomposition. arXiv preprint arXiv:2010.08531, 2020.
Zhang, X., Guo, Y., Stepputtis. S., Sycara, K., and Camp- bell, J. Understanding your agent: Leveraging large language models for behavior explanation. arXiv preprint arXiv:2311. 18062, 2023.
12

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
A. Assumptions
and
We make the following assumptions to facilitate the analy- sis:
j-1
J(TK)- - ((To)
2
Σ Ari. - 6, for all <
Assumption A.1 (Bounded Rewards), All reward functions, including the original reward R the true human feedback reward R. and the noisy human feedback reward Raíg. are uniformly bounded:
1=1
where ò is small bounded positive constant defined explic- itly later.
The proof proceeds in three steps:
[Rss a)) S Rmax.
Vs, a.
(18)
Step 1: Performance at positive improvement milestones
This assumption is standard in reinforcement learning to ensure stability and convergence (Sutton, 2018).
(A- = iz-1). By definition of iy, we clearly have:
Assumption A.2 (Learning Algorithm Convergence). Given a fixed reward function, the learning algorithm con- verges to a policy that is e-optimal with respect the ex- pected cumulative reward under that reward function:
J 1T ( = Σ (J(m)) - ((H
Ar
1=1
Step 2: Performance between positive milestones (1z-1 <
k < iz). For indices between ij-1 and i,, human feed- back results in negative or zero performance improvement (Ark I 0). According to algorithm and Eq. (15), the weight assigned to any unhelpful feedback reward func- tion is adjusted (reduced), and eventually clipped to zero. Specifically, whenever this negative improvement occurs, the adaptive weighting mechanism significantly reduces or eliminates the influence of this poor-quality feedback. Thus, the previously established optimal policy at timestep i5-1 remains largely unchanged. The largest possible degrada- tion from the previous gegh-perforancc point is therefore bounded by constant 8 > 0, independent of the total interaction length K:
lim ([))(=))) 2 (,8()() t0000
(19)
where (T)) is the expected cumulative reward for agent under policy ı and reward function R, and n is the optimal policy for agent i under R.
Assumption A.3 (Performance Estimation Accuracy). The estimate of the performance difference Art. J ('((T)) accurately reflects the true change in expected cu- mulative reward under the original reward function R between generations and + 1.
This assumption relies on having sufficient samples to esti- mate the performance difference accurately, which can be ensured through appropriate exploration and sample size.
J(Tx) - J(Ti))))) 5, M
Assumption A.4 (Independent Roll-outs). The X rollout trajectories {T(=))x J==1 collected under the fixed joint policy Ty are independent and identically distributed (i.i.d.).
Step 3: Combining two results. Combining the two pre- ceding steps directly, we establish the performance lower bound at any time step k within the interval:
B. Proof of Proposition 4.2
In this section, we provide a complete and structured proof for Proposition 4.2.
J(TK) - -
= (J(TL) - [(. (J(T
Let iy denote the generation index at which the incorpora- tion of human feedback explicitly yields positive policy improvement for the j-th time, and let r((K) represent the number of times positive improvements (helpful human feedback) occur until timestep K. Formally, each i, sat- isfies Ar:, > 0, and indices between i--1 and iy denote interactions with low-quality or unhelpful feedback where Ark- < O.
j-1
- +
Aru.
Since this holds for all intervals of indices, applying it specif- ically to the final interval (after the last helpful feedback index n( (()), we obtain the desired inequality:
n(K)
The proposition explicitly states that for any generation & in the interval 1j < k < ij. the performance satisfies:
J(TK) - J(mo) 2
Ari Ti
5.
j=1
J(TT)) - ((TO)- = Σ Aras for k = 1--1,
Thus, Proposition 4.2 follows directly.
l=1
13

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Explicit bound the constant 8. We next explicitly quantify the bounded constant ō appearing due to step of degraded performance. Consider the following general lemma:
C. Environment Details
In this section, we will introduce the details of the environ- ments we are using. We follow the setting from
Lemma B.1. Refer to Puterman (1990) and Lemma 6.2 in Bertsekas & Tsitsiklis (1996), Let Ti, 12,73 va be three bounded reward functions. Define combined rewards:
Goal. Three agents need to learn cooperating with each other to prepare a Tomato-Letttuc-Onnio salad and deliver it to the 'star' counter cell as soon as possible. The challenge is that the recipe of making tomato-lettuce-onion salad is unknown to agents. Agents have to learn the correct procedure terms of picking up raw vegetables, chopping, and merging in a plate before delivering.
(1 p) - R' =1 (L='--''-))++p' +
Let v. and nO be the optimal policies corresponding respec- tively to R and R. Then the following performance bound holds, for discount factor y E (0, 1):
State Space. The environment is 7x7 grid world involving three agents, one tomato, one lettuce, one onion, two plates, two cutting boards and one delivery cell. The global state information consists of the positions of each agent and above items, and the status of each vegetable: chopped, unchopped, or the progress under chopping.
VØ - -V I 1-7
|| RR-B R- loo
2 1 - [I/ + pl || ri r3ll0 lp- r211 ra [lo]
We now explicitly apply this lemma in our scenario. Con- sider the timestep & immediately after the last beneficial feedback index iy. with:
Primitive-Action Space. Each agent has five primitive- actions: up, down, left, right and stay. Agents can move around and achieve picking, placing, chopping and delivering by standing next to the corresponding cell and moving against (e.g., in Figure 2a, the pink agent can move right and then move up to pick up the tomato).
ri = Rori, the original environment reward.
r2 = weighted combination of previously beneficial feedback.
r3 = Rg. the current negative feedback reward.
Macro-Action Space. Here, we first describe the main func- tion of each macro-action and then list the corresponding termination conditions.
Then, we have:
Five one-step macro-actions that are the same as the primitive ones;
2 1-g
V
S
|| - q - Rello + 1o- //1l/r Rolloo]
(20)
Chop, cuts raw vegetable into pieces (taking three time steps) when the agent stands next to cutting board and an unchopped vegetable on the board, otherwise does nothing; and it terminates when:
By the current weight updating scheme (Eqs. 15), plicitly have:
1/(2+1) = 1 and 1+1/(2+)) 3+j
The vegetable on the cutting board has been chopped into pieces;
0 pーpp-p p' < p a = p(1 a²).
The agent is not next to cutting board;
Noting that the feedback probability p is bounded (c.g. 0 5 I 1) and that all reward functions satisfy ||r]|so < Rmax.
There is no unchopped vegetable on the cutting board;
we obtain uniform upper bound for 5. In particular,
The agent holds something in hand.
2 6 - 1-,11at, (1 (1-oeh)) bel Rea - Rellos + p(1 ax) ||re Ralboo
Get-Lettuce, Get-Tomato, and Get-Onion, navigate the agent the latest observed position of the vegetable, and pick the vegetable up if it is there; otherwise, the agent moves to check the initial position of the veg- etable. The corresponding termination conditions are listed below:
(21)
Since p S and |uri Rellso, ||| Rx Rmaxx the right-hand side is a finite constant independent of the total iteration count K. This uniform bound on 5 highlights the robustness of our -intgratintttgaation mechanism even when occasional puor-quality feedback occurs.
The agent successfully picks up a chopped or un- chopped vegetable;
This completes the full proof.
14

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
The agent observes the target vegetable is held by another agent or itself;
The agent attempts enter the same cell with other agent, but has lower priority than another agent.
The agent is holding something else hand;
The agent's path to the vegetable is blocked by another agent;
Observation Space: The macro-observation space for each agent is the same the primitive observation space. Agents are only allowed to observe the positions and status of the entities within a x view centered on the agent. The initial position of the items are known to agents.
The agent does not find the vegetable either the latest observed location or the initial location;
The agent attempts enter the same cell with an- other agent, but has a lower priority than another agent.
Dynamics: The transition in this task is deterministic. If an agent delivers any wrong item, the item will be reset to its initial position. From the low-level perspective, to chop vegetable into pieces on a cutting board, the agent needs to stand next to the cutting board and executes left three times. Only the chopped vegetable can put on plate.
Get-Plate-1/2, navigates the agent to the latest ob- served position of the plate, and picks the vegetable up if it is there; otherwise, the agent moves to check the initial position of the vegetable. The corresponding termination conditions are listed below:
The agent successfully picks up a plate;
-
The agent observes the target plate is held by an- other agent or itself;
Original Reward Function:
+10 for chopping vegetable,
+200 terminal reward for delivering correct salad (like tomato-lettuce-onion or tomato-lettuce salad), -5 for delivering any wrong entity, and -0.1 for every timestep.
The agent is holding something else in hand;
The agent's path to the plate blocked by another agent;
The agent does not find the plate either at the latest observed location or at initial location;
Episode Termination: Each episode terminates either when agents successfully deliver a tomato-lettuce-nnion salad reaching the maximal time steps, 200.
The agent attempts to enter the same cell with another agent but has a lower priority than another agent.
D. Implementation Details
navigates the agent to the corre- sponding cutting board with the following termination conditions:
D.1. Algorithm
In here, we list the complete algorithm, as shown in Algo- rithm.1.
The agent stops in front of the corresponding cut- ting board, and places in-hand item on it if the cutting board is not occupied;
D.2. Predefined Reward Function Templates
If any other agent is using the target cutting board, the agent stops next to the teammate;
To effectively incorporate human feedback into the learning process, we define a set predefined reward function tem- plates F that can be parameterized based on the feedback and entities present in the environment. These templates capture common interaction patterss between agents and their environment, facilitating automatic reward function generation aligned with human intentions.
The agent attempts enter the same cell with another agent but has lower priority than another agent.
Go-Counter (only available in Overcook- B. Figure 2 b), navigates the agent to the center cell in the middle of the map when the cell is not occupied, otherwise it moves to an adjacent cell. If the agent is holding object the object will be placed. If an object is in the cell, the object will be picked up.
Firstly, the distance-based reward function penalizes the agent proportionally to the Euclidean distance between two entities 61 and ez within the environment:
Deliver, navigates the agent to the 'star' cell for deliv- ering with several possible termination conditions:
a,, = - [[s[ee-pos]] s(e2-pos]ll. (22)
The agent places the in-hand item on the cell if is holding any item;
where sle; denotes the position vector of entity in state s, and || represents the Euclidean norm.
- any other agent is standing front of the 'star' cell, the agent stops next to the teammate;
Secondly, the action-based reward function provides
15

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Algorithm M HFF Multi-agent Reinforcement Learning
reward when the agent performs specific desired action Odesired:
from Multi-phase Human Feedback of Mixed Quality
Require: Number of agents N, Original Reward Functions { [ /N Predefined Reward Templates F, Environ- ment E, Initial Policies {T Ji=1. Total Generations
faction (S, Adesired) = I[a = adesited):
(23)
K
where a is the action taken by the agent, and I[-) is the indicator function, returning the condition is true and otherwise.
Ensure: Trained Policies { J
J==1
1: Initialize Reward Function Pools P; (TQq)) for each
agent
Thirdly, the status-based reward function rewards the agent when an entity attains a particular desired status statuSdesired:
2: for generation à = to - do
3: Multi-agent Training Phase
Eg..
4: for each agent do
5: Train policy zi,k using current reward function R a) (Eq. 12)
Istatus (8, a, e, statuStesired) = [tsesstaatus] = statuS
6: end for
(24)
7:
Rollout Generation
Sec. 4.1
where s(e.status] represents the current status of entity in state
8: if Periodic evaluation or performance stagnation
detected then Generate rollout (Se, at, !=) end if
trajectories
Tk
Additionally, we define composite reward function that allows for more nuanced feedback by combining multiple reward components:
10:
11:
Human Feedback Phase
Sec. 4.2
12: Human observes ти and provides feedback uy
Scump (s 0)= Enfi(s.a).
(25)
Feedback Parsing:
Use LLM M to parse us and assign feedback to
agents:
where fi(s, a) are individual reward components (e.g.. faia, Jstutss, and X are weighting coefficients that deter- mine the relative importance of each component.
ugg uk M(ug, N)
15: Reward Function Update
Sec. 4.3
16:
for each agent do Generate new reward function from feedback
For instance, given the human feedback "Agent 1 needs to get the onion," we might select distance-based reward template and parameterize as:
17:
(Eq. 10):
Ra,new = MM{,, u us
ue e)
18:
Add new to reward function pool P:
end for
Ra(8,a) =-l|s[Agentl pos] s(Oniionpos]]l (26)
20: Weight Update:
21:
for each agent do
22:
Initialize weight for new reward function:
This reward function encourages Agent to minimize its distance to the onion, thus aligning its behavior with the desired objective.
W..M [PT
23:
Apply weight decay to existing weights
(Eq.
15):
Furthermore, other templates can be incorporated depending on the environmental context and task requirements. For example, proximity-based reward function provides a reward when an agent within a certain distance of target entity:
Wi.m = Wi.m M-m, Vm E (1,..., M -13 Normalize weights:
..M,,
24:
Wi.o - ".. Vm E fl,..., M}
Compute performance difference:
Ar:
orj iik+1 on
26:
Adjust weight of newest reward function: if Ar, > max(O, Wi,M- B), otherwise
W.,M + B,
Tprox: if |fsee. s|s)|pos]]|2 < d, otherwise,
Wi,M
Spro (8, a,, e2,
Update final reward function (Eq. 12): R (s, a) MM Wi,m (s,
(27)
where Fprox is the reward assigned for being within distance d.
28:
end for
29: end for
16

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
time-based penalty be introduced encourage effi cient task completion:
Table 4. Hyperparameters used Overcooked-A, B, and
Hyperparameter
MPHF-IPPO
Baseline IPPO
Beseline-MAPPY
Training Generation
5
fime (s, a,t) = -3
(28)
Training lierations Training Episodes
1000
1000
1000
25k
25k
25k
Learnnng Rate
0.0001
0.0003
0,0003
where the current time step, and ,3 is penalty coefficient reflecting the cost of time.
Training Batch Size
5120
5120
5120
Minihatch Size
1024
1024
1024
Epochs
10
10
10
Discount Factor (:)
0.99
0.99
0.99
suceess-based reward provides reward upon achieving specific goal condition:
GAF Lambda
0.95
0.95
0.95
Clip Parameter
0.2
0.2
0.2
Valuc Function Clip Parameter
10.0
10.0
Earropy Coefficient
0.01
0.01
0.01
KL Coefficient
0.2
0.2
Gradient Clipping
0.5
fsuccess (s, a) [[goal_condition_mel))
Tsuccess
(29)
where T'ucess is the reward value assigned when the goal condition is met.
D.4. Prompts
Prompt 1: FEEDBACK PARSING PROMPT
An ensrgy-based penalty discourages unnecessary expen- diture of resources:
Given the following feedback for multi-agent system in an Overcooked environment, as- sign the feedback to appropriate agents or to all agents. The system has {num agents} agents.
Senergy as -r energy(a),
(30)
where (a) represents the energy cost associated with action and y is scaling factor.
Feedback: {Human Feedback]
By leveraging these templates, the system can systematically generate reward functions that align with human feedback, enabling agents to adapt their behavior effectively re- sponse to diverse instructions. This approach allows for the incorporation of mixed-quality human feedback into the learning process, enhancing the agents' ability to perform complex tasks in multi-agen environments.
The agent_ 1 is the chef in Green, agent 2 is the chef in Rose, agent.3 3 is the chef in Blue.
Return your response in the following JSON format:
{
D.3. Training Details
"agent.0": "feedback for agent O",
Our experiments were conducted on a heterogeneous com- puting cluster running Ubuntu Linux. The hardware configu- ration included a variety of CPU models, such as Dual Intel Xeon E5-2650, Dual Intel Xeon E5-2680 v2, and Dual Intel Xeon E5-2690 v3. For accelerated computing, we utilized 3 NVIDIA A30 GPUs. The total computational resources comprised 180 CPU cores and 500GB of system memory.
"agent_1": "feedback for agent 1",
"all": "feedback for all agents"
}}
Only include keys for agents that receive spe- cific feedback and 'all' if there's general feed- back.
We performed hyperparameter tuning for all baselines to ensure fair comparison. Specifically, for IPPO and MAPPO, we tuned learning rates, batch sizes, and gradient clip- ping values. For example, we systematically searched over learning rates in (3e-4, lc-4), #sgd iters in (5. 10), sgd_batch. batch_size in {1024, 5 5120), and entropy coefficient in (0.01, 0.05]. ultimately selecting the configurations with the best validation performance. For the macro-action based baseline, we directly adopted the best-performing hyper- parameters reported in Mac-based method (Xiao et al., 2022). We will explicitly include these details in our re- vised manuscript.
Prompt 2: REWARD FUNCTION BUILD PROMPT
Given the parsed feedback for an agent an Overcooked environment, select and parame- terize reward function template.
17

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
The observation space 32-length vector as described the task description.
[task_name)
The environment 7x7 grid with various objects and {num agents) agents.
Parsed Feedback: (feedback for this agent)
Observation Space (32-length vector for
Observation Space (32-length vector for each
each agent):
agent):
Tomato: position (2), status (1)
Tomato: position (2), status (1) (obs[0:2))
Lettuce: position (2), status (1)
Lettuce: position (2), status (1) (obs[3:5))
Onion: position (2), status (1)
Onion: position (2), status (1) (obs(6:8)
Plate 1: position (2)
Plate 1: position (2) (obs[9:10])
Plate 2: position (2)
Plate 2: position (2) (obs[11:12])
Knife 1: position (2)
Knife 1: position (2) (obs[13:14])
Knife 2: position (2)
Knife 2: position (2) (obs(15:16))
Delivery: position (2)
Delivery: position (2) (obs[17:18))
Agent 1: position (2)
Agent 1: position (2) (obs[19:20))
Agent 2: position (2)
Agent position (2) (obs[21:22)
Agent 3: position (2)
Agent 3: position (2) (obs[23:24))
Order: one-hot encoded (7)
Order: one-hot encoded (7) (obs(25:32))
MA-V1 Actions
(index
indicates macro
Available function templates:
action):
Distance-based -sqrt((agentxx x x)
0: No operation
+ (agentLy target_y)**2)
1: Move Up
2. Action-based: reward for specific actions
2: Move Right
(e.g., chopping, picking up)
3: Move Down
3. State-based reward for achieving specific
4: Move Left
states (e.g., holding an item)
5: Interact (pick up, put down, chop)
4. Time-based: penalty for time taken
5. Combination of the above
You will be provided with a video of the agents' gameplay, which may be lengthy. Your task is to:
Select a template and parameterize it based the feedback. Return your response Python lambda function that takes the observa- tion vector (obs) and action (act) as input.
1. Identify and summarize the key actions and strategies employed by the agents throughout the gameplay.
2. Provide constructive feedback based on your observations in single paragraph. Mark this paragraph with [SUGGESTION],
For example, Distance between agent and tomato
lambda obs, act: -sqrt((bb((19) obs
When generating feedback:
(obs[20] obs[1])**2] # Distance between agent and tomato
Address specific agents by their color (e.g., Green agent, Rose agent) or position (e.g., agent on the left, agent near the cutting board). Focus on aspects of gameplay that could be significantly improved for any or all agents.
Ensure that your function uses the correct in- dices from the observation vector as described the task description.
Offer specific, actionable suggestions that can be immediately applied.
Relate your feedback to the Overcooked environment, tasks, and overall efficiency, Prioritize improvements in teamwork, task allocation, or resource management.
Prompt 3: VLM FEEDBACK PROMPT
You are an Al assistant helping to manage an Overcooked environment with multiple agents. The task is prepare and deliver a
Consider how the suggestions could impact the agents' performance metrics.
18

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Avoid:
Using overly technical jargon complex explanations.
Giving vague or general advice not specific to their gameplay.
Mentioning anything outside the scope of the Overcooked game.
Using excessive praise or encouragement.
Provide a brief summary of the agents' actions, followed by single paragraph of feed- back marked with [SUGGESTION], addressing the agents directly about their gameplay in the Overcooked environment. Focus on concrete improvements for any all agents rather than motivational language.
E. Regarding the Scalability of Our Method
We extended our evaluation Google Football 5v5 (Ku- rach et al., 2020; Song al., 2024), a complex multi-agent benchmark. M3HF continues to outperform standard MARL baselines with the multi-phased human feedback. Full en- vironment details are provided in the Figure 5,6 and their captions.
19

---

Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Total Player Number
10
Training Iteration
100
Training Episodes
8000
Episode Length
3000
Training Timesteps
2.4c7
Deterministic
False
Offsides
True
End Episode on Score
False
end_episode ootoooooo_pppaayyyoooooaaaaapppa
False
lundeepisoee.o.pgg www.oooosossssoocooooioooo
False
Bot Level
1.0
(a) Rollout Screenshot
(b) Start Point
(c) 5-vs-5 full-game (5v5) configuration
Figure 5. Google Research Foothall Environment (5-vs-5 Full Game): We evaluate M3HF in the GRF 5-vs-5 HARD Built-in AT scenario, complex multi-agent benchmark widely used in prior work (Kurach et al., 2020; Song et al., 2024). Each team controls players (10 agents total); always control the yellow-shirt (left) team, which initiates the kickoff. Each episode lasts 3000 steps, with the second half beginning at step 1501. The simulation is accelerated: in-game minute equals real-world seconds, so a full 90-minute match takes 4.5 minutes. For human feedback collection, typically present the first attacking phase (60s rollout), corresponding 20 in-game minutes. (a) shows a rollout snapshot during the opening phase. (b) depicts the initial player formation around the center circle. (c) presents the full environment configuration. Observations follow dictionary structure with keys "obs" (state features) and "controo roll 1 ed pl ayer index (agent IDs: 00- for yellow team, 5-9 for blue). The action space is [Discrete (1 (19) 1, where each agent selects one of 19 discrete actions (e.g., pass, sprint, tackle) represented one-hot 19D vector.
Performance Metrics Comparison in Google Research Football
(Baseline)
MAPPO
Gen
Gen
Good Shot
Interception
Score
8.5
1.4
0.8
1.2
8.0
1.0
7.5
2.6
7.0
0.6
0.4
0.4
0.2
0.2
0.0
Geneo
Gen
Gen
Gen
Gen
Gen
Gen
Geh
Gen
Gen
20
40
en
80
150
20
40
60
B0
106
20
45
60
80
100
Iteration
Iteration
Iteration
Total Move
Total Possession
Win Rate
900
Value
Valuu
700
0.4
600
502
Gen
Gen
Gen
Gen
400 Gen
Gen
Gen
Gen
Gen
Gen
20
40
60
80
150
20
40
66
80
136
20
40
60
80
100
Iteration
Iteration
Iteration
Figure 6. Performance Comparison of HH setting. challenging auett benchmark; Baselines include IPPO, MAPPO, and macro-action- based MACCS; Metrics include Win Rate, Goal Difference, Total Move (indicating spatial coordination). Good Shot (quality shot attempts), and Interception (defensive awareness); MPHF outperforms all baselines in most metrics, particularly in Win Rate and coordination-related metries like Total Move and Good Shot; this is due to multi- phase human feedback that encourages off-ball movement, proactive attacking, and cooperative behavior (c.g., "agents should run into space"); these behaviors result in better scoring opportunities and team coordination; Conclusion: MHHF scales effectively to GRF and enables efficient, high-quality policy learning with minimal human input.
HF vs. Baselines in GRF 5-vs-5 Full Game. We evaluate MPHF in the GRF 5v5-HARD
20


Reviews:
[{"review_id": "rWBV4IJWpd", "reviewer": "Reviewer_6vBu", "summary": "This paper introduces M3HF, a framework for integrating multi-phase human feedback of varying quality into multi-agent reinforcement learning (MARL). The authors propose a Multi-phase Human Feedback Markov Game (MHF-MG) that extends standard Markov Games to incorporate iterative human guidance. The framework uses large language models (LLMs) to parse human feedback, converts it into structured reward functions through predefined templates, and employs adaptive weight adjustment mechanisms to handle feedback of mixed quality. The authors provide theoretical analysis justifying their approach and demonstrate empirical results in the Overcooked environment, showing that M3HF outperforms several baseline methods in complex coordination tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reply to Reviewer 6vBu\n\nWe sincerely thank the reviewer for your positive remarks on our framework’s structure, scalability, and consistent performance. We hope the following responses address your concerns.\n\n---\n### 1. Regarding the Novelty of Our Work\n\nWhile existing work on LLM-based reward design mainly focuses on **single-phase** and **single-agent** settings, our core novelty lies in jointly addressing the challenges of **multi-phase** human feedback and **multi-agent** reinforcement learning.\n\nThe challenge in **multi-phase** feedback lies in integrating guidance of varying focus and intent across stages. For example, initial feedback may reflect general observations about teamwork, while later rounds provide more concrete corrections based on improved understanding of agent behavior. The **multi-agent** challenge involves assigning human feedback—whether agent-specific or team-level—to appropriate reward functions. For instance, when a user suggests “place ingredients in the center desk” in Overcooked, our method can propagate this suggestion to all relevant agents by shaping their respective reward functions accordingly.\n\nOur novelty was also noted by other reviewers: “human feedback within a MARL training loop” (Reviewer sgwG) and “an innovative approach to sparse rewards” (Reviewer PGeQ).\n\n\n---\n### 2. Regarding the Comparison with Other Methods\n\nThe methods mentioned in your review—PbMARL and MOTIF—differ significantly from M3HF in both setting.\n\n* **PbMARL** (renamed from “MARLHF”) follows a different offline setting, using only pairwise preference comparisons from a pre-collected dataset generated by a simulated policy. In contrast, M3HF collects natural language feedback dynamically during training and supports diverse reward templates, enabling more flexible and scalable reward shaping.\n* **MOTIF** does not use any human feedback. It is designed for single-phase, single-agent settings and relies on LLM-generated intrinsic rewards. As such, it does not address challenges in multi-phase human-in-the-loop learning or multi-agent coordination.\n\nTo the best of our knowledge, no prior work addresses human feedback in a multi-phase, multi-agent setting, leaving no directly comparable baselines. We therefore compare against strong backbones (MAPPO/IPPO) and SOTA coordination-focused methods (Mac-based). Additionally, Tab.1 includes controlled variants with modified feedback settings to further validate our framework’s effectiveness.\n\n---\n### 3. Regarding Practical Applicability\n\nRegarding **practical efficiency**, M3HF achieves strong sample efficiency with minimal human input. For example, in Overcooked-B: L-T-S, it reaches high performance within \\~15k eps (\\~600 iters, \\~2–3 h of training) using just 2 rounds of feedback—whereas MAPPO/IPPO typically require 80k–100k eps (\\~4k iters, \\~12–18 h), and macro-action methods need 50k–75k eps (\\~3k–4k iters, \\~9–15 h). This represents a 3×–6× speedup. Empirically, each feedback query—including watching rollouts and composing responses—took about 3–5 mins, keeping the total human effort under 25 mins while saving several hours of training time.\n\nRegarding **feedback quality**, explicitly quantifying it is non-trivial—if a ground-truth metric existed, it could replace human input entirely. Instead, we use adaptive weighting to handle feedback variability, and we show through experiments (in Fig.4c and App. D) that our method remains robust even when some feedback is noisy or incorrect.\n\n---\n### 4. Regarding the Scalability of Our Method\n\nWe extended our evaluation to Google Football 5v5, a complex multi-agent benchmark. M3HF continues to outperform standard MARL baselines with the multi-phased human feedback. Full environment details are provided in the figure caption.\n\n[Anonymous Link: For the Football Env Results](https://drive.google.com/file/d/1wdKthshHkqb7h5u9ko9q7X-8Qw1ubhw-/view?usp=sharing)\n\n---\n### 5. Regarding the Robust Analysis of Weight Adjustment\nWe provide an empirical robustness analysis of the weight adjustment strategy in Sec.4.4, Sec.5 (Question 2), and App.D in our paper.\n\nSpecifically, Fig.4c shows that M3HF maintains strong performance even under deliberately misleading feedback. Tab.1 compares M3HF with variants lacking weight adjustment or using fixed weights, confirming the advantage of our adaptive strategy. Tab.2 evaluates robustness under correct, partially incorrect, and fully misleading feedback, demonstrating resilience to varying feedback quality.\n\n---\n### 6. Q&A\n> **Q1: How many training iterations and feedback rounds are needed?**\n\nEmpirically, M3HF saves 10–15 hours of training with at most 5 rounds of human feedback (1k iters/25k eps).\n\n> **Q2: Is human feedback used on-policy or off-policy?** \n\nFeedback is collected on-policy, directly influencing the next learning phase.\n\n> **Q3:** Comparisons with other methods\n\nPlease check Sec.2 of this rebuttal.\n> **Q4:** Scalability of M3HF\n\nPlease check Sec.4 of this rebuttal.", "claims_and_evidence": "The paper claims that M3HF significantly outperforms state-of-the-art MARL methods by leveraging human feedback across multiple training phases. The evidence provided includes performance comparisons in the Overcooked environment with varying layouts and recipe complexities. The results do show consistent improvements over the baselines (IPPO, MAPPO, and macro-action-based methods).\n\nHowever, the evidence is somewhat limited in scope. The experiments are confined to a single environment (Overcooked) with variations, and the performance gains could be attributed to the additional information provided by human feedback rather than the specific mechanisms of the M3HF framework. The ablation studies help address some of these concerns by showing the value of LLM parsing and weight adjustment, but more diverse environments would strengthen the claims.", "methods_and_evaluation_criteria": "The proposed methods are generally sound for the problem at hand. Using LLMs to parse human feedback and convert it into structured reward functions is a reasonable approach, and the weight adjustment mechanism to handle mixed-quality feedback is well-motivated.\n\nThe evaluation criteria focus on the average episode return in the Overcooked environment, which is appropriate for measuring task performance. The authors compare against relevant baselines and include ablation studies to isolate the effects of different framework components.\n\nHowever, the paper lacks clear metrics for evaluating the quality of human feedback and how effectively it is incorporated into the learning process. Additionally, there is limited discussion of the computational overhead introduced by the LLM parsing and reward function generation.", "theoretical_claims": "The paper presents two theoretical propositions: one justifying the use of rollout-based performance estimates and another analyzing the framework's robustness to noisy human feedback. The proofs appear sound, though they rely on several assumptions that may not always hold in practice, such as the ergodicity of the Markov chain induced by the policy and the zero-mean nature of the noise in human feedback.", "experimental_designs_or_analyses": "The experimental design is generally sound, with appropriate baselines and ablation studies.", "relation_to_broader_scientific_literature": "The paper positions itself within the literature on MARL, reward design, and learning from human feedback. While the authors provide a reasonable overview of related work, the novelty of their contribution is somewhat limited. Personally speaking, using LLMs for automated reward design based on human feedback has become quite common in recent literature.", "essential_references_not_discussed": "A significant omission is the lack of comparison with other methods that incorporate human feedback into MARL, such as \"Multi-Agent Reinforcement Learning from Human Feedback.\" This work addresses a similar problem space, and a direct comparison would provide valuable context for understanding the unique contributions of M3HF.\n\nGiven that human feedback provides additional information not available to the baseline methods, it would be important to compare against methods that also leverage this type of input to isolate the specific benefits of the M3HF framework.", "other_strengths_and_weaknesses": "* The paper presents a well-structured framework (M3HF) that integrates human feedback into multi-agent reinforcement learning in a systematic way, addressing the challenge of reward design in complex MARL environments.\n* The authors provide a theoretical analysis of their approach, particularly regarding the robustness to noisy human feedback, which adds credibility to their method and helps explain why their approach works.\n* The experimental results demonstrate consistent performance improvements across different environments with increasing complexity, showing the scalability and effectiveness of the proposed method in challenging coordination tasks.", "other_comments_or_suggestions": "The paper has several limitations that significantly impact its contribution:\n\n1. The innovation is somewhat limited. Using LLMs for automated reward design has become a common concept in recent literature. For instance, \"Motif: Intrinsic Motivation from Artificial Intelligence Feedback\" explores similar ideas. Unfortunately, the paper doesn't leverage this automation at scale, which would have been a more significant contribution. The core idea of using LLMs to parse human feedback and generate reward functions is incremental rather than transformative.\n2. The practical implementation details raise concerns about the method's applicability. I do not clearly know how many training iterations are required or how many human feedback instances are needed throughout the process. This raises questions about whether the feedback data is used in an on-policy manner or can be reused off-policy. The requirement for humans to repeatedly provide feedback during training seems cumbersome and potentially impractical for real-world applications, especially if frequent interventions are needed.\n3. The baseline comparison is inadequate. The paper fails to compare against other human-feedback-based MARL methods, such as \"Multi-Agent Reinforcement Learning from Human Feedback.\" Without such comparisons, it's difficult to assess whether the performance improvements come from the specific approach proposed or simply from the additional information provided by human feedback. Since human feedback inherently introduces external knowledge, performance improvements without proper baselines could be considered trivial.\n4. The weight adjustment mechanism, while theoretically justified, seems simplistic and potentially brittle in practice. The paper doesn't thoroughly explore how sensitive the performance is to different weight decay parameters or how the system behaves when feedback quality varies dramatically across generations. A more robust analysis of these aspects would strengthen the paper's contribution.", "questions_for_authors": "* How many training iterations and human feedback rounds are typically required to achieve the reported results? This information would help assess the practical applicability of your approach.\n* Is the human feedback data used in an on-policy manner, or can it be reused across training iterations (off-policy)? This distinction has significant implications for the amount of human involvement required.\n* Did you compare M3HF with other methods that also incorporate human feedback, such as \"Multi-Agent Reinforcement Learning from Human Feedback\"? Such comparisons would help isolate the specific contributions of your framework from the general benefits of human knowledge.\n* Have you considered the scalability of your approach to larger multi-agent systems or more diverse environments? The current evaluation is limited to the Overcooked environment with three agents.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ITUk80iWNl", "reviewer": "Reviewer_PGeQ", "summary": "The paper introduces a novel framework named M3HF (Multi-phase Human Feedback for Multi-agent Reinforcement Learning), designed to address the challenges of sparse or complex reward signals in multi-agent reinforcement learning (MARL) by incorporating multi-phase human feedback, including feedback of varying quality. The M3HF framework extends Markov games to incorporate human inputs and leverages large language models (LLMs) to parse and integrate human feedback, enabling agents to learn more effectively.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reply to Reviewer PGeQ\n\nWe thank the reviewer for acknowledging our M3HF framework as an \"innovative approach\" with \"real-world application potential\" and for recognizing its \"originality and innovation\" in addressing \"challenges of sparse or complex reward signals in multi-agent reinforcement learning.\"\n\n---\n> Question 1 and Weakness 2: \"The paper mentions that visual-language models (VLMs) currently lack specificity in critical feedback areas, offering vague suggestions like \"improve coordination\". Could you elaborate on how this limitation impacts the overall performance of M3HF and what steps might be taken to mitigate these issues in future iterations?\"\n\nOur experiments (Figure 4b-c) indicate that while VLMs can produce human-like feedback, their current lack of specificity impacts their utility in complex coordination scenarios, leading to performance gaps compared to precise human feedback. To mitigate this, future work includes: \n(1) fine-tuning VLMs on specialized multi-agent tasks, (2) enhancing their reasoning capabilities to identify and provide actionable coordination improvements, and (3) exploring hybrid systems that combine VLM-generated feedback with minimal human refinement, aiming to balance scalability and effectiveness.\n\n---\n> Weakness 1: Assumptions and Generalizability\n\nWe acknowledge that assuming Gaussian-distributed noise facilitates theoretical analysis but may not fully capture real-world complexities. However, our empirical results have shown robust performance across varied conditions, and a future research direction could be to relax this assumption under more realistic feedback distributions.", "claims_and_evidence": "The claims about overall performance and resilience to mixed-quality feedback appear well-supported by the experimental results. However, the claim regarding the effectiveness of VLMs as an alternative to human feedback might require further substantiation with more comprehensive performance comparisons. Gemini-1.5-Pro-002 was used to generate feedback based on video rollouts similar to those observed by humans in the experiments. While an example of VLM-generated feedback is mentioned, detailed comparative performance metrics between human and VLM feedback are not fully elaborated upon in the given excerpts. Therefore, while the concept is promising, the current evidence may be insufficient to conclusively support the claim that VLMs can serve as a scalable and effective alternative without additional data demonstrating comparable or superior performance metrics.", "methods_and_evaluation_criteria": "1. Complex Reward Structures: The M3HF approach aims to enhance cooperation among agents in challenging reward environments by introducing mechanisms such as performance-based weight adjustments and weight decay to mitigate the impact of low-quality feedback. This is particularly important in multi-agent systems where the complexity of interactions and potential conflicts between agents' objectives can make reward design difficult.\n2. Experimental Validation: The effectiveness of M3HF is validated through experiments conducted in a complex multi-agent environment based on the game Overcooked. In this setting, agents must learn to cooperate to prepare the correct salad and deliver it to a designated location. This scenario serves as an excellent testbed because it requires coordinated actions to achieve goals, which is a common challenge in multi-agent systems.\n3. Benchmark Comparisons: The study compares M3HF with several strong baseline methods, including MAPPO, IPPO, and macro-action-based baselines. These comparisons demonstrate M3HF's consistent superiority across different environments and recipe setups, especially its ability to quickly converge to optimal strategies in simpler recipe scenarios. This indicates that the proposed method significantly improves learning efficiency and final performance.\n4. Exploration of VLMs as an Alternative to Human Feedback: While not fully detailed, exploring Visual-Language Models (VLMs) as scalable and effective alternatives to human feedback is a promising direction, especially considering ways to reduce dependency on human input. However, this claim would benefit from further substantiation with specific data comparing the performance of feedback generated by VLMs versus human feedback.", "theoretical_claims": "I have roughly checked the proofs, and they appear to be correct with the conclusions likely valid. However, I did not meticulously verify the details of the proofs. In my opinion, the theoretical aspect is not critically important because the assumption of Gaussian distribution seems too strong and significantly deviates from reality.", "experimental_designs_or_analyses": "1. **Performance Across Different Environments**: The study evaluates M3HF in various Overcooked layouts (e.g., Layout C), demonstrating its effectiveness and adaptability. This is a robust approach as it tests the framework under different complexities, ensuring that M3HF can perform well in diverse scenarios. The comparison with backbone algorithms like IPPO shows consistent superiority, which supports the claim of M3HF's effectiveness.\n\n2. **Impact of Mixed-Quality Human Feedback**: Experiments were conducted to assess how M3HF handles feedback of varying quality. An example given is where agents exhibited suboptimal behavior due to poor coordination, yet the system maintained performance close to baseline algorithms even when receiving inaccurate or unhelpful feedback. This experiment demonstrates the robustness of M3HF against noisy feedback through its weight adjustment mechanisms. The analysis appears sound, showing that the method effectively mitigates the impact of low-quality human input.\n\n3. **VLM-Based Feedback Generation**: The potential of Vision-Language Models (VLMs) as an alternative to human feedback was explored. Gemini-1.5-Pro-002 was used to generate feedback based on video rollouts. While this concept is promising, the provided information does not fully elaborate on how VLM-generated feedback compares to human feedback in terms of performance. More detailed comparative metrics are needed to substantiate claims about the scalability and effectiveness of VLMs as a substitute for human feedback.\n\n4. **Robustness Analysis to Noisy Feedback**: Theoretical analysis and stochastic approximation theory were employed to analyze the robustness of M3HF under noisy human feedback. Proposition 4.2 suggests that zero-mean noise does not degrade the expected performance over time, supported by empirical evidence showing minimal performance drops under mixed-quality feedback conditions. This theoretical foundation adds credibility to the experimental findings but could benefit from more comprehensive empirical validation across different noise levels and types.", "relation_to_broader_scientific_literature": "1. Multi-phase Human Feedback: Earlier works such as Yuan et al. (2022) and Sumers et al. (2022) have considered the role of iterative and multi-phase human feedback in reinforcement learning, but these approaches often rely on predefined communication protocols or need human demonstrations that can be restrictive. In contrast, M3HF allows for more flexible and dynamic adjustments to the importance of feedback, accommodating varying qualities and phases of human input.\n2. Reinforcement learning from human feedback. While RLHF has been successfully applied to train Large Language Models (Ouyang et al., 2022; Shani et al.,2024), these approaches primarily focus on aligning LLM outputs with human preferences through single-turn interac-\ntions and scalar reward signals. M3HF differs by incorporating multi-phase, mixed-quality human feedback directly into the reinforcement learning loop of agents in a multi-agent environment.\n3. Language Models in Reward Design and Policy Learning. Previous works are limited in single-agent setting, while this one focuses on multi-agent setting.", "essential_references_not_discussed": "To the best of my knowledge, this paper provides a sufficiently thorough discussion of all closely related works.", "other_strengths_and_weaknesses": "### Strengths\n\n1. **Originality and Innovation**:\n   - The paper presents an innovative approach by introducing the M3HF framework, which creatively designed to address the challenges of sparse or complex reward signals in multi-agent reinforcement learning (MARL) by incorporating multi-phase human feedback, including feedback of varying quality.\n\n2. **Significance**:\n   - By addressing challenges related to sparse or complex reward functions and offering solutions that improve learning efficiency and performance, the proposed method could lead to more robust and adaptable AI systems.\n   - The exploration of Vision-Language Models (VLMs) as an alternative to human feedback introduces a scalable solution for reducing dependency on human input, which is particularly important for large-scale deployment.\n\n3. **Clarity**:\n   - The clarity of presentation is commendable, with detailed descriptions of experimental setups, comparisons with baseline methods, and theoretical underpinnings. The use of visual aids like figures and tables helps in understanding the performance metrics and outcomes.\n\n4. **Real-World Application Potential**:\n   - The application-driven nature of this ML paper is evident through its focus on practical issues such as handling mixed-quality feedback and the potential for using VLMs to reduce reliance on human input. These aspects are crucial for deploying AI systems in real-world settings where human resources may be limited.\n\n### Weaknesses\n\n1. **Assumptions and Generalizability**:\n   - While the assumption of Gaussian distribution facilitates theoretical analysis, it might limit the generalizability of the findings to real-world scenarios where data distributions can be significantly more complex and varied.\n   \n2. **Depth of Analysis on VLM Feedback**:\n   - Although the idea of using VLMs for generating feedback is promising, the depth of analysis and empirical evidence supporting this aspect is currently limited. More comprehensive comparative studies between human and VLM-generated feedback would strengthen the claims made about the scalability and effectiveness of VLMs.", "other_comments_or_suggestions": "Page 11 row 594 the equation exceeds the restriction of the length", "questions_for_authors": "The paper mentions that visual-language models (VLMs) currently lack specificity in critical feedback areas, offering vague suggestions like \"improve coordination\". Could you elaborate on how this limitation impacts the overall performance of M3HF and what steps might be taken to mitigate these issues in future iterations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fSnM82jeXn", "reviewer": "Reviewer_7Tji", "summary": "This paper addresses the challenge of designing effective reward functions in multi-agent reinforcement learning for complex, cooperative tasks with sparse or misaligned rewards. This paper proposes M³HF, a framework that integrates multi-phase human feedback of mixed quality into MARL by extending the Markov Game to include iterative human guidance. This paper provides a way of leveraging large language models to parse feedback into agent-specific reward functions using predefined templates (e.g., distance-based or action-based rewards) and adaptively weighting adjustment mechanisms that balance new feedback against prior rewards via decay and performance-based updates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reply to Reviewer 7Tji\n\nWe thank the reviewer for agreeing that our \"theoretical analysis demonstrate robustenss to noisy feedback,\" and that the experimental results demonstrate \"faster convergence and higher asymptotic performance.\"\n\n\n> Proposition 4.2 assumes zero-mean noise in feedback, but real human errors can be biased (e.g., consistently incorrect advice), which is not addressed. The theoretical analysis does not account for systematic human errors or adversarial inputs.\n\nThank you for your insightful comment. You raise an important point about biased human errors. As Reviewer sgwG also observed, our original zero-mean noise assumption was made primarily for theoretical tractability. We recognize that real human errors (e.g., consistently incorrect advice) may indeed be systematic or biased. However, our adaptive weighting mechanism inherently mitigates such systematic errors by continually reducing the influence of consistently detrimental feedback, as supported by our empirical results (Figure 4). Explicitly modeling and analyzing biased or adversarial feedback is indeed a valuable future direction, and we plan to address it in subsequent research.\n\n\n> While Table 3 lists hyperparameters, there is no indication that baselines were retuned for fairness.\n\nWe performed hyperparameter tuning for all baselines to ensure fair comparison. Specifically, for IPPO and MAPPO, we tuned learning rates, batch sizes, and gradient clipping values. For example, we systematically searched over learning rates in {3e-4, 1e-4}, #sgd_iters in {5, 10}, sgd_batch_size in {1024, 5120}, and entropy coefficient in {0.01, 0.05}, ultimately selecting the configurations with the best validation performance. For the macro-action based baseline, we directly adopted the best-performing hyperparameters reported in Mac-based method [1]. We will explicitly include these details in our revised manuscript.\n\n\n\n> Experiments use only three random seeds (e.g., in Figure 3 and Tables 1-2), which is insufficient for MARL, where high variance is common.\n\nThank you for raising this point. While three seeds are commonly used in MARL studies, and our current results already show clear and consistent improvements, we agree that additional seeds could further strengthen the statistical confidence. We are currently running more seeds, which will be included in our revisions.\n\n---\n### Reference\n\n[1] Xiao, Y et al., \"Asynchronous actor-critic for multi-agent reinforcement learning.\" NeurIPS, 2022.", "claims_and_evidence": "Proposition 4.2 assumes zero-mean noise in feedback, but real human errors can be biased (e.g., consistently incorrect advice), which is not addressed. The theoretical analysis does not account for systematic human errors or adversarial inputs.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Propositions 4.1 and 4.2 seem correct, but not checking carefully.", "experimental_designs_or_analyses": "While Table 3 lists hyperparameters, there is no indication that baselines were retuned for fairness.\n\nExperiments use only three random seeds (e.g., in Figure 3 and Tables 1-2), which is insufficient for MARL, where high variance is common.", "relation_to_broader_scientific_literature": "Related work on multi-agent reinforcement learning, reinforcement learning from human feedback, and multi-phase human feedback are provided.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Theoretical analysis claims to demonstrate robustness to noisy feedback, and the experimental results in Overcooked environments seem to show M³HF outperforms baselines (IPPO, MAPPO) by up to 50% in complex tasks, achieving faster convergence and higher asymptotic performance.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uAoG7XZJuR", "reviewer": "Reviewer_sgwG", "summary": "In this work, the authors introduce a technique for training agents in MARL settings using human feedback as a substitute for hand-designed reward functions. Specifically, their training pipeline involves (1) collecting human language feedback about a rollout video after a period of training, (2) using an LLM to design a reward function based on the feedback and reward templates, and (3) adjusting weights of reward functions to improve training. They validate their training on Overcooked and find significant improvements in final performance relative to existing MARL baselines.\n\n### Update After Rebuttal\n\nThe rebuttal clarified my original confusion between MAPPO and IPPO, and I am satisfied with the additional clarifications regarding theoretical assumptions and proofs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reply to Reviewer sgwG\nWe thank the reviewer for recognizing our pipeline as a \"novel\" contribution and for the positive comments on clarity and presentation. We address your concerns below.\n\n---\n### 1. Regarding the Framing of our work\nThanks for your thoughtful comments on our paper's scope and framing. We agree that human feedback plays a critical role in improving human-AI alignment. In our revision, we will clarify our motivations and explain how our work contributes to addressing key challenges for human-AI alignment.\n\n---\n### 2. Regarding Theoretical Assumptions\n> Proposition 4.1’s relevance:\n\nYou are correct that Overcooked is not ergodic due to episode resets and the timestep feature. We included Proposition 4.1 primarily for theoretical completeness, highlighting that rollouts can approximate policy performance under idealized conditions. We will clearly state this limitation and clarify the scope of this proposition in the revision.\n> Proposition 4.2’s noise assumption:\n\nIndeed, as you correctly observe, the reward function generated by the LLM is deterministic, making the original assumption of zero-mean random noise less reflective of our practical setup. Your suggestion—dropping the noise term and explicitly treating the human-generated reward function as potentially faulty—is more consistent with our actual implementation. This perspective strengthens our conclusions, as our adaptive weighting approach does not fundamentally rely on randomness and remains effective even in the presence of deterministic errors. We will explicitly clarify this point in the revision.\n\n---\n### 3. Baselines and Intrinsic Reward Comparisons\n>MAPPO vs. IPPO performance:\n\nIn our experiments, MAPPO employs a shared policy among agents with a centralized value function, while IPPO uses independent policies. We observed that IPPO often performs better in coordination-intensive scenarios like Overcooked, potentially due to reduced interference during training. Prior works [1,2] similarly report IPPO matching or outperforming MAPPO, even without centralized critics.  We will clarify this in the revision.\n\n> Comparison to intrinsic reward methods:\n\nOur method is orthogonal to intrinsic reward approaches, as M3HF assigns human feedback to each agent’s reward function, effectively serving as a form of tailored intrinsic reward. We also agree that it is valuable to compare intrinsic reward baselines directly; thus, we choose IRAT [3], an intrinsic reward method that combines manually defined reward functions with the original task objective. Since Overcooked lacks pre-built intrinsic rewards, we implement three manually constructed reward variants (rw_1–3) based on increasing levels of agent coordination:\n* rw_1: rewards any pickup or chop of ingredients\n* rw_2: rewards reaching the knife after picking up ingredients\n* rw_3: rewards chopping after reaching the knife and picking up.\n| Iters. |400|600|800|1000|\n|-|-|-|-|-|\n| IPPO| 19.2 ±4.5| 23.1 ± 2.7|23.2 ± 3.3|27.4 ± 4.9|\n| IRAT-rw_1|68.9 ± 10.1| 52.5 ± 11.3   | 78.2 ± 14.5    | 94.9 ± 10.7     |\n| IRAT-rw_2|1.1  ± 2.1| 9.29   ± 11.4  | 16.0  ± 8.1   |34.5 ± 14.0|\n| IRAT-rw_3|10.8  ± 9.1|17.3 ± 10.6|21.3 ± 8.7|33.8 ± 9.9|\n| M3HF| **164.8 ± 1.2**|-|-|-|\n\nThe table shows that IRAT variants perform better than vanilla IPPO, but they lag behind M3HF, especially in early training. This is because IRAT defines rewards before rollout, without observing actual policy behavior—often leading to coordination issues (e.g., agents crowding the same ingredient). In contrast, M3HF leverages post-rollout human feedback and reward assignment to precisely target behavioral failures, resulting in faster and more effective cooperation.\n\n---\n### 4. Domain knowledge in reward templates:\nThank you for highlighting this limitation. While our templates rely on domain knowledge, they are adaptable. In the football environment, we reuse categories like distance-based and action-based rewards, adjusting only to the environment’s observation and action spaces. We will clarify this in the revision.\n\n---\n### 5. Additional Clarifications and Typos\n>Equation 8 (undiscounted occupancy)\n\nThank you—we agree. We will revise Eq. (8) to include appropriate discounting weights for theoretical consistency.\n> Clarification on \"Long-term Exploration Failure\"\n\nIn implementation, this mechanism rolls back to a prior policy and requests new feedback if learning stagnates. However, this scenario did not arise in our experiments. We will clarify this in the revision.\n> Typos\n\nWe appreciate your attention to detail and will correct all mentioned typos and figure caption issues in the revision.\n\n---\n* [1] C.S. de Witt et al., \"Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?\", arXiv, 2020\n* [2] Yu C. et al., \"The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games\", ICLR, 2022\n* [3] Wang L. et al., \"Individual Reward Assisted Multi-Agent Reinforcement Learning\", ICML, 2022", "claims_and_evidence": "Claims are generally supported, though I listed specific issues with some claims in later sections of the review.", "methods_and_evaluation_criteria": "Yes, the proposed method and benchmark makes sense for this problem.", "theoretical_claims": "- Proposition 4.1 is valid but irrelevant to the game studied in this setting since the Markov Game is not ergodic. Specifically, we know Overcooked is not ergodic since the “timestep” feature changes at each step (and periodic episode resets violate aperiodicity).\n    - I generally felt that the section on “Approximation of Policy Performance via Rollouts” in Section 4.1 was unnecessary and unclear. In particular, it seems like the empirical distribution of states+actions in any single rollout trajectory won’t approximate the “true distribution” but this is easily solved by just having multiple rollouts.\n- Proposition 4.2 is based on the assumption that the noise is zero-mean for all states and actions, but this contradicts the premise that this noise is based on “misunderstanding, lack of expertise, or other factors” since the reward function generated by the LLM is deterministic. However, I think the proof still works if we drop the noise term and just state the “true” human feedback reward function is faulty.", "experimental_designs_or_analyses": "- The experimental results show that MAPPO consistently underperforms IPPO, which makes me question the hyperparameter tuning and network architectures used to generate the results. The only difference between MAPPO and IPPO (according to the original MAPPO paper) is the fact that MAPPO has centralized value function inputs, which was found to be broadly helpful. The only other possible difference is that parameter-sharing is used for one implementation but not for the other. I was not able to find details regarding the differences in MAPPO or IPPO in the main text or appendix, so this point needs to be clarified.\n- In general, there are other baselines that are simply not presented that attempt to solve the same problem of sparse extrinsic rewards (i.e. the techniques listed in the MARL section of the Related Work). Without comparing to existing baselines, it is unclear how helpful human feedback is relative to intrinsic rewards.", "relation_to_broader_scientific_literature": "The key contribution of the work is bringing human feedback (though a code-gen LLM) within a MARL training loop as a form of reward shaping. This method appears to be novel to me.", "essential_references_not_discussed": "The related works section is comprehensive.", "other_strengths_and_weaknesses": "- The paper reads very well and the diagrams are very clear. Overall, I understand the intuition behind the techniques and why it helps improve performance.\n\n- I disagree with the claim that “designing appropriate reward functions” is a fundamental and significant challenge in MARL (beyond the challenge in single-agent RL). In particular, although naive self-play often converges to suboptimal equilibria, this is often due to the dynamics of learning compatible conventions, not fundamental flaws in the reward function.\n    - One possible framing of this work is that human feedback helps align AI behavior to human conventions (though this may come at the cost of stronger self-play performance)\n- Despite the framing of the paper as a technique to help multi-agent RL, the technique has little to do with the multi-agent setting other than the fact that Overcooked was chosen as the setting of interest. A difficult single-agent setting could’ve been sufficient for demonstrating the pipeline.\n    - Again, framing this as human-AI alignment and conducting human-AI user studies would’ve resolved this concern.\n- The “reward templates” require significant domain knowledge and seem to limit the applicability of the technique to new domains.", "other_comments_or_suggestions": "- Multiple typos of word “function” in Figure 1\n- In Figure 2’s caption it seems like the descriptions of B and C are swapped?\n- The empirical occupancy measure in eq 8 is undiscounted, which means it does not approximate the discounted occupancy measure in eq 6.\n- “formal” should be “former” in line 306\n- I don’t think “the” should be in the section title of 4.4", "questions_for_authors": "- “Long-term Exploration Failure” for rollout generation is a bit unclear. It seems like the experiments presented only ask humans for feedback every 200 iterations, so how does the long-term exploration failure fit into your final method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes integrating human feedback during the training process of multi-agent reinforcement learning, and interpreting those feedbacks using LLMs so as to improve the learning outcomes of multi-agents. The reviewers agreed that the paper studies a well motivated problem, and proposes a novel solution. The reviewers pointed out ways to significantly improve the evidence backing the proposed approach (e.g. testing on other domains, testing against other baselines, making the writing clearly list experiment details) that the authors adequately addressed in their rebuttal. Including all of the details that the authors provided in their rebuttal into the draft will substantially strengthen the paper.
Decision: Accept (poster)