Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning
Authors: Peihong Yu
Manav Mishra
Alec Koppel
Carl Busart
Priya Narayan
Dinesh Manocha
Amrit Singh Bedi
Pratap Tokekar
Venue: Accepted by TMLR ()

Abstract: Multi-Agent Reinforcement Learning (MARL) algorithms face the challenge of efficient exploration due to the exponential increase in the size of the joint state-action space.  While demonstration-guided learning has proven beneficial in single-agent settings, its direct applicability to MARL is hindered by the practical difficulty of obtaining joint expert demonstrations. In this work, we introduce a novel concept of personalized expert demonstrations, tailored for each individual agent or, more broadly, each individual type of agent within a heterogeneous team. These demonstrations solely pertain to single-agent behaviors and how each agent can achieve personal goals without encompassing any cooperative elements, thus naively imitating them will not achieve cooperation due to potential conflicts. To this end, we propose an approach that selectively utilizes personalized expert demonstrations as guidance and allows agents to learn to cooperate, namely personalized expert-guided MARL (PegMARL). This algorithm utilizes two discriminators: the first provides incentives based on the alignment of individual agent behavior with demonstrations, and the second regulates incentives based on whether the behaviors lead to the desired outcome. We evaluate PegMARL using personalized demonstrations in both discrete and continuous environments. The results demonstrate that PegMARL learns near-optimal policies even when provided with suboptimal demonstrations and outperforms state-of-the-art MARL algorithms in solving coordinated tasks. We also showcase PegMARL’s capability of leveraging joint demonstrations in the StarCraft scenario and converging effectively even with demonstrations from non-co-trained policies.

Published in Transactions on Machine Learning Research (01/2025)
Beyond Joint Demonstrations: Personliieed Expert Guidance for Efficient Multi-Agent Reinforcement Learning
Peihong Yu University of Moryland
peshong@uumd.ed edu
Manav Mishra
mishma200hhisrrbbaccc
HSSR Bhopal
Alec Koppel JP Morgon Chase Co.
alec.koppe@@hppcchase.com
Carl Busart
carl. e. cisart..iv@ormm... mill
DEVCOM Army Research Laboratory
Priya Narayan
prigg..amayynnnn.cdellrmmmy
DEVCOM Army Research Laboratory
Dinesh Manocha
dmanocha@umd.coom edu
University of Maryland
Amrit Bedi University of Central Florida
amrithedi@rucf edu
Pratap Tokekar
tokekar@umdd edu
University of Maryland
Reviewed on OpenReview: https: // operreveew....... net/ forum? id- kzPNHQ8By
Abstract
Multi-Agent Reinforcement Learning (MARL) algorithms face the challenge of efficient exploration due to the exponential increase in the size of the joint statc-action space. While demoosttraloon-guiieed learning has proven beneficial in single-agent settings, its direct applicability to MARL is hindered by the practical difficulty of obtaining joint expert demon- strations. In this work, introduce novel concept of personalized expert demonstrations, tailored for each individual agent or, more broadly, each individual type of agent within heterogeneous team. These demonstrations solely pertain to single-agent behaviors and how each agent can achieve personal goals without encompassing any cooperative elements, thus naively imitating them will not achieve cooperation due to potential conflicts. To this end, we propose an approach that selectively utilizes personalized expert demonstrations as guidance and allows agents to learn cooperate, namely personalized expert-guided MARL (PegMARL). This algorithm utilizes two discriminators: the first provides incentives based on the alignment of individual agent behavior with demonstrations, and the second regulates incentives based on whether the behaviors lead to the desired outcome. We evaluate PegMARL using personalized demonstrations in both discrete and continuous environments. The experimental results demonst that PegMARL outperforms state-of-the-art MARL algorithms in solving coordinated tasks, achieving strong performance even when provided with suboptimal personalized demonstrations. We also showcase PegMARL's capability of leveraging joint demonstrations in the StarCraft scenario and converging effectively even with demonstrations from non-co-trained policies.

---

Published in Transactions on Machine Learning Research (01/2025)
1 Introduction
The use of expert demonstrations! has been proven effective in accelcrating learning in singlo-agent reinforce-
ment learning, as evidenced by studies such as Kang et al. (2018) Chen & Xu 2022 Rengarajan et al. (2022).
This approach has since been extended to Multi-Agent Reinforcement Learning (MARL) (Lee & Lee 2019
Qiu et al. 2022,, which typically assumes the availability of high-quality collaborative joint demonstrations However, from a practical standpoint, collecting joint demonstrations can be labor-intensive, demanding one per agent in cooperative scenarios. Furthermore, these demonstrations are not scalable. Tf change the number of agents or introduce new types of agents, will need to gather new set of demonstrations to learn from.
In contrast, it is much easier to obtain demonstrations for individual agents, or even better, for each type of agents in a heterogencous sctting. We thus ask the following research question: could we leverage individual- wise lask demonstrulions instend? In this work, we refer lo such expert demonstrations that address single-agent behaviors for personal objectives as personalized demonstrations (see Figure [ Since the personalized demonstrations will not necessarily reflect how the agents collaborate and may even conflict with each other in the joint setting, naively mimicking the demonstrations will not achieve cooperation. Therefore, purely imitation learning-based approaches would not be effective. We need an approach that selectively utilizes suitable personalized expert demonstrations as guidance and allows agents to learn to cooperate via collecting reward signals from the environments.
To this end, we present our algo- rithm, Personalized Expert-Guided MARL (PegMARL), which carries out personalized occupancy matching as form of guidance through reward- shaping. We implement this via two discriminators. The first, a personal- izod behavior discriminator, evaluates local state-action pairs, providing pos- itive incentives for actions that align with the demonstration and negative incentives for divergent ones. The second, a personalized transition dis- criminator, assesses whether a local state-action pair induces desired change in dynamics similar that observed in the demonstration, ad- justing the incentive weight accord- ingly. We demonstrate the elfective- ness of PegMARL on both discrete gridworld and continuous multi-agent particle environments (Lowe et al. 2017 Mordatch & Abbeel 2017 The main contributions of this paper are summarized as follows:
Existing Methods utilize Joint Demonstrations for Joint-Policy Learning
Ours utilizes Personalized Single-Agent Demonstrations for Joint-Policy Learning Agents of the some type can utilize the demonstrations
Demonstrations need be recollected if agentt configurations changes
Top demonstrotions; row: learning
Agents of different types
Gools
Figure 1: Joint demonstrations are costly to collect but offer rich information on collaborative behaviors. Personalized demonstrations are easier to collect, but solely focus on individual agent goals, so they lack cooperative elements.
(1) We propose PegMARL, the first approach that enables utilizing personalized demonstrations for policy learning in heterogencous MARL environments, which (i) avoids demonstration recollection regardless of the number and lype of agents involved, and (ii) is compatible with most MARL policy gradient methods.
(2) We demonstrate PegMARL's effectiveness with personalized demonstrations in both discrete and continuous environments. Our algorithm outperforms state-of-the-art decentralized MARL algorithms, pure multi-agent imitation learning, and reward-shaping techniques in terms of scalability and convergence speed, and achieves robust performance even when provided with suboptimal demonstrations.
In our context, the "expert. guidance" refers to demonstrations that provide meaningful, above-random performance in complex environments, even if the demonstrations necessarily optimal.
2

---

Published in Transactions on Machine Learning Research (01/2025)
Objective: The green agent opens the door, allowing the red agent to retrieve the key.
Personalized Demonstrations
Our algorithm learns cooperative policies
No cooperating behaviors are demonstroted
guided by personalized demonstrations
MAPPO failed to learn meaningful policies
for the red agent: Novigoting key locotion (the door openy
Red ugent enters the gnidtdle the dioor
agent stops away from the square red agent pass
for the green agent: Novigating to green square open the door door green agent moves away)
Green agent
due the sparseness of the environmental reward signals
green square open ond hold
agent navipates the gool after the green agent sleps
Figure 2: An example of utilizing personalized demonstrations to learn cooperative multi-agent policies. To learn successful cooperation, the agents are required not only to imitate the demonstrations to achieve personal goals but also to learn how to avoid confficts and collaborate. We visualize the state visitation frequency of the personalized demonstrations and the joint policies learned by our algorithm and MAPPO, where darker color means higher value. We observe that the demonstrations guide the agents in exploring the state space more efficiently than in MAPPO.
(3) We showcase PegMARL's capability to also leverage joint demonstrations, regardless of whether they are sampled from co-trained or non-co-trained policies Experimental results on the StarCraft environment (Samvelyan et al. 2019) demonstrate that PegMARL converges effoctively even with demonstrations from non-co-trained policies, which include potentially conflicting behaviors.
2 Related Works
Imitation Learning (IL). methods seek to replicate an expert policy from demonstration data generated by that policy. Behavior Cloning (BC) commonly employed IL technique (Pomerleau, 1991 Bojarski et al, 2016), where the expert policy is estimated through supervised learning on demonstration data. However, BC is susceptible to compounding errors resulting from distribution shifts (Ross et al., 2011 Another thread of IL research is Inverse Reinforcement Learning (IRL) (gg et al., 2000; Ziebart et al. 2008). in which the underlying reward function is estimated from the demonstration data and then used for policy learning, To alleviate the computational overhead of IRL, Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) was introduced,, allowing direct policy learning from observed data without intermediate IRL steps. Song al. (2018) extended this approach introduce Multi-Agent GAIL (MAGAIL), which adapts GAIL to high-dineensioal environments featuring multiple cooperative competing agents. In general, IL approaches can rarely perform better than demonstrations. Therefore, they not directly suitable for scenarios where only personalized expert demonstrations that do not demonstrate how to collaborate are available.
Learning from Demonstration (LfD). In contrast to IL methods, LfD aims to leverage demonstration data to facilitate learning rather than simply mimicking expert behavior. Existing LID works incorporate demonstration data into replay buffer with a prioritized replay mechanism accelerate the learning process. Due to the off-policy nature of the demonstration data, most methods are value-based (Hester et. al. 2018 Vecerik et al. 2017). There has been some recent work where the demonstrations
used to aid exploration,
especially in cnvironments with large stato-action spaces (Kang et al. 2018 Chen & Xu. 2022 Rengarajan el al. 2022). For instance, POID (Kngg et al. 2018) learns an implicit reward from the demonsttationn dala using a discriminator and incorporates it into the original sparse reward. LOGO (Rengarajan et al.) 2022) uses the demonstration data to directly guide the policy update: during each update iteration, the algorithm seeks policy that closely resembles the behavior policy within trust region. However, these approaches are primarily focused on single-agent settings.
2=Co-trained palicies" refers policies that have been trained together in the environment with shared experiences. We refer the readers Figure 3 for an illustration and Section 6.2 of Wang (2023) for more information.
3

---

Published in Transactions on Machine Learning Research (01/2025)
In multi-agent settings, Qiu et al. (2022) suggest using demonstrations pretrain agents through imitation learning as warm start, followed by optimization of the pretrained policies using standard MART. algorithms. Lee & Lee (2019) augment the experience buffer with demonstration trajectories and gradually decrease the mixing of demonstration samples during training to prevent the learned policy from being overly influenced by demonsttations. Similar to POID, DM2 (Wang et al. 2023) enables agents to enhance their task-specific rewards by training discriminators as well. Each agent matches toward a target distribution of concurrently sampled trajectories from a joint expert policy to facilitate coordination. These approaches, however, require joint demonstrations of the same team configurations sampled from cohesively trained policies. While effective, this requirement can be cumbersome and limiting, especially in real-world scenarios where such coordinated demonstrations are difficult to obtain (as mentioned in Section D
Our approach differs in that we leverage only personalized expert demonstrations to learn a cooperative policy. Notably, PegMARI. can also utilize joint demonstrations when they are available, including those from tor-coe policies. By accommodating these diverse and potentially conflicting demonstrations, PegMARL can leverage a wider range of available data, offering greater flexibility and robustness across various multi-agent applications.
3 Preliminaries
We start by considering a Markov Decision Process AS 7,r,7) for a cooperative multi-agent setting with N agents. Here, S denotes the global state across all the agents, which can be decomposed as the product of N local spaces St as S = S1 x S2 x.... x SN. By noting the local state of agent as 8 E S, the global state is == (2, ... ,,, .N)). Similarly, we define the global action space as A == A x A x ... x AN, meaning that for any a E A, we may write to s' after laking a joint action a is denoted by (()'(s,a) = II; a)). All agents share a common reward function learning and define the global policy as (o(a))), where 0 E O are the policy parameters. Specifically, we have 0 = (01, 02.... 0x) as the factorized global policy parameters, and we can write Ta(als) = II, no, (as(8) using policy factorization.
(21, ........ .ax) with a; E A₁. The transition probability from state
a)s,a), and 7 E [0, 1] is the discount factor. In this work, we focus decentralized
Personalized Tasks and MDPs. To introduce the notion of personalized demonstration, we need to extract the individual lasks of each agent from the collective tasks of multiple agents. For example, in Figure 2. the green agent's personalized task is to open the door and the red agent's 's personalized task is to reach the key without the other's presence. We then define Personalized Markov Decision Processes (PerMDPs) As, Qs, Te, 7) for each agent or, more generally, for each type of agents that share the same objective within a heterogeneous team. Here, a type refers to behaviorally identical agents who share the same state and action spaces, as well as the same reward function, using the formalism from Bettini et al. (2023), and we provide the PerMDP definition for each agent for simplicity. We assumne that the state space S, and action space A, of the personalized task T, are the same as the local state and action spaces of agent from the joint task. The transition probability from state 8; to s after taking action a; is represented as Qs(5) si,)). By following an arbitrary policy (a(ass))) for this personalized task, we can collect a set of personalized demonstrations B; [(6% at) Ho. where H is the episode horizon. We'd like to emphasize that while we assume the personalized tasks and the joint task are conducted in the same environment map, the underlying transition dynamics are different. Additionally, the joint reward r may not necessarily equal the summation of rewards ry for each personalized task.
Occupancy Measures. For a joint policy क (TI, T2, NN), we can write the global state action occupancy measure X (T(s.a)
(² Σn'
r
( TT)
P
(1)
t=
and write the corresponding local cumulative state-action occupancy measure
(7,, = Σ n" IP o = si, oj = ai π
(2)
1=0
4

---

Published in Transactions on Machine Learning Research (01/2025)
Objective: agent ond swap positions
Co-trained Joint Demonstrations
Personalized Demonstrations Agent behaviors be conflicting
Personalized demos elicit both compatible and conflicting joint strategies
Agent behaviors are compatible
Peth
Peti
Pach
Path
1
Danger
Donger
Donger
Path
Pach
Donger
2
Pashu
Peshu2
2
Pazh
Path
Figure When joint demonstrations are sampled from co-trained policies, the agents' behaviors exhibit compatibility. In contrast, personalized demonstrations solely focus how each agent achieves its individual goal and lack cooperative elements, potentially leading to conflicts.
for Va, E s; Sj An interesting observation to note here is that we can write the local occupancy measure as the marginalization the global occupancy measure with respect to all other agents. Mathematically, it holds that
as) = E Σ
(²
(3)
aEta; (a
with A-1 and =
4 MARL with Personalized Expert Demonstrations
Now, we are ready to present the main problem we are interested in solving in this work. Assume each agent is associated with personalized task T, We collect one set of expert demonstrations for each agent, or, equivalently, cach personalized task T. By letting expert user perform each personalized task in the respective personalized MDP, we obtain collection of expert demonstrations denoted by {BE, Br, Br, J. We assume that the underlying expert policy associated with Bg, is TE, (a])s;, and x is the occupancy measure following the expert's policy TE, for agent Note that while we establish the problem formulation and develop our algorithm a fully obscrvable setting, the experiments are conducted in both fully and partially observable settings, encompassing both discrete and continuous environments (details in Section
4.1 Formulating Personalized Expert-Guided Guided MARL
In standard Multi-Agent Reinforcement Learning, agents aim to discover optimal joint policies that maximize the long-term return R(T)) == (X* r) = v
A Eis(A (ATo .).. Typically, the learning process commences with raudom exploration, which is often inefficient due to MARL's exponentially growing exploration spaces, especially when rewards are sparse. Beginning with the intuition of leveraging personalized demonstrations guidance for how each agent should accomplish their personalized tasks to promote effective exploration, akin to Kang et al. (2018) for single-agent cases, demonstrations in mulli-agent settings follows:
define the objective for learning from personalized
1 N Σ (Q Dus (A || x=x)))
(4)
where 17 is weighting term balancing the long-term reward and the personalized policy similarity. The Jensen-Shannon (JS) Divergence terms enable individual agents to align their actions with their respective personalized demonstrations and facilitate the achievement of their specific objectives.
However, only imitating the personalized demonstration may not always yield favorable outcomes and can even impede the learning process. Previous works in MARL have predominantly utilized joint demonstrations guidance. As demonstrated by DM2 WWng al. (2023)), , trajectories must be sampled from co-trained joint expert policy for their joint action-matching objective to converge. The crucial aspect that jointly coordinated
5

---

Published in Transactions on Machine Learning Research (01/2025)
demonstrations offer, which personalized demonstrations lack, is compatibility. To illustrate, consider the example depicted in Figure where two agents aim to swap positions without entering the danger region. Two sets of optimal joint policies exist for this task: agent takes path 1 and agent b takes path 2, or vice versa. When sampling joint demonstrations from a set of co-trained policies, the agents' behaviors will be naturally compatible. In contrast, if personalized demonstrations were provided, agents could indifferently choose between both paths to reach their goals, potentially resulting in conflicting strategies through naive imitation.
Personalized Demonstration
Hence, we pose the following question: How cnn we leverage the benefits of using personalized demonstrations to enhance explo- ration efficiency while circumeenting the imitation of conflicting behaviors that might hinder joint-policy training?
Imitate t in multi agent Env
Succeet
Consider the scenario depicted in Figure where the blue agent seeks to replicate its personalized demonstration within multi- agent environment. The blue agent can successfully transition to the neighboring grid on the right it's unoccupied; other- wise, the transition fails. Viewing the blue agent as the focal agent and the green agent as part of the environment, success- ful imitation hinges on the agent's "local transition" matching those of the environment. where the personalized demonstra- tions originated. We define P(s[]ss, a;) as approximation for the i-th agent's local transition probability in the multi-agent environment, which is governed by the true transition dynamics P(s"|s, (ss a) and the policy Tg. With this insight, we further refine the objective function as follows:
Fail
Figure 4: motivating example illustrating the imitation of personalized demonstrations in a multi-agent environment. The primary technical challenge lies in the discrepancy between the transition dynamics in the per- sonalized MDP and the local transition dy- namics for each agent in the multi-agent environment.
max Σ (A (270. - (x AED
where Ano and E have the same definition the original x and A but with restricted domain:
domgro dom {(s E S, x Dss (P.(s[)ss,)a) || Q=(s/|s,, a.)) I c).
(6)
where ((s][s;, a:) represents the true transition dynamics of the personalized MDP from which demonstrations are collected, and is a threshold parameter that theoretically controls transition mismatch tolerance. By adjusting the learning objective in this manner, the occupancy matching only happens on those local state-action pairs that guide us toward the desired next local state under the current policy.
4.2 Solving Personalized Sxpert-Guided MARL
Now, we can begin the development of practical algorithm. By adopting Theorem 2 from Kang et al. 2018 we can substitute the JS divergence between occupancy measures with
Das (A 11 )=E.) x sup Exx, (Log(1 Di(s,, a:)) Ens, [log(Di(s,, (D((ss
where D(s;, a.) S, A, + (0, 1) is a discriminative classifier to discern if the (s,, a,) pair is from the demonstration or the current policy. However, this doesn't account for the constraint equation D Therefore, by introducing an indicator function 1(s;; a;) for whether (si, a;) belongs to the domain of 275 and ATR, we can obtain
Dus (250 || (Tm)
sup Erg [1(ss, a.) log(1 Di(si, ETR, [1((s 0:) log (s(s,,))))))))))) D.
(8)
We refer the reader to Kang 2018 for the full derivation. We use the same calculations with the change of notation Ar + To and fg E..

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm Personalized Expert-
Expert-Guided Guided MARL (PegMARL)
Input: Number of agents N; environment enc; personalizod expert trajectorics :g,..., BEN batch size M;
weight parameters {r/))
Initialize: Policies { discriminators {Do.) and & where i ..,
Output: Learned policies {TT,
1: for iteration == 0, 1, 1... do
2: Gather trajectories of multi-agent rollouts from env, B ROLLOUT(TTeRU)
3: for agent i = 0, 1,.,, 1 do
Sample M tuples of (s,, a, s() from demonstration E and B*; Update personalized behavior discriminator Dæ.:
max [EgB [log(1 De. (3t, a:))] + EBE [log (s).aa))))) 6:
(9)
6:
Update personalized transition discriminator Dø.:
(8is ai (61) llog(1 Dē (si, ais <(13] + RBBE [log (8,
max &
Ege
(10)
Estimate the reshaped reward as P DD (8,, us 8/) log(1 - Dos (8 (k)
8:
Update agent policy To,-
9: end for
10: end for
Since direct access to the transition distributions is unavailable, verifying whether each (s,, a;) pair is within the domain is not directly feasible. To address this, further approximate the indicator function using another discriminative classifier (ss,; az, s() S; x A; x S + (0, 1), estimating the likelihood of a (s1, uz 8() tuple being from the demonstrations. In essence, it quantifies the likelihood that the corresponding (st, a.) induces a desired change in local state transition as observed in the demonstrations, effectively serving as a learned proxy for the e-constrained domain in Equation D
After approximating the inner JS divergence term equation 5 with two discriminators, we integrate the discriminative rewards derived from these discriminators into the environmental reward for the outer problem. Specifically, estimate the reshaped reward A, A, = nD (st, a, s,) log(1 - D&, (st, a:)), where &; and Φ are parameters for the two discriminators. The personalized behavior discriminator D&, evaluates local state-action pairs, providing positive incentives for actions that align with the demonstration and negative incentives for divergent ones, while the personalized transition discriminator Dz, assesses a local state-action pair induces dosired transition in local state akin to that observed in the demonstration, adjusting the incentive weight accordingly. Subsequently, policy optimization is conducted by maximizing the long-term return with the reshaped reward.
Our PegMARL algorithm, detailed in Algorithm | is compatible with any policy gradient methods, with MAPPO Yu et al. (2021h) adopted our implementation. While presented in a fully observable setting, PeIMARL. can be adapted for partially observable scenarios. In such cases, we process observations from joint environment rollouts by removing dimensions not observable in the Personalized MDP for discriminators inputs. Notably, PegMARL
also utilize joint demonstrations; in this case, the observation processing step is unnecessary as both demonstrations and environment rollouts are derived from the same joint MDP, Compared to 2M (Wang et al., 2023), which also uses discriminator-base approach for multi-agent learning from demonstrationss, MARL introduces the personalized transition discriminator D as a key innovation. This additional discriminator helps address the challenge illustrated in Figure 3 by evaluating whether local stato-action pairs induce desired transitions similar to those in the demonstrations, effectively filtering out less desirable actions while retaining beneficial ones. This component enables PegMARL handle both personalized demonstrations and joint demonstrations from various sources, including those
7

---

Published in Transactions on Machine Learning Research (01/2025)
sampled from non-co-trained policies, while DM requires demonstrations from co-trained policies to achieve convergence.
5 Experiments
In this section, we empirically evaluate the performance of PegMARL, focusing on the following questions: (1) How does PegMARL, which leverages personalized demonstrations, compare to state-of-the-art MARL techniques? (2) How does PegMARL scale with an increasing number of agents and in the case of continuous state-action spaces? (3) How does the sub-optimality of personalized expert demonstrations affect the performance of PegMARL? and (4) How does PegMARL perform when trained with joint demonstrations sampled from co-trained non-co-trained expert policies?
5.1 Main Results with Personalized Demonstrations
We evaluate PegMARL with personalized demonstrations on discrete gridworld environments (Figure 5) and a continuous multi-agent particle environment (MPE, Figure Bay. The gridworld environment is fully observable with discrete state and action spaces, while the MPE is partially observable with a continuous state space and discrete action space. Both environments feature sparse reward signals, which adds to the challenge for standard MARL algorithms learn effectively.
Our comparison includes four strong baselines: MAPPO (Yu et al. 2021 )), leading decentralized MARL
algorithm;; MAGAIL (ong et al. (2018)). ), state-of-the-art multi-agent imitation learning algorithm; DM² Wang et al. 2023 which combines distribution matching reward with environmental reward; and ATA [he et al. (002 ) one of the best multi-agent reward-shaping methods. Notably, MAGAIL and DM2 were not originally designed for personalized demonstrations. Hence, we have adapted them for use in personalized demonstration settings through necessary modifications. Specifically, we processed the observations from joint environment rollouts by removing dimensions not observable in the Personalized MDP, similar our approach in PeRMARL. This adjustment ensures that the inputs to their respective discriminators the personalized demonstration data. In this context, MAGAIL serves the environmental signal and transition discriminator, while DM2 represents an ablation lacking only the transition discriminator. Details on algorithm implementation and hyperparameter choices can be found in Appendix C
compatible with ablation of PegMARL lacking
For cach scenario, we collect two sets of personalized demonstrations for each agent: optimal and suboptimal, by training single-agent. RL in simplified environments that focus on each agent's specific tasks. The average episodic rewards of suboptimal demonstrations details can be found in Appendix Bj. We execute each method in every environment with 10 distinct random initializations and plot the mean and variance across all runs. Oracle denotes the best possible return achievable with optimal policies.
approximately half of their optimal counterparts (more
agents
agents
agents
easy
hard
(b) The door scenario: the agents are heterogeneous, the assistant agent (green) must reach the green square and remain there to hold the door open, allowing the other agent (red) to reach the goal.
(a) The lava scenario: the agents are homogeneous, aiming reach corresponding diagonal positions with- out entering the lava. The episode ends any agent steps into the lava.
Figure 5: The discrete gridworkd environments. Circles indicate the starting locations of the agents, while squares of the same color denote their respective goal locations. More details be found in Appendix A.1

---

Published in Transactions on Machine Learning Research (01/2025)
PegMARL
PogMARL
DM2
DM2
MAGAIL
MAGAIL
ATA
MAPPO
Oracle
2 aagents
3 agents
agents
10
10-
Episode Rewards
Episode Rewards
Epitode Rewards
6
Number Timesteps
166
Number Timesteps
166
Number Timesteps
Figure 6: Learning curves of PegMARL versus other baseline methods under the lava scenario. PegMARL
converges to higher rewards and generalizes betler to larger numbers of agents. The star symbols (*) in the
legend indicate that suboptimal personalized demonstrations are adopted.
easy
hard
Episode Rewards
Episode Rewards
24
0
2
5
5
Number of Timesteps
1c6
Number of Timesteps
166
Figure 7: Learning curves of PegMARL versus other baseline methods under the door scenario. PegMARL shows better robustness in terms of convergence and generalizability.
How does PegMARL scale with an increasing number of agents? The lava scenario (Figure 5a contains three variations, each involving different quantities of agents and escalating levels of complexity. The agents receive a positive reward only upon reaching the goal, incur penalties for collisions, and instantly die and terminate the episodes if they step into the lava, making learming in this environment challenging. As observe from Figure D MAPPO struggles to develop meaningful behavior across all scenarios due to the sparse reward structure. MAGAIL performs worse, primarily due to the absence of environmental reward signals. The naive initation of personalized agent ramonnstrations leads to frequeni collisions among agents. While ATA can learn suboptimal policies in the 2-agent setting, its learning efficacy declines drastically as the number of agents increases. This underscores the challenges of training as the complexity of the multi-agent environment increases.
In contrast, PegMARL demonstrates superior generalizability across scenarios with more agents, maintaining stable performance despite suboptimal demonstrations. As DM2 has been adapted to accommodate person- alized demonstrations, it closely resembles PegMARL but lacks the personalized transition discriminator component. As the number of agents increases, we would expect the space of possible interactions among agents to expand, and therefore for the personal demonstrations to be potentially misleading. Consequently, 2M experiences a decrease in convergence speed as the number of agents increases. This highlights the importance the personalized transition discriminator in PegMART, enabling effective handling of inter-agent. interactions in personalized demonstration scenarios, thereby ensuring its efficacy across varying agent counts.
How does PegMARL perform under the heterogeneous setting? The door scenario (Figure 5b) contains two variants with varying levels of difficulty. The easy case is fairly straightforward. success should be achievable by adhering to personalized demonstrations, which illustrate how cach agent navigates to their respective goal locations. As shown in Figure 7 most algorithms,, except for MAGAIL, showcase proficient performance. Notably, PegMARL exhibits the swiftest convergence. We attribute MAGAIL's 's failure in this case to its inability to dircct the green agent to remain positioned at the green square a behavior not explicitly demonstrated. This underscores the importance of environmental reward signals when integrating personalized demonstrations into multi-agent learning paradigms. The hard case necessitates higher degree
9

---

Published in Transactions on Machine Learning Research (01/2025)
of agent cooperation: once the red agent gains entry to the middle room, the green agent must move aside from the green square to enable the red agent's s passage into the right room. In this complex setting, only PegMARL and DM demonstrate commendable convergence. PegMARL, equipped with the personalized transition discriminator, maintains faster convergence compared to DM
How does PegMARL perform in the continuous setting? We modified the cooperative navigation task from the multi-agent particle environment (Lowe al. (2017); Mordatch & Abbeel (2017)) to evaluate the performance of our algorithm in continuous environment (more details can be found in Appendix A.2). Figure 8b demonstrates the learning curves of PegMARL versus MAPPO and DM?. This validates the ability of PogMARL in continuous environments.
Episode Rewards
60
-80
PegMARL
DM2
-10
MAPPO
Number of Timesteps
167
(a) The modified cooperative navigation scenario: the agents need to navigate around the wall to occupy both landmarks.
(b) Learning curves of PegMARL, versus MAPPO and DM2 under the modified cooperative navigation scenario
Figure (a) The modified cooperative navigation scenario. PegMARL is effective in continuous environments.
(b) The learning curves demonstrate that
5.2 Results with Joint Demonstrations
While PegMARL is primarily designed to leverage personalized demonstrations, it can also be extended to utilize joint demonstrations. Unlike other MARL algorithms like DM² (Wang et al. (2023)) ) that require compatible joint demonstrations sampled from co-trained policies to achieve convergence, PogMARL has the flexibility to utilize joint demonstrations even from non-co-trained policies, which potentially contain conflicting behaviors. To demonstrate, we conduct comparative experiments within the StarCraft Multi-Agent. Challenge (SMAC) environment (Samvelyan al. (2019)). We adopted DM original implementation and parameters for accurate results replication. We employed the same two tasks that were used in DM?: 5mv6m, which involves 5 Marines (allies) against 6 Marines (enemies), and 3sv4z, which features 3 Stalkers (allies) versus 4 Zealots (enemies). The win rates of the demonstrations in both cases are approximately 30%.
How does PegMARL scale with joint demonstrations? Figure presents the learning curves of PegMARI. alongside the DM2 baseline in both the 5mv6m and 3sv4z configurations within the SMAC environment. When the joint demonstrations are sampled from co-trained policies, PegMARL achieves a comparable, and in some cases even higher, success rale compared to DM2 in both tasks.
When the joint demonstrations are sampled from policies that are non-co-trained, the success rate of 2 cxhibits a significant decline in both tasks. In the 5mv6m task, DM success rate drops to nearly 0, indicating failure to learn. In contrast, PegMARL maintains a similar level of performance compared to results from co-trained demonstrations, with only a slight decrease in convergence speed. These findings underscore the versatility and effoctiveness of MARL in scenarios requiring collaboration among diverse agents, demonstrating its applicability and robustness across various multi-agent scenarios and demonstration types.
10

---

Published in Transactions on Machine Learning Research (01/2025)
PegMARL RegMARL(dim
DM2
IPPO
PagMARL
DM2
IPPO
DM 2idif
PegMoRLidim
DM2
5m vs 6m
35 vs 4z
1.0
Mean Battle Wan Rate
Moan Battle Wan Rate
0.8-
0.8-
0.6
0.6
0.4-
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps
1e7
Timesteps
Figure 9: Learning curves of PegMARL versus DM in two tasks under the SMAC scenarios. The suffix "diff" in the legend indicates that the joint demonstrations used are sampled from non-co-trained policies. Otherwise, the demonstrations are sampled from co-trained policies.
6 Conclusions
In this work, we introduce PegMARL, a novel approach for Multi-Agent Reinforcement Learning, which adopts personalized expert demonstrations as guidance and allows agents to learn to cooperate. Specifically, the algorithm utilizes two discriminators to dynamically reshape the reward function: one provides incentives to encourage the alignment between the policy behavior with provided demonstrations, and the other regulates incentives based on whether the behavior leads to the desired objective. We demonstrate PegMARL's effectiveness, scalability, and robustness in both discrete and continuous environments, where it outperforms thate-of-hheart decentralized MARL algorithms, pure imitation learning, and reward-shaping techniques. We observe that PegMARL can achieve near-optimal policies even with suboptimal demonstrations. Furthermore, We showcase PegMART's capability to leverage joint demonstrations and converge successfully, regardless of whether they are sampled from co-trained or non-co-trained policies.
Applicability and Limitations: PegMARL is most applicable when: a) The task allows for some degree of independent decision-making, allowing individual agent behaviors to be isolated and demonstrated,, even if only partially. b) The environment provides feedback on cooperative behaviors, enabling agents to learn coordination beyond individual remonsttattion.. For example, in the door scenario, agents gain exploration guidance for door operation and navigation from personalized demonstrations. At the same time, environmental feodback on joint success drives them to learn coordination about timing and positioning (e.g,, the green agent learning to make way for the red agent). Similarly, in the lava scenario, personalized demonstrations provide guidance for lava-avoiding paths. Still, environmental penalties for collisions and rewards for collective goal achievement encourage the emergence of cooperative navigation strategies.
However, PegMARL may face challenges in tasks requiring continuous, seamless collaboration among agents, such as cooperative object lifting and relocation by multiple agents. In these scenarios, where constant, integrated teamwork is crucial, the reliance on personalized demonstrations may not provide sufficient guidance for complex, ongoing coordination. Despite this, PegMARL's capability to utilize joint demonstrations from diverse sources, including non-co-trained policies, suggests the potential for addressing this limitation. Tt. would be interesting future work to explore PogMARL's capabilities to handle more intricate and continuously collaborative lasks.
Another limitation of our work is the absence of formal convergence guarantees. While DM2 establishes convergence for individual agents optimizing distribution matching objectives, our approach introduces addi- tional complexity through the transition discriminator's dynamic adjustment of demonstration influence. The interplay between the dual discriminators and multi-agent dynamics poses challenges for theoretical analysis, particularly since the joint distribution cannot be directly recovered from individual agent demonstrations our setting. While our extensive empirical results demonstrate robust convergence across diverse scenarios, establishing theoretical convergence guarantees remains interesting direction for future work.
11

---

Published in Transactions on Machine Learning Research (01/2025)
References
Matteo Bettini, Ajay Shankar, and Amanda Prorok. Heterogeneous multi-robot reinforcement learning, arXiv preprinl arXiv:2301007337 2023.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint Xiw:1604407331 2016.
Jie Chen and Wenjun Xu. Policy gradient from demonstration and curiosity. IEEE Transactions Cybernetics, 2022.
Tim Franzmoyer, Matcusz Malinowski, and Joao F. Honriques. Learning altruistic behaviours in reinforcement learning without external rewards. In International Conference on Learning Representations, 2022. URL https://openreviee wtt.:/oorrriiaaaaaaooyyy.y.
Todd Hester, Matej Vecerik, Olivier Pictquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, lan Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advonces in nearal information processing systems, 29, 2016.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations, In International conference machine learning, pp. 2469 2478. PMLR, 2018.
Hyun-Rok Lee and Taesik Lee. Improved cooperative multi-agent reinforcement learning algorithm augmented by mixing demonstrations from centralized policy. In Proceedings of the 18th International Conference on Autonomous Agents and MulliAgent Systems, pp. 1089-1098, 2019.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiie environments. Newral Information Processing Systems (NIPS), 2017.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv: 703. 708.04908, 2017.
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, pp. 2, 2000.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Newral computation, 3(1):88 97, 1991.
Yunbo Qiu, Yuzhu Zhan, Yue Jin, Jian Wang, and Xudong Zhang. Samplo-efficient multi-agent reinforcement learning with demonstrations for flocking control. arXiv preprint KXKi:2209008351, 2022.
Desik Rengarajan, Gargi Vaidya, Akshay Sarvesh, Dileep Kalathil, and Srinivas Shakkotti. Reinforcement learning with sparse rewards using guidance from offline demonstration. In International Conference on Learning Representations, 2022. URL
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation lcarning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint 1rXiv:1902.0404.. 2019.
Jennifer She, Jayesh Gupta, and Mykel Kochenderfer. Agent- time attention for sparse rewards multi-agent reinforcement learning. ar Xixio preprint or Xiw:2210. 1.100.17..4. 17540, 2022.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martín Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint wrXiw:107.088111 2017.
Caroline Wang, Ishan Durugkar, Elad Liebman, and Peter Stone. Dm2: Decentralized multi-agent reinforce- ment learning via distribution matching. 2023.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprinl 05X5:22003.01555, 2021.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aani, volume 8, pp. 1433-14 103--443 1438. Chicago, II, USA, 2008.
13

---

Published in Transactions on Machine Learning Research (01/2025)
A Environment Details
A.1 Discrete Gridworld Environments
We use two gridworld environments with discrete state and action space: the lava scenario and the door scenario (Figure 5). The agents the lava scenario are homogencous because they have the same objective: navigating to their corresponding goal location. The door scenario has heterogeneous agents: the assistant. agent (green) must open the door while the other agent (red) must reach the goal.
This environment has 6-by-6 lava pond in the center (Figure 5a variations of this scenario, each with varying numbers of agents and increasing levels of complexity. The main goal for the agents is to efficiently navigate to their assigned goal locations while avoiding stepping into the lava. An episode terminates when all agents reach their respective goals (succeed), or if any agents step into the lava or the maximum episode length is reached (fail).
The lava scenario:
We provide three
In this scenario, the green agent must. navigate to designated green square and maintain its presence there to sustain the open state of the green door, thereby enabling the entry of red agent into the right side room. An episode ends when the red agent reaches the red goal location (succeed) or the maximum episode length is reached (fail).
The door scenario: This enviromment is adapted from Franzmeyer et al. (2022) (see Figure 5b
Each agent's ' local state space is its {a. coordinates in the map. We concatenate all the local states together to form the global state and assume all agents have access to the global state, which has dimension of Rax2 (n is the agent number). The local action space includes five actions: left, right, up, down, and stay. A sparse reward is granted when episode succeeds, while small penalty will be subtracted according to the steps taken (10 step count/max step). Agents will receive penally of - if they collide with each other.
A.2 Cooperative Navigation
We modified the cooperative navigation task of multi-agent particle environment Lowe et al. (2017): Mordatch & Abbeel (0177) to evaluate the performance of our algorithm in a continuous environment. The modified environment consists of 2 agents, goal landmarks and 1 wall between the agents and the goals. The agents need to navigate around the wall to occupy both landmarks. Each agent has a partial observation of the environment, including its position and velocity as well as the relative positions of other agents and landmarks. The action space is discrete, consisting of moving left, right, up, down, and staying in place. We sparsify their original reward as follows:
m.n, min ((a,g)) + Σ gEG
(0.3, min d(a.g) atA
(11)
a
gEG
A.3 The SMAC Environment
The StarCraft Multi-Agent Challenge (SMAC) Samvelyan et al. (2019) is an environment specifically designed for evaluating multi-agent reinforcement learning (MARL) algorithms. Built on the StarCraft I game, SMAC presents a variety of challenging scenarios where multiple agents must collaborate to control individual units in combat. Each agent has partial observation of its surroundings within a certain sight range. The action space is discrete, including move (dreetiie attack/enemy id], stop, and no-op. A global reward is given at each timestep based the total damage dealt to the enemy units, with additional bonus of 10 or 200 awarded for killing each enemy unit or for winning combat respectively. We consider the same two tasks as used in DM2 (Wang et al. (203))):
5m vs v 6m: The allied team consists of 5 Marines and the enemy team controls 6 Marines.
3svvs vs 4z: The allied team controls 3 stalker units and faces 4 enemy Zealot units.
14

---

Published in Transactions on Machine Learning Research (01/2025)
B Demonstration Details
B.1 Personalized Demonstrations
We let each individual agent perform their designated task in the samne environment map without the other agents' presence to collect personalized demonstrations. This can be implemented through two main methods: a) recording expert human demonstrations for each agent type, and b) training single-agent RL in simplified environments that focus on their specific tasks. In our experiments, we adopted the latter option. takes about 2 hours to Irain a single-agent PPO for each personalized task.
Discrete Gridworld Environments:
Figure shows an example of personalized demonstrations for the visualizes the personalized demonstrations for the door scenario. We summarize the details of the suboptimal demonstrations for the gridworld environment in Table 1 where the average
lava scenario, and Figure 2
episodic rewards are approximately 4.5, about half of their optimal counterparts.
Agent ld
S
A
Samples
Average Episodic Reward
Goal
1
R²
300
4.42
2
R2
440
4.79
3
R2
344
1.3
4
R²
360
4.41
(a)
The lava scenario
Start
Case
Agent
A
Samples
Average Episodic Reward
red
R²
5
643
4.01
Figure 10: An example of person- alized demonstrations for the lava scenario (we did not visualize all the optimal paths). There is only one agent in the environment. The agent may take both the left and the right path toward the goal.
easy
green
R²
5
612
4.39
red
R²
5
607
4.34
hard
green
R²
5
593
4.36
(b) The door scenario
Table 1: The details of suboptimal demonstrations for the gridworld environment.
We provide visual representations of the policy and state occupancy measure corresponding to suboptimal demonstrations in Figure II for the lava scenarios and in Figure 12 for the door scenario. The red square symbolizes the agent's initial position in these visualizations, while the green square designates its respective goal location. Arrows within the figures denote available actions at each state, with arrow length indicating the probability associated with each action.
Cooperative Navigation: For the cooperative navigation task from the multi-agent particle environment (Lowe al. (2077); Mordatch & Abbeel (2017)), the collaborative goal is for the two agents cover both goals, regardless of which agent covers which goal. Therefore, design the personalized task such that single agent covers either of the two goals without the other agent's 's presence. An illustration of personalized demonstration for this task is shown in Figure 13
B.2 Joint Demonstrations
Figure 13: An illustration of person- alized demonstrations for the coopera- tive navigation task. There only one agent in the environment. The agent's 's objective is to reach one of the two goal locations.
The SMAC Environment:
Regarding the joint demonstrations utilized in the SMAC environment (Samvelyan et al. (219))) ), we adopted the demonstrations provided by the authors of DM2 (Wang et al. (2023)). For joint demonstrations sampled from co-trained policies, they are derived from jointly trained expert policies that achieve approximately a 30% win rate. For joint demonstrations sampled from non-co-trained policies, they are obtained from expert
policies that were trained independently in separate teams but executed together in the same environment.
15

---

Published in Transactions on Machine Learning Research (01/2025)
(a) Agent 1
Agent 2
(c) Agent 3
(d) Agent 4
Figure 11: Agent policies and state occupancy measures estimated from the suboptimal demonstrations for the lava scenario.
(a) The easy case
(b) The hard case
Figure 12: Agent policies and state occupancy measures estimated from the suboptimal demonstrations for the door scenario. The top row is for the red agent, and the bottom row is for the green agent.
16

---

Published in Transactions on Machine Learning Research (01/2025)
C Network architecture and hyperparameters
C.1 Training with Personalized Demonstrations
The algorithin are implemented based MAPPO Yuu et al. (2021)) )) with each agent having separate policy and critic networks, discriminators, and optimizers. Both the policy and critic networks two-layer MLPs with hidden dimension of 64 and Tanh activation. The discriminators, including the personalized behavior discriminator and the personalized transition discriminator, are three-layer MLPs with hidden dimension of 64 and Tanh activation. The personalized behavior discriminator takes (8,, as) as input, while the personalized transition discriminator takes (s,, 0g sy) as input.
In partially observable settings, let's assume that the agent's observation the multi-agent MDP is denoted 0f O₁, while in the PerMDP; it is denoted as of E O Since the PerMDP involves single agent and the multi-agent MDP involves multiple agents, the obscrvation space for cach agent cannot be the same in these two settings. For experiments on cooperalive navigation, the dimensions that is not within the observation space O2 before passing them to the discriminators.
heuristic function to convert or by removing
All baselinc algorithms, except ATA, use the same codebase we implemented. MAGAIL is trained with the personalized behavior discriminator but does not use the envirommental reward. DM2 uses both the environmental reward and the personalized behavior discriminator. PegMARL utilizes the environmental reward, the personalized bchavior discriminator, and the personalized transition discriminator. ATA's 's original implementation is adopted for comparison. We summarized the common hyperparameters used for PegMARL, DM2 and MAGAIL in Table The goil rew coef in DM2 is set to be 0.02 for all the gridworkl scenarios and 0.1 for the cooperative navigation task. The coefficient in PegMARL is 0.05 for all the gridworld scenarios and 0.2 for the cooperative navigation task.
epochs 4096
buffer size
0.2 clip Ir
0.0001
Table 2: MAPPO Hyperparamters
C.2 Training with Joint Demonstrations
We implement PegMARL based on the original DM2's codebase to conduct experiments in the SMAC environment, augmenting it with our personalized transition discriminator. The hyperparameters for IPPO and GAIL were set to be identical to those used in DM2, as shown in Table 3 and PegMARL adopted the same hyperparameters as DM2.
5v6
3sv4z
5v6
3sv4z
gail rew coef
0.3
0.05
epochs
10
15
discr epochs
120
120
buffer size
1024
1024
buffer size
1024
1024
gain
0.01
0.01
batch size
64
64
clip
0.05
0.2
n exp eps
1000
1000
Table 3: DM2's IPPO Hyperparameters
Table 4: DM2's GAIL Hyperparameters
C.3 The influence of the weighting term !)
In our experiments, we chose n) through a grid scarch to optimize performance. Interestingly, we found that PegMARL is relatively insensitive 7,, which we attribute to the dynamic weighting provided by the personalized transition discriminator. This discriminator adaptively adjusts the influence of demonstration
17

---

Published in Transactions on Machine Learning Research (01/2025)
Episode Rewards
Episode Rewards
Episode Rewards
2 2.
PezitAnL 2-0
PegMARL,
MAII ge0.03
PegisanL p-0.1
PugMAIL p=01
Pag MAIL ge0.1
Pegioni, 0:0.5
FegMARL, 0:10
Peg MARI. 0:15
PegWMni,
PagMARL,
MABL
Number of Timesteps
Number of Timesteps
Number of Timestops
(a) agents
(b) 3 agents
(c)
agents
Figure 14: Ablation study the weighting term v the lava environment using optimal personalized demonstrations.
PegMAALLing-.55)
PegMARL(diff PegMARLLEdif,
demo
PogMARLEGII, ry=0. 3]
demo
5m, vs 6m
5m VS. 6m
0.8-
0.8
Mean Battle Won Rate
Mean Battle Won Rate
0.6-
0.6
0.4
0.2
0.2
0.0
0.0
0.0
0.4
1.0
0.0
0.8
Timesteps
1e7
Timesteps
1e7
Figure 15: Ablation study over the weighting term on the 5m vs
6m SMAC map. Left: results using joint
demonstrations sampled from co-trained policies; Right: results using joint demonstrations sampled from non-co-trained policies.
alignment based on the likelihood of local state-action pairs leading to desired outcomes, thereby mitigating the impact of suboptimal 1 values.
D Additional Visualizations
We additionally depict the state visitation frequencies of the joint policies learned by PegMARL with suboptimal demonstrations and MAPPO for both the lava (Figure 16) and the door scenario (Figure 17
PegMARL
MАPPO
PegMARL
MΑPPO
PegMARL
MАPPO
(a) 2-agent case
(b) 3-agent case
(c) 4-agent case
Figure 16: State visitation frequency the joint policies learned by PegMARL (with suboptimal demonstra- tions) and MAPPO for the lava scenario. The darker color means a higher value. MAPPO failed learn any meaningful policies in all three settings.
18

---

Published in Transactions on Machine Learning Research (01/2025)
PegMARL
MАPPO
PegMARL
MАPPO
(a) The easy case
(b) The hard case
Figure 17: State visitation frequency of the joint policies learned by PegMARL (with suboptimal demonstra- tions) and MAPPO for the door scenario. The darker color means a higher value. MAPPO converges to a suboptimal policy the easy case, while failing learn the hard case.
19


Reviews:
[{"review_id": "2o8gRvGMan", "reviewer": "Reviewer_ZE3i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- PegMARL is an interesting new MARL problem that could be of interest to the RL community.\n- The motivating examples are helpful in illustrating the main intuition of the paper.\n- The experimental results are quite strong.\n\nWeaknesses\n\n- My main concern is on the feasibility of using only personalized demonstrations in MARL. Specifically, personalized demonstrations can be viewed as the marginals. However, in general, we know that the joint distribution cannot be recovered solely from the marginal distributions of each random variable, unless there is some additional condition, e.g., independence. Therefore, the problem appears somewhat ill-posed, and this limitation has not been well explained. It would be very helpful to describe to what extent PegMARL can work. \n\n- Moreover, the resulting algorithm (cf. Algorithm 1) appears to be a direct extension of distribution matching in DM^2 to the PegMARL. However, given the first point mentioned above, the convergence property under joint demonstrations (i.e., convergence of independent GAIL learners in the DM^2 paper) no longer holds here. Intuitively, the algorithm shall have something fundamentally different to address the issue.\n\n- The comparison in SMAC could be further explained, and there are several missing pieces: 1) How is PegMARL adapted to the setting of joint demonstrations? 2) Why can PegMARL outperform DM^2 with co-trained policies (given that DM^2 is designed specifically for this setting)? 3) The performance of DM^2 appears somewhat weaker than those reported in the DM^2 paper. Some explanation about this would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies a new multi-agent RL (MARL) problem – personalized expert-guided MARL, where the demonstrations are collected for each individual type of experts. By adapting the DM^2 method to this setting, the authors propose PegMARL, which learns the augmented reward for each agent via a GAIL-like subroutine. To handle the restricted domain, this paper proposes to approximate the indicator function by a second discriminator. The proposed PegMARL is evaluated in multiple MARL domains, including both discrete-action environments (like minigrid) and the continuous-action environment (like SMAC) against multiple decentralized MARL benchmark methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses above. \n\nSome additional detailed questions / comments:\n- Eq. (6): How is the $\\epsilon$ determined? This parameter does not appear anywhere in the formulation.\n- What’s the difference between PegMARL and the modified DM^2 with personalized demonstrations in Section 5?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bEypXCK1Ia", "reviewer": "Reviewer_C3De", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very clear, the approach is well explained and it is also sound. Even though the experiments are not many, nor the tasks seem particularly difficult, there appears to be clear benefit over the baselines. \n\nOn the downside:\n- the paper lacks a contextualization of in which scenarios personalized demonstrations can be obtained, and how to obtain them. The availability of personalized demonstrations significantly limits the approach. In multi-agent settings, the co-ordinated policy may frequently be completely unknown. \n- there is no discussion or validation on the choice of the parameter $\\eta$, which should be quite important for the results, and problem-dependent: if the demonstrations are very good and the global task is individually decomposable in personalized tasks, possibly $\\eta$ should be higher. It would important to have experimental validation on the sensitivity of the algorithm to the parameter.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes PegMARL, an approach that leverages personalized (of individual agents) expert demonstrations for cooperative multi-agent reinforcement learning. The approach makes use of reward shaping, using two discriminators that aim at aligning each agent's policy with the demonstration. The approach also includes a parameter that allows to tune how much of the objective is the alignment an how much of it is the collective reward. Finally, the approach is validated in a couple of discrete and continuous environments and shown to outperform the baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I group the requested changes in two:\n- first, the paper should include a discussion on how to obtain expert demonstrations, and in which types of scenarios obtaining them should be easy and not. In relation to this point, in the case of the experiments on this paper, it is important to know how much computation was needed to obtain the personalized demonstrations: MAPPO, for instance, that does not have that added computational time, so the comparison on efficiency may be unfair.\n- second, it is important to have a discussion on the choice of $\\eta$, as well as experimental validation for the robustness of the method to the parameter. Not exclusively, the authors should explain how their parameter $\\eta$ was chosen. It would also be important to see a figure with a metric of performance (for instance the area under curve of the performance and the performance at the end of training) as a function of different values of $\\eta$. Finally, how were the hyperparameters of the baselines chosen, for instance for MA2?\n\nThe requested changes are important and should be addressed by the authors in their rebuttal, as well as integrated in the document if the authors agree on their sense.", "broader_impact_concerns": "I do not expect particular concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E8NwsdGoil", "reviewer": "Reviewer_LZmN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well-motivated and mostly well-written. The experimental results are comprehensive. The figures in the paper effectively conveys the key points of the paper.\n\n\nWeaknesses: \n\nOverall I have a positive view of the paper. However, I think the paper still has some important limitations described as follows:\n\n1. The distinction between homogeneous and heterogeneous agents is confusing throughout the paper. How exactly do the authors make this distinction (do heterogeneous agents have different state space, action space, and reward functions)? Further, why is this distinction not reflected in the personalized MDP setting described in Section 3?\n\n2. I did not fully understand the Lava scenario. While the paper describes some increase in complexity and the number of agents, I am not sure I follow the argument here. All the three sub-figures in Figure 5(a) look\nsimilar (why is one more complex than the other)? Also when testing for increasing numbers of agents in such a simple setting, more agents should have been considered (at-least tens of agents as opposed to just four agents). How is the expert guidance obtained in this setting?\n\n3. The baselines and their implementations have not been clearly described. What are the major reasons for superior performances of PegMARL seen in Figure 6?\n\n4. The experimental section does not clearly describe the expert guidance in any of the environments? How is expert guidance obtained in each environment? What is different in the case of StarCraft (why is it called\njoint demonstrations)?\n\n5. If the guidance is not accurate, why are these called expert guidances? For instance in StarCraft it is described that only 30% win rate is obtained by the demonstrations.\n\n6. In the Introduction, the paper claims that PegMARL is able to obtain near-optimal policies even with suboptimal demonstrations. I think this has not been demonstrated anywhere in the paper. Can the authors clarify where exactly does PegMARL show this behaviour?\n\n\nMinors:\n\n1. Section 1: We prove the effectiveness -> We demonstrate the effectiveness (this is not a theoretical work!).\n2. Section 5.1: both discrete gridworld -> discrete gridworld\n3. Section 5.1: Ways for agents to interact to increase -> Unclear.\n4. Section 6: alignment between policy behavior -> alignment between the policy behavior\n5. Section 2: Qiu et al. suggests -> Qiu et al. suggest", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of using personalized expert demonstrations that pertain to single-agent behaviours, to improve cooperation in multi-agent cooperative environments. The objective is to use the available expert demonstrations to learn to cooperate using an algorithm that tries to align the behaviours of the agent to the behaviours of the demonstrations and improve the policy using reward shaping. The associated algorithm: PegMARL is tested on several cooperative multi-agent tasks including both discrete and continuous action spaces. PegMARL is demonstrated to outperform several state-of-the art MARL baselines in a series of multi-agent environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my comments in the Weaknesses above.", "broader_impact_concerns": "N?A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: - One common criticism in the first round of reviews was that it was not clear how to obtain the expert demonstrations. Authors seemed to have incorporated explanations on this issue in the manuscript, which is good.

- One concern from the reviewers that does not seem to be addressed is about the following claim that appears in the Abstract, Introduction (contribution number (2)) and Conclusion. The text claims:

> "The algorithm obtains near-optimal policies even when provided with suboptimal demonstrations."

However it is not clear what this refers to. Can you please add justifications/explanations for this claim and what it refers to in the results? Conclusion states that this above claim is observed but it is hard to find a clear explanation in Section 5. Please address this in the revision and clearly explain which experiments this claim refer to.

- Another concern is about the lack of convergence guarantees. This is already acknowledged in your rebuttal, can you please incorporate this discussion in your main text and describe difficulties on getting convergence guarantees compared to previous work? This is an empirical work and the reviewing team agrees that the empirical contribution is sufficient but it is good to still have explanations about theory to motivate future work.

- Please explain, so your readers understand better, how Eq. 9 and Eq. 10 are derived and how they are computed in practice. Looking at the reference Kang et al. (2018), one can see where these equations come from but it is better to make your paper self-contained.

- One reviewer pointed out the similarities between the proposed algorithm and DM^2 from Wang et al. (2023). The authors acknowledged this and explained the differences in their rebuttal. I am not sure if this discussion is incorporated in the text. In particular, please include this discussion when you introduce Algorithm 1 to highlight the similarities and differences compared to DM^2.

- Two non-technical suggestions. 1. Please change phrases such as "However, blindly imitating, ..." which appears in Section 4.1. You may say "However, only imitating, ..." 2. Typo in the title of Appendix D: "Addtional"
Decision: Accept (poster)