Select to Perfect: Imitating desired behavior from large multi-agent data
Authors: Tim Franzmeyer
Edith Elkind
Philip Torr
Jakob Nicolaus Foerster
Joao F. Henriques
Venue: ICLR 2024 poster (2024)
Keywords: multi-agent systems
imitation learning
credit assignment
learning from human data
Primary area: general machine learning (i.e., none of the above)

Abstract: AI agents are commonly trained with large datasets of demonstrations of human behavior.
However, not all behaviors are equally safe or desirable.
Desired characteristics for an AI agent can be expressed by assigning desirability scores, which we assume are not assigned to individual behaviors but to collective trajectories.
For example, in a dataset of vehicle interactions, these scores might relate to the number of incidents that occurred. 
We first assess the effect of each individual agent's behavior on the collective desirability score, e.g., assessing how likely an agent is to cause incidents.
This allows us to selectively imitate agents with a positive effect, e.g., only imitating agents that are unlikely to cause incidents. 
To enable this, we propose the concept of an agent's \textit{Exchange Value}, which quantifies an individual agent's contribution to the collective desirability score. 
The Exchange Value is the expected change in desirability score when substituting the agent for a randomly selected agent.
We propose additional methods for estimating Exchange Values from real-world datasets, enabling us to learn desired imitation policies that outperform relevant baselines. The project website can be found at https://tinyurl.com/select-to-perfect.

Published as a conference paper at ICLR 2024
SELECT TO PERFECT: IMITATING DESIRED BEHAVIOR FROM LARGE MULTI-AGENT DATA
Tim Franzmeyer"
Edith Elkind
Philip Torr
Jakob Foerster
João F. Henriques'
University of Oxford
ABSTRACT
AI agents are commonly trained with large datasets of demonstrations of human behavior. However, not all behaviors are equally safe or desirable. Desired char- acteristics for Al agent can be expressed by assigning desirability scores, which we assume are not assigned to individual behaviors but to collective trajectories. For example, in a dataset of vehicle interactions, these scores might relate to the number of incidents that occurred. We first assess the effect of each individual agent's behavior on the collective desirability score, e.g., assessing how likely an agent is to cause incidents. This allows to selectively imitate agents with positive effect, e.g., only imitating agents that are unlikely to cause incidents. To enable this, we propose the concept of agent's Exchange Value, which quan- tifies an individual agent's contribution to the collective desirability score. The Exchange Value the expected change in desirability score when substituting the agent for randomly selected agent. We propose additional methods for es- timating Exchange Values from real-world datasets, enabling us learn desired imitation policies that outperform relevant baselines. The project website can be found patttp://ttiyyee com/ seleeeeeeeeeeeeee perfectt
INTRODUCTION
Imitating human behaviors from large datasets is promising technique for achieving human-Al and AI-AI interactions in complex environments (Carroll et al. 2019 FAIR He et al. 2023 Shih et al. 2022). However, such large datasets can contain undesirable human behaviors, making direct imitation problematic. Rather than imitating all behaviors, may be preferable to ensure that AI agents imitate behaviors that align with predefined desirable characteristies. In this work, assume that desirable characteristics are quantified as desirability scores given for each trajectory in the dataset. This is commonly the case when the evaluation of the desirability of individual actions is impractical or too expensive (Stiennon et al., 22020). Assigning desirability scores to collective trajectories may be the only viable option for complex datasets that involve multiple interacting agents. For instance, determining individual player contributions in a football match is difficult, while the final score a readily-available measure of team performance.
We develop an imitation learning method for multi-agent datasets that ensures alignment with de- sirable characteristics - expressed through Desired Value Functior (DVF) that assigns score to each collective trajectory. This scenario is applicable to several areas that involve learning behavior from data of human groups. One example a dataset of vehicle interactions, desirability scores in- dicating the number of incidents in collective trajectory, and the aim to imitate only behavior that unlikely to result in incidents (e.g., aiming to imitate driving with foresight). Similarly given dataset of social media conversation threads and desirability scores that indicate whether thread has gone awry one may want to only imitate behavior that reduces the chance of conversations (0199)
going awry Chang & Banescu-Nieuleseu-.
* "rttim @ robots..oooooo ax.ac.u
tequal supervision
'The DVF itself is not sufficient to describe desired behavior completely, as it possibly only covers subset of behavior, e.g., safety-relevant aspects. It is complementary the more complex and nuanced behaviors that obtained by imitating human demonstrations, providing guardrails additional guidance.

---

Published as a conference paper at ICLR 2024
Given: Dataset collective
Step 1: Compute Exchange Value (EV) agents from DVF scores
trajectories and desirability scores
EVE DVF
DDVF
(s, a a,S, ... ) DVF
Step 2:
Rank agents by computed Exchange Values
(s, a ...)... DVT
EVE 1-
EVC EVC
EV
(s, a a.S,... DVF(2
Step 3: Only imittte behavior of agents with high Exchange Values
Imitation Learning (8, 5,
We are given a dataset composed of multi-agent trajectories generated by many individual agents, e-g., a dataset of cars driving in urban environments. In addition, the Desired Value Function (DVF) indicates the desirability score of collective trajectory, e.g., the number of incidents that occurred. We first compute the Exchange Value (EV) of each agent, where a positive EV indicates that agent increases the desirability score (e.g. agent driving safely). We reformulate imitation learning to take into account the computed EVs, and achieve an imitation policy that is aligned with the DVF (e.g. only imitating the behavior of safe drivers).
Figure 1:
Assessing the desirability of an individual agent's behavior involves gauging its impact on the col- lective desirability score. For instance, it requires evaluating whether a driver's 's behavior increases the likelihood of accidents, or whether user's behavior increases the likelihood of conversation going awry. This is termed the credit assignment problem (Shapley 1953), akin to fairly dividing the value produced by a group of players among the players themselves. The credit assignment problem proves complex in real-world scenarios due to three main factors (see Figure 2 details): First, many scenarios only permit specific group sizes. This makes Shapley Values (Shapley 1953) concept commonly used in Economics for credit assignment inapplicable, as it relies the comparisons of groups of different sizes (e.g., Shapley Values are not applicable to football players, as football is game of 11 players and group of 12 is never observed.) Second, real-world datasets for large groups are almost always incomplete, i.e., they do not contain trajectories for all (com- binatorially many) possible groups of agents. Third, datasets of human interactions may be fully anonymized by assigning one- tie-tim--uee IDs. In this case, if an agent is present in two trajectories, it will appear in the dataset as if is two different agents, making the credit assignment problem de- generate. This requires incorporating individual behavior information in addition to the information about collective outcomes.
To address these challenges, we propose Exchange Values (EVs), akin to Shapley Values, which quantify an agent's contribution as the expected change in desirability when substituting the agent randomly. The EV of an agent can be understood as the expected change in value when substituting the agent with another randomly selected agent - or as comparing the average value of all groups that include the agent to that of all groups not including the agent (see Step in Figure D. EVs are applicable to scenarios with fixed group sizes, making them more versatile. We introduce EV- Clustering that estimates EVs from incomplete datasets by maximizing inter-cluster variance. We show a theoretical connection to clustering by unobserved individual contributions and adapt this method to fully-anonymized datasets, by considering low-level behavioral information.
We introduce Exchange Value based Behavior Cloning (EV2BC), which imitates large datasets by only imitating the behavior of selected agents with EVs higher than tuneable threshold (see Fig- ure D.. This approach allows learning from interactions with agents with all behaviors, without necessarily imitating them. This is not possible when simply excluding all trajectories with a low collective desirability score, i.e., selectively imitating based on collective scores instead of individual contributions. We find that EV2BC outperforms standard behavior cloning, offline RL, and selective imitation based on collective scores in challenging environments, such as the StarCraft Multi-Agent Challenge (Samvelyan et al. 2019). Our work makes the following contributions:
We introduce Exchange Values (Def. 4.1 to compute an agent's individual contribution to col- lective value function and show their relation to Shapley Values.
We propose EV- EV-Clustering (Def. 4.4 to estimate contributions from incomplete datasets and show a theoretical connection to clustering agents by their unobserved individual contributions.
We empirically demonstrate how EVs can be estimated from EV2BC (Def. 4.5) to learn policies aligned with the DVF, outperforming relevant baselines.
data and employ
2

---

Published as a conference paper at ICLR 2024
Example Secuazio
Dataset charactoriaties Sans
Example obrervation datanet of ageents al.
Stapley
Values
Exchange
Values
Exchange Value Computation
Applicable
Applicable
Hdenttile
DVTV&
Mdall
All sizes permitted.
possille combinations agezia (all
DVBi
DVR
DVB
DVB)
Eauct
groujs) son clterveal.
Group-lämiter
Only speeific
perminted. funeboll pame players
X
V
DVIV
DVI
DVEY
Exact
Low- Dale
Two foothall players might play for
Not all permitted grougs are ulserved
X
X
DVFF
DVF
X
Estinated (patentully with BY Cluseiing!
Degenerale
DVE
DVF
Istamteed with Clasiering and Lcheviour infommation froms
mineymized with one lime me IDs. New ID' player mach ployed.
clastrrng
Figure 2: Overview of different characteristics of real-world datasets, and whether Shapley Values and Exchange Values (EVs) are applicable compute contributions of individual agents the DVF.
RELATED WORK
Most previous work on aligning Al agents' policies with desired value functions either relies on simple hand-crafted rules (Xu al. 2020 [AIR), which do scale to complex environments, or performs postprocessing of imitation policies with fine-tuning (Stiennon et al., 2020; Ouyang et al., simulator.
2022 Glaese al. 2022 Bai al. 2022). which requires access the environment
In language modeling, Korbak et al. (2023) showed that accounting for the alignment of behavior with the DVF already during imitation learning yields results superior fine-tuning dfter-the-far however, their approach considers an agent-specific value function. In contrast, we consider learning a policy aligned with a collective value function, and from offline data alone.
Credit assignment in multi-agent systems was initially studied in Economics (Shapley 1953) Sub- sequently, Shapley Values (Shapley) 1953) and related concepts have been applied in multi-agent reinforcement learning to distribute rewards among individual agents during the learning pro-
cess (Chang et al. 2003 Foerster et al. 2018 Nguyen et al., 2018 Wang et al. 2020 Li et al.]
2021 Wang et al. 2022). Outside of policy learning, Heuillet et al. (2022) used Shapley Values to
analyze agent contributions in agent environments, however this requires privileged access to simulator, in order to replace agents with randomly-acting agents. In contrast Shapley Values, the applicability of EVs to all group sizes allows us to omit the need to simulate infeasible coalitions.
In contrast this work, existing work in multi-agent imitation learning typically assumes observa- tions to be generated by optimal agents, as well as simulator access (Le et al. 2017]] Song et al.) 2018 Yu al. 2019). Similar our framework, offline multi-agent reinforcement learning (Jiang & Lu 2021: Tseng et al. 2022 Tian et al. 2022) involves policy learning from multi agent demon- strations using offline data alone, however, assumes a dense reward signal be given, while the DVF assigns single score per collective trajectory.
In single-agent settings, a large body work investigates estimating demonstrator expertise to en- hance imitation learning (Chen et al. 2021 Zhang et al. 2021: Cao & Sadigh 2021 Sasaki & Yamashina 202T Beliaev et al.] [022
Yang et al.) 2021). However, these methods do not translate to the mueti- setting due to the challenge of credit assignment.
To the best of our knowledge, no prior work has considered problem imitating multi-agent datasets containing mixed behaviors, while ensuring alignment with collective value function.
3 BACKGROUND AND NOTATION
Markov Game. We consider Markov Games (Littman 1994), which generalize Markov Decision Processes (MDPs) to multi-agent scenarios. Markov Game, agents interact in a common envi- ronment. At time step I, each agent (the ith of total of m agents) takes the action af and the environ-
3

---

Published as a conference paper at ICLR 2024
ment transitions from state s' to s . A reduced Markov game (without rewards) is then defined by a state space S (s' E S), a distribution of initial states 77, the action space. A, (a E A,) of each agent i, an environment state transition probability P(s ai,..... am)) and the episode length T. We denote this Markov Game as M = A,,A, PT,. with collective trajectories 7 = (so, ap, ,0,...,,r)) 8r).
Set of multi-agent demonstrations generated by many agents. We consider Markov game M of m agents and a set of demonstrator agents N = {l, n where n 2 m. The Markov Game is further assumed to be symmetric (we can change the ordering of players without changing the game). The demonstration set D captures interactions among various groups of agents in M. Every entry D.- = (st, Tu, contains a trajectory тs, for group of agents s; S N. Notably, T. contains the collective trajectory of all agents in the group 8i.
Shapley Values. We now define the concept of the Shapley Value of an agent (Shapley) 1953), which is commonly used to evaluate the contributions of individual agents to a collective value function in a characteristic function game. Definition below somewhat unconventional but can be easily seen to be equivalent to the standard definition.
Definition 3.1 (Characteristic function game). A characteristic function game G is given by a pair (N, e), where N = {I,.... n is a finite, non-empty set of agents and 2 + R is characteristic function, which maps each group (sometimes also referred to as coalition) C' C N to real number vCC); it is assumed that v00)= = 0. The number v(C) is referred as the value of the group C.
Given a characteristic function game G = v), ,, let N denote the set of all permutations of N Mi.. i.c., one-to-one mappings from N [+] to itself. For each permutation π E [N we denote by (m)) the slice of π up until and including position m,; we think of S (m) as the set of all agents that appear in the first m positions in (note that S. = 0). The marginal contribution of an agent with respect to a permutation æ and a slice m in a game G = (N, v) is given by
((S U (i)) ((S))))).
m
This quantity measures the increase in the value of the group when agent i joins them. We can now define the Shapley Value of an agent i: it is simply the agent's average marginal contribution, where the average is taken over all permutations of the set of all other agents N (i) and all slices.
Definition 3.2 (Shapley Value). Given a characteristic function game G = (N, v) with |N] = n,
the Shapley Value of an agent i E N is denoted by (V) and is given by
SV,(G) 1/nt Σ Em= ErERM AC (
Def. 3.2 is important in the context of credit assignment, as possible solution for distributing collective value to individual agents. also has several consistency properties (Shapley 1953).
METHODS
Problem setting. Given dataset D of trajectories generated by groups of interacting agents and a Desired Value Function (DVF), the goal of our work is to learn an imitation policy for a single agent that is aligned with the DVF. We assume that fraction of the demonstrator agents' behavior is undesirable; specifically, their presence in a group significantly reduces the DVF. Further, we assume that the number of demonstrator agents is much larger than the group size.
Overview of the methods section. To evaluate agents' contributions in games that only permit specific group sizes, we first define the concept of EVs (Def. (Def..4.l) for regular characteristic function games (Def. 3.1. We then show that our definition extends naturally to characteristic function games with constraints on permitted group sizes. We finally derive methods to estimate EVs from real-world datasets with limited observations (sce Figure for an overview).
4.1 EXCHANGE VALUES TO EVALUATE AGENTS' INDIVIDUAL CONTRIBUTIONS
Note that each term of the Shapley Value, denoted (1), requires computing the difference in values between two groups of different sizes, with and without agent i (see Def. 3.2). If we wish
4

---

Published as a conference paper at ICLR 2024
to only compare groups with the same size, then a natural alternative is compute the difference in values when the agent at position m is replaced with agent i:
T m, (1) = (m - 1) - (sS, (m)))
We call this quantity the exchange contribution of i, given a permutation of agents sliced at posi- tion m. It represents the added value of agent in a group. Importantly does not require values of groups of different sizes.
We now define the EV analogously to the Shapley Value as the average exchange contribution over all permutations of N iH and all non-empty slices.
Definition 4.1 (Exchange Value). Given characteristic function game G = (NN, v) with [N] = n, the Exchange Value of an agent E N is denoted by (G) and given by
(V, = ((n - 1)! (n - (01-1))- 1)) Emel ExelNAto
NG ME,
In words, the EV of an agent can hence be understood as the expected change in value when substi- tuting the agent with another randomly selected agent, or as comparing the value of all groups that include the agent to that of all groups that do not include the agent (see Step I in Figure
Relationship between Shapley Value and Exchange Value. It can be shown that the Exchange Values of all agents can be derived from their Shapley Values by simple linear transformation: we subtract a fraction of the value of the grand coalition N (group of agents) and scale the _ (SV(GG) - 1/2 v v(N)). The proof proceeds by computing the
result by 7/n-1: GV) coefficient of each term B(C), C P N, in summations (T) and (3) (see Appendix AA.. Hence, the Shapley Value and the Exchange Value order the agents in the same way. Now, recall that the Shapley Value is characterized by four axioms, namely, dummy, efficiency, symmetry, and linearity Shapley 1953). The latter two are also satisfied by the Exchange Value: if (CC)(i)) = ((CU()) for all C CENN {i.j). we have (V) (G) (symmetry). and if we have two games G1 and G2 with characteristic functions U1 and U2 over the same set of agents N, then for the combined game G (N, v) with the characteristic function v given by v(C) = i1 (C) + (C) we have EV,(G) = EV]( + EV,(G2) (linearity). The efficiency property of the Shapley Value, i.e., LiEN SV; = v(N) implies that LIEN (V) 0. In words, the sum of all agents' EV is zero. The dummy axiom, too, needs to be modified: if an agent is a dummy, i... ((CUfi))==(( = for every C S N \ {i} then for the Shapley value we have (G) = 0 and hence (V, = (ife----(N)) o(N). In each case, the proof follows from the relationship between the Shapley Value and the Exchange Value and the fact that the Shapley Value satisfies these axioms (see Appendix A).
-1/n-1 1.
4.1.1 COMPUTING EXCHANGE VALUES IF ONLY CERTAIN GROUP SIZES ARE PERMITTED
For a characteristic function game G = (N, v) the value function u can be evaluated for each possible group C C N. We now consider the case where the value function is only defined for groups of certain sizes m E M, i.e. v is only defined for a subset of groups of certain sizes.
Definition 4.2 (Constrained characteristic function game). A constrained characteristic function game G is given by tuple (N, U, M), where N = {l...., n) is finite, non-empty set of agents, M S. {O....., n - 1} is a set of feasible group sizes and v : {C E 2 : |C| E M} - is a characteristic function, which maps each group C S N of size |C| E M to real number (C)).
Note that both the Shapley Value and the EV are generally undefined for constrained characteristic function games, as the value function is not defined for groups C of size CI & M. The definition of the Shapley Value cannot easily be adapted to constrained characteristic function games, as its computation requires evaluating values of groups of different sizes. In contrast, the definition of the EV can be straightforwardly adapted to constrained characteristic function games by limiting the summation to slices of size m E M++, where M = {m E M m > 0}. Hence, we define the Constrained EV as the average exchange contribution over all permutations of N MVii} and over all slices of size m E M
Definition 4.3 (Constrained Exchange Value). Given a constrained characteristic function game G = (N, v, M) with |N| = n. the Constrained Exchange Value of an agent i E N is denoted by GV) and is given by GV) = ((n 1)! |M+D-1 IM+D)-..mett . Exe ma (i).
LG
5

---

Published as a conference paper at ICLR 2024
We refer to the Constrained EV and EV interchangeably. they are applicable to different settings. some groups are not observed, we can achieve an unbiased estimate of the EV by sampling groups uniformly at random. The expected EV is GV) = (.-- [T m. (i) This expectation converges to the true EV in the limit of infinite samples.
As outlined in Step in Figure I the EV of an agent is comparison of the value of group that includes the agent and a group that does not include the agent, considering all permitted group sizes.
4.2 ESTIMATING EXCHANGE VALUES FROM LIMITED DATA
The EV assesses the contribution of an individual agent and is applicable under group size limita- tions real-world scenarios (see Group- Prooup-Limizee Limiliimeeeeeee- in Figure D. However, exactly calculating EVs is almost always impossible as real-world datasets likely do not contain observations for all (combi- natorially many) possible groups (Low-Dat.a in Figure 2 We first show sampling -bsed estimate (Section 4.2) of EVs, which may have high variance for EVs of agents that are part of only a few trajectories (outcomes). Next, we introduce a novel method, EV-Clustering (Section [42.1]. which clusters and can be used to reduce the variance. When datasets are anonymized with one-time-use IDs, each demonstrator is only observed as part of one group (see Degenerate in Figure ren- dering credit assignment degenerate, explained in Section 4.2.1 We address this by incorporating low-level behavior data from the trajectories 7.
4.2.1 EV-CLUSTERING IDENTIFIES SIMILAR AGENTS
In the case of very few agent observations, the aobove-introduce sampling estimate has a high vari- ance. One way to reduce the variance by clustering: if we knew that some agents tend to contribute similarly to the DVF, then clustering them and estimating one EV per cluster (instead of one EV per agent) will use more samples and thereby reduce the variance. Note that, as our focus is on accu- rately estimating EVs, we do not consider clustering agents by behavior here, as two agents may exhibit distinct behaviors while still contributing equally to the DVF.
We propose EV- WV-Clustering, which clusters agents such that the variance in EVs across all agents maximized. In Appendix A we show that EV-Clustering is equivalent to clustering agents by their unobserved individual contribution, under the approximation that the total value of a group is the sum of the participating agents' individual contributions, an assumption frequently made for theoretical analysis (Lundberg & Leel 2017 Covert & Lee 202T), as it represents the simplest trivial class of cooperative games. Intuitively, if we choose clusters that maximize the variance in EVs across all agents, all clusters' EVs will be maximally distinct. An example of poor clustering is a random partition, which will have very similar EVs across clusters (having low variance).
Specifically, we assign n agents to S clusters K {1...., k 1}, with individual cluster assignments u = {uo.... ..., un-1), where u, E K. We first combine the observations of all agents within the same cluster by defining a clustered value function i(C) that assigns a value to a group of cluster-centroid agents C C K by averaging over the combined observations, as i(C) = 1/n Em= Erelly ((S,(m) 1([u, 1j (mm) C),, where 1) is a normalization constant. The EV of an agent is then given as (V), with G = (K, 6). therehy assigning equal EVs to all agents within one cluster.
Definition 4.4 (EV-Clustering). We define the optimal cluster assignments u² such that the variance in EVs across all agents is maximized:
u' E arg max, (Er(
We show in Appendix B.I that this objective equivalent to clustering agents by their unobserved individual contributions, under the approximation of an additive value function.
4.2.2 DEGENERACY OF THE CREDIT ASSIGNMENT PROBLEM FOR TULLY-ANONYMIZED DATA
If two agents are observed only once in the dataset and part of the same group, equal credit must be assigned to both due to the inability to separate their contributions. Analogously, when all agents only observed once, credit can only be assigned to groups, resulting in the degenerate scenario

---

Published as a conference paper at ICLR 2024
Relation between (%) of abserved data and EV estirnaiion errer
Overcooked EV Error
Crompod Room
..
Table I: Resulting performance with re- spect to the DVF for different imitation learning methods in different Starcraft sce- narios.
ToC Vieu
notrin Ring
EV Error
TeC
Mothod
2x32
3s_18.52
6h.vs.8z
20%
31
BC
12.14 +:
13.10 +: 2.0
8.56 = 0.6
Group-BC
15:41 + 16.63
9.10
Figure 3: Mean error in estimating EVs with decreas- ing number observations. 'Deg.' refers to the fully anonymized degenerate case. Error decreases signif- icantly agents are clustered (green-shades area).
EV2BC (Ours)
17.38 + 1.6
20.31 24
10.0 +:0+000 0.91
that all agents in a group are assigned the same credit (e.g. are assigned equal EV). We solve this by combining low-level behavior information from trajectories T with EV-Clustering (see Sec. 5.1).
4.3 EXCHANGE VALUE BASED BEILAVIOR CLONING (EV2BC)
Having defined the EV of an individual agent and different methods to estimate it, we now define a variation of Behavior Cloning (Pomerleau 199T), which takes into account each agent's contribution to the desirability value function (DVF). We refer to this method as EV2BC. Essentially, EV2BC imitates only actions of selected agents that have an EV larger than a tunable threshold parameter.
Definition 4.5 (EV based Behavior Cloning (EV2BC)). For set of demonstrator agents N, dataset DD and a DVF, we define the imitation learning loss for EV2BC as
LEV2BC = -EEEN (Es (a]]s))) (a("))).EEVVV 1(EV >c)
(5)
where EV.VFF denotes the EV of agent n and where e is a tunable threshold parameter that trades off between including data of agents with higher contributions to the DVF and reducing the total amount of training data used.
EXPERIMENTS
The environments that we consider only permit certain group sizes, hence we use constrained EVs (see Def. 4.3) We run all experiments for five random seeds and report mean and standard deviation where applicable. For more details on the implementation, please refer to the Appendix. In the following experiments, we first evaluate EVs
measure of an agent's contribution We then assess the average estimation error for EVs as the number of observations in the dataset D decreases and how applying clustering decreases this error. We lastly evaluate the performance of Exchange Value based Behaviour Cloning (EV2BC, see Definition 4.5) for simulated and human datasets and compare relevant baselines, such as standard Behavior Cloning (Pomerleau 1991) and Offline Reinforcement Learning (Pan et al. 2022).
given DVF.
In Tragedy of the Commons (Hardin) 1968) (ToC) multiple individuals deplete shared resource. is a social dilemma scenario often studied to model the overexploitation of common resources (Di- etz al., 2003 Ostrom 2009). We model ToC as multi-agent environment and consider three DVFs representing different measures of social welfare: the final pool size Viiall the total resources consumed Uooal and the minimum consumption among agents Umin-
Overcooked (Carroll et al. 2019 is two-player game simulating a cooperative cooking task requir- ing coordination and is common testbed in multi-agent research. Within Overcooked, we consider the configurations Cramped Room and Coordination Ring (displayed in Figure 4 For each envi- ronment configuration, we generate two datasets by simulating agent behaviors using near-optimal planning algorithm, where we use parameter À to determine agent's behavior. For X agents act (ntar)-optimal, for = -1 agents act adversarially. We refer to À as the agent's 's trait, as acts as proxy for the agent's 's individual contribution to the collective value function. Each demonstration dataset D is generated by n = 100 agents, and trajectories т are of length 400. The adversarial dataset Dady is comprised of 25% adversarial agents with À = -1 and 75% (near)-optimal agents
7

---

Published as a conference paper at ICLR 2024
with =, while for the dataset D agents were uniformly sampled between À and X 1. The Dhuman dataset was collected from humans playing the game (see Carroll et al. (2019) ): it is fully anonymized with one-time-use agent identifiers, hence is degenerate dataset (see Figure bottom row). We consider the standard value function given for Overcooked as the DVF, i.e. the number of soups prepared by both agents over the course of a trajectory.
The StarCraft Multi-Agent Challenge (Samvelyan et al. 2019) is a cooperative multi-agent en- vironment that is partially observable, involves long-term planning, requires strong coordination, and is heterogeneous. We consider the settings 2s3z, 3s_vs_55 and 6h_vs_82, which involve teams of 3-6 agents. For each, we generate a pool of 200 agents with varying capabilities by extract- ing policies at different epochs, and from training with different seeds. We generate a dataset that contains simulated trajectories of 100 randomly sampled groups (out of 10 possible groups) and use the environment's ' ground truth reward function to assign DVF scores according to the collective performance of agents.
Exchange Values assess an agent's individual contribution to a collective value function. To analyze EVs measure for agent's individual contribution DVF, consider full datasets that contain demonstrations of all possible groups, which allows us to estimate EVs accurately.
In ToC, we find that the ordering of agents broadly reflects our intuition: Taking more resources negatively impacts the EVs, and agents consuming the average of others have less extreme EVs. The color-coded ordering of agents under different DVFs in shown in Figure in App. [ In Overcooked, we consider the two simulated datasets (Dadu and D but not the human dataset, as the individual contribution is unknown for human participants. We find that EVs of individual agents are strongly correlated with their trait parameter X, which is proxy for the agent's individual contribution, and
provide a plot that shows the relationship between À and EV in Figue 5 in App. B
5.1 ESTIMATING EVS FROM INCOMPLETE DATA
Estimation error for different dataset sizes. We now turn realistic settings with missing data, where EVs must be estimated (Sec. 4.2) For both ToC and Overcooked, we compute the mean estimation error in EVs if only fraction of the possible groups contained the dataset. As expected, we observe in Fig. 3 that the mean estimation error increases as the fraction of observed groups decreases, with the largest estimation error for fully anonymized datasets (see Fig, Deg.)
Estimating EVs from degenerate datasets with EV-Clustering. To estimate EVs from degen- erate datasets, first obtain behavior embeddings from the low-level behavior information given in the trajectories r in D. Specifically, in Overcooked and ToC, we concatenate action frequencies in frequently observed states. In Starcraft, we use TF-IDF (Spärck Jones, 1972) to obtain behav- ior embeddings. We then compute a large number of possible cluster assignments for the behavior embeddings using different methods and hyperparameters. In accordance with the objective of EV- Clustering, we choose the cluster assignment with the highest variance in EVs. We observe in Figure 3 that clustering significantly decreases the estimation error (see Deg. clustered).
5.2 IMITATING DESIRED BEHAVIOR BY UTILIZING EVS
We now evaluate EV2BC in all domains. In accordance with the quantity of available data, we set the threshold parameter such that only agents with EVs above the 90th, 67th, and 50th percentile are imitated in ToC. Starcraft, and Overcooked, respectively. We replicate the single-agent EV2BC policy for all agents in the environment and evaluate the achieved collective DVF score. As base- lines, we consider (1) BC, where Behavior Cloning (Pomerleau) 199T) done with the full dataset without correcting for EVs, (2) offline multi-agent reinforcement OMAR (Pan et al.] (202) with the reward at the last timestep set to the DVF's score for a given trajectory (no per-step reward is given by the DVF) and (3) Group BC, for which only collective trajectories with DVF score larger than the relevant percentile are included. While EV2BC is based on individual agents' contributions, this last baseline selectively imitates data based on group outcomes. For instance, if collective trajec- tory includes two aligned agents and one unaligned agent, the latter baseline is likely to imitate all three agents. In contrast, our approach would only imitate the two aligned agents.
8

---

Published as a conference paper at ICLR 2024
Table 2: Resulting performance with respect to the DVF for different imitation learning methods in the Overcooked environments Cramped Room (top) and Coordination Ring (bottom). In Tragedy of Commons: 12 agents experiment at the top, 120 agents experiment at the bottom.
Overcooked
Tragedy of Commons
Imitation method
DA
Dude
Dluas
Vyfinal
Viotal
Venón
BC Pomerreau 1991
10.8 2.14
40.8 = 12.7
153.34 11.5
2693.6 + 139.1
50.6 +
2.4 = 0.45
Group-BL
54.2 + 5.45
64.8 = 7.62
163.34 6.08
5324.2 + 210.8
100.01 + 20.08
4,60 = 1.01
OMAR Pan 2022
3.2
25.6 8.9
12.5 4.5
EV2BC (DurS)
91.6 4: 12.07
104.2 =: 10.28
170.89 +: 6.8
10576.8 307.4
342.8 1: 49.36
44.2 4: 6.4
BC Pomertcau 1991
15.43 +: 4,48
10.4 4: 6.8
104,89 +: 12.44
2028.8 = 60.9
38.9 + 10.4
18 = 0.4
Group
14.6 2.48
102.2
3400.5
77.1
3.51
OMAR (aan et al. 2022
12.43 3.35
9.5 3.5
12.4 + 6.0
EV2BC (surs]
30.2 6.91
12.4 2.65
114.89 5.08
8123.4 +: 600.8
270.0 50.0
33.1 7.1
ToC results. We imitate datasets of 12 agents and 120 agents, with group sizes of 3 and 10, spectively, evaluating performance for each of the three DVFs defined for the ToC environment. We do not consider the OMAR baseline policies that EV2BC outperforms the baselines by a large margin.
not learned but rule-based. TTable[Temoosttttt
Overcooked results. We now consider all datasets D D and Dhuman in both Overcooked environments. We evaluate the performance achieved by agents with respect to the DVF (the envi- ronment value function of maximizing the number of soups) when trained with different imitation learning approaches on the different datasets. EVs are computed as detailed in Section 5.1 Table I shows that EV2BC clearly outperforms the baseline approaches. We further note that EV2BC sig- nificantly outperforms baseline approaches on the datasets of human-generated behavior, for which EVs were estimated from a fully-anonymized real-world dataset. This demonstrates that BC on datasets containing unaligned behavior carries the risk of learning wrong behavior, but it can be alleviated by weighting the samples using estimated EVs.
Starcraft Results. We observe in Table T that EV2BC outperforms the baselines by substantial margin, underlining the applicability of our method to larger and more complex settings. We omitted the OMAR baseline, which is implemented offline MARL with the DVF as the final-timestep reward, as it performed significantly worse than BC.
6 CONCLUSION
Our work presents a method for training AI agents from diverse datasets of human interactions while ensuring that the resulting policy is aligned with given desirability value function. However, it must be noted that quantifying this value function is an active research area. Shapley Values and Exchange Values estimate the alignment of individual with group value function (which must be prescribed separately) and, as such, can be misused if they are included a larger system that is used judge those individuals in any way. Discrimination of individuals based on protected attributes is generally unlawful, and care must be taken to avoid any discrimination by automated means. We demonstrated novel positive use of these methods by using them to train aligned (beneficial) agents, that do not imitate negative behaviors in dataset. We expect that the benefits of addressing the problem of unsafe behavior by AI agents outweigh the downsides of misuse of Shapley Values and Exchange Values, which are covered by existing laws.
Future work may address the assumption that individual agents behave similarly across multiple tra- jectories and develop methods for a more fine- -e-igrained assessment of desired behavior. Additionally, exploring how our framework can more effectively utilize data on undesired behavior is an inter- esting avenue for further investigation, e.g., developing policies that are constrained to not taking undesirable actions. Lastly, future work may investigate applications to real-world domains, such as multi-agent autonomy scenarios.
Reproducibility. To help reproduce our work, we publish code on the project website at pttps: //tinyurl com/ wwweeeeeeeeeeeeeeeeeeeeeeeeeeeee to- tt----ttt- We provide detailed overviews for all steps the exper- mental evaluation In the Appendıx, where we also link to the publicly available code repositories that our work used. We further provide information about computational complexity at the end of the Appendix.

---

Published as a conference paper at ICLR 2024
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05866, 2022.
Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimat- ing expertise of demonstrators. In International Conference on Machine Learning, pp. 1732-1748. PMLR, 2022.
Zhangjie Cao and Dorsa Sadigh. Learning from imperfect demonstrations from agents with varying dynamics. IEEE Robotics and Automation Letters, 6(3):5231-5238, 2021.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Jonathan P. Chang and Cristian Danesau-Niculeseu-Miiill Conversations gone awry dataset [reddit emv version]. cessed: 2024-03-14.
htm? 2019. Ac-
Yu-Han Chang, Tracey Ho, and Leslie Kaelbling. All learning is local: Multi-agent learning in global reward games. Advances in neural information processing systems, 16, 2003.
Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-
supervised reward regression. In Conference on robot learning, pp. 1262-1277. PMLR, 2021.
Ian Covert and Su-In Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In International Conference on Artificial Intelligence and Statistics, pp. 3457-3465. PMIR, 2021.
Thomas Dietz, Elinor Ostrom, and Paul C Sterm. The struggle govern the commons. science, 302(5652): 1907-1912, 2003.
Meta Fundamental Research Diplomacy Team (FAIR) Anton Bakhtin, Noam Brown. Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human- level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378 (6624))1067-100) 2022.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counter- factual multi-agent policy gradients. In Proceedings of the AAAI conference artificial intelligence, vol- ume 32, 2018
Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. /4375, 2022.
Garrett Hardin. The tragedy of the commons: the population problem has technical solution; requires fundamental extension in morality. science, 162(3355): 1968.
Jerry Zhi-Yang He, Zackory Erickson, Daniel Brown, Aditi Raghunathan, and Anca Dragan. Learning representations that enable generalization assistive tasks. Conference on Robot Learning, pp. 2105- 21 14. PMIR, 2023.
Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz Rodriguzz.. Collective explainable ai: Explaining co- operative strategies and agent contribution in multiagent reinforcement learning with Shapley values. IEEE Computational Intelligence Magazine, 17(1):59-71, 2022.
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2108.0/4 001.0183, 2021.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint 20X.v22302.08882.. 2023.
Dieter Kraft. A software package for sequential quadratic programming. Forschungsbericht- Deutsche Forschungs- and Versuchsanstalt fur Luft- und Raunfahri, 1988.
Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In International Conference on Machine Learning, pp. 1995-2003. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2024
Jiahui Li. Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun Xiao. Shapley counterfactual credits for multi-agent reinforcement learning. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 934-942, 2021.
Michael Littman. Markov games as a framework multi-agent reinforcement learning. In Machine Learning Proceedings 1994. 1994. doi: 10.1016/b978-1 (a1-119988--558666 5-5600-333-660002-2
Scott Lundberg and Su-In Lee. unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Duc Thien Nguyen, Akshat Kumar, Hoong Chuin Lau. Credit assignment collective multiagent with global rewards. Advances in neural information processing systems, 31, 2018.
Elinor Ostrom. general framework for analyzing sustainability -colooccco systems. Science, 325 (5939):419 422, 2009.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback, Advances in Neural Information Processing Systems, 15:27730-27774, 2022.
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent re- inforcement learning with actor rectification. In International Conference on Machine Learning. pp. 17221- 17237. PMLR, 2022.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumnaeaa,, M. Brucher, M. Perrot, and E. Duches- nay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research. 12:2825-2835 2011.
Dean A. Pomerleau. Efficient Training of Artificial Neural Networks for Autonomous Navigation. Neural Computation, 3(1), 1991. ISSN 1899-7667. doi: 10. 16.1/e2//cco. 1991......999.......
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr. Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint #90iv:/902.000
2019.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In International Con- ference on Learning Representations, 2021.
Lloyd Shapley. A value for n-person games. Contributions to the Theory of Games, pp. 307-317, 1953.
Andy Shih, Stefano Ermon, Dorsa Sadigh. Conditional imitation learning multi-agent games. 2022 17th ACM/EEE International Conference on Human-Roul Robot Interaction (HRI), pp. 166-175. IEEE, 2022.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018.
Karen Spärck Jones. A statistical interpretation of term specificity and application retrieval. Journal of Documentation, 28(1):11-21, 1972.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Infor- mation Processing Systems. 33:3008 3021, 2020.
Robert Thomdiie. Who belongs in the family? Psychometrika, 18(4):267-776, 1953.
Qi Tian, Kun Kuang, Furui Liu, and Baoxiang Wang. lecarning from good trajectories offline multi-agent reinforcement learning. arXiv preprint 2rXiv:2211.15661 2022.
Wei-Cheng Tseng, Tsun Huua- Johnson Wang. Yen-Cher Lin, and Phillip Isola. Offline multi multi-agent rein- forcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35: 226-237, 2022.
Jianhong Wang, Yuan Zhang, Tac-Kyun Kim, and Yunjic Gu. Shapley quvalue: A local reward approach to solve global reward games. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 7285-7292, 2020.
Jianhong Wang Yuan Zhang. Yunjie Gu, and Tac- Kyun Kim. Shaq: Incorporating shapley value theory into multi-agent g-aarnnng. Advances Neural Information Processing Systems, 35:5941-5954, 2022.
11

---

Published as a conference paper at ICLR 2024
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for safety in open- domain chatbots. arXiv preprint arXiv:2010.0707,, 2020.
Mengjiao Yang. Sergey Levine, and Ofir Nachum. Trail: Near-optimal imitation learning with suboptimal data.
arXiv preprint arXiv:2110.14770, 2021.
Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning, pp.
PMLR, 2019.
Songyuan Zhang. Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation learning from demonstrations with varying optimality. Advances Neural Information Procexsing Systems, 34:12340- 12350, 2021.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX TO METHODS
A.1 AXIOMATIC PROPERTIES OF THE EXCHANGE VALUE AND ITS RELATIONSHIP WITH THE SHAPLEY VALUE
Fix characteristicc function game G with set of players N. It is well-known that the Shapley Value satisfies the following axioms (Shapley 1953):
(I) Dummy: an agent satisfies (i)) v(C) for all S N \ [i] then SVi(G) = O:
(2) Efficiency: the sum of all agents' Shapley Values equals to the value of the grand coalition, i.c.
LIEN SV((G) = v(N);
(3) Symmetry: for every pair of distinct agents N with u(C v(Cu (0) for all C N (N..)) we have (V) = (V)(G);
(4) Linearity: for any pair of games G₁ = (N, v1) and G2 (N, U2) with the same set of agents N, the game
G = (N, v) whose characteristic funciton is given by v(C') = (C + vz(C) for all C' S N satisfies SV SVi(G,) + (G2) for all E N.
Indeed, the Shapley Value is the only value for characteristic function games that satisfies these axioms (Shap-
ley 1953). It then natural to ask which of these axioms (or their variants) are satisfied by the Exchange Value.
To answer this question, we first establish a relationship between the Shapley Value and the Exchange Value. Proposition A.1. For any characteristic funnttoo game G = (N, v) and every agent i E N we have
EVi(G)
11-1 SVV(G) (NN
(6)
Proof. Fix an agent and consider an arbitrary non-empty coalition C & N (1).
the expression for the Shapley Value of the coefficient of v(C)
n! (IC1))(n - 1- IC])!
we subtract the fraction of permutations of N where the agents appear in the first |C] positions, followed by i. By the same argument, the coefficient of v(C u (1)) is
((C))((n - (CI)t.
Similarly, in the expression for the Exchange Value of i the coefficient of v(C) is
(n 1)l(n - 1) ((C))((n - 1 [C]]!:
each permutation of N \ (i) where agents in C' appear in the first |C| positions contributes with coefficient (m=1)((m-1)' By the same argument, the coefficient of uCC'U (i)) is
1
(n 1)((n -1) 1) (IC1)t(n-1 - 1- 1111)))
Now, if C = N (i). in the expression for SVGGG the coefficient of v(C) is - and the coefficient of (C) = (NV) is n. contrast, the expression for EV:(G) the coefficient v(C) is - - for each of (n- - 1)! permutations N \ {i] we subtract vCC) with coefficient (n-1))m-I] when we replace the last agent that permutation by On the other hand, e(N) never appears.
follows that, for every coalition C Ç N. if the value r(C) appears in expression for (G) with weight w then it appears in the expression for FV(G)) with weight = w. Hence
n n - SH.(G) 1 ((N)) n
EVI(G)
Example A.2. Consider characteristic function game G = (N, v). where = {1,2} and v is given by
v((1)) 2, x((2] = 4, v({1, 2}} = 10. We have
SHi (G) (2 - (10 - 4))
SHz(G) (4+ (10 2))/2 6
and
=V
EVz(G)= =400)=4-0 -4-2-2
Note that =G) 2(H (() FV²()) 2(SH))())(
-r)())
13

---

Published as a conference paper at ICLR 2024
can Proposition A.1 to show that the Exchange Value satisfies of the axioms satisfied by the Shapley Value, namely, linearity and symmetry.
Proposition A.3. The Exchange Value satisfies symmetry and linearity axioms.
Proof. For the symmetry axiom, fix characteristic function game (N, v) and consider two agents i,j E N with v(C (()) = v(C U (5}) for all C' CNN (i,j). We have
EVI(G) 1 (VV(G) 1
xNN) n-
(VV - FVV(C),
where the first and the third equality follow by Proposition A.I and the second equality follows because the Shapley Value satisfies symmetry.
For the linearity axiom, consider pair of games C₁ = (N, vi) and C2 (N, v2) with the same set of agents N and the game G = (N, v) whose characteristic funciton is given by e(C) vI(C) v2(C) for all C' S N. Fix an agent E N. We have
EV(() = n
SVi(G) 1 ((1((N) + (2((N)))
n
n 1-1 (VV(G)) - 1 .000( + n- (SV((Z)) - (2((N)))
EV.(G.) EV.(G))
Again, the first and the third equality follow by Proposition A.I and the second equality follows because the Shapley Value satisfies lincarity.
While the Exchange Value does not satisfy the dummy axiom the efficiency axiom, satisfies appropriately modified versions of these axioms.
Proposition A.4. For every characteristic function game G it holds that Σien EV((G) = O. Moreover, if is
dummy agent, i.e., ECCu (i)) VCC) for all C S N (1) then EVi(G) x(N)
Proof. We have
Σ EVI(C)
Σ n
(VV(C)
((())) = =Σ n EN n
SVI(C) n xNN)
IEN
iEN
n
n
v(N)
(NN) = 0,
Tt
n
where we use Proposition
A. and the fact the Shapley Value satisfies efficiency axiom.
Now, fix dummy agent i. We have
EVi(G) 1 -
n
v(N);
again, we use Proposition A.I and fact that the Shapley Value satisties dummy axiom.
A.2 DERIVATION OF CLUSTERING ORJECTIVE STATED EQ.
Inessential games and EVs. The assumption an inessential game commonly made to compute Shap- 2 In an inessential game, the value of a group is given by the sum of the individual contributions of its members, denoted as v(C) = Σieo Vs, where v, is an individual agent's unobserved contribution 24 The EV (sec Definition 4.1 of an individual agent i in inessential game given
ley Values more efficiently
EV. u 1/IN-11
E e -1+ JEN(()
1/1N -1 Σ R/
This expression represents difference between the individual contribution agent i, v;, and the average individual contribution of all other agents. The second term is independent of and remains constant across all agents.
see,, e.g., Covert, I. and Lee, S.I., 2020. Improving kernelshap: Practical shapley value estimation via linear regression, arXiv preprint 2rXiv:2012.01536.
14

---

Published as a conference paper at ICLR 2024
Figure 4: In the Overcooked environments Cramped Room (left) and Coordination Ring (right), agents must cooperate to cook and deliver as many soups as possible within a given time.
Derivation of equivalent clustering objective. We now consider the optimization problem defined by Equation 4 which defines optimal cluster assignments such the variance in EVs maximised
u E arg max,, Var( Var([EV (EVo EV. =V)))(G)).
Further, the clustered value function is defined as
(CC) 1/1 Em= Emeny ((5((m) 1((us 1 Sa(m)) C),
where the normalisation constant is defined 1) Σm Eneryy I(1uş E Sx (m() C). We denote by k the individual contribution of the agent that represents the agents in cluster i. The value k; is defined as the average individual contribution of all agents assigned to the cluster, i.e. k; 1/2 EyEN"; (u((i) = u(j)).
Here, the normalization constant is given as = ΣSEN (u(i) = u(j)).
Using the concept of the clustered value function é can express the EV for all agents assigned cluster as
EV.(G) (1 1/KK-11 b. 1/IXI-1
Σ k2
JEK
The second term, which is cluster-independent, can be omitted when computing the variance Var(IEVo(G).... ........ EV, (C)). as the variance is agnostic to shift in the data distribution. We will omit the scaling factor 1/1K-1 from here onwards.
Let denote the number of agents assigned to cluster j K. with EnO nu = N. By simplifying Equa- tion 4 we obtain:
K-1
Vc((EV(())...... EV = Σ
hi Ejoo my b//
This allows us to express the objective stated in Equation Jas
u E max, Var(Iko...., k--l)).
The objective stated in Equation 4 is therefore equivalent to assigning agents to clusters such that the variance in cluster centroids (centroids computed the mean of the unobserved individual contributions of all agents assigned given cluster) maximized.
Table 3: Dataset statisties in Overcooked.
Imiratinn Cramaped Room Coordination Cramped Room.
Cocecinatin Ring
10.39
40.64
Maximam
B OVERCOOKED EXPERIMENTS
We generate the simulated datascts using the planning algorithm given in Carroll et al. 20019 To be able simulate agents with different behaviors (from adversarial optimal), we first introduce latent trait parameter,
thub. com/ http:///wwwo.oooacooooooaaaaaaa/
15

---

Published as a conference paper at ICLR 2024
Excharge vale (oomataed)
Relation berweon batent behavburr EYs
Coorn netion Ring
Cramped Room
Ovarcukeed within cluster varlance
Relation between 1% data observed and within duster variace
Coordiation ling Varance ance CV.
Cramped fice Veraace Ct:
Cramped Reom: Behasier Ci:
Coondiation Ring Setavior
Lakend
single
Figure 5: Relation between an agent's trait Ả and its EV in Overcooked.
Figure 6: Within-cluster variance in relation to fraction of observations for simulated data in Cramped Room and Coordination Ring (Overcooked). Two cluster- ing methods shown (Behavior clustering and Variance Clustering). In the case of random cluster assignments, the within-cluster variance is 5.11 + 0.11, while under optimal cluster assignments, the variance is 0.156. See section for discussion.
A. which determines the level of adversarial optimal actions for a given agent. value of À represents a policy that always chooses the best action with certainty. As X decreases, agents are more likely select non-optimal actions. For < O, we invert the cost function create agents with adversarial hehavior. Notably, we assign a high cost (or low cost when inverted) to occupying the cell next to the counter in the Overcooked environment. Occupying the cell next to the counter enables adversarial agents to block other agents in the execution of their tasks.
human gameplay datasets, we utilized the raw versions the Overcooked datasets These datasets were used as-is, without manual pre-filtering.
we used either the full set of all possible groups or a fraction of it (see 3 for the relationship between dataset size and EV estimation error). For each observed grouping, we conducted 1O rollouts in the environment and calculated the average score across these rollouts account for stochasticity the environment.
EVs. To estimate agents' EVs according to Section 4.2
Imitation learning. For EV2BC, BC, and group- BC, we used the implementation of Behavior Cloning in Overcooked as given by the authors of (Carroll et al. 2019 We implement the offline multi-agent reinforce- ment learning method OMAR (Pan et al. 2022) using the author's 's implementation For the OMAR baseline, we set the reward at the last timestep the DVF's score for given trajectory, step reward signal is given, in contrast to the standard offline-RI. framework. We conducted a hyperparameter sweep for the following parameters: learning rate with options (0.01, 0.001, 0.0001). Omar-coe with options (0.1, 1, 10), Omar-iters with options (1, 3, 10), and Omar-sigma with options (1,2,3) The best-performing parameters were selected based on the evaluation results.
work assumes that per-
B. CLUSTERING OF AGENTS IN OVERCOOKED
Behavior clustering. The behavior clustering process in the Overcooked environment involves the follow- ing steps. Initially, we identify the 200 states that are most frequently visited by all agents in the given set of observations. As the action space in Overcooked is relatively small (5 actions), we calculate the empir- ical action distribution for each state for every agent. These 200 action distributions are then concatenated to form behavior embedding for each agent. To reduce the dimensionality of the embedding, we apply Principal Component Analysis (PCA), transforming the initial embedding space into three dimensions. Subsequently, we employ the k-means clustering algorithm assign agents to behavior clusters. The number of clusters (3 for Overcooked) is determined using the ELBOW method (Thorndike] 1953), while linear kernels are utilized for both PCA and k- memas.. It noteworthy that the results are found to be relatively inscnsitive to the parameters used in the dimensionality reduction and clustering steps, thus standard implementations are employed for both
aware r ciiee_rr/dattta htt///tt/huuaaa :ttp/aaaa/man//aannnnnnn/
https://github.ccmm//
16

---

Published as a conference paper at ICLR 2024
Take sedpa.
Tide soopt.
Tade 10 sedpp..
Toko ang. -3
Tale ovg, +3
Tado 10
Tale 196
Tide 395
Tale 10%
Talee awg.
Tolon
Take
Mals
Figure 7: Colour-coded ordering of EVs for agents with varying behaviors in Tragedy of the Com- mons. The brighter, the higher an agent's 's contribution to a given value function.
methods (Pedregosa et al, 2011). Importantly, this clustering procedure focuses exclusively on the observed behavior of agents, specifically the actions taken in specific states, and is independent of the scores assigned to trajectories by the DVF.
focuses solely on the scores assigned to trajectories by the DVF and disregards agent hehavior. The objective of variance clustering is maximize the variance in assigned EVs, stated in Equation 4 To optimize this objective, utilize the SLSQP non-linear constrained optimization introduced by Krafi 1988).
EV-Clustering. In contrast to behavior clustering, EV-Clustering (see Section 4.2.1
use soft cluster assignments and enforce constraints ensure that the total probability is equal to one for each agent. The solver is initialized with a uniform distribution and runs until convergence or for maximum of 100 steps. Given that the optimization problem may have local minima, we perform 500 random initializations and optimizations, selecting the solution with the lowest loss (i.e. the highest variance in assigned EVs).
Combining Behavior Clustering and EV Clustering. As described in Sections 4.2.2 and 5.1 behavior clustering (which utilizes behavior information but disregards DVF scores) and variance clustering (which utilizes DVF scores but disregards behavior information) are combined to estimate EVs for degenerate datasets. We initialize the SLSQP solver with the cluster assignments obtained from behavior clustering and introduce small loss term in the objective function of Equation 4 This additional loss term, weighted by 0.1 (selected in a small sensitivity analysis), penalizes deviations from behavior clusters. Similar before, we perform 500 iterations while introducing small amount of noise to the initial cluster assignments at each step. The solution with the highest variance in assigned EVs is then selected.
Ablation study. We present an ablation study examine the impact different components in the cluster- ing approach discussed in Section [5.1 We proposed two sequential clustering methods: hehavior clustering and variance clustering. This ablation study investigates the performance of both clustering steps when performed independently, also under the consideration of the fraction of the data that is observed. We assess performance the within-cluster variance in the unobserved agent-specific latent trait variable A, where lower within-cluster variance indicates higher performance. is important note that À solely used for evaluating the clustering steps and not utilized during the clustering process. The results the ablation study are depicted Figure
We first discuss EV-Clustering. EV-Clustering as introduced in Seciton 4 Igenerally leads to a significant de- crease in within-cluster variance in the unobserved variable X. More specifically, the proposed variance clus- tering approach (when 50% of data is observed), results in 89% reduction of the within-eluster variance X, which validates the approach of clustering agents by their unobserved individual contributions by maxi- mizing the variance estimated EVs. However, we observe Figure 66 that as the fraction of observed data decreases, the within-cluss variance increases, indicating a decrease in the quality of clustering. The highest within-cluster variance is observed when using only single observation ('single-obs' ), which corresponds to fully-anonymizee dataset. This finding is consistent with the fact that fully-anonymized dataset presents degenerate credit assignment problem, as discussed in Section 4.2.2
We now discuss behavior clustering. Figure 6 shows that behavior clustering generally results in a very low within-chu cluster variance. However, it is important to note that these results may world data, as the ablation study uses simulated trajectories. Note that such an ablation study cannot be con- ducted for the given real-world human datasets, these are fully anonymized.
not directly translate to real-
17

---

Published as a conference paper at ICLR 2024
C TRAGEDY OF THE COMMONS EXPERIMENTS
Clustering. model ToC as multi-agent environment where agents consume from a common pool of resources æt, which grows at fixed rate = 25% at each time step I: 2441 max ((1 + g) 2y - E, 0). with Cei as the consumption of the ith agent at time f and Zo = 200 as the starting pool. Hence, if all resources consumed, none can regrow and agents can consume more resources. The Tragedy of the Commons (ToC) environment features 4 different behavior patterns: Take-X consumes X units at every timestep, Take-X- x-dpl consumes units this does not deplete the pool resources, Take X% consumes X% of the available resources, and TakeAvg consumes the average of the resources consumed by the other agents at the previous timestep (0 in the first timestep). For the small-scale experiment of 12 agents, consider three agents for each pattern, with X values selected from the set 1, 3, 10. For the large-scale experiment of 120 agents, we simply replicate each agent configuration respectively, We generate a simulated dataset using agents with four different behavior patterns. We first collect a dataset of observations for small-scale scale experiment of 12 agents and simulate ToC for groups of three agents for 50 time steps (we later consider group of 120 agents).
times. We simulate both experiments for groups size and 10,
Due the continuous nature of the state and action spaces in ToC. we first discretize both and then apply the same clustering methods used in the Overcooked scenario. proceed by computing EVs for all agents as done in Overcooked (see Figure for results). We implement imitation policies by replicating the averaged action distributions in the discretized states.
C.I COMPUTATIONAL DEMAND.
We used an Intel(R) Xeon(R) Silver 4116 CPU and NVIDIA GeForce GTX 1080 Ti (only for training BC, EV2BC, group-BC. and OMAR policies). In Overcooked, generating dataset took maximum of three hours, and estimating EVs from given dataset takes a few seconds. Behavior clustering consumes a couple of minutes, while Variance clustering takes up to two hours per configuration (note that it is run 500 times). Training of the BC, group-BC. BCC and EV2BC policies took no OMAR baseline was trained for up to hours. Tragedy Commons, each rollout only consumes couple of seconds. Clustering times were comparable to those in Overcooked. Computing imitation policies is similarly only a matter of a few minutes.
than 30 minutes (using GPU), while the
D STARCRAFT EXPERIMENTS
Dataset generation. For each environment configuration, we first train agents with the ground truth reward function for different seeds. We then extract agents at three different timesteps (0%, 50%, and 100%) of the training. We randomly sample groups of the extracted agents to generate a large set of trajectory rollouts and record the average score achieved by group, which serves the observed DVF score. We anonymize the datasct by assigning one-time-use IDs all agents.
Clustering of agents and EV computation. We first extract high ditheiimnnsionll features from agent tra- jectories using Term Frequency-Ine Inverse Document Frequency (TF-IDF (Spärck Jones) 1972)) and compute agent clusters. We compute large number of possible cluster assignments using different hyperparameters for TF-IFF and different hyperparameters for spectral elustering. accordance with the objective of EV cluster- ing, we then choose the cluster assignment that results in the largest variance in EVs across all agents.
For each individual agent, we extract the action sequence (up to a cutoff of either 1000 or 50000 enviromment steps). We then convert the action sequence into string to apply ID-IDF. This can be thought of as representing each action individual word. This results in corpus that contains one document per agent.
For the transformation of the corpus into fcature space, we apply the TF-IDF vectorization, parameterized by:
Min.df: Minimum document frequency set at 0.05, ensuring inclusion of terms present in at least 5% of the documents.
Max_dff Maximum document frequencies tested were 0.3 and 0.9, to filter out terms excessively common across documents.
Max fcature:::: The upper limit the number of features considered were set to 100, 000 000, 000.
Ngram range: We explored n-gram ranges of (1,3), (1, and (1, 10), capture varying lengths of term dependencies.
Post TF-IDF vectorization, we use spectral clustering to achieve cluster assignments. We compute clustered EVs for each possible cluster assignment and choose the cluster assignment with the highest variance in EVs across all agents. The computed EVs for the highest-variance cluster assignment are then used as the estimated EVs for all agents.
18

---

Published as a conference paper at ICLR 2024
Imitation with EV2BC. We apply EV2Bc filtering for agents with high estimated BVs. We found that pre-filtering for trajectories with high collective scores significantly improves performance. In other words, we filter for agents with high estimated EVs in trajectories with high collective scores. We assume that this is the case as some agents might 'block' any progress in trajectories with low collective scores.
19


Reviews:
[{"review_id": "AvKPrkyCIS", "reviewer": "Reviewer_VYWJ", "summary": "The paper \"Who to Imitate: Imitating Desired Behavior from Diverse Multi-Agent Datasets\" proposes a novel imitation learning framework. It enables AI agents to learn desirable behaviors from large, mixed-quality multi-agent datasets by using a metric called Exchange Value (EV) to evaluate and imitate only those agents contributing positively to collective outcomes. The technique involves EV-Clustering to handle incomplete data and an Exchange Value based Behavior Cloning (EV2BC) method for learning policies aligned with desired outcomes. The approach is shown to outperform baselines and has applications in aligning AI behavior with human values in complex environments.", "questions": "1. Given that the quantification of what is considered desirable behavior is central to the proposed framework, can the authors provide additional insights into how the Desired Value Function (DVF) is defined and quantified across different environments and datasets?\n\n2. The paper assumes consistent behavior from individual agents across multiple trajectories. Could the authors discuss the potential implications of this assumption in environments where agent behavior is more dynamic and context-dependent?\n\n3. The paper suggests the potential for utilizing data on undesired behavior more effectively. Could the authors elaborate on possible approaches for leveraging this type of data to enhance the imitation learning process?\n\n4. How does the framework adapt to different environments, and what are the limitations when applying the proposed EV-Clustering and EV2BC methods to datasets that significantly differ from the ones used in the experiments?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Innovative Metric for Agent Evaluation: The introduction of Exchange Values (EVs) as a metric to compute individual agents' contributions to a collective value function is a significant contribution. EVs offer a method for identifying and imitating desirable behaviors within multi-agent systems, providing a novel way to approach imitation learning.\n\nEffective Handling of Incomplete Data: The paper presents EV-Clustering, a method that estimates contributions from incomplete datasets. This addresses a common challenge in real-world scenarios where datasets are rarely comprehensive, enabling more accurate modeling of agent behavior.\n\nAlignment with Desirable Outcomes: Through Exchange Value based Behavior Cloning (EV2BC), the paper proposes a mechanism to align the learning process with a Desired Value Function (DVF). This ensures that the learned policies reflect desirable outcomes, which is crucial for the practical application of AI systems trained on human data.", "weaknesses": "1. **Quantification of Desirability**: The process of quantifying the desired value function (DVF) is a complex task and an active area of research. The paper's methods depend on the DVF to guide the imitation learning process, so any limitations in accurately defining this function could impact the effectiveness of the approach.\n\n2. **Assumption of Consistent Agent Behavior**: The framework assumes that individual agents behave similarly across multiple trajectories. This assumption may not always hold true in complex, dynamic environments where agent behavior can vary significantly based on context.\n\n3. **Utilization of Undesired Behavior Data**: The paper points out that further research could explore how to utilize data on undesired behavior more effectively, such as developing policies that are explicitly constrained to avoid undesirable actions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5Aj2Hs5Up", "reviewer": "Reviewer_CrLG", "summary": "### Problem Statement\nThe paper tackles the challenge of extracting and imitating desirable behaviors from large multi-agent datasets, where desirability is quantified via collective trajectory scores. The problem arises as these scores reflect collective outcomes, making it difficult to ascertain individual agents' contributions, especially in real-world scenarios with fixed group sizes, incomplete datasets, and fully anonymized data.\n\n### Main Contributions\nThe key contributions include\n1. The introduction of \"Exchange Values\" (EVs) to quantify an individual agent's contribution to collective desirability.\n2. The proposal of \"EV-Clustering\" to estimate these values from incomplete datasets\n3. The development of \"Exchange Value based Behavior Cloning\" (EV2BC), a method that selectively imitates agents with high EVs estimated from anonymized data, thus aligning learned policies with desired characteristics, outperforming relevant baselines.\n\n### Methodology\nThe authors propose \"Exchange Values\", a modification to Shapley value computation that compares the Desired Value Function values between agent groups of the same size, making it amenable to games that have group size constraints. Based on the Exchange Values, clustering of agents can be done by maximizing inter-cluster EV variance, which is particularly useful for fully-anonymized data. Behavior cloning (BC) can be then confined to only mimicking agents with high Exchange Values.\n\n### Experiments\nTwo environments are used to evaluate the methods, namely the \"Tragedy of Commons\" and \"Overcooked\". Both synthesized and human generated data are used. The experiments show that the estimated EV values are meaningful and superior BC performance is attained with the guidance of EV for selecting trajectories to imitate.", "questions": "- What does the $m$ in line 220 denote? Is it a fixed value or can it take multiple possible values (since $m \\in M$). Why must $k \\geq m$? Should $k$ change when $m$ takes a different value?\n- Could authors further explain the definition of the \"cluster-centroid agents $C \\subseteq K$\" in line 223?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "### Originality and Significance\nEvaluating the agent quality / contribution from desirability scores of collective trajectories is a very realistic and meaningful problem. The proposed method is well-motivated and elegantly extending the well-known Shapley Value, which is innovative.\n\n### Quality\nThe problems the authors address are important and practical and the questions they try to answer are insightful.\n\n### Writing\nThe mathematical explanations of complex concepts are precise and consistent. In addition, the authors provide insightful intuition to help readers understand.", "weaknesses": "### Limited Environments\nOnly two environments are studied, while there are many environments that can further highlight the real-world value of the proposed method, e.g. public traffic.\n\n### Lack of more theoretical analysis of the properties of EV\nShapley Values are know to have good properties, e.g. symmetry, dummy (zero value for null players), additivity etc, which make it interpretable, appealing, and useful. It would be interesting to see analysis of Exchange Values with respect to these properties.\n\n### Lack of interpretation of EV\nMore detailed analysis can be added to the main text with respect to how different EV values can be connected to various behavior patterns. In particular, I think the $\\lambda$ values in both \"Tragedy of Commons\" and \"Overcooked\" can be linked to the estimated EVs to validate the method.\n\n### Lack of baseline\nI understand that this is the first work tackling this specific problem setup, but I'm interested to see whether Shapley values could be similarly useful for guiding imitation learning in multi-agent dataset when the group size constraint is absent (which should be possible in many cases, e.g. the Tragedy of the Commons).\n\n### Writing\nAlthough I in general enjoyed reading the paper, I still find many sentences throughout the article a bit repetitive and convoluted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jjhkLucGV9", "reviewer": "Reviewer_e3wr", "summary": "This work proposes a way to imitate the correct agents in a multi-agent setting, where correctness is measured by how an agent impacts the collective. This is termed the exchange value, and is formally presented as a way to quantify an agent’s contribution as the expected change in desirability when substituting the agent randomly. EVs require that different combination of agents are seen in the dataset. To counteract low-data regimes, EV-clustering is proposed. With this exchange values estimated for all agents in a dataset, the authors device EV2BC method learns behaviour cloning policies good agents only. Evaluation is performed on tragedy of commons and on the dataset collected from the Overcooked environment with diverse agent behaviours.", "questions": "- I'm a little confused about some experimental details. Specifically, the number of agents in the datasets, and the exact composition of the dataset is unclear. Can you clarify the composition of the $D^\\text{adv}$? Are there really two different types of agents, but $n=100$ agents?\n\n- In my opinion, the paper would be better organized by moving more results from the appendix into the main paper. One simple way of improvement is to move Figure 3 to the appendix and add the ablation study on the EV-clustering vs. Behavior clustering.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The work is well motivated and seems to solve practical issues in Shapely Values. \n\n- The presentation of the work is clear enough, and relatively easy to follow. I thought Sections 3 and 4 to be quite well written.\n\n- The results seem quite strong and convincing compared to other baseline methods. I thought Figure 4 (right) was quite convincing in showing the importance of clustering for the degenerate case.", "weaknesses": "- How scalable would this method be to let's say a dataset of multi-agent driving scenes? It seems to me like scalability is an issue here, specifically due to clustering. this brings me to an important point, the weaknesses of the proposed approach should be addressed. \n\n- I'm still left confused by the differences between behavioural clustering and EV-clustering. I understand the differences in the approaches, but the ablation study seems to point to behavioural clustering being more stable in low-data regiments. I see that the ablation study says to look at section 5.1 to show why behavioural clustering is not sufficient by itself, but I do not see the supporting results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Peh6crvcpS", "reviewer": "Reviewer_wrnL", "summary": "This paper addresses the problem of learning aligned imitation policies from multi-agent datasets containing unaligned agents. The authors argue that existing methods for estimating demonstrator expertise in single-agent settings do not translate to the multi-agent setting due to the challenge of credit assignment. This paper proposes a method for learning aligned imitation policies that takes into account the collective value function of the agents. Empirical evidence shows that their proposed method outperforms relevant baselines.", "questions": "Is EV a good measure if there exist complex/bipolar dynamics between agent behaviors? e.g. two agents work well if they are both in the team but horribly if only one present?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Learning human-aligned policies from a mixed multi-agent dataset is an important area of research that is relevant to a diverse set of applications, including autonomous driving. The proposed method takes into account the collective value function of the agents and is designed to address the challenge of credit assignment in multi-agent settings. \n\nThis work introduces a new metric called the Exchange Value (EV), which is used to estimate the individual contributions of agents to the collective value function. The paper provides empirical evidence that the proposed method outperforms relevant baselines, by showing that it can be applied to a social dilemma game and a cooperative task.", "weaknesses": "The proposed method assumes that the collective value function can be expressed as a sum of individual contributions. The authors should comment more on the class of problems that this is applicable to. \n\nExperiments evaluate the proposed method on a limited set of environments and tasks, and it is unclear how well the method would generalize to other domains and tasks.The authors motivated with a mixed driving dataset, and it would be useful to see how this method applies to driving benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WLPpbz4HbC", "reviewer": "Reviewer_fLoZ", "summary": "This paper proposes a method for imitating desired behavior from diverse multi-agent datasets. It introduces the concept of Exchange (EVs), which measures the individual contribution of an agent to a collective value function, and shows how they are related to Shapley Values. It further develops EV-clustering, a technique to estimate EVs from incomplete or anonymized datasets, by maximizing the variance of EVs across clusters of agents. It presents EV-based Behavior Cloning (EV2BC), an imitation learning method that only imitates the actions of agents with high EVs, and demonstrates its effectiveness in two domains: Tragedy of the Commons and Overcooked.", "questions": "- How do you deal with the uncertainty or variability in the EV estimates, especially when the data is incomplete or anonymized? How robust is your method to noise or outliers in the data?\n- How do you justify the choice of the DVF for each domain? How do you ensure that the DVF is aligned with the desired behavior and does not have any unintended consequences or biases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- It introduces the concept of Exchange Values, which measure the individual contribution of an agent to a collective value function, and shows how they are related to Shapley Values.\n- It develops EV-Clustering, a technique to estimate Exchange Values from incomplete or anonymized datasets, by maximizing the variance of Exchange Values across clusters of agents.\n- It presents EV based Behavior Cloning, an imitation learning method that only imitates the actions of agents with high Exchange Values, and demonstrates its effectiveness in two domains: Tragedy of the Commons and Overcooked.", "weaknesses": "- For the results, it is necessary to provide some reference for a better understanding of the performance, e.g., reporting the results using the ground truth identification, or the results using the shapely value or other credit assignment methods. \n- The paper uses a limited number of environments and datasets to evaluate the proposed method and does not consider more complex or realistic scenarios that involve heterogeneous agents, partial observability, communication, or coordination, e.g., applying the exchange value in proactive multi-camera cooperation[1] or SMAC[2].\n\nReference:\n[1] Ci, Hai, et al. \"Proactive Multi-Camera Collaboration For 3D Human Pose Estimation.\"  ICLR 2023.\n\n[2] Wen, Muning, et al. \"Multi-agent reinforcement learning is a sequence modeling problem.\" NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)