Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased
Authors: Chao Yu
Jiaxuan Gao
Weilin Liu
Botian Xu
Hao Tang
Jiaqi Yang
Yu Wang
Yi Wu
Venue: ICLR 2023 poster (2023)
Keywords: Human-AI Collaboration
Multi-Agent Reinforcement Learning

Abstract: There is a recent trend of applying multi-agent reinforcement learning (MARL) to train an agent that can cooperate with humans in a zero-shot fashion without using any human data. The typical workflow is to first repeatedly run self-play (SP) to build a policy pool and then train the final adaptive policy against this pool. A crucial limitation of this framework is that every policy in the pool is optimized w.r.t. the environment reward function, which implicitly assumes that the testing partners of the adaptive policy will be precisely optimizing the same reward function as well. However, human objectives are often substantially biased according to their own preferences, which can differ greatly from the environment reward. We propose a more general framework, Hidden-Utility Self-Play (HSP), which explicitly models human biases as hidden reward functions in the self-play objective. By approximating the reward space as linear functions, HSP adopts an effective technique to generate an augmented policy pool with biased policies. We evaluate HSP on the Overcooked benchmark. Empirical results show that our HSP method produces higher rewards than baselines when cooperating with learned human models, manually scripted policies, and real humans. The HSP policy is also rated as the most assistive policy based on human feedback.

Published as a conference paper at ICLR 2023
LEARNING ZERO-SHOT COOPERATION WITH HU- MANS, ASSUMING HUMANS ARE BIASED
Chao Yu' Jiaxuan Gao Weilin Liu', Botian Xu², Hao Tang', Jiaqi Yang", Yu Wang'! 1, Yi Wu Tsinghua University, 2 UC Berkeley, Shanghai Qi Zhi Institute
zoeyuchao gmail. com
ABSTRACT
There is recent trend of applying multi-agent reinforcement learning (MARL) train an agent that can cooperate with humans zero-shot fashion without using any human data. The typical workflow is to first repeatedly run self-play (SP) to build a policy pool and then train the final adaptive policy against this pool. A crucial limitation of this framework is that every policy in the pool is optimized w.r.t. the environment reward function, which implicitly assumes that the testing partners of the adaptive policy will be precisely optimizing the same reward function as well. However, human objectives are often substantially biased according to their own preferences, which can differ greatly from the environment reward. We propose a more general framework, Hidden- which explicitly models human biases hidden reward functions in the self-play objective. By approximating the reward space as linear functions, HSP adopts an effective technique to generate an augmented policy pool with biased policies. We evaluate HSP on the Overcooked benchmark. Empirical results show that our HSP method produces higher rewards than baselines when cooperating with learned human models, manually scripted policies, and real humans. The HSP policy is also rated the most assistive policy based human feedback.
Vidden-Utiltty Utility Self- (HSP),
INTRODUCTION
Building intelligent agents that can interact with, cooperate and assist humans remains a long- standing Al challenge with decades of research efforts (Klien et al. 2004 Ajoudani et al. 2018 Dafoe al. 2021 Classical approaches are typically basel-based, which (repeatedly) build an eitective behavior model over human data and plan with the human model (Sheridan 2016 Carroll jet al., 2019 Bobu et al., 2020 Despite great successes, this model-based paradigm requires an expensive and conscmunu data collection process, which can be particularly problematic for complex problems tackled by today's 's Al techniques (Kidd & Breazeal 2008 Biondi et al., 2019) and may also suffer from privacy issues (Pan et al. 2019).
Recently, multi-agent reinforcement learning (MARL) has become promising approach for many challenging decision-make making problems. Particularly in competitive settings, Als developed by MARL algorithms based on self-play defeated human professionals in a variety of domains (Sil- ver et al., 2018 Vinyals et al., 2019) |Berner et al. (019). This empirical evidence suggests a new direction of developing strong AIs that can directly cooperate with humans in similar "model-free" fashion, i.c.. via self-play.
Different from zero-sum games, where simply adopting Nash equilibrium strategy is sufficient, an obvious issue when training cooperative agents by self-play is convention overfitting. Due to the existence of a large number of possible optimal strategies in a cooperative game, SP-trained agents can easily converge to particular optimum and make decisions solely based on a specific behavior pattern, i.e., convention (Lowe et al., 2019 Hu et al.) 2020). of its co-trainers, leading to poor generalization ability unseen partners. To tackle this problem, recent works proposed two-staged framework by first developing a diverse policy pool consisting of multiple tr-trndded policies, which possibly cover different conventions, and then further training an adaptive policy against this policy pool (Lupu et al., 2021 Strouse et al. 2021 Zhao et al. 2021).
Despite the empirical success of this two-staged framework, a fundamental drawback exists. Even though the policy pool prevents convention overlitting, each SP-trained policy in the pool remains solution, which is either optimal or sub-optimal, fixed reward function specified by the under- lying cooperative game. This implies crucial generalization assumption that any test-time partner
"Equal Contribution
I Equal Advising

---

Published as a conference paper at ICLR 2023
will be precisely optimizing the specified game reward. Such assumption results in pitfall in the case of cooperation with humans. Human behavior has been widely studied in cognitive sci- ence (Griffiths 2015). economies (Wilkinson & Klaes] 2017) and game theory (Fang et al. (021). Systematic research has shown that humans' utility functions can be substantially biased even when a clear objective is given (Pratt 1978 Selten 1990; Camerer 201T Barberis [2013), suggesting that human behaviors may be subject to unknown reward lunction that is very different from the game reward (Nguyen et al. 2013). This fact reveals an algorithmic limitation of the existing SP-based methods.
In this work, we propose Hidden- Utility Self-Play (HSP), which extends the SP-hased two-staged framework to the assumption of biased humans. HSP explicitly models the human bias via an additional hidden reward function in the self-play training objective. Solutions to such a generalized formulation are capable of representing any non-adaptive human strategies. We further present tractable approximation of the hidden reward function space and perform a random search over this approximated space when building the policy pool in the first stage. Hence, the enhanced pool can capture a wide range of possible human biases beyond conventions (Hu et al.] [020] Zhao et al.) 2021] and skill-levels (Dafoe et al. (021)) w.r.t. the game reward. Accordingly, the final adaptive policy derived in the second phase can have a much stronger adaptation capability to unseen humans.
We evaluate HSP in popular human-Al AI cooperation benchmark, Overcooked (Carroll al., 2019), which is a fully observable two-player cooperative game. We conduct comprehensive ablation stud- ies and comparisons with baselines that do not explicitly model human biases. Empirical results show that HSP achieves superior performances when cooperating with behavior models learned from human data. In addition. we also consider a collection of manually scripted biased strategies, which are ensured to be sufficiently distinct from the policy pool. and HSP produces an even larger performance improvement over the baselines. Finally, we conduct real human studies. Collected feedbacks show that the human participants consistently feel that the agent trained by HSP is much more assistive than the baselines. We emphasize that, in addition to algorithmic contributions, our empirical analysis, which considers learned models, script policies and real humans as diverse test- ing partners, also provides a more thorough evaluation standard for learning
Als.
2 RELATED WORK
There is a broad literature on improving the zero-shot generalization ability of MARL agents to unseen partners (Kirk et al. 202 D. Particularly for cooperative games. this problem is often called ad hoc team play (Stone et al. 2010) zero-shot cooperation (ZSC) (Hu et al. 2020). Since most existing methods are based on selt-play (Rashid et al. 2018 Yu et al. 2021). how to avoid conven- tion overfitting becomes a critical challenge in ZSC. Representative works include improved policy representation (Zhang et al. 2020. Chen et al. 2020), randomization over invariant game struc- tures (Hu et al. 2020 Treutlein et al. 2021). population-based
training (Long et al., 2020 Lowe let al. 2020 Cui et al. 2021) and belief modeling for partial observable settings (Hu et al. [222; Xie
al.
2021). Fichiioos co-play (FCP) (Strouse et al. 202T) proposes a two-stage Tramework by lirst
creating a pool of self-play policies and their previous versions and then training an adaptive policy against them. Some techniques improves the diversity of the policy pool (Garnelo et al. 202T Liu
et al.
2021 Zhao et al. 2021 Lupu et al. 2021) for a stronger adaptive policy (Kottt et al. 202
We follow the FCP framework and augment the policy pool with biased strategies. Notably, tech- niques for learning robust policy in competitive games, such policy ensemble (Lowe et al.,
2017). adversarial training (Li al. 2019) and double oracle (Lanctot et al.) 2017). are complemen-
tary to our focus.
Building Als that can cooperate with humans remains a fundamental challenge in Al DDaooe et al.
2021). A critical issue is that humans can be systematically biased (Camerer 2011 Russell 2019)
Hence, great efforts have been made to model human biases. such as urrationality (Selten 1990
Bobu et al. 2020 Laidlaw & Dragan 2022). , risk aversion (Prati) 1978, Barberis 2013), and my-
opia (Evans et al. 2016). Many popular models further assume humans have hidden subject utility functions (guuen et al. 2013) Hadfield- Menell et al.) 2016, Eckersley 2019; Shah et al.) 2019). Conventional methods for human-Al col cooorattt require an accurate behavio model over human data (Ajoudani et al.) 2018 Kwon et al.) 2020 Kress- Gazit et al.) 2021 Wang et al. 2022). while consider the setting of no human data. Hence, explicitly model human biases as hidden utility function in the self-play objective to reflect possible human biases beyond conventions w.r.t. optimal rewards. We prove that such a hidden-utility model can represent any strategy of non- adaptive humans. Notably, it is also feasible to generalize our model to capture higher cognitive hierarchies (Canerer et al.) 2004). which we leave as a future direction.
We approximate the reward space by a linear function space over event-based features. Such linear representation is typical in inverse reinforcement learning (Ng & Russell 2000), policy trans-
2

---

Published as a conference paper at ICLR 2023
fer (Barreto al. 2017b), evolution computing (Cully et al. 2015) and game theory (Winterfeldt & Fischer [175]] Kiekintveld et al., 2013). Event-based rewards are also widely adopted as general design principle in robot learning (uu et al. 2018 Zhu al. 2019 Ahn et al. (022). We perform randomization over feature weights to produce diverse biased strategies. Similar ideas have been adopted in other settings, such as generating adversaries (Paruchuri et al. 2006), emergent team- formation (Baker ) and searching for diverse Nash equiibria in general-sum games (Tang) et al.] 2020p. In our implementation, out duplicatdd policies, which is inspired by the quality diversity method (Pugh et al.. 2016). There are also some works utilizing model- based methods to solve zero-shot cooperation Wu et al. (2021). Their focus is orthogonal to our approach since they focus more on constructing an adaptive agent, while our approach aims to find more diverse strategies. Besides, we adopt an end-to-end fashion to train adaptive agent, which is more general. Lastly, our final adaptive agent assumes zero-shot setting without any data from its testing partner. This can be further extended by allowing meta- adaptation at test time (Charakorn et al. 2021 Gupta et al. 2021 Nekoei et al. (02I), which we leave as a future direction.
use multi-reward signals as approximate metric to filter
3 PRELIMINARY
Two-Player Human-AI Cooperative Game: A human-AI cooperative game defined on world model, i.e., a two-player Markov decision process denoted by M = (S,A, P. R), with one player with policy TA being an Al and the other with policy TH being a human. S is a set of world states. A is a set of possible actions for each player. P is a transition function over states given the actions from both players. R is a global reward function. A policy Ti produces an action a E A given a world state 8t E S at the time step t. We use the expected discounted return JTAA, TH) = E aa E,y' de
(A) _ab ) as the objective. Note that H AA) can be simi-
larly defined, and we use A TH) for conciseness without loss of generality. Let PH II [0. 1] be the unknown distribution of human policies. The goal to find policy π that maximizes the expected return with an unknown human, i.e., ETH~PH [J(TH: x=)]] In practice, many works con-
struet or learn a policy distribution Pи to approximate real-word world human behaviors, leading to an approximated objective for т A. i.e., E B [J ( R.).
Self-Play for Human-Al Cooperation: Self-play (SP) optimizes ( R2) with two parametric policies I and T2 and takes I as A without use of human data. However, SP suffers from poor generalization since SP converges to a specific optimum and overfits the resulting behavior conven- tion. Population-based training (PBT) improves SP by representing πT₃ as a mixture of K individual
policies T:
(4)) [_ and runs cross-play between policies by optimizing the expected return (Long*
et al. Low) et al. |020 Cui et al.
(021). PBT can be further improved by adding a diversity Lupu et al., 2021).
bonus over the population (Gareeoo et al.
2021
Liu al. 2021
Fictitious Co-Play (FCP): FCP (Strouse et al.
2021 is recent work on zero-shot human-AI FCP extends PBT via two-stage framework. In ((T(*) , = = )
cooperation with strong empirical performances.
by optimizing ( (k) .T (k)
the first stage, FCP trains K individual policy pairs
_(k)) may quickly converge to distinct local optimum. Then for each k. Each policy pair (m1 ,52
FCP constructs a policy pool II = {=
(k) (
m2 ===1 with two past versions of each converged SP :
K
policy nI denoted by 72 In the second stage, FCP constructs a human proxy distribution PH: by randomly sampling from II, and trains TA by optimizing E ( ). We remark
that, for better cooperation, the adaptive policy aA should condition on the state-action history in an episode to infer the intention of its partner. Individual SP policies ensure P# contains diverse conventions while using past versions enables P!x to cover different skill levels. So, the final AA can be forced to adapt to humans with unknown conventions or sub-optimalities. Maximum Entropy Population- -basd Training (MEP) (Zhao et al., 2021) is the latest variant of FCP, which adopts the population entropy as a diversity bonus in the first stage to improve the generalization of the learned A.
COOPERATING WITH HUMANS IN Overcooked: A MOTIVATING EXAMPLE
Overcooked Game: Overcooked (Carroll et al., 2019) is fully observable two-player cooperative game developed as a testbed for human-Al cooperation. In Overcooked, players cooperatively ac- complish different soup orders and serve the soups for rewards. Basic game items include onions, tomatoes, and dishes. An agent can move in the game or "interact" to trigger some events, such as grabbing/putting an item, serving soup, etc., depending on the game state. To finish order, players
3

---

Published as a conference paper at ICLR 2023
Asymmetric Advantages
Coorclination Ring
Counter Circuit
Distani Tomaño
Many Ohders
Onion
Onious
Tomsto
Tomatoes
Dish
Disbes
Pot
Serving Area
Player
Player
Time
6 20
$ 20
201
20 20
Order
10
Figure I: Layouts in Overcooked. From left right are Asymmetric Advantages, Coordination Ring, Counter Circuit, Distant Tomato and Many Orders respectively, with orders shown below.
should put a proper amount of ingredients into the pot and cook for some time. Once a soup is fin- ished, players should pick up the soup with a dish and serve it to get a reward. Different orders have different cooking times and different rewards. Fig. demonstrates five layouts the first three onion-only layouts are adopted from (Carroll et 2019), while the latter two, Distant Tomato and Many Orders, are newly introduced to include tomato orders to make the problem more challenging: an Al needs to carefully adapt its behavior to either cook onions or tomatoes according to the other player's ' actions.
consider, where
A Concrete Example of Human Preference: Fig. illustrates motivating example in Distant Tomato (the 4th layout in Fig. [ There are two orders: requires three tomatoes. We run FCP on this multi-order scenario, and all the policies the FCP policy pool converge to the specific pattern of only cooking onion soup (Fig. 2a). Hence, the final adaptive policy by FCP only learns to grab onions and cook onion soups. Cooking tomato soup is a sub-optimal strategy that requires many extra moves, so the onion-only policy pool is exactly the solution to the FCP self-play objective under the enviromment reward. However, it is particularly reasonable for human to dislike onions and accordingly only grab tomatoes in game. To be an assistive AI, the policy should adapt its strategy to follow the human preference for tomatoes. On the contrary, as shown in Fig. 2b) the FCP policy completely ignores human moves for tomatoes and even results in poor cooperation by producing valueless wrong orders of mixed onions and tomatoes. Thus, make an FCP agent human-assistive the first-stage policy pool should not only contain optimal strategies (i.e., onion soups) of different conventions but also cover diverse human preferences (e.g., tomatoes) even if these preferences are sub-optimal under the environment reward.
requires three onions, and the other
What
Teunslo
Cnioon.
to didt
Soup
Homan
FCP
Onion
(a) FCP-FCP
(b) FCP's failure case when cooperating with a human player
Figure 2: Motivating example. (a) FCP converges the optimal onion soup strategy. (b) A failure
case of FCP with a human partner: FCP agent corrupts the human's plan of cooking tomato soups.
METHODOLOGY
We introduce a general formulation to model human preferences and develop a tractable learning objective (Sec. (.2). The algorithm, Hidden- Hidden-Urility Self-Play (HSP), is summarized in Sec. [.3
5.1 HIDDEN-UTILIT MARKOV GAME
The key insight from Sec. 4 is that humans may not truthfully behave under the environment reward. Instead, humans are biased and driven by their own utility functions, which are formulated below.
Definition: A two-player hidden utility Markov game is defined as (S, P. Rı). (S,A, corresponds the original game MDP with R, being the task reward function. Rw: denotes an additional hidden reward function. There are two players, Tas whose goal is maximize the task reward RG, and Tw. whose goal is to maximize the hidden reward Rng R, is only visible to Tw-
Let [TT, R2) denote the expected return under reward R with a policy कi and 72. During self-play, Ta optimizes J(T Fnd Re, while Tw optimizes J( Te (Rw). A solution policy pro- file (Iā Th)) to the hidden utility Markov game is now defined by Nash equilibrium (NE): [ Th (R,)) 2 ( <"u, ), V and J ( T (R,) 2 [ " Re), VT'
Intuitively, with a suitable hidden reward function R,,, we can obtain any possible (non-adaptive and consistent) human policy by solving the hidden-utility game induced by R...
Lemma 5.1. Given an MDP M AS, P, Re), for any policy π S A - [0. 1). there exists hidden reward function R,, such that the two-player hidden utility Markov game M' (S A, P, Ro, R2) has Nash equilibrium (Th at) where xt
4

---

Published as a conference paper at ICLR 2023
Lemma 5.1 connects any human behavior to hidden reward function. Then the objective of the adaptive policy A in Eq. (3) can be formulated under the hidden reward function space R. as follows.
O, there exists a mapping Twe where Ha Rww) denotes the derived policy Th in the NE of the hidden utility Markov game M (S. A. R,e Re) induced by Rw and distribution PR- R + [0, 1] over the hidden reward space R, such that, for any adaptive policy AA E arg max ER [ Fwo (Rw))) AA approximately maximizes the ground-iruth objective with at most e gga, i.e., Ex - Pn [TTA TH) 2 max,' Exy~Py [J TH)]
Theorem 5.1. For any
Theorem 5.1 indicates that it is possible to derive diverse human behaviors by properly designing a hidden reward distribution Pn. which can have a much lower intrinsic dimension than the policy distribution. In Overcooked, human preferences can be typically described by few features, such as interaction with objects or certain type of game events, like finishing an order or delivering a soup. By properly approximating the hidden reward distribution as PR. the learning objective becomes,
A = arg max E R._ Pn [T (R.o))
Eq. ① naturally suggests two-staged solution by first constructing policy pool ( (R) R ~ PRt from Pn and then training A to maximize the game reward w.r.t. the induced pool.
5.2 CONSTRUCT A POLICY POOL OF DIVERSE PREFERENCES
The fundamental question is how to design a proper hidden reward function space R. In general, a valid reward space is intractably large. Inspired by the fact that human preferences are often event-centric, we formulate R. as linear functions over event features, namely R = { R... Rw (8. 01 02) = Φ(s, 01, 0z)²
Event-based Reward Function Space:
(2)"u, w. ||wl|s> I Cmmx) Cnux is a bound on the feature weight wr while Ø S x A x A - RT specifies occurrences of different game events
when taking joint action (a1, (2) at state
Derive Diverse Set of Biased Policies: We simply perform random search over the feature weight w to derive set of diverse behaviors. We first draw N samples (w(( ]EE[ for the feature weight w where w is sampled uniformly from set of values Cj, leading to a set of hidden reward
]E[[)) For each hidden reward function R!! (1) of the hidden utility Markov game induced by R!f through self-play. The above process produces policy pool { He([]] that can cover wide range of behavior preferences.
functions {R(() R.()) (s, 01, 02) $(s, (2,,a))"u (1)
we find an approximated NE, Tw Tu
(3)
Policy Filtering: We notice that the derived pool of- ten contains a lot of similar policies. This is because the same policy can be optimal under set of reward functions. which is typical multi-objective optimiza- tion (Chugh et al. 2019 Tabatabaei et al., 2015). Du- plicated policies simply slow down training without any help to learn π. A. For more efficient training, we adopt behavior metric, i.e., event-based diversity, to only keep
Algorithm 1: Greedy Policy Selection
S <- [ia] where in ~ [N]: for = 1 1 K - 1 do
K' + arg max ED(SU
S SU():
end
let EC( denote the expected event count, L.e E 0(5t, 0z) TT =a We define event-based diversity for subset S C [N] by normalized pairwise EC differences, i.e., ED(S) = Eijes Ee G |EC| - ECτ"), where e is a frequency normalization constant. Finding subset S of size K with the optimal ED can be expensive. We simply adopt a greedy method in Algo. [ to select policies incrementally.
distinct ones from the initial pool. For each biased policy गh
Algorithm 2: Hidden-Utility Self-Play
5.3 HIDDEN-UTILIT SELF-PLAY
for
1 + N do
Given the filtered policy pool, we train the final adap- tive policy A over rollout games by A and randomly sampled policies from the pool, which completes our overall algorithm HSP in Algo. [2]
Train Te and (0)) under sampled r!):
end Run Algo. to only keep K policies; Initial policy .:: repeat
We implement HSP using MAPPO (Yu et al. 2021) as the RI algorithm. In the first stage, we use MLP policies for fast SP convergence. In practice use
Rollout with nA and sampled (()); Update π until enough iterations;
5

---

Published as a conference paper at ICLR 2023
half of the policy pool train biased policies and the other half to train MEP policies (Zhao et al.
2021 under the game reward. This increases the overall pool towards the game reward, leading
to improved empirical performances. For the final adaptive training, as suggested in (Tang et al,,
2020), , we add the identity of each biased policy as an additional feature to the critic. For event- based features for the reward space, we consider event types, including interactions with basic items and events causing non-zero rewards in Overcooked. Full implementation details can be found in Appendix D [
EXPERIMENTS
Baselines. We compare HSP with other SP-hased baselines, including Fictitious Co-Play (FCP), Maximum Entropy Population-based training (MEP), and Trajectory Diversity-based PBT (Traj- Div). All methods follow a two-stage framework with final pool size of 36, which we empirically verified to be sufficiently large to avoid performance degradation for all methods. More analysis on pool size can be found in Appendix F.2.1 The implementation details of baselines can be found in Appendix D.2 Each policy is trained for 100M timesteps for convergence over 5 random seeds. Full training details with hyper-parameter settings can be found in Appendix E.
Evaluation. We aim examine whether HSP can cooperate well with (1) learned human models, (2) scripted policies with strong preferences, and (3) real humans. We use both game reward and human feedback as evaluation metrics. We remark that since a biased human player may play a sub- optimal strategy, the game reward may not fully reflect the performance gap between the baselines and HSP. Our goal is to ensure the learned policy is effective for biased partners/humans. Therefore, consider human feedback investigate the impact of our design choices in HSP. In tables, maximum returns or comparable returns within a threshold of are marked bold. Full results can be found in Appendix
the fundamental metric. Ablation studies are also performed to
6.1 COOPERATION WITH LEARNED HUMAN MODELS IN ONION-ONLY LAYOUTS
For evaluation with learned human models, we adopted the models provided by Carroll et al. 2019), which only support onion-only layouts, including Asymm. Adv., Coord. Ring and Counter Circ.. The results are shown in Tab. [ For fair comparison, we reimplement all the baselines, labeled MEP, FCP, and TrujDiv, with the same training steps and policy pool size as HSP. We additionally take the best performance ever reported in the existing literature. labeled Existing SOTA in Tab, I Our implementation achieves substantially higher scores than Existing SOTA when evaluated with the same human proxy models. HSP further outperforms other reimplementations in Asymm. Adv. and is comparable with the best baseline in the rest. Full results of the evaluation with learned human models can be found in Appendix F. We emphasize that the improvement marginal because the learned human models have limited representation power to imitate natural human behaviors, which typically cover many behavior modalities. Fig in Appendix F.I.] shows trajectories induced by the learned human models only cover narrow subspace of trajectories played by human players. Further analysis of the learned human models can be found in Appendix F.I.I Furthermore, our implementation of baselines achieves substantially better results than the orginal papers (Carroll
al. 2019 Zhao et al., 202T), which also makes the improvement margin smaller.
Pos.
Asymm. Adv. Coord, Ring Counter Cire.
1
141.1
92.7(7.4)
54.5(2.3)
Existing SOTA
2
84.6(16.3)
107.3
55.8(3.6
1
282.8(94)
161.3.1.6
95.9(2.0)
FCP
2
203.8(8.2)
161.0(2.
92.7(1.3
1
291.7(4.6)
161.81077
108.8(42)
MEP
2
203.4(2.0)
164.2(2.1)
111.1(0.7)
1
289.3(s.1
150.8(3.1)
60.1(5.0)
TrajDiv
2
194.2
142. 1(23)
53.7(124)
1
300.3(2.2)
160.012.61
107.4(3.5)
HSP
2
217.1
160.6(33)
106.63.0)
Table 1: Comparison of average episode reward and standard deviation when cooperating with
learned human models. The Pos. column indicates the roles played by AI policies. Existing SOTA
is the best performance ever reported in the existing literature. HSP achieves substantially higher scores than Existing SOTA. And HSP further outperforms other methods in Asymm. Ady, and is comparable with the best baseline in the rest.
6.2 ABLATION STUDIES

---

Published as a conference paper at ICLR 2023
We investigate the impact of our design choices, includ- ing the construction of the final policy pool and the batch size for training the adaptive policy.
Epizode Rleard
300
Hil wus Fittring goel (bbwdd))
aba-33i
200
Policy Pool Construction: HSP has two techniques for the policy pool, i.e., (1) policy filtering to remove dupli- cated biased policies and (2) the use of MEP policies un- der the game reward for half of the pool size. We measure the performance with human proxies by turning these op- tions off. For "HSP w.o. Filtering", , we keep all policies by random search in the policy pool, resulting in a larger pool size of 54 (18 MEP policies and a total of 36 ran- dom search ones). For w.o. MEP", we exclude MEP policies from the policy pool and keep all biased policies without filtering, which leads to the same pool size of 36. The results are shown in Fig. 3 and the de- tailed numbers can be found in Appendix F.2.2 By excluding MEP policies, the HSP variant (HSP w.o, MEP) performs worse in the more complicated layout Counter Circ. while remaining compara- ble in the other two simpler ones. So we suggest including few MEP policies when possible. With policy filtering tumedd off, even though the policy pool size grows, the performance significantly decays in both Coord. Ring and Counter Cire. layouts, suggests that duplicated biased policies can hurt policy generalization.
100
Adymm, Adv
Coard.
Figure 3: Performance of different pool construction strategies. Results suggest that it is beneficial to incorporate MEP policies and filter duplicated policies.
We measure the training curves of the final adaptive policy under the game reward using different numbers of parallel rollout threads in MAPPO. More parallel threads indicate a larger batch size. The results in all five layouts are reported in Fig. 4 In general,
Batch Size:
observe that larger batch size often leads to better training performance, In particular, when the batch size is small, i.e., using 50 or 100 parallel threads, training becomes significantly unstable and even breaks in three layouts. Note that the biased policies in the HSP policy pool have particularly diverse behaviors, which cause a high policy gradient variance when training the final adaptive policy. Therefore, sufficiently large training batch size can be critical to stable optimization. We adopt 300 parallel threads in all our experiments for fair comparison.
Asymmuernc Advartages
Coandiaaion Ring
Courber Droat
Dutant Terrato
Mary Orders
toteree Bewwed
Epkinda
Spbodd
200
10
300
Tiradrag
ln7
Timeetepe
Timssdrpe
Tiradeep
1e7
Naulistop
Por kld0
Figure 4: Average game reward by using different numbers of parallel rollout threads in MAPPO to train the final adaptive policy. More parallel threads imply a larger training batch size.
Practical Remark: Overall, we suggest using pool size of 36 and including a few MEP policies for the best empirical performance. Besides, a sufficiently large training batch size can help stable optimization, and we use the same batch size for all methods for a fair comparison.
6.3 COOPERATION WITH SCRIPTED POLICIES WITH STRONG BEHAVIOR PREFERENCES
We empirically notice that human models learned by imitating the entire human trajectories cannot well capture a wide range of behavior modalities. So, we manually designed a set of script policies to encode some particular human preferences: Onion/Tomato Placement, which continuously places onions or tomatoes into the pot, Onion/Dish Everywhere, which keeps putting onions or dishes on the counters, Tomato/Onion Placement and Delivery, which puts tomatoes/onions into the pot in half of the time and tries to deliver soup in the other half of the time. For fair comparison, we ensure that all scripted policies are strictly different from the HSP policy pool. More details about scripted policies and full evaluation can be found in Appendix D.3
We remark that scripted policies are only used for evaluation but not for training HSP. Tab. 2 shows the average game reward of all the methods when paired with scripted policies, where HSP signif- icantly outperforms all baselines. In particular, Distant Tomato, when cooperating with a strong tomato preference policy (Tomato Placement), HSP achieves a 10x higher score than other base- lines, suggesting that the tomato-preferred behavior is well captured by HSP.
6.4 COOPERATION WITH HUMAN PARTICIPANTS
We recruited 60 volunteers (28.6% female, 71.4% male, age between 18-30) by posting the ex- periment advertisement on a public platform and divided them into 5 groups for 5 layouts. They provided with detailed introduction to the basic gameplay and the experiment process. Vol-
7

---

Published as a conference paper at ICLR 2023
Scripts
FCP
MEP
TrajDiv
HSP
Onion Placeeent 334.8(1.0))
330.5(142))
323.6(1700)
376.819.9)
Asymm. Adv.
Onion Place. DDelivrryy
297.7(3.4)
(98.5((34)
290.0(4.7)
300.1(4.1)
Onion Everywhere
109.1(7.9)
124.0(3.4)
116.9(8.9)
121.2(12.6)
Coord. Ring
Dish Everywhere
94.4(3.8
100.2(5.3)
107.3(5.3)
115.4(7.4)
Onion Everywhere
63.7(9.2)
88,9(5.1)
82.0(12.8)
107.5(3.5)
Counter Circ.
Dish Everywhere
57.0(5.3)
53.0(1.8)
57.2(2.2)
78.5(4.1)
Tomato Placement
15.6(5.2)
20. ((1066)
23.3(9.5)
277.9(14.3)
Distant Tomato
Tomato Plave.&Delivery
177.966.1
180.4(8.7)
164.8(196)
234.6(15.1)
Tomato Placement
282.6(16.2)
225. (65.860.8)
259.2(7.9)
317.8(9.3)
Many Orders
Tomato Pelvery
329.1(5.3)
328.1(12.6)
295.7(2.4)
324.5
Table 2:
Average episode reward and standard deviation with unseen testing scripted policies. HSP significantly outperforms all baselines.
unteers are fully aware of all their rights and experiments are approved with the permission of the department. A detailed description of the human study can be found in Appendix F4 The experiment has two stages:
MEP HSP
0.53
0.47
0.53
Participants could play the game freely to ex- plore possible Al behaviors. They are asked to rank Al policies according to the degree of assistance during free plays.
Warm-up Stage:
-0.53
0.033
0.2
Exploitation Stage: Participants are instructed to achieve a score as high as possible.
TrajDiv FCP
-0.47
-0.033
0.33
We note that our user study design differs from that of the original Overcooked paper (Carroll al.) 2019). The additional warm-up stage allows for diverse human behaviors under any possible pref- erence, suggesting a strong testbed for human-assistive Als.
-0.53
-0.2
-0.33
HSP
MEP
FCP TrajDiv
Figure 5: Human prefer- ence in the warm-up stage. The unit denotes the differ- ence between the percent- age of human players who prefer row partners column partners and hu- man players who prefer col- umn partners over row part- ners. HSP is consistently preferred by human partic- ipants with clear margin.
6.4.1 RESULTS OF THE WARM-UP STAGE
The warm-up stage is designed to test the performance of Al policies in the face of diverse human preferences. Fig. visualizes the human preference for different methods reported in the warm-up stage. The unit represents the difference between the percentage of human play- ers who prefer row partners over column partners and human players who prefer column partners over row partners. The detailed caleu- lation method can be found in Appendix [.4.3] HSP is preferred by humans with a clear margin. Since humans can freely explore any possible behavior, the results in Fig. 5jjmply the strong generalization
capability of HSP We also summarize feedback from human partici- pants in Appendix F.4.2]
6.4.2 RESULTS OF THE EXPLOITATION STAGE
The exploitation stage is designed to test the scoring capability of different Als. Note that it is possible that a human player simply adapts to the Al strategy when instructed to have high scores. So, in addition to final rewards, we also examine the emergent human-AI behaviors to measure the human-Al cooperation level. The experiment layouts be classified into two categories according to whether the layout allows diverse behavior modes. The first category contains simple onion-only layouts that are taken from (Carroll et al.) 2019), including Asymm. Adv., Coord. Ring and Counter Cire.. The second category contans newly introduced layouts with both onions and tomatoes, Dis- tant Tomato and Many Orders, which allow for much wider range of behavior modes.
Onion-only Layouts: Fig, 6a shows the average reward in onion-only layouts for different methods when paired with humans. Among these onion-only layouts, all methods have comparable episode reward in simpler ones (Asymm. Adv. and Coord. Ring), while HSP is significantly better in the most complex Counter Cire. layout. Fig. shows the frequency of successful onion passing between the human player and the AI player. The learned HSP policy is able to use the middle counter for passing onions, while the baseline policies are less capable of this strategy.
Layouts with Both Onions and Tomatoes: The results and behavior analysis in Distant Tomato and Many Orders are shown as follows,
Distant Tomato: In Distant Tomato, the optimal strategy is always cooking onion soups, while it is suboptimal to cook tomato soups due to the much more time spent on moving. Interestingly, our human-Al experiments found that humans may have diverse biases over onions and tomatoes. However, all learned baseline policies tend to have strong bias towards onions and often place onions into pot with tomatoes in it already. Tab. 3 reports the average number of such Wrong
8

---

Published as a conference paper at ICLR 2023
Episode Reward
FCP
Onion Pass
MEP
300
TraDiv
Frequency
200
10
100
Asymm. Adv.
Coord. Ring
Counter Circ.
0
FCP
MEP Tra/Dwr HSP
(a) Onion-Only Layouts
(b) Counter Circ.
Figure 6: (a) Average episode reward in onion-only layouts of different methods when paired with humans in the exploitation stage. HSP has comparable performance with the baselines in Asymm. Adv. and Coord. Ring, and is significantly better in the most complex Counter Circ. layout. (b) The onion passing frequency in Counter Circ. shows that HSP is the most capable, among other baselines, of passing onions via the counter, suggesting better capabilities to assist humans.
Placements made by different AI players. HSP makes the lowest number of wrong placements and the only method that can correctly place additional tomatoes into pot partially filled with tomatoes, labeled Correct Placements. This suggests that HSP is the only effective method to cooperate with biased human strategies, e.g., preferring tomatoes. In addition, as shown in Tab. 3 even when humans play the optimal strategy of cooking onion soups, HSP still achieves comparable performance with other methods.
Many Orders: In Many Orders, an effective strategy is utilize all three pots cook soups. Our experiments found that baseline policies tend ignore the middle pot. Tab. [4] shows the average number of soups picked up from the middle pot by different Al players. The learned HSP policy is much more active in taking soups from the middle pot, leading to more soup deliveries. Furthermore, HSP achieves substantially higher episode reward than other methods,
shown in Tab.
FCP
MEP
TrajDiv
HSP
Onion- Episode Reward
343.65
325.08
334.73
340.3
Wrong Placements
0.37
0.41
0.38
0.21
Correct Placements
0.0
0.0
0.0
1.41
Table 3: Average onion-preferred episode reward and frequency of different emergent behaviors Distant Tomato during the exploitation stage. Onion-Preferred Episode Reward is average episode reward when humans prefer onions. Wrong Placements and Correct Placements are the average numbers of wrong and correct placements into a pot partially filled with tomatoes. HSP makes the lowest number of wrong placements and is the only method that can place tomatoes correctly, suggesting that HSP is effective at cooperating with biased human strategies.
FCP
MEP
TrajDiv
HSP
Episode Reward t Number of Soups Picked Up from the Middle Pot t
316.81
320.61
323.52
382.52
1.93
2.03
1.33
5.64
Table 4: Average episode reward and average number of picked-up soups from the middle pot by different Al players in Many Orders during the exploitation stage. HSP achieves significantly better performance and is much more active in taking soups from the middle pot than baselines.
CONCLUSION
We developed Hidden-Utility Self-Play (HSP) tackle the problem of zero-shot human-Al cooper- ation by explicitly modeling human biases as additional reward function in self-play. HSP first generates pool of diverse strategies and then trains an adaptive policy accordingly. Experiments verified that agents trained by HSP are more assistive for humans than baselines in Overcooked. Although our work suggests a new research direction on this fundamentally challenging problem. there are still limitations to be addressed. HSP requires domain knowledge to design a suitable set of events. There exists some work learning reward functions rather than assuming event- based rewards (Shah al. 2019 Zhou al. (021), So future direction
utilize learning-based meth- ods to design rewards automatically. Another major limitation is the computation needed to obtain diverse policy pool. Possible solutions include fast policy transfer and leveraging a prior distribution of reward functions extracted from human data (Barrcto et al. 2017a). Learning and inferring the policy representations of partners could also provide further improvement. We leave these issues as future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research was supported by National Natural Science Foundation of China (No.U19B2019, 62203257, M-0248), Tsinghua University Initiative Scientific Research Program. Tsinghua-Meinn Joint Institute for Digital Life, Beijing National Research Center for Information Science, Technol- ogy (BNRist), and Beijing Innovation Center for Future Chips and 2030 Innovation Megaprojects of China (Programme on New Generation Artificial Intelligence) Grant No. 2021AAA0150000.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01692 2022.
Arash Ajoudani, Andrea Maria Zanchettin, Serena Ivaldi, Alin Albu-Schäffer, Kazuhiro Kosuge, and Oussama Khatib. Progress and prospects of the human-robot collaboration. Autonomous Robots, 42(5)9957-775, 2018.
Bowen Baker. Emergent reciprocity and team formation from randomized uncertain social prefer- ences. Advances in Neural Information Processing Systems, 23:15786-1574 2020.
Nicholas C Barberis, Thirty years of prospect theory in economics: A review and assessment. Journal of Economic Perspectives, 27(1):173-96, 2013.
Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Has- selt, and David Silver. Successor features for transfer in reinforcement learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017a. URL https: http:::://ppppeeedeeeeee 335Cb0881 (a..00081666122255555580
neur spsrrpp... pctpc///a/ppe//1///1eee 9b8c0b 9 pddf
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul. Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017b.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06660 2019.
Francesco Biondi, Ignacio Alvarez, and Kyeong-Ah Jeong. Human-vehicle cooperation in au- tomated driving: A multidisciplinary review and appraisal. International Journal of Human- Computer Interaction, 35(1 1):932-946, 2019.
Andreea Bobu, Dexter RR Scobee, Jaime Fisac. S Shankar Sastry, and Anca D Dragan. Less is more: Rethinking probabilistic models of human behavior. In Proceedings of the 2020 ACM/IEEE International Conference on Human- Rooua Interaction, pp. 429-437, 2020.
Colin Camerer. Behavioral game theory: Experiments in strategic interaction. Princeton univer- sity press, 2011.
Colin F Camerer, Teck-Hua Ho, and Luin-Kuan Chong. A cognitive hicrarchy model of games. The Quarterly Journal of Economics, 1 119(3):861-89, 2004.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility learning about humans for human-AI coordination. Advances in neural information processing systems, 32, 2019.
Rujikomn Charakom, Poramate Manoonpong, and Nat Dilokthanakul. Learning to cooperate with unseen agents through meta-reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1478-1477 1479, 2021.
Shuo Chen, Ewa Andrejezuk, Zhiguang Cao, and Jie Zhang. Aateam: Achieving the ad hoc team- work by employing the attention mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7095-71 1009, 2020.
Tinkle Chugh, Karthik Sindhya, Jussi Hakanen, and Kaisa Miettinen. A survey on handling com- putationally expensive multiobjective optimization problems with evolutionary algorithms. Soft Computing, 23(9):317-3116, 2019.
10

---

Published as a conference paper at ICLR 2023
Brandon Cui, Hengyuan Hu, Luis Pineda, and Jakob Foerster. K-level reasoning for zero-shot coordination in hanabi. Advances in Neural Information Processing Systems, 34, 2021.
Antoine Cully, Jeff Clune, Danesh Tarapore, and lean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):003-507, 2015.
Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel. Cooperative AI: machines must learn to find common ground, 2021.
Peter Eckersley. Impossibility and uncertainty theorems ai value alignment (or why your agi should not have utility function). In SafeAl@ AAAI, 2019.
Owain Evans, Andreas Stuhlmüller, and Noah Goodman. Learning the preferences of ignorant, inconsistent agents. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Fei Fang, Shutian Liu, Anjon Basak, Quanyan Zhu, Christopher 1) Kiekintveld, and Charles A Kamhoua. Introduction to game theory. Game Theory and Machine Learning for Cyber Security, pp. 26-41, 2021.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances neural information processing systems, 31, 2018.
Marta Garnelo, Wojciech Marian Czamecki, Siqi Liu, Dhruva Tirumala, Junhyuk Oh, Gauthier Gidel, Hado van Hasselt, and David Balduzzi. Pick your battles: Interaction graphs as population- level objectives for strategic diversity. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1501-1503, 2021.
Thomas L Griffiths. 21 21-23, 2015. Manifesto for a new (computational) cognitive revolution. Cognition, 135:
Abhinav Gupta, Mare Lanetot, and Angeliki Lazaridou. Dynamic population-based meta-learning for multi-agent communication with natural language. Advances in Neural Information Process- ing Systems, 34, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-eritic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Dylan Hadfield-Menel Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference on Machine Learning, pp. 4399 4410. PMLR, 2020.
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster Off-belief learning. In International Conference on Machine Learning, pp. 4369-4379. PMLR, 2021.
Cory D Kidd and Cynthia Breazeal. Robots at home:
Understanding long-term human-robot in- teraction. In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp- 3230-3235. IEEF, 2008.
Christopher Kiekintveld, Towhidul Islam, and Vladik Kreinovich. Security games with interval uncertainty. In Proceedings of the 2013 international conference Autonomous agents and multi-agent systems, pp. 231-238, 2013.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation in deep reinforcement learning. arXiv preprint 1rXhv:2111.09790 2021.
Glen Klien, David D Woods, Jeffrey M Bradshaw, Robert R Hoffman, and Paul J Feltovich. Ten challenges for making automation a "team player" in joint human-agent activity. IEEE Intelligent Systems, 19(6):91 95, 2004.
Paul Knott, Micah Carrol,, Sam Devlin. Kamil Ciosck, Katja Hofmann, Anca Dragan, and Rohin Shah. Evaluating the robustness of collaborative agents. In Proceedings of the 20th International Conference on Autonomous Agents and MulliA gent Systems, pp. 1560-1562, 2021.
Hadas Kress-Gazit, Kerstin Eder, Guy Hoffmman, Henny Admoni, Brenna Argall, Ruediger Ehlers, Christoffer Heckman, Nils Jansen, Ross Knepper, Jan Křetínský. et al. Formalizing and guaran- teeing human-robot interaction. Communications the ACM, 64(9):78 84, 2021.
11

---

Published as a conference paper at ICLR 2023
Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P Losey, and Dorsa Sadigh. When humans aren't optimal: Robots that collaborate with risk-aware humans. In 2020 15th ACM/EEEE International Conference on Human-Robot Interaction (HRI), pp. 43-52. IEEE, 2020.
Cassidy Laidlaw and Anca Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. In International Conference on Learning Representations, 2022.
Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Péro- lat, David Silver, and Thore Graepel. A unified game-theoretie approach to multiagent reinforce- ment learning. Advances in neural information processing systems, 30, 2017.
Shihui Li, Yi Wu, Xinyue Cui. Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 4213-4220 2019.
Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and
Yaodong Yang. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. Advances in Neural Information Processing Systems, 34:941-955 -922, 2021.
Qian Long*, Zihan Zhou* Abhinav Gupta, Fei Fang, Yi Wuł, and Xiaolong Wangt. Evolutionary population curriculum for scaling multi-agent reinforcement learning. In International Confer- ence on Learning Representations, 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettive environments, Advances in neural information process- ing systems, 30, 2017.
Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of
measuring emergent communication. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, pp. 693-701, 2019.
Ryan Lowe*, Abhinav Gupta*, Jakob Foerster, Douwe Kiela, and Joelle Pineau. On the interac- tion between supervision and self-play play in emergent communication. In International Confer- ence Learning Representations, 2020. URL https::/ nett forum?id
IJxGLIBtwH
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference Machine Learning, pp. 7204-7213, PMLR, 2021.
Hadi Nekoei, Akilesh Badrinaaraayanan, Aaron Courville, and Sarath Chandar. Continuous coor-
dination as a realistic scenario for lifelong learning. In International Conference on Machine Learning. pp. 8016-8024. PMLR, 2021.
Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In in Proc. 17th International Conf. on Machine Learning. Citeseer, 2000.
Thanh Nguyen, Rong Yang, Amos Azaria, Sarit Kraus, and Milind Tambe. Analyzing the effec- tiveness of adversary modeling in security games. In Proceedings of the AAAÍ Conference on Artificial Intelligence, volume 27, pp. 718-724, 2013.
Xinlei Pan, Weiyao Wang, Xiaoshuai Zhang, Bo Li, Jinfeng Yi, and Dawn Song. How you act tells a lot: Privacy-leaking attack on deep reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 368-376, 2019.
Praveen Paruchuri, Milind Tambe, Fernando Ordónez, and Sarit Kraus. Security in multiagent systems by policy randomization. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pp. 273-280, 2006.
John W Pratt Risk aversion in the small and in the large. In Uncertainty in economics, pp. 59-79.
Elsevier, 1978.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolu- tionary computation. Frontiers Roboties and AI, 3:40, 2016.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference on Machine Learning, pp. 4295 4304. PMLR, 2018.
Stuart Russell. Human compatible: Artificial Intelligence and the problem of control. Penguin, 2019.
12

---

Published as a conference paper at ICLR 2023
Reinhard Selten. Bounded rationality. Journal of Institutional and Theoretical Economics (JiTE)Zeisschrift für die gesamle Staatswissenschufi, 146(4):649-658, 1990.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In International Conference on Machine Learning. pp. 5670-5679. PMLR, 2019.
Thomas B Sheridan. Human-robot interaction: status and challenges. Human factors, 58(4):525- 532, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- self-play, Science, 362(6419):1140- 1144, 2018.
Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Aventy-Fourth AAAI Conference on Artificial Intelligence, 2010.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Mohammad Tabatabaei, Jussi Hakamen, Markus Hartikainen, Kaisa Miettinen, and Karthik Sindhya. A survey on handling computationally expensive multiobjectivv optimization problems using sur- rogates: non-nature inspired methods. Structural and Multidisciplinary Optimization, 52(1):1-25, 2015.
Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward random- ization. In International Conference on Learning Representations, 2020.
Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster. A new formalism, method and open issues for zero-shot coordination. In International Conference on Machine Learning, pp. 10413-10423. PMLR, 2021.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czareekki, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):550-354 2019.
Chen Wang, Claudia Pérez-D' Arpino, Danfei Xu, Li Fei-Fei, Karen Liu, and Silvio Savarese. Co- gail: Learning diverse strategies for human-robot collaboration. In Conference on Robot Learn- ing. pp. 1279-1290. PMLR, 2022.
Nick Wilkinson and Matthias Klaes. An introduction to behavioral economics. Macmillan Interna- tional Higher Education, 2017.
Detlof Von Winterfeldt and Gregory W Fischer. Multi-attribute utility theory: models and assess- ment procedures. Utility, probability, and human decision making, pp. 47-85, 1975.
Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman- Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collabora- tion. Topics in Cognitive Science, 13(2)4414432 2021.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning. arXiv preprint arXiv: 044888 2015.
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent repre- sentations to influence multi-agent interaction. In Conference on Robot Learning, pp. 575-588. PMLR, 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.0/955, 2021.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. Multi-agent collaboration via reward attribution decomposition. arXiv preprint arXiv:2010.0853), 2020.
13

---

Published as a conference paper at ICLR 2023
Rui Zhao, Jinming Song, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Max- imum entropy population based training for zero-shot human-ai coordination. arXiv preprint 12.11:2112.11701, 2021.
Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via soward-switching policy optimization. Deep RL Workshop NeurlPS 2021, 2021. URL https: //openzeview wwwpt//oooommddmmmmmmm
2IV2
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Ku- mär, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In Inter- national Conference on Learning Representations, 2019.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. entroyyentropp inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
14

---

Published as a conference paper at ICLR 2023
We would suggest httpggg...............oo.ooooe/a for more infor- mation.
A THEOREM PROOFS
For simplicity. assume state space and action space in analysis are both discrete and finite, which is exactly the case for Overcooked, and the rewards r are bounded: (r(s, a)s, S Rmax: Vs S.a E A.
Lemma 5.1. Given an MDP M = (S Ri), for any policy Tu x A - [0, 1]. there
exists hidden reward function R. such that the two-player hidden utility Markov game M' (S, A, P. Rw, Ra) has Nash equilibrium (Ta. mid)) where x Tw-
Proof. Our analysis is based on the maximum entropy reinforcement learning framework (Haarnoja al.) 2018: Ziebart al. 2008 Wulfmeier al. 2015). Given reward function I and policies of the two players W1 T2, we consider tollowing maximum entropy RL objective for policy Ty S 2).
(T E, Σ (R(s,, af [ a1(111))
We shall first constructs given policy Tw to satisfy Jz(TTw Ta Re)) (2 n'. Re), VTLa and secondly constructs Rw such that J1 (Tw; na Rtw) ) 2 J1 (T , Ta Rao), Vn is satisfied.
Step 1:
Construct na given Tw.
Given Tw, let Ta E arg max, J2( π Re).
Step 2: Construct Rw such that J1 (Tws Tal 2 J1 J1 Ta R..)). VT satisfied given Tw and Ta.
Given a fixed partner Na. by regarding Ta as part the environment dynamics, we could consider the dynamics for Tu in a single-agent MDP M' = (S,A, P', Rw. 7) where S is the state space, A the action space, p' denotes the transition probability and R. the reward function construct. More specifically, p' defined as,
P'(s'1s,a) -
P(s'ss,,a,)) (āls))
In .M, given reward Ry, the objective of nw becomes,
(B(( ((((( (HHta))) x(36)
max Ey,
(3)
The value function and the Q function could be defined as,
(s) = E, Σ (Bu ax) + a (st))) a am)) =
(4)
= Twl x (V7 (
(5)
Q a) = as) +y E.r (V(s))
(6)
It is sufficient to construct Ray such that ((s) is a stable point of the Bellman backup operator fon & Barto 2018) 7" under some R.,:
Sut-
(T)
max aH(d) + E A.C. d(a)=l
Now we assume (ss) is a stable point for Eq. 7 and construct R.,. For all & E S, n.. (-|s) should be solution to the following maximization problem,
15

---

Published as a conference paper at ICLR 2023
max aH((d) + Σ d(a)Q(s,a)
(8)
s.t. d(a)==1
(9)
Applying KKT conditions over the above optimization problem indicates that,
(-18) & exp(Q((,.)/a))
(10)
Let T (s) = arg max, (als), (s) = max, Q A(s, a) = Q(s, - Vs(ss. By Eq. also have
10
A(s, a) mw log Te (TT
(11)
By definition of value function V(s).
((s)
Σ
(12)
Σ
(13)
Σ
(14)
Σ
(( + a (s))
(15)
E Σ
Σ The ((s(")s)A(ss.a)) + (ss, az) + (.H((T (())))))))) |ae =
Th (+a)
(16)
Let b(s) = (s, Tt. (s)). Then (s)) is determined given tw and b,
V(s) = Er Σ x² Σ (w (dddss)(sss)))) + b(s)) + (HH(Tw((e)) - (17)
By A(s, a) allog The (als) log n
Q(s, -(s,a)-VV((1 Vs),,,
a(log To (als) log m. (m (s)(s) Ri (s, + 7
- (s)
(18)
(als)
as, log
(T)(( 7E ~ Ps) (*)
(19)
To summarize, for policy Tw. construct valid hidden reward function R. via following process,
1. Choose function b: 8' + R.
2. Compute A(s, a) by Eq. 11
3. Compute (s) and (*(8) by Eq.
4. Construct R,, (8, a) by R (8, (s)) = b(s) and Eq. 19
Now we show that, for any b: S' - R. under Rw constructed by the above process, (88) is stable point of the Bellman backup operator T This is straightforward. First, constructed R., ensures
16

---

Published as a conference paper at ICLR 2023
that a(log (als)) log Tw (s)(()))) Q(s.a) (*()) (Eq. 19 and therefore Tw(a/w) & exp(Q(s. a)/a), which means Tu is à solution for the maximization propllommooommmmmmmmmmmmm 8 So
(T)))(s) maxaH(d) d'
Σ (t)
(20)
= Σ Tu(als)Q(s,a) + (s) =
(21)
Theorem 5.1. For any E O, there exists a mapping Th where T., (Rw) denotes the derived policy
Tw in the NE of the hidden utility Markov game Mw- (S,A, P, Rus Ra) induced by Rw and
distribution PR: R + [0. 1] over the hidden reward space R, such that, for any adaptive policy TA arg max, E RR [J(',, #wo (R.(R.))), a approximately maximizes the ground-truth objective with at most an gap, ie., Ex Pu [ m 2 max,' Erg-Py [ TH)]
Proof. Let K(K >> [Al) be large positive integer. We construct a discretization of the policy space II by 1 = { o (als) K where i E [K], Vs E S, E A and Σ. T(als) 1, vs E S). Note that x is finite, i.e. | S (K + 1) - Al Let M = xc and Tı, "2, :* TM be an ordering of the policies in [ī,. For simplicity of notation, let 5 [A]
Given the discretization II, it's straightforward to specify the nearest policy स E 1 for any policy E II. Formally, for any policy x E II, let G(T)) = arg min, Es, (Tass) - n. (a(s)). An
obvious property of G is that, Vs S, [||(s|)) - G(s)( s = 5.
For two policies Ti and T2, consider Tı playing with and G( respectively. Since the action distribution of π2 and G(Tz) each state differ at most d, we have follows,
2Rmax
26
[J( - JTT, G(T2))) S Σ r (1 (1-
8) 8
S (1-
(22)
We can then derive discretized approximation of the ground-truth policy distribution Pи fol- lows,
(T( Pra' G(
(23)
We could show that the difference between the objective under the ground-truth policy distribution PH and that under the approximated policy distribution PH bounded. By Eq. 22 for any adaptive policy T.
E nn Pu (TTA, - Exy~ ( TH)l = |B PITAA G(TH)) - J (TTA (24)
28 Rmax
<
(25)
(1 7)2
On the other hand, consider following an iterative process find hidden reward functions for poli-
cies TT. For i 1..M, we find hidden reward function R(0 where R! (Rl@) و 1- and (()) could be constructed from as in Lemma 5.1. Notice that, by construction rule in Lemma 5. such 12(i) must exists since we can specify arbitrary b: IR
Let The The = Te, (T;)))) = .... M. We immediately see that, for any adaptive policy TA, the objective is equiv- alent under the approximated policy distribution Pиr and hidden reward function distribution Pn.
1...M and the hidden reward distribution PR be (R( =
ER P' [J ( 7n. RR)))) E Exuw-Py [J(TA, #H)]
(26)
Finally, for any adaptive policy F arg max En PP [J [ 7.n (Rw)) ))] and any policy x' E IT,
17

---

Published as a conference paper at ICLR 2023
28 Rmaxx
E (TT,,
E ma Pn [ TH
(27)
(1 - (1-
28 R R.na
= ER [J T Tw (w( (Bw)))
(28)
(1- -7
28 Rmax
2 BR ( Tw (Rw)))
(29)
(1 7)²
Rmax
Bx P [J(x', TH)
(30)
(1-
48 Rmax
2 Ban Pi [J(T', Tn)
(31)
(1 - 7/)
Let K Rmaxx and have B Laan-PpssTTAAAA [J(AA TH)) 2 max,/ Bmg Pm [1 [T)]]
c[1-7)²
B ENVIRONMENT DETAILS
Asymmetric Advantages
Coordination Ring
Counter Cirenit
Distant Tomate
Many Orders
Onion
Onions
Tomsto
Tomatnes
Dish
Disbes
Pot
Serving Area
Player
Player
Time
$
$
20
20
20
20
Order
10
1O
Figure 7: All 5 layouts used in our work (from left right): Asymmetric Advantage, Coordination Ring, Counter Circuit, Distant Tomato, and Many Orders, each featuring specific cooperation pat- terns we want study.
B. DESCRIPTION
The Overcooked Environment, first introduced in (Carroll et al. 2019), is based on the popular video game Overcooked where multiple players cooperate to hinsh many orders possible within time limit. In this simplified version the original game, two chiefs, each controlled by player (either human or Al), work in grid-like layouts. Chiefs can move between non-table tiles and interact with table tiles by picking up or placing objects. Ingredients (e.g., onions and tomatoes) and empty dishes can be picked up from the corresponding dispenser tiles and placed empty table tiles or into the pots. The typical pipeline for completing order is (1) players put appropriate ingredients into pot: (2) pot starts cooking automatically once filled and takes a certain amount of time (depending on the recipe) to finish; (3) a player harvests the cooked soup with an empty dish and deliver it to the serving area.
The observation for agent includes the whole layout, items on the counter and pots, player positions, orders, and time. The possible actions up, down, left, right. no-op, and "interacting" with the tile the player is facing. Reward is given to both agents upon successful soup delivery, with the amount varying with the type of soup. An episode of the game terminates when the time limit reached.
The environment used in (Carroll et al. 2019) has only onions as ingredients and onion soups as orders. In our work. evaluate all methods in three of them, namely Asymmetric Advantage, Coordination Ring, and Counter Circuit, each designed to enforce specific cooperation pattern.
Our work introduces order types to make cooperation more challenging. In Distant Tomato, dish of onion soup takes 20 ticks to finish and gives 20 rewards when delivered, while a tomato soup takes 10 ticks gives the same reward but needs more movements to get the ingredient. The two players need to agree on which type of soup to cook in order to reach high score. Failure in cooperation may result in nomato-onion soups that give reward. In many orders, there three types of orders: onion,
new layouts: Distant Tomato and Many orders, with new ingredients and
18

---

Published as a conference paper at ICLR 2023
tomato, and 2atiion--tomat... To fully utilize the three pots, the players need to work seamlessly in filling not just the pots near each of them but also the pot in the middle.
We show all the layouts in Fig 7 and conclude the cooperation pattern of our interest as follows.
Asymetrrc Advantage tests whether the players can choose strategy their strengths.
Coorlination Ring requires the players not to block each other when traveling between the two corners.
Counter Circuit embeds non-trivial but efficient strategy of passing onions through the middle counter, which needs close cooperation.
Distant Tomato and Many Orders both encourage the players reach an agreement on the fly order achieve a high reward.
B.2 EVENTS
In Overcooked, we consider the following events for random search in HSP and reward shaping during training of all methods:
putting an onion/tomato/dish/suupp on the counter,
picking up an onion/tomatto/lish/ssouu from the counter,
picking up an onion from the onion dispenser, picking up tomato from tomato dispenser,
picking up dish from the dish dispenser, picking up ready soup from the pot with dish,
placing an onion/tomato into the pot,
valid placement: after the placement, we can finish an order with a positive reward by placing other ingredients,
optimal placement: the placement is optimal if the maximum order reward we achieve for this particular pot is not decreased after the placement,
catastrophic placement:
the placement is catastrophic if the maximum order reward we can achieve for this particular pot decreases from positive to zero after the placement,
useless placement: the placement is useless if the maximum order reward we can achieve for this particular pot is already zero before the placement,
useful dish pickup: picking up dish is useful when there no dishes on the counter, and the number of dishes already taken by players is less than the total number unready and ready soups,
delivering soup to the serving area.
Additionally, in Distant Tomato, we consider the following events only for reward shaping,
placing a tomato into an empty pot,
optimal tomato placement: the placement is optimal and a tomato placement,
useful tomato pickup: the agent picks up tomato when the partner isn' 't holding tomato, and there is pot that not full but only has tomatoes in it.
C OVERCOOKED VERSION
In our experiments, we use two versions of Overcooked for fair comparison with prior works and introduce challenging layouts. One version, in which we tested Asymmetric Advantages, Coordina- tion Ring and Counter Circuit. consistent with the "ncurips2019" branch in the released GitHub repository of (Carroll et al. 2019). We remark that MEP (Zhao et al., (02T)) also follows this ver- sion. Following this also allows us to perform an evaluation with human proxy models provided in the released code of (Carroll et al.) 2019). The other version is an up-to-date version of Over- cooked, which supports tomatoes and defr-de orders. We notice that pot automatically starts cooking soup once there are three items in it in the former version, while requires an additional "interact" action to start cooking the latter version. This additional "interact" is required in the latter version since it supports orders with different amounts of ingredients. However, having an additional "interact" significantly influences human player's 's interactive experience. Therefore, we make modifications on the latter version to restrict orders to 3 items and support auto-cooking when there are 3 items. For more details, please refer to the released code.
19

---

Published as a conference paper at ICLR 2023
D IMPLEMENTATION DETAILS
D.1 HSP
Algorithm 3: Hidden-Utility Self-Play
for i
do Train The and n under sampled RW); end
Run greedy policy selection to only keep K policies; Initial policy TA
repeat
Rollout with TA and sampled :
Update TA
until enough iterations;
The pseudocode of HSP is shown in Algo. 3 We implemented HSP on top of MAPPO (Yu et al. 2021). Following the standard practice, we use multiprocessing to collect trajectories in parallel and then update the models. In the first stage, we use MLP policies, which empirically yield better RNN policies
results. In the second stage, tion of its partner by observing the history of its partner and make decisions accordingly for better adaptation. As suggested in Tang et al., 2020), we add the identities of the policies in the policy pool as an additional feature to the cn For better utilization of the computation resources, each environment sub- process loads a uniformly sampled policy and performs inference on CPUs, while the inference of the adaptive policy is batched across sub-processes in GPU.
that the adaptive policy could infer the inten-
D.2 BASELINES
For fair comparison, we implement all baselines to be two-staged and train layout-specific agents.
We remark that our implementation of MEP achieves substantially higher scores than reported in the original paper (Zhao et al. 2021) when evaluated with the same human proxy models as MEP All baselines are implemented with techniques stated above: loading policies from the pool per sub- process and the additional feature of identities of policies in the policy pool. We detail the baselines here and point out the difference with the original papers,
FCP Strouse et al., 2021): We list the differences between our implementation and the original FCP as tollows,
1. The original FCP uses image-based egocentric observations, while use feature-based observations as provided in Overcooked.
2. The original FCP uses a pool size of 96 while we use 36. We empirically found 36 a sufficiently large pool size in our experiments. As shown in Table 21 in the three layouts that have human proxy models, there is no significant difference between using pool size of 36 and of 72.
MEP et al. 2021): We list the differences between our implementation and the original MEP as follows,
While the released code of MEP uses MLP policy in the second training stage, we found RNN policy to work better. Intuitively, for better cooperation, the adaptive policy should infer the intention of its partner by observing the state-action history.
2. MEP uses pool size of 15 while we use 36.
3. MEP uses prioritized sampling in the second stage, which favors weak policies in the pool, while we adopt uniform sampling for MEP since we found prioritized sampling not helpful with our carefully tuned implementation (shown in Table 5)
4. In the released code of MEP, the policy updates performed data against only one policy from the pool, while we perform policy updates on data against many policies from the pool. This avoids the update from being biased towards some specific policies.
TrajDiv( et al. 2021): While the original Traj is tested in hand-crafted MDPs and Hanabi, we test TraiDIv in Overcooked. Although (Lupu et al. (021)) suggests training the adaptive policy and the policy pool together in single stage, we choose to follow MEP and FCP to have a two- staged design that trains the adaptive policy in the second stage.
20

---

Published as a conference paper at ICLR 2023
Pos.
Asy. Adv.
Coor. Ring
Coun. Cire.
291. (44.))
161.8(07)
108.8(4.2)
Uniform Sampling
203.4(2.0)
164.2(2.1)
111.1
1
284.6(3.2)
161.2(1.4)
94.4(2.3
Prioritized Sampling
218.8(2.4)
167(4.5)
99.8(18)
Table 5: Average episode reward and standard deviation (over 5 seeds) with different sampling methods MEP. The "1" and "2" indicates the roles played by Al policies.
Biased
Scripted
Asymm. Adv.
0.56
0.85
Coord. Ring
0.59
0.72
Counter Circ.
0.56
0.73
Dist. Tomato 0.70
1.90
Many Orders 0.55
1.00
Table 6: The average bast- difference of biased and scripted policies respectively.
D.3 SCRIPTED POLICIES
To evaluate all methods with policies that have strong preferences, we consider the following scripted policies.
Onion/Tomato/Dish Everywhere continuously tries to put onions, tomatoes dishes over the counter.
Onion/Tomato Placement always tries to put onion tomato into the pot.
Delivery delivers ready soup to the serving area whenever possible.
Onion/Tomato Placement and Delivery puts tomatoes/onions into the pot in half of the time and tries to deliver soup in the other half of the time.
For Counter Circuit, additionally consider scripted policy, named Onion Middle Counter, which keeps putting onions randomly over the counter the middle of the layout.
Input to these scripted policies the ground-truth state the game, which is accessible via the game simulator. When a scripted policy is unable to finish the event of its interest at some state, the scripted policy would walk to a random empty grid. For example, Onion Placement would choose random walk when all pots are full. We ensure that these scripted policies are strictly different from policies in the policy pool of HSP. For more details, please refer to the released code.
We also provide evidence show scripted policies sufficiently different from those in the train- ing pool. We use the expected event count scripted and biased policies to support our claim. Recall that expected event count for a pair of policy Ra Th is EC(T,, T = E EE>n Φ(st. (T) mı). Let THSP be the HSP adaptive policy, ( Jne[N] be the of biased policies in the training pool, and {-!()) IME be the set of scripted policies. For convenience, let II { JNe[N] U (m<"")) IEE be the union of biased policies and scripted policies. For each policy x' 11. we measure how close it is the rest of policies in II in the expected event count, i.e. the event-based difference minn (m') Σx C IEC, ( ( THSP) EC ( [H5)) where e₁ is a fre- quency normalization constant. Then large event-based difference indicates that n' is sufficiently different from other policies II. We calculate the event-based difference for all biased and scripted policies. Table. 6 reports the average event-based difference between biased and scripted policies, respectively. Serittdd policies consistently have a larger average event-based difference. indicating scripted policies are sufficiently different from biased policies, which are used for training the HSP adaptive policy.
EventDiff (x')
21

---

Published as a conference paper at ICLR 2023
E TRAINING DETAILS
E.I HYPERPARAMETERS
HSP and baselines all two-staged solutions by first constructing policy pool and then training an adaptive policy A to maximize the game reward w.r.t. the induced pool.
The network architecture in both two stages is composed of 3 convolution layers with max pooling. Hyperparameters of these layers are listed in Table 7 Each layer is followed by max pooling layer with a kernel size of 2. For MLP policies, we add two linear layers after the convolution. For RNN policies, we add -llayer GRU after the convolution and two linear layers after the GRU layer. The hidden sizes for these linear layers and the GRU layer are all 64. We use ReLU the activation function between layers and LayerNorm after GRU and linear layers except the last one. The output a 6-dim vector denoting the categorical action distribution.
Common hyperparameters for all methods in 5 layouts are listed in Table 8 and Table 9 Specifically, for MEP, we use the suggested hyperparameters from the original paper (Zhao et al. 202 I. Detailed hyperparameterss of MEP shown in Table [0] where population entropy coef. adjusts the impor- tance of the population entropy term. Detailed hyperparameters of TrajDiv are shown in Table II where traj. gamma is the discounting factor used in local action kernel and diversity coef. adjusts the importance of the diversity term. For each one of MEP, FCP and Traj we train 12 policies in the first stage and, following the convention of MEP (Zhao et al. 2021 l and FCP (Strouse et al.) 202I), take the init/middle/final checkpoints for each policy build up the policy pool, leading to pool size of 36. For HSP, we use a random search to first train 36 biased policies and then filter out 18 biased policies from them. We then combine these biased policies and past checkpoints of policies in the policy pool of MEP to build up the policy pool of HSP, again leading to a pool size of 36.
Layer
Out Channels
Kernel Size
Stride
Padding
1
32
3
1
2
64
3
1
3
32
3
1
1
Table 7: CNN feature extractor hyperparameters.
common hyperparameters
value
entropy coef.
0.01
gradient clip norm
10.0
GAE lambda
0.95
gamma
0.99
value loss huber delta
huber loss
10.0
mini batch size
batch size / mini-batch
optimizer optimizer epsilon weight decay network initialization use reward normalization
Adam
le-5
Orthogonal
True
use feature normalization
True
learning rate parallel environment threads
5e-4
100
ppo epoch
15
environment steps
10M
episode length
400
reward shaping horizon
100M
Table 8: Common hyperparameters in the first stage.
22

---

Published as a conference paper at ICLR 2023
E.2 CONSTRUCTING THE POLICY POOL FOR HSP
To construct the policy pool for HSP, we perform a random search over possible hidden reward functions. Each reward function is formulated as linear function over the event-based features, iee. R (Rw (s, 01, , o(s, 01, a2)a)) w. ||w]lc I Cmax] } where Φ S x A A + R specifies occurrences of different events when taking joint action (a1, .2) at state To perform random search, instead of directly sampling each wy from the section max: Cmax]. each w. from set of possible values Cj. We detail the C₁ for each event on each layout here. Tab. 12 shows Cj in Asymmetric Advantages, Coordination Ring and Counter Circuit. Tab. [3 and Tab. 14 show Cj in Distant Tomato and Many Orders respectively. A detailed description of the events shown in Sec. B.2 Note that in addition to events, we also include order reward one element random search.
sample
To filter out duplicated policies, we define an event-based diversity for a subset S, i.e. ED(S) EnJes Σx |EC| - EC(") where EC₁ is the expected number of occurrences of event type for biased policy T The coefficient сı. balances the importance of different kinds of events. We simply set ck normalization constant, i.e. Ck (maxieIN) EC
common hyperparameters
value
entropy coef.
0.01
gradient clip norm
10.0
GAE lambda
0.95
gamma
0.99
value loss
huber loss
huber delta
10.0
mini batch size
batch size / mini-batch
optimizer optimizer epsilon weight decay network initialization use reward normalization
Adam
le-5
0
Orthogonal
True
use feature normalization
True
learning rate parallel environment threads
5e-4
300
ppo epoch environment steps
15
100M
episode length
400
reward shaping horizon
100M
policy pool size
36
Table 9: Common hyperparameters in the second stage.
hyperparameters value
population entropy coef. 0.01
Table 10: MEP hyperparameters in the first stage.
hyperparameters
value
traj. gamma 0.5
diversity coef. 0.1
Table 11: TrajDiv hyperparameters in the first stage.
23

---

Published as a conference paper at ICLR 2023
Event
C2
Picking an onion from onion dispenser
-10, 0, 10
Picking up dish from dish dispenser
0, 10
Picking up a ready soup from the pot
-10, 0, 10
Placing an onion into the pot
-10, 10
Delivery Order reward
-10, 0
0,
Table 12: C3 for random search in Asymmetric Advantages, Coordination Ring and Counter Circuit.
Event
Cj
Picking up onion from onion dispenser
-5, 0,
Picking up a tomato from tomato dispenser Picking up dish from dish dispenser
0, 10, 20
0, 10
Picking up a soup Viable placement Optimal placement Catastrophic placement
-,
-10, 10
-10, 10
0,
Placing an onion into the pot
-10, 10
Placing a tomato into the pot Delivery Order reward
-10, 10
-10,0 0
0,
Table 13: Cj for random search in Distant Tomato.
E.3 REWARD SHAPING
We reward shaping during training in all layouts, detailed follows,
In the first stage, the reward shaping for Asymmetric Advantages, Coordination Ring and Counter Circuit is shown in Table. 15 and that for Distant Tomato and Many Orders is shown in Table. 17 Note that we do not use reward shaping when training biased policies for HSP in the first stage.
In the second stage, the reward shaping for Asymmetric Advantages, Coordination Ring and Counter Circuit is shown in Table. 16 Reward shaping for Many Orders is shown in Table. [8 and that for Distant Tomato is shown in Table. [9] The factor of shaped reward anneals Trom 1 during the whole course of training in alluussu except Distant Tomato, in which the factor anneals from to 0.5.
F FULL RESULTS
F.1 COOPERATION WITH LEARNED HUMAN MODELS
Table 20 shows average episode reward and standard deviation (over seeds) on 3 layouts for dif- ferent methods played with human proxy policies. All values within standard deviations of the maximum episode return are marked in bold. These three simple layouts may not fully reflect the performance gap between the baselines and HSP. The results with learned human models are reported for a fair comparison with existing SOTA methods. Besides, our implementation of the baselines achieves substantially better results than their original papers with the same human proxy models, making the improvement margin look smaller. We also remark that the learned human mod- els have limited representation power to imitate natural human behaviors that typically cover many behavior modalities. Here we give empirical evidence the learned human models failing to fully reflect human behaviors.
F.1.1 EMPIRICAL EVIDENCE
The original Overcooked paper (Carroll et al. 2019) collected human-play trajectories. We then collect game trajectories played by the learned human models and compare them with human-play
24

---

Published as a conference paper at ICLR 2023
Event
C
Picking an onion from onion dispenser
-5, 0,0.5
Picking up a tomato from tomato dispenser Picking dish from dish dispenser
10, 20
0,
Picking up soup
-5,
Viable placement
1,, 10
Optimal placement
-10,
Catastrophic placement
0,
Placing an onion into the pot -3,
-3,
Placing a tomato into the pot Delivery Order reward
-10, 0
0,
Table 14: Cj for random search in Many Orders.
Event
Value
Optimal placement Picking up a dish from dish dispenser Picking up ready soup from the pot
Table 15: Reward shaping for Asymmetric Advantages, Coordination Ring and Counter Circuit in the first stage.
1.0
1.-
Self Self Cooking Ratio
Self-Ceoking Ratio
Self-Cooking
0/1
0.1
02-
02- Ageir) nc
S0reEt Haman
spunzz: Hammnn
24
1.
96
Self-Dellvery Ratio
Self-Dellvery Ratio
Self-Dellvery Ratio
(a) Asymmetric Advantages
(b) Coordination Ring
(c) Counter Circuit
Figure 8: Trajectories induced by the learned human models and human players in Asymmetric Advantages, Coordination Ring and Counter Circuit. Each point triangle denotes trajectory with the X-axis coordinate being the self-cooking ratio, which the ratio onions the player places in the pot the total amount of placements in the trajectory, and the Y-axis coordinate being the self- delivery ratio, which is the ratio deliveries given by the player to the total number of deliveries in the trajectory. Triangles and points denote trajectories induced by human players and learned human models, respectively. Different colors stand for different player indices. "BC" represents the learned human models, and "Human" denotes human players. Clearly, trajectories induced by the learned human models can not fully cover those by human players.
trajectories by measuring self-delivery ratio, i.e., the ratio of deliveries by the specific player the total delivery number in a trajectory, and self-cooking ratio, which is the ratio onions that the these
player places in the pot to the total pot placement number in a trajectory. The distributions trajectories are demonstrated in Fig. 8 From the figure, we can observe that the learned human models can not fully cover human behaviors. This suggests that evaluation results with the learned human models
provide comprehensive comparison among different methods.
25

---

Published as a conference paper at ICLR 2023
Event
Value
Optimal placement
Picking up a dish from dish dispenser Picking up ready soup from the pot
Table 16: Reward shaping for Asymmetric Advantages, Coordination Ring and Counter Circuit in the second stage.
Event
Value
Picking up a dish from dish dispenser Picking up ready soup from the pot
Table 17: Reward shaping for Distant Tomato and Many Orders in the first stage.
F.2 ABLATION STUDIES
F.2.1 POOL SIZE
Table 21 shows the average episode reward on layouts with different sizes the final policy pool for trainig the adaptive policy. Since increasing the pool size to 72 gives little improvement as suggested by the result, we use 36 in our experiments for computation efficiency.
F.2.2 POLICY POOL CONSTRUCTION
HSP has two techniques for the policy pool, i... (1) policy filtering to remove duplicated biased policies and (2) the use of MEP policies under the game reward for half of the pool size. We measure the performance with human proxies by turning these options off. For "HSP w.o. Filtering", we keep all the policies by random search in the policy pool, which results in a larger pool size of 54 (18 MEP policies and a total of 36 random search ones). For HSP w.o. MEP", we exclude MEP policies in the policy pool and keep all the biased policies without filtering, which leads to the same pool size of 36. The results are shown in Table. 22
F.3 COOPERATION WITH SCRIPTED POLICIES WITH STRONG BEHAVIOR PREFERENCES
Table 23 illustrates average episode reward and standard deviation (over 5 seeds) in all layouts with scripted policies. All values within difference of from the maximum value are marked in bold.
F.4
HUMAN-AI EXPERIMENT
F.4.1 EXPERIMENT SETTING
We recruited 60 volunteers by posting the experiment advertisement on a public platform. They provided with detailed introduction the basic gameplay and the experiment process. The Overcooked game was deployed remotely on a server that the volunteers could access with their browsers. According to the feedback, over percent of volunteers had no prior experience with Overcooked. We uniformly divided 60 volunteers into 5 groups assigned to each the layouts. We designed the experiment to last around 30 minutes for each volunteer to ensure the validity of the data. Due to availabilities of volunteers, experiments days. The experiment has two stages. In the first stage, which is called the warm-up stage, the participants are encouraged explore the behaviors of 4 given Al agents without time limit. After the first stage, they are required to comment on their game experience, e.g., whether the Al agents are cooperative and comfortable to play with, and rank the agents accordingly. In the second stage, each participant is instructed to achieve scores as high as they could in 24 games (4 AI agents x player positions x 3 repeats).
conducted within two consecutive
We remark that, on the environment side, different from human-Al experiments performed by prior works (Zhao et a |021 Carroll al. (2019) in Overcooked, we slow down the Al agents so that the Al agents have similar speed with human players. More specifically, 7 idle steps are inserted before each step of the AI agent. Such an operation is necessary since. in our prior user studies, we find that human players commonly feel uncomfortable if the Al agent is much faster and human players could contribute little to the score.
26

---

Published as a conference paper at ICLR 2023
Event
Value
Picking up a dish from the dish dispenser 3
Picking up ready soup from the pot 5
Table 18: Reward shaping for Many Orders in the second stage.
Event
Value
Picking up dish from the dish dispenser Picking up a ready soup from the pot
3
5
Useful tomato pickup
10
Optimal tomato placement
5
Placing tomato into an empty pot
-15
Table 19: Reward shaping for Distant Tomato in the second stage.
F.4.2 HUMAN FEEDBACK
We collected and analyzed the feedback from the participants to see how they felt playing with AI agents. Here we summarize the typical reflections.
1. In Coordination Ring, the most annoying thing reported is players blocking each other during movement. To effectively maneuver in the ring-like layout, players must reach temporary agreement on either going clockwise or counterclockwise. HSP the only Al able to make way for the other player, while others can not recover by themselves once stuck. For example, both FCP and TrajDiv players tend to take a plate and wait next to the pot immediately after one pot is filled. But they can neither take a detour when blocked on their way to the dish dispenser nor yield their position to the human player trying to pass through. The video recorded in the human study can be found in Part 4.2 of https: pestps://sites google..oooooooooooooooooooo.... com/v vivviie weweew//ss hspp- hw/hsp-ichr iclr
2.
In Counter Circuit, one efficient strategy is passing onion via the counter in the middle of the room: a player at the bottom fetches onions and places them on the counter, while another player at the top picks up the onions and puts them into pots. HSP is the only AI player capable of this strategy in both top and bottom places and performs the highest onion passing frequency cooperating with human players shown in Figure. (6b]
3. In Distant Tomato, one critical thing is that mixed (onion-tomato soups give no reward, which means two players need agree on the soup to cook. All Al agents perform well when the other player focuses on onion soups. However, all Al agents except for HSP fail to deal with tomato-preferring partners as shown in Table. 3 FCP, MEP, or TrajDiv agents never actively choose to place tomatoes and keep placing onions even when a pot has tomatoes in it, resulting in invalid orders. On the contrary, HSP chooses to place tomatoes when there are tomatoes in the pot. Participants commonly agree that the HSP agent is the best partner to play with this layout. The video recorded the human study can be found in Part 4.2 of https:: / /tss/sstsss..././.ee//eeee.eeees google. com/ wwwew/hssp-iie
4. In Many Orders, most participants claim that HSP is able to pick up soups from all three pots, while other AI agents only concentrate on the pot in front of them and ignore the middle pot even if the human player attempts to use it. Table. shows that HSP agent picks up most soups from the middle pot and meanwhile gets the highest average episode reward.
F.4.3 HUMAN PREFERENCE ON DIFFERENT AI AGENTS
Figure. illustrates human preference for different Al agents. In all layouts except relatively restricted and simple layout, Coordination Ring, human players strongly prefer HSP over other AI agents, In Coordination Ring, though human players rank MEP above HSP, HSP is still significantly better than FCP and TrajDiv.
Calculation Method: Human preference for different methods is computed as follows. Assume we are comparing human preference between method A and method B. Let N be the total number of human players attending the experiments in one layout, NA be the number of human players who
27

---

Published as a conference paper at ICLR 2023
rank A over B, and Ng be the number of those who rank B over A. "Human preference for method
A over method B" is computed as N
N
Ng.
The rcp NPP HPp
PCP NEP HEP
0.077
0.77
HSP
The Div FCP NEP HEP
0.83
0.83
10.37
50.077
4..4
0.23
.0.5
40.17
-0.5
4.2
40,85
0.30
Tha
The
<0.17
28.077
4..59
4.23
40.67
40.7
844
-02
4.35
0.23
0.54
TraDiv
Tr.Div
HSP
НEP
FCP
(a) Asy. Adv.
(b) Coor. Ring
(c) Coun. Circ.
(d) Dis.
Toma.
(e) Many Ord.
Figure 9: Human preference for row partner over column partner in all layouts.
F.4.4 SCORES IN THE SECOND STAGE
Table 24 shows average reward per episode during the second stage in all layouts. All methods have comparable episode rewards in Asymm. Adv and Coord. Ring. There is no room for improvement since all the methods have reached the highest possible rewards. In Counter Circ., the most complex layout this category, HSP achieves better performance than baselines: HSP has 155+ reward while most competitive baseline MEP has reward of 134+. We remark that the reward differ- ence between HSP and MEP is around 20, which exactly the value of I onion soup delivery. This implies that the HSP agent can, on average, deliver one more soup than all the baselines per game episode with humans, which is significant improvement.
28

---

Published as a conference paper at ICLR 2023
Pos.
Asy. Adv.
Coor. R.
Coun. Circ.
282.8(9.4)
161.3(1.)
95.9(2m)
FCP
2
203.8(8.2)
161.0(27)
92.7(1.3)
291.7(4.6)
161.8(07))
108.8(4.2)
MEP
2
203.4(2.0)
164.2(2.1)
111.1
289.3(8.8)
150.8
60.1(5.0)
TrajDiv
194.2(0.7)
(42.1
53.7(124)
1
300.3/2.2)
160.0(2.6
107.4(3.5)
HSP
2
217.1(3.3)
160.6(3.3)
106.6(3.0)
Table 20: Average episode reward and standard deviation (over 5 seeds) layouts for different methods played with human proxy policies. All values within standard deviations of the maximum episode return marked in bold. The Pos. column indicates the roles played by Al policies.
Pos.
Asy. Adv.
Coor. R.
Coun. Cire.
Asy. Adv.
Coor. R.
Coun. Circ.
policy pool size = 36
policy pool size =
1
282.8(9.4)
161.3(1.6)
95.9(2.0)
278.3
158.9(0.))
91.9(7.5)
FCP
2
203.8(8.2)
161.0(2.7)
92.7(1.3)
200.9(13.2)
156.9
90.7(4.8)
1
291.7(4.6)
161.8(u.7)
108.8(4.2)
298.2(s
157.3
104.6(5.0)
МEP
2
((2.0
164.2(2.1)
111.1
207.8(7.3
158. 158.9(3.0)
105.012.21
289.3(8..
150.8(3.1)
60. (0..0)
270.8(2.5)
142.5(2.8)
70. (6.7))
TrajDiv
2
194.2(0.7)
142.1
53.7(12.4)
192.8(8.7)
137.3(4.9)
63.8(8.2)
Table 21: Average episode reward and standard deviation (over 5 seeds) on layouts for different methods played with human proxy policies. The Pos. column indicates the roles played by Al policies.
Pos.
Asy. Adv.
Coor. R.
Cou. Cire.
1
308.5(4.4)
157.5(3.0)
94.0(2.7
HSP w.o. MEP (pool size 36)
2
219.6
157.7(2.5)
100.4a.1
311.3(8.1)
139.2(5.6)
80.1(4.6:
HSP w.o. Filtering (pool size 54)
2
209.3(4.0)
138.5(3.1)
88.7(0.9)
300.3
160.0(26)
107.4
HSP (pool size 36)
2
117.1
160.66.3)
106.6(3.0)
Table 22: Average episode reward and standard deviation (over 5 seeds) on layouts for different methods played with human proxy policies. The Pos. column indicates the roles played by AI policies.
29

---

Published as a conference paper at ICLR 2023
Table 23: Average episode reward and standard deviation (over seeds) in all layouts with scripted policies. All values within Within difference of 5 from maximum reward are marked in bold. The "1" and "2" indicates the roles played by Al policies.
396.3
229 229.9 (4.9
121.7(10.9)
116.0
104.43.4)
77.8
147.1
278.9
231.9
319.96655
322.313.99
2
HSP
357.3
371.46.3
120.8
114.9
110.53.7
79.3
149.3
277.7..7)
237.2
315.7
326.7
350.7(8.)
221.8(3.4
118. 118. (66.7)
107.1(53)
78. (3.1(134)
55.0v...
107.9(55)
17.2
154.30766)
254.6337)
292.6.15
2
TrajDiv
296 296.5(25.2)
358.2(6.0)
115.7
107.5(52
86.0(1.1)
59.3r29;
109.5(3.6)
29.6.141)
175.3
263.8(10.1)
298.8(3.2)
365. 92288)
230.3(6.3
123.4
99.017.3
87.4(6.)
52.5
110.7
14.3:3.6)
174.8(74)
227 (77.660.55
327.2 227.2
MEP
295. 7)..7)
366.8
124.63.6)
101.43.1
90.6339
53.5(1.))
112.1
25.9
185.919.9
(61.2) (612)
329. (12.7)
224.
365.7 (5.3)
230.5
112.1
97.003.)
62.7(95)
56.1(52)
72. 72.4026)
17.87.6)
158.3(7.)
279.8 8x1.8
324.244)
FCP
304.0(20.8
364 92.9)
106.0 ((.7).
(1.)(
64.8 64.80(88)
57.9(5.3)
73.0m1.5)
13.5(2.6
(.4.9
285.4(195)
334.1(5.6)
Oni. Pla. Deli.
Oni Midd. Cou.
Toma. Pla &Deli.
Toma. &Deli.
Oni. Every.
Dish Every.
Oni. Every.
Dish Every.
Toma. Pla.
Toma. Pla.
Scripts
Oni. Pla.
Asy. Adv.
Coor. Ring
Coun. Cire.
Dis. Toma.
Many Ord.
30

---

Published as a conference paper at ICLR 2023
Pos.
Asy. Adv.
Coor. Ring
Cou. Cire.
Dis. Toma.
Many Ord.
339.3(38.17)
185.0n9.73)
127.7
352.3(62.25)
312.4(58.73)
FCP
321.3 (33.3()))80)
180.7(22.8)
1 188.322.20)
(20.5(66)49)
321.3(61.12)
329.7(45.91
193.3(22.11)
136,9(27.00)
341.9 (65.07)
322.0(30.53)
MEP
2
(39.2(39.93)
183. 183.6(26.75)
(24.5(28.63)
313.9(78.29)
319.2(52.98)
329.0<33.
184.7(28.00)
(12.6(25.88
327.1(71.04)
312.9(62.22)
TrajDiv 2
318.8 (748874877)
176.8
105.0(3105)
316.0(77.65)
334.2(57.99)
336.0(35.55)
(55.5(38.92)
158.0(28.56)
331.6661.33)
384.3(47.500
HSP
2
318.8(48.97)
188.9
(55.2(23.45)
305.9(58.61)
380.7(62.27)
Table 24: Average reward per episode in all layouts with human players in the second stage.
31


Reviews:
[{"review_id": "vnusrwGFfd", "reviewer": "Reviewer_hpUU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the Hidden-Utility Self-Play algorithm to explicitly model human bias via a modification to the reward function used during self-play training. This is used in the domain of the cooperative game Overcooked, where a particular reward modification is shown to be effective both in experiments where it is paired with learned human models and in experiments with actual humans. The framework is demonstrated to be especially effective with real human partners.", "summary_of_the_review": "This is an interesting paper that shows the effectiveness of incorporating models of human biases in a particular domain. The main weaknesses are the question of how domain-specific this reward is (and how much domain knowledge it requires) and inadequate details about the human study. For example, in Section F.4.1, slowing down the speed of the simulation is noted for making the users more comfortable in contributing; these are the kinds of factors that could substantially impact the effectiveness of a solution, in addition to algorithmic contributions. The qualitative details in F.4.2 could also be augmented with quantitative metrics of the feedback, if available.", "strength_and_weaknesses": "Strengths\n- This paper focuses on the benefits of incorporating human biases in cooperating with them, an important area.\n- The paper is written clearly, and the appendix incorporates many details of the experiments that aid reproducibility.\n- The ablation studies and other experimental details are presented in a thorough way.\n\nWeaknesses\n- It is not clear how well the approach presented here would generalize and requires domain knowledge to be effective.\n- The human study, which shows the most compelling evidence for the effectiveness of this framework, needs more details. How were the volunteers recruited, how did they participate in the experiment, how long did the experiments take, what was their experience with Overcooked, etc.? The volunteer group is also younger and heavily male. How may this impact the results?\n- The \"Distant Tomato\" regime has a clear optimal policy - would just telling the human participants what that is have substantially impacted results? Specifically, is this a question of a \"hidden reward function\" (as the framework presents) or inadequate skill or knowledge of the domain by the human participants?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hhw81hmr0DD", "reviewer": "Reviewer_SZ9c", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors train agents with self play that have diverse preferences that differ from the original task and show that having a richer pool of agents generated with this technique yields policies that are better adapted to human coordination. \n", "summary_of_the_review": "Based on the new method alone I would not accept this paper. However, the thoroughness of evaluation sets a new standard and I feel that I learned something new and important from these empirical analyses. I would like to cite this paper in the future and that should be sufficient for acceptance. I would raise my score further if the authors can more greatly emphasize these contributions in their work \n", "strength_and_weaknesses": "The empirical results are excellent. In particular, I think the combination of ablations, human experiments, scripted bots, and imitation-trained policies go beyond most any other works and clarify key issues that were not carefully analyzed in previous works. For instance, the authors clearly show that the techniques used in most evaluations (imitation-learned policies and human interaction) are highly confounded since people adapt, and the imitation-learned policies don’t show much diversity. Their method only weakly improves over baselines in these tasks. In contrast, the use of specific scripted probes and a more qualitative evaluation revealed large discrepancies between actual coordination performance in the most important edge cases. \n\nThe algorithmic contribution is a weakness as it depends on significant hand-tuning of custom features specific for these specific Overcooked environments. I do not see how this approach could be easily adapted to a new tasks (or even an Overcooked level with different dynamics). On its own, I do not think this algorithm is a sufficient contribution to literature. I would have also liked to see comparisons or thoughts on more model-based towards generating diversity in Overcooked for example: Wu, Sarah A., et al. \"Too Many Cooks: Bayesian Inference for Coordinating Multi‐Agent Collaboration.\" Topics in Cognitive Science 13.2 (2021): 414-432.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "SglfM9sXdOQ", "reviewer": "Reviewer_ufwf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for learning a collaborative agent that is able to handle arbitrary and/or suboptimal human reward functions. The method leverages self-play, specifically fictitious co-play, where the policies the target policy plays alongside are learned according to different reward functions. A variety of reward functions are used to train the pool of policy in order to train the target policy alongside a variety of different preference types and to teach the target policy to be robust to different, potentially previously unseen, preferences. The proposed method is evaluated on the Overcooked task, a standard benchmark for human-agent collaboration, alongside learned human models, scripted policies, and real humans. Across the different \"humans\", the proposed method outperforms the baselines. The experiments ablate several aspects of the approach: how the policies in the pools are trained and the batch size used to train the target policy. The results suggest that both filtering and MEP policies (trained with a single reward function) trained policies are important and large batch sizes are needed to stable target policy performance. ", "summary_of_the_review": "Overall the paper is strong with some areas for improvement in terms of clarity. The idea is relatively simple and based on the results appears to improve policy performance, especially in experimental conditions including real humans. My only concern about acceptance to ICLR is its fit. The paper seems like a better fit to CoRL, AAAI, AAMAS, or ICML as it does not have a representation nor representation learning focus.", "strength_and_weaknesses": "Strengths:\n- Overall a well written paper.\n- The method is well motivated for why it is needed and is well grounded in the existing literature.\n- The experiments appear to be solid and compare against important baselines to evaluate different aspects of the proposed contributions. The authors made sure to explicitly evaluate how well the approach works when the target policy is deployed to collaborative with agents that have reward functions not included in the policy training pool. Additionally, the authors evaluate against a variety of \"humans\", including real humans. Assessing the method across different \"humans\" provides a good idea of how the method extends to the real world. \n- The use of random reward search to create a diverse pool of policies seem simple, but also effective. \n- The authors discuss limitations of the current approach and propose future work to address the limitations.\n\nWeaknesses:\n- It would be nice to see a discussion about what the ablation study results tell us about the proposed method. What does it mean for the MEP policies to be helpful? What does it mean about the method for a larger batch size to be important? How does the need for a large batch size align with what the baseline methods needs?\n- It is unclear to me how relevant the paper is to the topic areas ICLR focuses on. There is little to no discussion of representations nor representation learning methods. The paper seems like it is a better fit for CoRL, AAAI, AAMAS, and ICML.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KbdYR7zpNXe", "reviewer": "Reviewer_gDCQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel method called Hidden-Utility Self-Play (HSP) to solve a realistic and challenging problem that there exist human biases in human-AI interaction. The main contribution of this paper is proposing a hidden reward function to model human biases. The experiments are solid and sufficient to verify the performance improvement of HSP.", "summary_of_the_review": "This paper proposes a novel practical research problem and a novel algorithm to solve it. Besides, it has done solid and sufficient experiments to verify the performance of the proposed algorithm.", "strength_and_weaknesses": "## Strength\n1. This paper is well written and easy to follow. The related works are sufficiently discussed.\n2. The experimental settings and the baseline algorithms are described in details.\n3. Although this paper is an application work, it still includes some theoretical analysis as the base to support the proposed algorithm. The theoretical analysis is neat and sufficient to support the idea.\n4. No obvious flaws in my view.\n\n\n## Weaknesses\n1. The construction of the hidden reward function is implemented as the counting-based method which is too simple. Since the hidden reward function is belonging to a type of intrinsic reward, there are plentiful literatures that study on the construction or learning this reward. I suggest the authors can have a review on these and improve the work.\n2. In the procedure of policy filtering, the greedy policy selection suffers computational burdern as the authors illustrate. I wish the authors give a discussion on the possible way to improve it in the future work, since I don't wish the performance improvement is just based on more computation which is not good for the following works.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster