Learning Altruistic Behaviours in Reinforcement Learning without External Rewards
Authors: Tim Franzmeyer
Mateusz Malinowski
Joao F. Henriques
Venue: ICLR 2022 Spotlight (2022)
Keywords: reinforcement learning
altruistic behavior in AI
multi-agent systems

Abstract: Can artificial agents learn to assist others in achieving their goals without knowing what those goals are? Generic reinforcement learning agents could be trained to behave altruistically towards others by rewarding them for altruistic behaviour, i.e., rewarding them for benefiting other agents in a given situation. Such an approach assumes that other agents' goals are known so that the altruistic agent can cooperate in achieving those goals. However, explicit knowledge of other agents' goals is often difficult to acquire. In the case of human agents, their goals and preferences may be difficult to express fully; they might be ambiguous or even contradictory. Thus, it is beneficial to develop agents that do not depend on external supervision and learn altruistic behaviour in a task-agnostic manner. We propose to act altruistically towards other agents by giving them more choice and allowing them to achieve their goals better. Some concrete examples include opening a door for others or safeguarding them to pursue their objectives without interference. We formalize this concept and propose an altruistic agent that learns to increase the choices another agent has by preferring to maximize the number of states that the other agent can reach in its future. We evaluate our approach in three different multi-agent environments where another agent's success depends on altruistic behaviour. Finally, we show that our unsupervised agents can perform comparably to agents explicitly trained to work cooperatively, in some cases even outperforming them.

Published as a conference paper at ICLR 2022
LEARNING ALTRUISTIC BEHAVIOURS IN REINFORCE- MENT LEARNING WITHOUT EXTERNAL REWARDS
Tim Franzmeyer University of Oxford frtimê robot s. ox. ac. uk
Mateusz Malinowski
João F. Henriques University of Oxford
DeepMind mateus z com
joao8robon ox uk
ABSTRACT
Can artificial agents learn assist others in achieving their goals without knowing what those goals are? Generic reinforcement learning agents could be trained to behave altruistically towards others by rewarding them for altruistic behaviour, i.e., rewarding them for benefiting other agents in given situation. Such an approach assumes that other agents' goals are known so that the altruistic agent can cooperate in achieving those goals. However, explicit knowledge of other agents' goals is often difficult to acquire. In the case of human agents, their goals and preferences may be difficult to express fully; they might be ambiguous beneficial to develop agents that do not depend external supervision and learn altruistic behaviour in task-agnostic manner. We propose to act altruistically towards other agents by giving them more choice and allowing them to achieve their goals better. Some concrete examples include opening a door for others or safeguarding them to pursue their objectives without interference. We formalize this concept and propose altruistic agent that learns to increase the choices another agent has by preferring to maximize the number of states that the other agent can reach in its future. We evaluate our approach three different multi-agent environments where another agent's ss success depends on altruistic behaviour. Finally, we show that our unsupervised agents can perform comparably to agents explicitly trained to work cooperatively, in some cases even outperforming them.
even contradictory. Thus,
INTRODUCTION
Altruistic behaviour is often described as behaviour that is intended to benefit others, sometimes at a cost for the actor (Dowding and Monroe 1997| Fehr and Fischbacher 2003). Such behaviour is a desirable trait when integrating artificial intelligence into various aspects of human life and society such as personal artificial assistants, house or warehouse robots, autonomous vehicles, and even recommender systems for and entertainment. By observing and interacting with us, we may expect that artificial agents could adapt to our behaviour and objectives, and learn to act helpfully and selflessly. Altruistic behaviour could be step towards value alignment (Allen al. 2005 Gabriel 2020), which aims to incorporate common-sense human values into artificial agents.
Typically, we could achieve such altruistic behaviour through various forms of supervision such as providing ground-truth actions at each time step. training agents with reinforcement learning (RL) and suitable rewards, or through imitation learning (Song et al. 2018). However, none of the approaches above scale up easily. They either require large amount of supervision carefully crafted rewards that can easily be misstated, leading to unwanted behaviour (Russell 2019 ch. 1).
How can one agent support another agent without knowing its goals? One clue might be the instrumental convergence hypothesis (Bostrom) 2017; Omohundro) 2008 Russell 2019). which states that intelligent agents with varied goals are likely to pursue common subgoals which are generally useful (instrumental). Some examples are resource acquisition, cognitive enhancement or self- self-preservation, which all increase agent's chance of achieving almost arbitrary final goals. This hypothesis has been validated theoretically under many models, including resource games (Benson- Tilsen and Soares 2016) and large classes of policies in discrete MDPs (Turner al. 2019).

---

Published as a conference paper at ICLR 2022
While instrumental convergence is central to the discussion of value alignment and safe AI (Bostrom, 2017), since many instrumental subgoals have harmful effects, we believe that it is also a key to supporting agents with ill-defined goals and values, such as humans. The reason is that enabling instrumental subgoals for other agents (or not impeding them) can be heneficial, for a wide variety of goals and preferences. Since these subgoals occur frequently for rational agents, enabling them has the highest chance of success in the absence of more information about the other agent's s preferences, even if is not guaranteed in the worst case.
We speculate that having the ability to reach many future states is one of the most general convergent subgoals. subsumes self seff-rreservation (avoiding absorbent states), resource acquisition (if they are prerequisites to some actions), and generally maintaining the ability to pursue many goals. There is theoretical evidence that many optimal agents pursue this subgoal (Turner et al. 2019) (see 3.2...2). Thus, we propose to train agents to support other agents by maximizing their choice (future state availability). This unsupervised approach learns altruistic behaviour without any extrinsic supervision such as rewards expert demonstrations.
We evaluate our methods in three diverse multi-agent environments. We always assume there are at least two agents: the leader agent that executes its own policy and can be trained using standard supervised methods, and an altruistic agent whose role is to help the leader. The performance of the altruistic agent is thus defined as the reward (success) achieved by the leader agent. In all our environments, the overall success of the leader agent depends on the altruistic agents' behaviour. We show that our unsupervised approach outperforms unsupervised baselines by a large margin and, in some cases, also outperforms the supervised ones. Finally, we demonstrate possible failure cases of our approach where maximising the leader agent's choice can lead to suboptimal behaviour.
Our work makes the following three contributions:
We devise multi-agent RL framework for intrinsically motivated artificial agents that act altruistically by maximising the choice of others.
We define and evaluate three task-agnostic methods to estimate the choice that an agent has in a given situation, which are all related to the variety in states it can reach.
We experimentally evaluate our unsupervised approach in three multi-agent environments and are able to match and, in some cases, outperform supervised baselines.
RELATED WORK
To the best of our knowledge. we are the first to experimentally evaluate unsupervised agents with purely altruistic objectives. However, there are many related concepts in the literature.
In human-robot cooperation, robotic agent aids human agent in achieving its goals (Pérez- D' Arpino and Shah 2015 Hadfield Menell et al. 2016 Baker et al. 2006 Dragan and Srinivasa,
2013 Fisac et al. 2017 2020 Javdani al. 2015 Dragan and Srinivasa 2013 Macindoe al,
2012 Pellegrinelli et al. 2016). Methods from Inverse RL (IRL) are often employed to infer human goals, which are then utilized by the robot agent support the human. IRL. itself aims learn objectives from observations and can be used in single-agent (Fu et al.) 2017) and multi-agent scenarios (Song et al. 2018 Yu et al., 2019 Jeon al. 2020). However, IRL relies the existence of expert demonstrations, which are often difficult to get at scale. In complex environments, it also often suffers from ambiguity of solutions (Arora and Doshi 202T).
In single-agent reinforcement learning, empowerment which measures an agent's capacity to used enable prrrinically-motivate exploration (Gregor et al., 2016) Volpi and Polani 2020). Empowerment is also used for multi- agent cooperation (Guckelsherger et al. 2016 Du et al.] 2020). [Du al. (020) use empowerment to develop helper agent that assists (simulated) human agent by maximizing the human's ' empowerment, constituting the research work most similar to ours. In contrast to our approach, requires privileged access an environment simulator and therefore does not allow to learn helpful or altruistic behaviour only from observation. Furthermore, the approach is not unsupervised.
affect its environment (Klyubin al.) 2005 2008)
There also mathematical formalizations of instrumental convergence (Bostrom, 2017). Benson- Tilsen and Soares (2016) analyze a MDP that makes finite resource allocation explicit, and find that optimal agents with arbitrary reward functions tend to deplete available resources. Turner
2

---

Published as a conference paper at ICLR 2022
et al. 2019) propose "power" convergent subgoal, which they define the average difference between the state value of an optimal policy and the reward in the same state. They show that, for environments with certain symmetries, larger proportion of optimal agents prefer states with higher power. In sec. 3.2 we will describe these symmetries and relate the result to our method.
METHODS
In this section, we formalize framework. We start with the generic definition describing multi- agent setting, Next, we describe our framework where we show various approaches to estimate choice for a single agent, and how it can be applied a two-agents Markov Game
Markov Game We consider Markov Game (Littman 1994), which generalizes a Markov Decision Process (MDP) to a multi-agent scenario. In a Markov Game, agents interact in the same environment. At time step t, each agent (the ith of a total of N agents) takes the action at, receives a reward ri. and finally the environment transitions from state sf to sh+1. A Markov Game is then defined by a state space S (s' E S), a distribution and reward function ri(s, a1..... aN) ) of each agent i, an environment state transition probability P(s*+1 [ss, (.....,,N) and finally the agents' discount factors 7
initial states 7), the action space A, (af E A,))
3.1 ESTIMATING CHOICE FOR A SINGLE AGENT
We first consider a single-agent scenario, i.e. N = 1, where only a leader agent, indicated by the subscript L, interacts with the environment through its pretrained stochastic policy TL. We assume that the leader acts Boltzmann-rationallyy i.e. that it chooses high-value actions with higher probability. We believe this to be a reasonable assumption, as, in comparison to deterministic policies, stochastic policies are more robust (Zhang et al., 2020), and often achieve better results in real- world-alkkee partially observable stochastic domains Kaelbling et al. 1998).
We denote the leader agent's generic choice in a given state (s.s), for which we propose concrete realizations below. Each method relies on the random variable S with values x s E S, which refers to the leader agent's state after environment transitions from a starting state 8t. Its probability mass function is defined as the n-step state distribution of the underlying single-agent MDP, conditioned on the current state: p(s p(s P(St+" S|TT, s)).
Discrete choice. Our first derived method simply defines the choice of the leader agent in state s as the number of states that it can reach within n transitions, which we refer to as its discrete choice:
(** Irange (s
(1)
where range(X) is the set of all values that a random variable X takes on with positive probability and |- measures the size of that set. While this count-based estimator of choice is intuitive and easily interpretable, can hardly be estimated practically in large or continuous state spaces. also discards information about the probability of reaching these states.
Entropic choice. It can be shown that the entropy of random variable X acts as lower bound for the size of the set of values that X takes on with positive probability (Galvin) 2014, Property 2.6), i.e. HXX) S log Irange(X) X)). We define a lower bound of the discrete choice by computing the Shannon entropy of the n-step state distribution, which we refer to as the agent's entropic choice:
ECT H(S (s') - Σ P(s s/TL, st) log (P(st*"=s]TL.S
= s|TL.
(2)
SES
which estimates the agent's s choice as the variety in its state after transitions. Unlike eq. can be computed in continuous state spaces or efficiently estimated by Monte Carlo sampling-
I FCE
Immediate choice. To further simplify entropic choice and reduce its computational complexity, we may limit the look-ahead horizon to = and assume an injective relationship from actions to states, i.e. no two actions taken at s² lead to the equivalent state g*+1, This assumption is often true in navigation environments, where different step-actions result in different states. We can then simplify the one-step state distribution of the leader agent to p(s (*)) = P(S s')
Taa;, als'), and compute simplified, short-horizon entropic choice, the immediate choice:
IC2(s') = H( |s')) (alss))))
(3)
3

---

Published as a conference paper at ICLR 2022
Immediate choice (IC) can be easily computed as the entropy its policy conditioned on the current state. Even though the assumptions made for immediate choice often do not hold in complex or real-world environments, we found empirically that this objective can yield good results.
3.2 OPTIMALITY OF CHOICE AN INSTRUMENTAL CONVERGENT SUBGOAL
Turner et al. (2019) amalyze the instrumental convergence of optimal agents on power-seeking subgoals and show that optimal policies tend keep their options open (Prop. 6.9). They consider two distinct actions and a' taken at state 8', leading into two sets of possible future states (for an infinite horizon). These sets of future states are represented as nodes in two graphs, respectively G' and G" (with edges weighted by the probability of transitioning from one state to another). They also assume that the states in G G' can only be reached from s' by taking actions a or a'. In the case where G' is "similar" to a subgraph of G", in the sense that they are equivalent up to arbitrary swapping of pairs of states, the authors prove that the probability of a being optimal is higher than the probability of a being optimal (for most reward function distributions). Therefore, if G' contains more states than G, an optimal agent will choose a' over a.
Turner et al. 2019) thus lend theoretical support our proposal: while there is guarantee that any one optimal policy (corresponding to a rational agent with arbitrary reward function) pursues higher choice, in expectation (over bounded space of reward functions) most policies do choose actions that lead to higher choice, all else being equal. As such, while we may not know a rational agent's 's concrete goals, there is high chance that choice works as an instrumental subgoal.
3.3 COMPARISON BETWEEN CHOICE AND EMPOWERMENT
The empowerment Klyubin al. 2005) of leader agent in a given state s and for horizon is (s" = max est 1(St";; A s') = (ax H (St+*((s)) - (S AA" 8*), with a" as sequence of actions of the leader agent and w a probing distribution over its n-step action sequences. When setting the probing distribution w equal to the leader agent's policy, equation 3.3 simplifies to Es's ECT H(S++1 A s'), with ECT(s") as the entropic choice of the leader agent introduced in equation 2 If we further assume deterministic environment transitions, then empowerment becomes equal to entropic choice, i.e. (s²) (s))
In contrast to the previously introduced methods to estimate choice of another agent, empowerment of another agent cannot be estimated from observations of the environment transitions. To estimate another agent's empowerment in a given state (E7, access to its action space as well as privileged access to an environment simulator are be required, which violates the main assumption of our research work, i.c. learning to assist others only from observations of the environment transitions. Even when assuming privileged access, computing empowerment in large or continuous- state environments often remains infeasible (Mohamed and Rezende 2015 Gregor et al. 2016 2016/Zhao) ctt al. 2020), as it requires maximizing over all possible probing distributions w of the leader agent. In contrast, estimating state entropy, as needed for the computation of the metrics introduced in this work, is feasible in large and continuous environments (Seo et al. 202T Mutti et al. 2020
3.4 BEHAVING ALTRUISTICALLY BY MAXIMIZING ANOTHER AGENT'S CHOICE
we now apply them a Markov Game of two agents. The main hypothesis is that maximizing the choice of another agent is likely to allow it to reach more favourable regions of the state-space (for many possible policies of the agent), thus supporting it without task-specific reward signal.
Having considered three methods estimate an agent's choice (eq. I 3
Altruistic agent's policy definition. In this Markov Game, one agent is the leader, with the subscript L, and another one is the altruistic agent, with the subscript A. We define the optimal policy of the altruistic agent as the one that maximizes the future discounted choice of the leader,
arginax A
A'A Cz(s'),
where the generic choice Cs'(s") can be estimated by one of several methods: discrete choice DCT entropic choice FC or immediate choice TCr(s").
4

---

Published as a conference paper at ICLR 2022
Conditional estimates of choice. As the agents interact in the same environment, they both have influence over the system state s. which contains the state of both agents. This makes applying single-agent objectives based on the state distribution (such eq. Tand 2 difficult to translate to multi-agent setting, since the states of both agents are intermingled. For example, an altruistic agent that maximizes entropic choice naively (eq. 2 will maximize both the state availability of the leader agent (which mirrors the single-agent entropic choice) and its own state availability (which does not contribute towards the altruism goal).
To maximize entropic choice without also increasing the entropy of the altruistic agent's actions, we propose to condition the choice estimate on the altruistic agent's ' actions over the same time horizon, denoted by the random variable A Att+n-
(C),($*) H( [Atittn1 s s").
(5)
In order to better understand eq. S we can use the chain rule of conditional entropy (Cover and Thomas 2005, ch. 2) to decompose it into two terms: ECT = H(S A 8')
ITL, respectively the joint entropy of the states and actions, and the entropy of the actions. Therefore, we can interpret this objective as the altruistic agent maximizing the variety of states and actions, but subtracting the variety of its own actions, which is the undesired quantity. We can also relate eq. 5 to discrete choice (eq. [1.. Using the fact that H(X|E) S log |range(P [ for a random variable X and event E (Galvin 2014| Property 2.1 we that eq. is a lower bound for a count-based choice estimate (analogous to eq.
I also conditioned
on the altruistic agent's ss actions: (s") S log (s') = log irange (S AA 1 . ,T,
However, assuming simultaneous actions, the immediate choice estimate (eq. 3) stays unchanged,
i.e (s') = Hx (als')la'a) = ((T (als))). The technical details of how these estimates can
be computed from observations of the environment transitions are given in Appendix A
4 EXPERIMENTAL EVALUATION
We introduce three multi-agent environments of increasing complexity in which the success of a leader agent depends on the behaviour of one or more additional agents. In each environment, we first evaluate a subset of the proposed methods for choice estimation (DCE. ECE and IC₁)) by comparing the estimated choice of the leader agent in minimalistic scenarios. We then evaluate approach of behaving altruistically towards others by maximizing their choice (section and measure performance of our approach as the reward achieved by the leader agent. We provide videos of the emergent behaviours in the supp. mat. (see appendix (F). We compare our method to both an unsupervised and a supervised approach. Note that the supervised approach has stronger assumptions, it requires direct access the leader agent's reward function. We do not consider inverse RL (IRL) as a relevant baseline, as it would rely on demonstrations of expert behaviour, which we do not assume. Even if perfect knowledge of the state transition probabilities is assumed, this does not allow generating expert demonstrations of the leader agent's 's policy, as its expert policy would in turn depend on the policy of the altruistic agent, which is yet to be found by IRL.
4.1 DISCRETE ENVIRONMENTS WITH CONTROLLABLE GATES
We start by considering three different scenarios on grid, illustrated in Fig. D (top row), with the starting positions of the leader (green) and an additional agent (blue) shown in faded colors, obstacles are gray, and agents may move in one of the four cardinal directions or stay still.
Choice estimate analysis. We first verify whether the estimated choice for each state (agent position) correctly maps to our intuitive understanding of choice (that is, the diversity of actions that can be taken). Therefore, we conducted an analysis of the estimated choice of the leader agent using a simplified version of the environment (Fig. [ top left), in which only the leader agent is present and selects actions uniformly at random. Fig. [ (bottom row) shows the three different methods of estimating choice evaluated for each possible cell position of the leader agent. We can observe that states in less confined areas, e.g. further away from walls, generally feature higher choice estimates, with the least choice being afforded by the dead end at the right. All three method's estimates are qualitatively similar, which validates the chosen approximations. In line
In appendix F. we evaluate performance in a non-spatial environment.
5

---

Published as a conference paper at ICLR 2022
10
10
8
8
2.1 2.1
1.9 1.9
1.2
1.1
1.4
1.1
1.1
1.1
0.7
10
11
14
14
12
10
2.1 2.3
2.4 2.4
2.2
2.1
1.4
1.4
1.6
1.6
1.1
1.6
1.4
1.1
10
10
10
10
2.1 2.1
2.1 2.1
1.7
1.1
1.4
1.1
1.1
1.4
1.1
1.1
Figure 1: Top row: Single-occupance grid environments in which agents either move to free adjacent cell or stay still. The green apple (reward +1) can only be obtained by the leader agent (in green) and other external rewards exist. Grey cells are blocked. Bottom row: Visualisation of the estimated choice of the leader agent when positioned at the respective cells. Left to right: discrete choice DCT, entropic ECE and immediate choice ICL (eq. and 3
with the simplifications made, the immediate choice (IC) estimates tend to be more local, as can be observed when comparing the estimates for the cell at row column 4. In conclusion, these results qualitatively agree with an intuitive understanding of choice of agent in grid environment.
Environment setup. In the Door Scenario (Fig. 1 top center), the door switch (row 1, col. 8) only be operated by the altruistic agent. The door (row 2, col. 4) remains open long the altruistic agent is on the switch cell and is closed otherwise. As the leader agent always starts to the left of the door and the altruistic agent the right, the leader agent only attain its goal, the apple (row 2, col. 6), the altruistic agent uses the door switch to enable the leader agent to pass through the door. In the Dead End Scenario (Fig. [ top right), the door always open, and the leader agent's target object (green apple) is moved to the top right cell. Hence, the leader agent can obtain the apple without additional help from the altruistic agent. However, the altruistic agent could potentially block the path by positioning itself at the entry to the dead end. This situation would be the opposite of altruistic behaviour and is, of course, undesired. We compare to a supervised approach, to Assistance via Empowerment (AvE, (Du et al. 2020 and random-policy baseline.
Assistance via Empowerment baseline. We compare with the recently-prooosed AvE, which has similar goal (Du et al. 2020). There are major differences: AvE is not unsupervised, and requires privileged access to an environment simulator to produce estimates. Hence, its use in real black-box environments is limited. We used the authors' implementation with fixed hyper- parameters, except for the crucial horizon n. for which we present a sweep in app. B
Training. We start by pretraining the leader agent with Q-Learning (Watkins Dayan 1992, with the altruistic agent executing a random policy. Hence, after convergence, the leader agent's policy targets the green apple. Appendix lists details and parameters. Afterwards, the leader agent's learning is frozen and the altruistic agent is trained; it always observes the position of the leader agent $L, its own position 8A, and the environment state Seav. which is composed of the door state (open, closed) and the food state (present, eaten). The altruistic agent is trained with Q-Learning to maximize the discounted future choice of the leader agent (see eq.. [ For that, it uses one of the three proposed methods such eq. eq. eq. [ detailed in appendix A.
Results. We investigate the developed behaviour of the altruistic agent after convergence for different choices of the hyperparameters look-ahead horizon n (1,3, 3, 12) (which determines the scale at which choices are considered) and discount factor E {0.11 0.7} (which defines whether the altruistic agent gives higher importance to the short-term or long-term choice Success is binary: either the leader agent attains its goal (green apple), or not.
leader agent).
In the Door Scenario (Fig. top center), we found that, for longer horizons n and higher discount factors Ya. the altruistic agent opens the door to allow the leader agent to reach its target, by occupying the switch position (square outline; row 1, col. 8). For smaller n and lower Tas the altruistic agent does not execute any coordinated policy and the leader does not succeed. Using the AvE method, we find that it only opens the door for n = 3, but fails to do so for n = and n 12.
In the Dead End Scenario (Fig. I top right), observe that, for longer horizons and large discount factors Ya. the altruistic agent stays out of the leader agent's way by occupying far-away cell (square outline; 1, col. 6). For short horizons and high discount factors Yas the altruistic

---

Published as a conference paper at ICLR 2022
Table Comparison of the estimated choice (IC) of leader agent in a high performance scenario (with a cooperative partner) vs. a low performance scenario (with randomly-acting partner). IC correlates with high performance.
Performance
Low
High
Figure 2: Left: In Level Based Foraging (LBF) two agents must cooperate to receive rewards for eating apples. The leader agent is green, altruistic agent is blue. Right: In Tag, leader (green) tries to escape from adversaries (red colors). Altruistic agents (who may help the leader) are blue.
Tag
IC
26.9%
56.7%
Reward
-7.55
-2.11
IC
19.1%
55.3%
LBF Norm, Reward
4.0%
94.3%
agent actively blocks the entry to the hallway that contains the target (circle outline; row 3, col. 7), to prohibit the leader agent from entering this region of low estimated choice (recall that the choice for each cell is visualized
Fig. bottom right). This failure case can be prevented by having large enough horizon and discount factor Ya. analogously to the selection of the temperature hyperparameter in maximum entropy single-agent RL (Haarnoja and Abbeel 2018). We find that this configuration performs consistently better than others in both scenarios, and hence is more preferred. On the other hand, the AvE method does not block the path of the leader agent for n = 1, and 12.
but blocks its path for
We found that the resulting behaviour of our approach is independent the used method for choice estimation, i.e. either discrete choice (eq. [ entropic choice (eq. 2 yield the same outcome, with immediate choice (eq. 3) being a special case of entropic choice. As for the AvE baseline, we hypothesize that the variance of results is due to the nature of the proxy used practice, which includes components of empowerment from both agents (sec. (.4). The binary outcomes for all hyperparameter combinations are given in appendix [ We also compare supervised baseline (receiving a reward when the leader obtains the apple), in which case the leader always succeeds.
4.2 LEVEL-BASED FORAGING EXPERIMENTS
Computational efficiency. Due the computational complexity resulting from the need to estimate long-term distribution of states, p(s"=1|s) (²"), we focus on immediate choice (IC) to estimate the leader agent's 's choice in the remaining sections. Furthermore, in rare state-action sequences, the assumptions made for IC, i.c. deterministic environment transitions and an injective relationship from actions states, may not hold. Nonetheless, we did find this adversely affect the results. Due to its dependence on access to the environment simulator and its computational complexity, we do consider the AvE baseline for the remainder of experiments.
Setup. We use oolly-bbbervvabl multi-agent environment that enables cooperation among agents (level-based foraging, LBF, [Christianos et al. (2020)) to evaluate the performance of altruistic agents in more complex environments with discrete state spaces. We compare our method to maximum-entropy approach from single-agent RL (Mutti et al. 2020) and a random-policy baseline. A visualization of the environment is depicted Fig. 2 (left). The two agents can forage apples by simultaneously taking positions at different sides of a targeted apple, yielding a fixed reward. We first train two agents which receive an equal reward for foraging using Deep Q-Learning (DQL. Van Hasselt et al. (2015). corresponding to sully-sue shared- reward in multi-agent reinforcement learning (MARL). We then take one of these pretrained agents that has learned to forage apples when accompanied by cooperating agent, freeze its policy. and place it as the leader agent (green) into the test scenario (additional details are provided in app. C).
assess the level of
Choice estimate analysis. We first qualitatively evaluate IC as an estimator for choice in Fig. by comparing representative scenarios. To quantitatively analyse IC estimator for the leader agent's 's choice, we compare the leader agent's average (over 100 episodes) two scenarios, one in which it can acquire many rewards, i.e. the other agent acts cooperatively, and one where it can

---

Published as a conference paper at ICLR 2022
1%
Policy Entropy of Leader Agent
Based For Foreging 6x6
85%
0.5
Reward of Leader Agent
1.0
0.4-
66%
0.3
0.6
Supervised
....
Random MaxEnt
Ours
a>
0.2
%
30
0
Environment Steps
ler
For two representative positions of the leader agent, its immediate choice (IC) estimates are given as a percentage of the maximum possible. Left (LBF)): The leader agent has low IC when having to wait for another agent to support it in foraging an apple and high IC when having the option of approaching multiple apples. Right (Tag): The leader has low when chased by the adversaries and high IC when it has multiple escape paths.
Figure 3:
Figure 4: Normalized reward of the leader agent (right vert. axis, green) when the altruistic agent is trained to maximize the leader's choice (ours), acting (random), receiving the same reward as the leader (supervised), or maximizing the state entropy (Mutti et al. (020). Left vert. axis (blue): internal reward our altruistic agent (estimated choice of the leader).
acquire only few rewards, i.e. the other agent takes random actions. We show the results in Table 1 We observe that the leader agent's estimated choice is substantially higher when it is able to acquire high rewards. Note that the IC estimate does not have privileged access the reward function of the leader agent, and so this experiment evaluates its worth as a generic proxy for the leader's ' reward. Assuming that an agent able acquire higher rewards when having more choice, these results indicate that IC is a reasonable estimator for the leader agent's 's choice in LBF.
Training. We now consider an environment that consists of the previously pretrained leader and an additional altruistic agent, which is trained from scratch and does not receive reward for foraging apples, but is rewarded according to the leader agent's choice. Its reward is given as the current estimate of the leader agent's IC (eq. 3 and is trained using DQL. To compute its internal reward signal, the altruistic agent would therefore need to estimate the state transition probabilities, as detailed in A.2 To decouple our approach's 's performance from that of the state transition estimator, instead directly compute the altruistic agent's 's reward using the leader agent's 's policy.
Results. We define the performance of the altruistic agent not as its achieved internal reward but as the reward achieved by the leader agent, i.e. its performance in enabling the leader agent to forage apples. Fig. 4 shows comparison of the altruistic agent's performance to that achieved by baselines (two unsupervised and one supervised), averaged over 5 random seeds, with the standard deviation as the shaded area. can be observed that the performance of the altruistic agent converges to a similar performance to that of the supervised agent, and outperforms the baseline approaches by a large margin. Furthermore, the IC improvement of the leader agent is correlated with its reward improvement, which supports using IC as reasonable proxy for the choice of the leader agent.
4.3 MULTI-AGENT TAG GAME WITH PROTECTIVE AGENTS
Setup. We multi-agent tag environment (Tag, Mordatch and Abbeel (2018); Lowe et al. (2077); Terry et al. (2020),, illustrated in Fig. 2 (right), to evaluate the capabilities of altruistic agents complex environments with continuous state spaces. Adversaries are rewarded for catching the leader, which in turn receives a negative reward for being caught or crossing the environment boundaries. To speed up training, altruistic agents additionally receive small negative reward for violating the environment boundaries. We pretrain the adversaries and the leader (without the presence of altruistic agents) using MADDPG (Lowe et al., 2017) and DDPG (Lillierap et al., 2016) respectively. After pretraining, the adversary agents have learned to cooperatively chase the leader agent, which in turn has learned to flee from the adversaries. Exact setup specifications and all parameters are given in appendix D
Choice estimate analysis. As done for LBF, we evaluate the IC of the leader agent in representative scenarios in Fig. 3 We also quantitatively evaluate IC as an estimator for the leader agent's s choice, by comparing the leader agent's IC per timestep for a scenario in which it receives high rewards to
8

---

Published as a conference paper at ICLR 2022
Table 2: Results of the Tag experiments (mean and standard deviation over 5 random seeds). Refer to sec. for more details.
Method
# Caught 4
None
0.55+00.56
Random
7.12
Cage
6.80
Supervised
6.94+1.13
Figure 5: Example behaviour of altruistic agents (blue) that learned to actively defend the leader agent (green) from the adversaries (red) in Tag. Obstacles are black. The trajectories taken by some of the agents in the last 10 steps are shown dotted lines.
Supervised + Cage
6.80=
Ours
2.87.10.96
where it receives low rewards. We again hypothesize that the leader agent is able acquire higher rewards when having more choice. Table shows that the estimated choice is substantially higher in the high-success scenario, indicating that IC is reasonable estimator also in Tag.
Training, We freeze the pretrained policies of the adversary agents and the leader agent and insert three additional altruistic agents which observe all agents but are not observed themselves. Each additional altruistic agent's internal reward signal is given as the IC of the leader agent (equation which is directly computed done in LBF (see 4.2).
Results. Performance of the altruistic agents defined the times per episode that the leader agent is caught by the adversaries, i.e. the lower the better. In Table 2 the performance of the team of three altruistically trained agents (ours) compared three relevant baselines, with the altruistic agents either removed (None), acting randomly (random), or solely receiving small negative reward for violating the environment boundaries (cage). In contrast to LBF, we do not compare to unsupervised exploration approach, as we are not aware of such an implementation for cooperative MARL.. Additionally, we report results for the case which the altruistic agents receive the same reward as the leader agent (supervised), possibly appended by negative reward for violating the environment boundaries (supervised + cage). can be observed that our approach outperforms all relevant baselines by substantial margin and also outperforms the supervised approach. We hypothesize this to be due to the dense internal reward signal that our approach provides, compared the sparse rewards in the supervised scenario: recall that in the supervised scenario the additional altruistic agents receive a large negative reward only when the leader agent is caught by the adversaries, whereas our approach provides dense reward signal that corresponds the current estimate of the leader agent's 's choice. Fig. 5 displays the emerging protective behaviour of altruistic agents trained with our approach. Results videos are found in the supplemental material.
CONCLUSIONS
We lay out some initial steps into developing artificial agents that learn altruistic behaviour from observations and interactions with other agents. Our experimental results demonstrate that artificial agents can behave altruistically towards other agents without knowledge of their objective any external supervision, by actively maximizing their choice. This objective is justified by theoretical work on instrumental convergence, which shows that for large proportion of rational agents this will be useful subgoal, and thus can be leveraged to design generally altruistic agents.
This work was motivated by desire to address the potential negative outcomes of deploying agents that are oblivious to the values and objectives of others into the real world. As such, we hope that our work serves both as a baseline and facilitator for future research into value alignment in simulation settings, and as a complementary objective to standard RL that biases the behaviour towards more altruistic policies. In addition to the positive impacts deployed altruistic agents outside of simulation, remark that altruistic proxy objectives do not yet come with strict guarantees optimizing for other agents' rewards, and identify failure modes (sec. 4.1) which are hyperpprrameter-deeeeddenn and which hope provide interesting starting points for future work.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
We addressed the relevant aspects in our conclusion and have no conflicts of interest to declare.
7 REPRODUCIBILITY STATEMENT
We provide detailed descriptions of our experiments the appendix and list all relevant parameters in [able All experiments were on single cores of Intel Xeon E7-8867v3 processors (2.5 GHz). Training times are given in the respective sections in the appendix. For the LBF and Tag experiments, we report mean and standard deviation over five different random seeds. The Gridworld experiments yield deterministic results. We will provide the source code for all experiments conducted with the final version of this publication. We created detailed instructions on how to run the code in order to replicate the experimental outcomes presented in this work.
8
ACKNOWLEDGEMENTS
We thank Thore Graepel and Yoram Bachrach for their helpful feedback. We are also grateful to the anonymous reviewers for their valuable suggestions. This work was supported by the Royal Academy of Engineering 201819)
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Allen, C., Smit, I. and Wallach, W. (2005), 'Artificial morality: Top-down, bottom-up, and hybrid approaches', Ethics and Information Technology 7(3).
Arora, S. and Doshi, (2021), 'A survey of inverse reinforcement learning: Challenges, methods and progress', Artificial Intelligence p. 103500.
Baker, C. L., Tenenbaum, J. B. and Saxe. R. (2006), "Bayesian models of human action understanding', Advances in neural information processing systems 18, 99.
Benson-Tilsen, T. and Soares, N. (2016), Formalizing convergent instrumental goals, in 'Workshops at the Thirtieth AAAI Conference on Artificial Intelligence'.
Bostrom, (2017), Superintelligence, Dunod.
Christianos, F., Schäfer, L. and Albrecht, S. (2020), 'Shared Experience Actor-Critic for Multi- Agent Reinforcement Learning' Advances in Neural Information Processing Systems 33.
Cover, T. M. and Thomas, J. A. (2005), Elements of Information Theory, John Wiley & Sons.
Dowding, K. and Monroe, R. (1997), 'The Heart of Altruism: Perceptions of Common Humanity', The British Journal of Sociology 48(3).
Dragan, A. D. and Srinivasa, S. S. (2013), 'A International Journal of Robotics Research 32(7), 790-805.
formalism for shared control', The
Du, Y, Tiomkin, S., Kiciman, E., Polani, D., Abbeel, P. and Dragan, A. (2020), 'Ave: Assistance via empowerment' Advances in Neural Information Processing Systems 33.
Fehr, E. and Fischbacher, (2003), 'The nature of human altruism'.
Fisac, J. F., Gates, M. A., Hamrick, B., Liu, C., Hadfield-Menell, D., Palaniappan, M., Malik, D., Sastry, S. S., Griffiths, T. L. and Dragan, A. D. (2017), Pragmatic-pedagogi value alignment'.
Fisac, J. F., Liu, C., Hamrick, J. B., Sastry, S., Hedrick, K., Griffiths, T. L. and Dragan, A. D. (2020), Generating plans that predict themselves, in 'Algorithmic Foundations Robotics XII', Springer, Cham, pp. 144-159.
Fu, J., Luo. and Levine, (2017), 'Learning robust rewards with adversarial inverse reinforcement learning'.
G. (2020), 'Artificial Intelligence, Values, and Alignment', Minds and Machines 30(3), 411- 437.
URL: http:////aairrm//////00011100000
Galvin, D. (2014), "Three tutorial lectures on entropy and counting', aarXiv preprint arXiv: 1406.877
Gregor, K., Rezende, D. J. and Wierstra, D. (2016), Variational intrinsic control, in 'arXiv preprint arXiv:1611.075077
Grover, A., Ai-Shedivat, M., Gupta, J. K. Yura, B. and Edwards, H. (2018), Learning policy representations in multiagent systems, in '35th International Conference on Machine Learning, ICML 2018", Vol. 4.
Guckelsberger, C., Salge, C. and Colton, S. (2016), Intrinsically motivated general companion NPCs via Coupled Empowerment Maximisation, in 'IEEE Conference on Computatonal Intelligence and Games, CIG', Vol. O.
Haarnoja, T. and Abbeel, P. (2018), 'Soft actor-critic algorithms and applications', arXiv preprint arXiv:1812.05905.
Hadfield-Menell, D., Russell, J., Abbeel, and Dragan, A. (2016), Cooperative inverse reinforcement learning, in 'Advances in neural information processing systems', pp. 3909-3917.

---

Published as a conference paper at ICLR 2022
Hong, Z. W., Su, S. Y., Shann, T. Y., Chang, Y. H. and Lee, C. Y. (2018), A deep policy inference Q-network for multi-agent systems, in Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS', Vol. 2.
Javdani, S., Srinivasa, S. and Bagnell, A. (2015), 'Shared autonomy via hindsight optimization', Robotics science and systems: online proceedings 2015.
Jeon, W., Barde, P., Nowrouzezahrai, D. and Pineau, J. (2020), 'Scalable multi-agent inverse reinforcement learning via actor-attention-criiic""
Kaelbling, L. P., Littman, M. L. and Cassandra, A. R. (1998), 'Planning and acting in partially observable stochastic domains', Artificial intelligence 101(1-2), 99-134.
Klyubin, S., Polani, D. and Nehaniv, measure of control, in '2005 IEEE Congress on Evolutionary Computation, IEEE CEC 2005. Proceedings', Vol. 1, pp. 128-135.
(2005), Empowerment: universal agent-centric
Klyubin, A. S., Polani, D. and Nehaniv, C. L. (2008), 'Keep your options open: An information- based driving principle for sensorimotor systems', PLoS ONE 3(12).
Lillicrap, T. P., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. and Wierstra, D. (2016), Continuous control with deep reinforcement learning, in '4th International Conference on Learning Representations, ICLR 2016 Conference Track Proceedings'.
Littman, M. L. (1994), Markov games as framework for multi-agent reinforcement learning, in 'Machine Learning Proceedings 1994', Elsevier.
Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Abbeel, O. P. and Mordatch, I. (2017). Multi-agent environments, 'Advances in neural information
actor-critic for mixed cooperative-comeete processing systems', pp. 6379-6390.
Macindoe, O., Kaelbling, L. and Lozano-Pérez, T. (2012), Pomcop: Belief space planning for sidekicks in cooperative games, in 'Eighth Artificial Intelligence and Interactive Digital Entertainment Conference'.
Mao, H., Zhang, Z., Xiao, Z. and Gong, Z. (2019), Modelling the dynamic joint policy of teammates with attention multi-agent ddpg. in 'Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS', Vol. 2.
Mohamed, S. and Rezende, D. J. (2015), Variational information maximisation for intrinsically motivated reinforcement learning, 'Advances in Neural Information Processing Systems', Vol. 2015-January.
Mordatch, I. and Abbeel, P. (2018), Emergence of grounded compositional language in multi-agent populations, '32nd AAAI Conference on Artificial Intelligence, AAAI 2018".
Mutti, M., Pratissoli, L. and Restelli, M. (2020), 'A Policy Gradient Method for Task- Taso-Agnostic Exploration'.
Omohundro, S. (2008), 'The basic ai drives. agi08proceediins of the first conference on artificial general intelligence'.
Papoudakis, G., Christianos, F. and Albrecht, S. V. (2020), 'Opponent modelling with local information variational autoencoders'.
Pellegrinelli, S., Admoni, H., Javdani, S. and Srinivasa, S. (2016), Human-robot shared workspace collaboration via hindsight optimization, in '2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)', IEEE, pp. 831-838.
Pérez-D' Arpino, C. and Shah, J. A. (2015), Fast target prediction of human reaching motion for cooperative human-robot manipulation tasks using time series classification, '2015 IEEE international conference robotics and automation (ICRA), IEEE, pp. 6175-6182.
Russell, S. (2019), Human compatible: Artificial intelligence and the problem of control, Penguin.
12

---

Published as a conference paper at ICLR 2022
Schulman, J., Levine, S., Moritz, P., Jordan, M. and Abbeel, P. (2015), Trust region policy optimization, in '32nd International Conference on Machine Learning, ICML 2015', Vol.
Seo, Y., Chen, L., Shin, J., Lee, H., Abbeel, P. and Lee, K. (2021), 'State Entropy Maximization with Random Encoders for Efficient Exploration' arXiv preprini arXiv:2102.09430.
Song, J., Ren, H., Ermon, S. and Sadigh, D. (2018), aglti- generative adversarial imitation learning, in 'Advances in Neural Information Processing Systems', Vol. 2018-December.
Tan, M. (1993), Multi-Agent Reinforcement Learning: Independent Cooperative Agents, in 'Machine Learning Proceedings 1993', Morgan Kaufmann.
Terry, J. K., Jayakumar, M., Santos, L.,, Black, B., Hari, A., Dieffendahl, C., Williams, N. L., Lokesh, Y., Sullivan, R., Horsh, C. and Ravi, P. (2020), 'Pettingzoo: Gym for multi-agent reinforcement learning'.
Tumer, A. M., Smith, L., Shah, R., Critch, A. and Tadepalli, P. (2019), 'Optimal policies tend to seek power', arXiv preprint urXiv:1912.016:
Van Hasselt, H., Guez, A. and Silver, (2015), 'Deep reinforcement learning with double q- learning', arXiv preprint arXhv:1509.0661
Volpi, N. C. and Polani, D. (2020), 'Goal-Directed Empowerment: Combining Intrinsic Motivation and Task-Oriented Behaviour', IEEE Transactions on Cognitive and Developmental Systems..
Watkins, C. C. H. and Dayan, P. (1992), 'Q-learning, Machine learning 8(3-4), 279-292.
Yu, L., Song, J. and Ermon, S. (2019), Multi-agent adversarial inverse reinforcement learning, in 36th International Conference on Machine Learning, ICML 2019', Vol. 2019-June.
Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D. and Hsieh, C.-J (2020), 'Robust deep reinforcement learning against adversarial perturbations state observations', arXiv preprint arXiv:2003.08938.
Zhao, R., Lu, K., Abbeel, P. and Tiomkin, S. (2020), Efficient empowerment estimation for unsupervised stabilization, in "International Conference Learning Representations'
13

---

Published as a conference paper at ICLR 2022
A ESTIMATION OF LEADER AGENT'S CHOICE FROM OBSERVATION
A.1 MODEL-BASED ESTIMATION OF CHOICE FROM OBSERVATIONS
We introduce model-based estimator of choice that is suitable for small-scale discrete-state environments, having the advantage that it is easily interpretable. Recalling how we compute the discrete choice and entropic choice estimates for the leader agent, an estimate of the n-step state distribution conditioned on the altruistic agent's actions is needed, i.e. P(s+t"|TT, A bitn-1 s To simplify this computation, we assume the altruistic agent's action to equal hold for the next n steps. More specifically, we assume that the altruistic agent's state is unchanged for the next steps. Furthermore assuming that both the state and the action space are discrete, we compute
P(s TL. aA #tn--1 ,,8) = , T's'''"
(6)
with
T(s'))) =P(s =3 s 8c, s 5'A))
where the state transition matrix As holds the transition probabilities between all possible states, function of the state of the altruistic agent SA. To compute T(s the system state is encoded into a one-hot vector $2.
The n-step discrete choice of the leader agent can then be computed as
D*T(s") || (lso)
(8)
its n-step entropic choice as and its immediate choice
ECI((s') = (s4 T)) (C2(s')=((=((()))
(9)
= (s1 T(sh))
(10)
In environments with a discrete state and action space, the altruistic agent can hence use an estimate of the state transition matrix T to estimate the choice of the leader agent using either of the proposed methods, i.c. DC, EC or IC. An estimate of T can be built over time, by observing the environment transitions and computing the transition probabilities relative frequencies of observed transitions.
A.2 MODEL-FREE ESTIMATION OF CHOICE FROM OBSERVATIONS
To limit the computational complexity, which is important for environments with large or continuous state spaces, we also consider immediate choice as an estimator for the leader agent's choice (s')
((*))+ As shown in section 3.1 this estimate can be simplified to H(S++11s²) Hari(alss) (als*)), under the named assumptions. Hence, to compute the immediate choice of the leader, the altruistic agent requires
estimate of the leader agent's 's policy entropy. which can be learned from observation using a policy estimation network (Hong et al., 2018
Papoudakis et al.
2020: Mao et al. 2019 Grover et al. 2018).
B GRIDWORLD EXPERIMENTS
B.
TRAINING PROCEDURE
B.
SETUP
The environment setup is described and displayed in section 4.1
AvE baseline. We evaluate the AvE. baseline for different horizons For each horizon, we tested the AvE baseline as implemented in the provided source codd2 using the haper-parameters suggested by the authors. The original implementation uses a look-ahead horizon = 10. We found
httpp:///i thw.t wwtp:htwwww.oo..t.cttt.cccc hgd/av/nnnn/////ee/eeeeee
14

---

Published as a conference paper at ICLR 2022
Table 3: Results on minimalistic environments (sec. 4.1 Two desired behaviours of the altruistic
agent, for each scenario, are listed as the two bottom rows. In the columns, 1 denotes Discrete Choice, E Entropic Choice, AvE the Assistance via Empowerment baseline (Du et al. 2020). and SV the supervised baseline. Refer to sec. 4.1 for a discussion of the results.
n
n
n 12
Tla = 0.1
No 0.7
7a = 0.1
Ta = 0.7
0.1
Ta 0.7
Category
Scen.
D
E
D
AvE.
D
E
AVE
AvE
Opens door
x
x
x
X
X
Non blocking
/
that results equal for both 10 and 12, which is why we only display results for n 12. We further evaluated the AvE. baseline for n. between and 12. For the Opens door task, we found that AvE yields success for 2, 3, 4, 5 and failing for the remaining. For the Non blocking task, we found that AvE yields success for n = 1, 2 and failing for the remaining.
B.1.2 PRETRAINING
We first pretrain the leader agent using tabular Q-Learning, with learning parameters given in Table During this pretraining, the altruistic agent takes random actions. We train until all Q-Values are fully converged, i.c. training runs for 300000 environment steps.
1.1.3 REWARD COMPUTATION FOR ALTRUISTIC AGENTS
The altruistic agent is then also trained using tabular Q-Learning, and its internal reward signal is given the choice estimate of the leader agent, i.e. either DC2 (s") or (s'), which is computed with the model based-estimation introduced in appendix A.I The altruistic agent records all environment transitions and frequently updates its estimate of the state transition matrix T(*A), which is needed to compute the internal reward signal for the altruistic agent. All training parameters can be found in Table Training time is about 15 minutes per experiment.
B.2 PERFORMANCE EVALUATION
Performance of the altruistic agent is reported for two different categories, as shown in Table For each category, we report success or failure for choice estimate look-ahead horizons E {1,3, 3, 12) and discount factors of the altruistic agent 7a E {0.1, the experiment setup, i.. 10 simulations were run for each setup which always yielded the same outcome. To estimate the leader agent's choice, the altruistic agent uses either discrete choice (D, equations and 8 entropic choice (E, equations 2 and 9 It must be noted that horizon n = 12 is equivalent to an infinite horizon look-ahead for the given environment size and that entropic choice is equivalent immediate choice (equations and 10 at
Success or failure was always
deterministie, conditioned
horizon n 1, as the environment satisfies the necessary conditions listed for equation
Table 3 displays the results of this experiment. In the first row, it is evaluated whether the altruistic agent opens the door at all times, such that the leader agent can eat the green apple. It can be observed that the altruistic agent only opens the door for longer horizons n, respectively higher discount factors %a-
Given the definitions of discrete choice (Equation MD and entropic choice (Equation 2 it can be assumed that the choice horizon determines the locality for which choice is considered and that the discount factor %o defines whether the altruistic agent gives higher importance to the short-term or long-term choice of the leader agent. This is in line with the observed results for the first category (Opens door). can be assumed that, for short horizons n. the altruistic agent does not open the door, as it does not estimate that this would lead to an increase in the leader agent's 's choice. A similar argumentation follows for low discount factors "a.
The bottom-row category evaluates whether the altruistic agent does not block the hallway that leads up to the leader agent's 's target apple in the top right environment cell. This category demonstrates possible failure horizons n and high discount factors Ya. the altruistic agent actively blocks the entry to the low- entropy hallway towards the top right cell by constantly occupying cell (2, 6) to prohibit the
of the proposed approach of maximizing another agent's 's choice. For short
15

---

be prevented Published as a conference paper at ICLR 2022
leader agent from entering this region of low estimated choice. This failure by an appropriate selection of the hyperparameters horizon n and discount factor %a- It is related to the selection of the temperature hyperparameter in maximum entropy single-agent RL (Haarnoja and Abbeel, 2018); if chosen incorrectly, the agent does not foster environment rewards in low- entropy regions. A possible solution to this problem would be to define constrained optimization problem, as shown by Haarnoja and Abbeel (2018).
B.3 ABLATION STUDY ON JOINT LEARNING
Training. To investigate the effects of joint learning of the leader agent's and the altruistic agent's policy, we adapted the training process described in section [4.1 for the Gridworld experiments as following. Instead of first learning the policy of the leader agent while the altruistic agent takes random actions, we initialized both policies from scratch and trained both agents simultaneously with the parameters given Table
Results. We evaluated the outcome for the same scenarios, i.e the scenarios described in section 4.1 We found that the results for the individual test cases were equivalent those achieved when training the leader and the altruistic agent subsequently, i.e. the results are equivalent to those displayed in Table 3
C LEVEL BASED FORAGING EXPERIMENTS
C.I TRAINING PROCEDURE
C.I.1 SETUP
We adopted the Level Based Foraging environment given in Christianos al. 2020 We only focus on two-agent scenarios and only consider the subset of possible environments that require full cooperation among agents, i.c. those where food can only be foraged by two agents cooperatively. We therefore only consider environments where both agents are at level one, and all present food is level two. In the original implementation, both agents have simultaneously select the eat action while docking at different sides of food object to forage the object and receive the reward. To reduce training time, we simplify this setup by reducing the action space to up, down, left, right, stay, i.c. remove the eat action and enable agents to forage food by being simultancously at different sides of a food object, with no further action required.
C.1.2 PRETRAINING
To obtain a pretrained leader agent, we first train two agents in the environment that are equally rewarded for foraging food. This setup corresponds to shared-reward cooperative MARL (Tan
1993). Both agents are trained using Deep Q Learning (DQL, (Van Hasselt et al.) 2015)). using fully connected neural network with two hidden layers and five output values, resembling the Q values of the five possible actions. The exact training parameters are listed in Table We then take either one of the two agents and set the pretrained leader agent for the subsequent evaluation of the altruistic agent.
C.1.3 TRAINING OF ADDITIONAL AGENTS
We then insert an additional agent into the environment that shall act altruistically towards the leader agent. This additional agent trained in the same fashion and with the same parameters as the previously trained leader agents. Only its reward signal is different, as laid out in the next section.
C.1.4 REWARD COMPUTATION FOR ADDITIONAL AGENTS
We compare four different approaches for how the reward of the additional agent is defined, respectively how it behaves. Random: The agent takes random actions. Supervised: The agent receives the same reward as the leader agent, i.e. shared reward as in cooperative MARL. Ours:
https://gi thw.. com www.eeewttaeeeeeeeeeeeeeea foragi ng
16

---

Published as a conference paper at ICLR 2022
The reward of the additional agent is defined the immediate choice of the leader agent, as detailed in equationB] We compute the leader agent's policy entropy by computing the entropy the softmax of the leader agent's 's Q values in the given state. We further consider an unsupervised baseline, as detailed in the next paragraph.
Unsupervised baseline (MaxEnt). As an unsupervised baseline, we implemented the MEPOL approach of Mutti et al. (020)) Their task-agnostic unsupervised exploration approach maximizes the entropy over the state distribution of trajectory rollouts. For this baseline, the additional agent trained with the implementation given by the authors which itself builds on TRPO (Schulman itt al., 2015). We leave all parameters unchanged but evaluate different learning rates; Ir E {le 6, le 5, le - 4, le 3, 2, le 1}. Best results were achieved for learning rate of 5, which was hence picked as the relevant baseline.
C.2 PERFORMANCE RVALUATION
Each experiment was run for different random seeds and mean and standard deviation are reported. Training progress is shown in Figure 4 Evaluations computed every 10000 environment steps for 200 episodes, with the exploration set to zero. Training time was about 14 hours for each run. Results shown in Fig.
D TAG EXPERIMENTS
D.1 TRAINING PROCEDURE
D.1.1 PRETRAINING
2020 5 which is unchanged as compared to the original implementation of Mordatch and Abbeel (2018 only fixing minor errors. We first adopt the original configuration and pretrain three adversaries and one good agent (leader agent) using the parameters listed in Table 4 We use MADDPG (Lowe et al. 2017 to train adversary agents, and modify the framework as follows. The last layer of each agent ss actor-networts outputs one value for each of the environment's five possible actions, over which the softmax is computed. We then sample the agent's 's action from the output softmax vector, which corresponds to the probabilities with which the agent takes specific action in given state. We train the leader agent with DDPG (Lillicrap et al. 2016).7 .7 where equally modify the output layer. Each actor and critic network is implemented as a fully-connected neural network with two hidden layers, with layer sizes as given in Table
We use the Simple Tag (Tag) implementation by Terry et al.
To make the environment more challenging for the leader agent, we decrease its maximum speed and acceleration to 70% of the original value. We next insert three additional agents into the environment whose observations include all agents and objects. These additional agents are not observed by adversary agents the leader agent. The additional agents of the same size the adversary agents, and their acceleration and maximum velocity are equal to that of the leader agent. To speed up training, we made the following changes the environment, which are applied our approach as well as to all baselines. First, we spawn the three additional agents in the vicinity of the leader agent, which itself is spawned a random position. Furthermore, we randomly pick two out of the three adversary agents and decrease their maximum acceleration and maximum speed by 50%. We made these changes to be able to observe substantial differences between the different approaches after training time of less than 24h.
D.1.2 TRAINING OF ADDITIONAL AGENTS
We train these three additionally inserted agents with the previously descrihed modified version of MADDPG. The reward for each agent is defined either according to developed approach, any of the given baselines, as detailed in the next section.
17

---

Published as a conference paper at ICLR 2022
Table 4: Detailed
used in the three experimental environments.
Gridworld
Tag
Level Based Foraging
Resources
Environment Steps
300000
7500000
50000000
300000
Episode Length
25
25
10/15
10
Learning Rate Actor
0.001
Learning Rate Critic/ Q-Learning
0.01
0.001
0.001
0.005
Exploration Noise
0
0.1
0
Epsilon start
0.1
1.0
0.1
Epsilon final
0.1
0.2
0.1
Discount Factor y
0.9
0.95
0,9
0.95
Target Network Update Rate
0.01
0.001
Replay Buffer Size
1000000
200000
Training Batch Size
256
256
Train every n steps
64
32
Layer Size Adversary Agent
64
Layer Size Leader Agent
64
64
Layer Size Altruistic Agent
128
64
Optimizer
Adam
Adam
Gradient Norm Clip
0.5
Activation Function
relu
relu
D.1.3 REWARD COMPUTATION FOR ADDITIONAL AGENTS FOR DIFFERENT BASELINES
We consider the following implementations for the reward computation of the additional agents, respectively different environment configurations.
None: For this scenario, the additional agents are removed from the environment.
The remaining approaches purely differ in the way that the reward of the additional agents is computed. No other changes are made.
Random: The additional agents take random actions.
Cage: The additional agents receive negative reward for violating the environment boundaries, which is equal to the negative reward that the leader agent receives for itself violating the environment boundaries (part of the original Tag implementation).
Supervised: The additional agents receive the same reward as the leader agent. That is, they receive reward of -10 if the leader agent caught by the adversaries and small negative reward the leader agent violates the environment boundaries.
Supervised + Cage: The additional agents receive the same reward the leader agent, and an additional small negative reward if they themselves violate the environment boundaries.
Ours: The reward of the additional agents defined as the immediate choice of the leader agent, as detailed in eq. To reduce the variance in the estimate of the leader agent's immediate choice, implement an ensemble of five pretrained nctor-networks for the leader agent, evaluate the policy entropy of each network, and take the median of the achieved values as the reward for the altruistic agents. Furthermore, the additional agents receive a small negative reward for themselves violating the environment boundaries.
D.2 PERFORMANCE EVALUATION
We train Cage, Supervised, Supervised + Cage and Ours for five different random seeds with parameters as detailed in Table We then compute the results listed in Table 2 by freezing all weights across all networks, setting the exploration noise to zero and computing the average and standard deviation over 500 rollout episodes.
18

---

Published as a conference paper at ICLR 2022
Table 5: Results on the resource environment. We evaluate two baselines (random and supervised), and four different implementations of choice. Performance is measured the average number of consume actions that failed, over all time steps and both agents (so it takes values between 0 and 1).
3
rand.
superv.
EC 1C)
DC
EC
DC
avg. failures | 0.336 0.312 0.081 0.185 0.079 0.183 0.082 0.182 0.079 0.185 0.071 0.181
E RESOURCE ENVIRONMENT
E.0.1 MOTIVATION AND OVERVIEW
This environment is a special case of the general resource-based MDP proposed by BBnsson-Tilsen and Soares (2016), which they used to show that intelligent agents pursue instrumentally useful subgoals. The motivation behind the choice for this environment is to evaluate our proposal in non-spatial and non-navigation environments.
In the environment, there are 3 resource types, which two "consumer" agents may consume as an action. Each consumer has different preferences (reward function), and so will only consume 2 of the resource types. A third, altruistic agent, receives resource unit of each type to distribute among the consumers, and its goal is to satisfy the preferences of the consumers without knowing their reward function. We define its performance as the average number of times that the consumers fail to consume their preferred resource (so lower is better). We compare our method to a supervised agent that is explicitly trained with the consumers' reward function, as well an agent that assigns the resources randomly.
E.0.2 ENVIRONMENT DESCRIPTION
The environment expressed a Markov Game (see section 3 The Markov game is composed
of two human- -umanninspired consumers with subscript C1, C2 and an altruistic agent with subscript A. Three types of resources exist, Rx, Ry and Rz. The environment state s given by the number of resources of each type available to each of the consumers. For example, means that agent C1 has one resource each of type X and available, while agent C2 only has one resource of type Z available. At the beginning of each time step, the altruistic agent is provided with one resource per category, i.e. Rx, Ry and Rz- The altruistic agent can assign each resource individually to any agent discard the resource. The altruistic agent's 's action space is hence defined by one sub-action per resource, i.e. a = (a axx a²). Each sub-action assigns the resource either one of the consumers discards it. The resources are then distributed according to the action taken by the altruistic agent and the environment state is updated. Resources cannot be stacked, which means that each agent can only have one resource per category available time. Next, the consumers attempt to consume one resource each, according to their preference. Agent C1 dislikes resource Rz, hence it chooses Rx or Ry with equal probability. Agent C2 dislikes resource Rx, hence it chooses Ry or Rz with equal probability. The actions of agents C1 and C2 are sampled accordingly and the environment state is updated. For each round, we record how many agents failed to consume resource that was not available.
[(1, 0, 1), (0, 1, 1)0)
E. TRAINING
The altruistic agent trained with Q-Learning Watkins and Dayan 1992) to maximize the discounted future choice of the consumers (see eq.
4. For that, it uses one of the three proposed which it estimates as detailed in appendix A.I The exact hyper-parameters are given in Table 4 We compare the performance of the altruistic agent that maximizes the choice of the consumers to that of a supervised agent. The reward of the supervised agent is the negative of the number of consumers that attempted to consume resource, in that time step, and failed. Further, we compare to a uandom-polic baseline that distributes the resources randomly but does not discard any resources.
objectives, namely IC (eq. 3). EC (eq. ( or DC (eq.
19

---

Published as a conference paper at ICLR 2022
E.2 RESULTS
Table [ shows that the results achieved by the altruistic agent trained with choice are equivalent those achieved by the supervised agent. Furthermore, they are significantly better than those achieved by an agent with random policy.
F VIDEOS OF BEHAVIOUR OF ALTRUISTIC AGENT
We provide videos for the most relevant outcomes of our experiments in the supplementary material.
F.1 VIDEOS FOR RESULTS OF GRIDWORLD EXPERIMENTS (SECTION 4.1)
F.1.1 DOOR SCENARIO IN FIG. T
TOP CENTER
01 Altruistic agent opens door for leader agent: It can be observed that the altruistic agent has learned to operate the door switch enable the leader agent pass through the door and reach its target on the other side.
02 Altruistic agent does not open door for leader agent (failure case): It can be observed that for an unfavourable choice of hyperparameters, the altruistic agent does not open the door.
F.1.2 DEAD END SCENARIO IN FIG. TOP RIGHT
03 Altruistic agent gives way to leader agent: It can observed that the altruistic agent does not get into the way of the leader agent, which is hence able to reach its target in the top right cell.
04 Altruistic agent blocks path of leader agent (failure case): can be observed that for an unfavourable choice of hyperparameters, the altruistic agent blocks the entry to the hallway towards the right side of the environment such that the leader agent cannot reach its target at the top right cell. This happens the altruistic agent forcefully maximizes the estimated choice of the leader agent by hindering it from entering the hallway, which is a region of fewer estimated choice.
F.2 VIDEO FOR RESULTS OF LEVEL BASED FORAGING (SECTION 4.2
05 Altruistic agent enables leader forage apples: can observed how the altruistic agent (blue) learned to coordinate its movements with the leader agent (green), enable the leader agent to forage apples. has learned this behaviour purely through optimizing for the leader agents choice and is itself not rewarded for foraging apples.
F.3 VIDEO FOR RESULTS OF TAG (SECTION 4.3
06 Altruistic agents protect leader from adversaries: It can be observed how the altruistic agents (blue colors) learned coordinate their movements to protect the leader agent (green) from its adversaries. The adversaries (red colors) try to catch the leader, which in turn tries to flee from them. The altruistic agents protect the leader by actively intercepting the paths of the adversaries. They have learned this behaviour purely through optimizing for the leader agents choice.
20


Reviews:
[{"review_id": "qh8rSIxSFa", "reviewer": "Reviewer_bgMP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is technically sound but very hard to follow through. For instance, I found it very hard to find which baselines they had implemented until I referred to appendix in table 3. \n\nI still don't understand what does \"choice\" of the agent mean and how it is different from sub-goals of the agent. I read it multiple times but its very confusing. \n\nFor the model free estimation of choice from observations, it is unclear how the altruistic agent estimates the leader agent’s policy entropy from observations. Can you provide the details such as the policy feature vector? FC layer? and a softmax layer? Additionally, what was the loss function used. \n\nIn the Unsupervised baseline (MaxEnt) implementation, it would be helpful to provide the entropy index since you are trying to maximize this entropy value over the state distribution of trajectory rollouts.\n", "summary_of_the_paper": "The paper posits a method for agents that learns altruistic behaviour from observations and interactions with other agents without knowledge of their objective fucntion by estimating the sub-goals of the agent and actively maximizing their choice.\n\nThe authors define 3 different task agnostic method to estimate the choices made by the agents. \n\nThe experimental evaluation is extensive with testing the agents in 3 different environments while reporting the results in comparison with three baselines - Discrete Choice, Entropic Choice, and Assistance via Empowerment.\n", "summary_of_the_review": "The authors are trying too many different(existing) methods to prove the central concept around how one agent can be altruistic to another by maximizing their reward. The paper is very hard to follow through. Although, the experiments are extensive. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pTumXbBqD3E", "reviewer": "Reviewer_4g8o", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points of the paper:\n\nThe main idea is quite conceptually simple and an interesting approach to develop altruism for MARL.\n\nThe experimental section of the paper is well executed, and the analysis of the results is thorough. Particularly the choice estimate analysis for each environment is helpful for understanding why the method may help. The analysis of the failure case is also quite insightful.\n \nWeak points of the paper:\n\nThe exposition in Section 3.2 about the theoretical support to the objective is rather nonspecific -- I feel that it would be helpful to introduce more specific, technical claims which follow from the arguments in Turner (2019)\n\nWhile the results on both the discrete gate and the continuous environment are strong, they seem less surprising because they are both navigation-type environments where the value of a state should correlate well with the “choice” afforded by being at that state. The analysis in the results of the hide-and-seek environment mention that it may have outperformed the supervised baseline because it provided a denser reward than the sparse supervised “catching” signal. So, these are environments where it’s unsurprising that the choice heuristic would work well. Because the contribution of this paper is quite dependent on the empirical performance of the method, I think that it needs to be evaluated on additional settings where it is less clear that “choice” is directly correlated with the reward, to be convincing as a generally useful metric when the true environment reward is unavailable.\n \nQuestions:\n\nIn Figure 3, for the LBF environment, I can see how the leader agent should have low IC when waiting at the apple at the top and lower IC when it can choose either of the two apples at the bottom. However, to me this seems to contradict the point that IC is a good proxy for the leader agent receiving high rewards, because the altruistic agent needs to help the leader to harvest the apple regardless, and the altruistic agent is closer to the apple at the top?\n\nWhat do the scores in Table 1 indicate? It is not described in the caption or in the text. Why is the score for LBF in percentages but it is not for Tag?\n", "summary_of_the_paper": "This paper introduces a method for developing altruistic agents in a multi-agent RL (MARL) setting. The core idea is that an altruistic agent, in the absence of any further reward or goal information, may try to increase the “choices” for the agent it is cooperating with as a proxy. The paper argues that this is a suitable proxy for the unknown true reward function in many environments, because optimal policies tend to choose actions which lead to greater choice (larger coverage of state visitation) in the future, using analysis of instrumental convergence. The method is evaluated on discrete environments where the altruistic agent has to help open a gate, a level-based foraging environment, and a continuous state space tag environment, and the paper shows that the method can lead to altruistic behavior and improved rewards obtained by the leader agent in these settings.\n", "summary_of_the_review": "The idea of this paper is conceptually simple and I quite like its application to this challenging setting of altruism without true rewards, as well as the analysis that the paper presents. However, mostly because of the choice of environments in the empirical evaluation, I feel that the paper hasn’t provided enough evidence for the claim of the applicability of this altruistic objective, and I think this is a borderline case.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F61TQTiJBm", "reviewer": "Reviewer_K6xu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1) I think the paper aims to tackle an important problem, namely that of inducing more altruistic behavior in artificial agents acting in the same environment with others and assisting others in achieving their goals without knowing what those are. \n\n2) I also like the proposed approach because it seems quite general and doesn't assume access to the leader's reward function, state goal, policy, or trajectories (unlike other methods in this space). \n\n3) The method is new as far as I know, although some of its elements have been used in other contexts, but I think the authors do a good job at explaining the connections to related work. \n\n4) I also found the analysis of the different choice formulations from Figure 1 to be quite insightful. \n\n\n## Weaknesses\n\n### Baselines \n1) In the paper, you write that for the AvE baseline, you used the hyperparameters suggested by the authors, but this doesn't seem fair since typically these methods need to be fine-tuned for each task / domain as it may require very different hyperparameters than the ones used in the original paper. Could you do a hyperparameter search for AvE and present the results with the best HPs found on the tasks used for evaluation?\n\n2) It wasn't very clear to me why you are not using the AvE baseline for the LBF and Tag domains and also not using the Supervised baseline for the Gridworld tasks. Could you please add these for completion or explain in more detail why they are not used for comparison?\n\n3) I think it would be useful to compare the methods with an oracle which would be the optimal policy for assisting the leader agent. This could provide insight into how far the current method is from such a policy and whether there is still potential for improvement on these tasks by future work. \n\n\n### Clarity\n\n1) One of my biggest concerns is that it seems like there might be some significant modes and implicit assumptions made by the method which are not openly discussed in the paper. First of all, it seems like the method assumes that the leader can solve the task during its training stage, while the altruistic agent is taking random actions. This implies that the altruistic agent doesn't need to learn a very complex behavior and is not necessary for the leader's success (even if it might help the leader achieve the goal sooner). A more realistic setting would be one in which the two are trained at the same time or at least, there are multiple training stages for both of them (that alternate). \n\nAren't there cases where the altruistic agent can hurt the leader's performance? For example, there might be dead end states in the environment which can be activate by the altruistic agent's actions but which would be better avoided by the leader. Given that the two agents are not training at the same time, the leader's policy may not be robust to such changes in the environment / new states, so they may end up in them if the altruistic agent aims to increase the number of reachable states. Would it be possible to bias the set of states enabled by the altruistic agent towards the set of states which are desirable for the leader to reach?\n\nCould the authors confirm whether this understanding is correct and explain why they decided to not tackle this setting / train in this way? I think these issues should at least be discussed in more depth in the paper. It would also be great if the authors can train the method on a similar scenario (where the approach isn't necessarily expected to do well) to better understand the limitations of this approach and when it can be expected to be effective.\n\n2) Another thing which is not openly discussed in the paper is the fact that it seems like your approach may actually require more training since you have two separate training stages (i.e. first you train the leader on the task and then you train the altruistic agent to assist the leader). Can you comment on the trade-off between final performance and learning efficiency and make this fact more transparent in the paper? It would be great to include a graph with performance as a function of the number of samples used for training for the entire training process, with a breakdown for the leader and altruistic agent.\n\n3) At the beginning of the paper, it is not very clear what are the metrics you are looking to improve, so I suggest mentioning that in the introduction. Initially, it is not clear whether the altruistic agent is supposed to help with a) percentage of times the leader can solve the goal, b) number of steps needed to solve the goal, c) leader's sample / computational efficiency, or something else.  \n\n\n### Limitations\n\n1) Could you extend this algorithm to non-deterministic environments? Would you just need to replace the unsupervised learning objective with empowerement?\n\n2) Along similar lines, is it possible to extend the algorithm so that the two agents learn at the same time? This would involve dealing with the non-stationarity of the leader's policy. This seems like it would be a more general setting and might be in a better position to handle more challenging tasks, so it would be great to at least discuss it in the conclusion section. ", "summary_of_the_paper": "This paper proposes an unsupervised learning method for training an agent to assist another agent (called the leader) in solve its task (thus displaying a type of altruistic behavior) without access to the other agent's reward function or policy. The authors propose to use the notion of maximizing the leader's choice which is formalized as maximizing the number of different states the leader can reach at any point (within a given number of steps from its current state). The authors propose three variants of this method and evaluate them on three different domains, while also comparing them with other approaches. ", "summary_of_the_review": "I think the paper tackles an important and rather neglected problem, proposes a novel and quite general method for this setting, and demonstrate significant empirical gains on multiple tasks. \n\nMy main concern is that some of the potential failure modes of this approach are not discussed in great detail. Because the method is so general, there are many settings where it may not be the most effective approach (which, from my perspective, is not a problem in regards to publishing the paper as long as this point is clear to the readers). I would really like to see an experiment (or more) that aims to tackle a case where this method isn't expected to shine to 1) either openly illustrate the method's failure modes and where it shouldn't be used or perhaps 2) show that it still works decently well even in such settings. Another big concern is the comparisons with other baselines, including the tuning of AvE, and the lack of comparisons with AvE and the Supervised baseline for some of the tasks.\n\nIn conclusion, I am leaning towards rejecting this paper, but I am willing to increase my score if the authors can successfully answer the questions above and address my concerns (especially the ones mentioned in the above paragraph).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4h6lkZ-dRU", "reviewer": "Reviewer_kHZT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not expert in the specified feild, but in my opnion:\n1. The writing of this paper is generally well-organized and of good quality. \n2. This paper, as far as I know, is the first to try to address this important problem.\n3 This paper can serve as a baseline and also propose three testbeds for future research. ", "summary_of_the_paper": "This paper is the first try to train agent behave altruistically towards others without knowledge of their objective or any external supervision. \nThe main idea is to train altruist agent giving the leader agents more choice and thereby allowing them to better achieve their goals.  They  introduce three multi-agent environments of increasing complexity to evaluate the proposed method. The results show that it can, in some cases, outperform the supervised baselines.\n ", "summary_of_the_review": "A novelty work, worth accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3REnKPz85hV", "reviewer": "Reviewer_U54a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=====Strengths=====\n\n1. It is an interesting and novel way to train goal agnostic helping policy by maximizing the choice of other agents. The choice-based modeling is clearly motivated and defined, well situated in prior work, sufficiently compared against the similar yet different framework, empowerment. \n\n2. The writing is very clear and the core ideas are conveyed successfully.\n\n3. The experiments and discussions are thorough and informative. The results are promising.\n\n4. The implementation was well documented and the details seem to be sufficient for reproducing the results.\n\n=====Weaknesses=====\n\nWhile the overall writing is clear and the experiments are comprehensive, I do however still have confusion/concerns about the comparison between this approach and the prior work, specifically i) empowerment and ii) MADDPG-like approach that learns other agents' policies. It seems to me that choice is similar to empowerment, with the addition of using agent-specific policy instead of arbitrary probing policy. If I understand this correctly, this means that empowerment only depends on the environment and is agent agnostic, while choice is agent dependent -- the definition of choice depends on a specific agent's policy. If this understanding is correct, then I have two questions:\n\nQ1: Can you train a policy using a MADDPG-like method that jointly learns the policy of other agents and the policy of its own? Crucially, this is closer to your approach than the vanilla single agent DRL is, since it also explicitly depends on the policies of others.\n\nQ2: Empowerment is agent agnostic, so in theory, it generalizes better to other situations where agents will have different policies than the ones seen during training. Could you comment on the generalization?\n", "summary_of_the_paper": "This paper provides an alternative to empowerment for goal agnostic assistance RL policy training -- choice-based optimization. The core idea is to model possible subgoals as the choices of others and train policy to maximize the choices of other agents for achieving goal agnostic assistance. Evaluation against standard deep RL and the recent assistance via empowerment (AvE) method shows that the choice-based approach can be effective in multiple tasks.", "summary_of_the_review": "I enjoyed reading this paper and I think the core idea is a valuable contribution to the multi-agent RL research. My main concern is about its comparison with empowerment and other related multi-agent RL work, which is detailed in my main review. I would appreciate some clarification from the authors and would be willing to increase my rating if my concern is addressed in the rebuttal and the revision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposed a method for encouraging an agent showing altruistic behaviour towards another agent (leader) without having access to the leader's reward function. The basic idea is based on the hypothesis that having the ability to reach many future states (i.e., called choice) is useful for the leader agent, no matter what it reward function is. The altruistic agent learns a policy that maximizes the choice of the leader agent. The paper defines three notions of choice, and evaluates them on four environments.

The reviewers believe that this work attempts to solve an important problem, proposes a novel approach, and performs reasonably good experiments. The reviewers are all on the positive side at the end of the discussion phase. Therefore, I recommend acceptance of the paper. I also suggest a spotlight presentation for this work because of the novelty of the problem, which might be of interest to other researchers.

The authors have already done some revisions to their paper (including adding a new environment). I encourage them to consider any remaining comments from reviewers in their final version.
Decision: Accept (Spotlight)