Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents
Authors: Ammar Fayad
Majd Ibrahim
Venue: ICLR 2022 Submitted (2022)
Keywords: Mutli-Agent Reinforcement Learning
Coordination
Intrinsic Motivation
Coordinated Exploration

Abstract: Discovering successful coordinated behaviors is a central challenge in Multi-Agent Reinforcement Learning (MARL) since it requires exploring a joint action space that grows exponentially with the number of agents. In this paper, we propose a mechanism for achieving sufficient exploration and coordination in a team of agents. Specifically, agents are rewarded for contributing to a more diversified team behavior by employing proper intrinsic motivation functions. To learn meaningful coordination protocols, we structure agents’ interactions by introducing a novel framework, where at each timestep, an agent simulates counterfactual rollouts of its policy and, through a sequence of computations, assesses the gap between other agents’ current behaviors and their targets. Actions that minimize the gap are considered highly influential and are rewarded. We evaluate our approach on a set of challenging tasks with sparse rewards and partial observability that require learning complex cooperative strategies under a proper exploration scheme, such as the StarCraft Multi-Agent Challenge. Our methods show significantly improved performances over different baselines across all tasks.

Under review as a conference paper at ICLR 2022
INFLUENCE-BASED REINFORCEMENt LEARNING FOR INTRINSICALLY-MOIVVEES AGENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Discovering successful coordinated behaviors central challenge in Multi-Agent Reinforcement Learning (MARL) since it requires exploring joint action space that grows exponentially with the number of agents. In this paper, we propose a mechanism for achieving sufficient exploration and coordination in a team of agents. Specifically, agents are rewarded for contributing a more diversified team behavior by employing proper intrinsic motivation functions. To learn meaningful coordination protocols, we structure agents' interactions by introducing a novel framework, where at each timestep, agent simulates counterfactual rollouts its policy and, through a sequence computations, assesses the gap between other agents' current behaviors and their targets. Actions that minimize the gap are considered highly influential and are rewarded. We evaluate our approach on a set of challenging tasks with sparse rewards and partial observability that require learning complex cooperative strategies under a proper exploration scheme, such the StarCraft Multi-Agent Challenge. Our methods show significantly improved performances over different baselines across all tasks.
INTRODUCTION
Deep Reinforcement Learning (DRL) has been applied to solve various challenging problems, where an agent typically learns to maximize the expected sum of extrinsic rewards gathered as a result of its actions performed in the environment (Sutton al. 1998). Multi-Agent Reinforcement Learning (MARL) refers to the task of training set of agents to maximize collective and/or individual rewards, while existing the same environment and interacting with each other.
Recent works have shown that agents with coordinated behaviors learn remarkably faster (Roy et al. 2019) since coordination helps the discovery of effective policies in cooperative tasks. Nevertheless. achieving coordination among agents still remains a central challenge in MARL (Jaques et al. 2019). Prominent works often resort popular learning paradigm called Centralized Training with Decentralized Execution (CTDE) (Lowe al.) 2017 Foerster al.] (018), where each agent is evaluated using centralized critic and has access to extra information about the policies other learning agents during training. At the time of execution, policies' actions restricted to local information only (i.c. their own observations). Our work is primarily motivated by the following natural questions:
In multi-agent settings, how can we quantify the effect of agent (or player) on other teammates' behaviors (particularly on their performance)? And to what degree exploiting this quantity can lead to coordinated behaviors and consequently better overall performances?
To that end, we propose a novel approach that aims at promoting coordination for cooperative tasks by augmenting CTDE MARL main return-maximinn stuun-maximization objective with an additional multi-agent objective that acts as a policy regularizer; we refer to the latter objective as the in flfunnnce function. To build intuition, chosen agent, which we call the "influencer". assesses the progress that other agents are making given its current policy and consequently learns behaviors that will result in an improved performance of its teammates. Concretely, an estimation of the dissimilarity between other agents' behaviors and their targets given the current behavior of The influencer is encouraged learn behaviors that expected to minimize that
formulate the influence of influencer

---

Under review as a conference paper at ICLR 2022
dissimilarity. We also propose two approaches to estimate the influence and empirically show that they yield unbiased estimates of the true value.
Agents acting upon the proposed coordination paradigm Iearn to efficiently exploit the observed joint action space using available information. However, and since the joint space grows exponentially with the number of agents, it is highly unlikely that agents will have access sufficient information to learn optimal behaviors to solve the task at hand; this problem arises in many scenarios such as sparse-reward environments, thus proper exploration scheme is often required. However, many existing multi-agent deep reinforcement learning algorithms still use mostly noise-based techniques (Liu et al. 2021; Rashid et al., 2018:: Yang al., 2018). Moreover, independent exploration proved to be infficiiett in cooperative settings (Roy et al.122019). Recently, this challenge was addressed through Intrinsic Motivation (IM) (Jaques al. [019]] et al., 2019; /hhou et al. 2020b). Many approaches employ IM to encourage exploration of state-space (Han et al. 20200] [Burda et al., 2019) or state-action space (Fayad Ibrahim| (02T) by identifying novel configurations and rewarding an agent for visiting them. We provide extension of these ideas into multi-agent settings and further build connection between reward shaping and coordinated behavior learning, where we choose an agent to act an influencer (i.e. regularize its standard objective using the influence function) while other agents learn to maximize the expected sum of both extrinsic and intrinsic rewards.
To up, main contributions threefold: 1) developing influence function to promote learning coordinated behaviors and improve team performance; 2) extending exploration via random network distillation to multi-agent settings by crafting "novelty" function that rewards under- explored behaviors; 3) formulating a novel intrinsic incentive to promote learning diverse team behaviors to help uncover complex behaviors in collaborative way.
We demonstrate the effectiveness of our methods on comprehensive set of challenging tasks which include. but not limited to, the StarCraft Multi-Agent Challenges (SMAC) (Samvelyan al. 2019) and the Multi-Agent Particle Environments (MAPE) (Mordatch & Abbeel, 2018 Lowe et al. 2017). Empirical results show a significant improvement over wide variety of state-of of-e--of-the-arr MARL approaches. We also conduct insightful ablation studies to understand the relative importance of each component of the approach individually.
2 BACKGROUND
2.1
MARKOV GAMES
Markov games (Littman 1994 Agarwal et al. 2020 Daskalakis et al. 202I are a superset of Markov decision process (MDPs) and matrix games, including both multiple agents and multiple states. Formally, we consider a setting with N > 2 agents ([N) == {1,2... N)) who repeatedly select actions in shared MDP. The goal of each agent is maximize their respective action value function. Formally, an MDP is defined as a tuple G = (S, N, (A,, r, hejN), Tr,, P). where:
S is the finite states space, where the initial states are determined by a distribution S + [0, 1]. Since we consider partially observable testing environments, we denote by s, the information observed by agent i about the global state E S.
A; is a finite action space for agent with generic element 0; E A, Using common conventions, we will write A [ and A_, = Aر to denote the joint action spaces of all agents and of all agents other than with generic elements a = (a:)ie(M) and a_; = =(d))E<JE(M) respectively. According to this notation, we have that a = (a,, a_;) Vi.
re : =Sx A + R is the individual reward function of agent i, i.c., r, a;, a is the instantaneous reward of agent i when agent i takes action a; and all other agents take actions a at state 8 E S.
T is the transition probability function, for which T(s'|s, a) is the probability of transitioning from s to s' when a E A is the action profile chosen by the agents.
7 discount factor for future rewards of the MDP, shared by all agents.
In a Markov game, each agent is independently choosing actions and receiving rewards. Convention-
ally, an agent & aims to maximize its own total expected return R₃ = -'t I'r where T is the time horizon.
2

---

Under review as a conference paper at ICLR 2022
2.2 MULTI-AGENT DEEP DETERMINISTIC POLICY GRADIENT
MADDPG (Lowe et al. 2017 is a multi-agent extension of the DDPG algorithm (Lillicrap et al. 2015). It adapts the CTDE paradigm, where each agent i possesses its own deterministic policy 1 for action selection and critic Q for state-action value estimation, respectively parameterized by ((6) and (60)) All parametric models are trained off- aff-policy from previous transitions Çr := (st, a Ft, St+1) uniformly sampled from a replay buffer D. Each centralized critic is trained to estimate the expected return for a particular agent from the Q-learning loss:
(st, (b610) EEED |10000 (s),, ar: (f
[ru, + Qungaa (5++ (1
(I)
Each policy is updated to maximize the expected discounted return of the corresponding agent i: po()) Ep (s,, at)
(2)
Notice that while optimizing an agent's 's policy, all agents' observation-action pairs are taken into consideration. By that, the value functions of all agents are trained in centralized, stationary environment, despite happening in a multi-agent setting. Moreover, this procedure allows for the learning of coordinated strategies, yet needs be augmented with efficient exploration methods that reward novel action configurations which may lead to the discovery of higher-return behaviors.
3 METHODS
3.1 BASIC INFLUENCE
Intuitively, can define coordination in team of agents the behavior of each individual agent being informed by other agents. Furthermore, agents' behaviors can be inter-affected either directly through communication for example or indirectly through task-specific shared goals and/or rewards or the dynamics of the environment. We hypothesize that when agents learn in a cooperative setting, they tend to affect each other's exploitation processes, confirm the hypothesis throughout the paper and build on that to formalize general method to foster influential interactions and learn meaningful coordination protocols. Specifically, consider setting where N agents with policies IT .T. ........ NN. We assume that the transition dynamics are dependent and use tasks that satisfy this assumption to validate our methods. We will write π, π to denote the joint policy of all agents and of all agents other than agent j. respectively. For two fixed agents and j. the influence of the behavior of j's policy on i's performance (expected return) can be quantified by "marginalizing" out all effects induced by other agents the expected outcomes of i. More precisely, the desired quantity Tj then for all (s, az) E S x Ay:
agents' actions (i.e. T(s'ss, a) A (ssss)))
T (8,
E a-
a)) + Ep(sl1s,as)
E
[ a
(8 a4)]
T('''ss, a)
II (auls) lej
One can note that, analogously to the action-value function Qi di T measures tuple values w.r.t i where the learning dynamics conditioned only on the behavior of j. Naturally, in expectation, g represents the sum of rewards that an agent is expected to receive when influenced by πTz- Based on above, we say that an agent is an optimal influencer when agents only influenced by its policy learn optimal behaviors. To that end, if we fix an agent (say 1), then its influence, F(T1). the rest of the team is defined as follows:
F(m)
Σ 04 Ea (s, max (1 u 2515N
n
where (ai)asisn are positive scaling factors that sum up to 1 and Q2 is the action value function of agent i. Intuitively, since F(T1) is the difference between the expected rewards of other agents conditioned on the current behavior ग and the maximum returns that can be achieved by other agents in the given task, minimizing F will encourage mi to take actions that lead other agents'
3

---

Under review as a conference paper at ICLR 2022
conditioned returns (q.") I closer to their maximum unrestricted returns (max, Qi u)). which matches our definition of an optimal influencer. Hence, the objective functions agents i e [N] are: (,,),) En Logest >'r,,] (F(m)))-=>> where 8 are the parameters of n, and A a hyperparameter called the influence importance temperature.
Theorem 1. (Infiuence Gradient)
((TL(
Σ
a.B. Ve, log ti ((rls))))sss
2SIEN
(5)
+2((s, (1(8,a1)Bs,,, Vo, log
where g(T) 9((,,1)) = a."h (8, a1) max. Qi(s, u).
has been shown empirically that the existence of one influencer can greatly improve the overall team performance. Thus, we only consider one influencer throughout our practical sections and upon that, we call our method "Asymmetric Learning for Influencing Team of Agents (ALITA)".
3.1.1 INFLUENCE WITH SINGLE ESTIMATOR
Practically, we quantify the influence F(x1) by initializing a network Q : - RR with parameters dren; the i-th component of Qeen (=:deen) estimates 9," by minimizing the following loss at each timestep t:
L( = E (xxx,xx)) [[]Qaen (x, a; ( - yll
(6)
where E R-N- r; + Q urga (x', (xx)); Q is the target critic of agent and C B and x1 $1.1, = a1,1) h. where B is a buffer storing all agents'
Dj = TT transitions. The reason for choosing such definition of D is that when fixing the current information of π (i.e. at timestep !) while updating Q we practically marginalize out all agents' experience (other than 1 and i). Later in this section, we show that this trick yields a fairly good approximation of o0 After training Qeen, we compute the influence F(TI):
F( (x....xx) [I|Q0ET (x)) =
where z [max, Q2(x', u) ..... max, Qx u)]". Note that the second term the expectation (i.e. the target vector z) is set to be undifferentiable with respect to _ 's parameters and thus does not propagate through its network.
3.1.2 INFLUENCE WITH MULTIPLE INDIVIDUAL ESTIMATORS
As seen earlier, agent estimates the gap between each agent's value and its target value by employing a single network. Another desirable approach is to additively decompose the centralized estimator Qeen, i.c to multiple estimators where each estimator, namely Q individually calculates a fairly good approximation of q, " To reduce computational costs and arrive at better estimates, each estimator's network is initialized with the parameters of the corresponding critic
network at each episode (i.c. Q(1) < Qt). The training is carried out similarly to that of the single estimator setting,
(()(0) = E(xa,)))) [IQ (x, a; (1)) - 1=
(8)
The influence function could be expressed
N-1 (x) (())) zi|
F(T1) E (x.....
Σ
Which approach yields better estimates of the true value of the influence? The influence of an agent π on a team of agents T was defined as a measure of the improvement the performance of T' given the current behavior of æ. However, measuring the influence using function approximators might result in inaccurate estimates. To resolve this concern, we plot the influence estimates of the two prior approaches time while they learn the Cooperative Navigation MAPE task (Mordatchl
4

---

Under review as a conference paper at ICLR 2022
& Abbeel 2018). where the number of agents is N 6. In Figure (I. graph the average influence estimates over 40000 episodes and compare it to the true value. The latter is averaged over 1000 episodes following the current policies of agents and is reported every 5000 episodes. The plots show a relatively small bias of both methods during learning. However, as Figure ① suggests, measuring influence using multiple individual estimators yielded more accurate values after enough training which substantiates its superiority over the shared network approach. Note that, although confirms our prior hypothesis, this experiment does not reflect the importance of employing the influence on the final performance of the agents as we will discuss that in Section
3.2 INTRINSIC MOTIVATION FOR DIVERSIFIED TEAM BEHAVIOR
In this section, introduce a framework for achieving coop- erative exploration by ensuring that agents are consistently tilted towards visiting under-explored state-action configu- rations; we start by providing a simple demonstration which shows that the number of environment steps required for all agents to randomly traverse all possible action configu- rations increases at least exponentially with the number of agents.
Single Eic
Proposition 1. Consider an L-action setting of agents. In expectation, the number of steps T needed to visit all L action configurations least once without coordinated exploration grows at least exponentially with the number of agents. More concretely, E[]] ((nL").
Figure I: Empirical evaluation of the bias in the proposed methods of mea- suring influence. The influence values estimated by single and multiple esti- mators are compared to the true values of F. The results for the estimated values are averaged across runs.
To mitigate this issue, we assign agent _ı a prediction error intrinsic reward to facilitate recognizing and learning novel behaviors:
(mt) T = ext
(10)
Where Φ an autoencoder network regularly trained on data generated by the policy x and A is hyper-parameter that balances the extrinsic and intrinsic reward terms. s/'s expression stems from the observation that when an autoencoder is trained on data from a particular distribution, it will be good reconstructing data from that distribution, while it will perform poorly if the data is from different distribution. Thus, by employing & as an intrinsic bonus, Ti rewards states' observations and actions that do not belong to the data generated by it. In practice, is designed network since we want it to be slightly overfitted to the training data so that it will not accidentally generalize behaviors that we may deem novel (Fayad & Ibrahim 2021 Zhang al. 2019)
be relatively large
Nevertheless, assigning each agent a z is not sufficient as it makes the case equivalent to independent exploration approaches. Thus, we propose coordinated exploration method that takes into account other agents' behaviors, encouraging agents to diversify team behavior while maintaining good performance.
Specifically, we assign agents { [ intrinsic penalty defined as:
(t, ar) exp ( w/bh( (11,,) (1,))) [| ai,e
(11)
This reward term aims at teaching the agents to recognize previous behaviors and synchronously select novel configurations. To build intuition, consider a case where N = 2. Whenever (M, (4) select an action tuple in the neighborhood of a frequently-visiter tuple (a1, (4))a2) in global state s, 1 will be
relatively small and the penalty, rilt, will be large. Conversely, if (T,, (4) encounter a novel tuple, say
(af, ab)) in s, the small penalty of μ (Eq. (1I) together with a large reward for π (Eq. 10 both agents to further explore this encounter.
will drive
In all, Fig. (2) shows how this framework can be augmented with the basic influence introduced earlier to reinforce learning and discovering coordinated behaviors.
5

---

Under review as a conference paper at ICLR 2022
Actor
Update
DN
Influencer
Environaneet
Actor
Influrecce
Update Critic Networks
Astor
Update (9,") T
Actic
Gradient
Figure 2: Architecture of the general proposed method
4 EMPIRICAL EVALUATION & ANALYSIS
The goals of our experiments are to: a) verify the performance of our method comprehensive set of multi-agent challenges (SMAC, MAPE, sparse-reward settings, and continuous control environ- ments); b) perform ablations examine which particular components of the proposed framework are important for good performance.
4.1 COOPERATIVE & MIXED GAMES
4.1.1 STARCRAFT MULTI- CHALLENGE
StarCraft provides a rich set of heterogeneous units each with diverse actions, allowing for ex- tremely complex cooperative behaviors among agents. We thus evaluate our method on several SC micromanagement tasks from the SMAC benchmarcchk (Samvelyan et al. 2019). where group of mixed-typed units controlled by decentralized agents needs cooperate to defeat another group of mixed-typed enemy units controlled by built-in heuristic rules with "difficult" setting; the battles can be both symmetric (same units in both groups) or asymmetric. Each agent observes its own status and, within its field of view, it also observes other units" statistics such as health, location, and unit rype (partial observability): agents can only attack enemies within their shooting range. A shared reward is received on battle victory as well as damaging or killing enemy units. Each battle has step limits set by SMAC and may end early. We consider battle maps grouped into Easy (2s3z), Hard (5m_vs vs_6m, $$_vs_____ vs_5z),, and Super Hard (corridor, 3s5z vs 3s6z) against 8 baseline methods using their open-source implementations based on PyMARL (Samvelyan et al. 2019): IQL (Tan 1993), VDN (Sunehag al. 2018), QMIX (Rashid al.) 2018 LIR (Individual Intrinsic Rewards) (Du et al. 2019), LICA (Implicit Credit Assignment) (hou et 2020b), and MAVEN (Variational Exploration) (Mahajan et 2019)) EDTI [Decision-Theoree Exploration).
EITI Innormation-Theeee Theoretic Exploration) (Wang et al. 2019).
The corridor map, in which Zealots face 24 enemy Zerglings, requires agents to make effective use of the terrain features and block enemy attacks from different directions. A properly coordinated exploration scheme applied this map would help the agents discover a suitable unit positioning quickly and improve performance, while 2s3z requires agents to learn "focus fire" and interception. For the asymmetric 5m vs 6m, basic agent coordination alone such as "focus firing" no longer suffices (Du et al.) 2019) and consistent success requires extended exploration to uncover complex cooperative strategies such as pulling back units with low health during combat. The 3s_vs_5z scenario features three allied Stalkers against five enemy Zealots. Since Zealots counter Stalkers, the only winning strategy for the allied units is to kite the enemy around the map and kill them one after another, causing the failure of independent learning algorithms learn good policies in this task. The 3s5z_vs. 3s6z is an extended scenario of 3s_vs_____________ vs__z where 3 Stalkers and 5 Zealots battle against 3 Stalkers and 6 Zealots- The extra enemy makes this scenario far more challenging. For this task, we randomly chose 1 S and Z to act as influencers because of the heterogeneity of the team. In the rest, agents are symmetric, hence the influencer randomly chosen the beginning of training.
For most of these scenarios, ALITA consistently shows the best performance with significant learning speed which confirms the effectiveness of our proposed methods. Detailed results are reported in Figure (3) as they present the median win rate during the training across 12 random runs.
whw.. wttp:hhww...o.....t.tetcc /

---

Under review as a conference paper at ICLR 2022
ALITA
QMIX
IQL
VDN
LIR
UCA
MAVEN
EDTI
EM
(a) 2x3z Easy
(h) 3s 3s_vs_57 Hard
(c) 5m_55m__vv_ vs_ vs_6m Hard
(d) corridor SuperHard
(e) 3s52 vs 3s6z SuperHard
Figure 3: The median test win % of various methods across the SMAC scenarios.
4.1.2 SPARSE-REWARD SETTINGS
We test on two additional tasks show the effectiveness our method on sparse-reward settings and compare it to famous influence-based coordinated exploration algorithms (Table ([): EDTI, EITI (Wang et al. 2019). and Social Influence (Jaques et al. 2019).
Sparse Push-Box: Box: A 15 x 15 room is populated with 2 agents and 1 box. Agents need to push the box to the wall in 300 environment steps to get a reward of 1000. Morcover, both can observe the coordinates of their teammate and the location of the box. However, the box is so heavy that only when two agents push the same direction at the same time can it be moved a grid. Agents need to coordinate their positions and actions for multiple steps earn reward.
Sparse Secret Room: 25 x 25 grid is divided into three small rooms on the right and one large room the left where 2 agents initially spawned. There is one door between each small room and the large room. switch in the large room controls all three doors. A switch also exists in each small room which only controls the room's door. The agents need to navigate rooms, i.c. the target room, to receive positive reward. The task is considered solved if both agents in the target room. The state vector contains (x, y) locations of all agents and binary variables to indicate if doors are open.
of the three small
Table 1: Results on the Push-Box and Secret Room tasks after 20000 updates across 10 runs.
Push-Box Team Performance
Secret Room
Agents
Performance std
Avg Success Rate
Success Rate std
ALITA
146.66
34.13
0.68
0.04
EDTI
135.84
45.20
0.34
0.02
Social Influence
86.67
65.81
0.25
0.10
EITT
75.09
78.54
0.46
0.06
notable reason for the good performance of ALITA on the two tasks that through the intrinsic rewards, agents tend to explore the majority of the possible encounters at the beginning of learning which is crucial for estimating the value of the influence. Exploiting the latter, agents tend to reinforce learning jointly rewarding configurations.
4.1.3 MULTI-AGENT PARTICLE ENVIRONMENTS
To understand how the proposed method helps agents achieve cooperative behavior in nonstation- ary settings, we conduct experiments the grounded communication environment proposed in
https:: http:///titubbbcoooo httpps://gittuubccom/mo/ httpr///oo/o/oooo//m//aaaaaaatae

---

Under review as a conference paper at ICLR 2022
(Mordatch & Abbeel 2018 Lowe et al. 2017). The chosen tasks are the Cooperative Navigation, Cuoperative Communication, and Physical Deception We trained with 10 random seeds and reported results in Tables
4.2 CONTINUOUS ENVIRONMENTS
To confirm the scalability of our algorithm large continuous settings, measure the performance of our algorithm on a suite of Py Bullet (Tan et al. 2018)) continuous control tasks, interfaced through OpenAl Gym (Brockman al. 2016). Gym environments, however, are mainly single-agent settings, thus to evaluate our approach, we reframe the problem by introducing an additional learning agent that acts as auxiliary agent. Crucially, both agents work collaboratively in order to find region of the solution space where an agent accumulates higher rewards, We use TD3 (Fujimoto et al. 2018 as our learning model and it against state-of-le sate-of-the-are art algorithms algorithm outperforms all baselines across all different environments (e.g. our method attains 131 % return of SAC final performance Humanoid-v3) For detailed results, see Appendix C
gym environments. Our
4.3 ABLATIONS
We further investigate the significance of each component along with symmetric extension of the proposed frame- work. Specifically, we consider the three cases: 1) No F: where the influence function does not contribute to the update rule to any of the policies; 2) No IM: where a ran- domly selected agent maximizes both the expected sum of extrinsic rewards along with the influence function, and other agents' policies are learned using the DDPG; 3) Symmetric: where all agents simultaneously play the role of influencer and influencee: they learn maximize an augmented reward function (extrinsic and intrinsic) along with the influence function.
Figure 4: Ablations for different compo- nents of our framework 2s3z scenario.
Results of the experiments conducted on the 2s3z SMAC scenario show that, in the absence of the intrinsic rewards
(No IM), the agents experience slightly decreased overall performance when compared to the significant decline induced by detaching the influence function (No F). In Figure (4). we observe that the agents following the Symmetric approach learn faster, and achieve a significantly higher median win rate. This approach, however, doubles the computational costs which restricts its applicability in larger settings.
RELATED WORK
We discuss recently developed methods for exploration in RL using intrinsic motivation, coordination in multi-agent RL, and influence-basee coordinated exploration methods subsequently.
Intrinsic motivation (IM) has been increasingly used both in single-agent RL and multi-agent RL. A core idea of IM is to encourage the agent to take new actions or visit new states, thus exploring the environment and obtaining more diverse behaviors. One common approach is approximate state or state-action visitation frequency and add reward bonus to states the agent rarely covers (Tang al. 2017 Bellemare et al. 2016) Martin et al.] (207)). more related IM approach to evaluate state Visitation novelty (Klissarov al. [019 Han et al. 2020; Burda al. (019) or state-action visitation novelty (Fayad & Ibrahim 2021). 1). Inspired by the latter, we provided a natural extension for this approach to the MARL settings by the learning of "novelty" function. Other works make use of
single-agent IM to construct their multi-agent intrinsic reward (Du et al. 2019 Iqbal & Shal (019)). Each agent in (Du et al., 2019) learns distinct intrinsic reward that the agents stimulated differently, even when the environment only feedbacks a team reward. This reward helps distinguish the contributions of the agents when the environment only returns collective reward. In (Iqbal &
Sha 2019) each agent has novelty function that assesses how novel an observation is to it, based
Full description in Appendix B
8

---

Under review as a conference paper at ICLR 2022
on its past experience. Their multi-agent intrinsic reward is defined based on how novel all agents consider an agent's 's observation. A recent work (Liu et al. 2021) assigns agents a common goal while exploring. The goal is selected from multiple projected state spaces via normalized entropy-batt based technique. Then, agents are trained to reach this goal in a coordinated manner.
Many works studied the cooperative settings in MARL; straightforward approach is to use inde- pendent learning agents (fully decentralized learning). This approach, however, is shown to perform inadequately both with Q-learning (Matignon et al. 2012) and with policy gradient (Lowe al. 2017). Therefore, we considered the CTDE paradigm, where each agent's policy takes its individual observation as many real life applications dictate, while the centralized critic permit for sharing of information during training. Policy gradient methods have been commonly used along with the CTDE paradigm MARI,, either by implementing a single centralized critic for all agents (Foerster et al. 2018), or one centralized critic for each agent (Lowe et al.) 2017). Adopting the latter, we enable agents with different reward functions to learn in competitive and mixed scenarios as well.
Some other works encouraged cooperative interactions between agents by sharing useful information (Yang et al. 2020 Hostallero et al. 2020). In Hostallero et al. (2020), each agent broadcasts a signal that represents an assessment of the effect of the joint actions that all agents take on its expected reward. Different from our approach, this signal encourages agents to behave as is expected of them and does not benefit exploration. As for Yang et al. (2020), each agent learns incentive function that rewards other agents based on their actions. Each agent's 's function aims to alter other agents' behavior to maximize its extrinsic rewards. To accomplish this, each agent requires access every other agent's 's policy, incentive function, and return making this approach difficult to scale and execute. Additionally, Roy et al. (2019) proposed two policy regularizers approaches to promote coordination in a team of agent, one of which assumes that an agent must be able to predict the behavior of its teammates in order to coordinate with them, while the other supposes that coordinated agents collectively recognize different situations and synchronously switch to different sub- sub-policies to react to them.
Similarly to our work, (Jaques et al. 2019) proposed a similar idea of rewarding an agent for having a casual influence on other agents' actions. Their method showed interesting results in terms of learning coordinated behavior. However, this casual influence is designed to reward policies for influencing other policies' actions without considering the "quality" of this influence. [Barton et al. (2018) propose causal influence way measure coordination between agents, specifically using Convergence Cross Mapping (CCM) to analyze the degree of dependence between two agents' policies. Our method also draws inspiration from the work of (Wang al. 2019), as they define influence-based intrinsic exploration bonus, called Value of Interaction ( (Vol), by the expected difference between the action-value function of one agent and its counterfactual action-value function without considering the state and action of the other agent. Particularly, the latter measures the expected behavior of an agent in situation where is not influenced by the other agent. Consequently, by maximizing the Vol, agents tend to explore meaningful interaction points as the distance between their action-vall value functions conditioned on other agents and their independent action-valle (only conditioned on self behavior) functions is maximized. The primary difference from ALITA lies in our definition of the influence.
CONCLUSIONS & FUTURE WORK
We introduced a novel multi-agent RI. algorithm for achieving coordination through assessing the influence an agent has on other agents' behaviors. Additionally, we proposed to learn an intrinsic reward for each agent to promote coordinated team exploration. We tested our algorithm on a wide variety of tasks with many challenges, such as partial observability, sparse rewards, and large spaces; these tasks include, but not limited to, SMAC, MAPE, as well as OpenAl gym continuous environments. Our methods achieved noticeable improvement over prominent algorithms all tasks. One promising extension of our algorithm is to use Graph Attention Networks (Veličković et al.) 2017 Zhou et al. 2020a) to learn the importance of the influencer in determining the influencees' policies and to establish a message -aasssngg architecture in networked systems. The investigation of the effectiveness of these methods is left for future works.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
We have provided an illustration of the proposed algorithm in Fig. ① along with implementation details and hyperparameters selection in Appendix [ Furthermore, code is submitted with Sup- plementary Material and each algorithm is evaluated at least 10 times using random seeds on all environments.
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory, pp. 64 66. PMLR, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 1rXiv:1607006450 2016.
Sean I. Barton, Nicholas R Waytowich, and Derrik E. Asher. Coordination-driven learning in multi- agent problem spaces. arXiv preprint / 2018.
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi
Munos. Unifying count-based exploration and intrinsic motivation. In 30th Conference Neural Information Processing Systems, 2016.
Greg Brockman, Vicki Cheung,
Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1rXiv:/606.01544 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network
distillation. In International Conference on Learning Representations, 2019. URL https
//openreview.oo http:/eeww.nee//ooom222oooo....
Gang Chen and Yiming Peng. Off-policy actor-critic in an ensemble: Achieving maximum general
entropy and effective environment exploration in deep reinforcement learning. arXiv preprint 100500100..05511 2019.
Constantinos Daskalakis, Dylan Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. arXiv preprint arXiv:2101.04233 2021.
Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individual
intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32:4403- 44144,4444442211 2019.
Ammar Fayad and Majd Ibrahim. Behavior Behavior-guided actor-critic: Improving exploration via learning policy behavior representation for deep reinforcement learning. arXiv preprint 2rXfv:2104.0442,, 2021.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.0947, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256 JMLR Workshop and Conference Proceedings, 2010.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR. 2018.
Gao-Jie Han, Xiao-Fang Zhang, Hao Wang, and Chen-Guan Mao. Curiosity-driven variational autoencoder for deep q network. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 764-775. Springer, 2020.
10

---

Under review as a conference paper at ICLR 2022
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
David Earl Hostallero, Daewoo Kim, Sangwoo Moon, Kyunghwan Son, Wan Ju Kang, and Yung Yi. Inducing cooperation through reward reshaping based on peer evaluations in deep multi-agent reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 520-528, 2020.
Shariq Iqbal and Fei Sha. Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning. arXiv preprint arXiv: /905. /2/27, 2019.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agen deep reinforcement learning. International Conference on Machine Learning, pp. 3040-3049. PMLR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412. 69800 2014.
Martin Klissarov, Riashat Islam, Khimya Khetarpal, and Doina Precup. Variational state encoding as intrinsic motivation in reinforcement learning. In Task-Agnostic Reinforcement Learning Workshop at Proceedings of the International Conference on Learning Representations, 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wwXiv:150.002977 2015.
Michael L Littman. Markov games framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Iou-Jen Liu, Unnat Jain, Raymond A Ych, and Alexander Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In International Conference Machine Learning, pp. 6826-6836. PMLR, 2021.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-competiive environments. arXiv preprint arXiv:1706.0000 02076.0225,..0 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. arXiv preprint 1eXivv1990.0700
2019.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Mareus Hutter. Count-based exploration in feature space for reinforcement learning. arXiv preprint (006:0070000000) 2017.
Lactitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforcement learners in cooperative markov games: survey regarding coordination problems. Knowledge Engineering Review, 27(1):1-31, 2012.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. International Conference on Machine Learning, pp. 4295-4304. PMLR, 2018
Julien Roy, Paul Barde, Félix G Harvey, Derek Nowrouzezahrai, and Christopher Pal. Promoting coordination through policy regularization in multi-agent deep reinforcement learning. arXiv preprint arXiv: /908 02269,.2222.... 2019.
Mikayel Samvelyan. Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: arrXiv/992..000 1902000043 2019.
11

---

Under review as a conference paper at ICLR 2022
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In 32nd International Conference on Machine Learning 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1777v:1707.06347, 2017.
Peter Sunehag, Guy Lever, A. Gruslys, Wojeiech Czarnecki, V. Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, K. Tuyls, and T. Graepel. Value-decomposition networks for cooperative multi-agent learning. ArXiv, hess/706.052998 2018.
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: 10332, 2018.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine Learning, pp. 330-337. Morgan Kaufmann, 1993.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. IEEE, 2012.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 2017.
Tonghan Wang, Jianhao Wang. Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. arXiv preprint arXiv: 2019.
Yuhuai Wu, Elman Mansimov, Roger B. Grosse, Shun Liao, and Jimmy Ba. Scalable trust- region method for deep reinforcement learning using tronecker-factone approximation. ArXiv, abs/1708. 051 44, 2017.
Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning incentivize other learning agents. arXiv preprint arXiv:2006. 06051, 2020.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In International Conference on Machine Learning, pp. 5571-5580. PMLR, 2018.
Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. International Conference on Machine Learning, pp. -492. PMLR, 2019.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neural networks: review of methods and applications. Al Open, 1:57-81, 2020a.
Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. arXiv preprint arXivv2007.02529, 2020b.
12

---

Under review as a conference paper at ICLR 2022
A
MATHEMATICAL DETAILS
Theorem 1. (Influence Gradient)
= Σ
aglEam Ve, log ((rls))s)) a1) 2g(s, a1 MBam [Va ( 2STSN
(s, a:)]
where g(T) g(5, a1) 1 (8, max, Qi(s, u).
Proof. We start by finding Vo, :
Volu (1) = 7
Vo, log (og=((a(ss)("" .4) + Vola. Vo (b)
(8,a)) (-1=)
- S A1, and ((T) = 7Lg Vo, log (arl(all)))) )a," (s',,4)) where the distribution
of s' is conditional on and a1. Thus, write Vo, 9, (T) as:
((T) + al Ve, 10 (T")
( (s'
(T)) E (T(T)) + t. of /V, o (T")
(new tuples T" are conditioned previous ones)
(repeatedly unroll V
Pr(T + T Tk;
T
km
lim Pr(S, = TT, ) (T")
(normalize the series term)
T
dr()()
Σ T"
dr(s') Ve. log logm (s", (dr is the discounted visitation distribution induced by TT)
E.
Ve, log lo (5)
(.)d
Vo, (4 (T')
(use T' simplify notation)
TTpp((T)
Since g(T) = 91'h (8, - max, Qi u), the gradient of F(T1) can be expressed as:
Vo, F(m) = Σ Er
Vo, log mı (a (()ls) 2g (T)
2SiEN
Σ
A + (T)
E
VA. log (4 (T')
T>ppl
Σ
A + (g( Er Ve, log lo (a 00 [T')l
In all,
F(T))
Σ 2
a.
Vo, log (a)) a.) Vo, log (ails))) (s, a:)]]
Proposition Consider an L-action setting of n agents. expectation, the number of steps T needed to visit all L action configurations at least once without coordinated exploration grows at least exponentially with the number of agents. More concretely, E = !(nL").
Proof. Let M = L" Since agents tend to visit different action configurations with no coordinated behavior, equivalently say that agents uniformly pick configuration of all I possible
13

---

Under review as a conference paper at ICLR 2022
configurations each step. Let Ta be the number of steps to visit the k-th distinct configuration after covering 1 distinct action tuples. Observe that:
M Σ
T
E
T[T
(12)
A=
Now, Pr[Tk- =i]
i] -M
k-1
meaning that Tp follows geometric distribution. Thus,
00
k
i-1
M
+
M
[
(13)
M
M
h
Getting back to Eq.
12
M
M
M
ETT = M
Σ
M
>
M
dr = M M nL"InL
(14)
M
1
1=1
k=1
And the conclusion follows.
B DETAILS OF MAPE TASKS
Cooperative Navigation: In this environment, N agents must collaborate to reach of N landmarks with known positions. Agents are rewarded based on how far any agent is from each landmark, meaning that the agents learn spread with each agent covering one landmark. The agents, which occupy a significant physical space, are aware of their relative positions to each other and are further penalized when colliding with each other.
Table Avg of collisions per episode and avg agent distance from a landmark in the cooperative navigation task, after 25000 episodes. we achieved optimal performance in the N = 3 case, with near-optimal performance in the N = 6 case, as agents focused more on not colliding with each other (lowest collision average)
N 3
N
6
Agent Average dist.
# collisions
Average dist.
# collisions
ALITA
1,559
0.185
3.349
1.294
MADDPG
1.767
0.209
3,345
1.366
DDPG
1.858
0.375
3.350
1.585
Cooperative Communication: Here, a stationary speaker must guide listener in environment consisting of three landmarks of differing colors. At each episode, one landmark of a particular color set as goal for the listener be reached, however, only the speaker can observe which landmark the listener must navigate to. Moreover, The speaker can produce communication output at each time step which is observed by the listener. The latter must navigate the environment reach the correct landmark. Agents are collectively rewarded at the end of an episode based on the listener's s distance from the correct landmark.
This environment consists of N agents and N landmarks, with one landmark as the target of all agents. The agents are rewarded based on the distance of the closest agent to the target landmark, making it sufficient for only one agent to reach it. adversary agent also tries to reach the target landmark, while the agents penalized it gets closer to the target. The adversary, however, does not know which landmark is the target and must deduce from the agents' behavior. For that reason, agents must cooperate trick the adversary by learning to cover all the landmarks. This task shows that our algorithm is applicable not only to cooperative interactions but to mixed environments as well.
Physical Deception:
In the cooperative communication and physical deception tasks, ALITA obtained the highest success rate across all baselines.
14

---

Under review as a conference paper at ICLR 2022
Table Percentage of episodes where the agent reached the target landmark and average distance from the target in the cooperative communication environment, after 25000 episodes.
Agent
Target reach %
Average distance
ALITA
90.3%
0.093
MADDPG
84.0%
0.133
DDPG
32.0%
0.456
DQN
24.8%
0.754
Actor-Critic
17.2%
2.071
TRPO
20.6%
1.573
REINFORCE
13.6%
3.333
Table 4: Results on the physical deception task, with N = 2 cooperative agents/landmarks. Success
(succ %) for agents (AG) and adversaries (ADV) is if they are within a small distance from the target landmark.
Agent
Adversary
AG suce %
ADV suce %
A suce %
ALITA
MADDPG
95.2%
45.1%
50.1%
MADDPG
MADDPG
94.4%
39.2%
55.2%
MADDPG
DDPG
92.2%
16.4%
75.8%
DDPG
MADDPG
68.9%
59.0%
9.9%
DDPG
DDPG
74.7%
38.6%
36.1%
C ADDITIONAL EXPERIMENTS ON CONTINUOUS ENVIRONMENTS
Since the formulation of F needs shared buffer, SAC and TD3 stand the best off-policy candidates to be incorporated with our framework, as they have shown great performances on many benchmarks. SAC, however, uses stochastic policies in general which makes infeasible combine with the formulation of pint Therefore, we use TD3 as our learning model to measure its performance on suite of Bullet (Tan al. 2018) continuous control tasks, interfaced through OpenAI Gym (Brockman et al. 2016). While many previous works utilized the Mujoco (Todorov et al. 2012) physics engine to simulate the system dynamics these tasks, we found better to evaluate our method on benchmark problems powered by PyBullet simulator since it is widely reported that PyBullet problems are harder to solve (Tan et al. 2018) when compared to Mujoco. Also, Pybullet is license-free free, unlike Mujoco that is only available to its license holders.
(a) Humanoid-v3
(b)
(c) Walker2D
(d) HalfCheetah
Reacher
Figure 5: Learning curves for the OpenAl gym continuous control tasks. The shaded region represents
quarter a standard deviation the average evaluation. Curves are smoothed for visual clarity.
We compare our method to the original twin delayed deep deterministic policy gradients (TD3) (Fujimoto et al.) 2018);; soft actor critic (SAC) (Haarnoja et al. 2018); proximal policy optimization (PPO) (Schulman et al. 2017), a stable and efficient on-policy policy gradient algorithm; deep deterministic policy gradient (DDPG); trust region policy optimization (TRPO) (Schulman al.
15

---

Under review as a conference paper at ICLR 2022
2015); Tsallis actor-critic (TAC) (Chen & Peng 2019). recent off-policy algorithm for learning maximum entropy policies, where we use the implementation of the author S and Actor- Critic using Kronecker-Factored Trust Region (ACKTR) (Wu et al. 2017). implemented by OpenAl's baselines repository | Each task is run for at least million time steps and the average return of 15 episodes is reported every 5000 time steps. To enable reproducibility, each experiment is conducted on 10 random seeds of Gym simulator and network initialization. Results of the best performing agent of the two across different methods are reported in Figure (5)
D TRAINING DETAILS
D.I GENERAL CONFIGURATIONS
We use a buffer-size of 10 entries and batch-size of 1024. We collect 100 transitions by interacting with the environment for each learning update. For tasks our hyper-parameter searches, we train the agents for 15, 000 episodes of 100 steps and then re-train the best configuration for each pair for twice as long (30, 000 episodes) to ensure full convergence for the final evaluation. We use discount factor y of 0.95, influence importance temperature À of 0.1, and a gradient clipping threshold of 0.5 in all experiments unless otherwise specified. Each cloned critic is updated time per step.
D.2 SPARSE PUSH BOX AND SPARSE SECRET ROOM, MAPE, & GYM
We use the Adam optimizer (Kingma & Ba 2014) to perform parameter updates. All models (actors, critics and proxy critics) are parametrized by feedforward networks containing two hidden layers of 128 units excpet for the autoencoder network where we use 7 hidden layers with dimensions (128, 64, 12, 3, 12, 64, 128), respectively. All models' parameters are initialized using Glorot Initialization method (Glorot & Bengio 2010); while the autoencoder's parameters are initialized using Kaiming method (He et al. 2015). We employ the Rectified Linear Unit (ReLU) as activation function and layer normalization (Ba et al. 2016) the pre-activane aetivviiiaioos unit to stabilize the learning.
Table 5: Best found hyper-parameter for the Sparse-reward tasks, MAPE, & Gym environments
PARR-EERR
PUSH Box
SECRET ROOM
МАРE
HUMANOID-V3
GYM (EXCEPT FOR HUMANOID-V3)
0.10
0.10
0.01
0.10
0.10
0.10
0.10
0.01
0.10
0.10
0.15
0.10
0.15
D.3 SMAC
The architecture of all agent networks is a DRQN (Hausknecht & Stone 2013) with recurrent layer comprised of a GRU with a 64-dimensional hidden state, with a conneccd layer before and after. All neural networks are trained using RMSprop (a 0.99 with no weight decay or momentum) with learning rate 5 10-4
Table Best found syper-parameters for the SMAC environments
Hyper-parameter
Corridor
5m_ vs 6m
3s v_v___o 57.
2s3z
0.09
0.03
0.01
0.01
(w:)
0.03
0.03
0.01
0.01
B
0.15
0.15
0.10
0.10
httass//athhbb.ccc/aaam/////
httass//iithbbb.ccm/mmmmm///a/a
16


Reviews:
[{"review_id": "paAJ8mo7m6b", "reviewer": "Reviewer_sFW9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper needs significant improvements in terms of better motivation and conceptual justification of the proposed methods, and with better choice of baselines in experiments. Technical writing also needs improvements to clarity and precision.\n\nThe proposed methods may be based on sensible intuition that the authors had in mind, but this is not successfully conveyed in the author's explication and justification of their method. One symptom of insufficient justification is that a reader still wonders why certain choices were made and why alternatives were not chosen. There are many examples of such in this paper:\n1. It is not clear which agent, or how many agents in a homogeneous team, should be chosen to be the influencer labelled $\\pi$ that is distinguished from all the other agents $\\mu_1,\\dotsc, \\mu_{n-1}$.\n2. In section 3.1.1, the function $Q^{\\text{cen}}$ is trained to predict the TD target that other agents compute using off-policy samples from a replay buffer. The function is fitted to data from some specific subset $\\mathcal{D}$, which was experienced during certain periods of training. Why, then, would it make sense to use this function as part of the regularization term later in equation (4), where the data in buffer $\\mathcal{B}$ may be completely out of the training distribution for $Q^{\\text{cen}}$?\n3. At convergence of a centralized critic, the TD target is equal in expectation to the Q-value. So why should equation (3) aim to predict the TD target rather than just the Q-value itself?\n4. Continuing from the previous point, why do we even need to use that special buffer $\\mathcal{D}$ that is restricted to experiences where one agent's input is constrained to a certain observation-action pair, to train a whole separate $Q^{\\text{cen}}$? If we have a well-trained centralized critic, then simply feeding it data that satisfies that constraint is enough to get the TD target (which asymptotically equals the critic's output).\n5. It is unclear how the \"counterfactual rollout\" is involved in computing eq (4) (also unclear whether that's even intended). Displaying pseudocode in the main paper will help.\n6. The given explanation for the use of $F_{\\pi}$ in equation (4) as a regularizer is unintelligible. $Q^{\\text{cen}}$ was trained to predict TD targets of other agents on a specific subset of experience. $F_{\\pi}$ measures the difference between the predictions and the TD targets on some other set of experience. Minimizing $F_{\\pi}$ (presumbly with respect to the parameters of $\\pi$, but this is not stated explicitly) means to find $\\pi$ whose behavior generates actions such that the pretrained $Q^{\\text{cen}}$ has small prediction error compared to the actual TD targets of other agents. This affects $\\pi$. This does not affect the other agents' policies $\\mu_k$ in any direct way. It is unclear how changing $\\pi$ to improve the prediction of $Q^{\\text{cen}}$ is equivalent to changing $\\pi$ to get other agents to \"reach their goals faster and more efficiently\".\n7. Page 4, authors write \"The influence of an agent $\\pi$ on a team of agents T was defined as a measure of the improvement in the performance of T given the current behavior of $\\pi$.\" One can have a large influence on a team and produce either signficant deprovement or improvement in performance. How, then, is \"improvement in performance\" a good measure of \"influence\"? Also, authors use the word \"was\", implying this definition was given previously. I can't find it.\n8. \"True value of influence\" is mentioned on page 4 many times. This paper does not even provide a formal definition of \"influence\".\n9. Again, the authors write \"distance between the true q-value and the true q-target of each agent\". This is zero in expectation.\n10. The authors use the inability of a VAE to reconstruct data that is out of the training distribution, as a measure of the difference between seen states and new states. The authors should look at previous work on exploration bonuses that measure the novelty of states in far simpler and effective ways. Put simply, why should one use a VAE when one can use any distance metric?\n11. equation (8) means that an agent with policy $\\mu$ will still get a penalty (i.e., negative reward) for exploring new states. Why should $\\pi$ be the one to receive positive intrinsic rewards rather than $\\mu$?\n\nThis paper was written to address multi-agent exploration and influence. There are well-known multi-agent methods for exploration and influence [1,2,3], some even cited by this paper, that are not compared in the experiments. Authors say they compare to LIIR and LICA in the text, but don't show the results in the figures.\n\nWhile the ablation experiments do show the impact of each component on overall performance, there are no results that allow one to conclude that a particular technical design choice has the specific effect for which it was designed. Examples: the regularizer in equation (4) is meant to generate more \"influential behavior\", but this concept is not defined and quantitatively measured.\n\nGiven all of these unclear design choices and lack of comparison to meaningful baselines, it doesn't even matter that the paper shows significant improvements versus standard MARL methods. No reader would be able to build on this work in its present form because of the disconnect between the technical method and the intuitive underpinnings.\n\nMore comments:\n1. page 1, introduction, \"dissimilarity between other agents' behaviors and their targets\": It is unclear at this point of the paper what the authors mean by \"targets\". The most likely guess is some kind of \"intended behavior\". But actually it turns out on page 3 that the authors actually mean the temporal-difference target when they say \"target\". This was very confusing on the first read.\n2. section 2.1, section header was used as a part of a sentence.\n3. inconsistent notation. Agent is labeled first with index $k$, then later labeled by its policy $\\pi$ or $\\mu$.\n4. In section 3.1, the authors write \"Intuitively, one can define coordination in a team of agents as the behavior of each individual agent being informed by other agents\". I can't tell whether the authors intended to give a definition of \"coordination\", or merely provide intuition .\n5. Wrong figure labels on page 4, authors probably meant figure 1 rather than figure 2 when talking about results.\n\n\n\nReferences\n1. Mahajan, Anuj, et al. \"MAVEN: multi-agent variational exploration.\" Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019.\n2. Iqbal, Shariq, and Fei Sha. \"Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning.\" arXiv preprint arXiv:1905.12127 (2019).\n3. Jaques, Natasha, et al. \"Social influence as intrinsic motivation for multi-agent deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "This paper is situated in the context of CTDE for cooperative MARL. The paper proposes new forms of intrinsic rewards to improve multi-agent exploration and a policy regularization term for an agent to have more influence over others. Experiments were conducted on existing benchmarks, such as StarCraft micromanagement and scenarios in the multi-agent particle environment, and two sparse-reward gridworld settings designed by the authors. The proposed method was compared with selected MARL baselines and ablations.", "summary_of_the_review": "This paper needs significant improvements in terms of better motivation and conceptual justification of the proposed methods, and with better choice of baselines in experiments. Technical writing also needs improvements to clarity and precision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RWaAxAN1UHV", "reviewer": "Reviewer_Pto8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strengths:\n\t- The paper aims to develop a framework for achieving better exploration and coordination among agents in multi-agent settings, which is definitely an important and interesting research direction.\n\t- The proposed method is tested in a large number of multi-agent tasks. \n- Weaknesses:\n\t- Many technical details of the proposed method are not very clear to me, which makes it very hard for me to judge if it is technically sound and how significant the contribution is (see my detailed comments below).\n\t- A lot of experimental results are not analysed at all in the paper. It is unclear to me how significant the experimental contributions are. \n- Major concerns/questions:\n\t- Many notations used in the method section are not explained or quite confusing to me, which makes it very hard for me to understand the proposed method. For instance, what is $\\mathbf{x}$ and $\\mathbf{a}$ in Eq. (3)? The joint observation and action? What is the difference between $\\mathbf{a}$ in Eq. (3) and $\\boldsymbol{\\mu}(\\mathbf{x})$ in Eq. (4)? What is the difference between Eq. (3) and Eq. (4), if there is any? What is the difference between $\\mathcal{D}$ and $\\mathcal{B}$? One example of confusing notations is that $\\pi$ was used to represent both an agent and a policy, so does $\\mu_{i}$. \n\t- Many technical details of the proposed method are not explained/not clear to me. Here are just a few examples:\n\t\t- How is DDPG combined with the proposed method? The method section never mentions DDPG, but in the experimental section, there is suddenly DDPG (our method). A naive extension of DDPG to multi-agent settings would be independent DDPG, or do you use something similar to MADDPG? What is the loss function for DDPG (our method)? How do you train the actors and critics?\n\t\t- The authors mention that \"$\\phi$ is designed to be a relatively large network since we want it to be slightly overfitted to the training data so that it will not accidentally generalize to behaviors that we may deem novel.\" This is very vague. How do you ensure that the network is **slightly** overfitted? How do you ensure that it will not generalize to **novel** behaviours? How do you define novel here?\n\t\t- How do you combine the proposed influence function with the intrinsic reward in your method?  Figure 2 should be explained to make this clearer. \n\t\t- How should we select the influencer agent seems to be a critical issue for the proposed method, but this was never discussed in the paper (or maybe I miss it?). In fact, it's not even clear to me how do you select the influencer agent in the experiments. \n\t- For the SMAC results, the authors mention that they compare against baselines including LIIR and LICA, but I couldn't find them in the result (Figure 3). Also, the performance improvement on map corridor does not look as significant as it seems. If you look carefully into the y-axis, the proposed method achieves about 28% win rate, while the second best method achieves about 22% win rate. \n   - For the results in Table 1-4, there seem to be no analysis/discussion at all. I think some discussion should at least be provided to explain why the proposed method can perform better than the baselines in some tasks. \n   - In sparse Push-Box, for DDPG(our method), why is the average success rate 0.68 while the team performance is 146.66? I assume the team performance is the average total return. The reward function in this task seems to be that the agents get a reward of 1000 once succeed, otherwise no reward. Then, if the average success rate is 0.68, the average total return should be around 680. Can the authors clarify this?\n\t- Minor: the font size on the x and y axis in Figures 1, 3, and 4 is too small. ", "summary_of_the_paper": "This paper introduces the idea of estimating the influence an agent has on other agents' actions, in order to achieve better coordination among agents. An agent is chosen as the influencer, which estimates the gap between other agents' actions and their targets given its current action. The influencer is encouraged to minimize this gap to lead other agents closer to their target returns. The paper also proposes to learn an intrinsic reward for each agent to encourage agents to learn more diverse team behaviour. The proposed method was tested in a wide range of multi-agent tasks. ", "summary_of_the_review": "I vote for rejecting this paper as many technical details of the proposed method are not very clear to me and many experimental results in the paper are not analysed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9wga2yCHMig", "reviewer": "Reviewer_HYex", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This problem domain is of considerable interest to a wide (and growing) audience. Hence, any substantial contribution has the potential for considerable impact. The paper demonstrates clear gains over baselines on a fairly wide array of popular benchmarks and considers a sizable set of useful baselines on at least a subset of these environments. \n\nMy primary critique of this paper is that the application of baselines is inconsistent across benchmarks in a way that makes it hard to say that a thorough head-to-head comparison has been done. In particular:\n\n-Both the SMAC and sparse-reward benchmarks have useful benchmarks and are individually fairly thorough, but the set of baselines is disjoint between the two. Given the popularity of several of the approaches as baselines in the sparse reward setting, it would make sense to include them in the SMAC environments. My understanding is that their utility isn't limited to sparse reward settings -- if there is a good reason to exclude these (e.g. they have already been shown to be dominated by other baselines used in previous work, which I am not aware of), it would be important to say this.\n\n-MAVEN (https://arxiv.org/abs/1910.07483) might be good to include.\n\n-The Multi-Agent Particle Environment and Continuous Control experiments appear not to have any of these baselines and instead only have more standard CTDE and completely decentralized (assuming that to be the case, not certain) baselines.\n\n\nMy secondary critique is on the clarity of the work. Specifically:\n\n-It's a bit hard to get the gestalt of the proposed method. My understanding is that one agent has an influencer regularizer to its optimization criterion, the influencer has reward (7), and the influencees have (8) added to their reward. It would be helpful if this were made more clear in  Figure 2, which appears not to differentiate how different agents receive intrinsic motivation, and has the term $\\tilde J$, which I take to be the influence-regularized objective of the influencer, but I cannot see where this is explicitly defined. More minor, but I find (what at least I think is) the use of symbols for both policy networks and agent indexing confusing.\n\n-An easy fix, but the method needs a name! Calling it DDPG(our method) in the results section is really confusing, especially in experiments that use vanilla DDPG. The figures have very small type.\n\n-Might be just me, but I wish I had a bit better intuition for why precisely these definitions work well. It is not entirely clear to me as to why attempting to help other agents reach targets, precisely, is always useful...it would be helpful to build our intuition a little.\n\n\nMore minor, but in the contributions, it is stated that the intrinsic motivation amounts to an extension of RND. I get that it is similar to RND, but the choice made here, with an autoencoder, is distinct. Is it a better choice? It's possible to define a more RND-like CTDE extension, so I wouldn't call it precisely that.\n", "summary_of_the_paper": "This paper introduces two techniques for Centralized Training with Decentralized Execution (CTDE) MARL. Specifically:\n(1) It proposes an \"influence\" training objective by which one agent is encouraged to help the other agents reach target returns.\n(2) It introduces intrinsic motivation and intrinsic cost terms aimed at encouraging efficient joint exploration of the Markov game.\nIt then goes on to give several empirical comparisons with baselines, showing the utility of the approach in Starcraft Multi-Agent Challenge (SMAC), two sparse reward settings, Multi-Agent Particle Environments, and OpenAI Gym continuous control environments. It also performs an ablation study in (SMAC), demonstrating the usefulness of each component of the proposed method.", "summary_of_the_review": "In sum, I think the method has the potential for high impact, but it would be considerably strengthened with more thorough head-to-head comparison with popular methods -- that different baselines are limited to different benchmark suites considerably limits the results. I also have clarity concerns, but these are more minor and can easily be rectified with an edit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1g_KJ5iR1SL", "reviewer": "Reviewer_dZZd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\nThe presented algorithm divides a group of agents into one “influencer” and other “influencees”, which is new and interesting. The implementation of influence-based regularization and curiosity-driven incentives is easy to implement (e.g., the influence function can use the one-step TD target directly), which is scalable for large-scale multi-agent systems. The empirical evaluation is extensive, in which the proposed algorithm achieves outperformance on a large set of challenging tasks, e.g., StarCraft II benchmark, MPE environments, and several gridworld tasks. This topic is of interest to the community because efficient exploration in MARL is a long-standing problem.\n\nWEAKNESSES\n\nThe main weaknesses and concerns are listed as follows:\n\n1) The core component of this paper, the formulation of influence functions (Eq. (4) and (6)), is not motivated. From the perspective of optimization in centralized training with decentralized execution (CTDE) paradigm, this regularization is equivalent to the one-step TD loss of the whole multi-agent system. Minimizing this influence-based regularization can be regarded as minimizing the empirical Bellman error during the centralized training process. Thus, it is not an additional component of the original cooperative MARL algorithms. The clearer motivation of this influence function is critical for this paper.\n\n\n2) The connection to EITI/EDTI (Wang et al., 2019) needs to be justified more clearly. For example, EITI/EDTI introduces an influence-based multi-agent exploration, in which they formulate the influence of other agents by the concept of Value of Interaction (Vol), and they also utilize decentralized exploration method, i.e., count-based exploration. As EITI/EDTI is closely related to this paper, this paper needs to discuss it in more detail.\n\n3) In general, the influencers of a multi-agent system may be dynamic during policy training or in different timesteps of an episode. This paper chooses the influencers without considering these factors. The author may justify in principle that this choice of influencers during policy learning is reasonable.\n\n4) This paper does not have a thorough comparison to some SOTA MARL baselines, e.g., MAVEN (Mahajan et al., 2019), QPLEX (Wang et al., 2021), MAAC (Iqbal and Sha, 2019), DOP (Wang et al., 2021), and FOP (Zhang et al., 2021), in which MAVEN is a popular exploration method, QPLEX is a strong value-based baseline, MAAC, DOP and FOP are advanced actor-critic algorithms. The empirical part of this paper would become stronger by including these baselines.\n\nMinor Question: \n\nThis paper claims that it extends random network distillation (RND) to the multi-agent setting for crafting a \"novelty\" metric, as shown in Eq. (7). In the RND, the curiosity-driven incentive on the timestep t is dependent on the prediction error of the next state and next action. However, in Eq. (7), the intrinsic reward on the timestep t is dependent on the current state and action. Why is the way to formulate curiosity-driven intrinsic rewards different from RND?\n\n[1] Mahajan, Anuj, et al. \"MAVEN: multi-agent variational exploration.\" Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019.\n\n[2] Wang, J., Ren, Z., Liu, T., Yang, Y., and Zhang, C. QPLEX: Duplex dueling multi-agent q-learning. International Conference on Learning Representations, 2021.\n\n[3] Iqbal, Shariq, and Fei Sha. \"Actor-attention-critic for multi-agent reinforcement learning.\" In International Conference on Machine Learning, pp. 2961-2970. PMLR, 2019.\n\n[4] Wang, Yihan, et al. \"Off-policy multi-agent decomposed policy gradients.\" International Conference on Learning Representations, 2021.\n\n[5] Zhang, Tianhao, et al. \"FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning.\" International Conference on Machine Learning. PMLR, 2021.\n", "summary_of_the_paper": "This paper introduces a new exploration method for cooperative multi-agent reinforcement learning (MARL), which utilizes influence-based regularization and curiosity-driven incentives to encourage coordinated and diverse exploration. This paper formulates the dissimilarity between other agents' behaviors and their one-step TD targets as the influence metric and extends the random network distillation (RND) to the multi-agent setting for crafting a \"novelty\" metric. Empirical results show that this method achieves improved performances on a comprehensive set of challenging tasks.", "summary_of_the_review": "This paper proposes a new method for MARL exploration and demonstrates extensive empirical performance on a large set of tasks. However, the reviewer has major concerns on the core component of this paper, i.e., the motivation of influence functions. In addition, the reviewer thinks that the proposed method has some implicit assumptions and the discussion and comparison of related work need to be improved. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: At this time, this work is not yet ready for publication. The core idea--influence functions--was poorly explained in the initial submission, and although major changes to the paper were made to rectify this, at least some reviewers of the remain unconvinced and it is unclear that the paper has been fully evaluated with this confusion resolved. There are a sufficient number of other concerns around the paper, that having rectified these more fully and outside the tight time constraints of the rebuttal period, I hope for an interesting resubmission in future.
Decision: Reject