Distributed Influence-Augmented Local Simulators for Parallel MARL in Large Networked Systems
Authors: Miguel Suau
Jinke He
Mustafa Mert Celikok
Matthijs T. J. Spaan
Frans A Oliehoek
Venue: NeurIPS 2022 Accept (2022)
Keywords: Simulation
Multi-Agent Reinforcement Learning
Influence
State Abstraction
Factorization.

Abstract: Due to its high sample complexity, simulation is, as of today, critical for the successful application of reinforcement learning. Many real-world problems, however, exhibit overly complex dynamics, making their full-scale simulation computationally slow. In this paper, we show how to factorize large networked systems of many agents into multiple local regions such that we can build separate simulators that run independently and in parallel. To monitor the influence that the different local regions exert on one another, each of these simulators is equipped with a learned model that is periodically trained on real trajectories. Our empirical results reveal that distributing the simulation among different processes not only makes it possible to train large multi-agent systems in just a few hours but also helps mitigate the negative effects of simultaneous learning.

Distributed Influence-Augmented Local Simulators for Parallel MARL in Large Networked Systems
Miguel Suau Delft University of Technology n.suaudecastro@uude ttudelft.n
Jinke He
Mustafa Mert Çelikok
Delft University of Technology j.he-4@udelft.n
Aalto University
mustafa.celikok @ aalto.fi
Matthijs T. J. Spaan
Frans A. Oliehoek
Delft University of Technology
Delft University of Technology
m.a.j.spaan& @tunneftu.nl
fa.olichock @ @tudelft.nl
Abstract
Due to its high sample complexity, simulation is, as of today, critical for the successful application of reinforcement learning. Many real-world problems, however, exhibit overly complex dynamics, making their full-scale simulation computationally slow. In this paper, show how to factorize large networked systems of many agents into multiple local regions such that we can build separate simulators that run independently and in parallel. To monitor the influence that the different local regions exert on one another, each of these simulators is equipped with a learned model that is periodically trained on real trajectories. Our empirical results reveal that distributing the simulation among different processes not only makes it possible to train large multi-agen, systems in just few hours but also helps mitigate the negative effects of simultaneous learning.
Introduction
Imagine we have to train team of agents to control the traffic lights of a very large city, so large that we simply cannot control all traffic lights using a single policy. The first step would be to split the problem into multiple sub-regions. A natural division would be to assign one traffic light to each agent. Then, since the agents act locally, we would limit their observations to contain only local information. This partial observability could affect their optimal policies but would also make each individual
problem more manageable (McCallum, 1995; Dearden and Boutilier, 1997). Moreover, we may also want reward agents only for what occurs in their local neighborhood such that we reduce the variance of the returns (Spooner et al., 2021) and facilitate credit assignment (Castellini et al., 2020). Finally, we could train all agents together on big traffic simulator that reproduces the global dynamics. However, if the city is truly large, could take weeks or even months to optimize their policies. That is assuming training actually converges.
One may argue that, since the agents' observations and rewards are local. we could as well train them on separate simulators that model only the local transition dynamics (i.e. cars moving within each of the sub-regions; van der Pol and Oliehoek 2016). This approach might work if the agents' local transitions are isolated from the rest of the system (Becker et al., 2003), but would probably break when the local regions are coupled. This is because the local simulators would fail to account for the fact that the agents' local regions belong to a larger system and depend on one another. A solution to model the influence the global system exerts on each local region. Fortunately, this does not necessarily imply modelling the entire system, else would just use the global simulator. In
'Source code is available at httpp: httpp::///ittbb.... wwt.luu..cc.mmmM/mM. IIIFLLLLEEEEEE/IIAALS
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

many scenarios, such as in the traffic problem, even though the local regions may be affected by many external variables (e.g. traffic densities in other parts of the city), they are only directly influenced by a small subset of them (e.g., road segments that connect the intersections with the rest of the city). This subset of variables is known as the influence sources. The theoretical framework of Influence- Based Abstraction (Oliehoek et al., 2021) shows that by monitoring the posterior distribution of the influence sources given the action local state history (ALSH), one can simulate realistic trajectories that match those produced by the global simulator. The resulting simulator, known the iulluence-aummente local simulator (IALS), has been proven effective in single agent scenarios when combined with planning (He et al., 2020) and reinforcement learning (RL) algorithms (Suau et al., 2022b).
In this paper, we extend the IBA framework to multi-agent domains. We show how to factorize large networked systems, such as the previous traffic example, into multiple sub-regions so that we can replace the GS by distributed network of IALSS that can run independently and in parallel. There one important caveat to this. The IBA framework assumes only single agent is learning at a time. This assumption is needed to make the influence distributions stationary. This implies that in our case since we want the agents to learn simultaneouslyy, previously computed influence distributions would no longer be valid after the agents update their policies. The naive solution would be to recompute new influence distributions every time any agent updates its policy. However, we argue that this is not only impractical, since recomputing the distributions is not without costs, but also undesirable. The theoretical results in Section 4.1 demonstrate that multiple (similar) joint policies may induce the same influence distributions and that even when they vary a little, they can still elicit the same optimal policies. Further, our insights in Section 4.3 hint that what seems to be problem at first, may in fact be an advantage since in many situations, maintaining the previous influence distributions implies that the local transitions, although biased, remain stationary.
Contributions The main contributions of this paper are: (1) adapting IBA to multi-agent rein- forcement learning (MARL).² and demonstrating that simultaneous learning is possible without incurring major computational costs, (2) showing that by distributing the simulation among different processes, can parallelize training and scale up systems with many agents, (3) revealing that the non-stationarity issues inherent to MARL are partly mitigated as a result of this training scheme.
2 Related Work
A few prior works have investigated the computational benefits of factorizing large systems into independent local regions (Nair et al., 2005; Varakantham et al., 2007; Kumar et al., 2011; Witwicki and Durfee, 201 1). Unfortunately, since local regions are often coupled to one another, such factor- izations are not always appropriate. Nonetheless, in many cases, the interactions between regions occur through a limited number of variables. Using this property, the theoretical work by Olichoek et al. (2021) influence-based abstraction (IBA) describes how to build simulators (IALS) of local-POMDPs, which model only the variables in the environment that are directly relevant to the agent while monitoring the response of the rest of the system with the influence predictor. The problem is that the exact computation of the conditional influence distribution is intractable, and we can only try to estimate it from data. Congeduti et al. (2021) provide theoretical bounds on the value loss when planning with approximate influence predictors. The work by He et al. (2020) has empirically demonstrated the advantage of this approach improve the efficiency of on- line planning in two discrete toy problems. Suau et al. (2022b) scale the method to high-dimensional problems by integrating the IBA framework with single-agent RL showing that the IALS can train policies much faster than the GS. In this paper, we extend the IBA solution to MARL and explain how to build a network of independent IALS such that we can train agents in parallel.
-aggmettd local
One of the consequences of training agents on independent simulators is that the non-stationarity issues arising from having the agents learn simultaneously are partly mitigated. There is sizeable body of literature that concentrates on this issue (Hermandez-Leal et al., 2017), we include a review of these works in Appendix B for completeness. However, we note that the main purpose of this paper is to scale MARL up to systems with many agents. Hence, we are not concerned here with comparing our method with those that exclusively target non-stationarity, especially given that, for scalability reasons, these cannot be applied to the high-dimensional problems we consider here.
2 Althoough the original IBA formulation (Olichoek et al., 2012) is already framed as multi agent, it assumes agents leam one at time while the other agents' policies are fixed.

---

3 Preliminaries
The type of problems we describe in the introduction can be formulated as factored partially observable stochastic games (Hansen et al., 2004), which are defined as follows.
Definition 1 (fPOSG). A factored partially observable stochastic game (fPOSG) is tuple (N, S, A. {R. Qt (O:3) where .N = {1.....)) n is the set of n agents, is the set of j state variables
{S) ,...,, 5* such that every state s E x = 8' = (s y m..y sh.)), A = X Ag is the set of joint actions a' = (af "1:... .,,,,,),, with A, being the set of actions for agent l' the transition function, with T(s*+1|s* ss a.),, (s', (ass)a)) .a9) is the immediate
k-dimensional vector
reward for agent i, S = x XveNS2, is the set of joint observations of (o ot,), with Q being the set of observations for agent i, and O is the observation function for agent i, (,(o(ss").
Solving the fPOSG implies finding the policy n; for each agent that maximizes the expected return G"; as defined in Sutton and Barto (1998). However, agents receive only partial observations o, of the true state 8, which are not necessarily Markovian. Therefore, optimal policies are in general history-dependee in POSG. Hence, we define the agents' policies n, {, (()) as mappings from action-observation histories (AOH), h (o) of.... df- ,)),,, to probability distributions over actions, such that agent i's optimal policy xę is the one that for every AOH hy selects the action with the highest Q-value Q (h', af) = E [G* hn, a', m.].
Given the structural assumptions we made in the introduction about the agents' only being able to observe and be rewarded for what occurs in their local neighborhood, we can narrow down the problem formulation and work with a specific class of IPOSGs called local-form IPOSGs (Oliehoek et al., 2021), which better encompass the problems we consider here.
Definition 2 (Local-form fPOSG). A Local-form fPOSG is fPOSG where O and R; depend only on subset of m state variables X; = {X] X,m' S S. with m I (number of state variables), and agent i's local states x; E x X X being vectors of = (I, 1,t
> . ,..,,,,???? ", such that Oo(o)]s") = (offl()))) and (s(s", = 12, af), where Ô and I?, are the local observation and reward functions for agent i.
3.1 Influence-Basee Abstraction
We now describe the IBA framework (Oliehoek et al., 2021) which intends to simplify the local-form fPOSG formulation by exploiting its structural properties. The framework assumes that there is single agent i learning perspective of agent i, the problem reduces to a POMDP (Kaelbling et al., 1996) where states are pairs (s', ,/ with h-, being the AOHs of all agents but agent i (Nair et al., 2003). Finding locally optimal solution to the Local-form fPOSG can be approached by 'alternating maximization' or 'coordinate ascent'. That is, sequentially iterating over all agents, possibly multiple times, and solving their respective POMDPs (Nair et al., 2003; Olichock and Amato, 2016).
time while all other agents' policies π-; are fixed. Hence, from the
Looking at the definition of Local-form fPOSG, one can argue that, when solving for agent i, sampling actions from the policies of all the other agents (ai and simulating the transitions T|s a²) of the full set of state variables is unnecessary, and while doing so is possible in small problems, it might become computationally intractable in large domains with many agents. Instead, we can define a new transition function T, that models only agent i's local state variables ry (a (ac++1pp [xt, at). The problem is that f+1 may still depend on the other agents' actions o²_ and the non-local state variables S \ X,, which means that T; is not well defined. Fortunately, in many problems, only a fraction of the non-local state variables will directly influence agent i's local region.
The diagram on the left of Figure is Dynamic Bayesian Network (DBN) (Pearl, 1988; Boutilier et al.. 1999) describing a particular instance of the transition dynamics for a generic agent i in local-form IPOSG. Agent i's local region, corresponds the variables that lie within the red box. I; E X = {?). The diagram also shows the non-local variables, known as influence sources "4 U, C S X,, that influence the local region directly. The three dots on the top indicate that there can be, potentially many, other non-local variables in S affecting the local variables X,. These are denoted by y1 Y; S SYX,UU, The diagram also shows that agent j can affect agent i's local region through its actions aj Aj. However, both Y; and A, can only influence X; via U.. Hence, given uf. 1-1 is conditionally independent of of and af. therefore P(x ++ |a4 vi. vl. at) P(a |a ut). x
3

---

sgaedfe
inflnncca
Global Simulator
Local Simulator
Sour002
local variables
lal motinn
11
boal natacia
62
Approximate
Influence Predictor
1=0
Figure 1: Left: A Dynamic Bayesian Network showing agent i's transition dynamics local-form IPOSG prototype. Right: conceptual diagram of the IALS.
The above implies that by inferring the value of the influence source ut, of the other agents and the non-local state variables and thus compute the local state transitions. This can be done by keeping track of the action-local-state history (ALSH) i (xc at..a...... ..., dj 4-1 xf)
monitor the influence
Definition 3 (IALM). An influence-and influence-augmented local Model (IALM) for agent i is tuple (Xi, Us, A,, Fi, Re S2,, Ö,, It). with local states E X x influence sources ui E XzEjU, local transition function [x lay, ut, at), local observation function Ó,(o,*1laj (1]"t) local reward function (x df). and influence distribution (u((1)).
Using the IALM we can compute agent i's local transitions as P( 11, ob)
Σ 7((2 7 |2,, ol)
(I)
Note that, as opposed to the Local-form fPOSG, the transition function 7; in the IALM is defined purely in terms of the local state variables and the influence sources. Moreover, since u, d-separates (Bishop, 2006) I₁ from yt, we only need to maintain a belief over ut, Ta (x((11)) rather than over the full of of state variables of and other agents' histories hí,;, P(s", h (=, All in all, this translates into a much more compact, yet exact representation of the problem (Olichock et al., 2021), which should be computationally much lighter than the original Local-form IPOSG.
3.2 Influence-Augmente Local Simulators
Here we briefly describe how the IALM formulation can be used in practice to build TALSS (Suau et al., 2022b), which consist of local simulator and approximate influence predictor.
Local simulator (LS): The LS is an abstracted version of the environment that only models a small portion of it. As opposed to global simulator (GS), which should closely reproduce the dynamics of every state variable, the LS focuses on characterizing the transitions of those variables X; that agent directly interacts with. 1; (x a ut. at).
Approximate influence predictor (AIP): The AIP monitors the interactions between agent i's local region X,, the external variables Y. and the other agents' actions A-,, by estimating (uf(()) Since, due to combinatorial explosion, computing the exact probability 4( ( is generally intractable (Oliehoek al., 2021), neural network is used instead to approximate the inffuence distribution. Thus, write İe, denote agent i's AIP, where 0g are the network parameters. The AIP io, is trained on a dataset D. of N samples role of the AIP is to estimate the conditional probability of the influence sources uf given the past ALSH, we can formulate the task as a classification problem and optimize the network using the expected cross-entropy loss (Bishop, 2006).
form (1), ut) collected from the GS. Since the
4 Distributed Influence--ugmeentee Local Simulators
As mentioned in the previous section, local-form fPOGGs are solved iteratively the IBA framework. This means that only single agent can update its policy at time. Here, we relax this assumption and discuss the advantages and disadvantages of simultaneous learning. Proofs for all the theoretical results in this section can be found in Appendix A.

---

4.1 Enabling Parallelization
Local Simulators
The main reason to disallow simultaneous learn- ing is that changes in the other agents' poli- cies can affect agent i's influence distribution Luu((()) which may become non-stationary. This renders previously computed influences useless because they no longer capture the true response of the global system.
Environment
This restriction, however, prevents IBA from unlocking its full potential. The fact that each agent's IALS is independent of the others means that the computations can be distributed among different processes that can run in parallel. Hence, putting aside the non-stationarity issue, and assuming overhead costs in spawning an increasing number of processes, the total runtime of the global system grew, either because the
Approximate Influence Predictors
Global Simulator
Figure 2:
conceptual diagram of the DIALS
the method would stay constant the dimensionality number of non-local variables or the number of agents increased. This is in contrast to having agents learn simultaneously in the same GS, in which case larger environments imply longer runtimes. Moreover, since each IALS simulates only portion of the environment the total amount of memory space needed would be split among the different processors. Hence, we could run the simulation on multiple machines with small memory rather than one big machine with very large memory.
In principle, one could prevent the AIPs from becoming stale by simply updating all {Is (u (4) HEN every time any the other agents changes its policy, However, this creates a difficult moving target problem and makes the whole method very inefficient since, especially in deep RL, policies are updated very frequently. Fortunately, as we argue in the following, in many cases, paying the extra cost of retraining the AIPs is neither necessary nor desirable.
4.1.1 Multiple joint policies may induce the same influence distribution
In the following, we show that multiple joint policies may often map onto the same influence distribution (((u)()) E V; for agent N.
Lemma 1. Let = Xle be the product space of joint policies with II, being the set of policies for agent i. Moreover, let V XiENV, V, be the product space of joint influences, with Ve, being the set of influence distributions for agent i. Every joint policy π E II induces exactly one inftuence distribution I,( E Lt for every agent E N.
Proposition 1. The space joint policies II = XieNll; is necessarily greater than or equal to the space of joint inftuences V XIEN W, II which the inequality is strict.
[ Moreover, there exist local-form fPOSGs for
The advantages of this result were shown empirically by Witwicki and Durfee (2010), who demon- strated that planning times can be reduced by searching the space of joint influences rather than the space of joint policies, which is often much larger. In fact, in the extreme case of local transition independence (Becker et al., 2003),² we have that for all joint policies π there is a single {Ti
Corollary 1. Let agent i's influence sources ot be independent of the other agents' actions a. Then, for any joint policy π E II, there is a unique influence distribution I E for every agent i E N and [I| V 1.
The result above implics that, in this particular case, we would only need to train the AIPs once at the beginning. Although we do not expect the situation in Corollary to be the norm, we do believe that in many scenarios, such as in the two environments we explore here, we would not need to retrain the AIPs very often because similar joint policies will influence the local regions in very similar, if not in the same, ways. Furthermore, the next result shows that even when this is not the case, an outdated I; computed from old joint policy might still produce the same optimal policy for agent i.
As opposed to IBA, Becker et al. (2003) assume agents tied by shared global reward.
5

---

4.1.2 Multiple influence distributions may induce the same optimal policy
We use the simulation lemma (Kearns and Singh, 2002) to prove that if two influence distributions are similar enough they will induce the same optimal policy.
Lemma 2. Let M; and M2 be two IALMS differing only on their influence distributions I (ufllt) and 13 (ut (.). Let 2Ns and OMz be the value functions induced by M; and M2 for the same Te. If I3 and I? satisfy
E P(CID (uflt) - (u21() s E. then | (hi, - Q (1i, s AA (H +)(H N.0
2
(2)
for all nu hiy and dis where is the horizon and R |8||o0
Intuitively. Lemma 2 shows that the difference in value between M, and M2² is upper-bounder by the maximum difference between I and 1, times a constant. Actually, if the action-gar gap (Farahmand, 2011) (i.e. value difference between the best and the second best action) in one of the IALMs is larger than twice the difference between the Q Pi and Q the IALMS share the same optimal policy.
Theorem 1. Let M, and M? be two IALMS differing only on their influence distributions I (ufl1) and 17 (. M2 and M2 induce the same optimal policy n² if. for some A,
2ig (1). 0 (1i, a.) 2A Vhl,a at * at with (1., Qis (hi) SA
Vld, me
where oi = arg m 2ip at)
Combining Lemma 2 and Theorem 1, we see that, because the difference in value between M, an M2 depends on & (Lemma 2), the closer the distributions T/ and I? are, the more likely that M, and M& share the same optimal policy. Note that we have no control over the action gap as it domain-dependent. In some domains, the gap might be large and we can be more relaxed about not retraining the AIPs. In some others, the gap might be small and we may need to retrain the AIPs more frequently.
4.2 Algorithm
After the analysis above, we are now ready to present our method, which we call Distributed Influence- Augmented Local Simulators (DIALS). Algorithm describes how we can train multi-agen systems with DIALS. As mentioned earlier, the key advantage of using DIALS is that simulations can be distributed among different processes, and thus training can be fully parallelized. This enables MARL to scale to very large systems with many learning agents. Moreover, following from our theoretical results, AIP training, which can also be done parallel, is performed only every certain number of timesteps. The hyperparameter F in Algorithm controls the AIPs' training frequency. The effect of F on the learning performance is empirically investigated in Section
Algorithm 1 MARL with DIALS
1: Initialize policies (m JeN and AIPs (is, JIN
2:
repeat
3: Collect datasets heN from GS
See Algorithm 2 in Appendix C
4: parallel, for
5: Train AIP fo dataset D;
D. See Section 3.2
6: end for
7: parallel, for
do
8: for F steps do
D F' is the AIPs' training frequency See Algorithm 3 Appendix Using any standard RL method
Simulate trajectories with IALS (T,, R., Ö Io.) Train policy क, 11: end for
9:
12:
end for
13:
until end of training

---

4.3 Mitigating the negative effects of simultaneous learning
The results in the previous section showed that the AIPs may not need be retrained every single time the policies are updated, as the influence distributions may often stay the same or vary only little. Yet, we now argue that, even when changes in the joint policy do affect the influence distributions (x((11)) significantly, it may be advantageous not to retrain the AlPs.
First, we have already mentioned that when all agents learn simultaneously in the same simulator the transition dynamics often look non-stationary from the perspective of each individual agent. This may result in sudden performance drops caused by oscillations the value targets (Claus and Boutilier, 1998). In contrast, when using independent IALS to train our agents, the transition dynamics remain stationary unless we update the AIPs, Hence, by not updating the AIPs too frequently, we get a biased but otherwise more consistent learning signal that the agents can rely on to improve their policies.
Second, we posit that the poor empirical convergence of many off thell Deep RL methods (Hernandez-I.ee) eal et al., 2017; Yu et al., 2021) is also because stochastic gradient descent updates often result in policies that perform worse than the previous ones. Thus, when learning together, agents may try to adapt to other agents' poor performing policies. These policies, however, are likely be temporary as they are just result of the inherent stochasticity of the learning process. Similarly, in many environments, agents shall take exploratory actions before they can improve their policies, which may also negatively impact cooperation if they learn simultaneously (Zhang al., 2009, 2010). In our case, we can again benefit from the fact that the AIPs need to be purposely retrained, and do so only when the policies of the other agents have improved sufficiently.
Even though further theoretical analysis would be needed to be more conclusive about the benefits of using independent simulators, the observations above give reasons to believe that what we initially described as a problem may in fact be an advantage. This view is also supported by our experiments.
5 Experiments
The goal of the experiments to: (1) test whether we can reduce training times by replacing GS with DIALS, (2) investigate how the method scales to large environments with many learning agents, (3) evaluate the convergence benefits of using separate simulators to train agents rather than a single GS, and (4) study the effect of the AIPs' training frequency F on the agents' learning performance.
5.1 Experimental setup
Agents are trained independently with PPO (Schulman et al., 2017)* on (1) the global simulator (GS), (2) distributed influence-augment local simulators (DIALS) with AIPs trained periodically on datasets collected from the GS using the most recent joint policy, (3) DIALS with untrained AIPs (untrained-DIALS))
To measure the agent's performance, training is interleaved with periodic evaluations on the GS. The results are averaged over 10 random seeds on all except on the largest scenarios (10 x 10) for which, due to computational limitations we could only run 5 seeds. We report the mean return of all learning agents. We also compare the simulators in termss of total runtime. For DIALS this includes the agents' training time, the AIPs' training time, and the time for data collection.
5.2 Environments
Traffic control The first domain we consider is a multi- variant of the traffic control benchmark proposed by Vinitsky al. (2018). In this scenario, agents are requested to manage the lights of a big traffic network. Each agent controls single traffic light and can only observe cars when they are inside the intersection's 's local neighborhood. Their goal is to maximize the average speed of cars within their respective intersections. To demonstrate the scalability of the method we evaluate DIALS on four different variants of the traffic network with 4, 25, 49, and even 100 intersections (agents). A screenshot of the traffic network with 25 intersections is shown in Appendix F. The GS and LS are built using Flow (MIT License) (Wu et al., 2017) and SUMO (Eelipse Public License Version 2)
'The vanilla PPO algorithm with decentralized value functions (independent PPO; IPPO) has been shown to perform exceptionally well on several multi- environments (de Witt al., 2020; Yu et al., 2021).
7

---

(Lopez et al., 2018). The GS simulates the entire traffic network while each LS models only the local neighborhood of each intersection i N (Figure 10 in Appendix F). We use the same LS for every igent-intersection but since, depending on where they are located, they are influenced differently by the rest of the traffic network, we have separate AIPs, (To, JieN, for each them. These are feedforward neural networks with the same architecture but different weights Q,, trained periodically with frequency F on datasets (Dhee collected from the GS. The influence sources u, are binary variables indicating whether or not car will be entering from each of the four incoming lanes.
Warehouse Commissioning The second domain we consider is a warehouse commissioning task (Suau et al., 2022b). A team of robots (blue) needs to fetch the items (yellow) that appear with probability 0.02 on the shelves (dashed black lines) of the warehouse (see Figure 9 in Appendix F). Each robot has been designated a 5 x 5 square region and can only collect the items that appear on the shelves at the edges. The regions overlap so that each of the 4 item shelves in a robot's 's region is shared with one of its neighbors. The robots receive a reward between [0, 1] when collecting an item. The exact value depends on how old the item is compared to the other items in their region. This is to encourage the robots to collect the oldest items first. The robots receive as observations a bitmap encoding their own location and set of 12 binary variables that indicate whether or not a given item needs to be collected. The robots, however, cannot see the location of the other robots even though all of them are directly or indirectly influencing each other through their actions. We built four variants of the warehouse with 4, 25, 49, and 100 robots (agents). A screenshot of the warehouse with 25 robots is shown in Appendix F. The GS simulates the entire warehouse while the LS models only a 5 x square region (Figure 10 in Appendix F). We use the same LS for every robot (agent) but since depending on where they are located they are influenced differently by the rest of the robots, we have separate AIPs, {fe, HEN, for each of them. These are GRUs (Cho et al., 2014) with the same architecture but different weights 0; which we train periodically with frequency F on datasets (Di) collected from the GS. Robot i's influence sources u² encode the location of the four neighbor robots. If its AIP fo, predicts that any of the neighbor robots is at one of the 12 cells within its region, and there is an active item on that cell, that item is removed and robot i can no longer collect it.
5.3 Results
GS vs. DIALS The two plots on the left of Figures 3a and 3b show the average return function of the number of timesteps obtained with GS, DIALS, and untrained-DIALS on the 4-agent traffic and warehouse environments. Shaded areas indicate the standard error of the mean. Agents are trained for 4M timesteps on all three simulators. The results reveal that, while agents trained on DIALS seem to converge steadily towards similar high-performinn GS often get stuck in local minima, hence the poor mean episodic reward and large standard error obtained with the GS relative to that of the DIALS. In contrast, the low performance of agents trained with the untrained-DIALE indicates that estimating the influences correctly is important for learning good policies. It is worth noting that the gap between the GS and the DIALS is larger in Figure 3b than in Figure 3a. We posit that this is because, in the warehouse domain, agents are more strongly coupled. For comparison, the dashed-black lines in the plots on the left of Figures 3a and 3b show the performance of hand-coded policies. For the traffic domain, we used fixed traffic light controllers that were extensively optimized by Wu al. (2017). For the warchouse domain, we hand-coded policies that follow the shortest path toward the oldest item in the agent's 's region. The learning curves for the other scenarios are provided in Appendix together with further discussion on these results. 56
policies, agents trained with the
Scalability The benefits of parallelization are more apparent when moving to larger environments. The two bar plots in Figures 3a and 3b depict the final average return and the total run time of training 4. 25, 49, and 100 agents far better than the GS to larger problem sizes, while also yielding better-performing policies. For example, training 100 agents the traffic network takes less than 6 hours with the DIALS, whereas training them with the GS would take more than 10 days. This is speedup factor of 40. In fact,
the two tasks for 4M timesteps. The plots show that DIALS scales
'A video showing the GS of the traffic network and of the IALS is provided at https: / /ttp//uoouuu///////// DgVE601QQ28.
'A video showing how agents trained with DIALS perform on the 100-agent variant of the traffic scenario is provided https:/ / /ywuutuuuuuuuuu be/G9Ethh-Geee G3vo.
8

---

13a)
14,
14.
(3b)
Average Return
200
Average Return
1os runtime (h)
Average Return
Average Return
180
1200 mun
leo
140
120
MTTHAITE--CALL handdeseed
CAAL hand-kooed
100
120
23
40.30
300
2:
300
23 40
100
21 42
100
Timesteps
Number of agents
Timesteps
Number of agents
(a) Traffic
(b) Warchouse
Figure 3: (1a) and (1b) Learning curves with the three simulators on the 4-intersection traffic and 4-robot warehouse environments.
(2a) and (2b): Final average return of agents trained with the three simulators for 4M timesteps. (3a) and (3b): Total runtime of training with the three simulators for 4M timesteps. The y-axis is in logg scale.
since the maximum execution time allowed by our computer cluster is week, the results reported for the GS in the scenarios of size 10 10 do not correspond 4M timesteps but the equivalent of 1 week of training. We would also like to point out that, disregarding the overhead costs associated with multiprocessing, the DIALS runtime should remain constant independently of the problem size. However, to update the AIPs, new samples are collected from the GS, which does increase the runtime. This explains the gap between DIALS and untrained-DIALS. That said, the number of samples needed to update the AIPs (80K for traffic and 10K for warehouse) is significantly lower than the samples needed to train the agents (4M), which is why the runtime difference between GS and DIALS is so large. A table with breakdown of the runtimes is given in Appendix G.
AIPs' training frequency Our first results have already demonstrated that isolating the agents in separate simulators and not updating the AIPs too frequently can be beneficial for convergence. We now further investigate this phenomenon by evaluating the agents' learning performance for different values of the hyperparameter F. The two plots on the left of Figures 4a and 4b show the learning curves for agents trained on DIALS where F is set 100K, 500K. 1M, and 4M timesteps. In the traffic domain, the gap between the green and the purple curve (Figure 4a) suggests that it is important to retrain the AIPs at least every 1M timesteps, such that agents become aware of changes in the other agents' policies. In contrast, in the warehouse domain (Figure 4b), we see that training the AIPs only once at the beginning (DIALS = 4M) seems sufficient. In fact, updating the AlPs too frequently (DIALS F = 100K) is detrimental to the agents' performance. This is consistent with our hypothesis in Section 4.3, The plots on the right show the average cross-entropy (CE) loss the AIPs evaluated on trajectories sampled from the GS. As explained in Section 4 since all agents learn simultaneously, the influence distributions (I( (w( are non-stationary. For this reason, we see that the CE loss changes as the policies of the other agents are updated. We can also see how the CE loss decreases when the AIPs are retrained, which happens more or less frequently depending on the hyperparameter F. Note that the CE not only measures the distance between the two probability distributions but also the absolute entropy. In the warchouse domain, the neighbor robots' locations become more predictable (lower entropy) as their policies improve. This explains why in the first plot from the right the CE loss decreases even though the AIPs are not updated. Also in the same plot, even though by the end of training DIALS F 4M is highly inaccurate, as evidenced by the gap between the purple and the other curves, it is still good enough to train policies that match the performance of those trained with DIALS F 500K and F 1M. This is in line with our results in Section 4. The same plots for the rest of the scenarios are provided in Appendix D.
Scope and Limitations
DIALS targets networked environments with well-defined local regions where the interactions between different regions occur through limited number of variables. There is plenty of examples of domains that have this particular structure including traffic, heating and water systems, logisties, relecommunications, etc. Knowledge of the influence sources U and how these affect the local regions required for building the DIALS. most cases, however, (as in the two environments we explored here) some domain knowledge suffices to be able to tell what the influence sources are.

---

6.71
ALD d6 Folll W. Fudi
fverage Return
200
inueer Loss
DAN SUAGEE
Awerage Return
1 5
Irfluctie Lass
10
DIALS 2-2224
DIALS
D1ALS PM3M
OIALS PM DWLS5
DIALS Fw
Runtimie (h)
Runtime th)
Runtime ih!
Runtime (h)
(a) Traffic 25 agents
(b) Warchouse 25 agents
Figure 4: Left (a) and (b): Learning curves with DIALS for different values of F on the 25-agent versions of the two environments. Right (a) and (b): Influence CE loss as a function of runtime averaged over the 25 AIPs.
Moreover, having or being able to build high-fidelity local simulators of these local regions is also a requirement. Fortunately, there exist plenty of simulators of real systems that can readily be used such as, SUMO (Lopez et al., 2018), Robosuite (Zhu et al., 2020), BRAX (Freeman et al., 2021). There is also a lot of commercial software for building custom-made simulators such as Mujoco (Todorov et al., 2012), or Unity (Juliani et al., 2018). Also, note that most (if not all) of the work that has applied RL real-world problems relies on simulation to train the policies offline (Bellemare et al., 2020; Degrave et al., 2022). Hence, we believe that DIALS can have strong impact on many real-world applications.
Finally, although the experiments reveal that DIALS can considerably accelerate training times, is also memory- demanddnn... As shown in Appendix H (Table 3), the total memory usage with DIALS increases exponentially with the number of simulators/procesees There is thus a trade-off between fast computation and total memory needed. Note, however, that the memory is split among the different processes. Hence, rather than using a big machine with large memory, DIALS can run on several smaller ones with less memory.
7 Conclusion
This paper has offered a practical solution that allows training large networked systems with many agents just a few hours. We showed how factorize these systems into multiple sub-regions such that we could build distributed influence-augmentie local simulators (DIALS).
The key advantage of DIALS is that simulations can be distributed among different processes, and thus training can be fully parallelized. To account for the interactions between the different sub-regions, the simulators are equipped with approximate influence predictors (AIPs), which are trained periodically on real trajectories sampled from a global simulator (GS). We demonstrated that, although using DIALS agents learn simultaneously, training the AIPs very frequently neither necessary nor desirable. Our results reveal that DIALS not only enables MARL to scale up but also mitigates the non- -on-staiinnaitty issues of simultaneous learning.
Future work could analyze this phenomenon from a theoretical perspective, study how to adapt DIALS more strongly coupled domains where frequent training of the AIPs is important, or design a method to directly estimate how the changes in the agents' policies affect the influence distributions. This is so that the AIPs can readily be updated without having to run the GS generate new samples.
Acknowledgements
This project received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innova- tion program (grant agreement No. 758824 -NNFLUENCE Mustafa Mert Çelikok is partially funded by KAUTE Foundation The Finnish Science Foundation for Economies and Technology.
erc
Earouer Seaeech Counc
10

---

References
Becker, R., Zilberstein, S., Lesser, V., and Goldman, V. (2003). Transition Transition-indeeeddeee decentral- ized Markov decision processes. In Proc. of the International Conference Autonomous Agents and Multiagent Systems, pages 41-48.
Bellemare, M. G., Candido, S., Castro. P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. (2020). Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-82.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
Boutilier, C., Dean, T., and Hanks, (1999). Decision-theoretie planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research, 11:1-94.
Castellini, J., Devlin, S., Olichoek, F. A., and Savani, R. (2020). Difference rewards policy gradients. arXiv preprint arXiv:2012.111 1258.
Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014).
Claus, C. and Boutilier, C. (1998). The dynamics of reinforcement learning in cooperative multiagent systems. In Proc. of the National Conference on Artificial Intelligence, pages 746-752.
Congeduti, E.,, Mey, A., and Oliehoek, F. A. (2021). Loss bounds for approximate influence- based abstraction. In Proceedings of the Twentieth International Conference on Autonomous Agents and MultiAgent Systems.
de Witt, C. S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P. H., Sun, M., and Whiteson, S. (2020). Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint S.ti:2001100553:0
Dearden, R. and Boutilier, C. (1997). Abstraction and approximate theisiin- planning. Artificial Intelligence, 89(1-2):219-283.
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., al. (2022). Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):4144477
Farahmand, A.mm. (2011). Action-gap phenomenon in reinforcement learning. Advances in Neural Information Processing Systems, 24.
Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. (2018a) (0). Learning with opponent-learning awareness. In Proceedings of the Seventeenth International Conference on Autonomous Agents and MultiAgent Systems.
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018b), Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence.
Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. (2021). Brax-a dif- ferentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106 13281,
Hansen, E. A., Bernstein, D. S., and Zilberstein, S. (2004). Dynamic programming for partially observable stochastic games. In Proc. of the National Conference on Artificial Intelligence, pages 709-715.
He, J., Suau, M., and Oliehoek, F. (2020). Inffuence-augmentee online planning for complex environ- ments. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 4392-4402. Curran Associates. Inc
Hernandez- P., Kaisers, M., Baarslag, T., and Cote, E. M. (2017). A survey of learning in multiagent environments: Dealing with non-stationarity arXiv preprint 1707.0970.

---

Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 98)):1335 1780.
Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., et al. (2018). Unity: A general platform for intelligent agents. arXiv preprini arXiv:1809.0262)
Kaelbling, L. P., Littman, M., and Moore, A. (1996). Reinforcement learning: A survey. Journal of Al Research, 4:237-285.
Kearns, M. and Singh, S. (2002). Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2):209-22 232.
Konda, V. and Tsitsiklis, J. (1999). Actor-critic algorithms. Advances in neural information processing systems, 12.
Kumar, A., Zilberstein, S., and Toussaint, M. (2011). Scalable multiagent planning using probabilistic inference. In Proc. of the International Joint Conference on Artificial Intelligence, pages 2140- 2146.
Laurent, G. J., Matignon, L., Fort-Piat, L.,, al. (2011). The world of independent learners is not markovian. International Journal of Knowledze-based and Intelligent Engineering Systems, 15(1):55-64.
Li, W., Wang, X., Jin, B., Sheng, J., and Zha, H. (2021). Dealing with non-stationarity in marl via trust-region decomposition. In International Conference Learning Representations.
Lopez, P. A., Behrisch, M., Bicker- Walz, L., Erdmann, J., Flötteröd, Y.-P., Hilbrich, R., Lücken, L., Rummel, J., Wagner, P., and Wießner, E. (2018). Microscopic traffic simulation using sumo. In The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-compettiive environments. Neural Information Processing Systems (NTPS).
Lyu, X., Xiao, Y., Daley, B., and Amato, C. (2021). Contrasting centralized and decentralized critics in agett--gentt reinforcement learning. Proceedings of the Twentieth International Conference on Autonomous Agents and MultiAgent Systems.
McCallum, A. K. (1995). Reinforcement Learning with Selective Perception and Hidden State, PhD thesis, University of Rochester.
Nair, R., Tambe, M., Yokoo, M., Pynadath, D. V., and Marsella, S. (2003). Taming decentral- ized POMDPs: Towards efficient policy computation for multiagent settings. In Proc. of the International Joint Conference Artificial Intelligence, pages 705-711.
Nair, R., Varakantham, P., Tambe, M., and Yokoo, M. (2005). Networked distributed POMDPs: synthesis of distributed constraint optimization and POMDPs. In Proc. of the National Conference on Artificial Intelligence, pages 13-1139.
van der Pol, E. and Oliehoek, F. A. (2016). Coordinated deep reinforcement learners for traffic light control. NIPS' Workshop on Learning, Inference and Control Multi-Agent Systems.
Olichock, F., Witwicki, S., and Kaelbling, L. (2021). A sufficient statistic for influence in structured multiagent environments. Journal of Artificial Intelligence Research, 70:789-88 870.
Oliehoek, F. A. and Amato, C. (2016). Concise Introduction to Decentralized POMDPs. Springer Briefs Intelligent Systems. Springer.
Olichock, F. A., Witwicki, S. J., and Kaclbling, P. (2012). Influence-based abstraction for multiagent systems. In AAAI12.
Pearl, J. (1988). Probabilistic Reasoning Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann.
12

---

Rabinowitz, N.. Perbet, F., Song, F., Zhang, C., Eslami, S. A., and Botvinick, M. (2018). Machine theory of mind. In International conference on machine learning, pages 4218 4227. PMLR.
Raileanu, R., Denton, E., Szlam, A., and Fergus, R. (2018), Modeling others using oneself in multi- agent reinforcement learning. In International conference on machine learning, pages 4257-4266. PMLR.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347.
Spooner, T., Vadori. N., and Ganesh, S. (2021). Factored policy gradients: Leveraging structure for efficient learning MOMDPs. Advances Neural Information Processing Systems, 34.
Suau, M., He, J., Congeduti, E., Starre, R. A., Czechowski, A., and Olichock, F. A. (2022a). Influence- aware memory architectures for deep reinforcement learning in pomdps. Neural Computing and Applications, pages 1-17.
Suau, M., He, J., Spaan, M. T., and Oliehoek, F. (2022b). Inftuence-augmenn local simulators: A scalable solution for fast deep rl in large networked systems. In International Conference on Machine Learning, pages 20604-20624. PMLR.
Sutton, R. S. and Barto, A. G. (1998). Reinforcement Learning: An Introduction. The MIT Press.
Tan, M. (1993). Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pages 330-337.
Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE
Varakantham, P., Marecki, J., Yabu, Y., Tambe. M., and Yokoo, M. (2007). Letting loose SPIDER on network of POMDPs: Generating quality guaranteed policies. In Proc. of the International Conference on Autonomous Agents and Multiagent Systems.
Vinitsky, F., Kreidieh, A., Flem, I.., Kheterpal, N., Jang, K., Wu, C., Wu, F., Liaw, R., Liang, E., and Bayen, A. M. (2018). Benchmarks for reinforcement learning in mixed-autonommy traffic. In Conference on robot learning, pages 399-409. PMLR.
Witwicki, S. J. and Durfee, E. H. (2010). Influence-basee policy abstraction for weakly- Dec-POMDPs. In Proc. of the International Conference on Automated Planning and Scheduling, pages 185-192.
Witwicki, S. J. and Durfee, E. H. (2011). Towards unifying characterization for quantifying weak coupling in Dec-POMDPs. In Proceedings of the Tenth International Conference on Autonomous Agents and Multiagent Systems, pages 29-36, Taipei, Taiwan.
Wu, C., Kreidieh, A., Parvate, K., Vinitsky, E., and Bayen, A. M. (2017). Flow: modular learning framework for autonomy in traffic. arXiv preprint 1rXiv:1710.05330
Yu, C., Velu, A., Vinitsky, E., Wang, Y., Bayen, A., and Wu, Y. (2021). The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955
Zhang, C., Abdallah, S., and Lesser, V. (2009). Integrating organizational control into multi- agent learning. Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pages 757-764.
Zhang, C., Lesser, R., and Abdallah, S. (2010). Self-organization for coordinating decentralized reinforcement learning. In AAMAS, volume 10, pages 739-746.
Zhu, Y., Wong, J., Mandlekar, A., and Martin-Martin, R. (2020). robosuite: A modular simulation
framework and benchmark for robot learning. arXiv preprint 0rXiv:2009.12293
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] proofs are included in Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] Code and instructions on how to reproduce all the experimental results are included in the supplemental material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1 for description of the experimental setup and Appendix I for full list of the hyperparameter configurations used for the experiments along with an explanation of how these were chosen.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] The shaded areas in the learning curves and the black vertical lines on the bar plots in Section 5.3 and Appendix D show the standard error of the mean.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The computer requirements and specifications along with a table showing a breakdown of the total runtimes for all experiments are provided in Appendix G.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "qFsP22wLJZw", "reviewer": "Reviewer_EtVj", "summary": "The authors extend the influence augmented local simulator paradigm to the multi-agent case and show that doing this leads to improvement in both run-time and training outcomes.", "questions": "### Questions:\n\n- Is it necessary to group every agent into its own simulator? Could the result be better if instead you grouped them into sets of 2x2? I’d love to see this experiment (though, note, it will not affect my review score I just think it’ll make for a better paper!)\n- I’m curious about the non-improvement using recurrent neural networks; is it possible that the RNNs are simply better able to “solve” their local environment and this causes failure in the global simulator due to overfitting to the local simulator? Is it necessary for the agents to not do too well in the local environment?\n- Is the time-index available to the influence estimators?\n- How many rollouts are in the training batch for PPO? This seems to be missing from the hyperparameter section.", "limitations": "It is not fully clear from the authors how much work went into tuning the simulation scheme and consequently it is not fully clear how robust the scheme is. This would be useful information to include in the appendix.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Improvements in runtime\n- Improvements in agent return\nWeaknesses:\n- Theorem bounds are very loose and consequently don't actually provide much intuition\n\n### Writing:\n\n- The caption in Figure 3 is quite hard to follow. It’s not clear what “first (a) (b)” is nor what the “third a and b” would be\n- In the definition of the fPOSG, it seems F is not defined?\n- Why does Figure 3 use the 4M agent when it has significantly worse return than some of the more frequent updating agents i.e. the 1M or 500K ones\n- It’s probably too late to change this but note that DIAL is already the name of a method in the MARL literature ([https://proceedings.neurips.cc/paper/2016/file/c7635bfd99248a2cdef8249ef7bfbef4-Paper.pdf](https://proceedings.neurips.cc/paper/2016/file/c7635bfd99248a2cdef8249ef7bfbef4-Paper.pdf)). Up to you what you do with this information but I figured it was good for you to be aware of this.\n\n### Correctness\n\n- Lines 116-118 contain a claim about being able to solve local form fPOSG by sequential iteration over agents and solving their POMDP. If this is true, it probably requires a reference or a pointer to the appendix; it is not obviously true from simply reading it.\n\n### Experiments:\n\n- Readers of this paper likely do not have intuition about the meaning of the reward in the traffic light and warehouse case. If given sufficient time, it might be worth including some metrics like speed or objects collected. Similarly, it might be useful to include a hand-tuned baseline (for example, in the traffic light case a fixed set of timing signals) to give a sense of whether the RL policy is learning anything good or is still underperforming simple baselines. Including these experiments or results will not improve my score but I do think it’ll make for a better paper!", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "m83hdlLMNQV", "reviewer": "Reviewer_iVuf", "summary": "This paper extends the influence-based abstraction (IBA) framework to the multi-agent reinforcement learning case (MARL). In the proposed approach, the global system simulator is decomposed into a distributed network of local simulators, each including a set of \"influence\" variables that model how the local simulators interact with each other. The idea is to independently train agents for each local simulator, in parallel, and periodically update approximate influence predictors (AIP) to account for changes in the other agents' simultaneously trained policies. The paper offers a bit of theoretical analysis justifying the choice of periodically training AIPs. The proposed method (DIALS) is evaluated in two multi-agent environments, showing that DIALS results in not only faster training time compared to training on the global simulator, but also better performance. The authors further numerically study the effect of the frequency of training AIPs, showing that re-training them occasionally can improve performance, but doing it too often can also be detrimental.", "questions": "**Main questions:**\n- My main question is about the constraints on the system simulator. Am I correct in my understanding that the ability to simulate local regions of the environment, one assigned to every agent, is a constraint placed on the system to be solved? (including freely resetting the local states). Can you discuss this during rebuttal, either explaining why my understanding is wrong, or why this is not too problematic in practice. \n- In the paragraph after Theorem 1 (line 236) it is said that one can chose $\\Delta$ according to Lemma 2 so guarantee that the conditions of Theorem 1 are met. This is clearly the case for the second condition relating the Q-values obtained with M1 and M2. However, to me this is not obviously true for the first condition of the Theorem, which requires the action-gap in M1 to be larger than $2\\Delta$. Am I missing something?\n- I would have liked to see more discussion about why training the global simulator (GS) leads to worse performance than using DIALS, since the training data resulting from GS is more accurate. Is this a model capacity issue? Could it be related to the choice of IPPO being better suited to learn with local simulators? Do you observe this when the base MARL method is different?\n\n**Some minor comments/questions:**\n  - The paragraph in line 257 discusses the \"poor empirical convergence of many Deep MARL methods\", but offers very little support for this claim in terms of references or particular experimental results this is referring to. Strong claims like this should be better motivated and discussed, in my opinion. \n  - The discussion about RNNs in the AIP paragraph that starts in line 157 seems somewhat out of place. At this point in the paper the discussion has been mostly theoretical, and it seems that all you require here is some generic estimator for the influence distribution. The use of RNN is just one specific way to do this, no? This might be better suited to be described as mostly a technical choice for the experiments. \n  - I didn't see a definition for $F$ in Definition 1 (line 88). \n  - The claim \"In this paper we extend the IBA framework to multi-agent domains\" seems overly strong. I took a look at the original IBA paper, and the framework is already tailored to multi-agent problems, so maybe you are referring to the reinforcement learning case? I think that some qualification is missing here. ", "limitations": "Not much discussion about limitations, including some I outlined above. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- *Originality:* I'm not too familiar with the literature of this area, but the proposed extension of IBA to MARL appears to be novel, judging from recent citations to the original IBA paper. As far as I understand, the paper offers a few novel technical contributions on top of the existing IBA theoretical framework, namely:\n  - Use parallel training of local simulators, each with their own single-agent policy.  \n  - Include periodically learned AIPs to account for non-stationarity due to other agents, with the retraining frequency as a hyperparameter controlling addressing non-stationarity vs stability.\n  - Proving Theorem 1, which justifies the above choice by essentially implying that one does not need exact influences to obtain optimal policies.\n- *Significance:* This is clearly a relevant contribution to the MARL literature, which is an active area of research. The experimental results show that this technique can be used to accelerate and improve the learning of multi-agent policies, and the underlying ideas are simple enough that I could see them being adopted by the community.\n- *Clarity:* The paper is reasonably well-written and easy to follow. \n- *Quality:* The main ideas are sensible and follow naturally from the IBA framework. I haven't checked the derivations, but the resulting claims are not surprising and appear correct. \n  \n**Weaknesses:**\n- As far as I can tell, applying DIALS requires one to be able to easily decompose the environment simulator into local pieces that can be executed in parallel. This seems like not only a theoretical limitation, but also a strong practical one, as it means that the environment's simulator must be implemented in a very particular way. I might be missing something, but it doesn't seem like DIALS is a method that can be applied out-of-the-box to any relevant multi-agent environment; it puts strong constraints on how the system is simulated. For example, I'm assuming the environments they used in their experiments were implemented by the authors; at least the wording in line 291 seems to imply this is the case for the traffic control experiments, but nothing is said for the warehouse environment. More generally, I didn't see a lot of discussion of this point in the paper, which I would argue is an important omission. \n- The experimental evaluation is somewhat limited, with only two domains and one baseline being considered. Some of the results are not necessarily intuitive and not a lot of discussion is offered (see my last main question below). \n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_FIBByyir93", "reviewer": "Reviewer_yftC", "summary": "This work focuses on factorizing large networked system simulators into smaller local simulators for training multi-agent reinforcement learning. The proposed method creates an influence augmented local simulator by training an influence predictor using data from the global simulator and applying this predictor to the local simulator. ", "questions": "see above", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Factorization is a common methodology in MARL training. This work proposes a different perspective which is factorizing the underlying simulator to improve scalability in the number of agents and the size of the simulated environment. The side effect of the method in mitigating the non-stationary issue is also worth further investigation. The manuscript is overall well written and easy to follow. A preliminary theoretical analysis is also provided. \n\nThe following are some of my questions and concerns:\n- Factorizing the large simulator into smaller simulators involves running multiple processes of smaller simulators and training/executing the AIP, the overhead of such a method should be evaluated. The tradeoff information between the number of local simulators and additional memory and computation overhead would be helpful. It gives us a better sense in terms of what size of simulator should be partitioned and how many partitions it should have.\n- The large simulator is factorized into many single-agent environments. To improve the training efficiency and resource utilization efficiency, it may be more beneficial to partition the large simulator into multiple multi-agent simulators. This should be considered in the evaluation section.\n- Although periodic update of AIP with slower frequency seems to stabilize the training. I wonder if that would affect exploration of the MARL training. For instance, if there is an optimal joint state that is harder to achieve, would the slow update of AIP slows down the exploration and convergence?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept