Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work focuses on factorizing large networked system simulators into smaller local simulators for training multi-agent reinforcement learning. The proposed method creates an influence augmented local simulator by training an influence predictor using data from the global simulator and applying this predictor to the local simulator. 
strengths_and_weaknesses: Factorization is a common methodology in MARL training. This work proposes a different perspective which is factorizing the underlying simulator to improve scalability in the number of agents and the size of the simulated environment. The side effect of the method in mitigating the non-stationary issue is also worth further investigation. The manuscript is overall well written and easy to follow. A preliminary theoretical analysis is also provided. 

The following are some of my questions and concerns:
- Factorizing the large simulator into smaller simulators involves running multiple processes of smaller simulators and training/executing the AIP, the overhead of such a method should be evaluated. The tradeoff information between the number of local simulators and additional memory and computation overhead would be helpful. It gives us a better sense in terms of what size of simulator should be partitioned and how many partitions it should have.
- The large simulator is factorized into many single-agent environments. To improve the training efficiency and resource utilization efficiency, it may be more beneficial to partition the large simulator into multiple multi-agent simulators. This should be considered in the evaluation section.
- Although periodic update of AIP with slower frequency seems to stabilize the training. I wonder if that would affect exploration of the MARL training. For instance, if there is an optimal joint state that is harder to achieve, would the slow update of AIP slows down the exploration and convergence?

questions: see above
limitations: yes
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors extend the influence augmented local simulator paradigm to the multi-agent case and show that doing this leads to improvement in both run-time and training outcomes.
strengths_and_weaknesses: Strengths:
- Improvements in runtime
- Improvements in agent return
Weaknesses:
- Theorem bounds are very loose and consequently don't actually provide much intuition

### Writing:

- The caption in Figure 3 is quite hard to follow. It’s not clear what “first (a) (b)” is nor what the “third a and b” would be
- In the definition of the fPOSG, it seems F is not defined?
- Why does Figure 3 use the 4M agent when it has significantly worse return than some of the more frequent updating agents i.e. the 1M or 500K ones
- It’s probably too late to change this but note that DIAL is already the name of a method in the MARL literature ([https://proceedings.neurips.cc/paper/2016/file/c7635bfd99248a2cdef8249ef7bfbef4-Paper.pdf](https://proceedings.neurips.cc/paper/2016/file/c7635bfd99248a2cdef8249ef7bfbef4-Paper.pdf)). Up to you what you do with this information but I figured it was good for you to be aware of this.

### Correctness

- Lines 116-118 contain a claim about being able to solve local form fPOSG by sequential iteration over agents and solving their POMDP. If this is true, it probably requires a reference or a pointer to the appendix; it is not obviously true from simply reading it.

### Experiments:

- Readers of this paper likely do not have intuition about the meaning of the reward in the traffic light and warehouse case. If given sufficient time, it might be worth including some metrics like speed or objects collected. Similarly, it might be useful to include a hand-tuned baseline (for example, in the traffic light case a fixed set of timing signals) to give a sense of whether the RL policy is learning anything good or is still underperforming simple baselines. Including these experiments or results will not improve my score but I do think it’ll make for a better paper!
questions: ### Questions:

- Is it necessary to group every agent into its own simulator? Could the result be better if instead you grouped them into sets of 2x2? I’d love to see this experiment (though, note, it will not affect my review score I just think it’ll make for a better paper!)
- I’m curious about the non-improvement using recurrent neural networks; is it possible that the RNNs are simply better able to “solve” their local environment and this causes failure in the global simulator due to overfitting to the local simulator? Is it necessary for the agents to not do too well in the local environment?
- Is the time-index available to the influence estimators?
- How many rollouts are in the training batch for PPO? This seems to be missing from the hyperparameter section.
limitations: It is not fully clear from the authors how much work went into tuning the simulation scheme and consequently it is not fully clear how robust the scheme is. This would be useful information to include in the appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper extends the influence-based abstraction (IBA) framework to the multi-agent reinforcement learning case (MARL). In the proposed approach, the global system simulator is decomposed into a distributed network of local simulators, each including a set of "influence" variables that model how the local simulators interact with each other. The idea is to independently train agents for each local simulator, in parallel, and periodically update approximate influence predictors (AIP) to account for changes in the other agents' simultaneously trained policies. The paper offers a bit of theoretical analysis justifying the choice of periodically training AIPs. The proposed method (DIALS) is evaluated in two multi-agent environments, showing that DIALS results in not only faster training time compared to training on the global simulator, but also better performance. The authors further numerically study the effect of the frequency of training AIPs, showing that re-training them occasionally can improve performance, but doing it too often can also be detrimental.
strengths_and_weaknesses: **Strengths:**

- *Originality:* I'm not too familiar with the literature of this area, but the proposed extension of IBA to MARL appears to be novel, judging from recent citations to the original IBA paper. As far as I understand, the paper offers a few novel technical contributions on top of the existing IBA theoretical framework, namely:
  - Use parallel training of local simulators, each with their own single-agent policy.  
  - Include periodically learned AIPs to account for non-stationarity due to other agents, with the retraining frequency as a hyperparameter controlling addressing non-stationarity vs stability.
  - Proving Theorem 1, which justifies the above choice by essentially implying that one does not need exact influences to obtain optimal policies.
- *Significance:* This is clearly a relevant contribution to the MARL literature, which is an active area of research. The experimental results show that this technique can be used to accelerate and improve the learning of multi-agent policies, and the underlying ideas are simple enough that I could see them being adopted by the community.
- *Clarity:* The paper is reasonably well-written and easy to follow. 
- *Quality:* The main ideas are sensible and follow naturally from the IBA framework. I haven't checked the derivations, but the resulting claims are not surprising and appear correct. 
  
**Weaknesses:**
- As far as I can tell, applying DIALS requires one to be able to easily decompose the environment simulator into local pieces that can be executed in parallel. This seems like not only a theoretical limitation, but also a strong practical one, as it means that the environment's simulator must be implemented in a very particular way. I might be missing something, but it doesn't seem like DIALS is a method that can be applied out-of-the-box to any relevant multi-agent environment; it puts strong constraints on how the system is simulated. For example, I'm assuming the environments they used in their experiments were implemented by the authors; at least the wording in line 291 seems to imply this is the case for the traffic control experiments, but nothing is said for the warehouse environment. More generally, I didn't see a lot of discussion of this point in the paper, which I would argue is an important omission. 
- The experimental evaluation is somewhat limited, with only two domains and one baseline being considered. Some of the results are not necessarily intuitive and not a lot of discussion is offered (see my last main question below). 

 
questions: **Main questions:**
- My main question is about the constraints on the system simulator. Am I correct in my understanding that the ability to simulate local regions of the environment, one assigned to every agent, is a constraint placed on the system to be solved? (including freely resetting the local states). Can you discuss this during rebuttal, either explaining why my understanding is wrong, or why this is not too problematic in practice. 
- In the paragraph after Theorem 1 (line 236) it is said that one can chose $\Delta$ according to Lemma 2 so guarantee that the conditions of Theorem 1 are met. This is clearly the case for the second condition relating the Q-values obtained with M1 and M2. However, to me this is not obviously true for the first condition of the Theorem, which requires the action-gap in M1 to be larger than $2\Delta$. Am I missing something?
- I would have liked to see more discussion about why training the global simulator (GS) leads to worse performance than using DIALS, since the training data resulting from GS is more accurate. Is this a model capacity issue? Could it be related to the choice of IPPO being better suited to learn with local simulators? Do you observe this when the base MARL method is different?

**Some minor comments/questions:**
  - The paragraph in line 257 discusses the "poor empirical convergence of many Deep MARL methods", but offers very little support for this claim in terms of references or particular experimental results this is referring to. Strong claims like this should be better motivated and discussed, in my opinion. 
  - The discussion about RNNs in the AIP paragraph that starts in line 157 seems somewhat out of place. At this point in the paper the discussion has been mostly theoretical, and it seems that all you require here is some generic estimator for the influence distribution. The use of RNN is just one specific way to do this, no? This might be better suited to be described as mostly a technical choice for the experiments. 
  - I didn't see a definition for $F$ in Definition 1 (line 88). 
  - The claim "In this paper we extend the IBA framework to multi-agent domains" seems overly strong. I took a look at the original IBA paper, and the framework is already tailored to multi-agent problems, so maybe you are referring to the reinforcement learning case? I think that some qualification is missing here. 
limitations: Not much discussion about limitations, including some I outlined above.
review_title: null

Meta-review: metareview: 
All the reviewers agree that the paper present novel and interesting contributions. As such I recommend acceptance.

Please incorporate the last feedback from Reviewer iVuf.

confidence: Certain
Decision: Accept