Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 

Summary:
The paper proposes a deep learning based algorithm for computing Multi agent Nash equilibrium, relying on the Forward Backward Stochastic Differential Equation representation (BSDE) of the solution together with Fictitious play algorithm. The authors present numerical examples on linear quadratic symetric stochastic games with 1000 agents, as well as solutions to partially observed 2 agents racing problem using extended Kalman filter. 

Strength of the paper:
- The derivation on a scalable  Deep learning based algorithm for solving FBSDEs and related stochastic game;
- The possibility to approximate the Nash equilibrium in a game with a large number of players (more than 1000)
- The presentation of examples in a partially observable framework, where Kalman filtering is necessary to derive the solution of the game.

Weakness of the paper:
- The BSDE presented in the paper is obtained through the consideration of the corresponding Hamilton Jacobi Bellman  Partial Differential Equation, although such representation could a priori be obtained in a more general and possibly non Markovian framework
- The BSDE representation is claimed to be different form the one introduced in Han & Hu, but this should be detailed more precisely, as few information on this crucial point is given in the current version of the paper. Is it e.g. related to the use of Pontryagin principle of value process approach for the FBSDE? 
- In the car racing example, it does not seem clear how the proximity to a Nash equilibrium is quantified. It is also surprising to focus on a 2 agents example, while the paper seems to intend to emphasize the scalability of the approach.
- It is a priori not clear how the presented BSDE representation differs from a related recent paper by Han, Hu and Long (Convergence of Deep Fictitious Play for Stochastic Differential Games). This should be discussed.
- It seems that this paper only focuses on games with unique Nash equilibrium, although the use of RL algorithm for games seems to become intricate whenever several Nash equilibrium shop up into the picture. 


Recommandation; Given the strengths of the paper in terms of FBSDE representation of the solution, together with the weaknesses presented above, I recommend to reject the paper, but am looking forward to get clarifications on the points raised above.

Minor points:
1. Typos: « drive fast than » p8.; « invarint layer » in Figure 5; 
2. Figure 2: Can you clarify the number of agents considered for such approximation? By the way, how does the solution changes as the number of agents increases? It could a priori seem to be quite stable.
3. Do the games considered here need to be non-cooperative by nature? For which purpose for the algorithm?
4. Some important references seem to be missing: some on the use of Fictitious Play for learning the Nash equilibrium in multi player games; some connecting BSDE representation to multi-player  stochastic games.


review_title: Review on Multi-agent Deep FBSDE Representation For Large Scale Stochastic Differential Games

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary

The paper introduces improved deep learning architecture for solving stochastic differential games by fictitious play. Compared to previous best model it uses LSTM instead of MLP to capture forward dynamics of the system, importance sampling and order invariant encoding of other agents to improve sample efficiency of learning.


Strong points

* Order invariant encoding uses inherent structure of the problem that improves sample efficiency compared to previous approaches. 
* Batching of some operations speeds up the algorithm.
* Two different domains are used to evaluate the model.


Weak points

* Time complexity claims are problematic.
* The paper is sometimes difficult to follow. It seems like 8 pages aren't enough to describe the whole approach in detail. Many things in appendix seem central for the presented system. 
* Importance sampling that is claimed to be important ingredient isn't described well enough neither in the main text nor in the appendix.
* Quantitative evaluation is presented only for domains that can be already solved analytically. In other domains only qualitative results are presented.


Recommendation

I recommend rejecting the paper due to lack of clarity and lack of evaluation against other methods suitable for solving problems that can't be solved analytically.


Questions

* It seems to me that "propagate by batch" time complexities in Table 1 assume constant time computation of batches. Is that correct? That would however be a simplification, while batched computation is much faster on GPUs it isn't in general constant time. This view is further supported by Fig 4 where empirical time complexity also isn't constant as is suggested by Table 1.

* My understanding of empirical results is that the model is run in domains with analytical solution only as sanity check. The true value is in domains without analytical solution (e.g. Superlinear Simulation section). Current evaluation for superlinear setup and autonomous racing is only qualitative. I would like to see comparison to previous methods applicable to these domains if there are any (numerical solvers?). If there aren't any the paper should explain why this is the case. I understand that the model is an improvement over Han & Hu however does it allow us to do something we couldn't do before when we consider even non learning methods? Please reflect on this point to help me better understand main contribution of the paper. This broader context would help readers (like me) that aren't experts in stochastic differential games.


Possible improvements

* Since most of the results compare presented model against Han & Hu 2019 it would help to add a paragraph that briefly summarizes algorithmic/model differences. Is it true that current system is Han & Hu with LSTM instead of MLP + importance sampling + invariant layer? 

* In table 2 unit of "Total time" isn't specified.

* "memory complexity of LSTM with respect to time is O(1)" --- this is true for inference time, at train time O(T) activations still have to be stored. (If they aren't to be recomputed again for each t < T). The text can be more clear on that.

* In Figs 5 and 6 captions say that it compares current system against Han & Hu however the legend shows FC and LSTM with and without invariant layers. I assume that Han & Hu = FC and current system is LSTM + invariant layer. However being more explicit about that would  help. Does FC model here use importance sampling or not?

Typos
* solutions exist only few special -> only FOR few
* stochastic optional control problem -> OPTIMAL control
* drive fast than -> drive fastER
* global augment -> augmentED
* The paper would benefit from further proofreading. 
review_title: Some questions about clarity of presentation and significance of the approach

final_score: rating: 5: Marginally below acceptance threshold
review_content: Clarity:
Till page 3 the paper was easy to follow, i.e., the analytical expressions in eq(5), and the basic idea of Algorithm 1 (which is same as prior works by Han et al. , Wang et al., Periera et al.) are clear.  However, after page 3 the paper is hard to follow. The specific points are as follows:
1) It looks like both LSTM and FC have been tried in the past.  So what is the novelty here? Is it making the deep FBSDE framework scalable?  and if so how is this achieved? by introducing the invariant layer? or just using the symmetry property to share the networks?
2) Comparisons with Han& Hu (2019) is presented only in Table 2. Figure 5 says comparison with Han & Hu (2019), but only FC w/wo invariant layer and LSTM w/wo invariant layer have been presented. Is it true that Han & Hu (2019) is FC and LSTM without the invariant layer and the current approach in the paper is FC and LSTM with the invariant layer?
3) Figure 2, says comparison between analytical and deep FBDSDE solutions. Yet, 10 trajectories are displayed. Do they correspond to 10 different agents? If  so, what is the difference between the 10 trajectories and the deep FBSDE solution?
4) In the autonomous racing section,  the authors  mention "The results emphasizes the generalization ability of our framework" what is generalization here? Is it that the method also works on a different domain?

Quality:
1) The main contribution seems to be scalability. It will be a lot helpful if Section 3.2 is elaborated a bit more, where the `scalability' aspects of the current work is discussed in detail. As of now, Section 3.2 seems `rushed'.
2) Table 2 compares the proposed method with Han&Hu (2019). This is done for 10 agents. Yet, we do not know what happens as we increase the number of agents.
3) Figure 4 has number of agents in x-axis, yet it compares only the various implementations of the current work and prior work is not included. 


Novelty:
It appears that the paper puts together blocks/algorithms in prior works (say invariant layer from Zaheer et al. 2017, deep FBSDE from Wang et al., Periera et al., Han & Hu). This way novelty seems  to be limited. 

Significance:
Scalability is the main take away of this paper. Figure  5 and Figure 4 show that the proposed method is better than prior methods, yet the performance improvement is not significant. However, the savings could be in the time/space complexity, which the paper does not elaborate. In short, due to the fact that certain critical parts have not been explained it is hard to judge the significance.


Overall Feedback:  The critical information regarding the invariant layers, scaling, savings in the scaling is missing. Also, adding this information may not take significantly extra space. So, it will be great if the authors can provide this information.

review_title: Puts together prior works and exploits problem structure to achieve scaling. Yet, critical details related to scaling are missing.

Meta-review: This paper introduces a scalable method for FSP based on FBSDE. The method is theoretically derived then applied on two problems, one simple but with many (1000) agents, and one with only 2 agents but partial observability.

The main strength of this paper lies in the scalability and the time complexity of the proposed method. Computing Nash equilibriums for many agents is a difficult problem and this paper is interesting in this aspect.

However, the reviewers point out several weak points to this paper. The difference with a previous work by Han, Hu and Long needs to be highlighted. Some parts of the paper are not clear, and too much of the important results are pushed into the appendix. Maybe this work is not best fitted to a conference format, and should be submitted to a journal? Another concern raised by the reviewers is that the experimental section does not show significant enough results, and that it is surprising to see a 2-agents problem as an illustration of a method that is aiming at addressing scalability with respect to the number of agents.

Reviewers agree on rejection for this paper, although by a small margin. I therefore recommend rejection. I think that if the authors improve this paper by following the reviewer suggestions, it can be accepted in a future venue.
Decision: Reject