Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This article proposes *Deep Schrödinger Bridge (DeepGSB)*, a numerical algorithm for solving large-scale (in state dimension) stochastic Mean Field Games (MFG) with hard distributional target constraint, continuous state space and flexible mean-field (MF) interaction function.  The authors adapted the use of Hopf-Cole transformation and Schrödinger factors, similar to in [1], to transform MFG PDEs (with hard target) to a system of Schrödinger Bridge (SB) PDEs that depend on an MF interaction term $F$.  In order to solve this system, the authors extended the Forward-Backward Stochastic Differential Equation (FBSDE) representation for (SB) from [2] to account for $F$.  DeepGSB solves this FBSDE by fitting neural network approximators in an alternating manner, similar to in [2] and past works, with an added TD-like loss function to account for $F$. Experiments are performed for a crowd-navigation use case, similar to in [3, 4], and an opinion depolarization use case (based on the party model [5]).

[1] Wasserstein Proximal Algorithms for the Schrödinger Bridge Problem: Density Control with Nonlinear Drift. K.F. Caluya, A. Halder.

[2] Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory. T. Chen, G. Liu, E. Theodorou.

[3] A machine learning framework for solving high-dimensional mean field game and mean field control problems. Ruthotto, et. al.

[4] Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games. Lin, et. al.

[5] Polarization in geometric opinion dynamics. Gaitonde, et. al.
strengths_and_weaknesses: Strengths
- A neural-based numerical solver for MFGs with exact terminal distribution and flexible mean field interaction can be highly useful and desirable in practice. In the crowd motion use case, the method is shown to respect the obstacles (e.g., flexible MF interaction) without losing convergence to the target, implying that in a real-world scenario, the agents are moving safely until they finally reach their intended destination.  Present methods are only able to solve for a smaller class of MFGs.
- In addition to exactness and flexibility, the method is empirically shown to be scalable by solving a high-dimensional ($d=1000$) instance. Having surveyed related works, as far as I can tell, this seems to be the largest documented to date.
- The algorithm is as simple as/simpler than APAC-Net [1] and [2], yet it solves a wider class of MFGs.  Setting the TD terms as regressand to allow account for $F$ is a principled way to handle a flexible of choice of $F$.  As the paper states, this can be generalized further to make use of other computational tools in reinforcement learning (RL).  As it stands, the design choices make use of best practices in RL, such as replay buffers and using multi-step TDs.
- The use case of opinion depolarization by adopting the party model in an MFG seems to be novel and it seems very realistic to have a high-dimensional opinion space.  This could be a good test bed for future (and past) numerical MFG solvers.
- To obtain an FBDSE system for an MFG, the authors extended and proceeded very similarly to an established result in [3] (Theorem 3 and its proof) originally intended for generative models.  This reuse unlocks a numerical solver for a wide class of MFGs.

Weaknesses
- The ML-based methods in [1] and [2] seem to be designed to solve high-dimensional MFGs.  This paper seems to indicate that they are not able solve for $d=1000$ but it's not clear why (too much computational resource needed?).  It might be informative to compare the computational complexity and/or running time of the methods (instead).
- The direct comparisons against present methods are done visually (Figure 5 and 7, in the appendix). It can be more rigorous to compare the Wasserstein distance to target of the proposed method against that of the present methods.  I see that it can be hard to put all the methods on an equal footing, as the proposed method solves a wider class of MFGs.  It would be a fair comparison though if we restrict to a class of MFGs that are solvable by all the methods -- it seems that crowd motion is a use case that is also showcased in [1,2].  A dynamical optimal transport problem can also be a good test bed, as per Example 1 in [1].
- There are some clarity issues -- while the paper is mostly well-written, it would be beneficial for non-expert readers to make some changes:
1. (Line 64) I don't think it's correct to write that SB is an emerging machine learning *model*, it might be more accurate to introduce the topic by stating what the SB problem is.
2. In equation 1, what is the domainof $G$? Shouldn't it be $\mathbb{R}^d \times \mathcal{P}(\mathbb{R}^d)$? If so, then the value function should be defined as: $u(x,T)=G(x,\rho(\cdot,T))$.
3. (Line 49-52) As the authors discuss the drawbacks of present methods, it would be good to include citations for every drawback (which method has which drawback).
4. Proposition 2 does not seem to be proven.   Please provide at least a proof-sketch -- it is not obvious to me how exactly the random terms in equations 14(a,b) are dealt with.
5. There are grammatical mistakes, but they do not seem to impact my understanding.  It would be beneficial if the authors were to edit the article.
6. It is not clear to me why DeepGSB is more robust to hyperparameters (Line 247).  Can you please clarify via a theoretical and/or empirical argument?

[1] Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games. Lin, et. al.

[2] A machine learning framework for solving high-dimensional mean field game and mean field control problems. Ruthotto, et. al.

[3] Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory. T. Chen, G. Liu, E. Theodorou.

questions: Every item under "Weaknesses" contains a question and/or suggestion that can be answered/addressed.
limitations: Adequately addressed
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors apply diffusion Schrodinger Bridges (DSB) to mean field game applications, this generalizes existing DSB approaches to include mean field interactions. The authors utilise a novel loss term based on temporal difference objectives within each IPF-like iteration.

The experiments are validated on 2D crowd navigation and opinion depolarisation experiments.
strengths_and_weaknesses: Strengths
- This is an interesting paper and connects DSB beyond generative modelling to interesting mean field game tasks.
- The contributions are simple but effective, adding a mean field term to SB and deriving a new loss based on temporal difference

Weaknesses
- Line 159 is not very clear, the objective of FB-SDE is only a log-likelihood at convergence and appears to only be justified by its equivalence to IPF 
- Is the statement that IPF methods will not work with MF interaction true for the the regression objective of [1]?, between line 164 and 165
- Experiments appear relatively simple, does the method scale to more complex examples?
- It does not appear that the code is available, however the authors have stated that the code is available in the checklist. Have I missed the link?
- Some experimental details such as number of IPF iterations, diffusion steps, training details are missing.
- Not clear how to tune training procedure in FBSDE approach. 
    - If number of training steps per IPF iteration, denoted K in Algo 1, is very small then this will just map noise to noise per IPF iteration hence not solve SB and surely cannot work?
    - If like in the original FBSDE SB paper [2] the forward/ backward networks are trained for a large number of training iterations in the first IPF step then fine-tuned by a small number of steps in later IPF steps, do they get stuck in local minima? is fine tuning doing much or is this essentially the same as score based generative modelling? If fine tuning and not following IPF by training each network to completion at each IPF step, is this actually approximating Optimal Transport? There appears to be no empirical evidence this is close to OT
   
Minor
- Line 4 typo, "preferences needs not available" does not make sense
- Capital "L" needed for Laplacian operator, line 37
- Typo line 73, "need not be continuous" is fine
- Line 203, "testify" seems to be the wrong word here

[1] Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling, Bortoli 2021
[2] LIKELIHOOD TRAINING OF SCHRÖDINGER BRIDGE USING FORWARD-BACKWARD SDES THEORY, 2022
questions: - Is the code available?
- Is there a reason to use the FB-SDE SB approach over the simpler original DSB approach given they have the same objective?
- How many IPF iterations were used? And how many training steps, K, per IPF iteration?
- What is the run time/ training time?
- How is $\hat{TD}_0$ computed if it requires the log of the target density?, line 182

limitations: The authors have not discussed limitations. Given the iterative nature of the procedure, similar to other SB approaches, it appears difficult to scale. Experiments are relatively toy-ish.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed the DeepGSB, a framework for solving (a variant of) mean-field games including non-differentiable interaction terms. The key idea is to replace the usual terminal condition on the control by a hard constraint on the target distribution, and then invoke the Schrodinger Bridge (SB) framework.The authors provide promising empirical evidence for the proposed method.
strengths_and_weaknesses: Strength:

- This paper is well-written, with ideas clearly presented.
- The proposed model is novel and interesting.
- The connection to TD-like objectives is intuitive and may potentially be useful in practice.


Weakness:
- The motivation for replacing the terminal control constraint with the hard distributional constraint is not very convincing to me. Under what circumstance does such a scenario arise? To me, it seems like for most applications, the target distribution is almost never available. In the experimental section, the authors simply assume that the target density is some simple distributions (such as Gaussians), for which I fail to grasp the reason why this is reasonable.
questions: My two major questions are:

1. Is there any chance to relax the hard constraint on the target density? For me, this seems quite hard as the SB is mostly expressed in the two fixed end-points formulas.

2. Is there any practical motivation for considering the hard distributional constraint formulations of MFGs? 
limitations: As mentioned in my review, although this paper aims at solving MFGs, its proposed variant does not seem particularly interesting to me (although the technique is elegant). I'm willing to raise my score if the authors can convince me of the value of their setting.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors of this work propose and explore extensively a unique and elegant generalization of SBPs to Mckean-Vlasov type of SDEs / MFGs. Whilst these types of systems have already been proposed and explored in the literature, the connection to SBPs and the reinterpretation as a generalization is novel and elegant. One of the contributions of this work is that leveraging some new interpretations the authors propose new ML aided numerical algorithms to solve these systems and provide extensive experimental evidence showing their success and some initial theoretical motivation. 
strengths_and_weaknesses: Overall there are issues issues in the way the paper is written and how some previous ideas / methods are underplayed (and arguably innacurately portrayed). I believe this was not the author's intention as they merely seeked to showcase / emphasize their contribution however I do believe it has a negative rather than a positive effect when reading the paper with the potential of confusing the reader. Furthermore there is a gap connecting the practical optimisation scheme used to the central theoretical results in the paper (main), whilst the connection may be in the appendix or possible to infer it is not clearly outlined. 

Pros: 
1. Sufficient and Extensive successful experimental criteria
2. Clear and accessible lit review of SBPs and MFGs
3. Cleanly presented and readable results/theorems 

Cons:
1. There is a gap between Theorem 1 and the training algorithm which makes it difficult for a reader to motivate the origin of the algorithm
2. Lack of clarity when connecting (or arguing significant differences) to IPF
3. Overall arguments in the paper are not well connected making it difficult for a reader to assert the validity of the final approach in particular bridging the theory and the practice.
4. As a result there seems to be no convergence proofs or guarantees for algorithm 1 , its somewhat heuristically motivated by IPF and Theorem 1 but there is no overarching argument establishing / justifying its convergence. 

questions: Questions / Corrections (suggestions):

1. The claim that IPF in all generality does not suffice based on a naive marginal based iteration of IPF specifically in termed “likelihood training objectives” proposed by [27] seem like a major overstatement and not a rigorous one either. At the very best the authors should state that the IPF flavor proposed in [27] is not suitable for the generalized SB since that is what they motivate. In the most general setting as a coordinate ascent algorithm IPF and Sinkhorn probably still have theoretical guarantees and could potentially admit practical iterations for this problem, the authors have not proved otherwise and should not state otherwise. It's not helpful for the reader to inflate claims in this manner, especially without proof (you do provide an argument but that proves something much more specific).  This is my main counterpoint. If the authors are happy to clarify and state this more humbly then I believe we have an excellent paper (there is no need for this really … ). 
2. Despite making criticisms towards a specific instance of IPF (which is only nicely satisfied / justified when Y,Y_hat are at equilibrium rather than approximately parameterized) the final form of the proposed algorithm is very IPF-esq, and whilst the differences are outlined as they are developed it feels they are not clearly emphasized enough. And as mentioned in the previous point its not necessarily proved that the alternating objective provided can not be re-written or re-interpreted as alternating path KLs with constraints. There is a gap here in the comparison or relationship between the schemes and the small paragraph claiming IPF is not applicable falls very short. 
3. Line 140 the notation is not particularly nice. I would advise subscripting Y and Y_hat with time indices as done in [27] and in the Appendix.
4. App A.2 Eq 26, whilst their are prior results which have not being acknowledged (eg [2] Equation 16 and Theorem 5 or [3] Theorem 3) trading off scores with divs in the context of diffusions, it is not clear how the cross terms involving the score q vanish after applying Green’s identity, its clear how divergences arise on Z_theta, but terms involving  q and Z_theta/phi typically would require further identities/steps (i.e. using the FPK equation) to substitute div(f) , this is missing a lot of clarity and steps. Whilst I know this result to be true I find the sketch incomplete and very uninformative.  
5. FBSDE Theory asserts conditions at which some particular system of Ito Processes satisfies certain HJB equations. Typically reaching said conditions numerically (see [1]) is achieved by minimizing some MSE styled error whose fixed point trivially matches the conditions in Theorem 1 , the authors fail to detail how they go from Theorem 1  to the objectives L(theta) and L(phi).
6. Overall this paper is lacking an additional proposition which proves that a combined objective function L(\phi, \theta) is minimized when the conditions of theorem 1 are satisfied, whilst this is informally discussed and partially derived/motivated  it is not formally stated or argued anywhere. Then after this a further lemma or remark asserting how a coordinate ascent scheme would lead from this objective to a fixed point satisfying the conditions in Theorem 1 would substantially improve the presentation and results.  Going through the appendix it does seem that this can be achieved without too much additional new work but with substantial change in the write up. It is not the job of the reviewer/reader to infer (and basically have to prove) this from partial information and semi-connected statements.  
7. To maybe emphasize further the previous point there are no convergence guarantees proved for algorithm 1 the way its sketched / presented comes across as mostly heuristic with some components derived but without any overall guarantee presented.



[1] Nüsken, N. and Richter, L., 2021. Solving high-dimensional Hamilton–Jacobi–Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space. Partial Differential Equations and Applications, 2(4), pp.1-48.

[2] Huang, C.W., Lim, J.H. and Courville, A.C., 2021. A variational perspective on diffusion-based generative models and score matching. Advances in Neural Information Processing Systems, 34, pp.22863-22876.

[3] Song, Y., Durkan, C., Murray, I. and Ermon, S., 2021. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34, pp.1415-1428.

[27] Chen, T., Liu, G.H. and Theodorou, E.A., 2021. Likelihood Training of Schr\" odinger Bridge using Forward-Backward SDEs Theory. arXiv preprint arXiv:2110.11291.

limitations: As detailed in the suggestions / questions the paper contains incomplete proofs, and disconnected statements. Overall the text is not well connected and the reviewer feels it is not well written for a machine learning auidence, even one specialising in the relevant subfields covered in this work. Thus due to the lack of clarity and verifiability I cannot recommend this work for publication in its current form, it feels that the author expects readers to complete and guess many intermediate and missing steps in proofs to the point where in some cases the overall desired statements you would expect for a new numerical scheme are not actually proved or even stated.  Overall the way in which the theory is presented does not prove/outline overall convergence guarantees for the final numerical scheme, the paper requires significant re-writing and its overall presentation in order to be clear/accesible.
review_title: null

Meta-review: metareview: This paper proposes a novel, simple but effective algorithm to solve Mean Field Games. Reviewers found the paper well written, presenting an exact and flexible method. Despite its simplicity, the method solves a wide class of MFGs. Authors were also able to demonstrate the computational advantage of their method, providing good experimental data.
confidence: Certain
Decision: Accept