Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper considers the problem of representing and learning the behavior of a large population of agents, in an attempt to construct an effective predictive model of the behavior. The main concern is with large populations where it is not possible to represent each agent individually, hence the need to use a population level description.  The main contribution of the paper is in relating the theories of Mean Field Games (MFG) and Reinforcement Learning (RL) within the classic context of Markov Decision Processes (MDPs). The method suggested uses inverse RL to learn both the reward function and the forward dynamics of the MFG from data, and its effectiveness is demonstrated on social media data. 
The paper contributes along three lines, covering theory, algorithm and experiment.  The theoretical contribution begins by transforming a continuous time MFG formulation to a discrete time formulation (proposition 1), and then relates the MFG to an associated MDP problem. The first contribution seems rather straightforward and appears to have been done previously, while the second is interesting, yet simple to prove. However, Theorem 2 sets the stage for an algorithm developed in section 4 of the paper that suggests an RL solution to the MFG problem. The key insight here is that solving an optimization problem on an MDP of a single agent is equivalent to solving the inference problem of the (population-level) MFG. Practically, this leads to learning a reward function from demonstrations using a maximum likelihood approach, where the reward is represented using a deep neural network, and the policy is learned through an actor-critic algorithm, based on gradient descent with respect to the policy parameters. The algorithm provides an improvement over previous approaches limited to toy problems with artificially created reward functions. Finally, the approach is demonstrated on real-world social data with the aim of recovering the reward function and predicting the future trajectory. The results compare favorably with two baselines, vector auto-regression and recurrent neural networks. 
I have found the paper to be interesting, and, although I am not an expert in MFGs, novel and well-articulated. Moreover, it appears to hold promise for modeling social media in general. I would appreciate clarification on several issues which would improve the presentability of the results.  
1)	The authors discuss on p. 6 variance reduction techniques. I would appreciate a more complete description or, at least, a more precise reference than to a complete paper. 
2)	The experimental results use state that “Although the set of topics differ semantically each day, indexing topics in order of decreasing initial popularity suffices for identifying the topic sets across all days.” This statement is unclear to me and I would appreciate a more detailed explanation.  
3)	The authors make the following statement: “ … learning the MFG model required only the initial population distribution of each day in the training set, while VAR and RNN used the distributions over all hours of each day.” Please clarify the distinction and between the algorithms here. In general, details are missing about how the VAR and RNN were run. 
4)	The approach uses expert demonstration (line 7 in Algorithm 1). It was not clear to me how this is done in the experiment.

review_title:  The paper relates the theories of Mean Field Games and Reinforcement Learning  within the classic context of Markov Decision Processes. The method suggested uses inverse RL to learn both the reward function and the forward dynamics of the MFG from data, and its effectiveness is demonstrated on social media data. 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper attacks an important problems with an interesting and promising methodology.  The authors deal with inference in models of collective behavior, specifically at how to infer the parameters of a mean field game representation of collective behavior. The technique the authors innovate is to specify a mean field game as a model, and then use inverse reinforcement learning to learn the reward functions of agents in the mean field game.

This work has many virtues, and could be an impactful piece. There is still minimal work at the intersection of machine learning and collective behavior, and this paper could help to stimulate the growth of that intersection.  The application to collective behavior could be an interesting novel application to many in machine learning, and conversely the inference techniques that are innovated should be novel to many researchers in collective behavior.

At the same time, the scientific content of the work has critical conceptual flaws.  Most fundamentally, the authors appear to implicitly center their work around highly controversial claims about the ontological status of group optimization, without the careful justification necessary to make this kind of argument.  In addition to that, the authors appear to implicitly assume that utility function inference can be used for causal inference. 

That is, there are two distinct mistakes the authors make in their scientific claims:
1) The authors write as if mean field games represent population optimization (Mean field games are not about what a _group_ optimizes; they are about what _individuals_ optimize, and this individual optimization leads to certain patterns in collective behaviors)
2) The authors write as if utility/reward function inference alone can provide causal understanding of collective or individual behavior

1 - 

I should say that I am highly sympathetic to the claim that many types of collective behavior can be viewed as optimizing some kind of objective function.  However, this claim is far from mainstream, and is in fact highly contested.  For instance, many prominent pieces of work in the study of collective behavior have highlighted its irrational aspects, from the madness of crowds to herding in financial markets.

Since it is so fringe to attribute causal agency to groups, let alone optimal agency, in the remainder of my review I will give the authors the benefit of the doubt and assume when they say things like "population behavior may be optimal", they mean "the behavior of individuals within a population may be optimal".  If the authors do mean to say this, they should be more careful about their language use in this regard (individuals are the actors, not populations).  If the authors do indeed mean to attribute causal agency to groups (as suggested in their MDP representation), they will run into all the criticisms I would have about an individual-level analysis and more.  Suffice it to say, mean field games themselves don't make claims about aggregate-level optimization.  A Nash equilibrium achieves a balance between individual-level reward functions. These reward functions are only interpretable at the individual level.  There is no objective function the group itself in aggregate is optimizing in mean field games.  For instance, even though the mean field game model of the Mexican wave produces wave solutions, the model is premised on people having individual utility functions that lead to emergent wave behavior.  The model does not have the representational capacity to explain that people actually intend to create the emergent behavior of a wave (even though in this case they do).  Furthermore, the fact that mean field games aggregate to a single-agent MDP does not imply that that the group can rightfully be thought of as an agent optimizing the reward function, because there is an exact correspondence between the rewards of the individual agents in the MFG and of the aggregate agent in the MDP by construction.

2 -

The authors also claim that their inference methods can help explain why people choose to talk about certain topics. As far as the extent to which utility / reward function inference can provide causal explanations of individual (or collective) behavior, the argument that is invariably brought against a claim of optimization is that almost any behavior can be explained as optimal post-hoc with enough degrees of freedom in the utiliy function of the behavioral model. Since optimization frameworks are so flexible, they have little explanatory power and are hard to falsify.  In fact, there is literally no way that the modeling framework of the authors even affords the possibility that individual/collective behavior is not optimal.  Optimality is taken as an assumption that allows the authors to infer what reward function is being optimized. 

The authors state that the reward function they infer helps to interpret collective behavior because it reveals what people are optimizing.  However, the reward function actually discovered is not interpretable at all. It is simply a summary of the statistical properties of changes in popularity of the topics of conversation in the Twitter data the authors' study. To quote the authors' insights: "The learned reward function reveals that a real social media population favors states characterized by a highly non-uniform distribution with negative mass gradient in decreasing order of topic popularity, as well as transitions that increase this distribution imbalance."  The authors might as well have simply visualized the topic popularities and changes in popularities to arrive at such an insight. To take the authors claims literally, we would say that people have an intrinsic preference for everyone to arbitrarily be talking about the same thing, regardless of the content or relevance of that topic.  To draw an analogy, this is like observing that on some days everybody on the street is carrying open umbrellas and on other days not, and inferring that the people on the street have a preference for everyone having their umbrellas open together (and the model would then predict that if one person opens an umbrella on a sunny day, everybody else will too).

To the authors credit, they do make a brief attempt to present empirical evidence for their optimization view, stating succinctly: "The high prediction accuracy of the learned policy provides evidence that real population behavior can be understood and modeled as the result of an emergent population-level optimization with respect to a reward function." Needless to say, this one-sentence argument for a highly controversial scientific claims falls flat on closer inspection. Setting aside the issues of correlation versus causation, predictive accuracy does not in and of itself provide scientific plausibility. When an n-gram model produces text that is in the style of a particular writer, we do not conclude that the writer must have been composing based on the n-gram's generative mechanism.  Predictive accuracy only provides evidence when combined in the first place with scientific plausibility through other avenues of evidence.

The authors could attempt to address these issues by making what is called an "as-if" argument, but it's not even clear such an argument could work here in general.  

With all this in mind, it would be more instructive to show that the inference method the authors introduce could infer the correct utility functions used in standard mean field games, such as modeling traffic congestion and the Mexican wave.  

--

All that said, the general approach taken in the authors' work is highly promising, and there are many fruitful directions I would be exicted to see this work taken --- e.g., combining endogenous and exogenous rewards or looking at more complex applications.  As a technical contribution, the paper is wonderful, and I would enthusiastically support acceptance.  The authors simply either need to be much more careful with the scientific claims about collective behavior they make, or limit the scope of the contribution of the paper to be modeling / inference  in the area of collective behavior.  Mean field games are an important class of models in collective behavior, and being able to infer their parameters is a nice step forward purely due to the importance of that class of games.  Identifying where the authors' inference method could be applied to draw valid scientific conclusions about collective behavior could then be an avenue for future work.  Examples of plausible scientific applications might include parameter inference in settings where mean field games are already typically applied in order to improve the fit of those models or to learn about trade-offs people make in their utility functions in those settings.

--

Other minor comments:
- (Introduction) It is not clear at all how the Arab Spring, Black Lives Matter, and fake news are similar --- i.e., whether a single model could provide insight into these highly heterogeneous events --- nor is it clear what end the authors hope to achieve by modeling them --- the ethics of modeling protests in a field crowded with powerful institutional actors is worth carefully considering.
- If I understand correctly, the fact that the authors assume a factored reward function seems limiting. Isn't the major benefit of game theory it's ability to accommodate utility functions that depend on the actions of others?
- The authors state that one of their essential insights is that "solving the optimization problem of a single-agent MDP is equivalent to solving the inference problem of an MFG." This statement feels a bit too cute at the expense of clarity. The authors perform inference via inverse-RL, so it's more clear to say the authors are attempting to use statistical inference to figure out what is being optimized.
- The relationship between MFGs and a single-agent MDP is nice and a fine observation, but not as surprising as the authors frame it as. Any multiagent MDP can be naively represented as a single-agent MDP where the agent has control over the entire population, and we already know that stochastic games are closely related to MDPs.  It's therefore hard to imagine that there woudn't be some sort of correspondence. 
review_title: The technical contributions are good, but the scientific claims should be revised to avoid making unjustified controversial claims

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: The paper proposes a novel approach on estimating the parameters  
of Mean field games (MFG). The key of the method is a reduction of the unknown parameter MFG to an  unknown parameter Markov Decision Process (MDP).

This is an important class of models and I recommend the acceptance of the paper.

I think that the general discussion about the collective behavior application should be more carefully presented and some better examples of applications should be easy to provide.  In addition the authors may want to enrich their literature review and give references to alternative work on unknown MDP estimation methods cf. [1], [2] below. 

[1] Burnetas, A. N., & Katehakis, M. N. (1997). Optimal adaptive policies for Markov decision processes. Mathematics of Operations Research, 22(1), 222-255.

[2] Budhiraja, A., Liu, X., & Shwartz, A. (2012). Action time sharing policies for ergodic control of Markov chains. SIAM Journal on Control and Optimization, 50(1), 171-195.
review_title: Great paper

Meta-review: The reviewers are unanimous in finding the work in this paper highly novel and significant.  They have provided detailed discussions to back up this assessment.  The reviewer comments surprisingly included a critique that  "the scientific content of the work has critical conceptual flaws" (!)  However, the author rebuttal persuaded the reviewers that the concerns were largely addressed.
Decision: Accept (Oral)