Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper applies the well-known idea of meta learning to learning in games, across various game types and settings including zero-sum games, potential games, general-sum games and Stackelberg games. The authors show several theoretical results for convergence to relevant performance metrics in these games, depending on natural notions of similarity between games encountered by the learner. In general, meta-learning serves to improve the performance of standard algorithms, recovering the known guarantees in static games. The authors also experiment with meta learning poker (zero-sum) endgames, with meta learning used to determine appropriate initializations for the learners. The experiments show that the technique greatly improves performance compared to vanilla methods.

strength_and_weaknesses: Strengths:
- This paper is very thorough in its exposition of related work and in extending the results in each of the settings studied. It is very ambitious to analyze so many different game types and settings, but I feel that the authors managed to do so in a way which still feels digestible and natural.
-  As far as I can tell, despite the breadth of the paper's analyses, the theoretical results are presented clearly and the proofs (while typically derivative of the standard techniques in the field) contain some non-trivial ideas which would certainly be of independent interest in static game settings. A primary example of this is the analysis of the extragradient algorithm using a regret-proxy, allowing for an RVU-type bound to be written even though extragradient is not a no-regret algorithm.

Weaknesses:
- A somewhat minor weakness of the paper is that a lot of the results seem unsurprising, in the sense that by selecting natural similarity metrics and having agents learn initializations based on a 'stacking'-type setup with no-regret algorithms, one would certainly expect the performance to improve over time. It is of course still useful and interesting to formally analyze the performance improvements, but I would have liked to see more focus on the empirical results.
- The experimental results in the paper are quite restricted, in the sense that despite a broad range of theoretical results, the authors only show experimental improvements for zero-sum games. It would have been interesting to see a visual representation of the improvements in potential, general-sum and even stackelberg games using meta-learning. 
- Finally, the choice of similarity metric in each setting, while usually appropriately motivated, leaves some room open for debate. In particular it would have been interesting to empirically compare different similarity metrics in settings where multiple metrics can be devised. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the framework for meta learning in games is clear and easy to understand. Moreover the technical results are all well-argued for and overall the paper has a nice flow to it. I feel the paper is less novel in the sense that it applies a well-known technique in machine learning to learning in games, and the resulting analyses, while interesting, are mostly extensions of known results. Overall, however, the central idea is well motivated and potentially practically useful for future work.

summary_of_the_review: In summary, this paper introduces a framework for meta learning in games which seems to be a useful and interesting idea to me. Many theoretical results in different game settings are described, all of which improve upon vanilla bounds and some new techniques are even shown in the supplementary material. The paper is well written enough to not feel bloated, but there is a lack of empirical evidence for the meta learning techniques in games beyond zero-sum. Overall I would recommend the paper for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces meta-learning for equilibrium finding and learning to play games including two-player zero-sum games, general-sum
games, and Stackelberg games. The authors define somewhat natural notions of similarity between the sequence of games encountered. The work evaluates
the meta-learning algorithms on endgames faced by the poker agent Libratus. The experiments show that related games can be solved significantly faster using the meta-learning techniques than by solving them separately.
strength_and_weaknesses: 

I think this is very good paper. I really have nothing to complain about about the results, the results also are a big enough advance.
The only issue I have is with the writing - I understand that this is a difficult paper to write, in fact I feel this paper is better suited for a journal than a venue which is so much page constrained. I found the informal theorem statements quite very informal, and the formal statement corresponding  to the theorem in the appendix requires tracking a few other theorems in the appendix. I think the appendix is more interesting than the main paper.

Examples of too informal: 
In Thm 3.1 it is stated "meta-learning algorithm for the initialization" - which algorithm? - later in the text below it is stated as Alg 1 but it is confusing to state something without introducing it first.
Thm 3.2 does not mention anything about meta-learner (or even the init strategy in the text before the Thm)
In definition of smooth games, OPT is not specified
clarity,_quality,_novelty_and_reproducibility: The paper is novel and of good quality.
The writing in the main part of the paper is confusing in parts.
summary_of_the_review: Solid paper, main paper can be written more clearly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies a framework on meta-learning in multi-agent games and considered a range of popular game structures, including zero-sum games, potential games, general-sum multi-player games, and Stackelberg security games. The general game is composed by  a sequence of multiple sub-games, where each sub-games have a number of iterations.

The authors analyzed theoretically the rates of convergence to different game equilibria which is dependent on the similarity between the sequence of sub-games, when each of the agents is performing some type of gradient descent updates. Experimentally, the game dynamics were demonstrated with two public Poker endgames.
strength_and_weaknesses: Strength:
- I found the meta-learning in game perspective interesting, and the analysis and results in the paper directly lead to several problems for future works, such as more general game structures where the single-game results are known but the meta-game analysis remains open.
- The authors were able to consider a set of common game structures and provide a comprehensive set of results. The experiments on the Poker endgames also nicely complement the theoretical analysis.
- Overall the paper is well-written and the structure was very clear to follow. 

Weakness:
- in the theoretical guarantees the task similarity notion significantly affects the convergence rate. It would be good to see more interpretation on the "task similarity" notion. What are some examples of sequences of games that exhibit a high/low task similarity?
- In the experiments, how does the proposed algorithm compare with the one in [Zhang et al 2022b]? Given that their solution is for a more general setup that applies in the paper's setting, it would be good to see such comparisons.
clarity,_quality,_novelty_and_reproducibility: Overall the paper is clearly written and clear to follow. The theoretical analysis and implementation details are both provided. I was not able to find the implementation code.
summary_of_the_review: Overall the paper studies a new framework for meta-learning in several common game structures. The results cover a nice set of game settings and open up a few problems for future works. I suggest the authors to add further interpretations on the theoretical bounds, and experimental comparison with one of the prior works ([Zhang et al 2022b].




-------- post rebuttal --------

I have read the author's response and remain the original score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes and investigates the problem of meta-learning in games. In particular, it builds the framework that uses meta-learning techniques to learn Nash equilibrium sequentially for a set of games by exploiting similarities between these games. Theoretical guarantees under this framework are derived for many classes of games and problems including two-player zero-sum games, potential games, optimal social welfare problems and stackelberg games. It also provides experiments on two Libratus endgames and shows supreme performance of the proposed meta-learning algorithms.
strength_and_weaknesses: ### Strengths
The results provided in this paper is very extensive, showing that the proposed framework can be readily applied to many classes of meta-learning problems related to games. Meanwhile, these results also provide connections to many different directions, opening an area of future research.

### Weaknesses

- One major weakness of this paper in my concern is in its organization. Since the coverage is too extensive, the current main content is more like a list of results. That is, it looks very difficult to get the core idea of the proposed algorithm by solely reading the main content. If my understanding is correct, most of the theoretical results are derived based on Theorem B.3. Therefore, it's probably better to put more discussion about Theorem B.3 and less discussion about specific problems into the main content. Otherwise, considering **submitting this paper to a journal** can also be an alternative choice.
- Another major weakness of this paper in my concern is in its assumptions. In particular, in this paper, the algorithm requires exact Nash equilibrium about previous games, while previous meta-learning algorithms usually don't require so much information about previous tasks (like the "Online Meta-learning"). Is there any unique properties in games making this amount of information necessary? Meanwhile, why this is a well-motivated assumptions in some applications, which seems not to be discussed in appendix?
- Meanwhile, from my perspective, the setup in this paper is more close to "online meta-learning" instead of "meta-learning", so it's probably more appropriate to call it "Online Meta-learning in Games".
clarity,_quality,_novelty_and_reproducibility: In general, the proposed framework is considered to be novel. However, the clarity of the paper may need some improvement. In particular, as mentioned in the Weakness part, it's difficult to get the core idea of the algorithm by solely reading the main content. Meanwhile, the background for many problems it addresses can only be provided in appendix. One potential resolution is to have less discussion about results in specific problems. Another potential resolution is to consider **submitting this paper to a journal**, which has much less restrictions on the length of the main content.

### Questions
- What is the definition of average potential game (or average zero-sum game) in Theorem 3.4 (or in Theorem 3.2)? Based on the proof in appendix, it seems what actually happens is that policy for some game at some iteration reaches $O(\epsilon)$-approximate Nash equilibrium.
- It is not very clear what is the "Meta component" in the proposed algorithm, which I refer to something similar to $\mathbf{w}\_{\mathrm{MAML}}$ in "Online Meta-learning". In particular, $\mathbf{w}_{\mathrm{MAML}}$ captures the high-level similarities among different tasks. Is there anything in the algorithm or analysis that captures the similarities among different games?
- Is it possible to meta-learn these games in a batch fashion like MAML instead of an online fashion?
summary_of_the_review: This paper proposes a novel framework to study meta-learning in games and provides extensive results in many classes of problems. However, some assumptions need to be justified and it looks inherently difficult to put everything it covers into the limited number of pages in a clear way.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper works in the meta-learning twist into the online learning in games literature. The main premise is not surprising: If you have a class of similar games, learning from observed games somehow helps with the convergence speed of a new one. 

While the writing is a bit informal at times, the overall improvements in practice are significant to warrant publication. The background summary is quite well-done, though there are some additional recent work on weak MVIs that the authors might want to include in camera ready. 

While the title is catchy, the paper is making a slight over-reach into the meta-learning field, which covers much more than what the authors use in their paper. 
justification_for_why_not_higher_score: The meat of the paper is in the supplementary material. Hence, the paper itself could benefit from a major rewriting to improve clarity. At the same time, while the improvement on the numerical example is quite convincing, there are a few overstatements at times, in particular a comparison to [Zhang et al 2022b], which covers the theoretical setting here, is avoided carefully with theoretical justifications. Moreover, the meta-learning aspects are missing (i.e., standard meta-learning aspects are missing).   
justification_for_why_not_lower_score: While the results are stated a bit informally, and there is a bit of overreach, the paper still has a solid contribution with a great background review.
Decision: Accept: poster