Kati8/ICLR_2023_2025 huggingface dataset:
authors: Tim Cooijmans
authors: Milad Aghajohari
authors: Aaron Courville
reviews: 
Comments: The paper considers the task of opponent modeling in general sum games. The paper attempts to introduce a LOLA-based method that is able to judge policies over a longer horizon than one-step. The paper introduces a value-based method for a meta-game of optimization. This allows the method to avoid directly modeling the policy space updates for each player. Prior art in LOLA assumes that the opponent is a naive learner and uses a one-step lookahead. The paper implements a self-consistent version of LOLA that relaxes these assumptions. The paper evaluates on small matrix games and a logistic game in order to show how the method may help in scenarios similar to that which LOLA was evaluated in.
Strengths: Opponent modeling is an important domain of multi-agent learning. There is much potential for opponent modeling iff the community is able to create a scalable method. The idea of any LOLA-based paper is to relax some of the assumptions that restrict its scalability. The paper attempts to make a more scalable method through the use of a value-based method that does not rely on policy updates. However, the evaluation is restricted to simple domains.
Weaknesses: The biggest worry with the method is the reliance on the REINFORCE algorithm. The sample complexity required in order to perform this method may be incredibly high once scaled to sufficiently difficult action and state spaces. Though the authors make a note of the limit of this method in terms of learning with neural networks and suggest a direction to help scale their method in the future. This is also a general worry about the LOLA-based works. My question is, does the improvements in the paper lead LOLA-based methods closer to scaling to a feasible solution? If so, how?

Another question is how can this method scale beyond two-player multi-agent settings to general multi-agent scenarios. Even an understanding of how this methodology works in three-player games would be quite an interesting topic.

The results appear to show that M-FOS and MeVa are equivalent in performance. I am confused as to what the takeaway here is. Is it that the performance of MeVa is able to do this without the policy gradient? Why not use the policy gradient as those methods are more scalable?
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: This paper proposes an algorithm that allow agents to model learning processes in the extended future with a value function. This generalizes previous work, namely LOLA, which only models the myopic one-step ahead learning dynamics.
It is pointed out that there are previous works, and other generalizations of LOLA, that also learn meta-value functions.
Unlike other attempts at genearlizing LOLA, the claim is that this algorithm is "self-consistent" meaning that "the algorithm does not assume the opponent is naive and it is aware of its own learning process as well".
Results are shown on logistic and matrix games, showing that meta values outperform LOLA uniformly across the 4 environments and that meta-values tend to be competitive with M-FOS.
Unfortunately, head-to-head results between meta-values and M-FOS are not presented due to computational restrictions.

# Decision

While I like several things about this paper and find the proposed algorithm promising, I think the paper has too many issues to warrant acceptance. I am not sure whether these can be adequately addressed in a rebuttal, but I am tentatively rating the paper below the acceptance threshold.
Strengths: - An overall interesting idea that applies important ideas from meta-learning, single-agent reinforcement learning and multi-agent RL. It is also well motivated by the successes of LOLA and other algorithms that they build on. And the theoretical foundations, while not formally rigorous, are convincing.
Weaknesses: - The paper makes a few statements that are vague and which can be true, but require more specificity. Please see the detailed comments.
- Experimental evaluation seems limited from my point of view, but perhaps this is typical for multi-agent work? The policy spaces are small, and even in these small problems the authors allude to computational concerns for head-to-head results between M-FOS and meta-values. I would also like to see error bars on the performance. Overall, I consider this a weakness, but less so than the first weakness which is the primary factor in my decision.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The paper proposes the MeVa method designed for two-player zero-sum meta-games. By extending the concept and form of "looking ahead" methods, and incorporating a discounted sum over returns, it allows for an extrapolated approximation of the meta-value. Additionally, from a practical standpoint, MeVa employs $U(x)$ to approximate the extrapolated value, which helps avoid the detachment from reality often encountered with bootstrapped value functions. Experimentally, the study analyzes the method's behavior on a toy game and makes comparisons to previous work on repeated matrix games.
Strengths: * **Novelty**: This work innovatively extends the concept and form of "looking ahead" methods, while introducing a discounted sum over returns. This allows for an extrapolated approximation of the meta-value, sidestepping the need for approximating the gradient of the policy.
* **Presentations**: The paper presents its viewpoints and theoretical discussions in a lucid and straightforward manner, making it easy for readers to grasp its underlying premise and theoretical implications.
* **Experimental Analysis**: The study compares MeVa with methods like LOLA, HOLA, and COLA. In the meta-games, MeVa demonstrates superior performance.
* The paper fairly analyzes the limitations and prospects of the proposed method.
Weaknesses: * **Theoretical Guarantees**: Although the method leverages the concept of looking ahead, given its use of meta-value, I believe there should be some theoretical analysis regarding its convergence and computational complexity, etc. However, I did not find such discussions in the paper.
* **Experiments**: The experiments primarily compare against baseline methods that are not specifically designed for meta scenarios. I believe it would be more informative to design experiments comparing with other meta-specific methods beyond M-FOS, such as Meta-PG, meta-MAPG.
* **Reproducibility**: The source code is not submitted, making reproducibility uncertain.
* **The targeted scenarios are somewhat restrictive**:
    * The scope of the approach seems somewhat limited. As it's currently tailored for two-player zero-sum meta-games, it might be challenging to expand to tasks with more complex state and action spaces.
    * The paper assumes that one can observe the strategy parameters of the opponent. This assumption might be difficult to uphold in real-world tasks.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper proposes Meta-Value Learning (MeVa), a general framework for learning with learning opponent awareness in MARL. MeVa uses a meta-value method to account for longer-term behaviours of opponents and does not require policy gradients. It is consistent and far-sighted, avoiding the need to explicitly represent the continuous action space of policy updates. Many evaluations are conducted on various games, including the Logistic Game, Iterated Prisoner's Dilemma, Iterated Matching Pennies, and the Chicken Game, demonstrating MeVa's effectiveness in opponent shaping and cooperation for MARL. The method shows its merit in achieving cooperation where self-interest warrants it, without being exploitable by other self-interested agents.
Strengths: **Originality:**

MeVa is a novel method based on meta learning for dynamic opponent modelling in MARL. Unlike previous methods, MeVa is consistent, meaning it does not assume its opponent is naive and is aware of its own learning process. MeVa is far-sighted, i.e., it looks more than one step ahead through a discounted sum formulation, which allows it to account for longer-term and higher-order interactions.

**Quality:**

Overall, MeVa is a high-quality method. It extends previous LOLA and other methods with value learning, which is based on value learning and does not require policy gradients anywhere. MeVa can be applied to optimization problems with a single objective.

**Clarity:**

Overall, the writing is good. It introduces comprehensive background and related works, which is easy for readers to follow. 

**Significance:**

MeVa brings new insights into the MARL community, including its consistency on learning the full dynamics of the other agents, far-sightedness, value learning and implicit Q-function.
Weaknesses: 1. Scalability: The main weakness of MeVa is its scalability, particularly when learning in environments with more agents. The method may struggle to handle large parameter vectors and complex multi-agent interaction when more agents are involved.

2. Writing in the methodology section: This section can be improved by using similar notations from previous works, such as LOLA, Meta-PG and Meta-MAPG. It will make it consistency in notation and easy to follow.

3. Algorithm 1 is hard to follow. It would be great to add more explanations.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: I was asked to give a last minute review. I understand the authors will not have a chance to respond, which is not ideal; I hope to mostly bring up points that are touched on by existing reviews.

The authors introduce MeVa, a method which is a consistent and far-sighted method for opponent-shaping. MeVa works by applying Q-learning/DDPG-like method to the meta-game.
Strengths: Originality:

- Q-Learning-based methods have not been applied to the meta-game in opponent-shaping scenarios.

Quality:

- The experiments are thorough.

Clarity:

- The paper is clearly written

Significance:

- Opponent shaping is becoming increasingly important as more real-world AI systems are deployed
Weaknesses: - The paper makes heavy comparisons to meta-learning based methods, but notably completely omits them from the introduction (which is quite important for understanding the contributions of this paper). For example, when factoring in M-FOS's existence into the contributions and claims made in the Introduction, the main one is that MeVa is value-based and not policy-gradient-based. I'm not sure if this counts as a significant contribution. By omitting these methods from the introduction, the paper makes it seem like solving LOLA's inconsistency and myopia are novel contributions.

    - This is largely related to points brought up by Reviewers c4Mz and 1uYc.

- I don't understand the difference between "learning with opponent learning awareness" and "acting with opponent learning awareness" the authors mention in Section 2.3. The authors seem to be implying that learning = gradients. Under this distinction, "in-context learning" is "in-context acting", which I don't think most researchers would agree with.

    - This is related to points brought up by Reviewer VzC1

    - I'm assuming MeVa uses fast policy changes in IMP, which seems to defeat the whole point of the "gradual dynamics that are characteristic of learning".

I concede that these concerns could be alleviated with some simple discussion or minor modifications. However, I do not feel strongly positive enough about the paper to overrule the existing reviews.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 3 good
Clarity: 3 good