reviews: 
Comments: Recent works in multi-agent learning focus on reasoning about inter-agent interactions. One such algorithm is Learning with Opponent Learning Awareness (LOLA), which looks ahead utilizing gradient descent. The work builds on the LOLA framework by casting the problem as a meta-game of optimization. Meta-Value Learning (MeVa) learns a meta-value function (expected discounted return over future iterates) and applies a form of Q-learning to evade representing the continuous action space. The inner loop corresponds to collecting a policy optimization trajectory using gradient descent on iterates of the meta-value. The outer loop corresponds to learning the meta-value function by minimizing the TD error. MeVa incorporates design considerations such as empirical corrections in Bellman updates, variable discounting and gaussian noise as exploration. Results in a logistic game and repeated matrix games demonstrate improvements over LOLA agents.
Strengths: * The paper is well-written and positioned within the learning with learning awareness literature.
* Authors have highlighted relevant design considerations and their motivation.
Weaknesses: * **Variance in Iterates:** My main concern is the variance in iterates of the optimization process. Using a correction-based formulation leads MeVa to estimate $\bar{\nabla} f(x)$ using REINFORCE which is known to have high variance in the policy gradient. Specifically, it is essential that the gradient of the iterates is consistent. It would be helpful if the authors can highlight any practical considerations to mitigate high variance in the gradients. Authors could also provide standard deviations of different trials for Tables 2 and 3 in order to evaluate the effect of noisy gradients (if any).
* **Ablations:** Appendix F provides a range of ablations for design considerations utilized in MeVa. However, an important comparison would be to assess the importance of the meta-value function formulation itself. Authors should compare the surrogate formulation of Eq. 6 with the naive infinitely discounted sum of Eq. 5. Similarly, the choice of a sophisticated exploration strategy, which is Gaussian noise, could be evaluated by comparing it with standard exploration schedules used for TD learning such as $\epsilon$-greedy exploration or action noise. Current ablations only enable/disable exploration which do not reason about the choice of schemes and their importance.
* **Related Work:** The paper cites relevant works from multi-agent learning and opponent awareness literature. However, their organization within the paper could be improved. Authors could organize recent literature in a dedicated related works section or discuss the improvements of MeVa over prior methods (as done for LOLA in Introduction).

[1]. Foerster et al., "Learning with Opponent-Learning Awareness", AAMAS 2018.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good