Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies Stackelberg equilibria and presents a general framework for implementing Stackelberg equilibria search as a multi-agent RL problem, which encapsulates both previous approaches and some new design spaces such as contextual policies. Some numerical experiments are also provided.
strength_and_weaknesses: Strengths:
* Propose a general framework for implementing Stackelberg equilibria search as a multi-agent RL problem.
* Numerical experiments are provided to demonstrate the usefulness of the proposed framework.

Weaknesses:
* The notation is confusing and inconsistent. For example, in Definition 2, it is unclear how these notation are related to the Markov Game notation in the previous page. Also, it seems that the authors are confusing strategies $s$ with actions $a$. In particular, if ${\bf s}_F$ is the follower strategies, then what is ${\bf a}_F$? Similarly, if $s_L$ is the leader strategy, then what are $s_t$ and $a_L$?
* Theorem 1 is either trivial or very hard to follow, making the claim about a general framework for implementing Stackelberg equilibria search as a multi-agent RL problem unconvincing. The general case seems to be a simple restatement of the problem formulation, and the proof in the appendix also seems to suggest this. The special case is hard to understand. and also $s_L(o)$ and $\mathcal{O}_L$ are undefined. The authors should try to make these statements and proofs more formal, instead of using informal statements like "acting the same during queries as they would in $\mathcal{M}$". 
* The authors seem to only compare with Balaguer et al. (2022) as the benchmark. However, there are tons of Stackelberg RL algorithms that the authors fail to compare with. See [A, B] for example.

[A] Tianmin Shu and Yuandong Tian. M3RL: Mind-aware multi-agent management reinforcement learning. In ICLR, 2019.

[B] Runsheng Yu, Xinrun Wang, Youzhi Zhang, Rundong Wang, Bo An, ZhenYu Shi, and Hanjiang Lai. Learning expensive coordination: An event-based deep RL approach. In ICLR, 2020.
clarity,_quality,_novelty_and_reproducibility: The writing is hard to follow, and neither the theoretical nor the numerical results are convincing. The reproducibility is okay given the provided codes.
summary_of_the_review: Given the comments above, I think the paper is not qualified for publication in ICLR due to the drawbacks in both writing quality and solidity of the results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a framework for convergence to Stackelberg equilibria in multi-agent RL. The main contribution of the paper can be summarised by Theorem 1 in the paper. Theorem 1 states 3 conditions for convergence so that the solutions for the leader and the follower form a Stackelberg equilibrium. Additionally, 2 more conditions are stated that are required to consider the problem of the leader as a POMDP. Finally, the authors experimentally evaluate when Theorem 1 holds in several matrix games and scenarios.
strength_and_weaknesses: Strengths:
- The paper is clearly written, and the concepts are nicely described.
- The background information is described concisely.

------------------------------------------------------------
Weaknesses

- Page 2, last paragraph: "We focus here mainly on this memory-less case." Why do you only consider memory-less cases?
- Page 3, first line: "total reward" ---> this should be "total return"
- The paper is heavily based on three works of Brero et al, which however only one of those is published in an archival venue.

- I would recommend including the payoff matrices of the games in the appendix.

- Page 6, 3rd paragraph: Could you please elaborate more on this sentence: "The leader should receive reward only when followers are playing their best-response equilibrium, and the entire follower learning process should be one episode for the leader." Why do we need such a constraint? 

- There are small typos and incorrect use of articles in the paper, which, however, do not make the paper hard to read. For example, on page 8 in the last paragraph: alllow ---> allow

- I would like to see the baselines of Balaguer et al. (2022) included in Figures 1 and 2. Otherwise, the reader has to go to the paper of Balaguer et al. (2022) to see the achieved returns of the baselines.

- I feel that ICLR is not a suitable venue for this paper. ICLR, as its name indicates, focuses on learning representations, and I do not see how this paper fits in this scope. I think conferences such as AAMAS/AAAI/IJCAI/ICML/NeurIPS/COLT are more suitable for this paper.

- Why are the matrix games POMDPs?

clarity,_quality,_novelty_and_reproducibility: **Clarity**: The paper is very well-written. Every piece of background is clearly and concisely described. 

**Novelty**: I believe that the concepts that are mentioned are not very novel. The main idea behind the paper is to propose a unified framework and show that it converges to Stackelberg equilibria. However, I think that the ideas are 

**Reproducibility**: The source code to reproduce the experiments is provided in the supplementary materials.
summary_of_the_review: I recommend a weak reject for this paper. I do not feel that it significantly alters our understanding of Stackelberg equilibria. Additionally, I believe that ICLR is not a suitable venue for this paper. I would expect higher dimensional experiments.
My ranking reflects my current understanding of the paper. I would be happy to change my score if the authors satisfactorily refute my main points of criticism.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper combines recent multi-agent RL approaches for finding stackelberg equilibria into one framework, with conditions on convergence. Also, ideas from meta-RL are used to improve the follower. 

strength_and_weaknesses: This paper provides a nice overview of the existing literature for solving sequential Stackelberg games. I especially like table 2. I think that a version of this paper that does a better job experimentally comparing against all the existing approaches and with better theory characterizing when one would expect approaches to not work could be useful to the community. 
However, as it currently is, the theory seems to give a result that is already known, the experiments do not sufficiently compare across all the baselines, and the novel algorithmic contribution does not have good enough experiments to back it up.

The experiments should be a lot more extensive and compare against all the related works shown in the table

clarity,_quality,_novelty_and_reproducibility: The conclusions from theorem 1 seem too strong. Just because some method doesn’t satisfy the conditions of the theorem doesn’t mean it isn’t principled. I could see a lot of approaches working where the leader updates while the follower learns
Theorem 1 also seems pretty obvious and already known from the literature. I think the work should be pretty reproducible. I thought the writing was clear and concise. 
summary_of_the_review: Overall, I think this paper needs a bit more work, with better theory that proves existing methods won't work, with extensive experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper deals with the question of learning Stackelberg equilibria -- games with a leader-follower structure -- an important topic with many relevant applications. The main contribution is to show that, given access to a best-response oracle for the follower, this problem can be reduced to learning in a POMDP consisting of two phases. In some simple games, the authors then implement this reduction experimentally. A crucial requirement is that in the first phase of the POMDP, the leader does not adapt their policy; the authors also experimentally test this point.
strength_and_weaknesses: Strengths: The paper presents a very clear paradigm which unifies many existing approaches. This is an important topic, as shown by the cited prior work that involves using RL-style approaches in Stackelberg games, but much prior work has somehow lacked the clarity of perspective which this paper provides.

Weaknesses: the main theorem is not very exciting in some sense. And the experiments are not particularly impressive (they mainly look at very toy matrix games, when there are many more interesting problems that in principle can be described by this framework).
clarity,_quality,_novelty_and_reproducibility: Clarity: the work is well-written and easy to follow.

Novelty: in some sense, the work is novel, as it presents an original viewpoint, but it also is just building on/generalizing previous work e.g. the cited Brero et al. paper.

Reproducibility: reproducibility seems fine, the code included looks after looking through it briefly.

Quality: the work is high quality although the main theorem and experiments, while they are enough to support the argument, are not too impressive.
summary_of_the_review: I recommend weak acceptance simply because in the end, I was glad to read this paper and think it is a positive contribution to the community.

Update after reading author's rebuttal: due to the presence of new experiments (and clarification of comparison to previous methods) which are indeed more convincing, I've increased my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the problem of computing a Stackelberg equilibrium and proves the conditions for this problem to be formulated as a partially observable Markov decision process (POMDP).  The main theorem essentially states that the leader policy at a Stackelberg equilibrium is the optimal policy for a POMDP given the access to the best response of followers.  Motivated by this theorem, the paper proposes a reinforcement learning method for computing the Stackelberg equilibrium, where the key difference from the prior work is that the proposed approach ignores the samples when the followers are not best responding.  The effectiveness of the proposed approach is empirically validated with a small normal form game and a strategic game of a modified Atari environment.

The strength of the paper is in the general framework that unifies existing approaches to computing Stackelberg equilibria.  However, it is unclear what new insights that Theorem 1 gives beyond the state-of-the-art knowledge about the Stackelberg equilibrium.  Since the general case directly follows from the definition, what is essential is the Query-Oracle Special Case. While it has been improved during discussion, further clarifications and details are needed to precisely understand the implications of the theorem.  

* Following is a personal opinion of the area chair:

Reviewer 671a suggested that "a constructive and explicit POMDP formulation should be stated in the main text", but I find that the POMDP formulation in the proof in the appendix is unclear.  This is partly because it is unclear what can be observed by each party.  I suggest writing the POMDP formulation in more detail and with more clarity.
justification_for_why_not_higher_score: The novelty and implications of the main theorem is unclear.
Decision: Reject