Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper explores the problem of mechanism design which studies how to design reward functions and environmental rules defining mathematical games. The applications of mechanism design spans across many domains from optimizing social welfare with economic policies to designing governmental policies. The conventional problem in this space is that it is often expensive to understand the effect of changes to a mechanism design in the real world. Thus, it is often convenient to study the mechanisms in simulations before deploying them in real world. 

The paper presents a deep RL approach to mechanism design that learns a world model and uses meta-learning to learn a mechanism that can be adapted quickly to unseen test agents. The approach called MERMADE consists of a planner that has an associated cost for intervening an agent and the goal of this planner is to achieve the designer’s objective. The learning agents maximize the rewards they experience from an environment. 

The approach is evaluated on one-shot adaptation performance of the planner in matrix game setups. 


strength_and_weaknesses: Strengths:
The paper is well-written.
Presents the idea of mechanism design with MERMADE in a clear manner.
Experiment setups and results are well-presented and the performance of the approach is demonstrably strong wrto baselines. 

Weaknesses: 
Experiments are limited to Matrix games and Bandit settings.
More random seeds and baselines are needed in the experiments. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear and is novel as far as I can tell. 
Most of the experiment setup and description of the algorithm is present in the main text, making it easier to reproduce the results.
summary_of_the_review: 
MERMADE is interesting because it merges meta-learning with mechanism design. The idea looks very promising. 
How do these ideas scale to larger/challenging domains (for example, in MDPs with continuous state spaces, with multi-agents environments)? What kind of research questions need to be addressed to make this idea to scale?

How about including a baseline that is not adapted at test time? What does the performance of the baseline look like when it is evaluated in a zero-shot manner at test time? How large is the gap between MERMADE and this baseline? This will be useful to understand the contribution made by MERMADE over a baseline that is trained similarly but held fixed at test time.

The experiment results seem to be averaged across 3 random seeds. The experiment setup should run fast and should not be a challenge to report results from many more random seeds. Have the authors considered looking into reporting results from more seeds?

How are the hyperparameters tuned for MERMADE and for the baseline methods? 

In the experiments, it seems like it is possible to measure the optimal performance of an oracle in mechanism design. It would be interesting to see the difference between MERMADE and such an oracle in the experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed MERMADE, a deep RL approach to mechanism design that fuses model-based methods and gradient-based meta-learning methods to design a mechanism with fast adaptability. The authors analyze the one-shot adaptation performance of a meta-learned planner in a matrix game setting, under both perfect and noisy observations for the agent and the planner. They show that meta-training reliably finds solutions that one-shot adapt well, and characterize how the planner’s out-of-distribution performance depends on its observable information about the agent.
strength_and_weaknesses: a)	Strength:
Different from the way of mechanism design in traditional game theory, this paper combines model-based and meta-learning theory in reinforcement learning to design a mechanism with good robustness and adaptability.
b)	Weaknesses: 
In model-based reinforcement learning, the author lacks sufficient analysis of model error and policy shift and lacks a detailed description of how the model is established. The experimental results are not fully displayed graphically, and the paper is not very easy to understand. Experimental results do not have multiple sets of seeds to eliminate probabilistic errors.

clarity,_quality,_novelty_and_reproducibility: Clarity: Lack of a clear graph to reflect the structure and novelty of the algorithm.
Quality: The charts are rough, and lack detailed instructions.
Novelty: The algorithm proposed in this paper combines model-based reinforcement learning and meta-learning. Compared with traditional methods, the designed mechanism has better adaptability.
Reproducibility: The author did not provide the source code, so I cannot confirm it

summary_of_the_review: This paper proposes a mechanism design algorithm based on the model-based method and meta-learning, and learns the adaptive mechanism, but lacks theoretical analysis, insufficient experimental design, insufficient chart production, and lack of graphics to illustrate the overall structure of the algorithm.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a MAML approach to meta-training a model based RL policy which will learn to "nudge" no-regret bandit learning algorithms into cooperating in a single-player prisonners dilemma. Experimental evaluation shows that the model based method allows "level-2" reasoning about the "level-1" decision and adaption rule of the bandit algorithm.
strength_and_weaknesses: Strengths:

- I think this is an interesting first step towards learning best mechanism design. Nudging two agents playing prisoners dilemma towards cooperating and being able to identify the minimal number of interventions *dynamically* by meta-adapting is an intuitive "minimal" mechanism design
- the experiments are well thought out, explained and reasoned for
- it is interesting to see that the model based mechanism can learn to recognise agent behaviour and 

Weaknesses:

- the biggest problem I have with the paper is that as of now, it isn't really mechanism design, it's simply curriculum learning for bandit algorithms. For it to truly be mechanism design, I would expect there to be a *game* between no-regret learners (as above, nudging *two* players, making for a three player game). As of right now, one uncharitable way of characterising the paper would be "empirical evaluation that MAML vs no-regret bandit algorithm games converge" (which, I must emphasise, is still a fine and nontrivial contribution). I think this can be alleviated by discussing the limitations the choice to focus on single agent poses, or even better, by evaluating the system in the 2-players, 1 planner setting, with "no planner" being a natural baseline. To be honest, I expect this to be *much* more difficult, but even a negative result would strengthen the contribution of this paper by establishing that a single adaptive algorithm (albeit simple) is "steerable" while a game might not be anymore
clarity,_quality,_novelty_and_reproducibility: Reproducibility is lacking due to no code being provided and the notorious difficulty of reproducing (META)-RL methods.
Evaluation is mismatched with the paper claim as described in weaknesses, and I am also missing error bars (unless the parenthesis denote std, which should be noted in table caption). Also, 3 seeds of a single META algorithm is too little, unless I'm missing a an explaination that this is justified.
Clarity is well done otherwise.
summary_of_the_review: Overall, I'd say this paper *barely* misses the threshold to being accepted. If the additional experiment on 2 players can be added I think it would be a clear accept, if not it would depend on the quality of discussion on why this is justified.

---

Post-rebuttal edit: the reframing of narrowing of scope from "mechanism design" to "learning to align learners" done by the authors after our discussion and the additional cross-algorithm experiments made me change my score from 5 to 6
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Mechanism design studies the problem of constructing a game that elicits a desired behavior from its' players. Often players adapt their policy with feedback through a learning algorithm to improve their future performance. This raises the question of how to design a mechanism that elicits a specified behavior while being agnostic to its' players' learning algorithms. This manuscript proposes an algorithm that learns to modify a single-agent player's behavior by altering said agent's reward function. 

The authors claim the following contributions:
 - A model-based meta-learning mechanism designing algorithm for few-shot games with unknown agents.
 - Empirically demonstrate their algorithm on single-stage matrix games with perfect and noisy observation.
 - Empirically demonstrate their algorithms on multi-stage matrix games with learning agents.

strength_and_weaknesses: **Strengths**
 - The exposition following their framing of intervened rewards and the planner was very clear.
 - Inclusion of the per-opponent generalization in Table 1 provides better insights into generalization, and the contribution of the model-based additions to their algorithm. 
 - Mechanism Design is an understudied field within multiagent reinforcement learning methods and the area needs more attention.

**Weaknesses**
 - Meta-learning appears to be an exceptionally complex approach for mechanism design in the provided experiments. It would embolden the author's work to either show analytic solutions and how their algorithm offers an advantage, and/or demonstrate their algorithm on a game where we cannot compute analytic or Bayesian solutions. 
 - The authors only investigate a single-agent environment and analyze repeated interactions between the agent's behavior and implementation of mechanisms derived from a co-learner independently. This is strange for a piece on mechanism design, because the incentives of both "players" (the agent and mechanism designer) are aligned. There is no strategic component present such as an agent not being truthful about their policy. I am curious if the authors have invested applying this method to general-sum games with >1 player  and if their mechanism selects equilibrium that are strategy-proof, efficient, or balanced?
 - Furthermore, as this work focuses on single-agent environments, this work is effectively in inverse reinforcement learning. I was hoping the authors could comment on how their algorithm differs from the existing work in this space? I would also ask that they include this discussion in their paper. They should also include baselines from this field in this work. 
 - A major point in their claims is that their algorithm can quickly adapt to unseen test agents. As a result, it's surprising to me that the results all assume that the train and test agents are using the same underlying algorithm (with different exploration coefficients). Could the authors comment on the generalization performance of their algorithm compared to baselines _across_ algorithms? Moreover, comment on the diversity of the population exhibited by the coefficients are chosen, it's not clear to me if beta=0.17 or beta=0.27 create meaningfully different behaviors in such a simple game.
 - I found their usage of test/train/meta-test/meta-train and its relation with varying k-shot regimes to be confusing. Perhaps that's just me, but for example ".... adapted quickly on a single test-time trajectory", is ambiguous as if the adaptation must occur within the trajectory, or if the adaptation is conditioned on a single trajectory. Moreover, if either is the problem setting, then why is there commentary about K-shots?
 - The name of the algorithm is never explained?
clarity,_quality,_novelty_and_reproducibility: I cannot comment on the originality of this work as I expect that it has exceptional similarity to work in Inverse Reinforcement Learning, of which I am not well-read. If it is novel in that area, then comparing it to [1] there is a novelty in looking at having the mechanism designer incur a cost for implementation. 

The overall clarity, quality, and reproducibility of this work is better than average; however, I would not expect the results to be replicated without the source code, and would encourage the authors to release it.

[1] Fu, et al.. Evaluating Strategic Structures in Multi-Agent Inverse Reinforcement Learning. JAIR, 2021.
summary_of_the_review: Overall, I'm excited to see work in this area, but the lack of discussion of inverse reinforcement learning makes it hard to understand where this paper fits into the literature. Moreover, without IRL baselines and narrative, it's unclear if this work contributes a new idea or one that is competitive in this domain. As these are major scientific concerns with this work I would not recommend publishing this work in its current state, but encourage the author's to continue to pursue this topic.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses a general sum game where a principal nudges adaptive agents to align goals with that of the principal by interacting with the agents while having a constraint of minimizing the number of interactions. The proposed method MERMAIDE learns a world model and uses gradient-based meta-learning to learn a principal policy that can be quickly adapted to perform well on unseen out-of-distribution agents.

The paper has an interesting formulation of an important problem. Reviewers pointed out many limitations of the work, such as framing the problem as mechanism design, connection to inverse reinforcement learning, and inclusion of additional baselines, which resulted in substantial changes in the paper. The changes warrant another round of reviewing. Reviewers also showed concern that the latest draft may have significant inconsistencies due to many changes. For example, the first paragraph of the introduction still builds up framing for mechanism design. The work would also benefit from using RL baseline algorithms stronger than Reinforce. I strongly encourage the reviewers to resubmit the paper to the next venue after careful editing.


justification_for_why_not_higher_score: The paper went through substantial changes, bringing inconsistencies and requiring another round of reviewing.
Decision: Reject