MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning
Authors: Arundhati Banerjee
Soham Rajesh Phade
Stefano Ermon
Stephan Zheng
Venue: Accepted by TMLR ()

Abstract: We study how a principal can efficiently and effectively intervene on the rewards of a previously unseen learning agent in order to induce desirable outcomes. This is relevant to many real-world settings like auctions or taxation, where the principal may not know the learning behavior nor the rewards of real people. Moreover, the principal should be few-shot adaptable and minimize the number of interventions, because interventions are often costly. We introduce MERMAIDE, a model-based meta-learning framework to train a principal that can quickly adapt to out-of-distribution agents with different learning strategies and reward functions. We validate this approach step-by-step. First, in a Stackelberg setting with a best-response agent, we show that meta-learning enables quick convergence to the theoretically known Stackelberg equilibrium at test time, although noisy observations severely increase the sample complexity. We then show that our model-based meta-learning approach is cost-effective in intervening on bandit agents with unseen explore-exploit strategies. Finally, we outperform baselines that use either meta-learning or agent behavior modeling, in both $0$-shot and $1$-shot settings with partial agent information.

Published in Transactions on Machine Learning Research (11/2023)
MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning
Arundhati Banerjee*
arundhat@ics.cma
School of Computer Science Curnegie Mellon University
Soham Phade
poham_phhdde@bbekkkeee.eeee eduu
Salesforce
Stefano Ermon
ermon edu
Department of Computer Science Stanford University
Stephan Zheng Asori AI
stephan @asari.et
Reviewed on OpenReview: https: // openreview. net/ hta//forur forum? id= H5 H5VRCCCC
Abstract
We study how a principal can efficiently and effectively intervene on the rewards of a previously unseen learning agent in order to induce desirable outcomes. This is relevant to many real-world settings like auctions or taxation, where the principal may not know the learning behavior nor the rewards of real people. Moreover, the principal should be few-shot adaptable and minimize the number of interventions, because interventions are often costly. We introduce MERMAIDE, model-based meta-learning framework to train principal that can quickly adapt to out-of of-distribution agents with different learning strategies and reward functions. We validate this approach step-by-step. First, in a Stackelberg setting with a best-response agent, we show that meta-learning enables quick convergence to the theoretically known Stackelberg equilibrium at test. time, although noisy observations severely increase the sample complexity. We then show that our model-based meta-learning approach is cost-cffective in intervening on bandit agents with unseen explorc-cxploit strategies. Finally, we out perforrorm baselines that use either meta-learning or agent behavior modeling, in both 0-shot and 1-shot settings with partial agent information.
1 Introduction
In many application domains, such revemue maximization in auctions (Milgrom & Milgrom 2004 economic policy design for social welfare (Zheng et al. 2022 or optimizing skill acquisition in personal ized education (Maghsudi et al.) 2021), a principal seeks to incentivize an adaptive agent to achieve the principal's goal. In this work, we assume that both the principal and the agent are learners and the principal incentivizes by directly intervening on the rewards of the agent. However, the principal does not know neither the exact value of the agent's 's rewards nor its learning algorithm (or its parameters). For instance, a government (principal) may want incentivize the use of environmentall--frieddgy products by levying green taxes, but needs to consider that. people (agents) may change their consumption behavior taxes change. Here, common agent models based rationality forms of bounded rationality often do not fully describe real-world behavior. Hence, interacting with the agents is required for the principal to learn how agents change their behavior, but such interactions are not. " "free" or without risk. For example, measuring the impact of lax change
* Work done during internship Salesforce

---

Published in Transactions on Machine Learning Research (11/2023)
consumers takes effort, while it may incur economic costs if the new tax is unfair or not well calibrated. In similar vein, for designing intelligent tutoring systems, the tutor (principal) has to adapt to different. students (agents) and learn to incentivize the students to understand different concepts, without the designed curriculum being too difficult or too casy (incurring cost for the principal). In personalized health-monitoring apps, the app (principal) has to adapt to the user's (agent) lifestyle and preferences to maybe create a plan that can incentivize the user to fulfill their health-related goals.
To reduce the need for real-world interactions, we can use simulations with deep reinforcement learning (RL) agents. This is an attractive solution framework: deep neural networks are expressive enough to imitate real-world entitics and simulations can be run safely and at will. Morcover, we can use deep RL to learn intervention policies that are effective even in the face of complex agent behaviors in sequential principal-agent problems.
However, this approach also faces several challenges. When deploying the learned policies the real world, interventions can lypically only be applied few times, due to implementation costs, and rarely under identical cireumstances; in contrast to simulations, we cannot reset the real world. Even though principals may adapt their policies to new conditions, they cannot realistically know the true rewards or learning strategy of the agent. Hence, our goal is to learn the principal's 's intervention policy that 1) can perform well even when agents learn, 2) can be quickly adapted, 3) is robust to distribution shifts in agent behaviors, and 4) is effective despite having only partial information.
Contributions. To address these challenges, we propose MERMAIDE (Meta-learning for Model-basee Adaptive Incentive Design), deep RL approach that 1) learns world model and 2) uses gradient- baased meta-learning to learn a principal policy that can be quickly adapted to perform well on unscen test agents. We consider a principal and single-agend setting wherein the principal intervenes al a cost on the agent's 's learning process to incentivize the agent to learn to act to achieve the principal's objective. We assume that the agent behaves in a first-order strategic manner and the principal in a second-order strategic manner. Here, the agents optimize their experienced rewards and minimize their regret, but do not account for their influence on the principal's actions. In contrast, the principal intervenes explicitly to influence the agent's 's actions.
Claims. In summary, our work demonstrates the challenges faced in the principal and single agent setup with adaptive learning agents. MERMAIDE is an effective framework to address these challenges.
Our empirical results validate the advantages, but also show the limitations of a purely model-based approach to address the non-stationarity in the environment for the principal and the adaptive agents. Our proposed purely model-
solution, MERMAIDE, shows that model- based meta-learning is able to further improve based approach and Icarns a cost-cffective few-shot adaptable intervention policy for the principal.
More specifically, in this work, we empirically verify the following claims about MERMAIDE:
In a single-round Stackelberg game between the principal and an agent, where the principal (leader) acts first. by deciding whether to intervene or not, and the agent. (follower) acts second according to best-respouse policy (Von Stackelberg) 2010), our meta-trained principal reliably finds solutions that onc-shot adapt well with a best response agent under both perfect and noisy obscrvations for the agent and the principal. Furthermore, the principal's out-of-distribution performance depends on its observable information about the agent.
Claim 1.
Claim 2. In the multi-armed bandit setting, MERMAIDE finds well-performing reward intervention policies in repeated interactive setup with the adaptive bandit agents. MERMAIDE's test-time performance and robustness against out-of-distribution bandit learners depends on the agent's 's level of exploration and their level of pessimism in the face of uncertainty, which are unknown to the principal. In particular, this holds for both K = 0-shot and K = 1-shot evaluation.
To the best of our knowledge, prior works studying the principal-agent sctup, even in an adaptive setting, overlook several of these problems with more restrictive assumptions on either the observations available or the types of agents and principal.
2

---

Published in Transactions on Machine Learning Research (11/2023)
2 Related Work
Mechanism design. The principal-agent problem [Esennhrdt]] 1989) studies how a principal can incentivize and align the agent with the principal's 's goals, in particular in the face of information asymmetry (e.g., the principal does not know the agent's rewards). This relates to the general mechanism design setting in which a principal typically interacts with multiple agents (Hurwicz & Reiter 2006). Learning a mechanism with agents who also learn is a bilevel optimization problem, which is NP-hard (Ben-Ayed & Blair 1990] Sinha et al. 2017). Possible solution techniques include branch-and- bound and trust regions (Colson et al., 2007]. In particular, solving bilevel optimization using joint learning of the mechanism and the agents can be unstable, since agents continuously adapt their behavior to changes in the mechanism. This can be stabilized using curriculum learning (Zheng el. al. 2020), but generally bilevel problems remain challenging, especially with nonlinear objectives constraints. Moreover, unlike with typical curriculum learning (Bengio et al. 2009), in our sctting, the principal's 's intervention essentially changes the task for the adaptive agent, thus presenting a non-stationary learning problem for both the principal and the agent at each time step.
Adaptive mechanism design. Previous work in mechanism design usually does not consider learning how to learn to incentivize across agents of different types. Pardoe et al. (2006) found that form of meta-learning that adapts the learning process itself can design English auctions (sequential bidding) that perform better with adaptive bidders who are loss-averse, and is still effective when the distribution of bidder behaviors (slowly) shifts. Prior work has studied algorithms for incentivizing exploration in bandit agents (Chen et al.) 2018: Wang et al.) 2021 where the principal can assign incentives to temporary and myopic agents for choosing different arms so that the principal can passively determine the global preferences of the agent population. (Shi et al. (021)) extend this line of work to consider non-myopic strategic agents, but unlike our principal-agent problem formulation, they assume that the agent is aware of the principal's I's incentive before choosing an arm and the agent always selects the incentivized arm. Moreover, their setting does not focus on few-shot generalization to unseen test agents. Our work expands on this theme by explicitly modeling agents that learn, considering shifts in the learnóng algorithm of the agents, and using deep RL with meta-learning. This combination enables learning incentives that generalize well across more complex tasks
Meta-learning and inverse R.. In recent years, gradient-based meta-learning has proven effective in learning initializations for complex policy models that generalize well to unseen tasks (Finn et al.] 201 7a Nagabandi et al. 2018). Luketina et al. (2022) study meta-gradient for adapting in environments with controlled sources of non-stationarity, but ignore non-stationarity from interactions between stratogic agents that learn. Prior works in imitation learning (Argall et al.) 2009) and inverse RL (Abbeel & Ng] 2004 assume access to export demonstrations with a fixed policy to imitate or learn the reward function of, whereas Jacq let al. (2019); Ramponi et al. (2020) consider inverse RL with observers from learners that improve their policies, but they do not feature principal that actively intervenes. In contrast, our principal aims to learn policy that can strategically alter the behavior of such demonstrators (our agents), who are themselves learning during an episode of the demonstration. Recently, Boutilier et al. (2020) studied meta-learning for bandit policies, while Guo et al. (2021) introduced the inverse bandit setup for learning from low-regret demonstrators. However, these works do not consider shifts in the bandit learning algorithm between training and test time.
Modeling agents. A key challenge in multi-agent learning is that each agent experiences a non-stationary environment if other agents are learning, As such, agents can benefit from having world model, e.g., to know what the policy or value function of the other agents are. World models can stabilize multi-agent RL (Lowe ct al.] 2017) and enable higher-order learning methods (Focrster et al. 2018), and are a form of model-basod RL. However, this may require large amount of observational data or prior knowledge, which may be hard to acquire. We show that world models make principals much more efficient in our setting.
3

---

Published in Transactions on Machine Learning Research (11/2023)
3 Problem formulation
Overview. We model a principal who aims to incentivize an agent to (learn to) execute the principal's preferred action. To do so, the principal can intervene and change the agent rewards at a cost. Without. interventions, the agent may learn to prefer an action different than the principal's.
For example, consider consumers who can use either environmentally "clean" or "dirty" goods. Indifferent at first, consumers may gradually learn to prefer dirty goods those are consistently cheaper than clean ones, whereas the government may want them to prefer clean goods. Hcre, the agent's 's reward is the negative of the cost of consumption, for instance, and an intervention changes the price of goods through taxes or subsidies. If we can use simulation, the principal can compute an optimal intervention. However, the simulation might be inaccurate and real-world agents might behave differently. As example of such test-time distribution shift, simulated agents may be quick to change their consumption patterns, while real agents may be slow. A "good" principal (trained in simulation) could quickly be fine-tuned to intervene more in the latter case and adapt quickly if such behavior is observed during deployment.
In particular, we focus on learning a principal policy that can adapt quickly at test-time (e.g., deploying taxes and subsidies in the real world), and that is effective when the agent's 's learning algorithm differs from that during training.
We now formalize this setting. In this work, we focus agents in stateless environment for ease of exposition. For all variables and their meaning, see Appendix A
The agent. The agents are characterized by their action space A and a base reward function r: r:A A + R. We call it base reward as the agent experiences an intervened reward
Fs (a:) = (a.) + r (a:),
where the intervention rf is provided externally (by the principal) for the agent action ar We index time as t = T.. At each time step f, the agent's policy computes a distribution over its actions based on the observations for the agent up to timestep and exccutes a ~ T₁. We assume that the principal has a preferred action a" E A that the agent should execule, whereas the agent's optimal policy can prefer a different action than a' without intervention. Finally, at time t, the agent learns using an update rule f (Te, a, f.) + M441 to maximize the agent's 's intervened rewards, e.g., under UCB (Lai et al.. 1985). f updattss the confidence bounds for the action selected at time t.
The principal. In this work, from the principal's point of view, the world (environment) consists of the agent who maximizes F. A standard assumption is that agents are rational and they may have a private state (referred to as its type) which the principal cannot see. Although the agent faces a stateless problem, the principal faces a stateful problem with partial observability. The full state S includes the principal's internal state hy (e.g., the principal's belief about the value of the private agent information), and all information about the agent, including its past actions, reward function, and policy model; often, the latter two are private.
More formally, the principal can be modeled as a POMDP (S, "S, P). It receives observations o" (a part of the world state s), AP is its action space of interventions, rP is its reward, 7 is a discounting factor, and P are the environment dynamics, e.g., as caused by the agent's actions. At time t, the principal samples action ay Tp (a]lo[1: which determines its intervention on each possible agent action o, i.e. ay = rr /AI]
Adaptive intervention policy learning To model distribution shift at Lest time, we follow the meta- learning terminology (Finn et al., 2017b) and view each distinct agent as a task -i. The principal has access to a train set of agents 7¹ E Tuaaini i I...., Puain and is evaluated on a test set of agents r E Tres; i 1...., Nestt- We emphasize that during task, both the principal and agent may learn and adapt, both at train and test time.
4

---

Published in Transactions on Machine Learning Research (11/2023)
Here, focus two key challenges: K-shot adaptation and distribution shift. First, the principal gets only K episodes for fine-tuning for each test task (but can train indefinitely for each train task). Second, the principal faces two types of distribution shift: 1) across tasks and 2) intra-task non-stationarity. The train and test tasks may differ (significantly) in their temporal distribution of actions, c.g., due to different agent updates f' or the agent rewards ri being centered around different values (e.g,, average price levels are higher in the real world vs in the simulation). Within a task, the agent's learning is affected by the principal's interventions that change its reward f. This gives rise to non-stationarity in the agent's environment, as its learning objective may shift over time. These forms of distribution shift distinguish our adaptive intervention policy learning setting from most prior work in meta- meta-learning, which often assume stationarity within task and also assume similar task distributions at train and test times.
Objectives. The principal's objective is to maximize how often test-time agents choose a" during learning and have them converge to a policy that always chooses a". To do so, the principal aims to maximize the cost-adjusted test-time return Jest (*²,'*) x²) EE=1 --1 (rf acı), where the agent executes its (optimal) policy œ² [x*) in response to xp and the principal incurs a cost er if it intervenes. rf (a, a²), 0.
T E (---((aa)) aci)
T arg max max EE En E (xP))
(2)
A simple cost function is q [r * i.e., the cost is constant across non-trivial interventions, where a >> 0 is a constant. Note that if intervention were free (c, 0), trivial solution is to always add a large r' (a*) >> O for its preferred action a', such that it always yields the highest reward. Hence, we focus on learning non-trivial strategies when intervention is costly, which forces the principal to strategically alter the agent's 's learning behavior.
During an episode of time steps, each agent starts with a uniformly initialized action probability distribution Tú and optimizes मं subject to interventions T² to maximize its return: E,,, Σt F (at af) Here, we assume that T and y are sufficiently large so the agent converges to its optimal policy under F, using its learning algorithm f. That is, we assume that the objective Equation (2) is sufficient to describe the principal's objective of ensuring the agent converges to preferring a" some t T.
In the K-shot adaptation setting, at test time, the principal gets K episodes to interact with any agent, cach episode of length T steps. The principal has fixed policy during an episode and it can updale its policy at the end of an episode. The agent is reset across episodes, and within each episode, the agent follows its own learning strategy in response to the principal's interventions. On the K + th cpisode, the principal evaluates its K-shot adapted policy on the agent. Note this assumes that the principal has a separate copy of the test time agent for evaluation.
4 MERMAIDE: Learning to Align Learners
MERMAIDE learns an intervention policy align the agent's preferred action with the principal's one, using:
1) recurrent world model parameterized by w that outputs distribution over an agent i's actions at the next time step t: 2t (dlat_1,a{_1 hi-1), conditioned on the planner's intervention and the observed agent action at t --1 1. h is the hidden world model state. of xt.
2) a recurrent intervention policy which outputs a distribution over interventions of o (aflat_1: ah_,, at, h) conditioned on its previous intervention, the observed agent action and the world model's predicted next agent action at = max, Aw (f ((1)) hi_ is the hidden state of the policy network.
We train this with gradient- basod meta-learning and RL (Algorithm n² indicates the mean or basc reward function for agent i and j' is the reward function after the principal's intervention of time t. Please refer to Table and Table 4 for comprehensive list of the notations used. Here, the principal maximizes
5

---

Published in Transactions on Machine Learning Research (11/2023)
Trian
Intervention policy
dq
i
T
f
di
World model
W
of
Figure Overview of MERMAIDE. Left: Flow of principal and agent observables, rewards, and actions. Right: The principal's world model and intervention policy. Also Algorithm
Algorithm MERMAIDE (Notations also in Table
Initialize principal (0o, wo), and hidden states hg. hg.
2: for meta-train epoch = 1,..., Farai do Update world model parameters w = we (Equation 3 Perrinn do 5: Initialize agent: (u', mo), task specific principal policy parameter (76) 8.. for k l,..., Kuain do
4: for agents (tasks) == 1, .,..., Meroin
7: for time = 1,...,T do Predict ai arg of &n (f
9.
Th Intervention: fi' = n² + af, at Fo(r)) a. di
10:
Agent acts: of mद and receives reward rt N (Øt ( tt mlt1. end for
11:
12:
Locally update 0 (Tk) + ( {Using REINFORCE.)
13:
end for (Rollout for meta-update; Dmntta (7')= =3 for t= 1..... T do Predict aj = arg max,, fhr a: Intervention: Ø² = n¹ + oi of x" ((hhra)))
14:
15:
16:
(aglai- dh hi,
17:
Agent acts: at xt, receives reward rf N (²) Updates oं + nias.
Collect Dmett (79 u ais af TP ( (Tkkea)))
18:
19:
end for
20:
end for
21:
Meta-update + le+1 using Dmeta Ups Dmota (7'). {Using MAML.)
22:
end for
J similar to the objective in Equation (2). The base RL algorithm REINFORCE (Williams) 1992) and the meta-learning update uses MAML (Finn al. 2017b). The agent optimizes its cumulative intervened reward (see Section 6 The world model d.. trains by maximizing the log-likelihood of the observed of. using
Adam (Kingma & Ba 2014)
T log the (f
arg max Eny Egg
Σ
(3)
Note that the principal's parameters are updated after each T-step episode, while the agent continuously learns during each episode. Also, the agent reset between episodes. At time 0, the world model makes a prediction based on zero initialization. We use single world model for all agents. At meta-test time, only the intervention policy is updated by one-shot adaptation to new agent (Algorithm) 2

---

Published in Transactions on Machine Learning Research (11/2023)
5 Evaluating Meta-Learning for the Principal
We now compare the learning bchavior of principal that is meta-trained (MAML) (Finn et al. 2017b) versus one trained with standard policy gradients (RL), in a simple Stackelberg game between principal and an agent. The Stackelberg game considers a leader-followee approach where the principal (leader) acts first, deciding to intervene or not on the agent (follower) who acts second according to a best-response policy. The key characteristic of a Stackelberg equilibrium is that once the leader has made their move, the followers have no incentive to deviate from their optimal responses, given the leader's action. This results a stable, strategic outcome where each player is maximizing their utility or payoff based on the actions of their opponent. Our goal is to learn intervention policy for the principal that can adapt to different agent types and find the Stackelberg equilibrium.
The agent's s actions are "cooperate" and "defect", while the principal can choose whether or not to intervene. Assuming the row player is the agent and the column player is the principal, we define a 2 x payoff matrix
No intervention (NI) Intervene (IN) u,
Cooperate (C) Defect (D)
(
u + 1,1
1 u,
- U, -C
where u e (0,1) and is the cost of intervention (c <c<D) 1). The agent's base payoff is its type. The principal prefers cooperation: it gets 1 if the agent cooperates and O if the agent defects (minus the cost if it intervenes). Note that intervention incentivizes cooperation (u + > -u).
We now analyze three scenarios with increasing difficulty:
1) First, we assume that the principal knows u. Here, there is a unique Stackelberg cquilibrium at (C, NI) when u 2 2, and at (C, IN) when 2-
2) Second, the principal observes a noisy version of u. In both cases, the agent first observes the principal's action and plays its best-response (knowing the payoffs).
Note that both 1) and 2) are single-round Stackelberg settings.
3) Finally, we study a repeated (multi-round) Stackelberg setting where the agent camnot see the principal's action. Instead, we assume that the agent keeps a running average for the experienced payoffs for each of its actions. In equivalent single-round setting this would correspond to the principal committing to mixed action and then the agent choosing its best response. When u 2 1, the Stackelberg equilibrium occurs at (C, NI). When < 2: at the Slackelberg equilibrium the principal has a mixed action where it chooses to intervene for 2u+1 fraction of times and the agent always cooperates.
2
Given these expected Stackelborg equilibrium strategies, our goal is to learn a policy for the principal that predicts its probability of intervention and study the adaptivity of MAML vs RL trained policies on unseen agents. We set 0.75. Given a set of training agents with different types u ~ U (0, (0,1), the principal learns the optimal policy parameters 8* arg max, E. (u) [r" (a*)]. Note that here we do not use world model, rather we focus on model-free policy learning. We study the quality of the initialization 0MAML fu. by evaluating the one-shot adaptation performance of the trained principal on unseen test agents.
With perfect observability. In this setting, assume that the principal observes agent's exact payoff parameter u. Figure shows the principal's meta-test time probability of intervening with 3 different agents from the test set, across training epochs. The principal and agent should be at different Stackelberg equilibria depending on the type u, as discussed above. We see that a principal trained from scratch on the test agents using standard policy gradients is unable to adapt to different agents in single-shot adaptation setting. In contrast, with meta-lcarning, the principal learns policy n ThMAM.. that is ono-shot adaptable to agents of different types and converges to the correct. Stackelberg equilibrium at meta-lest time.
With noisy observations for the principal. Here, we emulate a principal with partial observability of the agent, by letting the principal observe with added i..dd. Gaussian noise. The agent can see all
7

---

Published in Transactions on Machine Learning Research (11/2023)
2-4
U=0.93
10
0=2.2
1.0
U=0.07
Ruino netge
0.8-
MANLIno nobel
8.8
0.8
MANLI En nose!
6
Intervention
64-
0.4
8.4
RUno noisel
RLino rosel
02
NAMiino no52)
6.
MANIIno notsc!
MAML/whh nosel
0.6
20
60
80
100
20
40
100
9.
20
40
80
100
Training apochs
Training spochs
Training opochs
(b)
(c)
Figure 2: Single round game. REINFORCE (RL) does not adapt to expected Stackelberg equilibrium during evaluation. MAML's adaptability suffers under observation noise.
Um0.93
Um0.07
Number Intervertions
MAVI
HANI Rt
0.0
RL
0.8
0.6
0.6
2.0
100
200
160
400
500
160
250
3in
<ōn
s/o
79.01
Training spachs
Training apochs
(b)
(c)
Figure Multi-round game. (a) Principal's optimization trajectory in the expected payoff landscape during training. Axes are PCA directions in the policy parameter space. Colorbar indicates the principal's expected payoff over the training agents. (b) MAML adapts (single shot) to Stackelberg equilibrium with
best response agent. The number of interventions are normalized by the episode length T = 100.
payolfs and chooses the best response to achieve a Stackelberg equilibrium. Figure 2 shows that with noisy observations, the meta-learned principal policy requires more training time to be one-shot adaptable to the equilibrium intervention policy. This cmpirically indicates the increasod difficulty of learning an adaptive intervention policy due incomplete information about the agent, especially under limited adaptation time with unseen agents. It therefore motivates us to adopt a model-based approach for the principal to better estimate the agent type and learn an adaptive intervention policy.
Comparing Figure and preferred action (u) and the agent's s preferred action (1 -1-1 u) also impacts the one-shot adaptability of the principal with noisy observations. This informs our analysis in Section D
and Figure 2c when u < 2, the difference in unintervened payoffs between the principal's
Multi-round repeated game with noisy rewards. In this setting, the principal and agent repeatedly play an iterated game over T 100 steps. In each round, the principal observes the agent's type with added i.i.d. Gaussian noise. The agent cannot the principal's actions, and best responds given its payoff imates. Whenever the agent selects an action, it receives a noisy observation of the true payolf and updates its estimate. Compared to the single-round setting, here the agent's best response behavior may change across rounds in the game depending on its observed payoffs, giving rise to non-stationarity in the principal's 's environment. The planner, in turn, has to learn to intervene so that the agent's best response to cooperate.
Figure 3a compares the optimization trajectory followed using 1) policy gradicnts and 2) meta-lcarning for the principal. Starting from the same initialization, the meta-learned policy's parameters lie in region of the payoff landscape with higher expected value over the training agents. Moreover, Figure 3b and Figure 3c show the onc-shot adaptability of the principal's policy for two different agent types at meta-test. time. Meta-learning helps learn a better intervention strategy that is robust to the principal's observation noise as well as the agent's 's evolving best. response strategy.
This analysis shows the benefits of meta-learning over standard policy gradients in lcarning a principal's intervention policy at equilibrium with strategic agents and under partial observability for the principal in a simple Stackelberg setting. Our observations motivate using meta-training and model learning in general
8

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: Test-time principal performance on agents with different test-time learning parameters (3 random seeds). Left column: Principal's algorithm (e.g., MERMAIDE); training agent type (e.g., UCB with B 0.17). Other columns: Test-time scores (mean and standard error (s.e.) on agents with the same algorithm, but different hyperparameters. MERMAIDE (K 0) indicates zero-shot evaluation. Rest are evaluated with one-shot adaptation (K = 1). MERMAIDE outperforms all baselines in almost all cases in both the K = 0 and K = cases, with K = 1 principals doing better than K = 0.
Train on UCB, / = 0.17
Test on B = 0.17
3 = 0.27
B
0.42
8 = 0.5
8 = 0.67
No intervention
3 (0)
(0)
8 (0)
10 (0)
12 (0)
MF-RL
119 (2)
109 (2)
98 (2)
90 (2)
77 (1)
MF-MAML
133 (2)
125 (3)
107 (1)
97 (1)
77 (0)
WM-RL
123 (7)
112 (6)
100 (4)
92 (2)
75 (1)
MERMAIDE (K 1)
154 (2)
151 (1)
129 (1)
129 (1)
87 (0)
MERMAIDE (K 0)
148 (2)
138(1)
120(1)
103(2)
89(1)
Train on e-greedy, 0.1
6 0.1
= 0.2
==0.3
6= 0.4
6 0.5
No intervention
3 (0)
(1)
(0)
9 (1)
11 (0)
MF-RL
115 (5)
94 (4)
54 (19)
39 (6)
22 99
MF-MAML
122 (4)
97 (3)
58 (5)
40 (2)
12 (1
WM-RL
115 (4)
94 (5)
70 (1)
55 (3)
38 (1)
MERMAIDE (K =
138 (1)
112 (2)
85 (3)
66 (2)
37 (4)
MERMAIDE (K = 0)
133 (2)
109 (3)
86 (2)
65 (3)
37 (1)
Train on UCB, 0.67
0.17
0.27
-
8= 0.5
0.67
No intervention
3 (0)
(0)
(0)
10 (0)
12 (0)
MF-RL
103 (3)
101 (3)
92 (2)
85 (1)
74 (1)
ME-MAML
124 (2)
116 (1)
102 (1)
94 (1)
80 (1)
WM-RL
100 (4)
89 (0)
85 (1)
85 (1)
74 (0)
MERMAIDE (K =
132 (1)
130 (1)
123 (2)
115 (2)
99 (1)
MERMAIDE (K 0)
115 (3)
114 (3)
103 (4)
100 (3)
89 (3)
Train on e-greddy, 0.5
= 0.1
c= 0.2
c 0.3
c 0.4
c 0.5
No intervention
3 (0)
4 (1)
7 (0)
9 (1)
11 (0)
MF-RL
4 (5)
(3)
5 (0)
11 (5)
7 (1)
MF-MAML
(0)
(0)
6 (0)
8 (1)
11 (1)
WM-RL
102 (6)
79 (10)
(3)
47 (1)
30 (2)
MERMAIDE (K
87 (42)
102 (3)
78 (6)
69 (1)
46 (2)
MERMAIDE (K 0)
113 (20)
85 (15)
71 (16)
48 (14)
21 (15)
principal-agent settings. MERMAIDE uses both these components; next, analyze its performance against. learning agents with complex adaptive behavior under repeated interventions.
6 Evaluating MERMAIDE on Bandits
We now consider principal intervening sequentially Lamme bandit instance with action set A having base reward r = [rı..... [Ii]....,T]]]] At each time step t, the agent chooses an aims to maximize its cumulative reward over horizon of T sleps. The agent can only observe the reward for the chosen action, and hence faces a explore-exploit dilemma addressed by bandit algorithms like UCB [Lai ct al. 1985). We assume there is a unique arm with the highest base reward: = arg max, ra, i.c., the agent's s preferred action without any intervention.
adaptive no-regret learner agent, modeled by
and gets reward sampled from N (ra,o²). o²). We assumne E (0, Va. The agent
Costly interventions. To analyze the effect of the cost of intervention e, on the principal's learnt policy, we assume that the principal decides among three different intervention levels |r| E {0. 0.5, 1) such that a |r']. Across different bandit agent tasks 7 with distinct base rewards p² and reward gaps å max.e r'(a] [ [a").

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: K=0-shot evaluation across different agent algorithms: test-time principal scores (3 random seeds). Left column: principal's algorithm (e.g., MERMAIDE), training agent type (e.g., UCB, 8 0.42). Other columns: Test-time scores (mean and s.e.) on agents with different algorithm and hyperparameters.
Train on UCB, 3 0.42 No intervention
Test on 0.1
0.2
0.3
0.4
0.5
3 (0)
4 (1)
7 (0)
9 (1)
11 (0)
WM-RI
91 (4)
62 (8)
68 (1)
28 (4)
MERMAIDE (ours)
103 (1)
67 (2)
30 (2)
8 (1)
Train e-greedy, e = 0.3 Test on 3 = 0.17
B 0.27
8
0.42
B = 0.5
8 0.67
No intervention
3 (0)
5 (0)
8 (0)
10 (0)
12 (0)
WM-RL
127 (7)
95 (2)
80 (5)
80 (5)
61 (4)
MERMAIDE (ours)
138 (2)
102 (6)
116 (2)
96 (5)
77 (2)
the principal should learn to appropriately incentivize the agent while minimizing the total cost of intervening. We then define the experienced reward Va + a", (a, a" E A)
fa[a"| = r +
[a] =aaaa]-
(4)
Note that this ensures the agent always experiences an intervention, no matter which action it chooses. During each episode, the agent learns but the principal's policy is fixed; the principal can update its policy only at the end of each episode (Algorithm D Also, we assume that the principal can only observe the agent's ' actions ot but not its base reward r² or policy update function ft. We measure the performance of the principal using Equation [2 with 7 = 1.
World model. The world model predicts the agent's 's next action (given the principal's prior observations) to characterize the agent's 's behavior. We do not train the principal's world model to estimate the base rewards, because bandit agents with distinct base rewards could still execute the same sequence of actions, depending on the agent's explore-cxploit algorithm and its obscrvations.
Challenges in the sequential setting. Compared to the setting in Section learning to intervene on sequential (bandit) learners (Appendix B.1 creates more challenges:
1) Bandit agents may use different strategies to maximize their experienced reward. The agent's rate of exploration may be constant (e.g., e-greedy) or it can reduce with time (e.g., UCB) within an episode. This creates a highly non-stationary environment for the principal: its intervention policy must adapt to different intra-episode explore-exploi behaviors for the same agent. When the agent explores a larger action space, it further exacerbates these challenges as the principal only has partial information about the agent.
2) Bandit agents are sequential learners and feedback (at, ft) can update the policy n' differently at different f. This may depend on how optimistic (e.g,, UCB) pessimistic (e.g., EXP3) the agents are about their reward estimates. Hence, an intervention o" may change agent behavior differently at different t. Since interventions have different costs, a strategic principal must decide when to intervene and how much (Ir'l) based on its observations the agent's actions.
Appendix B.3 further illustrates the difficulty of this setting.
Results. Here, use 15 bandit agents for training and 10 bandit agents for testing, each with different base rewards (both within and across train and test sets). |A] 10. We consider two agent lcarning algorithms (UCB and e-greedy) and a range of exploration vs exploitation characteristics determined by their exploration coefficients: 8 E {0.17, 0.27, 0.42, 0.5, 0.67) for UCB (higher 3 gives more exploration) and c E {0.1, 0.2. 0.3, 0.4, 0.5) for e-groody (higher gives more exploration). These constants were chosen such that they yield, on average, the same number of exploratory actions with either UCB or e-greedy without. any intervention (see Appendix B.2h.
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1 shows the one-shot adapted principal's score each test set over T = 200 time steps. We compare MERMAIDE. with 1) model-free baselines (MF-RI. using REINFORCE, MF-MAMI using MAMT)) and 2) REINFORCE with world model (WM-RL); see Appendix B.4 We also include zero-shot evaluation of the trained MERMAIDE intervention policy, showing that it outperforms onc-shot adapted basclines on unseen test agents. We further include a "No Intervention" " baseline to show how agents behave by default. In all, our results show that MERMAIDE's model-based meta-lcarning approach is highly effective: the principal obtains a higher score across agents with different learning algorithms and explore-exploit behaviors.
performance. T the principal's score when evaluated on test agents with the same algorithm but different exploration constant than train agents. Using meta-learning (MF-MAML) and using& a world model to predict the agent's s behavior (WM-RL) both have advantages for training a robust and one-shot adaptable intervention policy. A world model is advantageous when 1) the test agent is more exploratory than the train set (e.g., 6 = 0.1 at training, = 0.4 at test), or 2) the agent explores throughout an episode and is likely to often select actions other than the one with its current maximum mean reward estimate (e.g., e = 0.5 at training). Because we evaluate on K 1, fine-tuning on only a single test-time episode, trained world model provides a useful prior belief for the principal. Indeed, the MF-RL results show the hidden state representation of the model-free principal might be unable to adapt to high euvironment non-stationarity without trained next-agent-action world model.
Compared to an e-greedy agent, the UCB agent explores mostly at the start of an episode, for all B. Hence, with UCB agents, the principal learns an effective onc-shot adaptable intervention policy using meta-learning (MF-MAML) only (even without a world model), as the agents cause less distribution shift across different B. Tt further emphasizes the effectiveness of meta-learning for adaptive policy learning: unlike MF-MAMT, neither the world model nor the intervention policy meta-learned in WM-RL. Moreover, it also shows that for the same amount of distribution shift (characterized in Appendix B.2), the relative benefit of a world model or meta-learning depends on the nature of the agent's exploration strategy (which is unknown to the principal).
Table D Appendix C includes observations from training with additional random seeds for the experiments
with MERMAIDE (K- = 1) in Table
Agent exploration vs intervention cost. In order to intervene effectively, the principal should learn when to intervene and how much to incentivize the agent while minimizing its incurred cost. This is a challenging learning problem for the principal not just during training, but more so during one-shot adaptation at test time. Bandit algorithms like EXP3 (Auer et al. 2002) use pessimism in the face of uncertainty, and encourage continued exploration. This increases the non-stationarity for the principal. In order to effectively incentivize such agents to prefer a", the principal needs to accurately predict the agent's policy from its observations; otherwise it can incur a high cost for intervening ineffectively and lowering its score, and learn to stop intervening. Indeed, our results when training on c = 0.5-greody agents show that the MF-RL and MF-MAML principal stop intervening. In contrast, in that setting, MERMAIDE learns an effective intervention policy that outperforms all baselines, even under distribution shift between train and test agents.
Cross-algorithm evaluation. Table 2 shows the principal's scores the zero-shot generalization setting when the training agent and test agents are of different types (different algorithms and different exploration coefficients). We consider only the WM-RL baseline to evaluate the generalization ability of the world model with standard policy gradients vs world model with meta- gradients, without adaptation to unseen test agents. We observe that when trained with UCB agents, MERMAIDE outperforms WM-RL for generalizing to e-greedy agents that have a lower exploration coefficient = 0.1 or 0.2. In contrast, when trained with e-greedy agents, MERMAIDE outperforms WM-RL for generalizing to UCB agents with both higher and lower levels of exploration. Note that the behavior of UCB agents less stochastic than c-groddy agents. More generally, a meta-learning principal that is trained on a stochastic agent generalizes well to an equal or less stochastic agent in the zero-shot setting.
11

---

Published in Transactions on Machine Learning Research (11/2023)
7 Limitations and Future Work
We have shown that MERMAIDE is an effective framework to learn principal intervention policies that adapt and generalize well to agents with unseen learning behavior. But our focus has been on stateless, sequential adaptive agents. Extending this setup to RL agents that solve non-Markovian settings (e.g,, Markov Decision Processes) would introduce a more challenging learning problem for the principal and may require different. neural network architectures for the principal's world model and intervention policy. Future work could also extend MERMAIDE to settings with multiple learning agents who may coordinate, compete, combination thereof. While our results show that MERMAIDE can adapt to bandit agents whose learning algorithm differs between training and test time, we would like to note that the meta-lcarning framework only guarantees adaptivity within the same mela-distribution over the training and test. lasks. Finally, it would also be interesting to extend MERMAIDE to agents that adapt adversarially to the principal's intervention policy, which poses a challenging non-stationary problem for the principal.
References
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty- first international conference on Machine learning, pp. 1, 2004.
Brenna Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robolics and aulonomous systems, 57(5):469 483, 2009.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit, problem. SIAM journal on computing, 32(1):48-77 177 2002.
Omar Ben-Ayed and Charles E Blair. Computational difficulties of bilevel linear programming. Operations Research, 38(3):556 160-56666066666 1990.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Craig Boutilier, Chih-wei Hsu, Branislav Kveton, Martin Mladenov, Csaba Szepesvari, and Manzil Za- heer. Differentiable meta-learning of bandit policies. In H. Tarochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2122 2134. Curran Associates, Inc., 2020. URL https://prrcceddin htt.::snnnurrrrcppcpeeeeee heeprrpp..c//pppp////0//ff1 777.ebbbbbB.4/f118222/777668883 Paperr..p
Bangrui Chen, Peter Frazier, and David Kempe. Incentivizing exploration by heterogencous users. In Conference On Learning Theory, pp- 798 818. PMLR, 2018.
Benoît Colson, Patrice Marcotle, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153(1):235-255) 256, 2007.
Kathleen M Eisenhardt. Agency theory: An assessment and review. Academy of management review, 14(1): 57-74, 1989.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic mela-learning for fast adaptation of deep networks. In International conference on machine learning, pp.
PMLR, 2017a.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Xrxi 1xXii:173.000 03400 [cs/, July 2017b. URL
03400
Jakob N. Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Tgor Mordatch. Learning with Opponent-Learning Awareness. (to:1709.04329 [es}, September 2018. URL 04326
http:://arxi..org////
Wenshuo Guo, Kumar Krishna Agrawal, Aditya Grover, Vidya Muthukumar, and Ashwin Pananjady. Learning from an exploring demonstrator: Optimal reward estimation for bandits, 2021. URL https
14866.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Leonid Hurwicz and Stanley Reiter. Designing economic mechanisms. Cambridge University Press, 2006.
Alexis Jacq, Matthieu Geist, Ana Paiva, and Olivier Pietquin. Learning from a learner. In International Conference on Machine Learning, pp.
PMLR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.0900 2014.
Tze Leung Lai, Herbert Robbins, et al. Asymptotically efficient adaptive allocation rules, Advances in applied mathematics, 6(1):4-22, 1985.
Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAI Picter Abbocl, and Igor Mordatch. Multi- Environments. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6379-6390. Curran Associates, Inc., 2017.
Actor-Critae for Mixed Cooperative-Compe
Jelena Luketina, Scbastian Flennerhag, Yannick Schroocker, David Abel, Tom Zahavy, and Satinder Singh. Meta-gradients in non-stationary enviromments. In ICLR Workshop on Agent Leurning in Ohen-Bndednesss, 2022.
Setareh Maghsudi, Andrew Lan, Jie Xu, and Mihaela van Der Schaar. Personalized education in the artificial intelligence era: what to expect next. IEEE Signal Processing Magazine, 38(3):37-50, 2021.
Paul Milgrom and Paul Robert Milgrom. Putting auction theory to work. Cambridge University Press, 2004.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Picter Abbecl, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In International Conference on Learning Representations, 2018.
David Pardoe, Peter Stone, Maytal Saar- Tsechansky, and Kerem Tomak. Adaptive mechanism design: A metalcarning approach. In Proceedings of the 8th International Conference on Electronic Commerce:
The New e- e-Commerce: Innovations for Conquering Current Barriers, Obstacles and Limitations to Conducting Successful Business on the Internel, ICEC '06, pp. 92-102, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933921. doi: 10.1145/1514544115545 URL https:
7/doi org/ 1145/1151454. 1151480
Giorgia Ramponi, Gianluca Drappo, and Marcello Restelli. Inverse reinforcement learning from a gradient- based learner. Advances in Neural Information Processing Syslems, 33:2458 2468, 2020.
Chengshuai Shi, Haifeng Xu, Wei Xiong, and Cong Shen. (almost) free incentivized exploration from decentralized learning agents. Advances in Neural Information Processing Systems, 34:560-571, 2021.
Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review on bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation, 22(2):276 295, 2017.
Richard S Sutton and Andrew G Barto. Introduction to reinforcement learning, 1998.
Heinrich Von Stackelberg. Markel structure and equilibrium. Springer Science & Business Media, 2010.
Huazheng Wang, Haifeng Xu, Chuanhao Li, Zhiyuan Liu, and Hongning Wang. Incentivizing exploration in linear bandits under information gap. arXie preprint arXiv:2104. 038660, 2021.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256, 1992.
Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbock, David C. Parkes, and Richard Socher. The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies. arXiv:2004.138 00120004-13922 fes, econ, q-fin, statf, April 2020. URL http://arrc...rg//aa//00....1.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances, 8(18): eabk2607, 2022. doi: 10.1126/sciis sciadv.abbdd0...... URL wwtp://iienneee./oooooo//oo 1126/
sciadv.abk2607
14

---

Published in Transactions on Machine Learning Research (11/2023)
Table Overview of notation.
Variable
Symbol
Time
Principal
p
Agent
State
State vector
State space
S
Agent's action space
A
Principal's action space
A²
Action sequence
01:T
{01,
02 ......
GT)
Agent i's reward sequence
Principal's reward sequence
Transition function
P
Agent i's policy
Principal's intervent ion policy
Agent's 's mean estimate intervened rewards for action
Number of adaptation steps Number of meta-tasks for the planner
K
N
Principal's history of interventions and observed agent actions upto time L.
Hl' {a" ai. 02
of af-13
Agent's 's history of actions taken and rewards observed upto time
Hi = {{ai al d
Table Notation for MERMAIDE See Section Section for their use.
Principal's policy parameter
E θ
Agent i's learning algorithm Agent i's true action Agent i's intervened action mean rewards Principal's action at time t
I E
rewards
n¹ ~u
jt'
af ~ Th of ((
Hidden state space of the principal's recurrent world model
H
Agent's action at time
af कo (ailH) .t
Agent's reward at time
N (īr
Principal's world model estimate of the agent's action probability distribution Principal's world model estimate of the latent state of the environment
A AP
H A (A), $1,0 E
L A x AP x H + H, Iús (H)
Principal's world model hidden state embeding in the LSTM architecture
h = g (af a
-h hv-1), t == 2.....
B.....TT T hjj = g
A Notation
For overview of all symbols and variables used in this work, Table 3 and Table 4
B Additional Results
B.1 Description of the bandit algorithms
We provide brief overview of the learning algorithms referred to in Section
15

---

Published in Transactions on Machine Learning Research (11/2023)
UCB. This is Upper Confidence Bound based exploration-cxplottaee algorithm that follows the principle of optimism in the face of uncertainty. At each time step 1, the bandit agent selects an action
de arg max Poo + 3 logt
The
where na is the number of steps until in which it previously selected the action Pe is its corresponding mean estimate for the experienced rewards F for action a and / is the exploration constant that balances the amount of exploration vs. exploitation across a time horizon T. A higher value of B makes the agent less optimistic and explore its action space more. The UCB agent's 's tendency to explore is also affected by the difference in the mean reward estimates of its actions. In the context of our principal agent problem formulation, if the UCB agent has larger value of 8 max, ta ra-, without any intervention at the beginning of an episode, its confidence bounds would quickly converge to exploiting the action arg máx, Ta- So principal that intervenes only lowards the later stages of an episode with this agent would have to provide much more incentives (higher r') to alter the agent's ' preferred action to be a", thus incurring a larger cost e as compared to a principal that intervencs more at the beginning of an episode when the UCB agent is still exploring its action space. This is also illustrated in Section 5 with a simpler best response agent in the single round game setting. As shown in Figure 2b and Figure 2c under observation noise (partial information), the meta-trained principal has better one-shot meta-test-time performance when the agent's 's base payoff has higher difference between the principal's preferred action and the agent's 's intrinsic preference without any intervention. Additionally, Appendix
B.3 provides an illustration of this behavior.
c-greedy. A simple exploration-exppooee exploitatioon strategy in the bandit setting is the e-greedy rule (Sutton & Barto 1998) wherein the agent selects with probability 1 1 the action = arg max, ja and with probability it selects random action. In our setting, we consider to be constant during an episode, which results in uniform exploration rate throughout. In contrast to the UCB agent, the e-greedy algorithm simulates less optimistic, more exploratory agent for which the principal requires a robust belief representation of the agent's predicted behavior conditioned uniform exploration rate for the agent, the principal has to continue intervening intermiltently throughout an episode, especially when d is large and the agent could obtain a higher reward for an action a e a" by exploring its action space when the principal does not intervene.
the principal's 's past observations (Table [ Since there is
EXP3. The paaneential-weighh algorithm for Exploration and Exploitation (EXP3) (Auer et al., 2002
follows a more pessimistic approach to exploration-cxploitttion in the bandit setting. It maintains set of weights for each agent action which are updated using the experienced rewards f follows:
(a,)
w
+ (1 w)
exp (Sar.)
(6)
[A]
Σ En.eAA 17 exxp (Sa..)
where
Sa, Σ 1 1as = a,}
Fayd
W
(7)
[
Here, w is the variable that determines the extent of uniform random exploration in the action space. This presents a very challenging problem to learn a suitable belief representation for such agents that can be utilized by a principal to guide its intervention policy. In Section P we exclude EXP3 from Table [ since it is primarily designed for an adversarial bandit setup, whereas we do not consider an agent to have such biases under our current problem formulation.
B.2 Characterizing the distribution shift in our evaluation setup
Bandit agents having the same base reward make different explore-cxploit decisions depending on their algorithm (eg. UCB, e-greedy) and also their prior observations. In Section 0 we consider agents with the same set of base rewards, but following different bandit algorithms. Both UCB and e-greedy have tunable
16

---

Published in Transactions on Machine Learning Research (11/2023)
Table 5: Experiment design choice. Frequency of agent selecting o, * arg max, Ta with UCB and e-greedy algorithms on the same set of base rewards (without any intervention) with a horizon T 200, averaged across 3 random seeds.
B
UCB
e-greedy
e
0.17
33 (0)
33 (0)
0.10
0.27
47 (0)
47 (4)
0.20
0.42
70 (0)
68 (9)
0.30
0.50
80 (0)
81 (3)
0.40
0.67
99 (0)
99 (1)
0.50
parameters that determine their explore-exploit tradeoff. In order to measure the robustness of the learnt principal policy to different agent behavior (leading to different levels of non-stationarity in the principal's environment between training and test agents), we vary the amount of exploration performed by the agent by varying the respective parameters: 3 for the UCB agent and 6 for the e-greedy agent. Table 3 shows the average (and standard error) frequency of exploration by the agents for our choices of B and in Section 6 We vary 3 and such that they pairwise comparable in Table [ and would lead to similar change in exploration frequency for both UCB and e-greedy agents. In other words, following Table [ principal trained with UCB agents having / = 0.17 when evaluated with UCB agents having B E {0.17, 0.27, 0.42, 0.50, 0.67) will encounter a similar shift in the agent's exploration frequency as in the case of training with e-greedy agents with e = 0.1 and evaluating on e-greedy agents having e E {0.1, 0.2, 0.3, 0.4, 0.5). In that case, the difference in achieved scores between the UCB and e-greedy agents can be attributed to the way in which they distribute their exploratory actions: UCB agent being more optimistic focuses most of its exploration the beginning of an episode, whereas the e-greedy agent is more stochastic with uniform random exploration throughout.
B.3 Visualizing the effect of | and the effect of different types of principal's interventions on the behavior of a UCB agent
We will consider three different instances of base rewards for a UCB agent and characterize its behavior when
unintervened
the principal intervenes once every 10 time steps for T = 200 (strategy S1)
the principal intervenes continuously until the first 20 time steps for T = 200 (strategy S2)
for 3 E {0.17, 0.27, 0.42, 0.50, 0.67). Instead of a learned stochastic intervention policy, we will analyze the effect of the deterministic policics S1 and S2 in aligning the preferred action of an agent with the preferred action of the principal. Note that both S1 and S2 incur the same total intervention cost.
First, consider a UCB agent whose base reward is r = [0.16, 0.11, 0.66, 0.14, 0.20, 0.37, 0.82, 0.10, 0.84, 0. 0.10] (Figure 4b. The principal prefers the action with base reward 0.82, whereas the unintervened agent would prefer the action with base reward 0.84. For such small value of 8 = max ra - r'(aa] = 0..2, the agent can be incentivized align its preferred action with a* more easily than if were larger. Figure 4 depicts the behavior of the UCB agents with different exploration coefficients under different principal-agent interaction conditions over T 200 as follows. In Figure 4a a value of indicates the time step when the unintervened agent selects the action with base reward 0.84, whereas Figure 4b similarly shows its frequency of selecting a" without the principal's intervention. As expected, these frequency distributions are quite similar since 8 = 0.02. Given the same budget for interventions, Figure 4c shows the agent's 's response to the principal's 's intervention strategy S1 and Figure 4d] shows its response to the intervention strategy S2. In these figures, a value of indicates the time step when the agent selects a". Both the principal intervention policies are able to align the action preference of the UCB agent with the principal's preference. But S2 receives a higher score, especially for agents with lower values for B. This is because the UCB algorithm gradually shifts from
17

---

Published in Transactions on Machine Learning Research (11/2023)
exploration to exploitation the episode progresses, and since S2 uses its intervention budget in the initial 20 time steps of the episode, the principal is able to incentivize the agent by effectively changing the observed reward i at the beginning of the episode. It experimentally verifies our discussion in Section G in sequential learners, the time step when the principal intervenes determines how effective the intervention will be in helping align the agent's 's action preference with that of the principal. This becomes more pronounced for higher values of 6, which we analyze next.
Figure 5 demonstrates the behavior of UCB agent with base reward r [0.32, 0.67, 0.13, 0.72, 0.29, 0.18, 0.59, 0.02, 0.83, 0.01]. The principal prefers the action with base ward 0.59. Without any interventions, the agent prefers the action with base reward 0.83. In this case, 8 = maxac [a[a]= --a'--aaa" [a= 0.24. Figure 5b indicates the frequency with which the unintervened agent selects a whereas Figure 5a shows the frequency of selecting the action with base reward 0.83 without the principal's intervention. Note that the unintervened agent would rarely pick a", even less so for smaller values of the exploration coefficient 3. Figure 5c indicates the frequency of the agent selecting a' with principal's intervention policy S1 and Figure 5d indicates its frequency of selection a" with principal's intervention policy S2, We observe that $2 outperforms S1 in aligning the agent's preferred action with the principal's preferred action. Since UCB agents tend to explore their action space more at the beginning of the episode, intervening on the agent's experienced reward during the initial time steps (S2) has a more noticeable effect in influencing the agent's 'ss preferred action than intervening with a fixed interval (S1).
Similar observations also hold in Figure G where the UCB agent has a base reward r [0.79, 0.53, 0.57, 0.93, 0.07, 0.09, 0.02,0.83, 0.78, 0.8. and the principal prefers the action with base reward 0.02. In this case, ō maX r[[a] - raaa] 0.91. As Figure 6b shows, this implies that the unintervened agent would almost never select a" even when it has a higher exploration coefficient B. Even with intervention policy S1, over T = 200, the principal wouldn't 't be able to align the agent's 's preferred action with its own as shown in Figure 6c| In contrast, Figure 6d shows that a principal with intervention policy S2 would outperform S1 and achieve a higher score, but the agent eventually discovers its own preference when the intervention stops in S2 and then no longer selects a". This further highlights the exlent of non-stationarity in the environment that affects the intervention policy of the principal and also the learning behavior of the agent. It also demonstrates the difficulty of the learning problem in our setup and the importance of lcarning cost-efficient few-shot adaptable principal intervention policy to effectively intervene on unknown adaptive agents.
B.4 Description of baselines
We now describe the details of our evaluated baselines in Section G alonn with their variations that assume access to an agent state oracle.
Rule based intervention with an agent state oracle (RB): Given an oracle that correctly identifies the action ae to be taken by an agent in the next time step, a simple rule based approach for the principal to intervene at time when a * a*. We assume that the principal always intervenes with a fixed incentive (r' 0.5 or 1) and we compute the principal's 's maximum possible score. Note that this is not a realistic solution for the principal since it is impractical to expect the availability of such oracle, especially for out of distribution test agents.
Model-free learning based intervention policy: In this framework, we assume that the planner has a recurent intervention policy that outputs distribution over interventions of ~ Th (af [f]laf-r.a" a" hpr1)) conditioned on the planner's intervention and observed agent action at t-1. The policy network is trained using REINFORCE for the MF-RL baseline and using MAML for the MF-MAML baseline.
Learned intervention policy with an agent state oracle: In this setting, the principal learns a recurrent intervention policy that outputs a distribution over interventions of ~ z (af k(_)) conditioned on the true agent action at time provided by an oracle. The policy network is trained using REINFORCE for the SB-RL baseline and MAML for the SB-MAML baseline.
18

---

Published in Transactions on Machine Learning Research (11/2023)
531
-00
511 28g8
828888893
(b)
032
S7.00
2 A3888183855
(d)
UCB agent with base rewards 00.16, 0.11, 0.66, 0.14, 0.20, 0.37, 0.82, 0.10, 0.84, 0.10). The agent prefers the action with base reward 0.84, while the principal prefers the action with base reward 0.82. Horizontal axis indicates time steps t= (1..., 200}. Vertical axis indicates agents following UCB with different exploration coefficient B, Values are either 0 or (a) Frequency distribution of agent selecting its unintervened preferred action with base reward 0.84. (b) Frequency distribution of agent selecting a* without principal's intervention. (c) Frequency distribution of agent selecting a' under principal's intervention S1. (d) Frequency distribution of agent selecting a" under principal's intervention S2. For small 5 max A r [a] - r [a*] 0.02, both S1 and S2 affect the agent's behavior quite similarly.
Figure 4: Charactorizing agent's behavior.
19

---

Published in Transactions on Machine Learning Research (11/2023)
027
224
(a)
(b)
022
-54
-32
-00
-38
(c)
(d)
Figure Characterizing agent's behavior. 10.32, 0.67, 0.13, 0.72, 0.29, 0.18, 0.59, 0.02, 0.83, 0.01]. The agent prefers the action with base reward 0.83, while the principal prefers the action with base reward 0.59. Horizontal axis indicates time steps t= (1..., 200}. Vertical axis indicates agents following UCB with different exploration coefficient B, Values are either 0 or 1. (a) Frequency distribution of agent selecting its unintervened preferred action with base reward 0.83. (b) Frequency distribution of agent selecting a* without principal's intervention. (c) Frequency distribution of agent selecting a' under principal's intervention S1. (d) Frequency distribution of agent. selecting a" under principal's intervention S2, For different values of B, the UCB agent acts differently based when the principal intervened following S1 $2. S1 intervenes periodically whereas $2 intervencs only at the beginning. The action selected by the agents in episode clearly reflects the effect that this has in being able to align the agent's 's preference with that of the principal.
UCB agent with base rewards
20

---

Published in Transactions on Machine Learning Research (11/2023)
037
027
542
24
(a)
(b)
022
544
-2
-00
-38
(d)
UCB agent with base rewards 0.87]. 0.7. The agent prefers the action with base reward 0.93, while the principal prefers the action with base reward 0.02. Horizontal axis indicates time steps t= (1...., 200}. Vertical axis indicates agents following UCB with different exploration coefficient B, Values are either 0 or 1. (a) Frequency distribution of agent selecting its unintervened preferred action with base reward 0.93. (b) Frequency distribution of agent selecting a* without principal's intervention. (c) Frequency distribution of agent selecting a' under principal's intervention S1. (d) Frequency distribution of agent. selecting a" under principal's intervention S2. Since å max r r-'[[a]]] 0.91 is large and the UCB agent explores its action space in the initial time steps, S2 is able intervene more effectively than S1 and achieves higher score. Since the agent is sequential learner, it discovers its own preferred action once the principal stops intervening and does not select a".
Figure Characterizing agent's behavior. 10.79, 0.53, 0.57, 0.93, 0.07, 0.09, 0.02, 0.83,
21

---

Published in Transactions on Machine Learning Research (11/2023)
Table 6: Principal (with oracle agent state input) scores across random seeds. These baselines are not applicable in practice since they cheat by assuming access to an oracle that always informs them of the agent's next action. We include them here as a form of standardization with respect a (perfect) system that does not face the challenges of partial observability out-of-distribution generalization in setting.
Train UCB, 8 0.17
Test
0.17
B
0.27
B
0.42
0.5
8
0.67
No intervention
3 (0)
5 (0)
8 (0)
10 (0)
12 (0)
RB SB-RL
173 (0)
166 (0)
154
146 (0)
126 (0)
168 (3)
138 (27)
128 (26)
122 (24)
107 (22)
SB-MAML
169
169 (1)
155 (2)
148 (1)
128 (2)
Train on c-greedy, 0.1
c=
<=
c 0.4
0.5
No intervention
3 (0)
4 (1)
7 (0)
9 (L)
11 (0)
RB
156 (3)
130 (1)
105 (4)
87 (4)
62 (6)
SB-RL
148 (2)
119 (3)
87 (4)
75 (6)
50 (2)
SB MAML
152 (1)
126 (2)
105 (3)
66 (3)
30 (9)
Train on UCB, / = 0.67
0.17
B = 0.27
B 0.42
8 = 0.5
8
0.67
No intervention
3 (0)
5 (0)
8 (0)
10 (0)
12 (0)
RB
173 (0)
166 (0)
154 (0)
146 (0)
126 (0)
SB-RL
166 (3)
(2)
150 (3)
146 (2)
128 (2)
SB-MAML
173 (1)
170 (0)
159 (0)
152 (0)
133 (0)
Train e-greedy, 0.5
0.1
6= 0.2
0.3
0.4
0.5
No intervention
3 (0)
4 (1)
7 (0)
(1)
(0)
RB
156 (3)
130 (1)
105 (4)
87 (4)
62 (6)
SB-RL
49 (46)
51 (35)
64 (29)
61 (15)
28 (17)
SB-MAML
93 (15)
62 (32)
32 (13)
58 (25)
24 (17)
Learned intervention policy with a world model without meta-learning (WM-RL): In this setting, we use our proposed recurrent world model with a recurrent intervention policy trained using REINFORCE. Here, the policy network outputs a distribution over interventions of ~ z (aflat, MP_)) where az = arg max, Au of
We would like to highlight an implementation detail our baselines indicated 'RI'' in Section Since we evaluate our learnt principal policy in the K-shot adaptation setting which common in the meta-learning literature, ensure that the principal policies that not meta-trained are also allowed to K-shot adapt at test time. This means that the 'RI'' policies are also updated at test time, before evaluation, using K rounds of principal-agent interactions. This is in contrast to Section 5 where 'RL' was trained from scratch during test time adaptation. further shows that even with pre-training (on the same set of train agents as used by 'MAML'), standard policy gradient update does not lead to effective test time K-shot adaptation on test agents.
In Table D compare the test time scores for the principal policy having access to state based oracle. We observe that overall, the meta-trained principal policy (SB-MAML) achieves higher score with distribution shift across different bandit algorithms and different levels of exploration, compared to the SB-RL baseline. The rule based baseline also shows strong performance but we note its scores do not reflect adaptation to distribution shift. However, none of these baselines that assume the principal has access to an oracle that correctly predicts the agent's action at the next time step are realistic. We can only treat the scores Table 6 as gold standards in a perfect system that does not account for the challenges faced by a principal in practice.
Training details. In Soction the principal policy R² fully connected neural network (MLP) with one hidden layer and ReLU activation. Given an (noisy) observed value of the agent type as input, predicts the probability of intervention: nf"- The principal's action at time is af ~ Bern (r²)).
22

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm MERMAIDE (K-shot Adaptation)
1: Initialize principal with trained parameters (Ometa, Wernin); and hidden states hg ng.
2: for agents (tasks) i = I,..., Thest do Initialize agent: (μ², m)), task specific intervention policy parameter (Tr) meta.
for k = l,...... K do
for time 1, .....: do
6:
Predict aj arg m Turnan 16 hQ-1) using the world model.
7:
Intervention: μt¹ μ² + of, of ~ Th (o(t)) (afla)-1- a - a. h(-1)- Agent acts: af oi and receives reward rf N के ++ o end for
10: Locally update 0 (Ti) 0 (Té+1). {Using REINFORCE.)
11: end for
12: for / 1= 1=1, 1....,,77 do
Predict 0 = arg m Aurmio 1, hi-1) using the world model. Intervention: j² n¹ of, of Th (F(k) (aflo)_,,ah_, a hb-1). Agent acts: of ~ Ti. receives reward ri ~ N (z), Updates of + T(41-
13:
14:
15:
16:
Update principal's score.
17:
end for
18:
end for
For the 'RT'' principal, it is trained on the test agents starting from scratch over episodes before evaluation. For the MAML principal, it is meta-trained to learn an initial parameterization with different set of training agents and evaluated with K-shot adaptation on the test agents.
In Section D the recurrent world model and policy networks are GRUs with 2 layers and hidden state dimension 128. For meta-training, the inner gradient update loop uses SGD optimizer with a learning rate of 7 x 10-4 whereas the mela-update step uses Adam with a learning rate of 0.001. The world model is trained only with the set training agents, it is not adapted at test time: only the policy network is K-shot adapted.
We plan to relcase the code for our implementation with the publishod paper.
B.5 Overview of K-shot adaptation with MERMAIDE:
Algorithu 2 outlines our framework for K-shot adaptation of the meta-trained principal to test agents. our experiments, K 1.
C Additional experimental results with bandit agents
Comparing the stability of MERMAIDE for different sets of random seeds. We trained and evaluated MERMAIDE for additional random scods in the == 1-shot setting from Table I In Table 7 show the scores (mean and standard error) for both sets of seeds and all the six seeds combined. Our results indicate that the training and evaluation of MERMAIDE is stable, with reasonable and explainable variability across random sceds. More specifically, models trained with different sets of random seeds result in similar mean scores with small standard error when K = 1-shot evaluated on the less stochastic UCB agents. In contrast, evaluation with the more stochastic e-greedy agents results in comparably larger standard error and more variation the mean scores for models trained with different sets of random seeds. Due to the computational costs involved, we were unable evaluate all basclines with six seeds but on the basis of these observations, we do not expect significant deviation from the claims made in Section t and the values originally reported in Table and Table 2 even with more seeds.
23

---

Published in Transactions on Machine Learning Research (11/2023)
Table 7: Comparing the variability of MERMAIDE test scores with a trained model across different sets of random sceds. Sct 1 uses the sceds {11, 26, 90) and Set 2 uses the seeds {12, 27, 91}. "Combined" indicates mean and s.e. scores for models trained with the seeds {11, 12, 26, 27, 90, 91}. Overall, we observe that training MERMAIDE, with different sets of random seeds shows little variance in the evaluation scores within error bounds, especially with the less stochastic UCB agent. For the e-greedy agent, we observe a higher standard error in the K = 1-shot evaluation scores, but in line with the stochasticity associated with e-greedy action selection in the bandit agent.
Train on UCB, 0.17
Test on 3 = 0.17
8 == 0.27
8
0.42
8= 0.5
8
0.67
MERMAIDE (Sct 1; 3 sceds)
154 (2)
151 (1)
129 (1)
129 (1)
87 (0)
MERMAIDE (Set 2; 3 seeds)
144
133 (3)
122 (2)
108 (2)
81 (2)
MERMAIDE (Combined; 6 seeds)
151 (3)
142 (4)
125 (2)
118 (5)
84 (2)
Train on UCB, = 0.67
β = 0.17
8= 0.27
Ø ==0 0.42
8= 0.5
8
0.67
MERMAIDE (Set 1; 3 seeds)
132 (1)
130 (1)
123 (2)
115 (2)
99 (1)
MERMAIDE (Set 2; 3 seeds)
119 (7)
118 (5)
110 (3)
104 (3)
87 (2)
MERMAIDE (Combined; 6 sceds)
126 (4)
124 (3)
116 (3)
110 (3)
93 (3)
Train on e-greedy, e = =.1 0.1
6 0.1
6 0.2
0.3
6 0.4
0.5
MERMAIDE (Set 1; 3 seeds)
138 (1)
112 (2)
85 (3)
66 (2)
37 (4)
MERMAIDE (Set 2; 3 sceds)
133 (2)
133 (4)
133 (4)
133 (4)
133 (4)
MERMAIDE (Combined; 6 seeds)
136 (2)
123 (5)
109 (11)
99 (15)
85 (22)
24


Reviews:
[{"review_id": "ZqtWV5Nbre", "reviewer": "Reviewer_jA9T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThis is an interesting setup, and is motivated by pertinent scenarios in governmental policy that the reviewer finds important and novel in terms of applications of RL.  The paper is very well written and for the most part easy to follow (corner cases are covered in the 'Requested Changes' section).  \n\nExperimental results are quite exhaustive within the scope of evaluated scenarios (bandits).\n\n# Weaknesses\nIn the single-shot adaptation setting with full observability (p.7), each agent's $\\mu$ is observed by the principal, it is therefore surprising that the REINFORCE agent does not quickly understand that $\\mu>1/2 \\rightarrow$ interventions and inversely.  Can you elaborate why you think this is not working?\n\nI would have also liked to see experiments with actual RL followers and not just bandits.  Can you clarify why you did not include evaluations of this type?  I would assume the MERMAIDE agent could learn to influence these situations as well, is there some reason you did not believe it would work?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes using meta-learning for Stackelberg games.    Stackelberg games are a setup where there are two agents, a principal and a a follower, and the principal tries to influence the follower to adhere to a particular policy.   The principal can affect the follower's policy by modifying its reward function at every step, however each modification of the reward incurs a certain cost, and therefore reward modification (or interventions) should be performed sparingly.  The authors propose using a meta-learning approach to learn an optimal prinicpal that can adapt to various follower behaviors.  They introduce the use of a model-based meta-learning policy that explicitly learns a model of the follower's behavior, which is then fed into the prinicpal's policy as an additional observation.  \n\nThe authors demonstrate that their approach is able to find Stackleberg-optimal behaviors in simple full and noisy observation setups, and that in more complicated bandit-based setups it performs better than more naive implementations.  Pertinent ablations are performed to suggest that both the MAML and model-based mechanisms are necessary to obtain good performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Although not uncommon, I'm not sure a Stackelberg game is so well known that it should not be at least briefly redefined within the paper's introduction (I had to look it up to remember what it was).  There is a brief definition in Sec. 5 but placing this earlier could be nice.  I also think a brief discussion of what the Stackelberg equilibrium corresponds to would also be pertinent.\n\nP 4, 'Within a task, the agent's learning is affected by the prinicpal's interventions that change its reward r.  This gives rise to non-stationarity in the agent's environement....these forms of distribution shift distintguish our adaptive intervention policy learning setting from most prior work in meta-learning\"  Non-stationarity is present both for the prinicpal as well as the agent.  In this section it seems that you are suggesting that the agent is affected by the principal's interventions thus creating a non-stationary learning problem for the agent, but then you go on to state that this non-stationarity distinguishes this setup from other meta-learning setups.  This is a bit confusing as the meta-learning is being (IIUC) performed on the prinicpal, and not on the agent.  And the principal /also/ has a non-stationary task, but as it is phrased it seems to suggest that it is the agent's non-stationarity which makes this meta-learning setup novel.  Can you clarify?\n\nOn page 5, after each episode of the principal's adaptation, the agent is reset, what do you mean by this?  The agent's episode is terminated or the agent is reset to an untrained state?  Please clarify here as well :)\n\nAlgorithm 1 has notations (such as mu) that seem to be only defined in Table 4, the reader should be able to fully interpret paper contents without referring to the appendix (esp. for key notations).\n\nSec. 5, the C/D vs NI/IN matrix is a bit confusing.  Are the 'u,1' tuples corresponding to reward and intervention cost?  \n\nI am pretty confused by the setup where the agent cannot see the principal's interventions\n\nP. 7 \"Given this equilibirium analysis....the quality of initilaization learnt using MAML than RL for adaptation..\" this sentence might be missing some words? I was not able to make sense of it.\n\np. 11 \"Our results when training on $\\eps = 0.5$-greed agent show that MF-RL nad MF-MAML principals stop intervening - is this illustrated anywhere?", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QJtNgvOqyE", "reviewer": "Reviewer_euPP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n  - The application of deep RL to mechanism design is not something I’ve seen attempted before. While deep RL has been used for a long time in multi-agent domains, historically these settings have not made assumptions on the opponent and action space that would correspond to principal-agent problems.\n\n  - The approach uses simple, easy-to-understand algorithms (REINFORCE + MAML) that are widely regarded as the standard baselines in their respective fields, making it easy to interpret the results without ablations.\n\n  - The method is evaluated under a set of tasks that incorporate distinct and interesting features, including the observability of the agent’s utility function, noise in the observations, the agent’s behavior policy, and the dimensionality of the agent’s action space.\n\n  - The use of a world model (or perhaps more accurately a meta-imitation learner to infer the agent’s policy) in conjunction with meta-learning is a creative idea.\n\n  - I appreciate the clear decomposition of the claims of the paper given in the introduction.\n\n**Weaknesses**\n\n  - The paper could be better motivated. The examples of principal-agent problems given in the introduction are not settings where I would ever expect employing a deep RL algorithm to be practical (e.g. government taxation policy has such a long period between steps that it would take many centuries before the RL agent would have meaningfully updated its parameters). \n\n  - One of the main weaknesses of the paper is that the constraints on the principal’s policy to be markovian could significantly reduce the performance of the optimal policy in the principal’s hypothesis class. Presumably, in most cases the expected benefit of an intervention will depend on its effect on the agent’s propensity to cooperate in the future as well, and so interventions earlier in the trajectory will likely pay off more in terms of discounted return. It doesn’t look like the current formulation of the problem setting permits this.\n\n  - Similarly, only using the agent’s next-step action to predict the efficacy of an action seems like it may not be the most effective use of a world model compared to evaluating the cumulative effect of an intervention on an agent’s later policies. \n\n  - The only baselines compared against in the paper are other deep RL algorithms. Given that there has been a great deal of work on mechanism design in the game theory community, I would assume that (particularly for the stackelberg dynamics evaluation task) some online learning method could be used to learn a good intervention policy as well.\n\n  -  For example, how does the deep RL approach perform compared to a principal which estimates the value of u based on prior experience and then follows an epsilon-greedy policy w.r.t. this estimate?\n\n  - I would ideally like to also see some smarter online algorithms compared against as well. It is one thing to show that it is _possible_ to use deep RL in this setting, but I think it is also on the authors to show why it would be _desirable_.\n\n  - The RL baseline using REINFORCE doesn’t seem to be doing any learning. It is possible that this is due to an innate failure of RL algorithms in this domain, but I could also believe, given the instability of REINFORCE, that the learner simply never converged. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the setting of principal-agent problems where a principal may intervene at a cost on the payoffs given to agents for taking different actions, with the goal of incentivizing the agents to take a desired action. It proposes MERMAIDE, a deep reinforcement learning-based approach to learn a policy for the principal in this principal-agent problem, which leverages a combination of meta-learning and a world model on top of a deep RL algorithm. This approach is evaluated and ablated on both Stackelberg games and in a multi-armed bandit setting on a variety of agent policies. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I have the following questions and requested changes:\n- Provide examples of practical realizations of the problem setting where it would be beneficial to employ a deep RL algorithm.\n- The authors should provide an explanation as to why they do not allow the principal’s policy to depend on the tilmestep t (or if I misinterpreted the paper, make it clearer that it does consider non-Markovian policies). I would also like to see a more rigorous discussion and analysis of the potential performance gap induced by this choice.\n- While not necessary to include in revisions, at least in the author response I would like to see some justification for why an LSTM is used as the world model instead of a transformer which receives as context the entire history of the episode (and prior episodes in the few-shot setting). Since T is not too large in these settings, this should be computationally feasible and might allow for faster adaptation to the new agent via in-context learning.\n- Provide some evidence that the vanilla RL agent’s poor performance is due to a limitation of RL in general, rather than being due to the instability of the particular algorithm used. For example, do the parameters converge to a fixed policy or do they diverge/oscillate? Does the loss go down? \n- More generally, more discussion about *why* the methods that don’t use a world model or meta-learning tend to do worse would be useful to understand MERMAIDE.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cw2skxbniO", "reviewer": "Reviewer_7t2x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pro:**\n 1. Very well written paper, particularly the introduction and proposal of the method. The experiments section gets a bit dense at times, but still at a high-quality level of writing.\n 2. Two very clearly spelled out claims that are tested. While the intro refers to a broader set of potential claims (which is good to situate the work), the “hard” take-aways are clearly spelled out.\n 3. Empirical evaluation includes important ablations and out-of-distribution experiments.\n\n**Con:**\n 1. No explicit discussion of limitations of the current study (also w.r.t. the broader goals alluded to in the intro). Would be great to see a paragraph towards the end of the paper.\n 2. One promise of the method is that it lends itself well to training a principal via imprecise simulations which then allows for rapid transfer to real-world incentive/reward design. The OOD experiments certainly support that claim but a user-study with humans would make such a claim much stronger. I do not consider this necessary for publication, but if included it would make the paper even stronger.\n 3. Going from a bandit setting to a full RL setting is likely to make the learning problems for the principal quite a bit harder. While results on the bandits look promising; showing results in some RL setting (e.g. gridworlds) would further strengthen the paper. Similar to 2., I do not consider this necessary for publication.\n\n**Verdict:**\nOverall I think the main idea of the paper is technically sound, well executed, and empirical results clearly support the two main claims (and the method more generally). The paper does a good job motivating the particular algorithm that is proposed. Perhaps the two biggest improvements for making the paper even stronger are on the experimental side: showing that the learning problem for the principal is still easily solvable in a full RL environment (and whether/how the principal generalizes to learners in new environments); and an empirical study showing that the meta-trained principal transfers well to human learners. I do not consider either necessary for publication (just suggestions for improvements). Taking all of this together I think the paper is ready for publication. Looking forward to the other reviews and the authors’ feedback.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\nThe paper investigates how to learn how to intervene on a learner’s reward function such that the learner takes a desired action with high probability, when (1) such interventions are costly and (2) the learner’s type (precise reward function, exploration strategy, and policy update function) are unknown and may be different between train- and test-time. The proposed solution is to meta-train a principal over many learner types such that it generalizes to learners from the same distribution at test time, and empirically performs well on learners from different distributions. The approach is split into two parts (learned simultaneously): learning a (world-) model over how a reward-intervention in a particular time-step affects the learners’ highest-probability action; and learning an optimal policy over interventions (by using the world-model) that takes into account the cost of intervening. Importantly, both parts do not explicitly observe the learners type, but through meta-training over many learner types behave as if inferring the learner’s type on the fly and adapting accordingly (amortized inference). Overall this leads to a partially observable, non-stationary learning problem for both the principal and the learners. The proposed approach is empirically validated in a series of experiments on a simple one-shot strategic game (Stackelberg game), with ablations regarding partial observability and non-stationarity. A second set of experiments is shown on a bandit task, where out-of-distribution performance of the principal is also evaluated. On both types of experiments the proposed algorithm performs well compared to baseline methods and alternative approaches.\n\n**Contributions:** \n 1. Formulation of incentive/reward design under unknown learner (-type) and intervention-cost as a meta-learning problem. The formulation is sensible and quite general (ranging from fully observable one-shot settings to partially observable non-stationary problems). While this may not be the very first formulation as a meta-learning problem, the paper executes it well, provides good motivation and a nice introduction / discussion of design choices.\n 2. Proposal of novel algorithm (MERMAIDE) using a meta-trained principal (gradient-based meta-learning and RL with a learned world-model). A sound proposal, with good discussion of (most) design choices. The algorithm itself is applicable to other coupled cooperative/competitive problems with two interacting learners (one doing first order strategic optimization, the other one doing second-order strategic optimization), making it relevant beyond alignment/incentive design.\n 3. Empirical evaluation. Well executed but somewhat limited in scope. A big plus is that there are a number of ablations and out-of-distribution evaluations. One small point of criticism, given the intro, is that all results are simulation only (the advertised sim-to-real advantage of the method is not evaluated).\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major (please address):**\n 1. Include a dedicated limitations paragraph towards the end of the paper, discussing to which extent the broader motivation at the beginning of the paper has been addressed and what the most important shortcomings of the current algorithm and experiments are.\n\n**Minor (suggestions, typos, etc. - not crucial for publication):**\n\n 2. Strictly speaking, meta-leraning only guarantees generalization at test time within the meta-distribution. In practice it has of course been observed many times that meta-trained systems generalize (sometimes considerably) beyond the meta-distribution. Please mention this.\n 3. “In this work, we focus on agents in a stateless environment for ease of exposition” - fine, but it may also make the learning problem for the principal significantly harder if the environments were stateful (the current phrasing sounds a bit like it would just make the math. notation more complex). Please mention this in the limitations paragraph.\n 4. (Similar to 3.) P4:  “These forms of distribution shift distinguish our adaptive intervention policy learning setting from most prior work in meta-learning, which often assume stationarity within a task and also assume similar task distributions at train and test times” At least implicitly (since meta-learning does the heavy lifting) MERMAIDE also somewhat relies on the second assumption (similar task distribution at train and test times) - whether or not the meta-trained principal generalizes to a new task distribution is not theoretically guaranteed but can only be empirically determined.\n\n 5. P5: MERMAIDE step 2: why only use the most likely predicted action rather than the full distribution over next actions provided by the world model? Could it not make a difference to the principal’s policy if the principal’s preferred action is the second most likely or the least likely action according to the world model?\n\n 6. Algorithm 1, line 10 and 17: Why does the agent’s action not appear in the Gaussian reward. I would have expected something like a negative squared error between $a_t^i$ and $\\tilde{\\mu^i}$ with added Gaussian noise: $r_t^i \\sim \\mathcal{N}(-(a_t^i - \\tilde{\\mu^i})^2, \\sigma^2)$.\n\n 7. P2 last paragraph: sentence starts with ‘However’ - a bit unclear what that refers to (is something missing?).\n\n 8. P7 typo: “using MAML than RL”\n\n 9. P7: At test-time (for the Stackelberg experiments) what is the distribution that agent types are drawn from? The same as the meta-training distribution?\n\n 10. Fig 3a: what does the colorbar indicate (label/description in caption missing; though given in main text)?\n\n 11. Fig 3 b,c: Is the number of interventions normalized by episode length (why does it lie between 0 and 1)?\n\n 12. P10: “Results.”: How were the base rewards for the train- and test-time agents constructed?\n\n 13. Table 2: Why the two empty entries (last column, row 3 and 4)?\n", "broader_impact_concerns": "No significant broader impact concerns. \n\nAs with any type of incentive design, there is potential to abuse the technology to 'nudge' humans towards actions that maximize economic profit or benefit a political agenda. I think a brief broader impact statement would be good, but would not object to the paper's publication if there is none (since I'd consider the paper quite far removed from an actual application for human/user incentive design).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers found the paper interesting and worth publishing and the requested changes were made by the authors. The reviewers found the made changes mostly appropriate. The only remaining concern was regarding the comparison of the proposed approach to additional non-RL baselines for the Stackelberg game but this was also not considered as a reason for not accepting the paper. Hence, in line with the reviewers' recommendations, I am recommending acceptance of the paper.
Decision: Accept (poster)