Title: Contextual Bilevel Reinforcement Learning for Incentive Alignment
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Bilevel Optimization

Contextual MDPs

Environment Design

Model Design
Abstract: The optimal policy in various real-world strategic decision-making problems depends both on the environmental configuration and exogenous events. For these settings, we introduce Contextual Bilevel Reinforcement Learning (CB-RL), a stochastic bilevel decision-making model, where the lower level consists of solving a contextual Markov Decision Process (CMDP). CB-RL can be viewed as a Stackelberg Game where the leader and a random context beyond the leader’s control together decide the setup of many MDPs that potentially multiple followers best respond to. This framework extends beyond traditional bilevel optimization and finds relevance in diverse fields such as RLHF, tax design, reward shaping, contract theory and mechanism design. We propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm to solve CB-RL, and demonstrate its convergence. Notably, HPGD uses stochastic hypergradient estimates, based on observations of the followers’ trajectories. Therefore, it allows followers to use any training procedure and the leader to be agnostic of the specific algorithm, which aligns with various real-world scenarios. We further consider the setting when the leader can influence the training of followers and propose an accelerated algorithm. We empirically demonstrate the performance of our algorithm for reward shaping and tax design.

Contextual Bilevel Reinforcement Learning for Incentive Alignment

Abstract

The optimal policy in various real-world strategic decision-making problems depends both on the environmental configuration and exogenous events. For these settings, we introduce Contextual Bilevel Reinforcement Learning (CB-RL), a stochastic bilevel decision-making model, where the lower level consists of solving a contextual Markov Decision Process (CMDP). CB-RL can be viewed as a Stackelberg Game where the leader and a random context beyond the leader's control together decide the setup of many MDPs that potentially multiple followers best respond to. This framework extends beyond traditional bilevel optimization and finds relevance in diverse fields such as RLHF, tax design, reward shaping, contract theory and mechanism design. We propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm to solve CB-RL, and demonstrate its convergence. Notably, HPGD uses stochastic hypergradient estimates, based on observations of the followers' trajectories. Therefore, it allows followers to use any training procedure and the leader to be agnostic of the specific algorithm, which aligns with various real-world scenarios. We further consider the setting when the leader can influence the training of followers and propose an accelerated algorithm. We empirically demonstrate the performance of our algorithm for reward shaping and tax design.

1 Introduction

In Reinforcement Learning (RL), Markov Decision Processes (MDPs) [53] provide a versatile framework for capturing sequential decision-making problems across various domains such as health care [76], energy systems [52], economics [14], and finance [33]. A considerable amount of work has been devoted to solving standard MDPs [2, 62, 67]. However, in many applications, MDPs can be configured on purpose or affected by exogenous events, both of which can significantly impact the corresponding optimal policies. For example, in a simplified economic framework, the optimal decision of an individual household depends both on public policies and economic uncertainties [19, 34]. The policy maker in turn has to make decisions, anticipating the best response of differentlyminded individual agents to its policies and exogenous events outside the policy maker's control.

To study such problems, we introduce Contextual Bilevel Reinforcement Learning (CB-RL), a hierarchical decision-making framework where followers solve a contextual Markov decision

processes (CMDP) [32], configured by the leader. CB-RL is formalized as:

where x represents the model configuration of the CMDP chosen by the leader, ξ represents the context that followers encounter, and the function J λ,x,ξ denotes the (entropy-regularized) reward of the CMDP for a policy π , a given model parameters x , a context ξ , and a regularization parameter λ .

In our framework, the leader chooses x to configure various aspects of the CMDP, such as state transitions, the initial state distribution, and the followers' reward functions. Modeling the problem as a CMDP instead of a standard MDP is essential when modeling situations where the environment is influenced by side information or personal perferences. For example, the context ξ can capture a wide range of real-world scenarios such as: (1) there is one follower, who responds optimally not only to the leader's chosen x but also to a side information ξ , such as weather or season, (2) there are multiple followers, each aiming to maximize their own utility, in which case ξ represents different possible follower preferences, and (3) there are multiple followers, each facing an uncertain contextual variable, i.e., ξ = ( i, η ) represents the i -th follower encountering a specific context η ∼ P η .

The proposed framework extends the concept of contextual bilevel optimization [37], where the follower engages in static contextual stochastic optimization rather than sequential decision-making. It also expands upon traditional MDP model design [78, 15] and configurable MDPs [50, 55], where typically only one follower attempts to solve an MDP, as opposed to CMDPs. We defer a full discussion of the related works to Appendix A. Beyond these fields, our framework finds various applications in Principal-Agent problems [8], RLHF [13, 58], dynamic Stackelberg games [27, 65], Security Games [60, 46], dynamic mechanism design [18], contract theory [41, 69], and tax design [19, 82, 34]. See Appendix B for the concrete CB-RL formulations of these applications.

Despite its wide applicability, to the best of our knowledge, there are no algorithms specifically designed for CB-RL. The closest is an algorithm from model design for MDPs [15], which can be adapted to our setting after some modifications. However, [15] requires the follower to solve the MDP deterministically using soft value iteration. Moreover, the full hypergradient is computed in each iteration, as part of which the leader requires access to the lower-level computations. Both of these aspects significantly restrict how well the method can scale to larger settings.

Instead, in this work, we propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm for the leader that solely relies on trajectory data from followers. The followers can use a variety of possibly stochastic learning algorithms to find an approximately optimal policy for the lower-level CMDPs. The leader in turn is agnostic of the exact algorithm used, as the hypergradient is estimated using only trajectory samples generated from the follower policy. The fact that both the lower-level and the hypergradient computation are stochastic makes HPGD salable to large problem settings.

We show non-asymptotic convergence of HPGD to a stationary point and validate these findings through experimental evidence. In scenarios where followers grant the leader full control over their training procedure, as posited in prior work [15], we present an accelerated HPGD algorithm, designed to minimize the number of lower-level iterations.

Our Contributions

We introduce Contextual Bilevel Reinforcement Learning (CB-RL) that captures a wide range of important applications (Sec. 2). It is the first bilevel reinforcement learning framework that through the context ξ allows multiple followers and side information. We summarize the key differences to the previous literature in Table 1.

We propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm that performs stochastic gradient descent on the upper-level objective (Sec. 3.2). Importantly, we are the first to estimate the hypergradient from lower-level trajectory samples instead of computing it exactly, while further providing convergence guarantees. Furthermore, our approach is agnostic of the learning dynamics of the agent , enabling followers to utilize a wide range of algorithms to solve the lower-level CMDPs. We only assume the leader can sample lower-level trajectories from an inexact oracle. For several widely-used RL algorithms, we explicitly show how to use them to build the inexact oracle needed by HPGD. Noteably, we are the first to consider stochastic lower-level learning algorithms, such as soft Q-learning.

Multi: Multiple followers. Side Info: Side information. Context: CMDP instead of MDP. r x , P x , and µ x denote the dependence of rewards, transitions, and initial state distribution on x . Agnostic vs. Control: Whether leader can influence the training of the follower(s). Deter vs. Stoch: Requiring full knowledge of hypergradient or estimating it from samples. Complexity is based on ∥∇ F ( x ) ∥ 2 ≤ δ 2 instead of ∥∇ F ( x ) ∥ 2 ≤ δ . *[15] assumes convexity of F in x and considers F ( x ) -min F ( x ) ≤ δ . VI: Value Iteration. PMG: Policy Mirror Gradient. PI: Policy Iteration. NPG: Natural Policy Gradient. Q: Q-learning. RT-Q: Randomly Truncated Soft Q-learning.

We establish the non-asymptotic convergence rate of HPGD to a stationary point of the overall objective-despite the nonconvex lower-level problem (Sec. 3.2). When assuming full hypergradient information, i.e., deterministic updates, the outer iteration complexity of HPGD reduces to O ( δ -2 ) , recovering previous results. Moreover, we discuss how to estimate the hypergradient if the upper-level loss function admits a specific form of cumulative costs (Sec. 3.3).

When the leader is allowed to control the follwers' learning dynamics (Sec 4), we propose a stochastic accelerated algorithm denoted as HPGD RT-Q (Alg 8). It greatly reduces the number of lower-level soft Q-learning iterations from ˜ O ( δ -2 ) to O (log( δ -1 )) , such that we recover the rate for deterministic lower-level updates. For this result we leverage several techniques, such as mini-batches, multilevel Monte Carlo [29, 37], and importance sampling.

We demonstrate the performance of HPGD for principal agent reward shaping and tax design (Sec. 5). In certain cases, the stochastic updates of HPGD are beneficial as they avoid local minima. Moreover, HPGD needs fewer outer iterations compared to benchmark algorithms.

2 Problem Formulation

We consider a bilevel optimization problem, where the follower solves a Contextual Markov Decision Process (CMPD) and the leader controls the configuration of the CMDP. In particular, the leader chooses a parameter x ∈ X ⊆ R d and nature chooses a random context ξ according to a distribution P ξ . Together ( x, ξ ) parameterizes an MDP M x,ξ , which the follower aims to solve. M x,ξ is defined by a tuple ( S , A , r x,ξ , P x,ξ , µ x,ξ , γ ) , where S denotes the state space, A denotes the action space, r x,ξ ( · , · ) : S ×A → R is the reward function, P x,ξ ( · ; · , · ) : S ×S ×A → [0 , 1] denotes the transition kernel, µ x,ξ indicates the initial state distribution, and γ is the discount factor. The subscript x, ξ implies that rewards, transitions, and initial state distribution depend on the leader's decision x and the context ξ . Connecting to previous works, for a fixed x , M x,ξ is a contextual MDP [32] with respect to ξ . For a fixed ξ , M x,ξ generalizes a configurable MDP [50]. Given M x,ξ , the follower maximizes an entropy-regularized objective by choosing a policy π x,ξ , where π x,ξ ( a ; s ) denotes the probability of choosing action a in state s .

where s 0 ∼ µ x,ξ , a t ∼ π ( · ; s t ) , s t +1 ∼ P x,ξ ( · ; s t , a t ) and H ( π ; s ) = -∑ a π ( a ; s ) log π ( a ; s ) . We call λ > 0 the regularization parameter and V π λ,x,ξ the value function. As standard in RL literature,

we define the related Q and advantage functions as:

The unique optimal policy for (2) is given by π ∗ x,ξ ( s ; a ) ∝ exp( Q ∗ λ,x,ξ ( s, a ) /λ ) , i.e., the softmax of the optimal Q-function [51]. 2 Given x, π ∗ x,ξ , ξ , the leader in turn incurs a loss f ( x, π ∗ x,ξ , ξ ) ∈ R , which it wants to minimize in expectation over P ξ . To do so, it needs to choose x to align the follower's policy π ∗ x,ξ with the leader's objective. CB-RL can thus be formulated as the following stochastic bilevel optimization problem:

Equation (4) is well-defined due to entropy regularization ensuring the uniqueness of π ∗ x,ξ . It further ensures π ∗ x,ξ is differentiable, stabilizes learning and appears in previous works [15]. Moreover, the difference between the regularized and unregularized problem generally vanishes as λ → 0 [15, 26].

CB-RL captures many important real-world problems, including RLHF, dynamic mechanism design, tax design, and general principal-agent problems. We discuss these in detail in Appendix B.

3 Hyper Policy Gradient Descent Algorithm for CB-RL

In this section, we derive a simple expression for the hypergradient of CB-RL. We present HPGD and prove non-asymptotic convergence. HPGD can be combined with a large class of lower-level MDP solvers satisfying a mild inexact oracle assumption. We show this is the case for several popular RL algorithms. Furthermore, we present results for two important special cases of our problem: (1) when the upper-level objective decomposes as a discounted sum of rewards over the lower-level trajectories, and (2) when the leader can direct the lower-level algorithm. We defer all proofs to Appendix E. and make the following standard assumptions on how x and ξ influence the setup of the CMDP.

Assumption 3.1. We assume the following conditions:

f is L f -Lipschitz continuous and S f -smooth in x and π , uniformly for all ξ , i.e.

3.1 Hypergradient derivation

The leader's loss f depends on x directly and indirectly through π ∗ x,ξ . Therefore, the derivative of f with respect to x is commonly referred to as the hypergradient to highlight this nested dependency. It is possible to obtain a closed-form expression of the hypergradient, using the implicit function theorem [28]. However, this involves computing and inverting the Hessian of the follower's value function, which can be computationally expensive and unstable [24, 47]. Instead, we leverage the fact that π ∗ x,ξ is a softmax function to explicitly compute its derivative with respect to x , which is given by dπ ∗ x,ξ ( s,a ) dx = 1 λ π ∗ x,ξ ( a ; s ) ∂ x A π ∗ x,ξ λ,x,ξ ( s, a ) [15], where ∂ x A = ( ∂ x 1 A,. . . , ∂ x d A ) . Applying the Dominated Convergence Theorem to switch derivative and expectation, we arrive at Theorem 1. Theorem 1. Under Assumption 3.1, F is differentiable and the hypergradient is given by

where ν is any sampling distribution with full support on the state space S .

Algorithm 1 Hyper Policy Gradient Descent (HPGD)

Input: Iterations T , Learning rate α , Regularization λ , Trajectory oracle o , Initial point x 0

The first term captures the direct influence of x on f , and the second the indirect influence through π ∗ x,ξ . For now we assume the leader knows ∂ 1 f ( · , π, ξ ) and ∂ 2 f ( x, · , ξ ) . It remains to compute ∂ x A π ∗ x,ξ λ,x,ξ ( s, a ) , i.e. the partial derivative with respect to x evaluated for a given policy. For this, cf. (3), we need ∂ x Q π ∗ x,ξ λ,x,ξ ( s, a ) . We derive an expression for the latter in Theorem 2. The proof adapts the analysis of the policy gradient theorem to account for the dependence of P x,ξ , µ x,ξ and r x,ξ on x .

Theorem 2. For given π, x, ξ , it holds that:

Note, Theorems 1 and 2 generalize existing results in model design for MDPs to CMDPs [15, 78].

3.2 HPGD Algorithm and Convergence Analysis

Computing the exact hypergradient is computationally expensive and thus infeasible in larger settings. Instead, to minimize F ( x ) , one would ideally sample unbiased estimates of the hypergradient in Equation (5) and run stochastic gradient descent (SGD). However, the leader does not have access to π ∗ x,ξ and generally no control over the training procedure of the lower level. Instead, we assume the follower adapts any preferred algorithm to solve the MDP up to a certain precision δ and the leader can observe trajectories from the follower's policy. Such a setting is well-motivated by economic applications.

Assumption 3.2. For any x,ξ , the leader has access to an oracle o , which returns trajectories

M .

We will show that Assumption 3.2 is relatively mild and holds for a variety of RL algorithms. Given access to trajectories generated by π o x,ξ , the leader can construct an estimator of ∂ x A π o x,ξ λ,x,ξ ( s, a ) by rolling out π o x,ξ for T steps, where T ∼ Geo (1 -γ ) . We defer the construction (Algorithm 2) and proof of unbiasedness (Proposition 3) to the Appendix. Using this estimator, we introduce HPGD in Algorithm 1. As F is generally nonconvex due to the bilevel structure [28], we demonstrate non-asymptotic convergence to a stationary point of F , which matches the lower bound for solving stochastic smooth nonconvex optimization [1].

Theorem 3. Using HPGD, under Assumptions 3.1 and 3.2, for α ≤ 1 / (2 S f ) , we have the following:

For α = O (1 / √ T ) and δ = O (1 / √ T ) , HPGD converges to a stationary point at rate O (1 / √ T ) .

Proof sketch . Using the smoothness of F and the fact that ˆ x T is uniformly sampled from all iterates, we upper bound the left side of (6) by the sum of three terms. The first is | F ( x 0 ) -min x F ( x ) | / ( αT ) . The second depends on the bias of our gradient estimate, which we show is linear in δ . The last term depends on α times the variance of our estimator, which is bounded.

To the best of our knowledge, Theorem 3 is the first result that shows convergence when using stochastic estimates for the hypergradient. When the hypergradient can be computed exactly, the last O ( α ) term vanishes and we recover the deterministic convergence rates of previous works [15, 13, 58].

Another major advantage of HPGD is that the follower can use any (possibly stochastic) algorithm satisfying Assumption 3.2 to solve the lower-level MDP, while the leader only needs access to generated trajectories. While Assumption 3.2 certainly holds if the follower solves the MDP exactly, for example with an LP-solver, we are interested in verifying it for common RL algorithms, which can scale to larger state and action spaces. In Appendix E.8, we prove non-asymptotic convergence to π ∗ x,ξ for Soft Value Iteration, which converges at rate O (log 1 /δ ) (Proposition 5); Q-learning, which converges at rate of O (log(1 /δ ) /δ 2 ) (Proposition 6) and Natural Policy Gradient, which converges at rate of O (log 1 /δ ) (Proposition 8). Additionaly, we show Vanilla Policy Gradient converges asymptotically in Proposition 7. All these Algorithms thus satisfy Assumption 3.2, which makes HPGD scalable and widely applicable to settings where followers might use a variety of model-free or model-based algorithms.

3.3 Upper-Level Discounted Reward Objective

So far we assumed the leader knows ∂ 1 f ( · , π, ξ ) and ∂ 2 f ( x, · , ξ ) . In this subsection, instead, we assume f can be written as the negative expected sum of discounted rewards over the lower-level trajectories and show how to estimate the hypergradient from trajectory samples without explicit knowledge of ∂ 1 f ( · , π, ξ ) and ∂ 2 f ( x, · , ξ ) . In many practical applications, such as reward shaping, or dynamic mechanism design (cf. Appendix B), the loss f satisfies:

Here r x,ξ represents the reward of the leader, which is generally distinct from the follower's reward. The expectation is taken over trajectories induced by the lower-level π ∗ x,ξ . In this case, the leader does not know the partial derivatives of f but can still estimate the hypergradient from trajectory samples. The following proposition follows from a similar anlysis as the policy gradient theorem.

Proposition 1. If f decomposes as in Equation (7) , then dF ( x ) dx can be expressed as follows:

where for compactness, we slightly abuse notation to express µ x,ξ ( s 0 ) as P x,ξ ( s 0 , a -1 , s -1 ) .

Here V x,ξ , Q x,ξ are the (unregularized) value and state action value functions with respect to r x,ξ . Comparing to Theorem 1, note that the expectation is over trajectories with starting states distributed according to the actual initial distribution µ x,ξ instead of some ν . We discuss how to construct estimators for (8) in Algorithm 5 (Appendix D) and prove unbiasedness in Proposition 4 (Appendix E.7). A special case of Equation (8) appeared in [15], where they consider model design for MDPs, that does not take into account contextual uncertainty or the possibility of multiple followers, i.e when the support of ξ is a singleton.

4 Accelerated HPGD with Full Lower-Level Access

Previously, we assumed that the leader does not know the solver used in the lower level and queries trajectories from an oracle. However, in certain settings, such as model design [15], and dynamic mechanism design [18], the leader can additionally influence how the followers solve the CMDP. In this section, we focus on the case when the followers use a stochastic tranining procedure, which usually require a polynomial number of steps in terms of δ -1 , to learn the optimal lower-level policy. We argue that if the leader has influence on the followers' training procedure, we can greatly reduce the number of lower-level iterations.

Let us assume that the lower level is solved using vanilla soft Q-learning (Algorithm 4 in Appendix D). According to Proposition 6 (Appendix E.7), the follower needs to run T = O ( K 2 K ) iterations

O

O

to ensure that E ∥ π T x,ξ -π ∗ x,ξ ∥ 2 ∞ ≤ 2 -K and thus ∥ ∥ ∥ E [ dF ( x ) dx -̂ dF T dx ]∥ ∥ ∥ ∞ = O (2 -K/ 2 ) , where π T

denotes the learned policy after running T -th Q-learning iterations and ̂ dF T dx denotes the corresponding hypergradient estimator.

To reduce the lower-level iteration complexity, we propose a randomized early stopping scheme over the lower-level soft Q-learning iterations, denoted as randomly-truncated soft Q-learning (RT-Q). The pseudocode is given in Algorithm 8 (Appendix E.5). We illustrate the high-level idea below.

Without loss of generality, consider a subsequence t k := O ( k 2 k ) such that t K := T . Let d dx F T denote the hypergradient estimator, based on the T -th policy iterate π T . It holds that:

where p k denotes a truncated geometric distribution, such that p k ∝ 2 -k . The above shows that d dx F t 1 + p -1 k [ d dx F t k +1 -d dx F t k ] with k ∼ p k is an unbiased estimator of d dx F T . Using this estimator, the follower does not need to run O ( K 2 K ) soft Q-learning iterations but in expectation only ∑ K -1 k =1 p k t k = O ( K 2 ) iterations. This implies that if the leader can direct how the followers learn and observe behaviors sampled from their learned policies, we can generate a hypergradient estimator with the same bias as d dx F T but a much smaller lower-level iteration complexity. We formalize our results in the following Theorem.

Theorem 4 (Improved iteration complexity using RT-Q) . Using Randomly Truncated soft Q-learning (RT-Q) instead of vanilla soft Q-learning to estimate the hypergradient, we achieve the bias, variance, and lower-level iteration complexity results summarized in Table 2.

The idea has been previously studied for contextual bilevel optimization under the name randomly truncated multilevel Monte-Carlo [29, 36, 37]. The reduction in the iteration complexity generally comes at the expense of an increased variance of the hypergradient estimator. In [37], this increase is logarithmic as the lower-level problem is a static optimization problem and samples generated to estimate the hypergradient are independent from the lower-level decision variable. This structure is crucial for controlling the increased variance of the hypergradient estimator. However, for CB-RL, rollouts generated from π t x,ξ are used to estimate the hypergradient. These trajectory samples thus depend on the lower-level decision and it is not possible to control the variance as in [37]. To address this issue, we notice that the major source of randomness in our hypergradient estimators stems from the estimator ̂ ∂ x A π t k ( s, a ) computed by GradientEstimator (Algorithm 2). We control this randomness by sampling multiple trajectories with a random length, which is in expectation O (1) . We further sample an action a once from π t k +1 and then use it to compute both ̂ ∂ x A π t k +1 ( s, a ) and ̂ ∂ x A π t k ( s, a ) , using importance sampling. Combining all these tricks with multi-level Monte Carlo, RT-Q achieves a variance of O ( K ) , where K = O (log( δ -1 )) .

5 Numerical Experiments

We illustrate the performance of HPGD in the Four-Rooms environment and on the Tax Design for Macroeconomic Model problem [34, 15] that we extend to multiple households with diverse preferences. We use Adaptive Model Design (AMD) [15] and a zeroth-order gradient approximation algorithm for comparison. Details on the zeroth-order algorithm are deferred to Appendix F.1.2. We note that AMD is not directly applicable to CB-RL as it was designed for solving an MDP without

S

G

1

G

2

+1

0

2000

4000

6000

8000

10000

Outer iterations:

T

0

.

4

0

.

6

0

.

8

1

.

0

Upper-level objective:

F

HPGD

AMD

Zero-order

Figure 1: Four-Rooms State Space and Performance. Left : S denotes the start state while G 1 and G 2 denote goal states that are considered separate tasks. +1 denotes the target cell to which the upper-level aims to steer the lower-level MDP. Right: HPGD escapes local optima achieving higher performance than comparison algorithms.

context. We apply it with modifications described in Appendix F.1.1. To our knowledge, such a zeroth-order gradient method is also the first of its kind for CB-RL. The main distinction between the algorithms is the zeroth-order algorithm requires two oracle queries for each gradient calculation while HPGD and AMD require only one. However, the zeroth-order method only needs to observe the function value of the upper level while the latter two require first-order information about the lower-level contextual MDP. In particular, AMD assumes complete access to the MDP to calculate the exact hypergradient while HPGD relies only on trajectory samples. Technical details about the implementation 3 are deferred to Appendix (F.2).

5.1 Four-Rooms Environment

Figure (1a) depicts the lower-level CMDP for the Four-Rooms environment. S denotes the initial position while G 1 and G 2 are goal states. We consider the two goal states as separate tasks and define ξ in Equation (4) to be the uniform distribution over the set of tasks, i.e., ξ ∼ Uniform( { 1 , 2 } ) . We denote the goal state in each task by G ξ . The state space S is defined by the cells of the grid world while the actions are the movements in the four directions. In each step t , with probability 2 / 3 , the agent moves to s t +1 following the chosen direction a t while it takes a random movement with probability 1 / 3 . The reward is always zero except when s t = G ξ where r ( s t , a t ) = 1 , and the episode resets. To incentivize taking the shortest path, we set the discount factor as γ = 0 . 99 .

For the upper level, we let x parameterize an additive penalty function ˜ r x : S × A → [ -0 . 2 , 0 . 0] 4 , such that the follower receives a reward of r + ˜ r x , as in the Principal-Agent problem [8]. The goal of the leader is to steer the followers through the cell marked with +1 in Figure 1a, denoted by s +1 , while keeping the penalties allocated to states to their minimum. We define r in Equation (7) as

where 1 is the indicator function and the second term defines the cost associated with implementing the penalties for the lower level. Note that there is a trade-off between the terms in r depending on the context variable ξ . If ξ = 2 , the desired change in the follower's policy can be achieved with small interventions since the shortest path from S to G 2 is already going through the bottom-left room. When ξ = 1 , the leader must completely block the shortest path from S to G 1 to divert the follower through the desired state. An efficient algorithm for this CB-RL problem therefore must avoid the local optimum of setting ˜ r = 0 and find the balance between the follower visiting state s +1 and implementing too large penalties in the CMDP.

Figure (1b) depicts the upper-level's objective function over the learning iterations t with hyperparameters λ = 0 . 001 and β = 1 . 0 . HPGD outperforms both AMD and the Zero-Order algorithms

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

↑

一

in this instance in terms of overall performance. The major difference in their performances is that HPGD successfully escapes the local optimum of ˜ r = 0 after about 5000 steps and assigns all the additive penalty budget to states in the gridworld. On the contrary, AMD and Zero-Order converge to the local optimum of minimizing the implementation penalty term in r . They only utilize 38% and 26% of the available budget of -0 . 2 to divert the follower when ξ = 2 but neglect the goal state G 1 .

Figure 2 shows the value of additive penalties ˜ r in the state space with the highest probability paths for the goal states. HPGD successfully blocks the follower when ξ = 1 and diverts its shortest path from S to G 1 along the other rooms, while AMD and Zero-Order fail to assign sufficient penalty to the upper corridor to cause the same effect. All algorithms are successful in ensuring that the shortest path through the bottom-left room is going through the marked state.

The parameters λ and β were chosen for demonstration purposes to highlight the capability of HPGD to escape local minima, as has been observed for SGD [72]. However, we emphasize that in the majority of the cases, the three algorithms perform equally as shown in Table 3. We provide the figures for the remaining hyperparameters in Appendix F.2.3. The slightly higher performance of AMD and low standard error among initializations is expected since this algorithm calculates the gradient of f deterministically while HPGD and Zero-Order rely on stochastic estimates yielding more variations, especially for the Zero-Order approach.

5.2 Tax Design for Macroeconomic Models

We test HPGD on a bilevel macroeconomic model evaluating taxation schemes based on [34, 15]. The economic model consists of a finite set of consumption goods i ∈ { 1 , . . . , M } each with price p i . We choose M = 3 with unit prices. On the lower-level, at each time step t , s t denotes the accumulated assets of a household which in turn chooses the number of hours worked n t

0

200

400

600

800

1000

Outer iterations:

T

35

40

45

50

Upper-level objective:

F

HPGD

Zero-order

0

200

400

600

800

1000

Outer iterations:

T

0

.

2

0

.

4

0

.

6

0

.

8

Tax rates

HPGD

x

HPGD

y

1

HPGD

y

2

HPGD

y

3

Zero-order

x

Zero-order

y

1

Zero-order

y

2

Zero-order

y

3

and consumption c i,t . The environment updates the accumulated assets of the household as s t +1 = s t +(1 -x ) wn t -∑ M i =1 c i,t + ϵ where ϵ is sampled from a normal distribution with mean 0 and standard deviation ς . We clip the accumulated assets to the range [ -100 , 100] for numerical stability. The utility of a household is given by u t = σ ( s t ) -θn 2 t + ∏ M i =1 ( c i,t / ( p (1+ y i ))) α i where the product-ofconsumption term corresponds to the Cobb-Douglas function [57] and σ ( · ) is the value of accumulated assets. We define P ξ as the distribution of households representing different socio-economic groups in the economy. In particular, we define two equal-sized groups with α = (0 . 6 , 0 . 3 , 0 . 1) and (0 . 1 , 0 . 7 , 0 . 2) that model their different preferences over the goods in the economy and optimize their consumption behavior using regularized deep Q-learning [26] with λ = 0 . 2 . On the upper-level, a tax designer is optimizing the discounted sum of social welfare by setting the income tax x ∈ [0 , 3] and value-added tax y i ∈ [0 , 3] for i ∈ { 1 , . . . , M } . In each time step t , the social welfare is defined as v t = ω ( s t ) + ∑ M i =1 c i,t / (1 + y i ) + ϕ log( ∑ M i =1 c i,t y i / (1 + y i ) + wxn t ) where ω ( · ) is the utility of accumulated assets and ϕ is a positive hyperparameter.

Figure 3a demonstrates that HPGD can successfully optimize the hyperparameters even in continuous complex problems. Benefiting from first-order information, HPGD converges faster than the ZeroOrder algorithm and increases the social-welfare by about 25% . Additionally, HPGD shows better qualitative results for the optimised tax rates in Figure 3b. HPGD swiftly increases the income tax to improve its objective by increasing tax revenues while sets value-added tax rates according to the preferences of the household groups. Households on average prefer the second good the most, therefore, y 2 is set low to increase consumption and therefore maximize the second term in v t while the third good is the least preferred for which the tax rate is increased 5 since consumption is already low. While Zero-Order shows equivalent performance on Figure 3a to HPGD it fails to distinguish goods when setting value-added tax rates. We defer further details on the implementation, hyperparameter choices, and additional results to Appendix F.3.1.

6 Conclusion

We introduce CB-RL, a class of stochastic bilevel optimization problems with lower-level contextual MDPs that capture a wide range of important applications, where a leader aims to design environments and incentive structures that align the followers' policies with the leader's upper-level objective. We propose HPGD, an oracle-based algorithmic framework, and analyze its convergence. Importantly, HPGD works with any existing algorithm that solves the lower-level CMDP to near-optimality, making it suitable in various regimes when the leader can only observe trajectories of followers. Moreover, HPGD is the first provably convergent algorithm in this area, which uses stochastic estimates of the hypergradient. We further propose RT-Q, a more efficient algorithm and study its bias, variance, and cost when the leader can fully control the followers' training. Numerical results further validate the expressiveness of the proposed model and the performance of our algorithm. Future directions include 1) applying HPGD in various real-world applications, 2) studying the setting when the lower-level problem is a game, and 3) studying single-loop algorithms for bilevel reinforcement learning with when the lower-level is just an MDP.

References

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199(1):165-214, 2023. (Cited on page 5.)

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine , 34(6):26-38, 2017. (Cited on page 1.)

Kavosh Asadi and Michael L. Littman. An alternative softmax operator for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 243-252, Sydney, NSW, Australia, 2017. JMLR.org. (Cited on page 49.)

Jan Balaguer, Raphael Koster, Christopher Summerfield, and Andrea Tacchetti. The good shepherd: An oracle agent for mechanism design. In ICLR 2022 Workshop on Gamification and Multiagent Solutions , 2022. (Cited on page 19.)

Jonathan F Bard. Practical bilevel optimization: algorithms and applications , volume 30. Springer Science & Business Media, 2013. (Cited on page 19.)

Tobias Baumann, Thore Graepel, and John Shawe-Taylor. Adaptive mechanism design: Learning to promote cooperation. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-7. IEEE, 2020. (Cited on page 19.)

Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028 , 2023. (Cited on page 21.)

Omer Ben-Porat, Yishay Mansour, Michal Moshkovitz, and Boaz Taitler. Principal-agent reward shaping in mdps. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 9502-9510, 2024. (Cited on pages 2, 8, 20, and 21.)

Mathieu Blondel and Vincent Roulet. The elements of differentiable programming, 2024. (Cited on page 46.)

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. (Cited on page 8.)

Seth Brown, Saumya Sinha, and Andrew J. Schaefer. Markov decision process design: A framework for integrating strategic and operational decisions. Operations Research Letters , 54:107090, May 2024. (Cited on page 19.)

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):25632578, July 2022. (Cited on page 52.)

Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang. PARL: A unified framework for policy alignment in reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations , 2024. (Cited on pages 2, 3, 6, 19, and 20.)

Arthur Charpentier, Romuald Elie, and Carl Remlinger. Reinforcement learning in economics and finance. Computational Economics , pages 1-38, 2021. (Cited on page 1.)

Siyu Chen, Donglin Yang, Jiayang Li, Senmiao Wang, Zhuoran Yang, and Zhaoran Wang. Adaptive model design for markov decision process. In International Conference on Machine Learning , pages 3679-3700. PMLR, 2022. (Cited on pages 2, 3, 4, 5, 6, 7, 9, 17, 19, 20, 21, 46, and 52.)

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems , 34:25294-25307, 2021. (Cited on page 19.)

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. (Cited on page 20.)

Michael Curry, Vinzenz Thoma, Darshan Chakrabarti, Stephen McAleer, Christian Kroer, Tuomas Sandholm, Niao He, and Sven Seuken. Automated design of affine maximizer mechanisms in dynamic settings. Proceedings of the AAAI Conference on Artificial Intelligence , 38(9):9626-9635, March 2024. (Cited on pages 2, 6, 19, and 21.)

Michael Curry, Alexander Trott, Soham Phade, Yu Bai, and Stephan Zheng. Learning solutions in large economic networks using deep multi-agent reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , AAMAS '23, page 2760-2762, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems. (Cited on pages 1, 2, and 19.)

Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: Convergent reinforcement learning with nonlinear function approximation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1125-1134. PMLR, 10-15 Jul 2018. (Cited on pages 19 and 48.)

Stephan Dempe. Foundations of bilevel programming . Springer Science & Business Media, 2002. (Cited on page 19.)

Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. (Cited on page 19.)

Manfred Diaz, Charlie Gauthier, Glen Berseth, and Liam Paull. Generalization games for reinforcement learning. In ICLR Workshop on Agent Learning in Open-Endedness , 2022. (Cited on page 19.)

Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3133-3144. PMLR, 13-18 Jul 2020. (Cited on pages 4 and 19.)

Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , AAMAS '18, page 122-130, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems. (Cited on page 19.)

Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning , pages 2160-2169. PMLR, 2019. (Cited on pages 4, 10, and 19.)

Matthias Gerstgrasser and David C. Parkes. Oracles & followers: Stackelberg equilibria in deep multi-agent reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the

40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 11213-11236. PMLR, 23-29 Jul 2023. (Cited on pages 2 and 19.)

Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018. (Cited on pages 4, 5, 19, and 34.)

Michael B. Giles. Multilevel monte carlo methods. Acta Numerica , 24:259-328, 2015. (Cited on pages 3 and 7.)

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 1352-1361, Sydney, NSW, Australia, 2017. JMLR.org. (Cited on page 49.)

Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca D. Dragan. Inverse reward design. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 6768-6777, Red Hook, NY, USA, 2017. Curran Associates Inc. (Cited on page 19.)

Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.02259 , 2015. (Cited on pages 2 and 3.)

Ben Hambly, Renyuan Xu, and Huining Yang. Recent advances in reinforcement learning in finance. Mathematical Finance , 33(3):437-503, 2023. (Cited on page 1.)

Edward Hill, Marco Bardoscia, and Arthur Turrell. Solving heterogeneous general equilibrium economic models with deep reinforcement learning. arXiv preprint arXiv:2103.16977 , 2021. (Cited on pages 1, 2, 7, 9, and 20.)

Yifan Hu, Xin Chen, and Niao He. Sample complexity of sample average approximation for conditional stochastic optimization. SIAM Journal on Optimization , 30(3):2103-2133, 2020. (Cited on page 34.)

Yifan Hu, Xin Chen, and Niao He. On the bias-variance-cost tradeoff of stochastic optimization. In Advances in Neural Information Processing Systems , volume 34, pages 22119-22131, 2021. (Cited on page 7.)

Yifan Hu, Jie Wang, Yao Xie, Andreas Krause, and Daniel Kuhn. Contextual stochastic bilevel optimization. Advances in Neural Information Processing Systems , 36, 2024. (Cited on pages 2, 3, 7, 19, and 27.)

Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning. Advances in Neural Information Processing Systems , 33:2759-2770, 2020. (Cited on page 34.)

Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15931-15941. Curran Associates, Inc., 2020. (Cited on page 19.)

Jiawei Huang, Vinzenz Thoma, Zebang Shen, Heinrich H. Nax, and Niao He. Learning to steer markovian agents under model uncertainty, 2024. (Cited on page 19.)

Dima Ivanov, Paul Dütting, Inbal Talgam-Cohen, Tonghan Wang, and David C. Parkes. Principalagent reinforcement learning, 2024. (Cited on pages 2, 20, and 21.)

Sham Kakade. A natural policy gradient. In Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic , NIPS'01, page 1531-1538, Cambridge, MA, USA, 2001. MIT Press. (Cited on page 52.)

Prashant Khanduri, Siliang Zeng, Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in neural information processing systems , 34:30271-30283, 2021. (Cited on page 19.)

Jeongyeol Kwon, Dohyun Kwon, and Hanbaek Lyu. On the complexity of first-order methods in stochastic bilevel optimization. arXiv preprint arXiv:2402.07101 , 2024. (Cited on page 19.)

Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. A fully first-order method for stochastic bilevel optimization. In International Conference on Machine Learning , pages 18083-18113. PMLR, 2023. (Cited on page 19.)

Joshua Letchford and Yevgeniy Vorobeychik. Optimal interdiction of attack plans. In AAMAS , pages 199-206. Citeseer, 2013. (Cited on page 2.)

Boyi Liu, Jiayang Li, Zhuoran Yang, Hoi-To Wai, Mingyi Hong, Yu Nie, and Zhaoran Wang. Inducing equilibria via incentives: Simultaneous design-and-play ensures global convergence. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 29001-29013. Curran Associates, Inc., 2022. (Cited on pages 4 and 19.)

Risheng Liu, Xuan Liu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A value-function-based interior-point method for non-convex bi-level optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6882-6892. PMLR, 18-24 Jul 2021. (Cited on page 19.)

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 6820-6829. PMLR, 13-18 Jul 2020. (Cited on pages 19, 43, 48, 51, and 52.)

Alberto Maria Metelli, Mirco Mutti, and Marcello Restelli. Configurable markov decision processes. In International Conference on Machine Learning , pages 3491-3500. PMLR, 2018. (Cited on pages 2, 3, and 19.)

Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 2772-2782, Red Hook, NY, USA, 2017. Curran Associates Inc. (Cited on pages 4, 48, and 50.)

ATD Perera and Parameswaran Kamalaruban. Applications of reinforcement learning in energy systems. Renewable and Sustainable Energy Reviews , 137:110618, 2021. (Cited on page 1.)

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014. (Cited on page 1.)

Guannan Qu and Adam Wierman. Finite-time analysis of asynchronous stochastic approximation and q -learning. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 3185-3205. PMLR, 09-12 Jul 2020. (Cited on pages 49 and 50.)

Giorgia Ramponi, Alberto Maria Metelli, Alessandro Concetti, and Marcello Restelli. Learning in non-cooperative configurable markov decision processes. Advances in Neural Information Processing Systems , 34:22808-22821, 2021. (Cited on pages 2 and 19.)

R. Tyrrell Rockafellar and Roger J. B. Wets. Variational Analysis . Springer Berlin Heidelberg, 1998. (Cited on page 46.)

Aaron Roth, Jonathan Ullman, and Zhiwei Steven Wu. Watch and learn: Optimizing from revealed preferences feedback. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 949-962, 2016. (Cited on page 10.)

Han Shen, Zhuoran Yang, and Tianyi Chen. Principled penalty-based methods for bilevel reinforcement learning and rlhf. arXiv preprint arXiv:2402.06886 , 2024. (Cited on pages 2, 3, 6, 19, and 20.)

Laixi Shi, Eric Mazumdar, Yuejie Chi, and Adam Wierman. Sample-efficient robust multiagent reinforcement learning in the face of environmental uncertainty. arXiv preprint arXiv:2404.18909 , 2024. (Cited on page 19.)

Arunesh Sinha, Fei Fang, Bo An, Christopher Kiekintveld, and Milind Tambe. Stackelberg security games: looking beyond a decade of success. In Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages 5494-5501, 2018. (Cited on page 2.)

Heinrich von Stackelberg. Marktform und Gleichgewicht . Klassiker der Nationalökonomie. Verlag Wirtschaft und Finanzen, Düsseldorf, 1934. (Cited on page 19.)

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018. (Cited on page 1.)

Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. Advances in Neural Information Processing Systems , 33:12151-12162, 2020. (Cited on page 19.)

Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. (Cited on page 21.)

Jing Wang, Meichen Song, Feng Gao, Boyi Liu, Zhaoran Wang, and Yi Wu. Differentiable arbitrating in zero-sum markov games. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , AAMAS '23, page 1034-1043, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems. (Cited on pages 2 and 19.)

Kai Wang, Lily Xu, Andrew Perrault, Michael K. Reiter, and Milind Tambe. Coordinating followers to reach better equilibria: End-to-end gradient descent for stackelberg games. Proceedings of the AAAI Conference on Artificial Intelligence , 36(5):5219-5227, Jun. 2022. (Cited on page 19.)

Xu Wang, Sen Wang, Xingxing Liang, Dawei Zhao, Jincai Huang, Xin Xu, Bin Dai, and Qiguang Miao. Deep reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022. (Cited on page 1.)

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992. (Cited on page 51.)

Jibang Wu, Siyu Chen, Mengdi Wang, Huazheng Wang, and Haifeng Xu. Contractual reinforcement learning: Pulling arms with invisible hands, 2024. (Cited on pages 2, 20, and 21.)

Shuo Wu, Haoxiang Ma, Jie Fu, and Shuo Han. Robust reward design for markov decision processes, 2024. (Cited on page 19.)

Li Xia, Luyao Zhang, and Peter W. Glynn. Risk-sensitive markov decision processes with long-run cvar criterion. Production and Operations Management , 32(12):4049-4067, December 2023. (Cited on page 19.)

Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations , 2021. (Cited on page 9.)

Chang Yang, Yuiyu Wang, Xinrun Wang, and Zhen Wang. A game-theoretic perspective of generalization in reinforcement learning. In Deep Reinforcement Learning Workshop NeurIPS 2022 , 2022. (Cited on page 19.)

Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning to incentivize other learning agents. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15208-15219. Curran Associates, Inc., 2020. (Cited on page 19.)

Yan Yang, Bin Gao, and Ya xiang Yuan. Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity, 2024. (Cited on pages 3 and 19.)

Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR) , 55(1):1-36, 2021. (Cited on page 1.)

Guanghui Yu and Chien-Ju Ho. Environment design for biased decision makers. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 , pages 592-598. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track. (Cited on page 20.)

Haifeng Zhang, Jun Wang, Zhiming Zhou, Weinan Zhang, Ying Wen, Yong Yu, and Wenxin Li. Learning to design games: strategic environments in reinforcement learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence , IJCAI'18, page 3068-3074. AAAI Press, 2018. (Cited on pages 2, 5, and 19.)

Haoqi Zhang and David Parkes. Value-based policy teaching with active indirect elicitation. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 1 , AAAI'08, page 208-214. AAAI Press, 2008. (Cited on page 20.)

Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Ba¸ sar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization , 58(6):3586-3612, January 2020. (Cited on page 46.)

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control , pages 321-384, 2021. (Cited on page 19.)

Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C. Parkes, and Richard Socher. The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances , 8(18):eabk2607, 2022. (Cited on pages 2, 19, and 20.)

Contents

A Related Work

Stochastic bilevel optimization has been extensively explored in the literature [21, 5]. In recent years, there is a pivotal shift to non-asymptotic analysis of stochastic gradient methods [28, 16, 43, 45, 44, 48]. [37] propose contextual stochastic bilevel optimization where the lower level solves a static contextual optimization. Our work generalizes to the lower level solving a contextual MDP. This poses unique challenges in terms of hypergradient estimation and sample generation. Comparing to bilevel optimization, leveraging the special structure of CB-RL, we avoid Hessian and Jacobian estimation of the lower-level MDP when computing the hyper policy gradient, which is crucial for scalability.

Configurable MDP [50] is an extension of a traditional MDP allowing external parameters or settings to be adjusted by the decision-maker, often referred to as the configurator . Only recently some works studied the case where the configurator has a different objective than the agent [55]. However, that work assumes access to a finite number of parameters that the configurator can control, while our model goes beyond this assumption. In addition, our model captures the variability and uncertainty that the agent could face in the same configuration environment.

Stackelberg games are a game theoretic framework, where a leader takes actions to which one or multiple followers choose the best response [61]. Several existing lines of work have studied solving variants of Stackelberg games. Examples include Stackelberg equilibrium solvers [24, 27], opponent shaping [25, 74], mathematical programs with equilibrium constraints [47, 65, 66], inducing cooperation [6, 4], steering economic simulations [19, 82]. These works are either too general with limited implications for our problem or consider entirely distinct settings.

Multi-agent RL (MARL) studies multiple agents interacting in a joint environment, i.e., their actions together determine the next state [81, 59]. In CB-RL the lower level CMDPs can be seen as a special instance of MARL where the interactions of the followers are restricted to jointly influencing the decision of the leader.

Bilevel RL studies how to design additional rewards or change the underlying MDP to achieve desirable learning outcomes. Many applications are formulated as bilevel RL, such as environment design for generalization [22, 23, 73], reward shaping [31, 39, 40, 70], safe reinforcement learning [63], optmizing conditional value at risk[71], and model design [15, 78, 11]. Previous work on bilevel RL considers a special case of our setting when there is only one lower-level MDP [15, 13, 58, 75]. In particular, [15] focus on the case when the leader has control on the follower's training procedure. [58] further extend from one single lower-level MDP to a lower-level min-max game. [13] and the concurrent work of [75] focus on the case when the leader can only influence the reward of the MDP.

The introduction of the context makes CB-RL harder to solve as there can be many followers, each with its own preferences, and their best response policies change even for the same leader decision x when facing different contextual uncertainties. Multiple followers and additional side information are very common, which highlights the practical relevance of our work. In addition, the algorithms in the aforementioned works focus on deterministic updates on the upper and lower level decisions, i.e., assuming access to the full hypergradient and performing exact policy gradient/value iteration, which is both computationally hard and not feasible for large-scale practical applications. To the best of our knowledge, we are the first to provide a convergence analysis for the stochastic case, when the hypergradient is estimated from samples and the lower level uses a stochastic update rule.

B Applications: RLHF, Tax Design, Reward Shaping, Contract Design, and Dynamic Mechanism Design

We list several applications of CB-RL below. For a clearer exposition, we omit the entropy regularization term at the lower level. However, we stress that some of the referenced works explicit use entropy regularization [18, 15] or make overlapping assumptions such as unique optimal policies [13]. Additionally, for problems without explicit entropy regularization we refer to [15, 49, 20, 26] who have shown that entropy-regularized RL approximates the unregularized problem as λ → 0 both in the upper and lower level.

Reinforcement Learning from human feedback (RLHF) considers the setting where an agent tries to learn a task from human feedback. The difficulty stems from the fact that the human feedback is

given as preferences over two possible trajectories and not directly as a reward [17]. The feedback is of the form { τ 0 , τ 1 , l } where τ 0 , τ 1 are two trajectories and l ∈ 0 , 1 indicates whether τ 0 is preferred over τ 1 or vice versa. It has been shown that this problem can be framed as CB-RL, where the upperlevel tries to learn rewards that minimize the cross-entropy loss, between the actual and predicted labels, using the Bradley-Terry Model and the lower level finds the optimal policy with respect to that reward function [13, 58],

Here H is the time horizon. D is the sampling distribution of trajectories using π ∗ x,ξ and

Note in the case of standard RLHF the context becomes trivial.

Tax Design for Macroeconomic Modeling considers a public entity setting tax rates and representative households responding optimally by balancing their short-term utility of consumption and long-term wealth accumulation [34, 15, 82]. A potential formulation of this problem as CB-RL is

where x and y denote the income and value-added tax rates, respectively, and ϕ defines the social welfare objective of the leader. The state s t defines the wealth of a household while their actions decide their working hours and consumption in each time step. The reward function r ξ W and r C y,ξ define the households' utility functions for wealth and consumption, respectively. The value-added tax rate y affects the consumption utility function r C y,ξ while the income tax x changes the transition kernel modeling wealth accumulation, i.e., s t +1 ∼ P x,ξ ( · ; s t , a t ) . ξ represents the preferences of the households over several consumption goods and their productivity in this problem formulation.

Population Principal-Agent Reward Shaping considers a principal aiming to craft a non-negative bonus reward function r B x , parameterized by x , to motivate an agent [8, 77, 79]. Commonly, a principal faces multiple agents that form a distribution. Each agent has its own individual reward function r ξ . This scenario, termed population principal-agent reward shaping is captured by our CB-RL framework.

Here E ξ denotes the expectation over the distribution of agents and the trajectories. The policy π ∗ x,ξ ( · ) is the optimal response of the ξ -th agent to the composite reward function r ξ + r B x . The principal's reward is r ( s t , a t ) when the agent visits the state action pair ( s t , a t ) .

Dynamic Contract Design studied by [41, 69] is similar to the above reward shaping. Generalizing it to a contextual setting, the problem consists of an agent of type ξ that incurs a cost c ξ ( s, a ) for taking action a in state s . The principal in turn gets a reward r ( s, s ′ ) for transitioning from state s to s ′ , but cannot observe the agent's action. It can however offer contracts x ( s, s ′ ) that get paid if the MDP transitions from state s to s ′ . These contracts are positive payments by the principal and are thus added to the lower-level objective and substracted from the upper-level objective.

Dynamic Mechanism Design considers the problem of a mechanism designer controlling an MDP for a group of n bidders, who get a reward based on the observed trajectories [18]. The context ξ parameterizes the bidders' reward functions r i,ξ , which they report to the mechanism designer. The latter wants to learn a policy for the MDP and charge payments to the bidders, to ensure eliciting truthful reward reports and also maximizize an objective L , e.g. the total sum of payments. In this setting, [18] propose to search for such a mechanism within the class of affine maximizers, as they guarantee truthful reports by all bidders. In these mechanisms, a set of agent-dependent weights x w,i and state-action dependent boosts x b is chosen by the mechanism designer, then a policy π is learned to maximize the corresponding affinely transformed social welfare E s t ,a t ∼ π [ ∑ T t =0 ( ∑ n i =1 x w,i r i,ξ ( s t , a t )) + x b ( s t , a t ) ] and bidders are charged for the learned policy depending on their reported reward functions. Searching for the optimal mechanism parameters x w,i and x b to maximize L in expectation over ξ , subject to the constraint that the mechanism's policy maximizes affine social welfare can be formulated as CB-RL. In this case x w,i and x b are the decision variable, the context parameterizes the bidders' preferences and the affinely transformed social welfare at each time step is the reward function of the lower-level MDP, as shown below:

Note, that all previous works in these application areas have either focused on the setting with a single representative follower [8, 15, 41, 69] or presented a problem-specific algorithm that cannot capture our CB-RL framework in its full generality [8, 18].

The CB-RL framework is also related to Meta reinforcement learning (Meta RL) , that aims to leverage the similarity of several RL tasks to learn common knowledge and use it on new unseen tasks [7]. One way to formulate Meta RL problems is to find a common regularization policy ˜ π for multiple tasks.

where ξ represents the distribution of multiple RL tasks and r ξ is the reward for the task indexed by ξ . Although the upper-level regularizer is different from entropy-regularization it still results in a unique softmax policy of the form π ∗ s ξ ( s, a ) ∝ exp ( Q s ξ ,π ( s,a ) λ +log(˜ π ( s, a )) ) [64]. Moreover, in Meta RL the leader does not change the transitions or rewards, but the target policy ˜ π , that goes into the KL regularization term of the followers.

C Frequently-Used Notation

T

D Algorithms

We give the pseudocode to certain algorithms/routines/procedures mentioned in the main text.

Algorithm 2 GradientEstimator ( ξ, x, s, a, o )

Input: ξ , x state s , action a , trajectory oracle o

Output:

̂ ∂ x A ( s, a ) ← ̂ ∂ x Q ( s, a ) -̂ ∂ x V ( s )

Algorithm 3 Soft Value Iteration

Algorithm 4 SoftQlearning ( T, π B , { α t } t ≥ 0 )

2: Result: Approximation Q λ ≈ Q ∗ λ , policy π λ ≈ π ∗ λ

1: Input: Number of iterations T , Behavioural Policy π B , Stepsizes { α t } t ≥ 0

3: Initialize Q λ = 0

4: Initialise s 0

5: for t = 0 to T do

6: Sample a ∼ π B ( · ; s t )

8: Q λ ( s t , a ) = Q λ ( s t , a ) + α t ( r ( s t , a ) + γλ log ( ∑ a ′ ∈A exp ( Q λ ( s t +1 ,a ′ ) λ )))

7: Observe next reward r ( s t , a ) and state s t +1 ∼ P ( ·| s t , a )

9: end for

11: return Q λ and π o λ

10: π o λ ( a ; s ) ← exp( Q λ ( a | s ) /λ ) ∑ a ′ exp( Q λ ( s,a ′ ) /λ )

Algorithm 5 DecomposableGradientEstimator

Algorithm 6 Vanilla Policy Gradient Algorithm

E Proofs

E.1 Overview

In this section, we provide proofs for the presented theorems and propositions. We provide the proof of Theorem 1, deriving the hypergradient of function F ( x ) ; the proof of Theorem 2, deriving the derivative of the action-value function with respect to x ; the proof of our main result, Theorem 3, which shows convergence of HPGD to a stationary point of F ( x ) .

For the propositions, we show how to estimate the upper-level gradient if f is decomposable in the proof of Proposition 1; In Proposition 2 we show how to compute the gradient of the optimal policy with respect to x ; the proof of Proposition 3, which shows we can achieve unbiased estimates of the advantage hypergradient; and the proof of Proposition 4, which shows the same for the special case when f decomposes.

We state and proof Propositions 5 to 8 which show convergence in L 2 to the optimal policy of soft value iteration, soft Q-learning, Vanilla Policy Gradient and Natural Policy Gradient respectively.

Lastly, we prove Theorem 4, regarding the reduced iteration complexity of RT-Q claimed in Section 4.

E.2 Proof of Theorem 1

Proof. The proof relies on on three main ideas.

Show that Dominated Convergence applies, i.e. all derivatives are uniformly bounded by an integrable function. Thus we can exchange derivative and expectation and compute the derivative of f instead of F .

Use Proposition 2 to get an expression for the derivative of the optimal policy with respect to x .

Use importance sampling with some distribution ν to get an expression of the hypergradient that we can cheaply sample, instead of having to multiply two matrices with size |S| × |A| .

By Proposition 2, it follows that

As the partial derivatives of f are bounded by L f (cf. Assumption 3.1), we can apply the Dominated Convergence Theorem to get

where we use Proposition 2 for eq. (9) and importance sampling with any distribution ν in the last equality, as long as ν has full support on S . Further, we note that ∂ 2 f ( x,π ∗ x,ξ ,ξ ) ∂π ∗ x,ξ ∈ Mat 1 , | S |×| A | ( R )

and ∂π ∗ x,ξ ∂x ∈ Mat | S |×| A | ,d ( R ) . Hence, we just explicitely write out the matrix multiplication for the second equality. The second equality follows from the multivariate chain rule and the first equality from the Dominated Convergence Theorem.

E.3 Proof of Theorem 2

Proof. Theorem 2 is important as ∂ x A π λ,x,ξ ( s, a ) and thus ∂ x Q π λ,x,ξ ( s, a ) allow us to compute dπ ∗ x,ξ dx (cf. Proposition 2). We will prove the theorem using an induction proof, which follows the analysis of the policy gradient theorem. However, instead of at each timestep π ( a t ; s t ) depending on a policy parameter θ , it will be the transition P x,ξ ( s t +1 ; s t , a t ) and reward r x,ξ ( s t , a t ) depending on x . Also note that we only consider the partial derivative with respect to x , evaluated at some policy π , which in the case of π ∗ x,ξ

In the following, we will show by induction that

where p x,ξ ( s, a → s ′ , a ′ ; t, π ) is the probability that the Markov Chain induced by π , starting from s, a reaches s ′ , a ′ after t steps. Note, the formulation is equivalent to the one stated in Theorem 2.

The proof follows the analysis of the standard policy gradient theorem. We drop here the dependence on x and ξ to simplify the notation. Assuming that Q π λ ( s, a ) is differentiable for all s, a , we show by induction that for all n ∈ N it holds that

The claim then follows as n →∞ .

Base case ( n = 0) It is easy to check that

We use the definition of Q π λ ( s, a ) in the first equality. The second follows by the product rule. The third equality follows by the definition of the value function. The last equality comes from rearranging terms.

Induction step ( n = ⇒ n +1) Assuming eq. (10) holds for n we show it holds for n +1 :

The first equality is simply the base case. The second equality follows from the definition of the Q-function. The third equality follows from the multivariate chain rule and the definition of the value function and the last equality is again rearranging terms.

E.4 Proof of Theorem 3

Proof. By the smoothness of f , we use the following bound from [37][Lemma 1] for α ≤ 1 / (2 S f ) :

The error term naturally decomposes into an initial error divided by T (1) , a bias term (2) , and a variance term (3) , which decreases with the stepsize α .

For (1) we do not need to simplify any further.

To prove our claim, we need to show that (3) is O (1) , and that (2) is O ( δ ) . The latter is the main challenge of this proof.

Let us begin by bounding the bias term (2) . The goal is to show that it can be upper bounded by a sum of terms, which are all linear in ∥ ∥ ∥ π o x,ξ -π ∗ x,ξ ∥ ∥ ∥ ∞ .

where the first equality is by definition and the first inequality follows from the trinalge inequality and we further use the fact that ̂ d dx A π o x t ,ξ λ,x,ξ is an unbiased estimator of d dx A π o x t ,ξ λ,x,ξ as shown in Proposition 3. (A) is relatively easy to bound. Indeed by the smoothness of f ( Assumption 3.1), it immediately follows that

To bound (B) we again use the triangle inequality to decompose:

(a) is relatively easy to bound. We have

where we use the assumption on the oracle in the first inequality, that f is Lipschitz continuous in the second inequality, and that ∥ ∥ ∥ ∂ x V π ∗ x,ξ λ,x,ξ ( s ) ∥ ∥ ∥ ∞ ≤ ∥ ∥ ∥ ∂ x Q π ∗ x,ξ λ,x,ξ ( s, a ) ∥ ∥ ∥ ∞ in the third inequality. Note that:

since the entropy of any policy is bound by log | A | (which follows from Jensen's inequality). Using the definition that

it thus holds that

Letting m := min s ν ( s ) , we thus have

For (b) we further simplify using the triangle inequallity:

Similar to (a) , we can bound (i) using the smoothness of f (Assumption 3.1):

Bounding (ii) and in particular ∥ ∥ ∥ ∂ x A π ∗ x t ,ξ λ,x,ξ ( s, a ) -∂ x A π o x t ,ξ λ,x,ξ ( s, a ) ∥ ∥ ∥ ∞ is the tricky part of this proof. We first need to show two intermediate results. First, we bound the difference in entropy between two policies and the difference in the regularized value functions of two policies. Once we have

that, we can tackle ∥ ∥ ∥ ∂ x A π ∗ x t ,ξ λ,x,ξ ( s, a ) -∂ x A π o x t ,ξ λ,x,ξ ( s, a ) ∥ ∥ ∥ ∞ . For the entropy, we denote by l 1 := min s,a,x,ξ π ∗ x,ξ ( a ; s ) the minimum probability of playing an action in any state under the optimal policy. Recall that ∀ x, ξ, s, a : | r x,ξ ( s, a ) | < R and that 0 ≤ H ( π ; s ) ≤ log |A| (by Jensen's inequality). Thus we have

Because π ∗ x,ξ ( s ; a ) ∝ exp( Q ∗ λ,x,ξ ( s, a ) /λ ) , it follows that

We assume now that δ is sufficiently small, i.e. δ ≤ l 1 / 2 , such that l 1 / 2 ≤ min s,a,x,ξ π o x,ξ ( a ; s ) . We need to have such a lower bound on the policies, touse the fact that the log function is Lipschitz continuous with parameter 1 a on an interval [ a, ∞ ) for any a > 0 . Hence we have

We use this to bound the difference in the value functions of π ∗ x,ξ and π o x,ξ .

The first inequality follows from the definition of the regularized value function and the triangle inequality. The second inequality follows from our derived bound on the difference of entropies. The third inequality is agian using the triangle inequality and the fourth inequality is bounding the Q-function using that the rewards are upper bounded by R and that Q π λ,x,ξ ( s, a ) = r x,ξ ( s, a ) + γ E s ′ ∼ P x,ξ ( · ; s,a ) [ V π λ,x,ξ ( s ′ ) ] . The last inequality then follows by iteratively plugging in the inequality for the term ∥ ∥ ∥ V π ∗ x,ξ λ ( s ) -V π o x,ξ λ ( s ) ∥ ∥ ∥ ∞ , which gives a geometric sum. We employ a similar technique to bound (ii) using the above results:

Here the first inequality follows from the Lipschitz continuity of f . For second inequality we use the definition of the advantage function, the triangle inequality and the following inequality:

where the last inequality follows from the trianlge inequality and Equation (12).

We bound the difference in Q-function derivatives as follows:

where the dots indicate multiplication over the linebreak. The first equality follows from plugging in the result from Theorem 2. The second equality follows by taking out all terms with t = 0 . The first inequality uses the triangle inequality. The second inequality plugs back in the definition from Theorem 2. The last inequality follows from the triangle inequality again.

Taking the expectation, we thus get:

where we use the intermediate result from before to bound the difference between the value functions, the upper bound on the Q-function derivative shown in Equation (12) and the assumption on the oracle to get the first inequality. The second inequality follows from the the resulting geometric sum and the last equality is just rearranging terms to show the linearity in δ .

Using this result, we can now bound (ii) :

With that we are done decomposing (2) . Combining everything, we have the following bound:

With that we have tackled terms (1) and (2) in Equation (11). It remains to bound the variance, i.e. term (3) . If we can show that (3) = O (1) then the last term in Equation (11) is O ( α ) as claimed and we are done. Indeed, bounding (3) is relatively easy, as all important terms are bounded by Assumption 3.1. We have:

where the third inequality follows directly, the second inequality uses the definition of the variance and the first inequality uses that

By the above, it suffices to bound the second moment:

where T Q is the random variable defined in Algorithm 2. The first inequality uses the definition of the advantage estimate (cf. Algorithm 2), the i.i.d. property of T Q and T V , as well as of T ′ Q and T ′ V , and Equation (16). The second inequality follows from Assumption 3.1.

Plugging in the above, we thus get:

where we repeatedly apply Equation (16) and the fact that the second moment of a geometric random variable is finite.

Now we can plug all our bounds back into Equation (11) to get the result of Theorem 3.

E.5 Proof of Theorem 4

Vanilla soft Q-learning

We give a brief overview of how HPGD is combined with vanilla soft Q-learning (Algorithm 4) to get a bias of O (2 -K/ 2 ) in Algorithm 7. In the algorithm we refer to t K = O ( K 2 K ) as the number of iterations soft Q-learning needs to achieve λ E ∥ π t K -π ∗ ∥ 2 ∞ ≤ E ∥ Q t K -Q ∗ λ ∥ 2 ∞ ≤ 2 -K (cf. Proposition 6). Note we slightly abuse notation, when we pass a policy instead of an oracle

to Algorithm 2. However, the policy can be equivalently used to sample trajectories. Note that the idea of HPGD with soft Q-learning is a double-loop methods that aims to find an δ -oracle. Similar a δ -oracle idea also appears in conditional stochastic optimization [38, 35], in bilevel optimization [28].

Algorithm 7 HPGD with vanilla soft Q-learning

Input: Iterations T , Precision param. K , Learning rate α , Regularization λ , Initial point x 0 , behavioural policy π B , Q-learning rates { α t } t ≥ 0

Randomly-Truncated soft Q-learning (RT-Q)

Let us now turn to RT-Q, for which we provide the pseudocode in Algorithm 8. As above, we denote by t k = O ( k 2 k ) the number of iterations soft Q-learning needs to achieve λ E ∥ π t k -π ∗ ∥ 2 ∞ ≤ E ∥ Q t k -Q ∗ λ ∥ 2 ∞ ≤ 2 -k . We slightly abuse notation in the Pseudocode, such that we do not just return the last soft Q-learning iteration but also the second last and the first. Moreover we denote by p k = 2 -k 1 -2 -K and we generally use ̂ x to denote estimates from a single sample and ˜ x to denote averaged estimates from multiple samples.

Algorithm 8 HPGD with RT-Q

Input: Iterations T , Precision param. K , Learning rate α , Regularization λ , Initial point x 0 , behavioural policy π B , Q-learning rates α t t ≥ 0

Proof. First let us specify that 'bias' refers to the bias of the hypergradient estimator, i.e.

Equivalently for 'variance' we mean the variance of the estimator i.e.

Moreover, the 'iteration complexity' is the number of soft Q-learning iterations needed to solve the lower level.

For Vanilla soft Q-learning, we can just combine previous results to compute the iteration complexity and variance to achieve a bias of 2 -K/ 2 . For RT-Q, we formalize the intuition of Section 4 to show we can achieve the same bias with only O ( K 2 ) iterations because we rarely perform many soft Q-learning iterations but assign a relatively higher magnitude to these few accurate hypergradient estimates. This necessarily increases variance and most of the proof will be spent on how to bound it. Here we 'divide and conquer' the variance until we have easy terms that depend linearly on

or related terms we can easily bound. To decompose the variance, our main tool will be the following identity

which we have already derived and used in the proof of Theorem 3 and also follows from the Parallelogram Law.

Vanilla soft Q-learning

Westart with the analysis of using HPGD with vanilla soft Q-learning to estimate dF ( x ) dx . In Theorem 3, we showed that where

In Proposition 6, we show that after t K = O ( K 2 K ) soft Q-learning iterations, it holds that

where π t K x,ξ denotes the t K -th iterate of the soft Q-learning algorithm. The results for complexity and bias follow directly. It remains to bound the variance. However, we have already shown in Theorem 3 that

Randomly-Truncated soft Q-learning (RT-Q)

We first show that dF RT t K dx has the same bias as its vanilla counterpart ̂ dF t K dx . For this, observe that the following estimators have the same mean:

The first equality holds because of importance sampling and the second inequality holds by the linearity of expectation. Plugging in these identities, we get

It follows directly that the hypergradient estimators obtained by RT-Q and vanilla soft Q-learning must have the same bias.

For a sampled k ∈ { 1 , . . . , K } , soft Q-learning has an iteration complexity c k of O ( k 2 k ) to build the following hypergradient estimator:

As we sample any k with probability p k = 2 -k 1 -2 -K , the expected iteration complexity is then given by

which proves our claim. The attentive reader will note that RT-Q runs GradientEstimator 2 k times instead of once and thus samples 2 k trajectories to estimate the advantage derivative instead of one like vanilla soft Q-learning. Nonetheless, RT-Q has the better sample complexity as both methods need to sample O ( k 2 k ) state action pairs for a given k to run soft Q-learning and then estimate the advantage derivative. The same analysis as for the iteration complexity thus shows that the sample complexity of vanilla soft Q-learning is O ( K 2 K ) and O ( K 2 ) for RT-Q.

It remains to show that the variance is of order O ( K ) . This is the most challenging part of the proof. As outlined previously, we will iteratively decompose the variance until we can bound all terms by O ( K ) .

For better readability we introduce the following notation for a given pair s, a :

Then for a given ξ, s, a (which are sampled at the beginning of RT-Q) we have that

Now let us decompose the variance of the RT-Q hypergradient estimator using the newly introduced notation:

We proceed to bound the individual terms.

Note that (3) is indepedent of k . Using Equation (15) in the analysis of Theorem 3, we can bound it as

(1) is also relatively easy to bound, as shown below:

In the first equality we simply use that k is sampled with probability p k = 2 -k 1 -2 -K . In the second equality we plug in the definitions of H k +1 (1) and H k (1) . In the first inequality we use the smoothness of f . The second inequality uses Equation (16). The third inequality follows from the fact that t k +1 is chosen to guarantee an expected distance of at most 1 2 k +1 to π ∗ x,ξ . The remaining equalities follow from plugging in p k and rearranging terms.

Now we want to repeat the same analysis again for (2) . As for (1) we get a sum over k with the factor 1 p k , which we need to compensate by bounding E [ ∥ ∥ H k +1 (2) -H k +1 k (2) ∥ ∥ 2 ∞ ] by O (2 -k ) . However, bounding the latter is more invovled than our analysis for (1) . In the following pages, we will iteratively apply Equation (16) until the terms get easy enough, such that we can use one of the following two facts about RT-Q:

2. ˜ ∂ x A π t k +1 ( s, a ) and related terms are an average estimate over 2 k trajectory samples, such that their variance (with respect to these random rollouts) is O ( 1 2 k ) .

We briefly note that in the analysis below we will sometimes write Q ( s, a ) or A ( s, a ) inside the infinity norm for ease of exposition. When we do so, the infinity norm is still interpreted as the maximum over all possible s, a and not as the absolute value of the Q-function or advantage for a specific s, a .

Let us start decomposing the numerator of (2) :

The first equality is just plugging in definitions. The first inequality follows from Equation (16) and the second inequality comes from the Cauchy-Schwarz inequality.

Let us begin with bounding (i) . First we want to tackle the fractions of the policies. For this, recall that ∀ x, ξ, s, a : | r x,ξ ( s, a ) | < R and that 0 ≤ H ( π ; s ) ≤ log |A| (by Jensen's inequality). Thus we have that:

The above bounds extend to any soft Q-learning estimate, which can be obtained by Algorithm 4, since by Assumption 3.1 the algorithm cannot observe rewards with greater magnitude than R . As Algorithm 4 returns a softmax policy, i.e.

it holds that:

Therefore we have:

Using this we can bound (i) . Let m = min s ν ( s ) , then:

The first inequality uses Equation (15). The second inequality uses Equation (16) and the last equality uses the convergence of soft Q-learning.

Now we turn to bound (ii) :

The first inequality uses M 2 to bound the policy fraction the definition of m = min s ν ( s ) and Cauchy-Schwarz. The second inequality uses Equation (16) and Cauchy-Schwarz. For the final inequality, we rearrange and use the Lipschitz continuity of f (cf. Assumption 3.1).

(A) is relatively easy to bound as follows:

In the first inequality we use the smoothness of f and Equation (15). In the second inequality we use Equation (16) and in the final inequality the convergence of soft Q-learning.

Now, we turn our attention towards bounding (B) . First, notice that both advantage derivatives are evaluated for the same state-action pair. This is because in Algorithm 8 we sample a once according to π t k +1 and reuse the same a for A π t k ( s, a ) by doing importance sampling. Without this trick, it would be hopeless to bound (B) .

When bounding (B) , we need to consider two sources of randomness. First, there is the randomness in the soft Q-learning iterations. Second, we have to account for the randomness over the trajectory rollouts of GradientEstimator to estimate ˜ ∂ x A π . GradienEstimator first separately estimates ̂ ∂ x V π t k and ̂ ∂ x V π t k from a trajectory and then returns ̂ ∂ x A π t k = ̂ ∂ x Q π t k -̂ ∂ x V π t k . We therefore denote by

the average over the 2 k value function derivatives estimated as part of the 2 k GradientEstimator procedures performed in RT-Q. Note because of the unbiasedness of GradienEstimator (cf. Proposition 3), it holds that ∂ x V π t k = E τ [ ˜ ∂ x V π t k ]

Using the above notation, we get the following bound by repeatedly applying Equation (16):

Here Q ∗ λ denotes the optimal Q-function, i.e. for the policy π ∗ x,ξ . In the equations above we have two flavours of terms. The first are the differences between the derivative estimator and its expectation for a given policy and the second are the differences between the expected derivative under the learned and optimal policy. We start by bounding the first kind of terms. Recall that

since we use 2 k sampled trajectories. The same analysis of course also holds for t k +1 and thus plugging this in, we get

In the first inequality, we simply plug in Equation (18). In the third inequality we use the convergence of soft Q-learning and in the second inequality, we use the following identity (cf. Equation (14)):

where we use Equation (16) for the last inequality.

To bound (B) , it only remains to upper bound E ∥ ∥ ∥ ∂ x Q π t k ( s, a ) -∂ x Q π ∗ ( s, a ) ∥ ∥ ∥ 2 ∞ and E ∥ ∥ ∥ ∂ x Q π t k +1 ( s, a ) -∂ x Q π ∗ ( s, a ) ∥ ∥ ∥ 2 ∞ . Below we will derive an upper bound for the former term. The same analysis yields an equivalent bound for the latter term. We denote by V ∗ λ the optimal regularized value function. Note, the analysis is similar to the one performed in the proof of Theorem 3,

The first equality follows from plugging in the result from Theorem 2. The second equality follows by taking out all terms with t = 0 . The first inequality used the triangle inequality. The second inequality plugs back in the definition from Theorem 2. The third inequality uses Cauchy-Schwarz.

The fourth inequality follows from simplifying. The fifth inequality uses Equation (12). The sixth inequality uses the geometric sum.

To simplify the bound above we want to expres ∥ ∥ ∥ V π t k ( s ′ ) -V ∗ λ ( s ′ ) ∥ ∥ ∥ ∞ using ∥ ∥ ∥ Q π t k -Q ∗ λ ∥ ∥ ∥ ∞ and ∥ π t k -π ∗ ∥ ∞ (as we know the latter two converge for soft Q-learning). We have previously seen in Equation (13) that

Here we use l 2 to denote the minimum possible value that any policy output by soft Q-learning can achieve. Note l 2 > 0 , which follows from Equation (17).

Plugging this result back in, we get

From [49][Lemma 24] we know

With that result we can simplify to

Therefore it follows that

where we use the convergence of soft Q-learning.

Using the above analysis, we can now bound:

This allows us to finish bounding (ii) as follows:

Plugging this result into (2) , we thus have:

Coming back to the start, we get for the variance our desired result as follows:

E.6 Proof of Proposition 1

Proof. In this proof we will derive an expression for

Applying the Dominated Convergence Theorem then directly gives the expression for the derivative of F ( x ) . As we focus on f we can drop any dependence on ξ below to make the proof more readable and concise.

Let

p ( µ → s, t, π, x ) denote the probability given the leader's choise x of reaching state s after t steps starting at µ and following policy π

p ( µ → s, a, t, π, x ) denote the probability under choice x of reaching state s after t steps and then taking action a starting at µ and following policy π

p ( µ → s, a, s ′ , t, π, x ) denote the probability under choice x of reaching state s ′ after t steps having previously been in state s and having taken action a , starting at µ and following policy π

Assuming V ( s ) is differentiable for all s , we show the following statement by induction.

Note that taking n →∞ then directly proves our claim.

Base case n =0 We prove the statement for n = 0

where we use the definition of f in the first equality. The second equality follows fromProposition 2 and the product rule. Rearranging terms gives the third equality.

Induction step n = ⇒ n +1 Assuming Equation (19) holds for n , we prove it for n +1 .

which proves our claim. In the first equality we use the definition of V and the product rule. The second inequality follows from collecting terms.

E.7 Auxiliary Results

Proposition 2 (Gradient of Best response Policy) . It holds that

Proof. For a given x, ξ , this result was previoysly shown by [15]. We give a short proof below.

The second equality follows from the quotient rule. The third and fourth equality follows from the definition of π ∗ x,ξ ( s, a ) . The remaining equalities leverage the definition of the advantage function.

Note, we can replace the total with the partial derivative of Q ∗ ( x ) = max π Q ( π, x ) , due to Rockafellar's thoerem because Q is continuously differentiable in x for all π and Π is compact and convex [9, 56].

Proposition 3 (Unbiased advantage derivative estimator) . The output ̂ ∂ x A π o x,ξ λ,x,ξ ( s, a ) of Algorithm 2

is an unbiased estimate of ∂ x A π o x,ξ λ,x,ξ ( s, a ) , i.e E [ ̂ ∂ x A π o x,ξ λ,x,ξ ( s, a ) ] = ∂ x A π o x,ξ λ,x,ξ ( s, a ) .

Proof. We drop any dependence on x, ξ, π o x,ξ for notational clarity. We further emphasize that the trick of truncating a rollout after a geomtrically sampled time to obtain unbiased gradients is commonly used in the RL literature for obtaining unbiased estimates of the standard policy gradient [80].

We show that the estimator ̂ ∂ x Q λ ( s, a ) given by

is unbiased. The same argument then holds for ̂ d dx V λ ( s ) and implies that ̂ d dx A λ ( s, a ) is unbiased. First of all, we have:

where we use Fubini's theorem for eq. (20) and the Dominated Convergence Theorem and the fact that T ′ Q ∼ Geo (1 -γ 0 . 5 ) in eq. (21). Because T Q and T ′ Q are sampled independently, it immediately follows that

We seperately show that the two summands are unbaised estimates. Then by linearity of expectation the result follows. For (1) using Fubini's theorem and Dominated Convergence Theorem it holds that

Similarly for (2), we have using Fubini and Dominated Convergence Theorem that

Plugging these results back in, we have

which proves the proposition.

Proposition 4 (Unbiased gradient estimator for F ) . The gradient estimator described in Algorithm 5 is unbiased for the given policy π o x,ξ .

Proof. We need to show that:

We can show the claim seperately for (1), (2) and (3). Note for (1) and (3) the claim directly follows from the proof of Proposition 3. And the proof for (2) works almost identical to the one for (3), relying on the fact that a truncation via a geometric distribution is identical to an infinite trajectory with a discount factor.

E.8 Convergence Results for Popular RL Algorithms

For the next Proposition, consider the following soft Bellmann optimality operator, which has been shown to be a contraction [20, 51].

Using Equation (22), one can define a standard soft value iteration algorithm (see Algorithm 3 in Appendix D). We show soft value iteration satisfies Assumption 3.2.

Proposition 5. Algorithm 3 converges, such that ∥ ∥ ∥ π ∗ x,ξ -π o x,ξ ∥ ∥ ∥ 2 ∞ ≤ δ 2 after T iterations, where T = O (log 1 /δ ) .

Proof. From [49][Lemma 24] we have

Moreover

where we use the contraction property shown in [20, 51] and the fact that we instantiate V λ with 0, such that no value iterate can ever be larger than higher than ( R + λ log |A| ) . The claim follows from δ ≤ O ( γ T ) .

As soft value iteration assumes knowledge of the transition function and scales badly when the state and action space are large, in practice stochastic methods such as soft Q-learning are used instead. For this method, consider the soft Bellman state-action optimality operator [3, 30]:

We can use Equation (23) to run soft Q-learning, as described in Algorithm 4 in Appendix D. Equivalently to soft value iteration, we can show soft Q-learning satisfies Assumption 3.2.

Proposition 6. Let π B be sufficiently exploratory, such that the induced Markov chain is ergodic.

Then soft Q-learning converges, such that E o [ ∥ ∥ ∥ π ∗ x,ξ -π o x,ξ ∥ ∥ ∥ 2 ∞ ] ≤ δ 2 after T iterations, where T = O ( log(1 /δ ) δ 2 ) .

We use the following Theorem from [54] to prove our claim:

Theorem [54] Let x ∈ R d , and F : R d → R d be an operator. We use F i to denote the i 'th entry of F . We consider the following stochastic approximation scheme that keeps updating x ( t ) ∈ R d starting from x (0) being the all zero vector,

where i t ∈ { 1 , . . . , d } is a stochastic process adapted to a filtration F t , and w ( t ) is some noise. Assume the following:

̸

Assumption 1 (Contraction) (a) Operator F is γ contraction in ∥·∥ ∞ , i.e. for any x, y ∈ R d , ∥ F ( x ) -F ( y ) ∥ ∞ ≤ γ ∥ x -y ∥ ∞ . (b) There exists some constant C > 0 s.t. ∥ F ( x ) ∥ ∞ ≤ γ ∥ x ∥ ∞ + C, ∀ x ∈ R d .

Assumption 3 (Sufficient Exploration) There exists a σ ∈ (0 , 1) and positive integer, τ , such that, for any i ∈ N and t ≥ τ, P ( i t = i | F t -τ ) ≥ σ .

Assumption 2 (Martingale Difference Sequence) w ( t ) is F t +1 measurable and satisfies E w ( t ) | F t = 0 . Further, | w ( t ) | ≤ ¯ w almost surely for some constant ¯ w .

Suppose Assumptions 1,2 and 3 hold. Further, assume there exists constant ¯ x ≥ ∥ x ∗ ∥ ∞ s.t. ∀ t, ∥ x ( t ) ∥ ∞ ≤ ¯ x almost surely. Let the step size be α t = h t + t 0 with t 0 ≥ max(4 h, τ ) , and h ≥ 2 σ (1 -γ ) . Then, with probability at least 1 -δ ,

Proof. Our algorithm can be seen as a stochastic approximation scheme where we update Q asynchronously just like x above in the following way

where

̸

and the errors:

We now show that F satisfies the assumptions of the Theorem from [54] and use the result to prove our own claim.

In the following we let F t be the σ -algebra generated by the random variables ( s 0 , a 0 , · · · , s t , a t ) . First we restate the following identity from [51]

We use it to show that T ∗ λ is a contraction. Indeed we have:

Moreover, it holds that

So we can set C = R + λ log |A|

Next we note that w t is F t +1 -measurable (it depends on s t +1 ) and that

Moreover w ( t ) is bounded by w = 2 γ ( R + λ log |A| ) 1 -γ .

Further we have assumed that the behavioural policy π B is sufficiently exploratory. Let ˜ µ be the corresponding stationary distribution, µ min = inf s,a ˜ µ ( s, a ) and t mix the mixing time. Then [54] show that for σ = 1 2 µ min and τ = ⌈ log 2 ( 2 µ min ) ⌉ t mix it holds that

Moreover, we note that Q ( t ) and Q ∗ λ are bound by x = R + λ log |A| 1 -γ .

Using the Theorem from [54] we thus have the following result:

Let α t = h t + t 0 with t 0 ≥ max ( 4 h, ⌈ log 2 2 µ min ⌉ t mix ) and h ≥ 4 µ min (1 -γ ) . Then, with probability at least 1 -p ,

We denote the bound above by (A) . Let us choose p = δ 2 . 6 With probability p , ∥ Q ( T ) -Q ∗ λ ∥ ∞ is not bounded by (A) . However, it is always upper bounded by 2( ¯ R + λ log( |A| )) 1 -γ .

Setting T = O ( log(1 /δ ) δ 2 ) , we get

Since δ < 1 , log(1 /δ ) > 0 and for x > 0 it holds that log( x ) x < 1 /e , such that we get:

Using [49][Lemma 24], we arrive at:

A popular class of RL algorithms are policy gradient methods such as REINFORCE [68]. For the entropy-regularised problem, it generally makes sense to choose a softmax parametrization for the policy, as we know π ∗ x,ξ ( s ; a ) ∝ exp( Q ∗ λ,x,ξ ( s, a ) /λ ) [49]. We defer the details to Algorithm 6 in Appendix D and present the following convergence result, which shows using vanilla policy gradient for the lower level also fulfills Assumption 3.2-at least asymptotically.

Proposition 7. Vanilla policy gradient with softmax parameterization converges, such that ∀ δ, ∃ T, ∀ t ≥ T : ∥ ∥ ∥ π ∗ x,ξ -π o t,x,ξ ∥ ∥ ∥ ≤ δ 2 , where π o t,x,ξ is the computed policy after t iterations.

Proof. As in most proofs we drop the subscripts for x, ξ . The proof is an adaptation of the one presented in [49][Lemma 16]. We denote by π t the iterates of the policies of the algorithm and by V π t λ ( µ ) the corresponding value function with starting distribution µ . It can be shown that V π λ is s -smooth for some s [49]. Choosing a stepsize of 1 /s , we have that the value functions increase monotonically, i.e.

At the same time, it holds that:

By monotone convergence it thus follows that V π t λ ( µ ) → V ∗ λ ( µ ) , where V ∗ λ ( µ ) is the maximum possible value.

Since π t ∈ ∆( A ) |S| and ∆( A ) |S| is compact it follows that { π t } t has a convergent subsequence { π t k } k . Denote by π ∗ the limit of this subsequence. It has to hold that V π ∗ λ ( µ ) = V ∗ λ ( µ ) and thus π ∗ is the optimal policy.

Now assume that { π t } t does not converge to π ∗ . In that case

Note that due to entropy regularization V ∗ λ ( µ ) is the unique maximum. This means that

It follows then that which implies V π t λ ( µ ) does not converge to V ∗ -acontradiction to our conclusion above. It therefore has to hold that π t → π ∗ .

The asymptotic guarantee of Vanilla Policy Gradient can be improved to non-asymptotic by using Natural Policy Gradient, as introduced by [42]. We restate the following result from [12].

Proposition 8 (Linear convergence of exact entropy-regularized NPG, [12]) . For any learning rate 0 < η ≤ (1 -γ ) /τ , the entropy-regularized NPG updates (18) satisfy for all t ≥ 0 , where

F Implementation Details

F.1 Baseline Algorithms

F.1.1 Adaptive Model Design [15]

As noted in Section 5, the Adaptive Model Design (AMD) algorithm [15] was proposed for the Regularized Markov Design (RMD) problem which is a special case of Contextual Bilevel Reinforcement Learning. In particular, when P ξ is a Dirichlet distribution CB-RL reduces to the RMD problem. To account for this difference, we modify the AMD algorithm (Algorithm 2 in [15]) as described in Algorithm 9. We denote the upper-level reward and value functions with the superscript u in the algorithm.

F.1.2 Zero-Order Algorithm

Algorithm 10 defines the zero-order gradient estimation algorithm described in Section 5. We parametrize the perturbation constant to decrease with the number of iterations such as u t = C t where C is a positive constant.

Algorithm 9 (Modified) Adaptive Model Design

Input: Iterations T , Inner iterations: K , Learning rate α , Regularization λ , gradient of the pre-learned model ∇ x log P , gradient of the reward function ∇ x r end for

Output:

Optimised parameter x T

Algorithm 10 Zero-Order Algorithm

F.2 Four Rooms

F.2.1 Implementation Details

We parametrize the penalty function ˜ r as the softmax transformation of x ∈ R d s +1 where the i -th entry of x corresponds to the i -th cell in the state space S and the additional dimension d s +1 is used to allocate the penalties not effective and also excluded from the penalty term received by the leader at the end of each episode. In particular,

where softmax( s ; x ) denotes the value of the softmax transformation of x at the entry corresponding to the state s . Note that this parametrization explicitly restricts the maximum available budget for penalties to -0 . 2 .

F.2.2 Hyperparameters

For the upper-level optimization problem, we use gradient norm clipping of 1 . 0 . The learning rate for each algorithm has been chosen as the best performing one from [1 . 0 , 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01] individually. Additionally, we tune the parameter C for the Zero-order algorithm on the values [0 . 1 , 0 . 5 , 1 . 0 , 2 . 0 , 5 . 0] . For Hyper Policy Gradient Descent, we sample 10 , 000 environment steps for each gradient calculation.

λ

F.2.3 Additional Figures

0

2000

4000

6000

8000

10000

Outer iterations:

T

0

.

2

0

.

4

0

.

6

0

.

8

Upper-level objective:

F

HPGD

AMD

Zero-order

0

2000

4000

6000

8000

10000

0

.

2

0

.

4

0

.

6

Upper-level objective:

F

HPGD

AMD

Zero-order

Outer iterations:

T

0

2000

4000

6000

8000

10000

Outer iterations:

T

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Upper-level objective:

F

HPGD

AMD

Zero-order

0

2000

4000

6000

8000

10000

Outer iterations:

T

0

.

2

0

.

4

0

.

6

0

.

8

Upper-level objective:

F

HPGD

AMD

Zero-order

0

2000

4000

6000

8000

10000

Outer iterations:

T

-

0

.

4

-

0

.

2

-

0

.

0

0

.

2

0

.

4

0

.

6

Upper-level objective:

F

HPGD

AMD

Zero-order

0

2000

4000

6000

8000

10000

Outer iterations:

T

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

Upper-level objective:

F

HPGD

AMD

Zero-order

0

2000

4000

6000

8000

10000

Outer iterations:

T

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

Upper-level objective:

F

HPGD

AMD

Zero-order

0

2000

4000

6000

8000

10000

Outer iterations:

T

-

0

.

4

-

0

.

2

-

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

Upper-level objective:

F

HPGD

AMD

Zero-order

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

→→→

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

→

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

↑

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

→

G

1

G

2

S

+1

HPGD

G

1

G

2

S

+1

AMD

G

1

G

2

S

+1

Zero-order

-

10

-

1

-

10

-

2

-

10

-

3

0

F.3 Tax Design for Macroeconomic Models

F.3.1 Implementation Details

In our experiments, we use σ ( s ) = max(0 , log( s/ 20 + 1)) and ω ( s ) = min( s, 0) and select the hyperparameters as θ = 0 . 1 , ϕ = 5 . 0 , ς = 5 . 0 , w = 1 . 0 and λ = 0 . 3 . We use 3 products in the simulated economy with unit prices and assume two equal-sized socio-economic groups with preferences α = (0 . 6 , 0 . 3 , 0 . 1) and (0 . 1 , 0 . 7 , 0 . 2) . Assets are initialized as s 0 ∼ N (0 , √ 2) and each episode is truncated after 200 steps. Both chosen consumption levels and hours worked are discretized with ranges [0 , 5] and [0 , 8] and cardinality 5 and 10 , respectively. All tax rates are initialized as 0 . 3 and optimized over the continuous domain [0 . 0 , 2 . 0] .

F.3.2 Effects of lower-level regularization

Figure 20 and Figure 21 shows the results for λ = 0 . 3 and λ = 0 . 1 , respectively. Changing the regularization has small effects on the final results and only marginally change the speed of convergence, especially for the Zero-Order algorithm.

F.4 Computational Costs

We ran our experiments on a shared cluster equipped with various NVIDIA GPUs and AMD EPYC CPUs. Our default configuration for all experiments was a single GPU with 24 GB of memory, 16 CPU cores, and 4 GB of RAM per CPU core. For all parameter configurations reported in Table 3, the total runtime of the experiments for HPGD, AMD, and Zero-Order were 17, 40, and 2 hours, respectively, totaling 59 hours. Our total computational costs including the intermediate experiments are estimated to be 2-3 times more.

0

200

400

600

800

1000

Outer iterations:

T

35

40

45

50

Upper-level objective:

F

HPGD

Zero-order

0

200

400

600

800

1000

Outer iterations:

T

0

.

2

0

.

4

0

.

6

0

.

8

Tax rates

HPGD

x

HPGD

y

1

HPGD

y

2

HPGD

y

3

Zero-order

x

Zero-order

y

1

Zero-order

y

2

Zero-order

y

3

(a) Performance on the Tax Design problem

(b) Tax rates over the outer iterations.

0

200

400

600

800

1000

Outer iterations:

T

35

40

45

50

Upper-level objective:

F

HPGD

Zero-order

0

200

400

600

800

1000

Outer iterations:

T

0

.

2

0

.

4

0

.

6

0

.

8

Tax rates

HPGD

x

HPGD

y

1

HPGD

y

2

HPGD

y

3

Zero-order

x

Zero-order

y

1

Zero-order

y

2

Zero-order

y

3

(a) Performance on the Tax Design problem

(b) Tax rates over the outer iterations.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our Abstract gives a short overview of our main contributions. Moreover in the Introduction (Section 1), we state the contributions clearly with bullet points and point to the relevant sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: For all proofs we clearly state the assumptions (see Assumption 3.1 and Assumption 3.2) we need to make and write the main limitations multiple times in the text. For the experiments we clearly point to the sensitivity of the regularization parameters.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer:[Yes]

Justification: Our Appendix E contains all proofs to all our statements. For our main theorem (Theorem 3) we further give a proof sketch in the main paper to help with intuition. Our Assumptions are clearly stated.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the main experimental setting in Section 5 while provide details explanation in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide all code related to the research in this paper at https://github. com/lasgroup/HPGD .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All experimental details including hyperparameter selection is described in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the average over 10 random seeds and the corresponding standard errors for the experimental results reported in Table 3 and on Figure 1b.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report our used resources in Appendix F.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics and conducted the research following the guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work has a theoretical focus without an immediate impact on society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: To our best knowledge there do not exist such risks for our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all codes and models used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We neither have crowdsourcing experiments nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:We neither have crowdsourcing experiments nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a general approach for bilevel optimization where the lower-level component is modeled as a Markov Decision Process (MDP). Traditional solvers for bilevel optimization often involve the computation of the Hessian term, which can be complex and computationally intensive. The main strategy employed here leverages the properties of entropy-based reinforcement learning (RL), which offers a closed-form solution. Consequently, the lower-level problem is simplified to a best response function, streamlining the training process. The authors also explore additional scenarios, including one where the upper-level objective is a discounted reward and another where the leader can further influence the follower.\n\nExperiments conducted in a four-room environment demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\nThe problem setting of bilevel optimization in reinforcement learning (RL) is crucial for the community. \n\nThe authors provide a detailed theoretical analysis of their proposed methods.\n\nWEAKNESSES:\n1) The methods proposed might only be used when the lower-level components have a closed form, as the algorithm relies on the closed form provided by entropy-based reinforcement learning (RL). What happens in cases where entropy RL is not used in the lower-level MDP, such as in meta RL?\n\n2) The authors propose various scenarios, including meta RL and macroeconomic modeling, but only test the method in the Four-Rooms environment. Could the authors also conduct experiments on meta RL or economic models, and compare relevant baselines?\n\n3) Proposition 9 follows the established work closely, which limits its theoretical contributions.\n\n4) The authors propose having full access to the lower-level components, but this does not seem to be utilized in the experiments. It would be beneficial to include experiments that make use of this access.\n\nQUESTIONS:\nPlease answer my question mentioned above.\n\nLIMITATIONS:\nPlease refer to the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper introduces the framework of bilevel optimization with lower-level contextual MDPs (BO-CMDP), which covers a wide range of practical decision-making applications. The authors of this paper propose the Hyper Policy Gradient Descent (HPGD) algorithm for solving BO-CMDP. They prove the convergence of HPGD theoretically and demonstrate the practical efficiency of HPGD in experiments.

STRENGTHS:
This paper introduces an interesting framework of BO-CMDP and shows that many practical applications can be formulated as BO-CMDP problems. The authors propose HPGD and prove its convergence. The authors also verify the practical efficiency of HPGD by experiments.

WEAKNESSES:
There are several typos in the statements and the proofs of the main results of this paper.

QUESTIONS:
- In line 551, the authors said they used the results from [30]. It seems like they used Lemma 1 in [30] which requires stepsize $\alpha\leq 1/(2S_f)$, but the authors did not mention this requirement for stepsize $\alpha$ in either the main text or the appendix of this paper. Moreover, the coefficient before $S_f$ in [30] is 1, whereas in this paper, the coefficient is 2. Could the authors explain the reasons for this discrepancy?

- Assumption 3.1 requires further clarification. In line 551, the authors used the infinity norm, but in Assumption 3.1, the specific norms used for $\nabla f$ were not specified. 

- Typos:

In line 551, should $L$ be $L_f$?

In line 188, the $(s,a)$ on the left side of the equation should have subscripts about $t$.

In line 599 and subsequent lines, $d\log P_x(s’;s,a)$ missed $/dx$.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-10

comments: SUMMARY:
This paper considers the bilevel reinforcement learning problem with lower-level contextual MDPs. As compared to existing works, the problem considered is more general. The paper proposes a hyper-gradient based method. The hyper-gradient can be directly evaluated by leveraging the close form optimal policy in terms of optimal value function, enabled by the entropy regularization in the lower level. Furthermore, the paper establishes non-asymptotic convergence for the proposed method.

STRENGTHS:
The strength of this work can be summarized as:
1) The paper is well-written and easy to follow.
2) Compared to previous works where the lower level problem does not include context, the lower-level contextual mdp considered in this work is more general and includes more applications.
3) The close form hyper-gradient facilitates practical implementation.

WEAKNESSES:
Though the paper offers experiments showing the effectiveness of their method, the experiments might be a bit boyish. I believe the work could benefit more from including more scaled up experiments like meta RL listed in the application section.

QUESTIONS:
How does the complexity of the proposed algorithm compare to the existing methods? A table of result comparison might be beneficial for putting this work in context.

LIMITATIONS:
no potential social impact
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces BO-CMDPs, a form of two-level optimization problems in which the inner optimization is a contextual MDP and the outer optimization problem selects the injected context based on some objective. The authors connect this formulation to a range of interesting domains (meta-RL, economics & reward shaping), before proposing a gradient-based algorithm for solving these problems. Depending on the level of insight the outer optimiser has into the workings of the inner different convergence guarantees are established before the algorithm is assessed on a toy example. An ablation into the role of several hyperparameters is also provided.\n\nSTRENGTHS:\nOverall the work is well-presented and easy to follow.\n\nThe introduced problem setting is well-motivated and connected to existing work. The example applications are of high interest to the community and lend further importance to the proposed formulation.\n\nTechnical contributions and theoretical analysis appears sound and the empirical examination supports the authors claims.\n\nWEAKNESSES:\nArguably, the main limitation of the paper is the simple nature of the toy example.\n\nWhile the provided results appear promising and the examinations of the learned incentive maps illustrate the viability of the algorithm, they leave questions regarding the computation complexity and scalability of the proposed method.\nI think such questions could be answered by returning to one of the motivating examples (e.g. Meta-RL or Reward Shaping tasks) and applying HPGD to some moderate-to-large-scale settings. This would also lend further credibility to the papers motivating settings.\n\nQUESTIONS:\nA minor point: Would it make sense to rename the problem setting to Bi-cMDP? \"BO\" is quite overloaded with Bayesian Optimisation and might be slightly confusing for some readers.\n\nLIMITATIONS:\nThe authors assumptions are stated were necessary.\nThe algorithms limitations and room for future work are briefly outlined in the final section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}