Title: Model-Based Transfer Learning for Contextual Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Reinforcement Learning

Zero-Shot Transfer

Generalization

Bayesian Optimization
Abstract: Deep reinforcement learning (RL) is a powerful approach to complex decision-making. However, one issue that limits its practical application is its brittleness, sometimes failing to train in the presence of small changes in the environment. Motivated by the success of zero-shot transfer—where pre-trained models perform well on related tasks—we consider the problem of selecting a good set of training tasks to maximize generalization performance across a range of tasks. Given the high cost of training, it is critical to select training tasks strategically, but not well understood how to do so. We hence introduce Model-Based Transfer Learning (MBTL), which layers on top of existing RL methods to effectively solve contextual RL problems. MBTL models the generalization performance in two parts: 1) the performance set point, modeled using Gaussian processes, and 2) performance loss (generalization gap), modeled as a linear function of contextual similarity. MBTL combines these two pieces of information within a Bayesian optimization (BO) framework to strategically select training tasks. We show theoretically that the method exhibits sublinear regret in the number of training tasks and discuss conditions to further tighten regret bounds. We experimentally validate our methods using urban traffic and standard continuous control benchmarks. The experimental results suggest that MBTL can achieve up to 43x improved sample efficiency compared with canonical independent training and multi-task training. Further experiments demonstrate the efficacy of BO and the insensitivity to the underlying RL algorithm and hyperparameters. This work lays the foundations for investigating explicit modeling of generalization, thereby enabling principled yet effective methods for contextual RL. Code is available at https://github.com/jhoon-cho/MBTL/.

Model-Based Transfer Learning for Contextual Reinforcement Learning

Abstract

Deep reinforcement learning (RL) is a powerful approach to complex decision making. However, one issue that limits its practical application is its brittleness, sometimes failing to train in the presence of small changes in the environment. Motivated by the success of zero-shot transfer-where pre-trained models perform well on related tasks-we consider the problem of selecting a good set of training tasks to maximize generalization performance across a range of tasks. Given the high cost of training, it is critical to select training tasks strategically, but not well understood how to do so. We hence introduce Model-Based Transfer Learning (MBTL), which layers on top of existing RL methods to effectively solve contextual RL problems. MBTL models the generalization performance in two parts: 1) the performance set point, modeled using Gaussian processes, and 2) performance loss (generalization gap), modeled as a linear function of contextual similarity. MBTL combines these two pieces of information within a Bayesian optimization (BO) framework to strategically select training tasks. We show theoretically that the method exhibits sublinear regret in the number of training tasks and discuss conditions to further tighten regret bounds. We experimentally validate our methods using urban traffic and standard continuous control benchmarks. The experimental results suggest that MBTL can achieve up to 43x improved sample efficiency compared with canonical independent training and multi-task training. Further experiments demonstrate the efficacy of BO and the insensitivity to the underlying RL algorithm and hyperparameters. This work lays the foundations for investigating explicit modeling of generalization, thereby enabling principled yet effective methods for contextual RL. Code is available at https://github.com/jhoon-cho/MBTL/.

1 Introduction

Deep reinforcement learning (DRL) has made remarkable strides in addressing complex problems across various domains [29, 37, 1, 4, 10, 12, 27]. Despite these successes, DRL algorithms often exhibit brittleness when exposed to small variations like different number of lanes, weather conditions, or flow density in traffic benchmarks [20], significantly limiting their scalability and generalizability [19]. Such variations can be modeled using the framework of contextual Markov Decision Processes (CMDP), where task variations can be parameterized within a context space [14, 32, 5].

There are two predominant solution modalities for CMDPs [23]: independent training and multi-task training. Independent training constructs a separate model for each task variant (say, N ≫ 1 ), which

0.7

OracleTransfer

performance

0.7

0.7

Independent

0.6

Multitask

0.6

0.6

0.5

0.5

0.5

Normalizec

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0

10

20

30

40

50

0.0

0.2

0.4

0.6

0.8

1.0

0

400

800

1200 1600 2000

GreenPhaseTime

PenetrationRate

InflowRate

is compute-intensive. At the other extreme, multi-task training constructs a single 'universal' policy, and thus can be more compute-efficient, but suffers from model capacity and negative transfer issues [22, 46, 2, 42]. There is thus a need for more reliable training methodologies for generalization across tasks variants. In this work, we consider training an intermediate set of K models, where N > K > 1 , in an effort to balance performance and efficiency; we refer to this strategy as multi-policy training .

We build upon zero-shot transfer, a widely-used practical technique which directly applies a policy trained in one context (source task) to another (target task) without adaptation. Figure 1 shows that multi-policy training with zero-shot transfer has the potential to improve the performance even over the independent training. In this article, we strategically select source tasks by explicitly modeling the generalization performance to estimate the value of training a new source task.

A note on terminology . For brevity, we refer to task variants as tasks in the remainder of this article. We also use the language of task and context interchangeably. We emphasize that this work focuses on within-domain generalization (e.g., traffic signal control for intersection scenario variants) rather than across-domain generalization (e.g., distinct traffic control tasks). Additionally, it is crucial to differentiate between training reliability, which concerns the ability to reliably train models across tasks, and model reliability (or robustness), which concerns the resistance of a trained model to differences in tasks. This article is concerned with training reliability.

The main contributions of this work are:

We introduce Model-Based Transfer Learning (MBTL) , a novel framework for solving CMDP sample efficiently (Figure 2). To the best of our knowledge, this is the first work to explicitly model generalization performance for contextual RL (CRL). As such, our work opens the door for further investigation into reliable model-based methodologies for CRL.

We provide theoretical analysis for the sublinear regret of MBTL and give conditions for achieving tighter regret bounds.

We empirically validate our methods in urban traffic and standard continuous control benchmarks for contextual RL, observing up to 43x improvements in sample efficiency. We further include ablations on the components of the algorithm.

The remainder of the paper is organized as follows. After introducing notation in Section 2, we formally define the problem in Section 3. A key contribution of our work is the introduction of a Gaussian process model acquisition function specifically tailored to the source task selection problem, which is detailed in Section 4. In Section 4.3, we provide a theoretical analysis of the regret bounds of our method, followed by an empirical evaluation across diverse applications in Section 5.

2 Preliminaries and notation

Contextual MDP. A standard MDP is defined by the tuple M = ( S, A, P, R, ρ ) where S represents the state space, A is the action space, P denotes the transition dynamics, R is the reward function, and ρ is the distribution over initial states [45]. A contextual MDP (CMDP), denoted by M =( S, A, P x , R x , ρ x ) x ∈ X , is a collection of context-MDPs M x parameterized by a context

Marginalgeneralization

(a)

Estimated training performance

(b)

/performance

Generalization gap

(GaussianProcess)

(linear)

Performance

Performance

Generalization

πT1

T2

πT1

T2

performance

Zero-shot

argmax

Transfer

X1X2X3..XN-4XN-3XN-2XN-1

xo

x1

x2

xN-4XN-3XN-2xN-1

Next

Context space X

Iteration

(d)

(c)

Acquisition function

Performance

Selectmax!

Acquisition

Terminate

function

Train on x!

T3

T1

T2

π3

TT2

Zero-shot

Transfer

argmax

1

variable x within a context space X (assumed bounded). The context variable x can influence dynamics, rewards, and initial state distributions [14, 32, 5]. We define source task performance J ( π x , x ; Alg ) as follows: we train a policy π x on a task with the context x ∈ X using RL algorithm Alg (e.g., PPO, SAC) and evaluate the policy by the expected return in the same MDP M x with context x . For brevity, we will write it as J ( π x , x ) . We distinguish between estimated values ˆ J ( x ) and observed outcomes J ( π x , x ) , with the latter measured after training and evaluation.

Generalization gap via zero-shot transfer. Consider zero-shot transfer from the trained policy π x from a source task (context-MDP) to solve another target task (context-MDP) with the context x ′ ∈ X in the CMDP. Zero-shot transfer involves applying a policy trained on a source task M x to a different target task M x ′ , with x, x ′ ∈ X . This experiences performance degradation, also called generalization gap [17, 23]. For instance, Figure 3 depicts that the performance degrades as the target task diverges from the source task, corresponding to an increasing generalization gap. Nonetheless, leveraging the notion that training is expensive but zero-shot transfer is cheap, we are interested in optimally selecting a set of source (training) tasks, such that the generalization performance on the target range of tasks is maximized. We observe the generalization performance , denoted by

1.0

performance

0.8

0.6

Normalized

0.4

0.2

0.0

0

20

40

60

80

100

Contextvariation

J ( π x , x ′ ) , by evaluating the target task x ′ based on the policy trained using source task x via zero-shot generalization. We define the generalization gap as the absolute performance difference in average reward when transferring from source task x to target task x ′ :

3 Problem formulation

Sequential source task selection problem. The selection of source MDPs from the CMDPs is key to solving the overall CMDP [3]. We therefore introduce the sequential source task selection

(SSTS) problem , which seeks to maximize the expected performance across a dynamically selected set of tasks. This problem is cast as a sequential decision problem, in which the selection of tasks is informed through feedback from the observed task performance of the tasks selected and trained thus far. The notation x k indicates the selected source task at the k -th transfer step, where k ranges from 1 to K . For brevity, we will denote π x k as π k . We denote the sequence x 1 , x 2 , ..., x k by x 1: k and π 1 , π 2 , ..., π k by π 1: k .

Definition 1 (Sequential Source Task Selection Problem) . This problem seeks to optimize the expected generalization performance across a CMDP M x ′ ∈ X by selecting a task x ∈ X at each training stage. Specifically, at each selection step k , we wish to choose a distinct task x k such that the expected cumulative generalization performance is maximized. This can be expressed by keeping track at each step, which policy to use for which task, and the corresponding generalization performance. Upon training the policy π x k for source task x k , the cumulative generalization performance, which we abuse notation to denote as J ( π 1: k , · ) = J ( π x k , · ; π 1: k -1 ) . Formally, this can be recursively defined based on previous observations { J ( π 1 , x ) , . . . , J ( π k -1 , x ) } for all x ∈ X as follows:

And J ( π 1:1 , x ) ≡ J ( π 1 , x ) . Then, the overall sequential decision problem can be written as:

The state at each step k is defined by the best generalization performance for each task, achieved by policies trained in earlier stages, represented as J ( π 1: k -1 , x ′ ) for each target task x ′ . The action at each step is choosing a new task x k . In general, SSTS exhibits stochastic transitions, for example due to randomness in RL training. For simplicity, in this work, we assume deterministic transitions; that is, training context-MDP x will always yield the same performance J ( π x , x ) and generalization gap ∆ J ( π x , x ′ ) , ∀ x ′ ∈ X . The problem's maximum horizon is defined by | X | , but can be terminated early if conditions are met (e.g., performance level, training budget).

4 Model-Based Transfer Learning (MBTL)

In this section, we introduce an algorithm called Model-based Transfer Learning to solve the SSTS problem. MBTL models the generalization performance in two parts: 1) the performance set point, modeled using Gaussian processes, and 2) generalization gap, modeled as a linear function of contextual similarity. MBTL combines these two pieces of information within a Bayesian optimization (BO) framework to sequentially select training tasks that maximize generalization performance.

4.1 Modeling assumptions

We consider a task set X that is continuous and the performance function J ( π, x ) , V ( x ) for a policy π to be smooth over the task space X . In practice, such as control systems, tasks often vary continuously and smoothly rather than abruptly. For example, adjusting the angle of a robotic arm by a small amount typically results in a small change in the system and optimal action. Inspired by the empirical generalization gap performance as observed in Figure 3, we estimate the generalization gap with a linear function of contextual similarity.

Assumption 1 (Linear generalization gap) . A linear function is used to model the generalization gap function, formally ∆ ˆ J ( π x , x ′ ) ≃ θ | x -x ′ | , where θ is the slope of the linear function and x and x ′ are the context of the source task and target task, respectively.

The relaxation of this assumption can yield additional efficiency benefits but also adds modeling complexity, and thus is an interesting direction for future work.

4.2 Bayesian optimization

Bayesian optimization (BO) is a powerful strategy for finding the global optimum of an objective function when obtaining the function is costly, or the function itself lacks a simple analytical form [31, 6]. BO integrates prior knowledge with observations to efficiently decide the next task to train by using the acquisition function. MBTL is a BO method which optimizes for promising source tasks by leveraging Assumption 1 in its acquisition function. The role of BO is to approximate V ( x k ; π 1: k -1 )

(see Equation 3) using the data acquired thus far. The next source task x k is then selected using this estimate.

Gaussian process (GP) regression. Within the framework of BO, we model the source training performance ˆ J ( π x ,x ) ∀ x ∈ X \ x 1: k using Gaussian process (GP) regression. Specifically, the function ˆ J ( π x ,x ) is assumed to follow a GP ( ˆ J ( π x ,x ) ∼ GP ( E [ ˆ J ( π x ,x )] , k ( x, ˜ x ))) , where k ( x, ˜ x ) is the covariance function, representing the expected product of deviations of ˆ J ( π x ,x ) and ˆ J ( π ˜ x , ˜ x ) from their respective means. Let D k -1 denote the data observed up to iteration k -1 , consisting of the pairs { ( x i , J ( π i , x i )) } i =1 ,...,k -1 . The estimated performance ˆ J k after querying k -1 samples is updated as more samples are obtained. The posterior prediction of ˆ J k at a new point x , given the data D k -1 and previous inputs x 1: k -1 , is normally distributed as P ( ˆ J k | D k -1 ) = N ( µ k ( x ) , σ 2 k ( x )) . µ k ( x ) and σ 2 k ( x ) are defined as µ k ( x ) = E [ ˆ J ( π x ,x )] + k ⊤ ( K + σ 2 I ) -1 y and σ 2 k ( x ) = k ( x, x ) -k ⊤ ( K + σ 2 I ) -1 k , with k being the vector of covariances between x and each x i in the observed data, i.e., k = [ k ( x, x 1 ) , . . . , k ( x, x k -1 )] , and K is the covariance matrix for the observed inputs, defined as K = [ k ( x i , x j )] 1 ≤ i,j ≤ k -1 . This enables the GP to update its beliefs about the posterior prediction with every new observation, progressively improving the estimation.

Acquisition function. The acquisition function plays a critical role in BO by guiding the selection of the next source training task. At each decision step k , the task x k is chosen by maximizing the acquisition function, as denoted by x k = arg max x a ( x ; x 1: k -1 ) . One effective strategy employed in the acquisition function is the upper confidence bound (UCB) acquisition function, which considers the trade-off between the expected performance of a task based on the current task (exploitation) and the measure of uncertainty associated with the task's outcome (exploration) [41]. Especially in our case, the acquisition function can be designed as UCB function subtracted by generalization gap and so-far best performance. It is defined as follows:

where [ · ] + represents max( · , 0) and we can use various forms of β k , which is the trade-off parameter between exploitation and exploration.

4.3 Regret analysis

We use regret to quantify the effectiveness of our source task selection based on BO. Specifically, we define regret at iteration k as r k = V ( x ∗ k ; π 1: k -1 ) -V ( x k ; π 1: k -1 ) , where V ( x ∗ k ; π 1: k -1 ) represents the maximum generalization performance achievable across all tasks, and ˆ V ( x k ; π 1: k -1 ) is the performance at the current task selection x k . Consequently, the cumulative regret after K iterations is given by R K = ∑ K k =1 r k , summing the individual regrets over all iterations. Following the framework presented by Srinivas et al. [41], our goal is to establish that this cumulative regret grows sublinearly with respect to the number of iterations. Mathematically, we aim to prove that lim K →∞ R K K = 0 , indicating that, on average, the performance of our strategy approaches the optimal performance as the number of iterations increases.

Regret of MBTL. Having established the general framework for regret analysis, we now turn our attention to the specific regret properties of our MBTL algorithm. To analyze the regret of MBTL, consider the scaling factor for the UCB acquisition function given by β k = 2log( | X | π 2 k 2 / 6 δ ) in Equation (4). It is designed to achieve sublinear regret with high probability, aligning with the theoretical guarantees outlined in Theorem 1 and 5 from [41].

Theorem 1 (Sublinear Regret) . Given δ ∈ (0 , 1) , and with the scaling factor β k as defined, the cumulative regret R K is bounded by √ KC 1 β K γ K with a probability of at least 1 -δ . The formal expression of this probability is Pr [ R K ≤ √ KC 1 β K γ K ] ≥ 1 -δ , where C 1 := 8 log(1+ σ -2 ) ≥ 8 σ 2 and γ K = O (log K ) for the squared exponential kernel.

Impact of search space elimination. In this section, we demonstrate that strategic reduction of the possible sets, guided by insights from previous task selections or source task training performance, leads to significantly tighter regret bounds than Theorem 1. By focusing on the most promising regions of the task space, our approach enhances learning efficiency and maximizes the policy's performance and applicability. Given the generalization gap observed in Figure 3, we observe that performance loss decreases as the context similarity increases. We model the degradation from

the source task using a linear function in Assumption 1. Training on the source task can solve a significant portion of the remaining tasks. Our method progressively eliminates partitions of the task space at a certain rate with each iteration. If the source task selected in the previous steps could solve the remaining target task sufficiently, we can eliminate the search space at a desirable rate. Formally, we can define the search space at k -th iteration as follows:

Definition 2 (Search space) . We define the search space X k at iteration k as a subset of X , with each element x ′ ∈ X k , such that J ( π 1: k -1 , x ′ ) ≤ ˆ J ( π x k , x ′ ) -∆ J ( π x k , x ′ ) .

Given the generalization gap observed in Figure 3, we model the degradation from the source task using a linear function in Assumption 1. While the figure might not strictly appear linear, the linear approximation simplifies analysis and is supported by empirical observations. Training on the source task can solve a significant portion of the remaining tasks. Our method progressively eliminates partitions of the task space at a certain rate with each iteration. If the source task selection in the previous step sufficiently addresses the remaining target tasks, we can reduce the search space at a desirable rate. Consequently, at each step, we effectively focus on a reduced search space.

We leverage the reduced uncertainty in well-sampled regions to tighten the regret bound while slightly lowering the probability δ in Theorem 1. For the regret analysis, we propose the following theorem based on the generalization of Lemma 5.2 and 5.4 in [41] to the eliminated search space.

Here, | X | denotes the cardinality of the set X , the number of elements in X . Theorem 2 matches the Theorem 1 when X k = X for all k . This theorem implies that regret has a tighter or equivalent bound if we can design the smaller search space instead of searching the whole space. The comprehensive proof is provided in Appendix A.3.1.

Here are some examples of restricted search space: If we consider an example where | X k | = 1 √ k | X | , the regret can be bounded tighter than that of Theorem 1.

Corollary 2.1. Consider | X k | = 1 √ k | X | . The regret bound would be R K ≤ √ C 1 β K γ K log K with a probability of at least 1 -δ ′ .

In cases where the search space is defined using MBTLGS, the largest segment's length would reduce geometrically, described by | X k | ≤ 2 -⌊ log 2 k ⌋ | X | .

Corollary 2.2. The regret bound for the | X k | ≤ 2 -⌊ log 2 k ⌋ | X | would be R K ≤ √ C 1 β K γ K π 2 / 6 with a probability of at least 1 -δ ′ .

Proofs for Corollaries 2.1 and 2.2 are provided in Appendix A.3.2 and A.3.3, respectively. Based on our experiments presented in Section 5, the rate of elimination of the largest segment for MBTL is shown in Figure 4.

5 Experiments and analysis

5.1 Setup

Our experiments consider CMDPs that span standard and real-world benchmarks. In particular, we consider standard continuous control benchmarks from the CARL library [5]. In addition, we study problems from RL for intelligent transportation systems, using [49] to model the CMDPs. Surprisingly, despite the relatively low complexity of the CMDPs considered, standard deep RL algorithms appear to struggle to solve the tasks.

Baselines. Weconsider two types of baselines when evaluating our proposed algorithm: canonical and multi-policy. The canonical baselines are selected to validate multi-policy training; the multi-policy training baselines are heuristic methods designed to validate the Bayesian optimization approach.

†Higher the better. Bold values represent the highest value(s) within the statistically significant range for each CMDP, excluding the oracle. Detailed results with variance for each method are provided in Appendix A.4.3.

‡AA: Advisory autonomy benchmark, Ring: Single lane ring, Ramp: Highway ramp, Acc: Acceleration guidance, Vel: Speed guidance.

The canonical baselines include: (1) Independent training , which involves independently training separate models on each task; and (2) Multi-task RL , where a single 'universal' context-conditioned policy is trained for all tasks. The multi-policy baselines include: (3) Random selection , where each training task is chosen uniformly at randomly; (4) Equidistant strategy , which selects training tasks by equally subdividing the context space based on a given training budget, and then trains them in lexicographical order; (5) Greedy strategy , which greedily selects the next source task based on Assumption 1 and fixed training performance; and (6) Sequential Oracle transfer , which has access to generalized performance corresponding to policies for all tasks (including those not yet selected) and uses that information to greedily select the best source task at each step.

Proposed method. We evaluate MBTL with the scaling factor β k = 2log( | X | π 2 k 2 / 6 δ ) .

DRL algorithms and performance measure. We utilize Deep Q-Networks (DQN) for discrete action spaces [29] and Proximal Policy Optimization (PPO) for continuous action spaces [36]. For statistical reliability, we run each experiment three times with different random seeds. We evaluate our method by the average performance across all N target tasks after training up to K = 15 source tasks or the number of source tasks needed to achieve a certain level of performance. We employ min-max normalization of the rewards for each task, and we provide comprehensive details about our model in Appendix A.4.1.

5.2 Traffic benchmark experiments

We consider three traffic control benchmarks. First, while most traffic lights still operate on fixed schedules, RL can be used to design adaptive (1) Traffic signal control to optimize traffic [8, 24]. However, considering that every intersection is different, challenges persist in generalizing across intersection configurations [19]. Given the significant portion of greenhouse gas emissions in the United States due to transportation [11], the second traffic domain is (2) Dynamic eco-driving at signalized intersections , which concerns learning energy-efficient driving strategies in urban settings. DRL-based eco-driving strategies have been developed [13, 47, 18] but still experience difficulties in generalization. Our final traffic domain is (3) Advisory autonomy , in which real-time speed or acceleration advisories guide human drivers to act as vehicle-based traffic controllers in mixed traffic environments [40, 7, 15]. The context space X is discretized into N = { 50 , 50 , 40 } contexts for the three domains, respectively. In Appendix A.4, we provide details about our experiments.

Results. Table 1 and Figure 5 summarize the results. Notably, the Oracle far outperforms the standard baselines (independent and multi-task training), indicating the potential for transfer learning and intelligent training of multiple models, respectively. MBTL rapidly approaches the Oracle within ≈ 10 transfer steps, indicating that the GP effectively models the training performance and linear generalization gap models the generalization performance. It is important to note that multi-task RL studies commonly consider Independent training as a strong baseline due to its avoidance of negative transfer and other training instability issues. Indeed, independent training often (but not always) outperforms multi-task training in our experiments. Yet, our experiments show that MBTL outperforms both independent and multi-task baselines and matches their performance with up to 30x improved sample efficiency . Among the multi-policy baselines, MBTL often outperforms the

TrafficSignalControl

Eco-Driving

AdvisoryAutonomy

(Inflow)

(GreenPhaseTime)

(Acceleration guidance in Ring)

performance

0.50

0.85

0.9

0.45

0.8

3x

0.80

nalized

13x

0.40

0.7

lorma

0.75

13x

0.6

0.35

0.5

0.70

104

105

104

105

104

105

Number of samples(Log scale)

Number of samples (Log scale)

Number of samples(Log scale)

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq.Oracle

†Higher the better. Bold values represent the highest value(s) within the statistically significant range for each CMDP, excluding the oracle. Detailed results with variance for each method are provided in Appendix A.4.3.

heuristic multi-policy baselines, indicating the value of adaptively selecting source tasks based on feedback. The multi-policy baselines, such as random, equidistant, and greedy strategy, also generally outperform Independent and Multi-task, indicating the general value of multi-policy training for solving CMDPs. More results are provided in Appendix A.4.

5.3 Continuous control benchmark experiments

Cartpole (Mass of Cart)

Pendulum (Time Step)

Halfcheetah(Friction)

1.0

0.9

1.0

performance

0.8

0.8

0.9

0.6

0.8

5x

0.7

20x

11x

Normalized

0.4

0.7

0.6

0.2

0.6

0.5

0.0

0.5

104

105

104

105

104

105

Number of samples (Log scale)

Number of samples (Log scale)

Number of samples (Log scale)

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq.Oracle

Step1

Step3

0.08

0.175

1.0

1.0

1.0

Normalized Performance

ance

0.150

ance

0.06

0.8

0.06

function

rform

0.8

0.125

function

0.8

0.05

function

0.6

0.04

Acquisition

Per

0.6

0.100

sition

erfor

Per

0.6

0.04

quisition

0.4

Normalize

0.4

0.075

cquisi

zed

Z!1

0.4

0.03

0.2

0.02

0.2

0.050

Ac

Norma

0.2

GPprediction

Source tasks

0.02

A

0.0

0.00

0.0

0.025

0.000

0.0

Acquisitionfunction

Next source task

0.00

0.01

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Source Task

Source Task

Source Task

DQN

PPO

A2C

performance

1.0

1.0

1.0

林

0.9

0.8

0.8

0.8

0.6

0.7

Normalized

0.6

0.6

0.4

0.5

0.4

0.2

0.4

104

105

104

105

104

105

Number ofsamples trained(Log scale)

Numberofsamplestrained(Log scale)

Numberofsamples trained(Log scale)

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq.Oracle

To probe the generality of MBTL, we utilize context-extended versions of standard RL environments from CARL benchmark library [5] to evaluate our methods under varied contexts. For the Cartpole domain, we considered CMDPs with varying cart mass, pole length, and pole mass. In Pendulum, we vary the timestep duration, pendulum length, and pendulum mass. The BipedalWalker was tested under varying friction, gravity, and scale. In HalfCheetah domain, we manipulated friction, gravity, and stiffness parameters. These variations critically influence the dynamics and physics of the environments. The range of context variations was selected by scaling the default values specified in CARL from 0.1 to 10 times ( N = 100 ), enabling a comprehensive analysis of transfer learning under drastically different conditions. We provide more experimental details in Appendix A.4.

Results. The results summarized in Table 2 demonstrate sample efficiency and competitive performance of multi-policy training including MBTL across diverse control domains, often closely trailing the Oracle only with a small number of trained policies. Figure 6 shows that with the exception of a few context variations, MBTL generally shows superior performance. Specifically, Figure 7 illustrates the detailed process of how MBTL utilizes GP for performance estimation and chooses the next source task that maximizes the acquisition function that evaluates the expected improvement of generalized performance. MBTL achieves comparable performance to multi-task or independent baselines with 5-43x fewer samples, highlighting its effectiveness in reducing training requirements.

5.3.1 Sensitivity analysis

DRL algorithms. Figure 8 shows that MBTL remains effective with different underlying DRL algorithms-DQN, PPO, and Advantage Actor-Critic (A2C) [30]-used for single-task training.

Acquisition functions. Figure 9 assesses the role of acquisition functions in Bayesian optimization. While expected improvement (EI) focuses on promising marginal gains beyond the current best, UCB utilizes both mean and

Normalized performance

1.0

EI

UCB (β=C1/k)

UCB (β =1)

UCB (β=C2log(k+1))

0.8

0.6

0.4

0.2

Cartpole

(Mass of

control

control

Pole)

(Road Length)

(Green time)

variance for balancing exploration and exploitation. Overall, we find that MBTL is not particularly sensitive to the choice of optimism representation in the acquisition function, which indicates that MBTL has a weak dependence on hyperparameters.

6 Related work

Contextual Reinforcement Learning. Robustness and generalization challenges in DRL are generally addressed by a few common techniques in the literature. The broader umbrella of such methods falls under CRL, which utilizes side information about the problem variations to improve the generalization and robustness. In particular, CRL formalizes generalization in DRL using CMDPs [14, 32, 5], which incorporate context-dependent dynamics, rewards, and initial state distributions into the formalism of MDPs. The contexts of CMDPs are not always visible during training [23]. When they are visible, they can be directly used as side information by conditioning the policy on them [39]. In this paper, we focus on a scenario where the learner can choose which context-MDP to train on. This contrasts with other CRL works that assume context-MDPs arrive from a fixed distribution.

Multi-task training. Multi-task methods can help address CRL by exploiting shared structure across tasks. Prior work has leveraged techniques such as policy sketches for task decomposition [2] and distilled policies that capture common behaviors [46]. However, a key limitation arises when the context is unobserved, effectively transforming the CMDP into a partially observable setting [23, 9], which complicates multi-task training. Another challenge is negative transfer, wherein training on tasks that are too dissimilar leads to instability or outright failure [22, 42, 44]. Although more recent multi-task approaches such as MOORE [16] and PaCo [43] have shown promise, they often focus on discrete task sets and are thus less suited to CRL, where tasks span a broad continuum of contexts. In this work, we include multi-task learning as a baseline to benchmark our methods.

Zero-shot transfer and policy reuse. Zero-shot transfer-where models trained for one environment directly perform in new, unseen settings without additional training [23]-is an important strategy in CRL settings. For solving CMDP problems, prior works attempted to utilize zero-shot transfer to solve CMDP problems by approximation on RL algorithm and hypernetworks that maps from parameterized CMDP to a family of near-optimal solutions [35]. Sinapov et al. [38] use meta-data to learn inter-task transferability to learn the expected benefit of transfer given a source-target task pair. Bao et al. [3] propose a metric for evaluating transferability based on information-theoretic feature representations across tasks. Taken together, these approaches highlight the importance of policy reuse, where efficiently selecting or adjusting a pre-trained policy accelerates learning and improves robustness in new contexts.

Source task selection. In the context of transfer learning, selecting appropriate source tasks is crucial. Li and Zhang [25] proposes an optimal online method for dynamically selecting the most relevant single source policy in reinforcement learning. Beyond RL, Meiseles and Rokach [28] emphasizes structural alignment in time-series source models to prevent performance degradation, while Poth et al. [33] finds that selecting aligned intermediate tasks in natural language processing boosts transfer effectiveness. Building upon these insights, we formulate the source task selection problem for CRL, enabling zero-shot transfer by estimating training performance online and leveraging structural generalization across context variations.

7 Conclusion

This study introduces a method called Model-based Transfer Learning (MBTL), which layers on top of existing RL methods to effectively solve CMDPs. Rather than independent or multi-task training, which trains N or 1 models, respectively, MBTL intelligently selects an intermediate number of models to train. MBTL has two key components: an explicit model of the generalization gap and a Gaussian process component to estimate training performance. MBTL achieves up to 43x improved sample efficiency on standard and real-world benchmarks. Furthermore, MBTL achieves sublinear regret in the number of training tasks. A limitation is that MBTL is designed for a single-dimensional context variation with a reliance on the explicit similarity of context variables. Promising directions of future work include studying high-dimensional context spaces and formalizing task similarity, as well as the development of new real-world CMDP benchmarks.

Acknowledgments and Disclosure of Funding

The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper. This work was supported by the National Science Foundation (NSF) CAREER award (#2239566), the Kwanjeong Educational Foundation Ph.D. scholarship program, and an Amazon Robotics Ph.D. Fellowship. The authors would like to thank the anonymous reviewers for their valuable feedback.

References

Abubakr O Al-Abbasi, Arnob Ghosh, and Vaneet Aggarwal. Deeppool: Distributed model-free algorithm for ride-sharing using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems , 20(12):4714-4727, 2019.

Jacob Andreas, Dan Klein, and Sergey Levine. Modular Multitask Reinforcement Learning with Policy Sketches. In Proceedings of the 34th International Conference on Machine Learning , pages 166-175. PMLR, July 2017. ISSN: 2640-3498.

Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An Information-Theoretic Approach to Transferability in Task Transfer Learning. In 2019 IEEE International Conference on Image Processing (ICIP) , pages 2309-2313, Taipei, Taiwan, September 2019. IEEE. ISBN 978-1-5386-6249-6. doi: 10.1109/ICIP.2019.8803726.

Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature , 588(7836):77-82, 2020.

Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian Döhler, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize Me - The Case for Context in Reinforcement Learning. Transactions on Machine Learning Research , June 2023.

Eric Brochu, Vlad M. Cora, and Nando de Freitas. A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning, December 2010.

Jung-Hoon Cho, Sirui Li, Jeongyun Kim, and Cathy Wu. Temporal transfer learning for traffic optimization with coarse-grained advisory autonomy. arXiv preprint arXiv:2312.09436 , 2023.

Tianshu Chu, Jie Wang, Lara Codeca, and Zhaojian Li. Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control. IEEE Transactions on Intelligent Transportation Systems , 21(3): 1086-1095, March 2020. ISSN 1524-9050, 1558-0016. doi: 10.1109/TITS.2019.2901791.

Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning , pages 2048-2056. PMLR, 2020.

Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897):414-419, 2022.

US EPA. Sources of Greenhouse Gas Emissions, 2023. URL https://www.epa.gov/ghgemissions/ sources-greenhouse-gas-emissions .

Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature , 610(7930):47-53, 2022.

Qiangqiang Guo, Ohay Angah, Zhijun Liu, and Xuegang (Jeff) Ban. Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors. Transportation Research Part C: Emerging Technologies , 124:102980, March 2021. ISSN 0968090X. doi: 10.1016/j.trc.2021.102980.

Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.02259 , 2015.

Aamir Hasan, Neeloy Chakraborty, Haonan Chen, Jung-Hoon Cho, Cathy Wu, and Katherine DriggsCampbell. Cooperative advisory residual policies for congestion mitigation. Journal on Autonomous Transportation Systems , 2024.

Ahmed Hendawy, Jan Peters, and Carlo D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. arXiv preprint arXiv:2311.11385 , 2023.

Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: Improving Zero-Shot Transfer in Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Learning , pages 1480-1490. PMLR, July 2017. ISSN: 2640-3498.

Vindula Jayawardana and Cathy Wu. Learning eco-driving strategies at signalized intersections. In 2022 European Control Conference (ECC) , pages 383-390. IEEE, 2022.

Vindula Jayawardana, Catherine Tang, Sirui Li, Dajiang Suo, and Cathy Wu. The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning. In Advances in Neural Information Processing Systems , volume 35, pages 23881-23893. Curran Associates, Inc., 2022.

Vindula Jayawardana, Baptiste Freydt, Ao Qu, Cameron Hickert, Edgar Sanchez, Catherine Tang, Mark Taylor, Blaine Leonard, and Cathy Wu. Mitigating metropolitan carbon emissions with dynamic eco-driving at scale. arXiv preprint arXiv:2408.05609 , 2024.

Vindula Jayawardana, Baptiste Freydt, Ao Qu, Cameron Hickert, Zhongxia Yan, and Cathy Wu. Intersectionzoo: Eco-driving for benchmarking multi-agent contextual reinforcement learning. arXiv preprint arXiv:2410.15221 , 2024.

Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11) , pages 521-528, 2011.

Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research , 76:201-264, 2023.

Li Li, Yisheng Lv, and Fei-Yue Wang. Traffic signal timing via deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica , 3(3):247-254, July 2016. ISSN 2329-9266, 2329-9274. doi: 10.1109/JAS. 2016.7508798.

Siyuan Li and Chongjie Zhang. An Optimal Online Method of Selecting Source Policies for Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), April 2018. ISSN 23743468, 2159-5399. doi: 10.1609/aaai.v32i1.11718. URL https://ojs.aaai.org/index.php/AAAI/ article/view/11718 .

Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic simulation using sumo. In The 21st IEEE International Conference on Intelligent Transportation Systems . IEEE, 2018.

Daniel J Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, et al. Faster sorting algorithms discovered using deep reinforcement learning. Nature , 618(7964):257-263, 2023.

Amiel Meiseles and Lior Rokach. Source model selection for deep learning in the time series domain. IEEE Access , 8:6190-6200, 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, February 2015. ISSN 0028-0836, 1476-4687. doi: 10.1038/nature14236.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning , pages 1928-1937. PMLR, June 2016. ISSN: 1938-7228.

Jonas Mockus. Bayesian Approach to Global Optimization: Theory and Applications , volume 37 of Mathematics and Its Applications . Springer Netherlands, Dordrecht, 1989. ISBN 978-94-010-6898-7 978-94-009-0909-0. doi: 10.1007/978-94-009-0909-0.

Aditya Modi, Nan Jiang, Satinder Singh, and Ambuj Tewari. Markov Decision Processes with Continuous Side Information. In Proceedings of Algorithmic Learning Theory , pages 597-618. PMLR, April 2018. ISSN: 2640-3498.

Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. What to pre-train on? efficient intermediate task selection. arXiv preprint arXiv:2104.08247 , 2021.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021. URL http://jmlr.org/papers/v22/20-1364.html .

Sahand Rezaei-Shoshtari, Charlotte Morissette, Francois R. Hogan, Gregory Dudek, and David Meger. Hypernetworks for Zero-Shot Transfer in Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence , 37(8):9579-9587, June 2023. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai. v37i8.26146.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484-489, January 2016. ISSN 1476-4687. doi: 10.1038/nature16961.

Jivko Sinapov, Sanmit Narvekar, Matteo Leonetti, and Peter Stone. Learning Inter-Task Transferability in the Absence of Target Task Samples. In Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2015) , Istanbul, Turkey, May 2015.

Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning , pages 9767-9779. PMLR, 2021.

Mayuri Sridhar and Cathy Wu. Piecewise Constant Policies for Human-Compatible Congestion Mitigation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC) , pages 2499-2505, Indianapolis, IN, USA, September 2021. IEEE. ISBN 978-1-72819-142-3. doi: 10.1109/ITSC48978.2021. 9564789.

Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting. IEEE Transactions on Information Theory , 58(5):3250-3265, May 2012. ISSN 0018-9448, 1557-9654. doi: 10.1109/TIT.2011.2182033.

Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 9120-9132. PMLR, 13-18 Jul 2020. URL https://proceedings. mlr.press/v119/standley20a.html .

Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter-compositional multi-task reinforcement learning. Advances in Neural Information Processing Systems , 35:21495-21507, 2022.

Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems , 33:8728-8740, 2020.

Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction . Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-03924-6.

Yee Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Marius Wegener, Lucas Koch, Markus Eisenbarth, and Jakob Andert. Automated eco-driving in urban scenarios using deep reinforcement learning. Transportation Research Part C: Emerging Technologies , 126:102967, May 2021. ISSN 0968090X. doi: 10.1016/j.trc.2021.102967.

Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA, 2006.

Zhongxia Yan, Abdul Rahman Kreidieh, Eugene Vinitsky, Alexandre M. Bayen, and Cathy Wu. Unified Automatic Control of Vehicular Systems With Reinforcement Learning. IEEE Transactions on Automation Science and Engineering , pages 1-16, 2022. ISSN 1545-5955, 1558-3783. doi: 10.1109/TASE.2022. 3168621.

A Appendix

Contents

A.1 Notation

Table 3 describes the notation used in this paper.

Figure 10 helps understand the discrepancy between the observed generalized performance and the predicted one. Figure 11 illustrates how to calculate the marginal improvement of expected generalized performance ( ˆ V ( x ; π 1: k -1 ) -V ( x 1: k -1 ) ).

Target task

x

′

U

J

(

π

x

1

,x

1

)

x

1

J

(

π

x

1

, x

′

)

ˆ

J

(

π

x

1

, x

′

)

Target task

x

′

U

J

(

π

1

,x

1

)

x

1

ˆ

J

(

π

x

,x

)

x

J

(

π

1

, x

′

)

ˆ

J

(

π

x

, x

′

)

(

ˆ

J

(

π

x

, x

′

)

, J

(

π

1

, x

′

)

)

ˆ

V

(

x

;

π

1

)

-

V

(

x

1

)

A.2 Model-Based Transfer Learning (MBTL) Algorithm

Model-based Transfer Learning (MBTL)

A.3 Theoretical analysis

A.3.1 Proof of Theorem 2

Theorem 2. For a given δ ′ ∈ (0 , 1) and scaling factor β k = 2log( | X | π 2 k 2 / 6 δ ) , the cumulative regret R K is bounded by √ C 1 β K γ K ∑ K k =1 ( | X k | | X | ) 2 with probability at least 1 -δ ′ .

Proof. The following Lemma 3 and 4 is basically considering the cardinality of restricted search space of X k instead of X upon the lemmas in literature [41].

Lemma 3. For t ≥ 1 , if | f ( x ) -µ k -1 ( x ) | ≤ β 1 / 2 k σ k -1 ( x ) ∀ x ∈ X k , then the regret r t is bounded by 2 | X k | β 1 / 2 k σ k -1 ( x ) / | X | .

Using Lemma 3, Lemma 5.3 in [41], Lemma 4, and Cauchy-Schwarz inequality, we can bound the cumulative regret as:

A.3.2 Proof of Corollary 2.1

Corollary 2.1. Consider | X k | = 1 √ k | X | . The regret bound would be R K ≤ √ C 1 β K γ K log K with a probability of at least 1 -δ ′ .

Proof. Recall that ∑ K k =1 1 k ≤ log K .

Calculating the sum of squares for the reduced segments, we have:

The cumulative regret can be bounded as below:

A.3.3 Proof of Corollary 2.2

Corollary 2.2. The regret bound for the | X k | ≤ 2 -⌊ log 2 k ⌋ | X | would be R K ≤ √ C 1 β K γ K π 2 / 6 with a probability of at least 1 -δ ′ .

Proof. Calculating the sum of squares for the reduced segments, we have:

The cumulative regret can be bounded as below:

A.4 Experiment details

A.4.1 Details about Gaussian process (GP) Regresstion

We use the GaussianProcessRegressor implementation from scikit-learn , which follows Algorithm 2.1 of [48]. Specifically, we construct a kernel by multiplying a constant kernel

by a radial basis function (RBF) kernel

with an initial length scale ℓ = 1 . 0 (constrained to lie in the range [10 -2 , 10 2 ] ). To determine the hyperparameters, we begin by generating synthetic data that aligns with our modeling assumptions, including constant training performance and a linear generalization gap, while introducing noise to degrade generalization performance by up to 10%, sampled from a uniform distribution. We vary the GP hyperparameters, including noise standard deviation over the set { 0 . 001 , 0 . 01 , 0 . 1 , 1 } , the number of restarts for the optimizer over { 5 , 6 , . . . , 15 } , and explore several kernel configurations on the synthetic data. We then select the hyperparameter configuration that maximizes the average predictive performance. Specifically, we choose a noise standard deviation of σ = 0 . 001 and perform 15 random restarts of the hyperparameter optimizer to reduce the risk of convergence to poor local minima. We use the same GP hyperparameter configuration across all experiments and benchmarks.

A.4.2 Accuracy of generalization gap assumption

In Figure 12, we report the Pearson correlation between the observed generalization gap and the estimated gap under our linear assumption (Assumption 1). Each histogram shows how strongly the two measures align across various tasks in standard control (blue) and traffic (red) benchmarks. Many tasks cluster around moderate positive correlations (0.3-0.5), suggesting that a linear function of context similarity can reasonably capture the gap in most scenarios. However, certain tasks-such as Eco-driving-exhibit higher correlations (above 0.6), whereas others-such as HalfCheetah-are closer to 0, indicating that the assumption holds more effectively in some domains than in others.

8

Control Benchmarks

Traffic Benchmarks

7

Traffic Signal

6

:Control

AdvisoryAutonomy

Frequency

5

Penidulum

4

iBipedalWafker

iCartpole

Eco-driving

3

iHalfCheetahi

2

1

0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Pearson Correlation

A.4.3 Results of table with standard deviation

A.4.4 Detailed sample complexity comparison results

Table 5 presents a comparison of sample complexity required for MBTL to perform as good as the best generalization performance of baselines (independent training and multi-task training) across various tasks in the CMDP. Each row lists a different domain, the specific context variation applied (e.g., changes in physical properties or environmental parameters), and two key values: k ∗ and N , where k ∗ represents the number of models required by MBTL to reach a performance level comparable to the baseline. This value is shown as a range (e.g., [3 , 5 , 3] ), indicating results from three random seeds. N represents the total number of contexts. The value N k ∗ helps represent the sample efficiency of MBTL.

A.4.5 Details about traffic signal control benchmark

Most traffic lights operate on fixed schedules, but adaptive traffic signal control using DRL can optimize the traffic flow using real-time information on the traffic [8, 24], though challenges persist in generalizing across various intersection configurations [19].

Figure 13 showcases the layout of traffic networks used in a traffic signal control task with several lanes and a signalized intersection in the middle. The state space represents the presence of vehicles in discretized lane cells along the incoming roads. Actions determine which lane gets the green phase of the traffic signal, and rewards are based on changes in cumulative stopped time, the period when speed is zero. The global objective is to minimize the average waiting times at the intersection. Different configurations of intersections (e.g., road length, inflow, speed limits) are modeled to represent varying real-world conditions. To represent various real-world conditions, we vary factors such as road length, inflow rate, and speed limits from 0.1 to 5 times; by default, the road length is 750 meters, the flow rate is 1000 vehicles per hour, and the speed limit is 13.89 m/s.

Inflow

Length

0000

Speed Limit

SPEED

LIMIT

35

Guided vehicle

Default vehicle

Training configuration We used the microscopic traffic simulation called Simulation of Urban MObility (SUMO) [26] v.1.16.0 and PPO for RL algorithm [36]. We utilized the default implementation of the PPO algorithm with default hyperparameters provided by the Stable-Baselines3 library [34]. All experiments are done on a distributed computing cluster equipped with 48 Intel Xeon Platinum 8260 CPUs.

License Traffic signal control benchmark falls under MIT License.

1.0

1.0

1.0

ance

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

Normal

0.2

0.2

0.2

OracleTransfer

Independent

Multitask

0.0

0.0

0.0

0

10002000300040005000

0.0

13.89 27.78 41.67 55.56 69.45

0

750

1500

225030003750

FlowRate

Speed Limit

LengthofLane

Potential of multi-policy training and zero-shot transfer Figure 14 shows how each approach adapts to variations in flow rate, speed limit, and lane length for a traffic signal control benchmark. The y-axis shows normalized performance, with higher values indicating better control policies. Oracle Transfer consistently achieves superior performance across these different settings, owing to its ability to leverage full task information in a zero-shot transfer manner. By contrast, independent and multi-task training exhibit more pronounced performance drops and greater instability when faced with shifts in problem parameters, underscoring the challenges of generalizing policies in traditional DRL approaches.

Transferability heatmap Figure 15 presents heatmaps of transferability for different traffic signal control tasks, each varying a specific aspect: inflow, speed limit, and road length. The heatmaps display the effectiveness of strategy transfer from each source task (vertical axis) to each target task (horizontal axis). In terms of inflow variation, transferability drops when transferring from tasks with lower vehicle inflow to those with higher inflow. In speed limit variation, the transferability shows uniform effectiveness, suggesting less sensitivity to these changes. In road length variation, distinct blocks of high transferability indicate that different road lengths may require significantly tailored strategies.

(a) Inflow variation

(b) Speed limit variation

(c) Road length variation

10

0.8

10

0.8

10

0.8

Source task

20

0.6

Source task

20

0.6

Source task

20

0.6

30

0.4

30

0.4

30

0.4

40

0.2

40

0.2

40

0.2

10

20

30

40

0.0

Q

10

20

30

40

0.0

10

20

30

40

0.0

Target task

Target task

Target task

Results Figure 16 illustrates the normalized generalized performance across various traffic control tasks: inflow, speed limit, and road length. The plots display how different strategies adapt with increasing transfer steps:

Inflow: Performance improves as the number of transfer steps increases, with MBTL strategy consistently achieving the highest scores, demonstrating their effectiveness in adapting to changes in inflow conditions.

Speed Limit: Here, performance levels are relatively stable across all strategies except for the multi-task training.

Road Length: There is a general upward trend in performance for all strategies, particularly for MBTL, indicating robustness in adapting to different road lengths.

This data suggests that MBTL and Oracle are particularly effective across varying conditions, maintaining higher levels of performance adaptability.

Inflow

Speed Limit

Road Length

0.90

0.950

performance

0.85

0.85

0.925

0.80

0.900

0.80

0.75

0.875

Normalized

0.70

0.850

0.75

0.65

0.825

0.60

0.800

0.70

104

105

104

105

104

105

Number ofsamples trained (Log scale)

Numberof samplestrained(Log scale)

Numberofsamplestrained(Logscale

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq. Oracle

A.4.6 Details about eco-driving control benchmark

Given the significant portion of greenhouse gas emissions in the United States coming from the transportation sector [11], eco-driving behaviors are critical for climate change mitigation. Deep reinforcement learning-based eco-driving strategies have been developed [13, 47, 18, 20] but still have some issues of difficulties in generalization. We also extend to various intersection configurations with different traffic inflow rates, penetration rates of eco-driving systems, and durations of green phases at static traffic signals to optimize vehicle behaviors for reduced emissions.

Figure 17 illustrates the traffic road network used in the eco-driving control task. The road network is depicted as traffic flowing vertically and horizontally, crossing the static phase traffic signal. There are both guided and default vehicles in the system. The state space includes the speed and position of the ego vehicle, the leading vehicle, and the following vehicles, supplemented by the current traffic signal phase and relevant context features, including lane length and green phase durations. The action space specifically focuses on the ego vehicle's acceleration control. The reward mechanism is designed to optimize the driving strategy by balancing the average speed of the vehicles against penalties for emissions, thereby promoting eco-friendly driving behaviors within the traffic system.

Inflow

0.0

00

0 0

D

Green phase

Penetration rate

Guided vehicle

Defaultvehicle

Training configuration We also used the microscopic traffic simulation called Simulation of Urban MObility (SUMO) [26] v.1.16.0 and PPO for RL algorithm [36]. For detailed experimental details and RL hyperparameter configurations, please refer to [20, 21].

Potential of multi-policy training and zero-shot transfer Figure 1 shows how each RL training paradigm adapts to variations in green phase time, penetration rate, and inflow rate in the ecodriving control benchmark. Oracle Transfer remains the strongest method across all configurations, benefiting from zero-shot transfer. independent training shows unstable performance across different task variations, performance, while multi-task training lags behind. Overall, the trends highlight the advantage of leveraging zero-shot transfer in traffic CMDPs.

Transferability heatmap Figure 18 displays heatmaps for the eco-driving control task, with each heatmap varying an aspect such as green phase, inflow, and penetration rate. These visuals illustrate the transferability of strategies from source tasks (vertical axis) to target tasks (horizontal axis), highlighting the impact of traffic light phases, vehicle inflow, and the proportion of guided vehicles on strategy effectiveness. Notably, longer green phases correlate with improved performance and transferability. For inflow variations, reduced inflow typically yields better outcomes. However, variations in the penetration rate of guided vehicles show minimal impact on performance differences.

(a) Green phase variation

(b) Inflow variation

(c) Penetration rate variation

0.7

Q

Q

0.7

0.7

10

0.6

10

10

0.6

0.6

0.5

0.5

Source task

20

Source task

20

0.5

Source task

20

0.4

0.4

30

30

0.4

30

0.3

0.3

0.2

0.3

40

40

40

0.2

0.1

0.2

10

20

30

40

0

10

20

30

40

0

10

20

30

40

0.0

Target task

Target task

Target task

Results Figure 19 illustrates the normalized generalized performance across variants of eco-driving control tasks, specifically looking at variations in green phase time, inflow, and penetration rate. The graphs depict performance enhancement over transfer steps for different strategies. Notably, MBTL consistently demonstrates superior performance across all variations, indicating robust adaptability to changing task parameters.

GreenPhaseTime

Inflow

PenetrationRate

Iperformance

0.50

0.6

0.5

0.45

0.5

0.4

Normalized

0.4

0.40

0.3

0.3

0.35

林

0.2

104

105

104

105

104

105

Number of samples trained (Log scale)

Numberof samplestrained(Log scale)

Numberofsamplestrained(Log scale

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq. Oracle

A.4.7 Details about advisory autonomy benchmark

Advisory autonomy involves a real-time speed advisory system that enables human drivers to emulate the system-level performance of autonomous vehicles in mixed autonomy systems [40, 7, 15]. Instead of direct and instantaneous control, human drivers receive periodic guidance, which varies based on road type and guidance strategy. Here, we consider the different frequencies of this periodic guidance as contextual MDPs since the zero-order hold action affects the transition function.

Figure 20 illustrates two distinct traffic network configurations used in the advisory autonomy task: a single-lane ring and a highway ramp. The single-lane ring features 22 vehicles circulating the ring, with only one being actively controlled, presenting a relatively controlled environment for testing vehicle guidance systems. The highway ramp scenario introduces a more complex dynamic, where vehicles not only travel along the highway but also merge from ramps, creating potential stop-and-go traffic patterns that challenge the adaptability of autonomous guidance systems.

Problem Definition In a single-lane scenario, the state space includes the speeds of the ego and leading vehicles, along with the headway. For highway ramp scenarios, additional states cover the relative positions and speeds of adjacent vehicles. Actions vary by guidance type: for acceleration guidance, the action space is continuous, ranging from -1 to 1 ; for speed guidance, it has ten discrete actions compared to the speed limit. Rewards are based on system throughput or average speed of all vehicles in the system.

Context Variations We explore different durations of coarse-grained guidance holds to test various levels of human compatibility, adjusting the model based on observed driver behaviors and system performance.

License Advisory autonomy benchmark falls under MIT License [40].

Guidance hold duration

8

(e.g.,10 sec)

0000

Single lane Ring

1.0

0.975

1.0

1.0

performance

Independent

0.8

0.950

0.8

0.9

Multitask

0.6

0.925

0.6

0.8

Normalized

0.900

0.7

0.4

0.4

0.875

0.6

0.2

0.850

0.2

0.5

0.0

0.825

0.0

0.4

0

1

2

3

4

0

1

2

3

4

0

1

2

3

4

0

1

2

3

4

Ring Acc

Ramp Acc

Ramp Vel

Potential of multi-policy training and zero-shot transfer Figure 21 shows that ring-road networks tend to yield higher performance and smaller gaps compared to highway ramp scenarios. In addition, independent training exhibits greater performance drop and variability due to training instability. Oracle Transfer retains clear potential improvements over other baselines.

Transferability heatmap Figure 22 showcases heatmaps of transferability for advisory autonomy tasks, each varying in specific aspects: acceleration guidance and speed guidance across a single lane ring and a highway ramp. These heatmaps demonstrate the effectiveness of strategy transfer from each source task (vertical axis) to each target task (horizontal axis), capturing how variations in task conditions influence adaptability. For acceleration guidance in a ring setup (a), transferability is generally higher among tasks with similar acceleration demands. In contrast, speed guidance on a ramp (d) reveals more variability in transferability, potentially due to the complexity of speed adjustments in ramp scenarios.

Results Figure 23 illustrates the comparison of normalized generalized performance for advisory autonomy tasks, specifically acceleration and speed guidance in a ring and acceleration guidance on a ramp. The graphs demonstrate that MBTL consistently exhibits higher performance across all tasks. Particularly, acceleration guidance in both ring and ramp scenarios shows significant performance improvements over transfer steps, with MBTL closely matching in some instances.

A.4.8 Details of control benchmarks

For this experimental phase, we selected context-extended versions of standard RL environments from the CARL benchmark library, including Cartpole, Pendulum, BipedalWalker, and Halfcheetah. These environments were chosen to rigorously test the robustness and adaptability of our MBTL algorithm under varied conditions that mirror the complexity encountered in real-world scenarios.

D

1.0

D

5

8'0-

0.95

5

0.8

0.9

10

10

10

10

8°0

Source task

15

0.6

Source task

0.90

Source task

1520

0.6

520

0.7

25

0.4

25

0.85

25

0.4

25

0.6

30

0.2

OE

0.80

30

0.2

30

0.5

35

35

0.75

35

35

0.4

0

5

10

15

Target task

20

25

30

35

0'0

0

5

10

15

Target task

20

25

3035

5

10

15

Target task

20

25

3035

Q

10

15

Target task

20

25

3035

(a) Ring with acceleration guide (b) Ring with speed guide (c) Ramp with acceleration guide (d) Ramp with speed guide

AccelerationguidanceinRing

Speed guidance in Ring

AccelerationguidanceinRamp

performance

0.98

0.9

林

0.97

0.6

0.8

0.96

0.4

Normalized

0.7

0.95

0.94

0.6

0.2

0.93

0.5

104

105

104

105

104

105

Numberof samples trained (Log scale)

Number ofsamples trained (Log scale)

Number of samples trained (Log scale)

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq.Oracle

Context Variations: In the Cartpole tasks, we explored CMDPs with varying cart masses, pole lengths, and pole masses. For the Pendulum, the experiments involved adjusting the timestep duration, pendulum length, and pendulum mass. The BipedalWalker was tested under different settings of friction, gravity, and scale. Similarly, in the Halfcheetah tasks, we manipulated parameters such as friction, gravity, and stiffness to simulate different physical conditions. These variations critically influence the dynamics and physics of the environments, thereby presenting unique challenges that test the algorithm's capacity to generalize from previous learning experiences without the need for extensive retraining. The range of context variations was established by scaling the default values specified in the CARL framework from 0.1 to 10 times, enabling a comprehensive examination of each model's performance under drastically different conditions.

We utilized the default implementation of the DQN, A2C, and PPO algorithm with default hyperparameters provided by the Stable-Baselines3 library [34].

License: CARL falls under the Apache License 2.0 as is permitted by all work that we use [5].

A.4.9 Details about Cartpole benchmark

Potential of multi-policy training and zero-shot transfer Cartpole may be considered a simpler benchmark than traffic benchmarks, yet independent and multi-task training methods still face notable difficulty when faced with context variations. Figure 24 shows that Oracle Transfer performs at the highest performance across problem variations, while independent training or multi-task training shows a larger variance in performance.

1.0

1.0

1.0

performance

0.8

0.8

0.8

0.6

0.6

0.6

Normalized

0.4

0.4

0.4

OracleTransfer

0.2

Independent

Multitask

0.2

0.2

0.0

1.0

2.0

3.0

4.0

5.0

0

2

4

6

8

10

0.0

0.2

0.4

0.6

0.8

1.0

LengthofPole

MassofCart

Mass ofPole

Transferability heatmap Figure 25 presents transferability heatmaps for the Cartpole task with variations in three physical properties: mass of the cart, length of the pole, and mass of the pole. Each heatmap illustrates how well strategies transfer from source tasks (vertical axis) to target tasks (horizontal axis), depicting the influence of each parameter on control strategy effectiveness. For the mass of the cart variation (a), transferability decreases as the mass difference increases. In the length of the pole variation (b), strategies are less transferable between significantly different pole lengths. Similarly, for the mass of the pole variation (c), variations show divergent transferability depending on the extent of mass change.

(a) Mass of cart variation

(b) Length of the pole variation

(c) Mass of pole variation

1.0

1.0

1.0

20

0.8

20

0.8

20

0.8

Source task

40

0.6

Source task

40

0.6

Source task

40

0.6

60

0.4

60

0.4

60

0.4

80

0.2

80

0.2

80

0.2

0

20

40

60

80

20

40

60

80

0

20

40

60

80

Target task

Target task

Target task

Results Figure 26 presents a comparison of normalized generalized performance for the Cartpole task across different strategies when varying the mass of the cart, length of the pole, and mass of the pole. In the mass of cart variation, performance generally increases with transfer steps, with MBTL strategies achieving the highest scores. This indicates robust adaptability to changes in cart mass. Similar trends are observed with length variation and mass of pole variation. MBTL shows close to Oracle performance.

A.4.10 Details about Pendulum benchmark

Potential of multi-policy training and zero-shot transfer Pendulum is also one of the simplest benchmarks in classic control. In the Pendulum benchmark, we vary three key parameters: time step (left), pole length (middle), and ball mass (right). As shown in Figure 27, a few well-trained policies in specific contexts transfer effectively to new tasks, particularly under Oracle Transfer. For certain configurations (e.g., shorter poles, lighter balls), Independent and Oracle Transfer both excel, while multi-task struggles. These results suggest a remaining performance gap that multi-policy training and zero-shot transfer could help bridge.

MassofCart

LengthofPole

MassofPole

performance

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

Normalized

0.4

0.4

0.4

0.2

0.2

0.0

0.2

104

105

104

105

104

105

Numberof samplestrained (Log scale)

Number of samplestrained(Log scale)

Number of samplestrained(Log scale)

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq. Oracle

1.0

1.2

1.2

OracleTransfer

performance

Independent

Multitask

0.9

1.0

1.0

0.8

0.8

0.8

Normalized

0.7

0.6

0.6

0.6

0.4

0.4

0.5

0.2

0.2

0.0

0.1

0.2

0.3

0.4

0.5

0

2

4

6

8

10

0

2

4

6

8

10

Time Step

LengthofPole

Mass of Ball

Figure 28 provides a visual mapping of which trained policy is being applied for each specific context in the Pendulum benchmark. CMDP with time step variation demonstrates that the tasks are covered by a few 'good" policies nearby. In addition, contexts with shorter poles and lighter balls often gravitate toward a single high-performing policy, whereas more challenging configurations may require a specialized or distinct policy. This illustrates how Oracle Transfer can seamlessly select from a suite of learned policies, demonstrating robust zero-shot transfer and stronger adaptability.

OracleTransfer

berformance

1.0

1.2

1.2

Independent

Multitask

0.9

1.0

1.0

0.8

p

0.8

alized

0.6

0.8

0.7

Norma

0.4

0.6

0.6

0.2

0.0

0.05

0.1

0.15

0.2

0.25

0

1

2

3

4

5

0

1

2

3

4

5

TimeStep

LengthofPole

Mass of Ball

Transferability heatmap Figure 29 presents transferability heatmaps for the Pendulum task with variations in three physical properties: timestep, length of the pendulum, and mass of the pendulum. Each heatmap illustrates how effectively strategies transfer from source tasks (vertical axis) to target tasks (horizontal axis), highlighting the impact of each parameter on control strategy effectiveness. For the timestep variation (a), there appears to be high consistency in transferability across different timesteps, especially around the diagonal axis. In the length of the pendulum variation (b), transfer-

ability decreases with greater length differences. Similarly, for the mass of the pendulum variation (c), transferability shows variability dependent on the extent of mass changes.

(a) Timestep variation

(b) Length of the pendulum varia-

(c) Mass of pendulum variation

D

1.0

1.2

1.2

0.9

20

0.8

20

1.0

20

1.0

Source task

40

0.7

Source task

40

0.8

Source task

40

0.8

0.6

0.6

60

0.5

60

60

0.6

0.4

0.4

80

0.3

80

0.2

80

0.4

0.2

0.0

0.2

0

20

40

60

80

D

20

40

60

80

0

20

40

60

80

Target task

Target task

Target task

tion

Results Figure 30 shows a comparison of normalized generalized performance for the Pendulum task across different strategies when varying the timestep, length of the pendulum, and mass of the pendulum. For the length of the pendulum variation and mass of the pendulum one, MBTL strategies demonstrate the highest scores, suggesting robust adaptability to changes in pendulum dynamics. MBTL shows performance close to that of the Oracle across all variations, indicating its effectiveness in handling dynamic changes in system parameters.

TimeStep

LengthofPendulum

MassofPendulum

0.9

performance

0.80

0.70

0.8

0.75

0.7

0.65

Normalized

0.6

0.70

0.60

0.5

0.65

0.55

104

105

104

105

104

105

Numberof samples trained (Log scale)

Number of samplestrained(Log scale)

Number of samples trained (Log scale

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq.Oracle

A.4.11 Details about BipedalWalker benchmark

Potential of multi-policy training and zero-shot transfer Figure 31 compares the performance of different RL training methods in the BipedalWalker benchmark. independent training typically performs nearly as well but suffers intermittent dips. Similarly, multi-task training experiences larger swings, occasionally collapsing to low performance in certain parameter regions. However, Oracle Transfer remains near-perfect across every setting. These patterns highlight how multi-policy training with zero-shot transfer and per-task training generally fare better than a single universal model when faced with diverse environment dynamics.

1.0

1.0

1.0

hance

e

0.9

0.9

0.9

erforma

0.8

0.8

0.8

OracleTransfer

e

p

0.7

0.7

0.7

Independent

e

Z

0.6

0.6

0.6

Multitask

2

0.5

0.5

0.5

0.4

0.4

0.4

0.0

5.0

10.0

15.0

20.0

25.0

0

20

40

60

80

100

0

60

120

180

240

300

Friction

Gravity

Scale

Transferability heatmap Figure 32 presents transferability heatmaps for the BipedalWalker task, focusing on three variations: friction, gravity, and scale. Each heatmap illustrates the effectiveness of strategy transfer from source tasks (vertical axis) to target tasks (horizontal axis), highlighting how each parameter influences control strategy adaptability. For friction variation (a), strategies show uniform transferability across different friction levels. In gravity variation (b), transferability is highly variable, suggesting that strategies need specific tuning for different gravity levels. For scale variation (c), the heatmap indicates variable transferability, reflecting the challenges of scaling control strategies.

(a) Friction variation

(b) Gravity variation

(c) Scale variation

0.9

0.9

0.9

20

20

0.8

20

0.8

0.8

0.7

Source task

40

0.7

Source task

40

0.6

Source task

40

0.7

60

0.6

60

0.5

60

0.6

0.4

0.5

80

0.5

80

80

0.3

0.4

0.4

20

40

60

80

20

40

60

80

20

40

60

80

Target task

Target task

Target task

Results Figure 33 shows the comparison of normalized generalized performance for all variations within the BipedalWalker task. There is no huge difference in performance for all three cases, but if we look into the tabualr results in Table 2, MBTL shows the highest performance across varying conditions, indicating their robustness in adapting to changes in physical parameters of the model. This suggests that these strategies are more effective in handling the complexities introduced by different frictions, gravities, and scales compared to other baselines.

A.4.12 Details about HalfCheetah benchmark

Potential of multi-policy training and zero-shot transfer In this HalfCheetah benchmark, each subplot examines how policies adapt to changing friction, gravity, and stiffness (Figure 34). Oracle Transfer maintains nearly perfect scores for all parameter ranges, indicating robust zero-shot adaptability. In contrast, independent training experiences larger fluctuations, while multi-task training remains consistent yet at a lower performance plateau. The clear gap between Oracle Transfer and the other methods highlights the advantage of leveraging specialized multi-policy training solutions that effectively transfer across diverse dynamics.

Friction

Gravity

Scale

0.98

formance

0.90

0.96

0.9

0.94

0.8

0.85

p

0.92

0.80

alized

0.7

0.90

ormal

0.75

0.6

ION

0.88

0.70

0.86

0.5

104

105

104

105

104

105

Numberof samples trained (Log scale)

Numberof samplestrained (Log scale)

Numberof samplestrained(Log scale)

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq.Oracle

1.0

1.0

1.0

performance

0.9

0.9

0.9

0.8

0.8

0.8

0.7

0.7

0.7

Normalized

0.6

0.6

0.6

0.5

0.5

0.5

OracleTransfer

0.4

0.4

Independent

0.4

Multitask

0.3

0.3

0.0

1.2

2.4

3.6

4.8

6.0

0.0

19.6

39.2

58.8

78.4

98.0

0

30000600009000012000050000

Friction

Gravity

Stiffness

Transferability heatmap Figure 35 displays transferability heatmaps for the HalfCheetah task, focusing on three physical properties: friction, gravity, and stiffness. Each heatmap demonstrates the transferability of strategies from source tasks (vertical axis) to target tasks (horizontal axis). For friction variation (a), there is uniform high transferability across different friction levels, indicating that strategies are robust to changes in friction. Gravity variation (b) shows less consistent transferability, suggesting a sensitivity to gravity changes that might require adaptation of strategies. Stiffness variation (c) similarly demonstrates variable transferability, highlighting the challenges of adapting to different stiffness levels in control strategies.

(a) Friction variation

(b) Gravity variation

(c) Stiffness variation

D

Q

Q

1.0

0.9

0.9

20

20

20

0.9

0.8

0.8

0.8

Source task

40

Source task

40

Source task

40

0.7

0.7

0.7

60

60

60

0.6

0.6

0.6

80

0.5

80

80

0.5

0.5

0.4

Q

20

40

60

80

20

40

60

80

20

40

60

80

Target task

Target task

Target task

Results Figure 36 presents a comparison of normalized generalized performance across various strategies for the HalfCheetah task with respect to the varied physical properties. The results indicate that the MBTL generally outperforms others, particularly in managing variations in gravity and stiffness, suggesting the superior adaptability of these models to physical changes in the task

environment. The trends across different parameters confirm the critical impact of task-specific dynamics on the effectiveness of the tested strategies.

Friction

Gravity

Stiffness

1.0

1.0

performance

0.9

0.9

0.9

0.8

0.8

0.8

Normalized

0.7

0.7

0.7

0.6

0.6

0.6

0.5

0.5

0.5

104

105

104

105

104

105

Numberofsamples trained (Log scale)

Number of samplestrained(Log scale)

Numberof samples trained (Log scale)

Random

Independent

Multitask

Greedy

Equidistant

MBTL

Seq.Oracle

A.4.13 Implementation of the recent multi-task baselines

In Figure 37, we compare two recent multi-task algorithms-PaCo [43] and MOORE [16]-with several our baselines tested on Cartpole CMDP benchmark. Although MOORE underperforms relative to our baseline multitask implementation, PaCo achieves competitive or even higher performance at certain points, demonstrating its potential to generalize across multiple tasks. Also, it is important to note that those algorithms are naively implemented without thorough investigation. These trends show that enhanced multi-task strategies can be beneficial in some CMDP settings, whereas not all multi-task methods readily adapt to broader parameter variations.

MassofCart

Length of Pole

MassofPole

Normalized performance

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.0

0.2

104

105

104

105

104

105

Number ofsamples(Log scale)

Number ofsamples(Log scale)

Numberofsamples (Log scale)

Random

Multitask

Equidistant

Seq.Oracle

Multitask(MOORE)

Independent

Greedy

MBTL

Multitask(PaCo)

A.5 Potential impacts

Our work has the potential to reduce the computational effort needed to solve complex real-world problems, offering scalable solutions for implementing deep reinforcement learning in dynamic environments. While there are no immediate negative societal impacts identified, ongoing research will continue to assess the broader implications of deploying these technologies in urban settings.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the contributions and scope, matching the theoretical and experimental results presented in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Conclusion (Section 7).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper includes all necessary assumptions and provides complete proofs in the Appendix A.3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Detailed experimental settings, including models and hyperparameters, are provided to ensure reproducibility. Also, the code is submitted by the authors.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code are submitted by the authors with zip files, with detailed instructions for reproducing the results included in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides comprehensive details about the experiments, including data generation, hyperparameters, and optimizer types, to ease understanding of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars and confidence intervals are reported for all significant experiments, with methods and assumptions clearly stated.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The type of compute resources, memory, and execution time for each experiment are specified to ensure reproducibility in Appendix A.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research adheres to the NeurIPS Code of Ethics, with no deviations from the guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper includes a section discussing both the potential positive and negative societal impacts of the research.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not involve data or models with high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper uses the CARL framework under the Apache License 2.0, with proper credit given and license terms mentioned in the Appendix A.4.8.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets, including code for the MBTL algorithm and experiments, are well-documented and submitted with an anonymized zip file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Model-Based Transfer Learning (MBTL) in Contextual RL, which solves multiple related tasks and enhances generalization across different tasks. MBTL strategically selects a set of source tasks to maximize overall performance and minimize training costs.\nThe paper theoretically demonstrates that the method exhibits regret that is sublinear in the number of training tasks. \nMBTL achieves greater performance than other baselines (e.g., exhaustive training, multi-task training, and random selection) on urban traffic and control benchmarks.\n\nSTRENGTHS:\n- The paper is well-written and organized, and includes a thorough illustrative diagrams and examples.\n- The problem formulation is well done, with clear mathematical representation. The analysis of Bayesian Optimization appears to be thorough.\n- The authors provide code for training and evaluating their proposed method. This significantly facilitates reproducing the experimental results and extending the introduced method.\n\nWEAKNESSES:\n- The assumptions are too tight, particularly Assumption 3, which models the generalization gap using linear constraints. This approach is unsuitable for complex environments and therefore lacks generalizability.\n- The experimental environments (urban traffic and control benchmarks) are overly simplistic, consisting solely of vector environments with low-dimensional state spaces. The study lacks comparisons with complex tasks, such as those in the CARL benchmark, including games and a real-world application of RNA design.\n- The ablations on DRL algorithms (DQN, PPO, A2C) utilize outdated methods. Why not use more recent RL baselines?\n\nQUESTIONS:\nBelow, I have a few questions and feedback for the authors:\n- How does the computational time consumption compare between MBTL and other baselines (exhaustive training, multi-task training, and random selection)?\n- I am curious to see experimental results in complex environments, such as visual environments.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new framework to estimate the expected generalization performance across different tasks where the differences have an explicit and model-based structure. To improve the expected generalization performance via training on selected tasks, the paper proposes naive and Bayesian optimization based method to effectively explore the task space to find a policy that have the optimal zero-shot performance given a selected task. The experiments demonstrate that the proposed MBTL method outperform other baseline method and reach the performance of the optimal transfer method assuming full knowledge.\n\nSTRENGTHS:\n- By and large, the paper is written well. I especially appreciated the detailed discussion on SSTS problem formulation and its relations to robust RL training.\n- The idea of Bayesian Optimization to search training tasks using generalized performance estimation where tasks differences are “model-based” or have an explicit structure is novel and useful.\n- The section on analysis of BO and its comparison with ES and PS are also interesting, and highlights the sublinear regret theoretical results.\n- The theoretical results appear to be correct.\n\nWEAKNESSES:\n- The empirical evaluation is by and large only on low dimensional systems. It would have been interesting to see how the method would scale with more challenging, high-dimensional tasks, such as those commonly found in vision control tasks in robotics.\n- It would have been interesting to see how well this method would work with policies/controllers not parametrized with neural networks (i.e. kernel machines).\n- Some additional commentary on how to use the GP to search for a trained policy given a selected task.\n\nQUESTIONS:\n- Could you comment on the challenges of applying MBTL-GP on vision-control tasks?\n\nLIMITATIONS:\nYes, the author has discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework called Model-Based Transfer Learning (MBTL) for solving contextual reinforcement learning problems. By modelling the performance loss as a simple linear function of task context similarity, the authors leverages Bayesian optimization techniques and provides theoretical analysis showing that MBTL exhibits sublinear regret in the number of training tasks, and discusses conditions to further tighten the regret bounds. MBTL is validated on a range of simulated traffic management scenarios and standard control benchmarks, demonstrating superior performance compared to baselines like exhaustive training, multi-task training, and random task selection.\n\nSTRENGTHS:\n1. In general, the paper is easy to follow and well-motivated. The figures are helpful for readers to quickly grasp the key concepts and problem settings.\n\n2. Active and transfer learning for contextual RL is an interesting and practical problem, especially given that exhaustive/multi-task training on RL tasks can be computationally expensive.\n\n3. The experiments are conducted on a relative broad range of benchmarks and context variations.\n\nWEAKNESSES:\nI do think there are several major concerns regarding the current form of the paper:\n\n1. The paper (especially the theoretical analysis) is based on too many assumptions: continuity of task space and performance function, linear genearalization gap and deterministic MDP transitions, which oversimplifies the problem and significantly limits the applicability range of the proposed method. The most concerning is Assumption 3 (Linear generalization gap), for which I have no clue why and when this can hold. Assuming Lipschitz continuity of the generalization gap seems much more reasonable to me, which makes assumption 3 an inequality instead of an (approximate) equality.\n\n2. In line 96-98, the paper assumes that a vector representation/context feature $x$ of each task is given a priori, based on which the continuity assumptions 1-2 are made.  Howeveer, as in line 333-335,  context features are often not visible. Assuming prior knowledge of such information further limits the applicability range of the proposed method.\n\n3. (2 continued) In fact, no matter whether context features are given a priori, there are many effective methods which perform transfer/meta-RL by learning a conditioned policy [1][2][3], which should be significantly better than the \"multi-task\" baseline adopted in the paper. However, there is currently no empirical evidence on how well the proposed method compared to these stronger baselines.\n\n4. Even though Theorem 2 and Corollary 2.1 2.2 give quantitative regret bounds assuming the search space can be narrowed down at each training step, there is no theoretical guarantee on how the proposed method, especially MBTL-GP, can effectively realize such restricted search space. Hence the current analysis is not complete in terms of proving the effectiveness of proposed method.\n\n[1] Rakelly, Kate, et al. \"Efficient off-policy meta-reinforcement learning via probabilistic context variables.\" International conference on machine learning. PMLR, 2019.\n\n[2] Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pages 9767–9779. PMLR, 2021.\n\n[3] Li, Lanqing, Rui Yang, and Dijun Luo. \"Focal: Efficient fully-offline meta-reinforcement learning via distance metric learning and behavior regularization.\" ICLR 2021.\n\nQUESTIONS:\nWhat's the foundamental difference between \"exhaustive training\" and \"multi-task RL\"? Based on the brief description in line 257-258, they seem similar to me. Also why is \"multi-task RL\" even worse than \"random\" most of the time (in Table 1, 2)?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-26"}