Model-Based Transfer Learning for Contextual Reinforcement Learning
Authors: Jung-Hoon Cho
Vindula Jayawardana
Sirui Li
Cathy Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Reinforcement Learning
Zero-Shot Transfer
Generalization
Bayesian Optimization
Primary area: reinforcement_learning

Abstract: Deep reinforcement learning (RL) is a powerful approach to complex decision-making. However, one issue that limits its practical application is its brittleness, sometimes failing to train in the presence of small changes in the environment. Motivated by the success of zero-shot transfer—where pre-trained models perform well on related tasks—we consider the problem of selecting a good set of training tasks to maximize generalization performance across a range of tasks. Given the high cost of training, it is critical to select training tasks strategically, but not well understood how to do so. We hence introduce Model-Based Transfer Learning (MBTL), which layers on top of existing RL methods to effectively solve contextual RL problems. MBTL models the generalization performance in two parts: 1) the performance set point, modeled using Gaussian processes, and 2) performance loss (generalization gap), modeled as a linear function of contextual similarity. MBTL combines these two pieces of information within a Bayesian optimization (BO) framework to strategically select training tasks. We show theoretically that the method exhibits sublinear regret in the number of training tasks and discuss conditions to further tighten regret bounds. We experimentally validate our methods using urban traffic and standard continuous control benchmarks. The experimental results suggest that MBTL can achieve up to 43x improved sample efficiency compared with canonical independent training and multi-task training. Further experiments demonstrate the efficacy of BO and the insensitivity to the underlying RL algorithm and hyperparameters. This work lays the foundations for investigating explicit modeling of generalization, thereby enabling principled yet effective methods for contextual RL. Code is available at https://github.com/jhoon-cho/MBTL/.

Model-Based Transfer Learning for Contextual Reinforcement Learning
Jung-Hoon Cho MIT jhooncho@mit.com edu
Vindula Jayawardana MIT
vindula@mit. edu
Sirui Li MIT siruil@mit. edu
Cathy Wu MIT cathynu@mit.edi
Abstract
Deep reinforcement learning (RL) powerful approach complex decision making. However, one issue that limits its practical application is its brittleness, sometimes failing to train in the presence of small changes in the environment. Motivated by the success of zero-shot transfer- where pre-trained models perform well on related tasks- we consider the problem of selecting good set of training tasks to maximize generalization performance across range of tasks. Given the high cost of training, it is critical select training tasks strategically, but not well understood how do so. We hence introduce Model-Based Transfer Learning (MBTL), which layers on top of existing RL methods to effectively solve contextual RL problems. MBTL models the generalization performance two parts: 1) the performance set point, modeled using Gaussian processes, and 2) performance loss (generalization gap), modeled contextual similarity. MBTL combines these two pieces information within Bayesian optimization (BO) framework to strategically select training tasks. We show theoretically that the method exhibits sublinear regret in the number of training tasks and discuss conditions to further tighten regret bounds. We experimentally validate our methods using urban traffic and standard continuous control benchmarks. The experimental results suggest that MBTL can achieve up to 43x improved sample efficiency compared with canonical independent training and multi-task training. Further experiments demonstrate the efficacy of BO and the insensitivity to the underlying RL algorithm and hyperparameters. This work lays the foundations for investigating explicit modeling of generalization, thereby enabling principled yet effective methods for contextual RL. Code is available at https///gluub.cc////oooooooooooo
linear function of
Introduction
Deep reinforcement learning (DRL) has made remarkable strides in addressing complex problems across various domains 29 37 I 4 10 12 27). Despite these successes, DRL algorithms often exhibit brittleness when exposed small variations like different number of lanes, weather conditions, or flow density in traffic benchmarks (20). significantly limiting their scalability and generalizability
19 Such variations can be modeled using the framework of contextual Markov Decision Processes
(CMDP), where task variations can be parameterized within a context space [14 32 5].
There two predominant solution modalities for CMDPs 23
independent training and multi-task
training. Independent training constructs a separate model for each task variant (say, N > 1), which
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

0.7-
Oracle Transfer
Normalized performance
0.7
0.7
Independent
0.6
Multitask
0.5
0.5
0.5
0.4-
0.4
0.4
0.3
0.3
0.3-
0.2
0.2
0.2
0
10
20
30
40
50
0.0
0.2
0.4
0.6
0.8
1.0
400
800
1200 1600 2000
Green Phase Time
Penstration Rate
Inflow Rate
Figure 1: Normalized performance comparison across different problem variations in Eco-Driving benchmark. Traditional DRL approaches (e.g., Independent training or multi-task training) exhibit greater training instability, whereas Oracle Transfer, zero-shot transfer with full information, shows the potential for performance improvement by multi-policy training.
is compute-intensive. At the other extreme, multi-task training constructs a single "universal" policy, and thus can be more compute-efficient, but suffers from model capacity and negative transfer issues 12
46 2 42 There is thus need for more reliable training methodologies for generalization across tasks variants. In this work, we consider training an intermediate set K models, where > K > 1, in an effort to balance performance and efficiency; we refer to this strategy multi- multi--olicy training.
We build upon zero-shot transfer, a widely-used practical technique which directly applies a policy trained in one context (source task) to another (target task) without adaptation. Figure I shows that multi-policy training with zero-shot transfer has the potential to improve the performance even over the independent training. In this article, we strategically select source tasks by explicitly modeling the generalization performance estimate the value of training new source task.
A note on terminology. For brevity, we refer to task variants as tasks in the remainder of this article. We also the language of task and context interchangeably. We emphasize that this work focuses on
generalization (e.g., traffic signal control for intersection scenario variants) rather than across-domal generalization (e.g., distinct traffic control tasks). Additionally, is crucial to differentiate between training reliability, which concerns the ability to reliably train models across tasks, and model reliability (or robustness), which concerns the resistance of trained model to differences in tasks. This article is concerned with training reliability.
The main contributions of this work are:
We introduce Model-Based Transfer Learning (MBTL), a novel framework for solving CMDP sample efficiently (Figure To the best of our knowledge, this is the first work to explicitly model generalization performance for contextual RL (CRL). As such, our work opens the door for further investigation into reliable model-based methodologies for CRL.
We provide theoretical analysis for the sublinear regret of MBTL and give conditions for achieving tighter regret bounds.
We empirically validate our methods in urban traffic and standard continuous control benchmarks for contextual RL, observing up to 43x improvements in sample efficiency. We further include ablations on the components of the algorithm.
The remainder of the paper is organized as follows. After introducing notation in Section - we formally define the problem Section 3 A key contribution of our work the introduction of Gaussian process model acquisition function specifically tailored to the source task selection problem, which is detailed in Section In Section 4.3 we provide theoretical analysis of the regret bounds of our method, followed by an empirical evaluation across diverse applications in Section 5
2 Preliminaries and notation
Contextual MDP. A standard MDP is defined by the tuple M (S, P,P,P where S repre- sents the state space, A is the action space, P denotes the transition dynamics, R is the reward function, and p is the distribution over initial states (45. contextual MDP (CMDP), denoted by M (S, A, Ps, Rg pu) is collection of context-MDPs Mr parameterized by context

---

Marginal generalization
Estimated training performance (Gaussian Process)
(b)
performance
Generalization gep (lineer)
Performanc
Performance
Géneralization performance
1F2
Zero-shot
orgmax
Transfer
Ra-s Ap-s Fy 2t-1
Context space
Iteration
(d)
Acquisition funation
Performence
Select maxt
Acquisition function
Terminate
Train xy!
753
2
35g
T2
Zero-shot Transfer
argmax
Xg X1
23 Xg
Xa-e Xame Agpa App-
X1 Xg Xg Xame Tauut Ranu Xpoa
Figure Overview illustration for Model-based Transfer Learning. (a) Gaussian process regression is used to estimate the training performance across tasks using existing policies; (b) marginal generalization performance (red area) calculated using upper confidence bound of estimated training performance, generalization gap, and generalization performance; (c) selects the next training task that maximizes the acquisition function (marginal generalization performance); (d) once the selected task is trained, calculate generalization performance using zero-shot transfer.
variable x within context space X (assumed bounded). The context variable can influence dynamics, rewards, and initial state distributions [4]
32 51. We define source task performance [ma, Alg) as follows: train policy лæ task with the context
X using RL algorithm Alg (e.g.. PPO, SAC) and evaluate the policy by the expected return in the same MDP M2 with context x. For brevity, we will write as J(m,, x). We distinguish between estimated values (xx and observed outcomes J( .), with the latter measured after training and evaluation.
Generalization gap via zero-shot transfer. Con- sider zero-shot transfer from the trained policy Ta from source task (context-MDP) to solve another target task (co)text-MDP)) with the context x' X in the CMDP. Zero-shot transfer involves applying
1.0
Noraalzzdd performance
0.8
policy trained source task M2 to different target task Max, with x, x E X. This experiences performance degradation, also called generalization gap [17 23) For instance, Figure deppics that the performance degrades as the target task diverges from the source task, corresponding to an increasing gener- alization gap. Nonetheless, leveraging the notion that training is expensive but zero-shot transfer is cheap, we are interested in optimally selecting set source (training) tasks, such that the generalization perfor- mance on the target range of tasks maximized. We observe the generalization performance, denoted by J(H,, x²), by evaluating the target task x' based on the policy trained using source task via zero-shot generalization. We define the generalization gap as the absolute performance difference in average reward when transferring from source task æ to target task x':
0.4
0.2
0.0
Context variation
Figure 3: Example generalization gap de- picted for Cartpole CMDP. The solid lines show the zero-shot transfer generaliza- tion performance across contexts. Source tasks are indicated by dotted lines.
A J =
J(TT
J[
Generalization
Source task performance
Generalization performance
3 Problem formulation
Sequential source task selection problem. The selection of source MDPs from the CMDPs key to solving the overall CMDP [3]. We therefore introduce the sequential source task selection

---

(SSTS) problem, which seeks to maximize the expected performance across dynamically selected set of tasks. This problem cast as a sequential decision problem. in which the selection of tasks is informed through feedback from the observed task performance of the tasks selected and trained thus far. The notation 2yK indicates the selected source task at the k-th transfer step, where l ranges from to K. For brevity, we will denote Tw,, as x-- We denote the sequence 21, P2. Fq by Ti:k and 11, #2; ..., Tk by Ti:k-
Definition 1 (Sequential Source Task Selection Problem). This problem seeks to optimize the expected generalization performance across CMDP Mx by selecting task E X at each training stage. Specifically, at each selection step k, we wish to choose a distinct task 2i such that the expected cumulative generalization performance is maximized. This can be expressed by keeping track at each step, which policy to use for which task, and the corresponding generalization performance. Upon training the policy Way for source lask FR. the cumulative generulization performunce, which we abuse notation to denote as (TTI::) = [(To T1k-- Formally, this can be recursively defined based previous observations {J( (TK-1, for all x E X as follows:
xT;; T1sk-1) max (J(Tk, ( (T1))_1,,2) Vx E if > 1.
And (TT11 = JT1, Then, the overall sequential decision problem can be written as:
max (I = max (((X) [J (TTx, m1dk-1) s.t. Ikk E ( 25%
(3)
The state at each step k is defined by the best generalization performance for each task, achieved by policies trained in earlier stages, represented as J J[T11=k11 x²) for each target task x'. The action at each step is choosing new task zg. In general, SSTS exhibits stochastic transitions, for example due to randomness in RL training. For simplicity, this work, we assume deterministic transitions; that is, training context-MDP z will always yield the same performance J [ x) and generalization gap A (Tzz x'), Vx' E X. The problem's maximum horizon is defined by [X but can be terminated early if conditions are met (e.g. performance level, training budget).
4 Model-Based Transfer Learning (MBTL)
In this section, we introduce an algorithm called Model-based Transfer Learning to solve the SSTS problem. MBTL models the generalization performance in two parts: 1) the performance set point, modeled using Gaussian processes, and 2) generalization gap. modeled as a linear function of contextual similarity. MBTL combines these two pieces of information within a Bayesian optimization (BO) framework to sequentially select training tasks that maximize generalization performance.
4.1 Modeling assumptions
We consider a task set X that is continuous and the performance function (TT, æ). V(x) for policy be smooth over the task space X. In practice, such as control systems, tasks often vary continuously and smoothly rather than abruptly. For example, adjusting the angle of robotic arm by small amount typically results in small change in the system and optimal action. Inspired by the empirical generalization gap performance as observed in Figure 3 we estimate the generalization gap with linear function of contextual similarity.
Assumption 1 (Linear generalization gap). A linear function is used model the generalization gap function, formally AJ x') 0|z ='1, where f is the slope of the linear function and x and x' are the context of the source task and target task, respectively.
The relaxation of this assumption can yield additional efficiency benefits but also adds modeling complexity, and thus is interesting direction for future work.
4.2 Bayesian optimization
Bayesian optimization (BO) is a powerful strategy for finding the global optimum of an objective function when obtaining the function is costly, or the function itself lacks simple analytical form
BO integrates prior knowledge with observations to efficiently decide the next task to train by using the acquisition function. MBTL. is a BO method which optimizes for promising source tasks by leveraging Assumption in its acquisition function. The role of BO is to approximate V (œzi T1k-1)
31
6

---

(see Equation 3D using the data acquired thus far. The next source task 2y is then selected using this estimate.
Gaussian process (GP) regression. Within the framework of BO, we model the source training performance TTT Vr E X (iik using Gaussian process (GP) regression. Specifically, the function (x) is assumed to follow a GP (J(=, ~ (P(E)[(TT,:)) k(x, R))). where k(ar, .) is the covariance function, representing the expected product of deviations of (TT,22) and (Tg,a) from their respective means. Let Dx-1 denote the data observed up to iteration k - consisting of the pairs {(T), J (TT, x.))) k-1 The estimated performance Jy. after querying k - samples is updated as more samples are obtained. The posterior prediction of Jj. at new point æ, given the data Da-1 and previous inputs 11:4-1, 1. is normally distributed as P(Jk Dk-1) = (2), (x((z) and (z) are defined as pu (z) [[ + k" (K + + and df(x)) k(x, kJ k"KK- - I ', with k being the vector of covariances between r and each z; the observed data, i.e., k- = [k)( ......... k( -x- =)], and K is the covariance matrix for the observed inputs, defined as K prediction with every new observation, progressively improving the estimation.
This enables the GP to update its beliefs about the posterior
Acquisition function. The acquisition function plays critical role in BO by guiding the selection of the next source training task. At each decision step k, the task I& is chosen by maximizing the acquisition function, denoted by 2h = arg max, a(x; Fi:k-)). One effective strategy employed in the acquisition function is the upper confidence bound (UCB) acquisition function, which considers the trade-off between the expected performance of a task based on the current task (exploitation) and the measure of uncertainty associated with the task's 's outcome (exploration) 4T.. Especially in our case, the acquisition function can be designed as UCB function subtracted by generalization gap and so-far best performance. It is defined as follows:
a(z;#1=k-1> 1) = Ex (1x- + 3t 0z-) - A (TT,
(4)
where ++ represents max(-; 0) and we can use various forms of Bg. which the trade-off parameter between exploitation and exploration.
4.3 Regret analysis
We regret to quantify the effectiveness of source task selection based on BO. Specifically, define regret at iteration k as rx = V ( - V(Ix;; Tis--), where V VIzg; ((a--) represents the maximum generalization performance achievable across all tasks, and V( m1:k-1) is the performance at the current task selection Zg. Consequently, the cumulative regret after K iterations is given by Rx = Σ Th, summing the individual regrets over all iterations. Following the framework presented by Srinivas et al. [41). our goal is to establish that this cumulative regret grows sublinearly with respect to the number of iterations. Mathematically, we aim to prove that lim Rx
K = 0, indicating that, on average, the performance of our strategy approaches the optimal performance the number of iterations increases.
Regret of MBTL. Having established the general framework for regret analysis, we now turn our attention to the specific regret properties of our MBTL algorithm. To analyze the regret of MBTL, consider the scaling factor for the UCB acquisition function given by 3x = 2 [og[[X|==k=+ /65) in Equation (4). It is designed to achieve sublinear regret with high probability, aligning with the theoretical guarantees outlined in Theorem and 5 from [41]
Theorem 1 (Sublinear Regret). Given cumulative regret Rg is bounded by VKC1BK7K with probability of at least 1 8. The formal expression of this probability is Pr RR S VKC,3KKK 2 1 5, where Ci [og[1+0-2] 2 8n²
(0, 1), and with the scaling factor BR as defined. the
and 7k- Olog K) for the squared exponential kernel.
Impact of search space elimination. In this section, we demonstrate that strategic reduction of the possible sets, guided by insights from previous task selections or source task training performance, leads to significantly tighter regret bounds than Theorem I By focusing on the most promising regions of the task space, our approach enhances learning efficiency and maximizes the policy's performance and applicability. Given the generalization gap observed in Figure we observe that performance loss decreases as the context similarity increases. We model the degradation from
5

---

the source task using a linear function in Assumption 1 Training on the source task can solve sigmificant portion of the remaining tasks. Our method progressively eliminates partitions of the task space at a certain rate with each iteration. If the source task selected in the previous steps could solve the remaining target task sufficiently, we can eliminate the search space a desirable rate. Formally, we can define the search space at k-th iteration as follows:
Definition 2 (Search space). We define the search space X5 at iteration k as a subset of X, with each element x' E Xg, such that J [TT 1 S 2 (T
Given the generalization gap observed in Figure 3 we model the degradation from the source task using a linear function in Assumption [ While the figure might not strictly appear linear, the linear approximation simplifies analysis and is supported by empirical observations. Training on the source task can solve significant portion of the remaining tasks. Our method progressively eliminates partitions of the task space at a certain rate with each iteration. If the source task selection in the previous step sufficiently addresses the remaining target tasks, can reduce the search space at desirable rate. Consequently, at each step, we effectively focus on a reduced search space.
We leverage the reduced uncertainty in well-sampled regions to tighten the regret bound while slightly lowering the probability d in Theorem For the regret analysis, we propose the following theorem based on the generalization of Lemma 5.2 and 5.4 in 41 to the eliminated search space.
Theorem 2. For given 8' E 10 and scaling factor Bu = 2
/66)) the cumulative
regret Rg bounded by V VI EK (1xx1) with probability at least 1 8'.
Here, [X| denotes the cardinality of the set X, the number of elements in X. Theorem 2 matches the Theorem 1 when X₂ X for all k. This theorem implies that regret has a tighter or equivalent bound if we can design the smaller search space instead of searching the whole space. The comprehensive proof is provided in Appendix [A.3.I]
Here are some examples of restricted search space: If we consider an example where [ = vR
X
the regret can be bounded tighter than that of Theorem [
Corollary 2.1. Consider [xx] = 1 |. The regret bound would be Rx S VCI log K with a probability of least 8' Vk
0.7
MBTL-GP (Average)
0.6
Corollary 2.1 Corallary 2.2
In cases where the search space is defined using MBTL-
GS, the largest segment's length would reduce geometrically, described by [X&]l S 2- 2oolo RJ x).
Max Space
0.5
0.4
0.3
Corollary 2.2. The regret bound for the [Xxl 2 20 RJ ||| would be RK= I /C13KTKT with a prob-
0.2
0.1
ability of at least 1 - 8'.
2 4
7 9 10 Transfer Steps
Proofs for Corollaries 2.1 and 2.2 are provided in Ap- pendix A.3.2 and A.3.3 respectively. Based experi- ments presented in Section 5] the rate of climination of the largest segment for MBTL shown in Figure
Figure 4: Empirical results the re- striction of search space by MBTL compared to two examples from Corollaries 2.1 and 2.2.
Experiments and analysis
5.1
Setup
Our experiments consider CMDPs that span standard and real-world benchmarks. In particular, consider standard continuous control benchmarks from the CARL library [5]. In addition, study problems from RL for intelligent transportation systems, using [49 to model the CMDPs. Surprisingly, despite the relatively low complexity of the CMDPs considered, standard deep RL algorithms appear to struggle to solve the tasks.
Baselines. We consider two types of baselines when evaluating our proposed algorithm: canonical and multi-policy. The canonical baselines are selected validate multi-policy training; the multi-policy training baselines heuristic methods designed to validate the Bayesian optimization approach.

---

Table 1: Comparative performance of different methods on traffic CMDPs (K 15)
Benchmark (CMDP)
Baselines
Multi-policy Baselines
MBTL
Oracle
Domain
Context Variation Independent Multi-task Random Equidistant
Greedy
Ours
Sequential
Number Trained Models
NN
k
k
NN
Traffic Signal
Road Length
0,9409
0.8242
0,9366
0,9337
0,9349
0,9364
0.9409
Traffic Signal
Inflow
0.8646
0,8319
0.8699
0.8712
0.8682
0.8432
0,8768
Traffic Signal
Speed Limit
0.8857
0.6083
0.8872
0.8872
0.8874
0.8867
0,8876
Eco- Eco-Driving
Penetration Rate
0.526
0.1945
0.6212
0.6278
0.5992
0.6148
0,666
Eco-Driving
Inflow
0.4061
0,2229
0,5077
0,5513
0.5299
0.5172
0.5528
Feo-Driving
Green Phase
0.385
0.4228
0.4724
0.4697
0.4678
0.4985
0.5027
AA-Ring-Ace
Hold Duration
0.8362
0.9209
0.9306
0,9225
0.9136
0.9382
0.9552
AA-Ring-Vel Vel
Hold Duration
0,9589
0.972
0,9819
0,9819
0,9819
0,9818
0,9822
A--aamp-Ace
Hold Duration
0.4276
0,5158
0.6750
0.6821
0.6882
0.6964
0.7111
AA-Ramp- Vel
Hold Duration
0.5473
0.5034
0.7170
0.7386
0.6918
0.7273
0.7686
Average
0.6778
0.6017
0,7600
0.7666
0.7563
0.7641
0.7844
HHigher the better. Bold values represent the highest value(s) wathin the stutistically sigmilicant range for each CMDP, excluding the oracle.
Detailed results with variance for each method are provided Appendix (3.43
$AA: Advisory auronomy benchmark, Ring: Single lane ring. Ramp: Highway ramp, Acc: Acceleration guidance, Vel: Suidannnaance.
The canonical baselines include: (1) Independent training, which involves independently training separate models on each task; and (2) Multi-task RL, where single "universal" context-conditioned policy is trained for all tasks. The multi pulti-policy baselines include: (3) Random selection, where each training task is chosen uniformly at randomly; (4) Equidistant strategy, which selects training tasks by equally subdividing the context space based on given training budget, and then trains them in lexicographical order; (5) Greedy strategy, which greedily selects the next source task based on Assumption [ and fixed training performance; and (6) Sequential Oracle transfer, which has access to generalized performance corresponding to policies for all tasks (including those not yet selected) and uses that information to greedily select the best source task at each step.
Proposed method. We evaluate MBTL with the scaling factor Bg = 2 | /65).
DRL algorithms and performance measure. We utilize Deep Q-Networks (DQN) for discrete action spaces (29) and Proximal Policy Optimization (PPO) for continuous action spaces [3). For each experiment three times with different random seeds. We evaluate our method by the average performance across all N target tasks after training up to K =15 source tasks or the number of source tasks needed to achieve a certain level of performance. We employ min-max normalization of the rewards for each task, and we provide comprehensive details about our model in Appendix 1.4.1
statistical reliability,
5.2 Traffic benchmark experiments
We consider three traffic control benchmarks. First, while most traffic lights still operate on fixed schedules, RL can be used to design adaptive (1) Traffic signal control optimize traffic [8] 24). However, considering that every intersection is different, challenges persist in generalizing across intersection configurations [9]] Given the significant portion of greenhouse gas emissions in the United States due to transportation M. the second traffic domain is (2) Dynamic eco-driving at signalized intersections, which concerns learning DRL-based eco-driving strategies have been developed [3] 47 [8] but still experience difficulties in generalization. Our final traffic domain is (3) Advisory autonomy, in which real-time speed or acceleration advisories guide human drivers to act as vehicle-based traffic controllers in mixed traffic environments 440 15. The context space X discretized into N = (50, 50, 40} contexts for the three domains, respectively. In Appendix A.4 we provide details about our experiments.
efficient driving strategies in urban settings.
Results. Table and Jsummarie the results. Notably, the Oracle far outperforms the standard baselines (independent and multi-task training), indicating the potential for transfer learning and intelligent training of multiple models, respectively. MBTL rapidly approaches the Oracle within = 10 transfer steps, indicating that the GP effectively models the training performance and linear generalization gap models the generalization performance. is important to note that multi-task RL studies commonly consider Independent training as strong baseline due to its avoidance of negative transfer and other training instability issues. Indeed, independent training often (but not always) outperforms multi-task training in our experiments. Yet, our experiments show that MBTL outperforms both independent and multi-task baselines and matches their performance with up to 30x improved sample efficiency. Among the multi-policy basclines, MBTL often outperforms the
7

---

Traffic Signal Control (Inflow)
Eco-Driving (Green Phase Time)
Advisory Autonomy
(Acceleration guidance in Ring)
Normalized performance
0.50
0.85
0.9
0.45
0.8
0.80
3x
0.40
0.7
0.75
13x
0.35
0.70
10
10
10"
10
10' samples (Log scale)
10
Number samples (Log scale)
samples (Log scale)
Independent
Multitask
Greedy
Equidistant
Seq.
Figure 5: Traffic CMDP results. Method comparison of normalized performance over N tasks. MBTL efficiently selects source training tasks. The black dotted line indicates the first training step within MBTL that exceeds both independent and multi-task baselines, with up to 30x fewer samples.
Table 2: Comparative performance of different methods standard control CMDPs (K 15)
Benchmark (CMDP)
Baselines
Multi-policy Baselines
MBTL
Oracle
Domain
Context Variation Independent Multi-task Random Equidistant Greedy
Ours
Sequential
Number of Trained Models
NV
K
k
Pendulum
Length
0.7383
0.6830
0.7607
0.7601
0.7774
0.7755
0. 0.7969
Pendulum
Mass
0.6237
0.5793
0.6647
0.6794
0.6887
0.6667
0.7132
Pendulum
Timestep
0.8135
0.7247
0.8331
0.8292
0.8497
0.8333
0.8801
Cartpole
Mass of Cart
0,9466
0.7153
0.8961
0.9044
0.8299
0.9356
0.9838
Cartpole
Length of Pole
0.9110
0.5441
0,9497
0.9484
0.9424
0.9488
0.9875
Cartpole
Mass of Pole
0.9560
0.6073
0.9870
0.9927
0.9916
0.9813
1,0000
BipedalWalker
Gravity
0.9281
0.7898
0.9654
0. 0.9666
0.9656
0,9655
0.9674
BipedalWalker
Friction
0.9317
0,9051
0,9739
0.9738
0,9738
0,9725
Waleer
Scale
0.8694
0.7452
0,8910
0.8975
0.8990
0,8962
0.9107
HalfCheetah
Gravity
0.6679
0.6292
0,9086
0,9000
0.9089
0.9214
0.9544
HalfCheetah
Friction
0.6693
0.7242
0.9314
0.9457
0.9184
0.9225
0. 0.9663
HalfCheetah
Stiffness
0.6561
0.7007
0.9191
0.9097
0.9295
0,9205
0.9674
Average
0.8093
0.6957
0.8901
0.8923
0.8896
0.8950
0.9255
[ligherr the betler. Bold values represent the highest valuels) within the statistically signilicant range each CMDP, excluding oracle. Detailed results with variance for each method are provided in AppendixAA.A
heuristic multi-policy baselines, indicating the value of adaptively selecting source tasks based on feedback. The multi-policy baselines, such random, equidistant, and greedy strategy, also generally outperform Independent and Multi-task indicating the general value of multi-policy training for solving CMDPs. More results are provided in Appendix A.4
5.3
Continuous control benchmark experiments
Cartpole (Mass of Cart)
Pendulum (Time Step)
Halfcheetah (Friction)
1.0
0.9
1.0
Normalized performance
0.8
0.8
0.9
0.6
0.8
5x
11x
0.4
0.6
0.5
10a
10a
10a
10a
10a
10a
Number samples (Log scale)
Number samples (Log
Number samples (Log
Random
Independent
Multitask
Greedy
Equidistant
MBTL
Oracle
Figure 6: Continuous control CMDP results. Method comparison of normalized performance over N tasks. The black dotted line indicates the first training step within MBTL that exceeds both independent and multi-task baselines, with up to 43x improved sample efficiency.

---

Step
Step 3
0..8
0.175
1.0
Nomalized Performance
0.06
Nomallze Performane
0.8
Nomaloed on
0.8
-0.125
Acquzation funzt on
0.0
0.05
Apo funett
0.6
0.100
0.04
3.04
0.4
0.075
0.003
0.2
3.02
0.
0.050
0.2
Source locks
GP pradicton
0.02
0.0
0.000
Nent source Lask
Acqusation function
0.01
20
40
00
90
100
40
80
Source
Source
Figure 7: The GP sequentially updates estimates of the performance function (blue) based on previously trained models. Then, MBTL selects the next source task that maximizes the acquisition function (red). (CMDP: Pendulum (Time step)).
DON
A2C
1.0
Normaized performance
0.9
0.8
0.8
0.7
0.6
0.4
0.4
10
10a
10
10
Number of samples trained (Log scale)
Number of samples trained (Log scale)
Number samples trained (Log scale)
Random
Independent
Multitask
Greedy
Equidistant
MBTL
Seq. Oracle
Figure 8: Sensitivity analysis the DRL algorithm underlying MBTL (DQN, PPO, and A2C), tested on Cartpole with varying length of pole. MBTL remains effective.
To probe the generality of MBTL, we utilize context-extender versions of standard RL environments from CARL benchmark library [5] to evaluate our methods under varied contexts. For the Cartpole domain, we considered CMDPs with varying cart mass, pole length, and pole mass. Pendulum, vary the timestep duration, pendulum length, and pendulum mass. The BipedalWalker tested under varying friction, gravity, and scale. HalfCheetah domain, we manipulated friction, gravity, and stiffness parameters. These variations critically influence the dynamics and physics of the environments. The range of context variations was selected by scaling the default values specified in CARL from 0.1 to 10 times (N 100), enabling comprehensive analysis of transfer learning under drastically different conditions. We provide more experimental details in Appendix A..4
Results. The results summarized in Table 2 demonstrate sample efficiency and competitive perfor- mance of multi-policy training including MBTL across diverse control domains, often closely trailing the Oracle only with small number of trained policies. Figure 6 shows that with the exception of a few context variations, MBTL generally shows superior performance. Specifically, Figure illustrates the detailed process of how MBTL utilizes GP for performance estimation and chooses the next source task that maximizes the acquisition function that evaluates the expected improvement of generalized performance. MBTL achieves comparable performance multi-task independent baselines with 5-43x fewer samples, highlighting its effectiveness reducing training requirements.
5.3.1 Sensitivity analysis
Normalized performance
1.0
UCB (8 cc/k)
UCB (8 11
UCB (8 cyloogk 1))
DRL algorithms. Figure g shows that MBTL remains effective with different
0.8
derlying DRL Sigorithms-DQN PPO, and
0.6
Advantage Actor- Critic (A2C) 30 used
for single-task training.
0.4
Acquisition functions. Figure assesses the role acquisition functions Bayesian opti- mization. While expected improvement (EI) focuses on promising marginal gains beyond the current best, UCB utilizes both mean and
0.2
Cartpole (Mass of Pole)
Traffic signal AA-Ramp-Vel Eco-driving control (Green time)
control (Road Length)
Figure 9: Sensitivity analysis on acquisition functions.

---

variance for balancing exploration and exploitation. Overall, we find that MBTL is not particularly sensitive to the choice of optimism representation the acquisition function, which indicates that MBTT. has a weak dependence on hyperparameters.
Related work
Contextual Reinforcement Learning. Robustness and generalization challenges in DRL are gen- erally addressed by few common techniques in the literature. The broader umbrella of such methods falls under CRL, which utilizes side information about the problem variations to improve the generalization and robustness. In particular, CRL formalizes generalization in DRL using CMDPs [14 32 5]. which incorporate context- -deeeneet dynamics, rewards, and initial state distributions into the formalism of MDPs. The contexts of CMDPs are not always visible during training [23] When they are visible, they can be directly used as side information by conditioning the policy on them [39]. In this paper, focus scenario where the learner can choose which coDtext-MDP to train on. This contrasts with other CRL works that assume context-MDPs arrive from a fixed distribution.
Multi-task training. Multi-task methods can help address CRL by exploiting shared structure across tasks. Prior work has leveraged techniques such policy sketches for task decomposition [2] and distilled policies that capture common behaviors [46]. However, a key limitation arises when the context is unobserved, effectively transforming the CMDP into partially observable setting 2 [ which complicates multi-task training. Another challenge is negative transfer, wherein training on tasks that are too dissimilar leads to instability outright failure 22
44 Although more recent multi-task approaches such as MOORE [6] and PaCo 43 have shown promise, they often focus on discrete task sets and are thus less suited to CRL, where tasks span broad continuum of contexts. In this work, we include multi-task learning as baseline to benchmark our methods.
Zero-shot transfer and policy reuse. Zero-shot transfer-where models trained for one environment directly perform in new, unseen settings without additional training [23]-is an important strategy in CRL settings. For solving CMDP problems, prior works attempted to utilize zero-shot transfer to solve CMDP problems by approximation on RL algorithm and hypernetworks that maps from parameterized CMDP to a family of near-optimal solutions [35]. Sinapov al. [38] use meta-data to learn inter-task transferability to learn the expected benefit of transfer given a source-target task pair. Bao et al. 3 propose a metric for evaluating transferability based representations across tasks. Taken together, these approaches highlight the importance of policy reuse, where efficiently selecting or adjusting a pre-trained policy accelerates learning and improves robustness in new contexts.
iiformationnthhorette feature
Source task selection. In the context of transfer learning, selecting appropriate source tasks is crucial. Li and Zhang [25) proposes an optimal online method for dynamically selecting the most relevant single source policy in reinforcement learning. Beyond RL, Meiscles and Rokach (28) emphasizes structural alignment in time-series source models to prevent performance degradation, while Poth al. 33 finds that selecting aligned intermediate tasks in natural language processing boosts transfer effectiveness. Building upon these insights, we formulate the source task selection problem for CRL, enabling zero-shot transfer by estimating training performance online and leveraging structural generalization across context variations.
Conclusion
This study introduces method called Model-based Transfer Learning (MBTL), which layers on top of existing RL methods to effectively solve CMDPs. Rather than independent or multi-task training. which trains N or 1 models, respectively, MBTL intelligently selects an intermediate number of models to train. MBTL has two key components: an explicit model of the generalization gap and a Gaussian process component to estimate training performance. MBTL achieves up to 43x improved sample efficiency on standard and real-world benchmarks. Furthermore, MBTL achieves sublinear regret the number of training tasks. limitation that MBTL is designed for a single-dimensionall context variation with a reliance on the explicit similarity of context variables. Promising directions of future work include studying hig high-dimensional context spaces and formalizing task similarity, as well as the development of new real-world CMDP benchmarks.
10

---

Acknowledgments and Disclosure of Funding
The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper. This work was supported by the National Science Foundation (NSF) CAREER award (#2239566), the Kwanjeong Educational Foundation Ph.D. scholarship program, and an Amazon Robotics Ph.D. Fellowship. The authors would like to thank the anonymous reviewers for their valuable feedback.
References
[1] Abubakr O Al-Abbasi, Amob Ghosh, and Vaneet Aggarwal. Deeppool: Distributed model-free algorithm for ride-sharing using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 20(12):4714 4727, 2019.
[2] Jacob Andreas, Dan Klein, and Sergey Levine. Modular Multitask Reinforeement Learning with Policy Sketches. In Proceedings of the 34th International Conference on Machine Learning, pages 166-175 PMLR, July 2017. ISSN: (598.
[3] Yajie Bao, Yang Li, Shao- Huang. Lin Zhang, Lizhong Zheng. Amir Zamir, and Leonidas Guibas. An Information Theoretic Approach to Transferability Task Transfer Learning. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2309-2313. Taipei, Taiwan, September 2019. IEEE. ISBN 978- 1- 5386-6249-6 doi: 10. 11110//CII.2211.88800
[4] Marc G Bellemare. Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-82, 2020.
[5] Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian Döhler, André Biedenkapp. Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize Me -hhe Case for Context in Reinforcement Learning. Transactions on Machine Learning Research, June 2023.
[6] Eric Brochu, Vlad M. Cora, and Nando de Freitas. A Tutorial Bayesian Optimization of Expensive Cost Functions, with Application Active User Modeling and Hierarchical Reinforcement Learning, December 2010.
[7] Jung-Hoon Cho, Sirui Li, Jeongyun Kim, and Cathy Wu. Temporal transfer learning for traffic optimization with coarse-grained advisory autonomy. arXiv preprint arXiv:2312.09436. 2023.
[8] Tianshu Chu, Jie Wang. Lara Codeca, and Zhaojian Li. Multi-Agent Deep Reinforcement Learning for Large-Scale Traffic Signal Control. IEEF Transactions on Intelligent Transportation Systems, 21))): 1086-1095, March 2020. ISSN 1524-9050, 1558-0016, doi: 10. 10.1109/TTTS.20 2019999...2229901111 79119999.
[9] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference machine learning, pages 2048-2056. PMLR, 2020.
[10] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Fwalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897)4144449) 2022.
[11] US EPA. Sources of Greenhouse Gas Emissions, 2023. URL sources- greenhouse- gas- emissions
[12] Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-533 2022.
[13] Qiangqiang Guo, Ohay Angah, Zhijun Liu, and Xuegang (Jeff) Ban. Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors. Trans- portation Research Part C: Emerging Technologies. 124:102980, March 2021. ISSN 0968090X. doi: 10. (10.://....2221.109999909
[14] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint 100.:0500.002259 2015.

---

[15] Aamir Hasan, Neeloy Chakraborty, Haonan Chen, Jung-Hoow Cho. Cathy Wu, and Katherine Driggs- Campbell. Cooperative advisory residual policies for congestion mitigation. Journal on Autonomous Transportation Systems, 2024.
[16] Ahmed Hendawy, Jan Peters, and Carlo D'Framo. Multi-task reinforcement learning with mixture of orthogonal experts. arXiv preprint arXiv:2311.11385, 2023.
[17] Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel. Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: Improving Zero-Shot Transfer in Rein- forcement Learning. Proceedings of the 34th International Conference on Machine Learning, pages 1480-1490. PMIR, July 2017. ISSN: 2640-3498.
[18] Vindula Jayawardana and Cathy Wu. Learning eco-driving strategies signalized intersections. In 2022 European Control Conference (ECC), pages 383 390. IEEE, 2022.
[19] Vindula Jayawardana, Catherine Tang, Sirui Li, Dajiang Suo, and Cathy Wu. The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning. Advances in Neural Information Processing Systems, volume 35, pages 23881 -33893. Curran Associates, Inc., 2022.
[20] Vindula Jayawardana, Baptiste Freydt, Ao Qu, Cameron Hickert, Edgar Sanchez, Catherine Tang, Mark Taylor, Blaine Leonard, and Cathy Wu. Mitigating metropolitan carbon emissions with dynamic eco-driving at scale, arviv preprint arXiv:2408.05609 2024.
[21] Vindula Jayawardana, Baptiste Freydt, Ao Qu, Cameron Hickert, Zhongxia Yan, and Cathy Wu. Inter- sectionzoo: Eco-driving for benchmarking multi-agent contextual reinforcement learning. arXiv preprint arXiv:2410.15221 2024.
[22] Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom share in multi-task feature learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-II), pages 521-528, 2011.
[23] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76:201-264, 2023.
[24] Li Li. Yisheng Lv, and Yue Wang. Traffic signal timing via deep reinforcement learning. JEEE/CAA Journal of Automatica Sinica, 3()):477-254, July 2016. ISSN 2329-9266. 2329-9274. doi: 10.1109/AS. 2016.7508798.
[25] Siyuan Li and Chongjie Zhang. An Optimal Online Method of Selecting Source Policies for Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018 ISSN 2374- 3468, 2159-5399. doi: 10. http//.aii.x2...11777 URL URI httph::tt////jopt aaail www.aaaaaaaaaaaaa.aa.....a.. indii.nddeppppp php/AAAT/ /
Stticle/view/1n 11718
[26] Pablo Alvarez Lopez. Michael Behrisch, Laura Bieker-Waas Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic simulation using sumo. In The 2 IEEE International Conference Intelligent Transportation Systems. IEFF, 2018.
[27] DanielJ Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern al. Faster sorting algorithms discovered using deep reinforcement learning. Nature, 618(796))2577 263, 2023.
[28] Amiel Meiseles and Lior Rokach. Source model selection for deep learning in the time series domain. IEEE Access, 8:6190-6200, 2020.
[29] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-53) February 2015. ISSN 0028-0836, 1476-4687. doi: 10. 1038//aauree12236
[30] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning, pages 1928-1937. PMLR, June 2016. ISSN: 1938-7228.
[31] Jonas Mockus. Bayesian Approach to Global Optimization: Theory and Applications, volume 37 of Mathematics and Its Applications. Springer Netherlands, Dordrecht, 1989. ISBN 978-94-011 040--6888 978 44-000909009.0 doi: 10. 111111000799---001
94-0009009090000000
12

---

[32] Aditya Modi. Nan Jiang, Satinder Singh, and Ambuj Tewari. Markov Decision Processes with Continuous Side Information. In Proceedings Algorithmic Learning Theory, pages 597-618. PMLR, April 2018. ISSN: 2640-3498.
[33] Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Tryna Gurevych. What to pre-train on? efficient intermediate task selection. arXiv preprint 1rXiv:2104.08247 2021.
[34] Antonin Raftin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 8, 2021. httt://///mmlr httpa//rrrrrpppp///22222/22/////
1364. html
[35] Sahand Rezaei-Shoshtari, Charlotte Morissette, Francois R. Hogan, Gregory Dudek, and David Meger. Hypernetwork for Zero-Shot Transfer in Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8):9579 9587, June 2023. ISSN 2374-3468, 2159-5399. doi: 10. 10.1609/aaai.
v37i8.26146.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal Policy Optimization Algorithms, August 2017.
[37] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Gracpel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484
January 2016. ISSN 1476-4687. doi: 10. 10.1038/naatte 1038//ature 16961.
[38] Jivko Sinapov, Sanmit Narvekar, Matteo Leonetti, and Peter Stone Learning Inter-Task Transferability in the Absence of Target Task Samples. In Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2015), Istanbul, Turkey, May 2015.
[39] Shagun Sodhani, Amy Zhang, and Joelle Pineau, Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pages 9767-9779 PMLR, 2021.
[40] Mayuri Sridhar and Cathy Wu. Piecewise Constant Policies for Human Comba---o Congestion Mitigation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2499-2500 Indianapolis, IN, USA, September 2021. IEEE ISBN 978-1-72819-111 doi: 10. 10.11//TSC44998820211 9564789.
[41] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information Ciformation-Thhorttiic Regret Bounds for Gaussian Process Optimization in the Bandit Setting. IEEE Transactions Information Theory, 58(5):2250-322 May 2012. ISSN 0018-9448, 1557-9654, doi: 10.110/912220111221 2122233.....
[42] Trevor Standley. Amir Zamir, Dawn Chen. Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9120-9133 PMLR, 13-18 Jul 2020. httphtttt///pceeeeeeee
mlr. pxess/vv199 st119//ttndley20es html
[43] Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter-compoottionna multi-task reinforcement learning. Advances in Neural Information Processing Systems, 35:21495- 2022.
[44] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33:8728-8740, 2020.
[45] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-69 0020022-2222222
[46] Yee Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[47] Marius Wegener, Lucas Koch, Markus Fisenbarth, and Jakob Andert. Automated eco-driving in urban scenarios using deep reinforcement learning. Transportation Research Part C: Emerging Technologies, 126:102967, May 2021. ISSN 0968090X. doi: 10.0016f..c.2021110399
[48] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge. MA. 2006.
13

---

[49] Zhongxia Yan, Abdul Rahman Kreidieh, Eugene Vinitsky, Alexandre M. Bayen, and Cathy Wu. Unified Automatic Control of Vehicular Systems With Reinforcement Learning. IEEE Transactions on Automation Science and Engineering, pages 1-16, 2022. ISSN 1545-5955, 1558-3783, doi: 10.1 109S..222.... 3168621.
14

---

A
Appendix
Contents
A.1
Notation
16
A.2
Model-Based Transter Learning (MBTL) A Algorithm
17
A.3
Theoretical analysis
17
A.3.I
Proof of Theorem
17
A.3.2
Proof of Corollary 2.1
18
A.3.3
Proof of Corollary
18
A.4
Experiment details
19
A.4.1
Details about Gaussian process (GP) Regresstion
19
A.4.2
Accuracy of generalization gap assumption
19
A.4.3
Results table with standard deviation
20
A.4.4
Detailed sample complexity comparison results
21
A.4.5
Details about traffic signal control benchmark
................................
21
A.4.6
Details about eco-driving control benchmark
................................
24
A.4.7
Details about advisory autonomy benchmark
25
A.4.8
Details of control benchmarks
26
A.4.9
Details about Cartpole benchmark
................................
27
A.4.10 10 Details about Pendulum benchmark
28
A.4.11
Details about Bipedal Walker benchmark
30
A.4.12
Details about HaltCheetah benchmark
31
A.4.13 Implementation of the recent multi-tas task baselines
33
A.5
Potential impacts
34
15

---

A.1
Notation
Table 3 describes the notation used this paper.
Table 3: Notation used in the problem formulation Description
Symbol
Source task (æ E X)
Target task (a' E X)
Wx
Trained policy from source task (x E X)
Selected source task at transfer step k (k = 1, K)
M J (Tayz) [TT, x') (J(
Contextual MDP parameterized by
Performance of task M2
Generalization performance (source: æ (or x), target: x')
Generalization gap (source: z, target: x')
V(o:: (Tx)
Expected generalization performance of source model æ evaluated on all x' X
Figure 10 helps understand the discrepancy between the observed generalized performance and the predicted one. Figure [T illustraae how to calculate the marginal improvement of expected generalized performance (V(z; 114--1 (z1
[TT ,011)
j (Taz x')
J xT)
Target task x'
Figure 10: Illustration of the discrepancy between observed (J) and predicted (f) generalized performance after training source task 2j and attempting zero-shot transfer to z'.
16

---

U
J (01)
max (J(Ta) 2t) (TTI)
(x; - ((1)
x(
J xT),
Target task x²
Figure 11: Step for choosing that maximizes the estimated marginal improvement (V (æ;T)) V(xi)). ((z ii) corresponds to the red area under the red line and V(x1) the area under J(1,z')
A.2 Model-Based Transfer Learning (MBTL) Algorithm
Model-based Transfer Learning (MBTL)
1: Input: CMDPs Ma. Task (context) set X, Training budget K
2: Initialize J, V 0 Va E X, π (}. =
3: while & < do
4: % Estimate training performance
5: 1e.0 +- --((E(((T.)))) k(r,, ž)))
6: % Calculate marginal generalized performance and acquisition function
7: Calculate a(x; x1:k-1) with Eq.
8:
% Select the next training task
9:
arg max, a(2; 11)--1)
Tk Train(Max)
11:
π + T (
k ++++1
13: end while
Zero-shot transfer and calculate generalization performance (I1,..., 2k)
15: Output: Set of policies and generalization performance V
A.3 Theoretical analysis
A.3.1 Proof of Theorem 2
Theorem 2. For given 8' E (0, and scaling factor B* 2log([XX|72k]/ /65). the cumulative
regrel Rg bounded by / CI ΣK
[Xx1] [X
with probubility least 8'.
Proof. The following Lemma and is basically considering the cardinality of restricted search space of X5 instead of X upon the lemmas in literature [1].
Lemma 3. Fort 2 1, ff) - N (x)1 < 3//2 (x)) Vx Xt. then the regret r is bounded by 2/ 1/2 (x)
17

---

Lemma 4. Setting 8 E (0,1), Px = 2 X
8 x|=222/600 and C1 = log(1+o 3)
2 80
we have
Pr
E
Th
X
S CI8KTK
-
Using Lemma] Lemma 5.3 in 41 Lemma cumulative regret
4 and Cauchy-Schward incquality, we can bound the
K
K
X
2
K
[M
2
K
[x
RR
Th
S
Σ
CI
(5)
[
[
[X]
k=1
A=1
k=1
k=1
A.3.2 Proof of Corollary
2.1
Corollary 2.1. Consider Xx 1
vi |1. The regret bound would be Rx I VC1 log with
probability of at least 1 - 8'.
Proof. Recall that Σ I log
Calculating the sum squares for the reduced segments, we have:
K
K
X
X S | log K
(6)
The cumulative regret can be bounded as below:
X X
Rk
Σ rk
C18KTK
S v CiBKi log K.
(7)
1=1
k=
A.3.3 Proof of Corollary 2.2
Corollary 2.2. The regret bound for the [ S -looo kl ||| with probability of at least -'.
would be R S VCBK7K02116
Proof. Calculating the sum of squares the reduced segments, we have:
K
K
$ n²
X
Σ 2
log, h:
|
<
x
|x|
(8)
b
6
k=1
The cumulative regret can be bounded as below:
K
X& X
C1
RK
Σ
S
CιßKTK
<
(9)
k=1
18

---

A.4
Experiment details
A.4.1
Details about Gaussian process (GP) Regresstion
We use the GaussiaarrooeesRRggessss implementation from scikit-leard which follows Algorithm 2.1 of 48 Specifically, we construct kernel by multiplying constant kernel
( 1.0, E (10-$,10%
by a radial basis function (RBF) kernel
X x'13
KRBF (x, (x,x;() = exp
202
with an initial length scale { = 1.0 (constrained to lie in the range [10-2, 10 To determine the hyperparameters. begin by generating synthetic data that aligns with our modeling assumptions, including constant training performance and linear generalization gap, while introducing noise to degrade generalization performance by up to 10%, sampled from uniform distribution. We vary the GP hyperparameters, including noise standard deviation over the set (0.001, 0.01, 0.1, the number of restarts for the optimizer over {5, 6..., 15), and explore several kernel configurations on the synthetic data. We then select the hyperparameter configuration that maximizes the average predictive performance. Specifically, we choose a noise standard deviation 15 random restarts of the hyperparameter optimizer reduce the risk of convergence to poor local minima. We use the same GP hyperparameter configuration across all experiments and benchmarks.
= 0.001 and perform
A.4.2 Accuracy of generalization gap assumption
report the Pearson correlation between the observed generalization gap and the estimated gap under our linear assumption (Assumption[]] [T). Each histogram shows how strongly the two measures align across various tasks in standard control (blue) and traffic (red) benchmarks. Many tasks cluster around moderate positive correlations (0.3-0.5), suggesting that a linear function of context similarity reasonably capture the gap in most scenarios. However, certain tasks-suse as Eco-driving- Fco-driving-xxhin exhibit higher correlations (above 0.6), whereas others- such as HalfCheetah- closer indicating that the assumption holds more effectively in some domains than in others.
In Figure 12
Control Benchmarks
Traffic Benchmarks
Traffic Signal
Conttoll
Advisory Autonomy
Frequency
5-
Periuuuum
4
Bipeda Walker
Cartpole
Eco-driving
3-
HallCheetahi
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Pearson Correlation
Figure 12: Accuracy of linear generalization gap assumption. Pearson correlation analysis on the observed generalization gap and the estimated gap under our linear assumption.
19

---

A.4.3 Results of table with standard deviation
Table 4: Comparative performance of different methods on context-variant traffic and benchmark
CMDPs (K = 15)
Benchmark (CMDP)
Baselines
Multi-policy Baselines
MBTL
Oracle
Domain
Context Variation
Independent Multi-task Random
Equidistant
Greedy
Ours
Sequential
Number of Trained Models
k
K
k
0.9409
0.8242
0.9366
0.9337
0.9349
0.9364
0.9409
Traffic Signall
Road Length
(0.0002)
(0.0538)
(0.0009)
(0.0044)
(0.0021)
(0.0059)
(0.0006)
0.8646
0.8319
0.8699
0.8712
0.8682
0.8432
0.8768
Traffic Signal
Inflow
(0.0009)
(0.0040)
(0.0011)
(0.0044)
(0.0008)
(0.0372)
(0.0012)
0.8857
0.6083
0.8872
0.8872
0.8874
0.8867
0.8876
Traffic Signal
Speed Limit
(0.0005)
(0.0403)
(0.0002)
(0.0004)
(0.0004)
(0.0002)
(0.0002)
0,5260
0.1945
0.6212
0.6278
0,5992
0.6148
0.6660
Eco-Driving
Penetration Rate
(0.0087)
(0.0057)
(0.0041)
(0.0332)
(0.0007)
(0.0145)
(0.0049)
0,4061
0.2229
0.5077
0,5513
0.5299
0.5172
0.5528
Eco-Driving
Inflow
(0.0094)
(0.0010)
(0.0114)
(0.0113)
(0.0456)
(0.0415)
(0.0096)
0.3850
0,4228
0.4724
0.4697
0,4678
0.4985
0.5027
Eco-Driving
Green Phase
(0.0063)
(0.0184)
(0.0069)
(0.0318)
(0.0147)
(0.0096)
(0.0058)
0,8362
0,9209
0.9306
0,9225
0.9136
0,9382
0,9552
AA-Ring-Ace
Hold Duration
(0.0048)
(0.0235)
(0.0087)
(0.0252)
(0.0193)
(0.0216)
(0.112))
0.9589
0,9720
0.9819
0.9819
0,9819
0.9818
0.9822
AA-Ring-Vel
Hold Duration
(0.0096)
(0.0035)
(0.0001)
(0,0003)
(0.0001)
(0.0002)
(0,0000)
0.4276
0,5158
0.6750
0.6821
0.6882
0.6964
0.7111
AA-Ramp-Ace
Hold Duration
(0.0066)
(0.045)
(0.0275)
(0.0809)
(0.0715)
(0.0522)
(0.0479)
0.5473
0.5034
0.7170
0.7386
0.6918
0.7273
0.7686
AA-Ramp-Vel
Hold Duration
(0.0222)
(0.014)
(0.0384)
(0.0410)
(0.084)
(0.0636)
(0.0581)
Pendulum
Length
(0.0034)
0.7383
(0.0008)
0.6830
(0.0072)
0.7607
(0.0074)
0.7601
(0.0041)
0.7774
(0.0215)
0.7755
(0.0123)
0.7969
Pendulum
Mass
0.6237
0.5793
0.6647
0.6794
0.6887
0,6667
0.7132
(0.0023)
(0.0041)
(0.0065)
(0.0376)
(0.0116)
(0.013)
(0.0129)
0.8135
0.7247
0.8331
0.8292
0.8497
0,8333
0.8801
Pendulum
Timestep
(0.0103)
(0.0488)
(0.0084)
(0.0189)
(0.0322)
(0.0130)
(0.0198)
0.9466
0.7153
0.8961
0.9044
0.8299
0,9356
0,9838
Cartpole
Mass of Cart
(0.0065)
(0,2195)
(0.0214)
(0.055)
(0.0392)
(0.0239)
(0.0131)
0.9110
0,5441
0.9497
0,9484
0.9424
0.9488
0.9875
Caripole
Length of Pole
(0.0065)
(0.1614)
(0.0044)
(0.042)
(0.031)
(0.0145)
(0.0047)
0,9560
0.6073
0.9870
0,9927
0,9916
0,9813
1.0000
Cartpole
Mass of Pole
(0.0128)
(0.0948)
(0,0050)
(0.0014)
(0.0030)
(0.0103)
(0.0000)
0.9281
0.7898
0.9654
0.9666
0.9656
0.9655
0.9674
BipedalWalker
Gravity
(0.0034)
(0.0928)
(0.0004)
(0.0022)
(0.0021)
(0.001)
(0.0014)
0.9317
0.9051
0.9739
0.9738
0.9738
0.9725
0.9778
BipedalWalker
Friction
(0.0074)
(0.0734)
(0.0003)
(0.0022)
(0.0013)
(0.0038)
(0.0013)
0.8694
0.7452
0.8910
0.8975
0.8990
0.8962
0.9107
WalkdalWalker
Scale
(0.0087)
(0.0938)
(0.0079)
(0.0118)
(0.0135)
(0.0123)
(0.004)
HalfCheetah
Gruvity
0.6679
0.6292
0.9086
0.9000
0,9089
0.9214
0.9544
(0.0162)
(0.0258)
(0.0078)
(0.0600)
(0.0235)
(0.0232)
(0.0221)
0.6693
0.7242
0.9314
0.9457
0.9184
0,9225
0.9663
HalfChectah
Priction
(0.0203)
(0.1056)
(0:0175)
(0.0295)
(0.0184)
(0.0392)
(0.0276)
0.6561
0,7007
0.9191
0.9097
0,9295
0.9205
0.9674
HallCheetah
Stiffness
(0.0101)
(0.1126)
(0.01)
(0.027)
(0.0169)
(0.0161)
(0.0286)
0.7775
0.6337
0.8309
0.8352
0.829
0.8355
0.8833
Average
(0.197)
(0.2395)
(0.0089)
(0.024)
(0.0198)
(0.0199)
(0.1467)
Note: Bold values represent the highest valucis) within the staistically significant range for each task, excluding the oracle. Standard deviation across multiple in the parenthesis.
LAA: Advisury autonomy Lasks, Ring: Single lane ring, Ramp: Highway rump, Ace: Acceleration guidance, Vel: Speed guidance.
20

---

A.4.4 Detailed sample complexity comparison results
T a comparison of sample complexity required for MBTL to perform as good as the best generalization performance of baselines (independent training and multi-task training) across various tasks in the CMDP. Each row lists different domain, the specific context variation applied (e.g., changes in physical properties or environmental parameters), and two key values: k* and N, where k* represents the number of models required by MBTL to reach a performance level comparable to the baseline. This value is shown as range (e.g., [3, 5, 3]) indicating results from three random seeds. N represents the total number of contexts. The value N helps represent the sample efficiency of MBTL.
k
Table 5: Sample complexity comparison to baseline performance on CMDP tasks
Task
Context Variation
k* N
Pendulum
Length
[4, 5, 5]
100
Pendulum
Mass
[3, 3,
100
Pendulum
Timestep
[8, 14]
100
Cartpole
Mass of Cart
[15, 25, 18]
100
Cartpole
Length of Pole
[10, 6, 14]
100
Cartpole
Mass of Pole
[9, 9, 12]
100
BipedalWalker
Gravity
[2, 3, 4]
100
BipedalWalker
Friction
[3, 2, 2]
100
BipedalWalker Walker
Scale
[7, 6, 1]
100
HalfCheetah
Gravity
[10,2, 5]
100
HalfCheetah
Friction
[1,2, 13]
100
HalfCheetah
Stiffness
[1,
100
AA- AA-Ring-Ace
Hold Duration
[5, 7,33]
40
AA-Ring- Vel
Hold Duration
[2, 1]
40
AA-Ramp-
Hold Duration
[1 2]
40
AA-Ramp- Vel
Hold Duration
[6, 4,
40
Traffic Signal
Road Length
[50, 50, 11]
50
Traffic Signal
Inflow
[2, 44, 41]
50
Traffic Signal
Speed Limit
[10, 4, 8]
50
Eco- Bro-Diiv
Penetration Rate
[3, 5, 1]
50
Eco-Driving
Inflow
[2, 1,
50
Eco-Driving
Green Phase
[3, 3,
50
A.4.5 Details about traffic signal control benchmark
Most traffic lights operate on fixed schedules, but adaptive traffic signal control using DRL can optimize the traffic flow using real-time information on the traffic [8] 24], though challenges persist
in generalizing across various intersection configurations [9]].
Figure 13) showcases the layout of traffic networks used in a traffic signal control task with several lanes and a signalized intersection in the middle. The state space represents the presence of vehicles in discretized lane cells along the incoming roads. Actions determine which lane gets the green phase of the traffic signal, and rewards are based on changes in cumulative stopped time, the period when speed is zero. The global objective is to minimize the average waiting times at the intersection. Different configurations of intersections (e.g., road length, inflow, speed limits) are modeled to represent varying real-world conditions. To represent various real-world conditions, we vary factors such as road length, inflow rate, and speed limits from to times; by default, the road length 750 meters, the flow rate is 1000 vehicles per hour, and the speed limit is 13,89 m/s.
21

---

Inflow
Length
DIO Di
Speed Limit
speen 35
Guided vehicle
Default vehicle
Figure 13: Illustration of the traffic networks in traffic signal control task.
Training configuration We used the microscopic traffic simulation called Simulation of Urban MObility (SUMO) [26] v.1.16.0 and PPO for RL algorithm [36]. We utilized the default implementa- tion of the PPO algorithm with default hyperparameters provided by the
library
[34]. All experiments are done on a distributed computing cluster equipped with 48 Intel Xcon Platinum 8260 CPUs.
License
Traffic signal control benchmark falls under MIT License.
1.0
1.0
Nornalized performance
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
Independent
Multitask
1000 2000
5000
1500 2250 Length of Lane
Flow Rate
Speed Limit
Figure 14: Normalized performance of three based methods- Transfer (gray), indepen- dent training (orange), and multi-task training (green)-unde, different traffic-signal benchmarks: flow rate (left), speed limit (middle), and lane length (right). While independent and multi-task training approaches exhibit higher variance and reduced asymptotic performance, Oracle Transfer benefits from zero-shot transfer with full information, demonstrating more stable and generally higher performance.
Potential of multi-policy training and zero-shot transfer Figure shows how each approach adapts to variations in flow rate, speed limit, and lane length for a traffic signal control benchmark. The y-axis shows normalized performance, with higher values indicating better control policies. Oracle Transfer consistently achieves superior performance across these different settings. owing to its ability to leverage full task information in a zero-shot transfer manner. By contrast, independent and multi-task training exhibit more pronounced performance drops and greater instability when faced with shifts in problem parameters, underscoring the challenges of generalizing policies in traditional DRL approaches.
22

---

Transferability heatmap Figure 15 presents heatmaps of transferability for different traffic signal control tasks, each varying a specific aspect: inflow, speed limit, and road length. The heatmaps display the effectiveness of strategy transfer from each source task (vertical axis) to each target task (horizontal axis). In terms of inflow variation, transferability drops when transferring from tasks with lower vehicle inflow to those with higher inflow. In speed limit variation, the transferability shows uniform effectiveness, suggesting less sensitivity to these changes. In road length variation, distinct blocks of high transferability indicate that different road lengths may require significantly tailored strategies.
105
64
(a) Inflow variation
(b) Speed limit variation
(c) Road length variation
Figure 15: Examples of transferability heatmap for traffic signal control.
Results Figure illlstttes the normalized generalized performance across various traffic control tasks: inflow, speed limit, and road length. The plots display how different strategies adapt with increasing transfer steps:
Inflow:
Performance improves as the number of transfer steps increases, with MBTL strategy consistently achieving the highest scores, demonstrating their effectiveness in adapting to changes inflow conditions.
Speed Limit:
Here, performance levels are relatively stable across all strategies except for the multi-tase task training.
Road Length: There is general upward trend performance for all strategies, particularly for MBTL, indicating robustness in adapting to different road lengths.
This data suggests that MBTL and Oracle are particularly effective across varying conditions, maintaining higher levels of performance adaptability.
Inflow
Speed Limit
Road Length
0.90
0.950
Normalized performance
0.85
0.85
0.925
0.80
0.75
0.875
0.70
0.850
0.75
0.825
0.60
0.800
10a
10a
10
10a
10a
10a
Number of samples trained (Log scale)
Number of samples trained (Log scale)
Number samples trained [Log scale
Random
Independent
Multitask
Greedy
Equidistant
Seq. Oracie
Figure 16: Comparison of normalized generalized performance of all target tasks: Traffic signal control.
23

---

A.4.6 Details about eco-driving control benchmark
Given the significant portion of greenhouse gas emissions in the United States coming from the transportation sector [[], eco-driving behaviors are critical for climate change mitigation. Deep reinforcement beaning-buseed based eco-driving strategies have been developed [13 47 18, 20] but still have
some issues of difficulties in generalization. We also extend to various intersection configurations with different traffic inflow rates, penetration rates of eco-driving systems, and durations of green phases at static traffic signals to optimize vehicle behaviors for reduced emissions.
Figure 17 illustrates the traffic road network used in the eco-driving control task. The road network is depicted as traffic flowing vertically and horizontally, crossing the static phase traffic signal. There are both guided and default vehicles in the system. The state space includes the speed and position of the ego vehicle, the leading vehicle, and the following vehicles, supplemented by the current traffic signal phase and relevant context features, including lane length and green phase durations. The action space specifically focuses on the ego vehicle's acceleration control. The reward mechanism designed to optimize the driving strategy by balancing the average speed of the vehicles against penalties for emissions, thereby promoting eco-friende coo-friendly driving behaviors within the traffic system.
Inflow
Green phase
Penetration rate
Guided vehicle
Default
Figure 17: Illustration of the traffic networks in eco-driving control task.
Training configuration We also used the microscopic traffic simulation called Simulation Urban MObility (SUMO) (26) 1.1. and PPO for RL algorithm (36] For detailed experimental details
and RL hyperparameter configurations, please refer to 20 21.
License Eco-driving benchmark falls under MIT License 21
Potential of multi-policy training and zero-shot transfer Figure shows how each RL training paradigm adapts to variations in green phase time, penetration rate, and inflow rate in the eco- driving control benchmark. Oracle Transfer remains the strongest method across all configurations, benefiting from zero-shot transfer. independent training shows unstable performance across different task variations, performance, while multi-task training lags behind. Overall, the trends highlight the advantage of leveraging zero-shot transfer in traffic CMDPs.
Transferability heatmap Figure 18 displays heatmaps for the eco-driving control task, with each heatmap varying an aspect such as green phase, inflow, and penetration rate, These visuals illustrate the transferability of strategies from source tasks (vertical axis) to target tasks (horizontal axis), highlighting the impact of traffic light phases, vehicle inflow, and proportion of guided vehicles on strategy effectiveness. Notably, longer green phases correlate with improved performance and transferability. For inflow variations, reduced inflow typically yields better outcomes. However, variations in the penetration rate of guided vehicles show minimal impact on performance differences.

---

10
20
25
(a) Green phase variation
(b) Inflow variation
(c) Penetration rate variation
Figure 18: Examples of transferability heatmap for eco-driving control.
Figure illustrates the normalized generalized performance across variants of eco-driving control tasks, specifically looking at variations in green phase time, inflow, and penetration rate. The graphs depict performance enhancement over transfer steps for different strategies. Notably, MBTL consistently demonstrates superior performance across all variations, indicating robust adaptability to changing task parameters.
Results
Green Phase Time
Inflow
Penetration Rate
Nomalized performance
0.50
0.5
0.5
0.4
0.40
0.3
0.3
0.35
10*
10a
104
10a
Number of samples trained (Log scale)
Number of samples trained (Log scale)
Number of samples trained (Log scale
Random -...
Independent
Multitask
Greedy
Equidistant
MBTL
Seq. Oracle
Figure 19: Comparison of normalized generalized performance of all target tasks: Eco-driving control.
A.4.7 Details about advisory autonomy benchmark
Advisory autonomy involves a real-time speed advisory system that enables human drivers to emulate the system-level performance of autonomous vehicles in mixed autonomy systems (40 7 [5]. Instead of direct and instantaneous control, human drivers receive periodic guidance, which varies based on road type and guidance strategy. Here, consider the different frequencies this periodic guidance as contextual MDPs since the zero-order hold action affects the transition function.
Figure 20 illustrates two distinct traffic network configurations used in the advisory autonomy task: single-lane lane ring and a highway ramp. The single-lane ring features 22 vehicles circulating the ring, with only one being actively controlled, presenting relatively controlled environment for testing vehicle guidance systems. The highway ramp scenario introduces a more complex dynamic, where vehicles not only travel along the highway but also merge from ramps, creating potential stop-and-go traffic patterns that challenge the adaptability of autonomous guidance systems.
single- scenario, the state space includes the speeds of the ego and leading vehicles, along with the headway. For highway ramp scenarios, additional states cover the relative positions and speeds of adjacent vehicles. Actions vary by guidance type: for acceleration guidance, the action space is continuous, ranging from -1 to 1; for speed guidance, it has ten discrete actions compared to the speed limit. Rewards are based on system throughput or average speed of all vehicles in the system.
Problem Definition
25

---

We explore different durations of coarse coarse-grained guidance holds to test various levels of human compatibility, adjusting the model based on observed driver behaviors and system performance.
Context Variations
License
Advisory autonomy benchmark falls under MIT License
40
a CNO
CNI
Guidance hold duration
(e.g., 10 sec)
Single lane Ring
0000
Highway Ramp
Figure 20: Illustration of the traffic networks in advisory autonomy task.
0.975
Crracle Transfer
Normaltedd pertomsres
0.8-
0.950
Independent
Multhask
0.6
0.4
0.875
0.2
0.850
0.2
0.0
0.825
0.0
Ring
Ring
Ramp
Ramp
Figure 21: Normalized performance of Oracle Transfer, independent training, and multi-task training under Advisory Autonomy benchmark with human compatibility task variations.
Potential of multi-policy training and zero-shot transfer Figure shows that ring-road networks tend to yield higher performance and smaller gaps compared to highway ramp scenarios. addition, independent training exhibits greater performance drop and variability due training instability. Oracle Transfer retains clear potential improvements over other baselines.
Transferability heatmap Figure 22 showcases heatmaps of transferability for advisory autonomy tasks, each varying in specitic aspects: acceleration guidance and speed guidance across a single lane ring and highway ramp. These heatmaps demonstrate the effectiveness of strategy transfer from each source task (vertical axis) to each target task (horizontal axis), capturing how variations in task conditions influence adaptability. For acceleration guidance in ring setup (a), transferability is generally higher among tasks with similar acceleration demands. In contrast, speed guidance on ramp (d) reveals more variability in transferability, potentially due to the complexity of speed adjustments in ramp scenarios.
Results Figure 23 illustrates the comparison of normalized generalized performance for advisory autonomy tasks, specifically acceleration and speed guidance a ring and acceleration guidance on ramp. The graphs demonstrate that MBTL consistently exhibits higher performance across all tasks. Particularly, acceleration guidance in both ring and ramp scenarios shows significant performance improvements transfer steps, with MBTL closely matching in some instances.
A.4.8 Details of control benchmarks
For this experimental phase, we selected context-extended versions standard RL environments from the CARL benchmark library, including Cartpole, Pendulum, BipedalWalker and Halfcheetah. These environments were chosen to rigorously test the robustness and adaptability of our MBTI. algorithm under varied conditions that mirror the complexity encountered in real-world scenarios.
26

---

(a) Ring with acceleration (b) Ring with speed guide guide
(c) Ramp with accelcration (d) Ramp with speed guide guide
Figure 22: Examples of transferability heatmap for advisory autonomy.
Acceleration guidance in Ring
Speed guidance in Ring
Acceleration guidance in Ramp
0.98
Normalized performence
0.9
0.6
0.8
0.96
0.7
0.95
0.6
0.94
0.2
0.93
0.5
10
10
10
104
10"
Number samples traiined (Log scale)
Number of samples trained (Log scale)
Number samples trained (Log scale)
Random
Independent
Multitask
Greedy
Equidistant
MBTL
Seq, Oracle
Figure 23: Comparison of normalized generalized performance target tasks: Advisory autonomy.
Context Variations: In the Cartpole tasks, we explored CMDPs with varying cart masses, pole lengths, and pole masses. For the Pendulum, the experiments involved adjusting the timestep duration, pendulum length, and pendulum mass. The BipedalWalker was tested under different settings of friction, gravity, and scale. Similarly, in the Halfcheetah tasks, manipulated parameters such as friction, gravity, and stiffness to simulate different physical conditions. These variations critically influence the dynamics and physics of the environments, thereby presenting unique challenges that test the algorithm's 's capacity to generalize from previous learning experiences without the need for extensive retraining. The range of context variations was established by scaling the default values specified in the CARL framework from 1.1 to 10 times, enabling a comprehensive examination of each model's performance under drastically different conditions.
We utilized the default implementation of the DQN, A2C, and PPO algorithm with default hyperpa- rameters provided by the
library 34]
License: CARL falls under the Apache License 2.0 as permitted by all work that we use
A.4.9 Details about Cartpole benchmark
Potential of multi-policy training and zero-shot transfer Cartpole may be considered simpler benchmark than traffic benchmarks, yet independent and multi-task training methods still face notable difficulty when faced with context variations. Figure 24 shows that Oracle Transfer performs the highest performance across problem variations, while independent training or multi-task training shows a larger variance in performance.
27

---

1.0
1.0
1.0
Normalized performance
0.8
0.8
0.6
0.4
0.4
Trensfer
Independent
Multitask
0.2
0.2
3.0
4.0
2
0.0
0.8
Length of Pole
Mass of Cart
Mass of Pole
Figure 24: Normalized performance Oracle Transfer, independent training, and multi-task training in Cartpole benchmarks.
presents transferability heatmaps for the Cartpole task with variations in three physical properties: mass of the cart, length of the pole, and mass of the pole. Each heatmap illustrates how well strategies transfer from source tasks (vertical axis) to target tasks (horizontal axis), depicting the influence of each parameter on control strategy effectiveness. For the mass of the cart variation (a), transferability decreases as the mass difference increases. In the length of the pole variation (b), strategies less transferable between significantly different pole lengths. Similarly, for the mass of the pole variation (c), variations show divergent transferability depending on the extent of mass change.
Transferability heatmap Figure 25
18
06
04
24
(a) Mass of cart variation
(b) Length of the pole variation
(c) Mass of pole variation
Figure 25: Examples transferability heatmap for Cartpole.
Results Figure 26 presents a comparison of normalized generalized performance for the Cartpole task across different strategies when varying the mass of the cart, length of the pole, and mass of the pole. In the mass of cart variation, performance generally increases with transfer steps, with MBTL strategies achieving the highest scores. This indicates robust adaptability changes cart mass. Similar trends are observed with length variation and mass of pole variation. MBTL shows close to Oracle performance.
A.4.10 Details about Pendulum benchmark
Potential of multi-policy training and zero-shot transfer Pendulum is also one of the simplest benchmarks in classic control. In the Pendulum benchmark, we vary three key parameters: time step (left), pole length (middle), and ball mass (right). As shown in Figure specific contexts transfer effectively to new tasks, particularly under Oracle Transfer. For certain configurations (e.g., shorter poles, lighter balls), Independent and Oracle Transfer both excel, while multi-task struggles. These results suggest remaining performance gap that multi-policy training and zero-shot transfer could help bridge.
few well-trained policies
28

---

of Cart
Length
of
1.0
1.0
Normalizdd performance
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.0
0.2
10
10
10"
10
10
10
Number of samples trained (Log scale)
Number of samples traiied (Log scale)
Number of samples trainnd (Log scale)
Random
Independent
Multitask
Greedy
Equidistant
MBTL
Seq. Oracle
Figure 26: Comparison of normalized generalized performance of all target tasks: Cartpole
1.0
Oracle Transfer
Noraaizedd performance
Independent
0.9
1.0
Multitask
0.8
0.8
0.8
0.7
0.6
35
0.6
0.4
0.4
0.5
0.2
10
Time Step
Length of Pole
Mass of Ball
Figure Normalized performance Oracle Transfer, independent training, and multi-tase training in Pendulum benchmarks.
Figure 28 provides visual mapping which trained policy is being applied for each specific context in the Pendulum benchmark. CMDP with time step variation demonstrates that the tasks are covered by few "good" policies nearby. In addition, contexts with shorter poles and lighter balls often gravitate toward single high-performing policy, whereas more challenging configurations may require a specialized or distinct policy. This illustrates how Oracle Transfer can seamlessly select from suite of learned policies, demonstrating robust zero-shot transfer and stronger adaptability.
Oracle Transfer
Normalized performance
1.0
1.2
1.2
Independent
Multitask
0.9
1.0
0.8
0.8
0.7
0.6
0.8
0.6
0.2
0.0
0.05
0.1
0.15
0.2
0.25
3
3
Time
Length
Figure 28: Visulazation on which policy is used to solve specific context-MDP in Pendulum CMDP
Transferability heatmap Figure 29
presents transferability heatmaps for the Pendulum task with variations in three physical properties: timestep, length of the pendulum, and mass of the pendulum. Each heatmap illustrates how effectively strategies transfer from source tasks (vertical axis) to target tasks (horizontal axis), highlighting the impact of each parameter on control strategy effectiveness. For the timestep variation (a), there appears be high consistency in transferability across different timesteps, especially around the diagonal axis. In the length of the pendulum variation (b), transfer-
29

---

ability decreases with greater length differences. Similarly, for the mass of the pendulum variation (c), transferability shows variability dependent on the extent of mass changes.
12
11
05
13
Suiget
Dinger
Derget
(a) Timestep variation
(b) Length of the pendulum varia-
(c) Mass of pendulum variation
tion
Figure 29; Examples of transferability heatmap for Pendulum.
Results Figure 30 shows a comparison normalized generalized performance for the Pendulum task across different strategies when varying the timestep, length of the pendulum, and mass of the pendulum. For the length of the pendulum variation and mass the pendulum one, MBTL strategies demonstrate the highest scores, suggesting robust adaptability changes in pendulum dynamics. MBTL shows performance close to that of the Oracle across all variations, indicating its effectiveness in handling dynamic changes in system parameters.
Time Step
Length Pendulum
Mass Pendulum
0.9
Normalized perfornance
0.80
0.8
0.7
0.65
0.70
0.60
10*
10a Number of samples trained (Log scale)
10
10a
Number of samples trained (Log scale)
Number of samples trained (Log scale
Random
Independent
Multitask
Greedy
Equidistant
MBTL
Seq. Oracle
Figure 30: Comparison of normalized generalized performance of all target tasks: Pendulum.
A.4.11 Details about Bipedal benchmark
Potential of multi-policy training and zero-shot transfer Figure 31 compares the performance of different RL training methods in the BipedalWalker benchmark. independent training typically performs nearly as well but suffers intermittent dips. Similarly, multi-task training experiences larger swings, occasionally collapsing to low performance certain parameter regions. However, Oracle Transfer remains near-perfect across every setting. These patterns highlight how multi-policy training with zero-shot transfer and per-task training generally fare better than single universal model when faced with diverse environment dynamics.
30

---

1.0
Normalized performance
0.9
0.8
Oracle Transfer
0.7
0.7
Independent
Multitask
0.5
0.5
0.4
0.4
5.0
15.0
20
300
Friction
Gravity
Scale
Figure 31: Normalized performance Oracle Transfer, independent training, and multi-tast training
in Bipedal BipedalWalker Walker benchmarks.
Transferability heatmap Figure presents transferability heatmaps for the BipedalWalker task, focusing three variations: friction, gravity, and scale. Each heatmap illustrates the effectiveness of strategy transfer from source tasks (vertical axis) to target tasks (horizontal axis), highlighting how each parameter influences control strategy adaptability. For friction variation (a), strategies show uniform transferability across different friction levels. In gravity variation (b), transferability highly variable, suggesting that strategies need specific tuning for different gravity levels. For scale variation (c), the heatmap indicates variable transferability, reflecting the challenges of scaling control strategies.
00
(a) Friction variation
(b) Gravity variation
(c) Scale variation
Figure 32: Examples transferability heatmap for BipedalWalker.
Results Figure 33 shows the comparison normalized generalized performance for all variations within the BipedalWalker task. There no huge difference in performance for all three cases, but we look into the tabualr results in Table 2 MBTL shows the highest performance across varying conditions, indicating their robustness in adapting to changes physical parameters of the model. This suggests that these strategies are more effective in handling the complexities introduced by different frictions, gravities, and scales compared to other baselines.
A.4.12 Details about HalfCheetah benchmark
Potential of multi-polic training and zero-shot transfer In this HalfCheetah benchmark, each subplot examines how policies adapt to changing friction, gravity, and stiffness (Figure (34). Oracle Transfer maintains nearly perfect scores for all parameter ranges, indicating robust zero-shot adapt- ability. In contrast, independent training experiences larger fluctuations, while multi-task training remains consistent yet a lower performance plateau. The clear gap between Oracle Transfer and the other methods highlights the advantage of leveraging specialized multi-policy training solutions that effectively transfer across diverse dynamics.
31

---

Friction
Gravity
Scale
0.98
Normalized performance
0.90
0.96
0.94
0.8
0.92
0.80
0.7
0.90
0.75
0.88
0.6
0.86
10
10
10
10
10
10*
Number samples trained (Log scale)
Number samples trained (Log scale)
Number samples trained (Log scale.
Random
Independent
Multitask
Greedy
Equidistant
MBTL
Seq. Oracle
Figure 33: Comparison of normalized generalized performance of all target tasks: Bipedal Walker. Sipeda/Walker.
1.0
1.0
Normaiized perforance
0.9
0.8
0.8
0.7
0.7
0.6
0.5
Oracie Transfer
0.4
0.4
Independent
Multitask
0.0
1.2
2.4
4.8
6.0
0.0
19.6
39.2
58.8
78.4
98.0
$000000000000020000000000
Friction
Gravity
Stiffness
Figure 34: Normalized performance Oracle Transfer, independent training, and multi-task training in HalfCheetah benchmarks.
Transferability heatmap Figure 35
displays transferability heatmaps for the HalfCheetah task, focusing three physical properties: friction, gravity, and stiffness. Each heatmap demonstrates the transferability of strategies from source tasks (vertical axis) to target tasks (horizontal axis). For friction variation (a), there is uniform high transferability across different friction levels, indicating that strategies are robust to changes in friction. Gravity variation (b) shows less consistent transferability. suggesting sensitivity to gravity changes that might require adaptation of strategies. Stiffness variation (c) similarly demonstrates variable transferability, highlighting the challenges of adapting to different stiffness levels in control strategies.
14
00
107
Targst 200k
Targst vock
Friction variation
(b) Gravity variation
(c) Stiffness variation
Figure 35: Examples of transferability heatmap for HalfCheetah.
Results Figure 36 presents a comparison of normalized generalized performance across various strategies for the HalfCheetah task with respect to the varied physical properties. The results indicate that the MBTL generally outperforms others, particularly in managing variations gravity and stiffness, suggesting the superior adaptability of these models to physical changes in the task
32

---

environment. The trends across different parameters confirm the critical impact of task-specifie dynamics on the effectiveness of the tested strategies.
Friction
Gravity
Stiffness
1.0
1.0
Nonhalized performanc
0.9
0.9
0.8
0.8
0.7
0.6
0.6
0.6
0.5
0.5
0.5
Number of samples trained (Log scale)
Number of samples trained (Log scale)
Number of samples trained (Log scale)
Random
-....
Independent
.....
Multitask
Greedy
Equidistant
MBTL
Seq. Oracle
Figure 36: Comparison of normalized generalized performance of all target tasks: HalfCheetah.
A.4.13 Implementation of the recent multi-task baselines
In Figure 37 compare two recent multi-task algorithms- PPCC and MOORE 16
43
with sev-
eral our baselines tested on Cartpole CMDP benchmark. Although MOORE underperforms relative our baseline multitask implementation, PaCo achieves competitive or even higher performance certain points, demonstrating its potential to generalize across multiple tasks. Also, it is important note that those algorithms naively implemented without thorough investigation. These trends show that enhanced multi-task strategies can be beneficial some CMDP settings, whereas not all multi-task methods readily adapt to broader parameter variations.
Mass of Cart
Length of Pole
Mass of Pole
1.0
1.0
Normalzzed performance
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.2
0.0
0.2
10
10
10a
Number of samples (Log scale)
Number of samples (Log scale)
Number of samples [Log scale)
Equidistant
Seq.
Mulbitask (MOORE)
Independent
Greedy
MBTL
Multitask (PaCo)
Figure 37: Normalized performance comparison of PaCo and MOORE 16 on Cartpole benchmark.
33

---

A.5
Potential impacts
Our work has the potential reduce the computational effort needed to solve complex real-world problems, offering scalable solutions for implementing deep reinforcement learning in dynamic environments. While there are no immediate negative societal impacts identified, ongoing research will continue to assess the broader implications of deploying these technologies in urban settings.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the contributions and scope, match- ing the theoretical and experimental results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Conclusion (Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

The paper includes all necessary assumptions and provides complete proofs in the Appendix A.3
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed experimental settings, including models and hyperparameters, are provided to ensure reproducibility. Also, the code is submitted by the authors.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
36

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code submitted by the authors with zip files, with detailed instructions for reproducing the results included in the supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic//uidee//ooeettliiiss wtt.:eeewuumiisioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplggguuuieeeeee
Saaeee//CoSSSubisssS.S.. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides comprehensive details about the experiments, including data generation, hyperparameters, and optimizer types, to ease understanding of the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars and confidence intervals are reported for all significant experiments, with methods and assumptions clearly stated.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
37

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The type of compute resources, memory, and execution time for each experi- ment are specified to ensure reproducibility in Appendix A.4
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research adheres to the NeurIPS Code of Ethics, with no deviations from the guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes section discussing both the potential positive and negative societal impacts of the research.
Guidelines:
38

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve data or models with high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper uses the CARL framework under the Apache License 2.0, with proper credit given and license terms mentioned in the Appendix A.4.8
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
39

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets, including code for the MBTL algorithm and experiments, are well-documented and submitted with an anonymized zip file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
40

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "IyeuYDV8wv", "reviewer": "Reviewer_21Qs", "summary": "The paper introduces a framework called Model-Based Transfer Learning (MBTL) for solving contextual reinforcement learning problems. By modelling the performance loss as a simple linear function of task context similarity, the authors leverages Bayesian optimization techniques and provides theoretical analysis showing that MBTL exhibits sublinear regret in the number of training tasks, and discusses conditions to further tighten the regret bounds. MBTL is validated on a range of simulated traffic management scenarios and standard control benchmarks, demonstrating superior performance compared to baselines like exhaustive training, multi-task training, and random task selection.", "questions": "What's the foundamental difference between \"exhaustive training\" and \"multi-task RL\"? Based on the brief description in line 257-258, they seem similar to me. Also why is \"multi-task RL\" even worse than \"random\" most of the time (in Table 1, 2)?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. In general, the paper is easy to follow and well-motivated. The figures are helpful for readers to quickly grasp the key concepts and problem settings.\n\n2. Active and transfer learning for contextual RL is an interesting and practical problem, especially given that exhaustive/multi-task training on RL tasks can be computationally expensive.\n\n3. The experiments are conducted on a relative broad range of benchmarks and context variations.", "weaknesses": "I do think there are several major concerns regarding the current form of the paper:\n\n1. The paper (especially the theoretical analysis) is based on too many assumptions: continuity of task space and performance function, linear genearalization gap and deterministic MDP transitions, which oversimplifies the problem and significantly limits the applicability range of the proposed method. The most concerning is Assumption 3 (Linear generalization gap), for which I have no clue why and when this can hold. Assuming Lipschitz continuity of the generalization gap seems much more reasonable to me, which makes assumption 3 an inequality instead of an (approximate) equality.\n\n2. In line 96-98, the paper assumes that a vector representation/context feature $x$ of each task is given a priori, based on which the continuity assumptions 1-2 are made.  Howeveer, as in line 333-335,  context features are often not visible. Assuming prior knowledge of such information further limits the applicability range of the proposed method.\n\n3. (2 continued) In fact, no matter whether context features are given a priori, there are many effective methods which perform transfer/meta-RL by learning a conditioned policy [1][2][3], which should be significantly better than the \"multi-task\" baseline adopted in the paper. However, there is currently no empirical evidence on how well the proposed method compared to these stronger baselines.\n\n4. Even though Theorem 2 and Corollary 2.1 2.2 give quantitative regret bounds assuming the search space can be narrowed down at each training step, there is no theoretical guarantee on how the proposed method, especially MBTL-GP, can effectively realize such restricted search space. Hence the current analysis is not complete in terms of proving the effectiveness of proposed method.\n\n[1] Rakelly, Kate, et al. \"Efficient off-policy meta-reinforcement learning via probabilistic context variables.\" International conference on machine learning. PMLR, 2019.\n\n[2] Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pages 9767–9779. PMLR, 2021.\n\n[3] Li, Lanqing, Rui Yang, and Dijun Luo. \"Focal: Efficient fully-offline meta-reinforcement learning via distance metric learning and behavior regularization.\" ICLR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the reviewer's constructive and insightful feedback. We kindly request the reviewer to also refer to our general comments, which address (1) concerns with the linear generalization gap assumption, (2) the application of MBTL to tasks with high-dimensional visual inputs, and (3) multi-task baselines. Should our response meet the reviewer's expectations, we would be grateful if they could consider increasing the score.**\n\n> The paper (especially the theoretical analysis) is based on too many assumptions: continuity of task space and performance function, linear genearalization gap and deterministic MDP transitions, which oversimplifies the problem and significantly limits the applicability range of the proposed method. The most concerning is Assumption 3 (Linear generalization gap), for which I have no clue why and when this can hold. Assuming Lipschitz continuity of the generalization gap seems much more reasonable to me, which makes assumption 3 an inequality instead of an (approximate) equality.\n\nThank you for your valuable comments. We refer to **General Response [GR1]**, where we detail interpretations of the linear generalization gap. Moreover, assuming Lipschitz’s continuity of the generalization gap could bound the performance range, it would provide a more flexible and realistic generalization gap estimation. However, incorporating Lipschitz continuity into our MBTL algorithm involves additional considerations on effectively utilizing this bound to optimize task performance. We acknowledge that further investigation is needed to understand the potential benefits of Lipschitz continuity.\n\n> In line 96-98, the paper assumes that a vector representation/context feature $x$ of each task is given a priori, based on which the continuity assumptions 1-2 are made. Howeveer, as in line 333-335, context features are often not visible. Assuming prior knowledge of such information further limits the applicability range of the proposed method.\n\nWe appreciate the reviewer's insightful comments. This paper specifically addresses the SSTS problem settings where context features are visible. Many real-world applications for solving multiple related tasks with visible contexts, such as robotics manipulations, recommendation systems, autonomous driving scenarios, and personalized healthcare, often have accessible context features (e.g., robot configurations, user preferences, traffic, environmental conditions, and patient health records). These contexts allow our method to be broadly applicable in these domains.\n\nWe acknowledge that there are also scenarios where context features might not be visible or available. Extending MBTL to handle such hidden context scenarios is a promising direction for future work. This could involve developing techniques to infer or estimate hidden contexts from observable data. We appreciate your suggestion and look forward to exploring this avenue in our future research.\n\n> (2 continued) In fact, no matter whether context features are given a priori, there are many effective methods which perform transfer/meta-RL by learning a conditioned policy [1][2][3], which should be significantly better than the \"multi-task\" baseline adopted in the paper. However, there is currently no empirical evidence on how well the proposed method compared to these stronger baselines.\n\nWe appreciate this feedback and agree that including comparisons with stronger baselines would strengthen our evaluation. We refer the reviewer to the **General Response [GR3]** and Figure R.3 about the multi-task baseline. Additionally, our method has an advantage over PaCo [1] because it does not require learning additional parameters during training, thanks to the simple modeling of performance loss in transfer.\n\n> Even though Theorem 2 and Corollary 2.1 2.2 give quantitative regret bounds assuming the search space can be narrowed down at each training step, there is no theoretical guarantee on how the proposed method, especially MBTL-GP, can effectively realize such restricted search space. Hence the current analysis is not complete in terms of proving the effectiveness of proposed method.\n\nThank you for your constructive comments. To address your concerns, we offer a more detailed discussion on the practical realization of a restricted search space and include empirical evidence to support our theoretical claims. The graph in Figure R.4 illustrates the impact of search space elimination on the performance of different strategies over multiple transfer steps. It compares the empirical search space of MBTL-GP in tasks provided in this paper with the examples given in Corollaries 2.1 and 2.2. Corollary 2.2, representing a greedy strategy (MBTL-PS), demonstrates a more aggressive reduction in search space, leading to the tightest regret bounds and superior performance. Corollary 2.1 also shows a rapid reduction in max space and improved performance. Although we cannot guarantee the theoretical performance, our empirical results indicate that MBTL-GP can achieve competitive performance compared to MBTL methods using simpler strategies.\n\n> What's the foundamental difference between \"exhaustive training\" and \"multi-task RL\"? Based on the brief description in line 257-258, they seem similar to me. Also why is \"multi-task RL\" even worse than \"random\" most of the time (in Table 1, 2)?\n\nThank you for your feedback, and we apologize for any confusion in differentiating the two. Exhaustive training involves training separate models for all different cMDP tasks, while multi-task RL trains a single universal model that can generalize across different cMDP tasks. As shown in our transferability heatmap, some environments have low transferability to other tasks, making it challenging for multi-task RL to derive a single model that can effectively solve a wide range of task variants. Random selection of models in SSTS can sometimes cover a broader context range better than a single multi-task RL model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uhQHLIBL69", "reviewer": "Reviewer_wJEK", "summary": "The paper proposes a new framework to estimate the expected generalization performance across different tasks where the differences have an explicit and model-based structure. To improve the expected generalization performance via training on selected tasks, the paper proposes naive and Bayesian optimization based method to effectively explore the task space to find a policy that have the optimal zero-shot performance given a selected task. The experiments demonstrate that the proposed MBTL method outperform other baseline method and reach the performance of the optimal transfer method assuming full knowledge.", "questions": "- Could you comment on the challenges of applying MBTL-GP on vision-control tasks?", "limitations": "Yes, the author has discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- By and large, the paper is written well. I especially appreciated the detailed discussion on SSTS problem formulation and its relations to robust RL training.\n- The idea of Bayesian Optimization to search training tasks using generalized performance estimation where tasks differences are “model-based” or have an explicit structure is novel and useful.\n- The section on analysis of BO and its comparison with ES and PS are also interesting, and highlights the sublinear regret theoretical results.\n- The theoretical results appear to be correct.", "weaknesses": "- The empirical evaluation is by and large only on low dimensional systems. It would have been interesting to see how the method would scale with more challenging, high-dimensional tasks, such as those commonly found in vision control tasks in robotics.\n- It would have been interesting to see how well this method would work with policies/controllers not parametrized with neural networks (i.e. kernel machines).\n- Some additional commentary on how to use the GP to search for a trained policy given a selected task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the reviewer's thoughtful comments and suggestions. We encourage the reviewer to review our general comments, which include additional experiments addressing (1) concerns with the linear generalization gap assumption, (2) the application of MBTL to tasks involving high-dimensional visual inputs, and (3) multi-task baselines. We hope that the new results will satisfactorily address the concerns raised, potentially leading to a reconsideration of our score.**\n\n> The empirical evaluation is by and large only on low dimensional systems. It would have been interesting to see how the method would scale with more challenging, high-dimensional tasks, such as those commonly found in vision control tasks in robotics.\n\nWe appreciate constructive comments on the scalability of our method to the high-dimensional tasks. With your suggestion, the authors conducted experiments with vision control tasks. We refer the reviewer to **General Response [GR2]** for visualizations and interpretations of our learning results.\n\n> It would have been interesting to see how well this method would work with policies/controllers not parametrized with neural networks (i.e. kernel machines).\n\nThank you for your valuable suggestion. One of the strengths of our work is its flexibility and extensibility, allowing the use of various methods beyond just reinforcement learning. Our method can be applied to other approaches such as kernel methods, radial basis functions (RBF), model predictive control (MPC), and optimal control.\nIn this paper, we were motivated by traffic examples and focused on developing a simple and practical algorithm to efficiently solve a wide range of CMDPs. While we have primarily concentrated on deep reinforcement learning algorithms using neural network parameterizations, exploring the applicability of our method with alternative approaches is indeed an interesting direction for future research.\nIn response to your comments, we conducted preliminary experiments with support vector machines (SVM), one of the most popular kernel machines, to solve the CartPole CMDPs. Though SVMs are not inherently designed for sequential decision-making, we used them in a supervised learning context where we trained the SVM on the best actions given certain states. After training the SVM model, we transferred it to other CMDPs and collected the rewards. \nOur preliminary result in Figure R.5 shows that an SVM-based controller trained on a default configuration actually solves other tasks. \n The results of these preliminary experiments in Fig. R5. indicate that our method shows promise when applied to kernel machines as well. This suggests that our approach can potentially be extended to other non-neural network-based methods. We appreciate your suggestion and will consider including more detailed experiments and discussions in future work.\n\n> Some additional commentary on how to use the GP to search for a trained policy given a selected task.\n\nThank you for your insightful comment. In our MBTL-GP method, the GP is utilized to predict the performance of a policy trained on a source task when applied to a target task. The modeled generalization gap helps predict the transferability of that source task. Using the GP model, we apply Bayesian Optimization (BO) to select the next training task. The acquisition function in BO balances exploration and exploitation by considering both the predicted mean performance and the uncertainty (variance) associated with the prediction. Specifically, our proposed modified UCB acquisition function is used. This allows us to strategically choose tasks that are likely to improve overall performance while reducing uncertainty. Once a new task is selected, the policy is trained on this task, and the resulting performance data is used to update the GP model. This iterative process continues, progressively refining the GP model and improving task selection. This approach allows the MBTL framework to efficiently search for trained policies by leveraging the predictive power of GP and the strategic task selection of BO. For a better understanding, we added Figure R.1 to illustrate the MBTL process.\n\n> Q1. Could you comment on the challenges of applying MBTL-GP on vision-control tasks?\n\nWe appreciate your suggestion, and we understand the importance of evaluating our method on more complex tasks with high-dimensional state spaces. In response, we conducted additional experiments on high-dimensional environments. Unfortunately, the CARL benchmark no longer supports the RNA design application, so we decided to focus on vision-control experiments instead. The experiments are based on the benchmark [1], which supports visual generalization for reinforcement learning. We ran preliminary experiments with this benchmark using the CartPole task, specifically examining contextual MDPs of the CartPole environment with different frame skip parameters. This setup is similar to the task variant of advisory autonomy in traffic tasks. We kindly request the reviewer to refer **General Response 2 [GR2]**. The preliminary results in Figure R.2 show that MBTL algorithms ranging from simple strategies to GP-based algorithm also work in complex high-dimensional state space. These experiments demonstrate the scalability and robustness of MBTL in more challenging and complex settings.\n\n### References\n- [1] Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou, “MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3461–3475, Mar. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HS6E0GwV3w", "reviewer": "Reviewer_ECSQ", "summary": "The paper proposes Model-Based Transfer Learning (MBTL) in Contextual RL, which solves multiple related tasks and enhances generalization across different tasks. MBTL strategically selects a set of source tasks to maximize overall performance and minimize training costs.\nThe paper theoretically demonstrates that the method exhibits regret that is sublinear in the number of training tasks. \nMBTL achieves greater performance than other baselines (e.g., exhaustive training, multi-task training, and random selection) on urban traffic and control benchmarks.", "questions": "Below, I have a few questions and feedback for the authors:\n- How does the computational time consumption compare between MBTL and other baselines (exhaustive training, multi-task training, and random selection)?\n- I am curious to see experimental results in complex environments, such as visual environments.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and organized, and includes a thorough illustrative diagrams and examples.\n- The problem formulation is well done, with clear mathematical representation. The analysis of Bayesian Optimization appears to be thorough.\n- The authors provide code for training and evaluating their proposed method. This significantly facilitates reproducing the experimental results and extending the introduced method.", "weaknesses": "- The assumptions are too tight, particularly Assumption 3, which models the generalization gap using linear constraints. This approach is unsuitable for complex environments and therefore lacks generalizability.\n- The experimental environments (urban traffic and control benchmarks) are overly simplistic, consisting solely of vector environments with low-dimensional state spaces. The study lacks comparisons with complex tasks, such as those in the CARL benchmark, including games and a real-world application of RNA design.\n- The ablations on DRL algorithms (DQN, PPO, A2C) utilize outdated methods. Why not use more recent RL baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The authors truly appreciate the reviewer’s positive feedback on our work. We invite the reviewer to also take a look at our general comments, which include additional experiments on (1) concerns with the linear generalization gap assumption, (2) application of MBTL to tasks with high-dimensional visual inputs, and (3) multi-task baselines. We hope that the new results will satisfactorily address the concerns raised, potentially leading to a reconsideration of our score.**\n\n> The assumptions are too tight, particularly Assumption 3, which models the generalization gap using linear constraints. This approach is unsuitable for complex environments and therefore lacks generalizability.\n\nWe acknowledge your concern about assumption 3. However, ironically, this assumption is the key strength of our MBTL algorithm. The authors kindly request the reviewer to look at the **General Response 1 (GR1)**. \n\n> The experimental environments (urban traffic and control benchmarks) are overly simplistic, consisting solely of vector environments with low-dimensional state spaces. The study lacks comparisons with complex tasks, such as those in the CARL benchmark, including games and a real-world application of RNA design.\n\nWe appreciate your suggestion, and we understand the importance of evaluating our method on more complex tasks with high-dimensional state spaces. In response, we conducted additional experiments on high-dimensional environments. Unfortunately, the CARL benchmark [1-2] no longer supports the suggested RNA design application, so we focused on vision-control experiments instead. These experiments are based on the MetaDrive benchmark [3], which supports visually generated observations for driving scenarios. The authors kindly request the reviewer to look at **General Response 2 (GR2)** for visualization of our results on high-dimensional state spaces. The preliminary results show that MBTL algorithms ranging from simple strategies to GP-based algorithm also work in complex high-dimensional state space. These experiments demonstrate the scalability and robustness of MBTL in more challenging and complex settings.\n\n> The ablations on DRL algorithms (DQN, PPO, A2C) utilize outdated methods. Why not use more recent RL baselines?\n\nWe apologize for any confusion caused by using the term \"ablation.\" To clarify, we intended this section as a \"sensitivity analysis\" to demonstrate that the MBTL selection process is robust across different types of RL algorithms, whether they are value-based or policy-based, and that these methods may or may not significantly affect the generalization gap. Our intention was to show that the effectiveness of MBTL is not heavily dependent on the specific RL algorithm used. The primary goal was to highlight that both value-based methods (like DQN) and policy-based methods (like PPO and A2C) are compatible with our framework, indicating the versatility of MBTL. Additionally, we aimed to maintain consistency with the RL algorithms used in the original papers from which the traffic experiments were derived. For instance, the eco-driving control paper [4] utilized PPO variants, and we adopted similar algorithms to ensure a fair comparison and reproducibility of results.\n\n> How does the computational time consumption compare between MBTL and other baselines (exhaustive training, multi-task training, and random selection)?\n\nThank you for your question on the computation time. If we assume the same training time for each model, the number of trained models presented in Table 1 and Table 2 provides the order of magnitude difference in computational time across different methods. For instance, Exhausitive Training or Oracle Transfer requires training all tasks, which will need $N(=|X|)$ models, while MBTL requires $k$ models to train. However, the calculation for multi-task reinforcement learning deviates from this, since it depends on the batch size though it trains a single model. Our results (Table 1 and 2 in the main text) indicate that MBTL significantly reduces training time compared to exhaustive training and multi-task training while achieving comparable or better performance. This efficiency is primarily due to the strategic selection of training tasks, which minimizes redundant training. On the other hand, we include a detailed comparison of the computational time required for MBTL source task selection in our experiments. Specifically, MBTL-GP requires more computation than simple methods.\n\n||MBTL-ES|MBTL-GS|MBTL-GP|\n|--|--|--|--|\n|Pendulum|4.2518E-05|0.00018432|1.61098456|\n|Cartpole|3.28488E-05|0.00026944|1.6663856|\n|BipedalWalker|3.25309E-05|0.00015042|1.64290924|\n|HalfCheetah|3.26369E-05|0.00014559|1.63489845|\n|Traffic Signal|3.49164E-05|0.000156|0.5571901|\n|Eco-Driving|3.16037E-05|0.00014793|0.62795639|\n|AdvisoryAutonomy|3.17097E-05|0.00014257|0.69461881|\n\nSpecifically, the running time for MBTL algorithms is relatively short compared to the actual computation time required for training RL models. When comparing the computation time for the SSTS process alone, simple strategies such as MBTL-ES (Equidistant Strategy) and MBTL-GS (Greedy Strategy, previously MBTL-PS) require almost negligible computation time. In contrast, MBTL-GP (Gaussian Process) requires additional computation time for the Bayesian optimization process.\n\nOverall, the strategic task selection in MBTL results in a substantial reduction in the number of models trained, which in turn reduces the overall computational burden. We will provide a detailed analysis of these findings in the updated results section of the revised manuscript.\n\n> I am curious to see experimental results in complex environments, such as visual environments.\n\nThank you for your comments. We refer the reviewer to **General Response [GR2]** for visualizations and interpretations of our results on visual environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**The authors appreciate each of the reviewers for their detailed and constructive comments. Here, we first respond to all reviewers before answering each reviewer’s specific question.**
### [GR1] Concerns with the Linear generalization gap assumption
Thank you for your valuable comments. Similar concerns were raised by Reviewer 21Qs and ECSQ, regarding the assumptions made in our method.
The purpose of our assumptions was not to oversimplify the problem but to design a straightforward algorithm that benefits from simple modeling. Assumptions 1-3 in Section 4.1 were made to streamline the empirical algorithm design rather than to constrain our theoretical analysis. Although it may appear to be an oversimplification, the simple modeling of training performance and the generalization gap are key strengths of this paper. Empirical findings indicate that even when these assumptions do not hold perfectly, our simple, principled algorithms remain effective. To address these concerns, we have included additional experiments that relax these assumptions. As one example of a non-linear generalization gap, we tested our algorithms with quadratic generalization gap assumptions. Preliminary results show that these methods can perform well, sometimes better than those assuming a linear generalization gap. Specifically, in tasks like CartPole variants, estimation with a quadratic function performs better than linear modeling since the actual generalization resembles a quadratic function.

Table R.1. Comparative performance of different methods with quadratic generalization gap function on CMDP tasks

|||Random|Exhaustive|MBTL-GP|MBTL-GP with quadratic function|Oracle Transfer|
|--|--|--|--|--|--|--|
|Cartpole|Mass of Cart|0.7221|0.9466|0.8212|0.7979|0.9838|
|Cartpole|Length of Pole|0.8121|0.9110|0.9124|0.926|0.9875|
|Cartpole|Mass of Pole|0.8858|0.9560|0.9351|0.9593|1|
|HalfCheetah|Gravity|0.8542|0.6679|0.9073|0.8253|0.9544|
|HalfCheetah|Friction|0.8567|0.6693|0.9274|0.8601|0.9663|
|HalfCheetah|Stiffness|0.8533|0.6561|0.9146|0.8423|0.9674|

Furthermore, non-linear generalization gaps require estimating more parameters than linear gaps, potentially complicating the achievement of effective performance. Despite this, our preliminary results are promising and indicate that our method can be effectively adapted to handle non-linear generalization gaps, enhancing its robustness and applicability across different tasks.

### [GR2] Application of MBTL to high-dimensional visual input task
We appreciate the feedback and comments from Reviewer wJEK and ECSQ regarding the applicability of MBTL to high-dimensional state space tasks. We understand the importance of evaluating our method on more complex tasks with high-dimensional state spaces.
In response to these comments, we conducted additional experiments on high-dimensional vision-control experiments. These experiments are based on the MetaDrive benchmark [1], which supports visually generated observations for driving scenarios.

We ran preliminary experiments with the MetaDrive benchmark in the three-lane four-way intersection traffic network with different traffic density variations (from 0.05 to 0.5) (Fig. R.2). The task involves controlling an autonomous vehicle in the presence of other vehicles. The controlled vehicle observations are generated from a low-level sensor using an RGB-based camera view with 200x100 pixels. Those inputs were passed through a three-layer CNN for feature extraction. The autonomous vehicle is controlled with steering and acceleration changes. 

The preliminary results show that MBTL algorithms, ranging from simple strategies to GP-based algorithms, are still effective in complex high-dimensional state spaces. These experiments demonstrate the scalability and robustness of MBTL in more challenging and complex settings, confirming the versatility of our approach in handling high-dimensional visual input tasks.

### [GR3] Multi-task baselines
We greatly appreciate the feedback from reviewer 21Qs and concur that including comparisons with stronger baselines would enhance our evaluation. In response, we thoroughly examined state-of-the-art multi-task reinforcement learning (MTRL) methods, including the suggested baselines. However, we believe the suggested CARE algorithm [2], which involves language embeddings, is not an appropriate comparison for our approach, where context variation is continuous and straightforward. Instead, we have compared our methods with the Parameter-compositional multi-task reinforcement learning (PaCo) [3] algorithm. In Figure R.3, Our preliminary implementation of PaCo [5] on CartPole CMDP variants indicates that MBTL remains competitive against these methods, demonstrating superior performance compared to our previous, more naive MTRL strategy. We would like to note that due to the limited time available for rebuttals, we could not offer comparisons with various MTRL baselines, as the training procedures require considerable computation time and effort. Also, it was unfortunate that there were a few MTRL works that didn’t release the codebase, had issues running the code, or could not be reproducible during the implementation of previous works. 

### References
- [1] Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou, “MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3461–3475, Mar. 2023.
- [2] Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pages 9767–9779. 2021.
- [3] L. Sun, H. Zhang, W. Xu, and M. Tomizuka, “PaCo: Parameter-Compositional Multi-Task Reinforcement Learning,” in Conference on Neural Information Processing Systems., 2022.

Meta-review: The paper aims at solving contextual reinforcement learning problems, with a focus on selecting which tasks to train on, such that the learned policies can be used zero-shot to generalize to unseen task. Towards this goal, author use bayesian sequential sampling strategy, where the acquisition function is based on the generalization gap of the learned policies on the sampled tasks. This allows the method to sample tasks where the policies perform the worst, and thereby quickly gaining coverage for good performance. 

Regarding this methodology, there was an active discussion where the reviewers raised several concerns regarding both theoretical and practical aspects of the paper. Particularly, regarding the linear generalization gap assumption, computational complexity when the number of tasks are large, applicability of the method to domains with higher-dimensional state features, and better baselines. In the rebuttal, authors provided several new results regarding all the concerns raised, and shared empirical validation where theoretical justification was more involved.

The overall idea is sensible, and opens a different avenue of storing few anchor policies (as opposed to having a global policy for all the tasks) to address the multi-task generalization problem. The experiments are also carefuly designed and results seem promising.
Decision: Accept (poster)