Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning
Authors: Chi Zhang
Ziying Jia
George K. Atia
Sihong He
Yue Wang
Venue: ICML 2025 poster (2025)
Keywords: zero-shot
transfer learning
distributed learning
reinforcement learning
Primary area: reinforcement_learning

Abstract: Transfer reinforcement learning aims to derive a near-optimal policy for a target environment with limited data by leveraging abundant data from related source domains. However, it faces two key challenges: the lack of performance guarantees for the transferred policy, which can lead to undesired actions, and the risk of negative transfer when multiple source domains are involved. We propose a novel framework based on the pessimism principle, which constructs and optimizes a conservative estimation of the target domain’s performance. Our framework effectively addresses the two challenges by providing an optimized lower bound on target performance, ensuring safe and reliable decisions, and by exhibiting monotonic improvement with respect to the quality of the source domains, thereby avoiding negative transfer. We construct two types of conservative estimations, rigorously characterize their effectiveness, and develop efficient distributed algorithms with convergence guarantees. Our framework provides a theoretically sound and practically robust solution for transfer learning in reinforcement learning.

Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning
Chi Zhang' Ziying Jia
George K. Atia
Sihong He² Yue Wang"
Abstract
et al., 2013), transportation (He et al., 2023a;b; Wang et al., 2023a), robotics (Nguyen & La, 2019; Liu et al., 2021) or natural language processing (Sharma & Kaushik, 2017; Uc- Cetina et al., 2023; Kirk et al., 2023a). These successes, however, often depend on access to vast amounts of data, which are critical for enabling agents to understand their environments and develop effective policies. In domains with limited data-due to high exploration costs other constraints- RL performance typically degrades. A promis- ing solution is transfer learning (TL), which leverages more data from related environments (source domains) to train policies that can be transferred to target environments. We focus on zero-shot transfer, where no target domain data available prior deployment, and multi-domain transfer. where multiple source domains are utilized.
Transfer reinforcement learning aims to derive a near-optimal policy for a target environment with limited data by leveraging abundant data from related source domains. However, it faces two key challenges: the lack of performance guaran- tees for the transferred policy, which can lead to undesired actions, and the risk of negative transfer when multiple source domains are in- volved. We propose novel framework based the pessimism principle, which constructs and optimizes conservative estimation of the tar- get domain's 's performance. Our framework effec- tively addresses the two challenges by providing an optimized lower bound on target performance, ensuring safe and reliable decisions, and by ex- hibiting monotonic improvement with respect to the quality of the source domains, thereby avoid- ing negative transfer. We construct two types of conservative estimations, rigorously character- ize their effectiveness, and develop efficient dis- tributed algorithms with convergence guarantees. Our framework provides a theoretically sound and practically robust solution for transfer learning in reinforcement learning.
perfoommacce
Despite its potential, TI. is often hindered by the Sim-to- Real Gap (Salvato et al., 2021; Zhao et al., 2020), which refers to performance degradation when transferring learned policies from source to target domains. TL. typically relies on constructing proxy to approximate target domain per- formance and transferring the proxy's optimal policy. While successful when source domains closely resemble the target domain (e.g., high-fidelity simulations), significant diver- gence between the source and target comains-caused by modeling errors, non non-stationarity. or perturbations -leads to inaccurate proxies and degraded transfer performance.
In multi-domain transfer, the presence of multiple source domains introduces additional challenges, particularly nega- tive transfer. Source domains that differ significantly from the target domain can skew the proxy, resulting in overly pessimistic or suboptimal policies that hinder the transfer process if they cannot be identified and treated properly.
1. Introduction
Reinforcement learning (RL) aims to learn a policy that optimizes an agent's performance when interacting with its environment. RL has achieved remarkable success in areas such as game playing (Silver et al., 2016; 2017; Mnih
DDepartment of Electrical and Computer Engineering, Univer- sity of Central Florida, Orlando, FL 32816, USA Department of Computer Science and Engineering, University Texas at Ar- lington, Arlington, TX 76019, USA 'Department of Computer Science. University of Central Florida, Orlando, FL 32816. USA. Correspondence to: Chi Zhang, Ziying Jia, George K. Atia, Si- hong He, Yue Wang <chi. tebbi.hhng@uuc @ucc. zxj3060@mav mavs. wttoo/@maaasst....o.oom utu. george.atia ucf. edu, sihong. @uta.cdu, wang @ @ucf.edu>>.
These challenges are critical in practical applications, where ineffective policies may have severe consequences, such as mechanical damage in robotics traffic accidents in au- tonomous driving. Existing TL methods, such as domain randomization (DR) imitation learning, lack systematic guarantees for transferred performance, as they fail to es- tablish a robust connection between the proxy and target domain outcomes. address these limitations, and inspired by the effectiveness of pessimism principle in various RL settings, particularly in scenarios where conservativeness is
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
advantageous, such as offline RL (Jin et al., 2020; Shi et al., 2022), we propose novel framework that incorporates the pessimism principle in proxy construction for TL. By using pessimistic estimation of target domain performance, our approach ensures that transferred policies achieve satisfac- tory outcomes while avoiding severe consequences from overly optimistic actions. Our framework also mitigates negative transfer in multi-domain settings. We rigorously characterize the effectiveness of this approach and design concrete algorithms with convergence guarantees. Our con- tributions are summarized as follows.
agent selects an action a' state s transitions to the next state s according the transition probability Pa and receives a reward r(s', a').
A stationary policy π S - A(A) maps each state to a probability distribution over A, and r(a|s) gives the probability of selecting action a at state 8. The perfor- mance of the agent following a policy is measured by the expected cumulative reward, defined as the value func- tion VB of policy ι with transition kernel P: Vā(s) & Ep - (iron"""s""a))) (((s",a')(s" a²) = s, =, for all E S. Alter- natively, the cumulative reward can also be characterized by the Q-function Qp for all (s, a) E S x A defined as p A Er Eizon"n(s ,at) = s, a a, aT
Introducing pessimism principle in transfer learning. We introduce the pessimism principle to transfer learning, enabling the transfer of conservative yet effective policies to the target domain. We characterize the framework's 's ef- fectiveness, showing that the target domain's 's optimality gap depends on the degree of pessimism used in proxy construc- tion. This motivates the design of proxies with minimal but sufficient pessimism, ensuring robust lower bounds on trans- ferred performance and avoiding the severe consequences common in other TI. methods.
The goal of an MDP is to learn the optimal policy n²:
π arg max (8),), E Well
which exists and can be restricted to the set of deterministic policies (Puterman, 2014). The optimal value functions are Vp - max, VB = VV and Qp - max, Qp- = Q7"
Construction of pessimistic proxy and design of concrete algorithms. Leveraging the inherent pessimism in robust RL frameworks, we construct pessimistic proxies by incor- porating prior knowledge of domain similarities, such as upper bounds on the distance between source and target domains. We develop several proxy types, demonstrate their effectiveness, and propose a distributed algorithm for multi-domand transfer. This algorithm ensures convergence and yields conservative but effective policies across diverse source domains.
Robust Markov Decision Process (RMDP). Robust RL aims optimize the performance under the worst case, when the environment is uncertain, and it is generally for- mulated as robust MDP. The transition kernel in a robust MDP is not fixed but lies in an uncertainty set P. In this pa- per, we focus on (s, -)rreananullar uncertainty sets (Nilim & El Ghaoui, 2004; Iyengar, 2005). Given a nominal kernel Po. the uncertainty set P is defined as P A @(s.a)ESxAPP PP. where Pa = {P* A(S) D(P". (Po)?) S r. Here, ((,,)) is distance metric between two probability distribu- tions, and r is the radius of the uncertainty set, measuring the level of environmental uncertainty.
Design of proxy and algorithm to avoid negative transfer. In multi-domain transfer, some source domains may signifi- cantly diverge from the target, leading to overly pessimistic policies if treated equally. We show that the pessimism prin- ciple inherently mitigates negative transfer by improving the proxy's 's value without overly optimistic assumptions. Building on this insight, we propose a refined proxy that avoids negative transfer and design a convergent algorithm to ensure effective policy transfer in such settings.
The robust value function and robust Q-function of a policy in a RMDP measure the worst-case performance over all transition kernels in the uncertainty set:
(*) min V$(s), (7) A min @7(s,a).
(1)
PEP
PEP
Then, the goal in the RMDP problem is to find the optimal robust policy '² that maximizes V/:: x² - arg max, VT. Similarly, we denote V.S" and Q by V and Qp-
2. Preliminaries and Problem Formulation
The robust value function 07 (and the optimal robust value function Qp. respectively) is the unique fixed point of the ro- bust Bellman operator T (and the optimal robust Bellman operator T, respectively) (Iyengar, 2005)?:
Markov Decision Process. A Markov Decision Process (MDP) serves as the standard framework for formulating RL. An MDP can be represented by tuple M = AS, 7,,,)), where S and A denote the state and action spaces, respec- tively, P S x AA- + A(S)' denotes the transition kernel, and we denote the probability vector under (s, a) -piir by Pa. r:
T (FQ(s, = r(s, + t0pp
Σ ((a))(()a)
(2)
- [0, 1] is the deterministic reward function, and 7 [0, 1] is the discount factor. At the 1-th step, the
aEA
TQ(s,a) = r(s,a) a) + Topy (max Q( a)),
(3)
aEA
(-)) denotes the probability simplex defined on the space.
Pap(V) = minpep PV the support function over P.
2

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
2.1. Problem Formulation
agents on randomized versions of the simulated domain (Tobin et al., 2017; Vuong et al., 2019; Mehta al., 2020; Tiboni et al., 2023; Tommasi, 2023). By introducing de- grees of variability in domain properties such as physics, or object appearances, the agent aims to optimize the ex- pected performance under a uniform distribution over these environments. DR is expected to enhance the robustness and generalizability of the learned policy so that performs well when deployed in the target domain. Although DR has proven effective in many tasks (Shakerimov et al., 2023; Niu et al., 2021; Slaoui et al., 2019; Jiang et al., 2023), it has limitations when the domain shift is large or the target en- vironment is significantly different from the simulated one, in which case the randomized environments do not provide enough information for the target domain. Furthermore, DR generally lacks theoretical guarantees and justifications for the performance in the target domain, except under some strict assumptions about the structures of the underlying MDPs (Chen et al., 2021; Jiang et al., 2022).
The goal of zero-shot multi-domain transfer RL is to optimize the performance under target MDP Mo r. 7), from which no data is available. Instead, there are K' related source domains M3 = (S,A, P:, generating much more data. For simplicity, we consider the case of identical reward. The only assumption is that, there exists an upper bound I' 2 D(P,, Pk), Vk, that is known by the learner. This assumption is reasonable, as such infor- mation is essential to achieve performance guarantee. And even if in the worst-case, we can set I = 1 (in total vari- ation) to construct an (overly) conservative proxy, which yet still avoids decisions with severe consequences in target domain, preferred in transfer learning settings.
In this work, we consider a more challenging setting of distributed, decentralized transfer learning with partial in- formation sharing. Specifically, we consider framework in which a central server or global agent collects and ag- gregates partial information (instead of all raw data) from multiple source domains and distributes updated results back to these domains. Each source domain then performs local learning independently based on the received updates and subsequently returns refined outcomes to the global agent.
Multi-task Learning: Multi-task learning (MTL) is another method that aims to leverage knowledge from multiple re- lated tasks to improve learning efficiency and generalization. Specifically, MTL aims to optimize a finite set of tasks si- multancously or to obtain Pareto stationary point (Yang et al., 2020; Wilson et al., 2007; Vithayathil Varghese & Mahmoud, 2020; Teh et al., 2017). When the target domain shares common properties with these source tasks, this in- formation will be maintained during the optimization of all source tasks and transferred. However, similar to DR, MTL also suffers from insufficient theoretical justification, requir- ing assumptions on the common representations of tasks. Moreover, MTL cannot avoid negative transfer, where some source tasks can be relatively distinct from the target task, thus slow down transfer learning. Also, solving an MTL problem can be difficult, due to the differences among the tasks known as the gradient conflict (Wang et al., 2024f).
A key aspect of this framework is its preeaccires nature. The underlying environments or raw data from the local domains are not directly shared with the central learner. Instead, only aggregated or processed results are exchanged, safeguarding the confidentiality of local data. Additionally, there is no direct communication between local domains, further ensuring data isolation and reducing the risk of information leakage.
This setting is both practical and relevant to many real- world applications where data privacy is a critical concern. For example, in a ride-sharing platform like Uber, each vehicle and its operational environment can be treated as an independent source domain. A central server may aim to optimize global dispatching or pricing policy for new target region, such as city where the platform is expanding. However, individual vehicles interact with their local enviromments independently and share only processed insights-such as aggregated trip statistics or anonymized performance metrics-wii with the server. This TL framework hence ensures effective policy transfer while maintaining the privacy of individual vehicle data.
Representation Learning for Multitask RL: Represen- tation learning aims to extract the shared structure across multiple tasks, and utilize such structure to reduce sam- ple complexity in downstream tasks with similar features (Teh et al., 2017: 7; Sodhani et al., 2021; Arulkumaran et al., 2022). Cheng et al. (2022); Agarwal et al. (2023); Sam et al. (2024) investigated multitask representation learning in an online RL setting, where the agent interacts with mul- tiple source tasks to extract common latent structure. Ish- faq et al. (2024) proposed Multitask Offline Representation Learning (MORL), which adopt the principle of pessimism and learn a shared representation from offline datasets across multiple tasks modeled by low-rank MDPs. However, these methods rely on the assumption that all tasks share low- rank latent structure and that common representation can be learned and reused, which may not hold in practical zero-shot transfer scenarios with large domain shifts.
3. Related Work
We briefly discuss the most commonly used transfer learning approaches (see Appendix A for additional discussions).
Domain Randomization: Domain randomization (DR) is widely used to reduce the Sim-to-Real Gap, by training
3

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
4. Pessimism Principle for Transfer Learning
Lemma 4.1. (Effectiveness of Pessimism for Transfer Learn- ing) Denote the level of pessimism of proxy f((x) by
4.1. Major Barriers in Prior TL Methods
5" _ V7 -(x)
Before developing our pessimism principle for transfer learning, we first identify two fundamental limitations in existing TL methods:
Then, the transferred policy T - max, (TT has the following sub- sub-optimality gap under the target environment:
(1) the lack of clear connection between the optimization proxy and the target domain performance, and (2) the inability to identify and exclude source information that may cause negative transfer.
VI.
Vig S |||1| A max
(4)
As discussed, most existing TL approaches optimize a prox- imal objective function / (T) and deploy its optimal policy in the target domain. However, there are limited guarantees regarding the relationship between the proxy f(r) and the target domain value function V except under some strong assumptions (e.g., (Chen et al., 2021)). Consequently, it is unclear whether the optimal policy for (T) will also per- form well under Po. leaving uncertainty in its performance. Furthermore, when multiple source domains are available, existing methods struggle to distinguish domains that are more dissimilar to the target, often treating all domains equally. This inability to prioritize relevant sources can lead to negative transfer, where the knowledge transferred from additional source domains hurt the performance of the target domain instead of improving it.
The result demonstrates that optimizing a pessimistic proxy guarantees the target domain performance in the worst-case scenario, thereby avoiding undesired decisions-a feat that is unattainable with prior transfer learning methods. More importantly, the result establishes a monotonic dependence between the sub-optimality gap and the level of pessimism: as the level of pessimism I|<* decreases, the transferred policy achieves smaller performance gap. This observation highlights key advantage of the pessimism principle: an automatic improvement from an enhanced proxy. As long as the proxy remains conservative, higher proxy value directly translates into better performance, eliminating the risk of overestimation as with previous methods.
Remark 4.2. As detailed in the methods presented later, |(_ can be controlled via the construction of local un- certainty sets. Consequently, the suboptimality gap can be bounded in terms of domain similarities, which is rea- sonable and subject to the problem nature. Moreover, if additional information becomes available (e.g., small set of target domain data or allowance of explorations), the radii of the uncertainty sets constructed can be tightened, leading to improved transfer performance.
As an example, in DR, the objective proxy is typically the average performance (w--Unif(s)) [VP.]] over an index set Q. As mentioned, this proxy often fails to accurately re- flect the target domain's performance in general settings. It may overestimate the target performance, resulting in an overly optimistic policy. Deploying such policy in the target domain can lead to suboptimal or even harmful outcomes. Additionally, the uniform distribution used in domain randomization cannot effectively exclude harmful source domains, increasing the risk of negative transfer.
In the following, we develop a robust RL based framework to provide concrete answers for the critical questions:
How can construct effective conservative proxies for pessimistic transfer, and how can we further improve the effectiveness by enhancing proxy values?
4.2. Pessimism Principle for Transfer Learning
Identifying these two limitations of prior works, we aim to develop more informative proxy that satisfies the following objectives: (1). It establishes concrete connections to the target domain performance, ensuring that optimizing the proxy guarantees the target performance. (2). It ensures the resulting policy remains conservative, thereby avoiding undesired decisions and their associated consequences.
4.3. A Robust RL Based Pessimism Principle
Robust RL is inherently pessimistic when tackling envi- ronment uncertainty. Specifically, when the environment of interest lies within the defined uncertainty set, the ro- bust value function naturally provides a conservative lower bound on performance. This property inspires our studies of robust RL transfer learning, as constructing an uncer- tainty set that includes the target domain is not challenging, and thus we obtain conservative proxy. Generally, the learner has some prior knowledge about the domain similarities to motivate transfer learning, and such similar- ities are usually captured in terms of the source and target domain kernels. For instance, a common characterization is the distance/divergence between the source and target
To achieve this, we propose the pessimism principle frame- work, where construct conservative proxy (T) X Vø. VT, then we transfer its optimal policy sy arg max; f (T). By optimizing this conservative proxy, we guarantee an optimized lower bound on the target domain performance, ensuring strong performance while mitigat- ing potential negative outcomes. We then characterize its effectiveness as follows (proof deferred to Appendix 4.)..
4

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
kernels, e.g., (Qu et al., 2024). Based on this information, an uncertainty set can be constructed to include the target domain kernel, and result in its robust value function as conservative proxy.
particularly valuable in safety-critical or high-stakes appli- cations.
5. Averaged Operator Based Proxy
In the case of multi-domain transfer, we similarly assume that the distance between any source domain Px, k > and the target domain Po is upper bounded by known I: (Po, Pt) S r.
As discussed, while robust RL-based conservative proxies are straightforward to construct, it is crucial to develop ones that can be efficiently optimized in a distributed setting, where each source domain avoids sharing its underlying MDPs or raw data. Constructing proxies that rely on the un- derlying environments, as discussed earlier, may be imprac- tical due to data privacy concerns in distributed frameworks.
Remark 4.3. Generally, such knowledge can be obtained through domain experts, or estimated from a small amount of target data. Moreover, our method remains applicable even without prior knowledge, by setting I to a known up- per bound on distributional distance (e.g., total variation between any two distributions is at most 1). While this yields over-conservative proxy, it still helps prevent sig- nificant drops when no similarity information is available a guarantee that DR methods do not offer.
Inspired by distributed and federated learning (Wang et al., 2024a), we propose an operator- proxy that enables the design of distributed algorithms where each local domain shares only updated Q-tables instead of raw data. Specifi- cally, we introduce averaged operator- basee proxy, which extends the design of vanilla federated learning to our pes- simism transfer learning framework.
For each source domain Pk. we can construct a local uncer- tainty set PB x,, (P&)), with
(P&) E E A(S): (Px))) <
5.1. Averaged Operator Based proxy
where I 2 P. In this way, Po E Pa and each ro- bust value function is a conservative proxy: VF, I VØ, Vk E K, V.. It is then of great importance to uti- lize these local uncertainty sets and robust value functions to construct conservative proxies with high value, to able effective transfer learning. Several potential straight- forward constructions are as follows:
By definition (2), we denote the robust Bellman operators for the k-th source domain by T Based on these, we construct an averaged operator and our conservative proxy.
Lemma 5.1. Define the averaged robust Bellmun operator as T a) = * Ek (T))s, Then, Too has a unique fixed point QAo
(1). Robust DR:
V= Σズ VE, (2). Proximal robust DR: VB. where
We use its fixed point to construct a conservative proxy, named the averaged operator based proxy for any given pol- icy m: (s) EnEA ग (aass)QRo(s this proxy is indeed conservative.
K
(P): {GE A(S): (()) D (g, Pg [P(P)) < <T (3). Maximal robust value function: max; V (4). Robust value function of the intersected uncertainty set (Px))
We first show
Theorem 5.2. (Pessimism of the proxy) The averaged oper- ator based proxy is conservative: Văo S V VT.
can be shown that all of these proxies are conservative estimations of V however, none of these proxies can be efficiently solved in a distributed setting. Specifically, (ro- bust) DR is generally intractable and is approximated by proximal DR objective (Jin et al., 2022). Meanwhile, soly- ing other proxies may require sharing local source domain models or data, which is often impractical or undesirable in a distributed setting. Therefore, we propose two conserva- tive proxies based on robust RL, and show that they can be efficiently optimized under the distributed setting.
The result indicates that the averaged operator based proxy effectively adopts the pessimism principle and provide a lower bound for the target domain performance.
Remark 5.3. The construction of the proxy is not direct extension of DR or multi-task RL. As mentioned, the DR method optimizes VB under the averaged environment P
ΣK, Ps, In the robust setting, its direct extension becomes VØ. the proximal robust DR proxy in Section 4.3. Due to the non-linearity of robust value functions w.r.t. the nominal kernel, VT does not coincide with the fixed point of the averaged robust Bellman operator, as in the non-robust case. However, we can show that our averaged operator based proxy is more effective than the robust DR.
Remark 4,4. Note that in certain cases, such as when the prior knowledge used to constructed the uncertainty set is inaccurate, our method may lead to conservative solutions that could be outperformed by DR. Nonetheless, our main contribution is to provide robust performance across all scenarios. Considering the inherent necessaries of conser- vativeness of robust transfer learning, such guarantee is
We further provide the following result, indicating that for the two most commonly used distance measures, Vğo is more effective than proximal robust DR.
We assume universal bound I' for simplicity, but our methods can directly extended to the case with different similarities Tx.
5

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Proposition 5.4. If uncertainty sets are defined through total variation or Wasserstein distance', then V I VAO- Remark 5.5. Notice that according the proof of Propo- sition 5.4, the result holds provided that o:: K ( S O. In Addition, for ly-norm, the support function has a duality: (VV) max, (PV. + f(r, Va)3 for some function f (Clavier et al., 2024). Thus
domains are more free to explore. We hence assume that each local source domain obtain unbiased estimate of its robust Bellman operator: for any vector Q, an unbiased estimate T&Q is available to each local agent. This is a mild assumption, as there are different ways to construct an unbiased estimator of the robust Bellman operator, e.g.. (Wang et al., 2023c; Kumar et al., 2023b; Yang et al., 2023; Liu et al., 2022). We also note that due to the linear structure of To, its unbiased estimation can be directly constructed as the average of all local estimations as TAo Σx= Te
1
O::-- - #R Ek= (Pig (V) = max, I + f(I, Va)} - R E k= max, (P&Vo + f(T, Va)3 - O, and the similar
result still holds.
On the other hand, to further avoid extensive communica- tion, the frequency of aggregation can be reduced. Specifi- cally, we update the local estimation of each source domain individually for E steps, and then aggregate all the domains once. Such a learning scheme is also used in federated or distributed learning, e.g., (Wang et al., 2024a; Jin al., 2022; Wang et al., 2024c; Woo et al., 2023).
The result shows that for the most widely used distance measures, the averaged operator based proxy is less conser- vative than the proximal robust DR, hence is more effective and is more likely to result in a better policy, due to the monotonic improvement property. It is then of interest to design a concrete algorithm optimize it, i.e., obtain the policy AO A arg max, VĀO-
We then present our Averaged Operator based multi multi-domain transfer (MDTL-Avg) algorithm as follows. The algorithm consists of two parts: local update to the local optimum, and global aggregation. Each local agent updates its own Q table with its data, and after E' steps, a global aggregation unifies all local Q tables. The convergence proof can be similarly decomposed in to the local (controlled by the local Bellman operator) and the global parts (controlled by aggregation). As mentioned, our algorithm does not require the knowledge
5.2. Efficient and Distributed Algorithm Design
Although TA is a contraction and Voo can be estimated exponentially fast, it can be impractical to estimate and compare Văo over all policies to find the AO-optimal pol- icy TAO = arg max, Voo. To efficiently optimize it, we construct an averaged optimal Bellman operator as follows, and develop fundamental characterizations for it and Ao-
Theorem 5.6. Recall T being the optimal robust Bellman operator for the k-th source domain as Equation (3), Define the averaged optimal Bellman operator as
Algorithm 1 MDTTL-Avg and MDTLL-Max Algorithms
1: Initialization: 0 < Qx S T vk = 1...,, K
2: for t= 0,..,, T-,,,-111 7 do
TsoQ(s,
A 1 K
Σ
K
a))
(5)
4: Qx(s,0) + (1 - (--A))QQssss + (Tx((u)(s.)).
3: for k = 1,.... K do
(s, E x xSx
/Local Update/
Then: (1). TAo is a contraction and has a unique fixed
end for
point QAo: (2). Let (s) e arg maNaeA Q a), then T. = arg max, VλO-
6:
if ( 0(mod E) then
7:
for s E S, A
8:
For MDTL-AVq:
The results imply that the optimal policy tAO can be ob- tained from the unique fixed point of the averaged optimal
9:
Q(s,a) + k Ex= (x(s,a)
10:
For MDTL-Max:
Bellman operator, hence it suffices to learn QAo by recur-
11:
Q(s,a) + Max-Aggregation of (Qu(s,a)
sively applying Too More importantly, due to the average structure over every source domain, the global learner does not require knowledge of each source domain, but only the update from each source domain, and hence can be trained in distributed and efficient scheme. Specifically, each lo- cal source domain maintains local Q-table to capture its Bellman operator, and sends the updated table to the global learner. After an average aggregation over all domains, a global Q-table is then sent back to each local domain.
12:
Q+(s. + Q(s, /synchronize/
13:
end for
14: end if
15:
end for
of each source domain, but only an unbiased estimator of the updated Q-table, which is more applicable under large- scale environments and better preserve privacy. Moreover, we reduce the aggregation frequency by E times, greatly enhancing the communication efficiency.
In practice, the exact source domains may be unknown, but extensive samples from them are available, as the source
We then develop the convergence analysis of Algorithm Theorem 5.7. Let E. S min } ( }. and X
*See Appendix D.3 for definitions.
6

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
(ogg((T)) Run Algorithm 1 for T steps. If [T = Tg. T(I T(--))
we define and characterize the minimal pessimism proxy. Theorem 6.1. The following results hold:
then it holds that
Σ k=l Qx K
1 TK T
(E-1)r)
(E-1))
1)r (6)
(1) Both Tup and TMP are "-contractions, and have unique fixed points. Qup and QMP, respectively.
E
QAO
Sō
(2) For any policy a. let (s)
EnEA ((p)s) a), then V S Moreover, VĀO S MMP
VMP S VF.
As the result shows, our algorithm enjoys partial linear speedup, implying the efficiency of our algorithm for multi- domain transfer. The second term measures the trade-off be- tween the convergence rate and communication cost, which is also observed in federated learning (FL) with heteroge- neous environments (Wang et al., 2024a;c). We note that the convergence rate of our stochastic algorithm still en- joys partial linear speedup, which matches the minimax optimal robust up to polylog factors (Vershynin, 201 20. It hence illustrates the efficiency of our algorithm, in terms of convergence rate and communication cost.
(3) Let TT arg max QMr(s, a). then QmP Vhi and arg max, 0MP
Similar to the previous section, our construction is based on a minimal pessimism principle operator, which is shown to be a contraction and admits a unique fixed point. Moreover, our results imply that the proxy constructed is conservative, and more importantly, it is better than the robust value func- tion of any source domain and the averaged operator based proxy. This hence avoids negative transfer by ruling out the more conservative source domains. As discussed, this less conservative proxy can result in a better estimate and is more likely to yield a better transferred policy.
Our algorithm design and convergence result thus provide the first effective, efficient multi-domain transfer method with conservative lower-bound guarantee.
The third result implies a practical approach to optimize Qhr. through obtaining the fixed point QMP of the optimal operator TMP- As it is contraction, recursively applying it will converge to Qhr Similarly, considering the partial in- formation from each source domain, we design distributed algorithm to achieve the fixed point as Algorithm 1.
6. Avoiding Negative Transfer: the Minimal Pessimism Principle
While our averaged operator-based method provides con- servative yet effective proxy for target domain performance, it remains vulnerable to a key challenge in multi-domain transfer: negative transfer. Similar to existing TL methods, the averaged operator-based proxy treats all source domains equally and cannot identify or down-weight the less relevant ones. Inspired by recent advances in personalized FL (Ari- vazhagan et al., 2019; Fallah et al., 2020; Deng al., 2020; Tan al., 2022), the global learner should assign higher weights to source domains that are more similar to the tar- get domain. However, identifying these similar domains without additional information remains challenging.
Notably, in MDTL-Max as Algorithm 1, the aggregation obtained through some function dax-gggregation. When each local Bellman operator is exact, i.e., Tx Tr, we sim- ply set Max-Aggregation of {Q+) to be max (Qx), and the resulting algorithm will converge to QMP (see The- orem E.2). However, when each local operator is stochas- tic, such an aggregation may result in biased estimation of TMP Specifically, even if Tr are unbiased estimations, the straightforward application of the maximum function results in a biased estimation: E[maake Qxl # max [[Qx]-
Our pessimism principle offers a solution to this problem. As shown in Lemma 4.1, a less conservative proxy leads to better-performing policy. Intuitively, more distinct source domains yield smaller robust value functions, allowing to assign higher weights to domains with higher robust val- ues to prioritize the more relevant ones. A straightforward proxy would be maxsex V which uses the maximum robust value function as the proxy. However, this approach may not be computationally efficient in a distributed setting. In this section, we propose refined proxy that addresses these limitations, achieving both higher effectiveness and computational efficiency.
To address this issue, it is essential to design alterna- tive maximal aggregation scheme that ensures convergence despite the presence of estimation errors. We propose an ag- gregation strategy to overcome this challenge. Specifically, we adopt the multi-level Monte-Carlo method (Blanchet Glynn, 2015; Blanchet al., 2019; Wang & Wang, 2022) to construct an operator MMLMC of the maximal aggregation based on estimated local operators. We defer the detailed construction to Appendix E.
Lemma 6.2. E|M (f (Q)) (. = maxMex E[Qa]
We define the minimal pessimism operator and the optimal minimal pessimism operator as ThnQ(s. a)
We hence constructed an unbiased estimation of the global minimal pessimism operator, which can then be adopted to implement MDTL--Max as Algorithm 1.
max#ek TEQ a), and T (s. A maxtex (s) based on which
Remark 6.3. The MLMC module for MDTLL-Max will in-
7

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
crease the sample/computation complexity, which can be viewed the price of improving effectiveness. Never- theless, it can be reduced along two potential directions: One is to control the level number through techniques like threshold-MIMCC (Wang et al., 2024g). Although it results in a biased estimation, the bias can be controlled and hence still implies convergence (see Appendix B.4). Another one to relax the uncertainty set constraint. For example, for total variation, the relaxation results in the solution PoV - -VSpan(V) S (V) (Kumar et al., 2023a), which is much easier to compute and remains conservative. Re- ducing the aggregation frequency is an alternative way to reduce the computational cost, but also introduces a trade- off in the convergence rute, as we will show the following results.
conservative bounds for the robust value function with re- speet to some uncertainty set around Po. As result, the transferred policy is guaranteed to perform well despite po- tential uncertainties in the target domain, thereby enhancing its generalizability against continuous perturbations of the environment. Specifically, the policy will also perform ef- fectively in environments similar to Po- We further validate this robustness numerically in our experiments.
Scalability. Our proposed algorithms are scalable, enabling effective transfer in large-scale problems. Firstly, our al- gorithms can be implemented in a model-free manner to enhance computational and memory efficiency. In particular, any model-fre algorithm for robust RL can be integrated into the local update step in Algorithm 1. To demonstrate this, we design a model-free variant of Algorithm I in Ap- pendix G, accompanied by a convergence guarantee:
We can then incorporate such a multi-level Monte-Carlo construction into Algorithm 1 as an unbiased update step. We further characterize its convergence.
Theorem 7.2. (Informal) Denote the oulput of the algo- rithms with suitable step sizes X and total steps T by QT. Let AT = QAo - Q" and Dr == Qui Q" for MDTL- Avg and MDTL-Max, respectively. With probability at least
Theorem 6.4. Let E- I min Mi kb. X
4log (TK)) and adopt MLMC MLLMC-Mar-Aggregation Then after T(1-)) T steps,
- 6, we have
QMP - max E
Q KEK
( T
1 TK
(E- 1)T )
(7)
P ATISS 50
(E 1)1
7)3 (1 (1- (1-7)3VTK
log SATK
8
(8)
The result implies the convergence of our algorithm in a more practical setting. More importantly, our algorithm converges more effective proxy than the averaged one, which potentially results in better transferred policy due to our pessimism principle. Also, our algorithm enjoys nearly optimal convergence rate, similar to Algorithm 1.
Notably, while previous methods, such as DR, lack con- crete algorithms, our approach is both effective and scalable, offering significant advantage.
Secondly, our method is not restricted to tabular setting, and can be integrated with function approximation or policy gra- dient. For example, when combine function approximation for robust RL (Zhou et al., 2023) or robust policy gradi- ent (Wang & Zou, 2022), the global step becomes parameter aggregation as in (Jin al., 2022).
7. A Addiionna Discussion
In the previous sections, we demonstrated that our pes- simism principle-based framework offers two key advan- tages: (1) it guarantees a well-performing lower bound on the target domain performance, avoiding severe undesired outcomes, and (2) it establishes a monotonic relationship between the pessimism level and transfer effectiveness, al- lowing us to improve transfer performance and mitigate negative transfer. In this section, we highlight two addi- tional advantages: improved robustness and scalability.
8. Experimental Results
In this section, we aim to verify the effectiveness of our pessimism principle, and its ability to avoid negative transfer. More details and additional experiments results are provided in Appendix B.
8.1. Effecciveness of Pessimism Principle for Transfer Learning
Robustness. By optimizing conservative estimation of the target domain's 's performance, our framework inherently enhances the robustness of the transferred policy: it ensures that the policy performs well even under model uncertainties within the target domain, such as -tattiaionnay parameters. We show the following result to justify its robustness:
We develop our simulation on the recycling robot problem (Sutton & Barto, 2018; Wang et al., 2023c). In this problem, mobile robot powered by a rechargeable battery is tasked with collecting empty soda cans. The robot operates with two battery levels: low and high. It has two possible actions: (1) search for empty cans; (2) remain stationary and wait for someone to bring it a can. When the robot's 's battery is low (high), it has a probability of o(3) of finding an empty
Proposition 7.1. There exists some connected uncertuinty set P such that Po E P, and VT, > Vz Z V > Văo.
This result implies that the proxies we construct are also
8

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
can and maintaining its current battery level, receiving some high reward. If the robot searches but does not find any, it will deplete its battery completely and receive large penalty. If the robot chooses to wait, it will remain at the same battery level and receive a relatively small reward.
date whether MDTL-Max can effectively mitigate negative transfer. The FrozenLake Gym environment provides an explicitly known transition kernel. This allows us to pre- cisely control the distance between source domains and the target domain, thereby offering more rigorous and inter- pretable results. In this example, we manually perturb the agent using total variation distances from the default model: D [0.01, 0.02, 0.3]. The source domain with D = 0.3 is intentionally designed to represent a domain that dif- fers significantly to the target, potentially causing negative transfer. For each algorithm, we run experiments across 10 random seeds. For each seed, the policy is evaluated over 10 independent episodes to compute the average return. As shown in Figure 2, MDTL-Max effectively leverages the most informative source domain, and hence avoids negative transfer.
In our experiments, we first compare our methods with the non-robust corresponding methods, including proximal non-robust DR and maximal aggregation algorithm (see Algorithm 2). We run 5 trials with different random seeds for each method and report means and standard deviations of values over training steps in Figure 1. We use a constant step size- = 0.1, and total training steps T = 5000. We set the target environment as a = 3 = 0.1, pre-specified radius of the uncertainty set R = 0.8. Then, we randomly generate K = 7 distinct source domains with ag. Byx E [0.85, 0.9] to construct the uncertainty set. Other parameters E 1, and y = 0.95. As the results show in Figure 1 I-left, both of the non-robust baselines achieve a low reward in the target domain, as they are overly optimistic and decide to search under both battery levels, whereas our pessimism principle methods tend to wait conservatively, avoiding undesired decisions and severe consequences.
bemann Performace
2.
Beger Demain Neformnnce
Target Domens Performance will Model Unceraaity
Time Step
Figure 2. Negative Transfer under FrozenLake Gym environment
1B% Aag
9. Conclusion
Figure 1. Recycling Robot Problem
In this paper, we studied zero-shot transfer reinforcement learning and identified two critical limitations of cxisting methods: the lack of guarantees for the safety and per- formance of transferred policies and the inability miti- gate negative transfer when multiple source domains involved. To overcome these challenges, we incorporate pessimism principle into TL to conservatively estimate the target domain's performance, and mitigate risk of un- desired decisions. Our framework establishes a monotonic dependence between the level of pessimism and target per- formance, effectively addressing the two identified issues by constructing less conservative estimations. We proposed and analyzed two types of conservative estimations, rigorously characterizing their effectiveness, and developed distributed, convergent algorithms to optimize them. These methods are well-suited for distributed transfer learning settings, offering performance guarantees and privacy protection. Our frame- work represents a foundational step toward zero-shot trans- fer RL with theoretical performance guarantees, providing robust and practical solution for real-world applications.
To illustrate the robustness and generalizability of our meth- ods against target model uncertainty, we evaluate the robust value functions of the learned policies against an uncertainty set centered at the target domain, which represents the worst- case performance under the model uncertainty. Figure 1- right shows that both of our pessimistic methods outperform the non-robust baseline, indicating that our approach can maintain a high performance, even with uncertainty or per- turbations in the target domain. We also conduct an ablation study to compare the robust value functions under different target domain uncertainty levels in Appendix B.I, to further illustrate the enhanced robustness of our approaches.
In summary, our method outperforms non-robust baselines, achieving 195.03% performance improvement in the tar- get domain and maintaining a 15.95% % improvement even under uncertainty. The experiment results hence verify that pessimism principle is more effective, especially when the source domains are relatively distinct from the target do- main, or when model uncertainty exists in the target domain.
8.2. Experiments on Negative Transfer
A Acknowledgment
We further tested our MDTL-Avg and MDTL-Max algo- rithms on Frozenl Lake Gym environments, aiming to vali-
This work was supported by DARPA under Agreement No. HIR0011-24-9-044 and NSF under Award CCF-2106339.

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
The authors thank the anonymous reviewers whose con- structive comments led to substantial improvement to the paper.
state simulation, quantiles, and other applications. arXiv 2019.
preprint
Chen, X., Hu, J., Jin, C., Li, I.., and Wang, L. Understanding
domain randomization for sim-to-real transfer. arXiv preprint arXiv:21 2021.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Cheng, Y., Feng. S., Yang. J., Zhang, H., and Liang,
Provable benefit of multitask representation learning in reinforcement learning. Advances in Neural Information Processing Systems, 15:31741-31755 2022.
References
Clavier, P., Shi, L., Le Pennec, E., Mazumdar, E., Wierman, A., and Geist, M. Near-optimal distributionally robust re- inforcement learning with general I-p norms. Advances in Neural Information Prucessing Systems, 37:1750-1810, 2024.
Agarwal, A., Song, Y., Sun, W., Wang, K., Wang, M., and Zhang, X. Provable benefits of representational transfer in reinforcement learning. In The Thirty Sixth Annual Conference on Learning Theory, pp. 2114-21888 PMLR, 2023.
Deng, Y., Kamani, M. M., and Mahdavi, M. Adap- tive personalized federated learning. arXiv preprint arXiv:2003.1 134611 2020.
Arivazhagan, M. G., Aggarwal, V., Singh, A. K., and Choud- hary, S. Federated learning with personalization layers. arXiv preprint arXiv: 191,00088,, 2019.
Derman, E., Geist, M., and Mannor, S. Twice regularized MDPs and the equivalence between robustness and reg- ularization. Proc. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Anllkumaran, K., Ashley, D. R., Schmidhuber, J., and Sri- vastava, R. K. All you need is supervised learning: From imitation learning to meta-rl with upside down rl. arXiv preprint arXiv:2202. /960, 2022.
Desai, S., Durugkar, I., Karnan, H., Warnell, G., Hanna, J., and Stone, P. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33:3917-3929, 2020.
Badrinath, K. P. and Kalathil, D. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In Proc. International Confer- ence on Machine Learning (ICML), pp. 511-520. PMLR, 2021.
Dong, J., Li. J., Wang, B., and Zhang, J. Online policy optimization for robust mdp. arXiv preprint 2rXiv:2209.1334 13841, 2022.
Benjamins, C., Eimer, T., Schubert, F., Mohan, A., Döhler, S., Biedenkapp. A., Rosenhahn, B., Hutter, F., and Lin- dauer, M. Contextualize me-the case for context in inforcement learning, arXiv preprint arXiv:2202.04455 2022.
Fallah, A., Mokhtari, A., and Ozdaglar, A. Personalized federated learning: A meta-learning approach. arXiv preprint arXiv:2002.079455 2020.
Beukman, M., Jarvis, D., Klein, R., James, S., and Rosman, B. Dynamics generalisation in reinforcement learning via adaptive context-aware policies. Advances in Neural Information Processing Systems, 36, 2024.
Hallak, A., Di Castro, D., and Mannor, S. Con- textual markov decision processes. arXiv preprint 2015.
He, S., Pepin, L., Wang, G., Zhang, D., and Miao, F. Data- driven distributionally robust electric vehicle balancing for mobility on-demand systems under demand and sup- ply uncertainties. In 2020 IEEE/RS International Con- ference on Intelligent Robots and Systems (IROS), pp. 2165-2172. IEEE, 2020.
Biedenkapp, A., Bozkurt, H. F., Eimer, T., Hutter, F,, and Lindauer, M. Dynamic algorithm configuration: Foun- dation of new meta-algori framework. In ECAI 2020, pp. 434. IOS Press, 2020.
Blanchet, J. and Glynn, P. W. Unbiased Monte Carlo for optimization and functions of expectations via multi-level randomization. In 2015 Winter Simulation Conference (WSC), pp. 3656-3667. IEEE, 2015.
He, S., Han, S., and Miao, F. Robust electric vehicle bal- ancing of autonomous mobility-on-deman system: A multi-agent reinforcement learning approach. 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5471-5478. IEEF, 2023a.
Blanchet, J. H., Glynn, P. W. and Pei. Y. Unbiased mul- tilevel Monte Carlo: Stochastic optimization, steady-
10

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
He, S., Wang, Y., Han, S., Zou, S., and Miao, F. A robust and constrained multi-agent reinforcement learning electric vehicle rebalancing method in amod systems. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5637-5644, IEEE, 2023b.
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Ham- bro, E., Grefenstette, E., and Raileanu, R. Understanding the effects of rlhf on Ilm generalisation and diversity. arXiv preprint arXiv:2310.0643 2023a.
Kirk, R., Zhang, A., Grefenstette, E., and Rocktäschel, T. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 201-264, 2023b.
Hua, J., Zeng, L., Li, G., and Ju, Z. Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning. Sensors, 21 (4): 1278, 2021.
Ishfaq, H., Beauyen-Tang, T., Feng, S., Arora, R., Wang, M., Yin, M., and Precup. D. Offline multitask representation learning for reinforcement learning. Advances in Neural Information Processing Systems, 77:70557-70616 2024.
Koppejan, R. and Whiteson, S. Neuroevolutionary rein- forcement learning for generalized helicopter control. In Proceedings of the 11th Annual conference on Genetic and evolutionary computation, pp. 145-1455555 152, 2009.
Iyengar, G. N. Robust dynamic programming Mathematics of Operations Research, 30(2):257-280, 2005.
Kumar, N., Derman, E.,, Geist, M., Levy, K. Y., and Mannor, S. Policy gradient for rectangular robust markov decision processes. Advances in Neural Information Processing Systems, 36:59477-59501 2023a.
Jia, Z., Dong, Z., Yin, M., and He, S. How robust rein- forcement learning enables courier-friendly route plan- ning for last-mile delivery? In ICML 2025 Workshop on Programmatic Representations for Agent Learning, 2025. URL https: http:///pppeeeeeeeeeeeeeeee netie/eweeeeeeeeeeeeeeeeeeeee forum? (d=OznWjA6XTK
Kumar, N., Levy, K., Wang. K., and Mannor, S. An ef- ficient solution to s-rectangular robust markov decision processes. arXiv preprint arXiv:2301.13642 2023b.
Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., and Dresp- Langley, B. Deep reinforcement learning for the con- trol of robotic manipulation: focussed mini-review. Robotics, 10(1):22, 2021.
Jiang, Y., Li, C., Dai, W. Zou, J., and Xiong, H. Vari- ance reduced domain randomization for policy gradient, 2022. URL
net/ forum?
id=vnF5gDNvcke
Liu, X.-Y,, Zhou, X.-H,, Li, G.-T,, Li, H., Gui, M.-.,, Xiang, T.-Y., Huang, D.-X., and Hou, Z.-G. Micro: Model-based offline reinforcement learning with a conservative bell- man operator. arXiv preprint arXiv:2312.0399) 2023.
Jiang, Y., Li, C., Dai, W., Zou, J., and Xiong, H. Variance reduced domain randomization for reinforcement learn- ing with policy gradient. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Liu, Z. and Xu, P. Distributionally robust off-dynamics rein- forcement learning: Provable efficiency with linear func- tion approximation. arXiv preprint arXiv:2402.15399, 2024.
Jin, H., Peng, Y., Yang, W., Wang. S., and Zhang, Z. Feder- ated reinforcement learning with environment heterogene- ity. In International Conference on A Artificial Intelligence and Statistics, pp. 18-37, PMLR, 2022.
Liu, Z., Bai, Q., Blanchet, J., Dong, P., Xu, W., Zhou, Z., and Zhou, Z. Distributionally robust Q-learning. In In- ternational Conference on Machine Learning pp. 13623- 13643. PMLR, 2022.
Jin, Y.. Yang, Z. and Wang, Z. Is pessimism provably efficient for offline RL? arXiv preprint arXiv:2012.15085 2020.
Kaddour, J.. Sæmunndsoon S., et al. Probabilistic active meta-learning. Advances in Neural Information Process- ing Systems, 1222022322022. 2020.
Lu, M., Zhong, H., Zhang, T., and Blanchet, J. Distribu- tionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithm. arXiv preprint arXiv:2404.03577 2024.
Khodadadian, S., Sharma, P., Joshi, G., and Maguluri, S. T. Federated reinforcement learning: Linear speedup under markovian sampling. In International Conference Machine Learning, pp. 10997-11057. PMLR, 2022.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and
y Arcas, B. A. Communication-efficceee learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1 PMLR, 2017.
Kim, H., Ohmura, Y., Nagakubo, A., and Kuniyoshi, Y. Training robots without robots: deep imitation learning for master-to-robor policy transfer. IEEE Robotics and Automation Letters, 8(5):2906-291 2023.
Mehta, B., Diaz, M., Golemo, F., Pal, C. J., and Paull, Active domain randomization. In Conference on Robot Learning, pp. 1 1162-1176. PMLR, 2020.

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Mnih, V., Kavukeuoglu, K., Silver, D., Graves, A., Antonoglou, I,, Wierstra, D., and Riedmiller, M. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:/312.5600, 2013.
Seo, Y., Lee, K., Clavera Gilaberte, I., Kurutach, Shin, J., and Abbeel, P. Trrjectory-wise multiple choice learning for dynamics generalization in reinforcement learning. Advances in Neural Information Processing Systems, 33: 12968-12979, 2020.
Modi, A., Jiang, N., Singh, S., and Tewari, A. Markov decision processes with continuous side information. In Algorithmic Learning Theory, pp. 597-618. PMLR, 2018.
Shakerimov, A., Alizadeh, T., and Varol, H. A. Efficient sim- to-real transfer in reinforcement learning through domain randomization and domain adaptation. IEEE Access, 11: 116809-136822 2023.
Nguyen, H. and La, H. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE international conference on robotic computing (IRC), pp. 590-595, IEEE, 2019.
Sharma, A. R. and Kaushik, Literature survey of sta- tistical, deep and reinforcement learning natural lan- guage processing. In 2017 International conference on computing, communication and automation (ICCCA), pp. 350-354. IEEE, 2017.
Nilim, A. and EI Ghaoui, L. Robustness in Markov decision problems with uncertain transition matrices. In Proc. Ad- vances in Neural Information Processing Systems (NIPS), pp. 839 846, 2004.
Shi, L., Li, G., Wei, Y., Chen, Y., and Chi, Y. Pessimistic Q Q-learning for offline reinforcement learning: Towards optimal sample complexity. International Conference on Machine Learning, pp. 19967-20025, 2022.
Niu, H., Hu, J., Cui, Z. and Zhang, Y. Dr21: Surfacing corner cases to robustify autonomous driving via domain randomization reinforcement learning. In Proceedings of the 5th International Conference on Computer Science and Application Engineering, pp. 1-8, 2021.
Shi, L., Li, G., Wei, Y., Chen, Y., Geist, M., and Chi, Y. The curious price of distributional robustness in reinforce- ment learning with a generative model. arXiv preprint arXiv:2305. 16589, 2023.
Panaganti, K. and Kalathil, D. Sample complexity of bust reinforcement learning with generative model. In International Conference Artificial Intelligence and Statistics, pp. 9582-9608 9602. PMLR, 2022.
Silver, D., Huang, A., Maddison, C. Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu. K., Graepel. T., and Hassabis, D. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587):484-489 2016.
Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. Robust reinforcement learning using offline data. arXiv preprint arXiv:2208.05// 2022.
Puterman, M. L. Markov decision processes: discrete
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., al. Mastering the game of Go without human knowledge. Nature, 550(7676):354 359, 2017.
stochustic dynamic programming. John Wiley & Sons, 2014.
Qu, C., Shi, L.,, Panaganti, K., You, P., and Wierman, A. Hybrid transfer reinforcement learning: Provable sample efficiency from shifted-dy -yymmicss data. arXiv preprint 1oXiv:241.008//1 2024.
Slaoui, R. B., Clements, W. R., Foerster, J. N., and Toth, Robust domain randomization for reinforcement learning. 2019.
Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., and Hadsell, R. Policy distillation. arXiv preprint arXiv: 1.002995, 2015.
Sodhani, S., Zhang, A., and Pineau, Multi-task rein- forcement learning with context-based representations. In International Conference on Machine Learning, pp. 9767-9779. PMLR, 2021.
Sun, Z., He, S., Miao, F,, and Zou, S. Policy optimization for robust average reward mdps. Advances in Neural Information Processing Systems, 37:17348-17372 2024.
Salvato, E., Fenu, G., Medvet, E., and Pellegrino, F. A. Crossing the reality gap: A survey on sim-to-real trans- ferability of robot controllers in reinforcement learning. IEEE Access, 9:153171-153189, 2021.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An
introduction. MIT press, 2018
Sam, T., Chen, Y., and Yu, C. L. The limits of transfer reinforcement learning with latent low-rank structure. Ad- vances in Neural Information Processing Systems, 37: 108262-108330, 2024.
Tan, A. Z., Yu, H., Cui, L.,, and Yang, Q. Towards person- alized federated learning. IEEE transactions on neural networks and learning systems, 34(12):9587-966 2022.
12

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Teh, Y., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
Wang, G., He, S., Jiang, L.,, Wang, S., Miao, F., Zhang, F,, Dong, Z.,, and Zhang, D. Fairmove: A data-driven vehicle displacement system for jointly optimizing profit efficiency and faimess of electric for-hire vehicles. IEEE Transactions on Mobile Computing, 23(6):6755-6802, 2023a.
Tessler, C., Efroni, Y., and Mannor, S. Action robust rein- forcement learning and applications continuous control. In International Conference on Machine Learning, pp- 6215-6224, PMLR, 2019.
Wang. H., He, S., Zhang, Z.,, Miao, F., and Anderson, J. Momentum for the win: Collaborative federated rein- forcement learning across heterogeneous environments. In International Conference on Machine Learning, pp. 50530-50560 PMLR, 2024a.
Tiboni. G., Klink, P., Peters, J., Tommasi, T., D' D'Eramo, C., and Chalvatzaki, G. Domain randomization via entropy maximization. arXiv preprint arXiv:2311.01888, 2023.
Wang. H., Shi, L., and Chi, Y. Sample complexity of offline distributionally robust linear markov decision processes. arXiv preprint arXiv:2403. 1946, 2024b.
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. Domain randomization for transferring deep neural networks from simulation the real world. In 2017 IEEE/RSJ international conference on intelligent robols and systems (IROS), pp. 23-30. IEEE, 2017.
Wang. M., Yang. P., and Su, L. On the convergence rates of federated q-learning across heterogeneous environments. arXiv preprint wwXiv:2409.03889 2024c.
Tommasi, T. Online vs. offline adaptive domain random- ization benchmark. In Human-Friendly Robotics 2022: HFR: 15th International Workshop on Human- Friennlyy Roboties, volume 26, pp. 158. Springer Nature, 2023.
Wang, Q.. Ho, C. P., and Petrik, M. Policy gradient in robust mdps with global convergence guarantee, 2023b.
Wang. Y. and Zou, S. Online robust reinforcement learning with model uncertainty. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 7193-72066 2021.
Traoré, R., Duselles-Dupré, H., Lesort, T., Sun, T., Cai, G., Díaz- Rodzigodrgg N., and Filliat, D. Discorl: Contin- ual reinforcement learning via policy distillation. arXiv preprint arXiv:1907.05880 2019a
Wang, Y. and Zou, S. Policy gradient method for robust re- inforcement learning. In Proc. International Conference on Machine Learning (ICML., volume 162, pp. 23484- 23526. PMLR, 2022.
Traoré, R., Caselles-Duppe H., Lesort, T., Sun, T., Díaz- Rodríguez, N., and Filliat, D. Continual reinforcement learning deployed in real-life using policy distillation and sim2real transfer. arXiv preprint 1006:::990.04444 2019b.
Wang. Y., Velasquez, A., Atia, G. K., ceater-Bennette,, A., and Zou, S. Model-free robust average-reward reinforce- ment learning. In International Conference on Machine Learning, pp. 36431-36469, PMLR, 2023c.
Uc-Cetina, V. Navarro-Guerero, N., Martin-Gonzaled A., Weber, C., and Wermter, S. Survey on reinforcement learning for language processing. Artificial Intelligence Review, 56(2):1543-1575, 2023.
Wang, Y., Sun, Z. and Zou, S. A unified principle of pessimism for offline reinforcement learning un- der model mismatch. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024d. URL //penrevie//eeeeee nnt/ forum? id-cBY66CKEbq,
Vershynin, R. High-dimensional probability: An introduc- fion with applications in data science, volume 47. Cam- bridge university press, 2018.
Vithayathil Varghese, N. and Mahmoud, Q. H. A survey of multi-task deep reinforcement learning. Electronics, 9()): 1363, 2020.
Wang. Y., Velasquez, A., Atia, G., Prater-Bennette, A., and Zou, S. Robust average-reward reinforcement learning. Journal of Artificial Intelligence Research, 80:719-800 193, 2024e.
Vuong, Q. Vikram, S., Su, H., Gao, S., and Christensen, H.
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? arXiv preprint arXiv: arXiv:/903.. 1990.117744 2019.
Wang, Y., Xiao, P., Ban, H. Ji. K., and Zou, S. Finite-time analysis for conflict-avoidee multi-task reinforcement learning. arXiv preprint arXiv:2405.1600 2024f.
Wang, G. and Wang. T. Unbiased multilevel Monte Carlo methods for intractable distributions: Mlme meets meme. arXiv preprint 2204 048008 2022.
Wang. Y., Zou, S., and Wang. Y. Model-free robust rein-
forcement learning with sample complexity analysis. In
13

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
The 40th Conference on Uncertainty in Artificial Intel- ligence, 2024g. URL htt.s: https/////ppeeeeeeeeeeeee Eorum?id-ban brrRwwsS5.....
Zhang, C., Farhat, Z. U., Atia, G. K., and Wang, Y. Model- free offline reinforcement learning with enhanced robust- ness. In The Thirteenth International Conference Learning Representations, 2025.
Wiesemann, W. Kuhn, D., and Sim, M. Distributionally robust convex optimization. Operations research, 62(6): 1358-1376, 2014.
Zhao, W., Queralta, J. P., and Westerlund, Sim-to-real transfer in deep reinforcement learning for robotics: survey, In 2020 IEEE symposium series on computational intelligence (SSCI), pp. 737-744. IEEE, 2020.
Wilson, A., Fern, A., Ray, S., and Tadepalli, P. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference Machine learning, pp. 1015-1022, 2007.
Zhou, R., Liu, T., Cheng, M., Kalathil, D., Kumar, P., and Tian, C. Natural actor-critic for robust reinforcement learning with function approximation. Advances in neural information processing systems, 36:97-133, 2023.
Woo, J., Joshi, G., and Chi, Y. The blessing of heterogene- ity in federated q-learning: Linear speedup and beyond. In International Conference on Machine Learning, pp- 37157-37216. PMLR, 2023.
Xu, Z., Panaganti, K., and Kalathil, D. Improved sample complexity bounds for distributionally robust reinforce- ment learning. In International Conference on Artificial Intelligence and Statistics, pp. 9728-9754 PMLR, 2023.
Yang. R., Xu, H., Wu, Y., and Wang, X. Multi-task rein- forcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777,, 2020.
Yang, R., Bai, Ma, X., Wang, Z., Zhang, C., and Han, L. Rorl: Robust offline reinforcement learning via con- servative smoothing. Advances neural information processing systems, 35:23851-23866 2022.
Yang. W., Zhang, L., and Zhang, Z. Towards theoreti- cal understandings of robust Markov decision processes: Sample complexity and asymptotics. arXiv preprini arXiv:2105.03863. 2021.
Yang, W., Wang. H., Kozuno, T., Jordan, S. M., and Zhang, Z. Robust markov decision processes without model estimation. arXiv preprint arXiv:2302.01248 2023.
Yim, J., Joo, D., Bae, J., and Kim, J. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4133- 4141, 2017.
Yin, H. and Pan, S. Knowledge transfer for deep reinforce- ment learning with hierarchical experience replay. In Proceedings of the AAA/ Conference Artificial Intelii- gence, volume 31, 2017.
Zare, M., Kebria, P. M., Khosravi, A., and Nahavandi, S. A survey of imitation learning: Algorithms, recent develop- ments, and challenges. IEEE Transactions on Cybernet- ics, 2024.
14

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
A. Additional Related Works
Imitation Learning and Policy Distillation: Imitation learning (IL) and policy distillation are extensively explored TL techniques that enable an agent to learn from demonstrations or from simpler policies. In IL, an agent mimics expert behavior through supervised learning, often using behavioral cloning or inverse RL (Hua et al., 2021; Desai al., 2020; Kim et al., 2023; Zare et al., 2024). Policy distillation transfers knowledge from a teacher policy to student policy by matching the teacher's action distribution (Rusu et al., 2015; Yim et al., 2017; Yin & Pan, 2017: Traoré et al., 2019b;a). The success of these approaches, however, rely on the assumption that the source domain's 's expert or teacher policy is highly aligned with the target domain, which may not always be the case.
Contextual Reinforcement Learning: In contextual reinforcement learning (cRL), the transition dynamics are assumed to be governed by an underlying context (Hallak et al., 2015; Modi et al., 2018). This context can represent physical properties-such as wind conditions (Koppejan & Whiteson, 2009) or the length of a pole in a balancing task (Seo et al., 2020; Kaddour et al., 2020; Benjamins et al., 2022) or more abstract features that characterize environment dynamics (Biedenkapp et al., 2020). Kirk et al. (2023b) identify cRI. as particularly relevant for studying zero-shot generalization in RL agents. The cRL framework allows for a systematic and principled analysis of how agents adapt to environmental changes by explicitly defining inter- and extrapolation distributions. Following the evaluation protocol introduced by Kirk et al. (2023b). Benjamins ct al. (2022) examined the generalization capabilities of various model-free RL agents on a benchmark that incorporates different physical properties as context information. Their approach assumed a naive utilization of context by directly concatenating with observations. In contrast, Beukman et al. (2024) proposed hyppemetwork-basee method to enable adaptable RL agents. However, these approaches still fail to address the fundamental challenges discussed earlier.
Federated Learning: FL is a decentralized RL approach that aims to optimize the overall performance of K agents (McMa- han et al., 2017). In this setting, each agent performs local updates based on its own environment and periodically communicates with a central server to aggregate the local models, without sharing raw trajectories with each other. While FL has been extensively studied (Jin et al., 2022: Wang et al., 2024c: Woo et al., 2023; Khodadadian al., 2022; Wang et al., 2024a), existing work focuses on optimizing the average performance across local environments, rather than addressing the more challenging problem of multi-domain transfer. A fundamental challenge in applying FL to transfer learning lies in the nature of update rules: standard FL methods rely on linear updates (non-robust operators), whereas our proposed methods involve non- non-linear robust updates account for uncertainties.
Robust RL; Robust RL (lyengar, 2005; Nilim & El Ghaoui, 2004) or DRO formulation (Wiesemann et al., 2014; He et al., 2020) offer a pessimism principle and a robust framework for addressing environmental uncertainty, which optimizes the worst-case performance within an uncertainty set of environments and can offer a theoretically sound lower bound on target domain performance when the target domain lies within this set. However, its application in transfer learning has been under explored, as robust RL often exhibits excessive pessimism, resulting in overly conservative policies for the target domain. Recent advances have extensively studied robust RL. as standalone problem (Wang & Zou, 2021; 2022; Wang et al., 2023b; Badrinath& & Kalathil, 2021; Dong et al., 2022; Lu et al., 2024; Liu & Xu, 2024; Yang et al., 2021; Xu et al., 2023; Panaganti& & Kalathil, 2022; Shi et al., 2023; Wang et al., 2024b; Panaganti et al., 2022; Yang et al., 2022; Liu et al., 2023; Wang et al., 2024d;e; Zhang et al., 2025; Sun et al., 2024), but its integration into transfer learning remains largely untapped. Leveraging robust RL principles in transfer learning offers an opportunity to develop methods that are not only theoretically grounded but also resilient uncertainties in target domains. This proposal seeks to explore research directions that harness the strengths of robust RL to create more effective, efficient, and reliable transfer learning frameworks.
B. Details on Experiments and Additional Experiments
All experiments are conducted on a MacBook Pro configured with an Apple M3 Pro chip, featuring a 11-core CPU and 18GB of unified memory, running macOS Sequoia 15.2. The experiments are performed using Python 3.12 in a Conda environment. Major libraries used include NumPy 1.2.1.
The baseline algorithms compared are presented as follows. Here, Ts is some unbiased estimation of the non-robust Bellman operator T(Q) rkk + P.Q.
15

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Algorithm 2 Proximal Non-Robust DR and Non-Robust MDTL-Max Algorithms
1: Initialization: 0 < Qx < T- Vk
,,.., K
2: for = O..... T do
3: for k = L,..., K do
4: (s((s) - max Q),
5:
Qx(s,a) + - X)Qu(s, a) (T.(Q))s..d) ((s,))
/Local Update/
6: end for
7:
i 0(mod E) then for s S, a do Proximal DR: Q(s.a) D Qx(s.a)
8:
9:
10:
11:
For Maximal Aggregation:
12:
((s.a)+ Max-Aggregation of (Qu(s,a)
13:
Qx(s,a) + a)s vk
/Synchronize/
14: end for
end if
end for
B.1. Ablation Study: experiments different target domain uncertainty levels
To further evaluate the robustness of our methods, conduct ablation study analyzing the impact of different levels of uncertainty. In this experiment, we systematically vary the test uncertainty set radius Riest and measure the optimal performance in the worst case over these uncertainty sets for the learned policies. Four different methods (MDTL-Avg, MDTL-Max, Non-robust DR, and Non-robust MDTL-Max) are evaluated across five levels of uncertainty, defined by the uncertainty set radius {0.01, 0.03, 0.05, 0.07, 0.1}. All testing experiments are conducted for 3 times. We also provide an optimal policy as benchmark, which trained on the target domaín using non-robust vanilla value iteration (Sutton Barto, 2018). The means and standard deviations of robust values of each method are reported over training time steps.
Let's conclude the ablation study before further analyzing the figures and tables: ablation study provides empirical evidence that our robust learning methods effectively mitigate the impact of target domain uncertainty. Specifically, our approach achieves at least 15.95% % and at most 151.36 % performance dominance over non-robust baselines in the target domain under model uncertainty. Compared to the optimal policy, our method achieves at least 81.32% and at most 90.66 % of the optimal performance under model uncertainty. These findings validate our pessimism principle, demonstrating its effectiveness in handling domain shifts and model uncertainty.
Method
Reest 0.01
Reest = 0.03
Riest 0.05
Riest 0.07
Riest = 0.1
MDTL-Avg
134.45
101.91
82.05
68.67
55.17
MDTL-Mal
134.45
101.91
82.05
68.67
55.17
Non-robust DR
53.49
52.06
50.70
49.41
47.58
Non-robust MDTL-Max
53,49
52.06
50.70
49,41
47.58
Non-robust Single-learn Nominal
148.30
115.48
95.35
81.71
67.84
Table J. Robot: Values 5 Methods under Different Target Domain Uncertainty Levels
Table provides a numerical comparison of the methods across different uncertainty levels. Our methods consistently outperform non-robust baselines across all uncertainty levels.
16

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
MDTL-Avg
140
Robust Value Function
100
60
R R_lest00.1
R_test-0.07
20
R_test00.05
test-0.03
1estt=00011
20
60
ao
100
120
140
Time Step
Figure 3. Robot: Values of MDTLL-Avg under Different Uncertainty Levels
Figure 3 shows the evolution of the robust value function for the MDTL-Avg method the uncertainty level increases. The results indicate that when the target domain uncertainty is low (Reest = 0.01), the learned policy achieves high rewards. However, as the uncertainty radius increases, the value function decreases, illustrating the inherent difficulty in handling a more uncertain environment. Despite this, our method still maintains a significantly higher value compared to non-robust baselines, confirming its adaptability to uncertainty.
MDTL-Max
120
Robust Value Function
100-
80
40
R_test=0.1
testt=0.007
test=0.05
R testt=0.003
R_test=0.01
00
100
Time Step
Figure Robot: Values of MDTL-Max under Different Uncertainty Levels
Figure presents the results for the MDTLL-Max method, demonstrating a similar trend. While performance declines as uncertainty increases, the robust approach still consistently outperforms the non-robust baselines. Notably, both robust methods maintain high robustness under uncertainty, reinforcing the effectiveness of our pessimism principle.
B.2. Experiments the HPC Cluster Management Problem
We also validate our methods in an HPC cluster management problem. Each time new task is submitted, the HPC cluster manager must decide whether allocate it immediately enqueue it for later processing. The HPC cluster operates in of three states: normal, overloaded, fully occupied, depending the number of active tasks. Under normal conditions, allocating task has probability p of transitioning to the overloaded state while yielding large reward. When the cluster is overloaded, allocating a task has a probability of pushing the system into the fully occupied state, where only a small reward is received. In the fully occupied state, all new tasks are automatically enqueued, and no further rewards are obtained.
In our experiments, we set the target environment parameters to p = q = 0.9. To model distinct domains, we randomly
17

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
in the above recycling robot problem. Specifically, we conduct trials for each method using different random seeds, with fixed step size of 0.1, a total of 5000 training steps, pre-defined uncertainty set radius of R 0.8, E = 1, and 0.95. The results, presented in Figure 5, where our pessimistic methods consistently outperform the non-robust baselines in target domains, both with and without model uncertainty. In summary, our method outperforms non-robust baselines, achieving a 183.28 % performance improvement in the target domain and maintaining 11.05% improvement even under uncertainty. This further confirms the effectiveness of our approach.
generate K 7 source environments with Pk; Qg E [0.1, 0.15,, while keeping all other parameters the
Target Domain Performance
Target Domain Performance with Model Uncertainty
Robust Va lue Function
Value Function
200
60
150
MDTL-Avg
MDTL-Avg
50
MDTL-Max
Non-robust
0-
Non-robust MDTL-Mex
Non-robust MDTL-Max
120
40
Time Step
Time Step
Figure 5. HPC Cluster Management Problem
Our ablation study highlights the substantial advantage of our approaches in handling different levels of uncertainty in target domain, Across different levels of model uncertainty, our method consistently outperforms the non-robust baselines. Specifically, approach demonstrates minimum improvement of 11.05% and maximum of 141.29% over the best- performing non-robust baseline in the target domain under model uncertainty. Moreover, when compared to the optimal policy, our method retains at least 77.52 % and at most 88.49 % of the optimal performance under varying uncertainty levels. These results further reinforce the effectiveness of our approach.
Method
Riest 0.01
Reest 0.03
Riest 0.05
Riest 0.07
Riest 0.1
MDTL-Avg
224.09
169.85
136.75
114.45
91.95
MITTL-Max
224.09
169.85
136.75
114.45
91.95
Non-robust DR
92.87
90.44
88.12
85.92
82.80
Non-robust MDTL-Max
92.87
90.44
88.12
85.92
82.80
Optimal policy
253.25
198.42
164.75
141.89
118.62
Table 2. HPC: Values of Methods under Different Uncertainty Levels
Table presents a quantitative comparison of the four methods under varying levels of uncertainty. Our approach consistently surpasses the non-robust baselines across all uncertainty settings.
18

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
MDTL-Avg
Robust Robust Value Function
100
R R_lest=0,1
R_test-0.07
R_test00.05
test-0.03
1estt=00011
20
60
ao
100
120
140
Time Step
Figure 6. HPC: Values of MDTI.-Avg under Different Uncertainty Levels
Figure illustrates how the robust value function of the MDTL-Avg method evolves with increasing uncertainty levels. The results show that when target domain uncertainty is minimal (Rest 0.01), the learned policy attains high rewards. As the uncertainty radius expands, the value function declines, reflecting the growing challenge of navigating more uncertain environment. Nevertheless, our approach consistently outperforms non-robust baselines, demonstrating its resilience in adapting to uncertainty.
MDTL-Max
200
Robust Value Function
150
100
R.test-0.1
R_test=0.07
teset=00
test-0.03
R_test=0.01
20
60
a0
100
120
140
Time Step
Figure 7. HPC: Values of MDTL-Max under Different Uncertainty Levels
Figure 7 depiets the performance of the MDTL-Max method, revealing a similar pattern. Although the value function decreases uncertainty grows, the robust approach consistently surpasses non-robust baselines. Notably, both robust methods exhibit strong resilience to uncertainty, further validating the effectiveness of our pessimism-driven strategy.
B.3. Experiments on Dynamic Vehicle Routing Problem
We further implemented our proposed method in real-world problem: DVRP (Jia al., 2025), to further validate the proposed method. DVRP extends the classic vehicle routing problem by incorporating dynamic elements such real- time customer requests, and environmental uncertainty, making it more representative of practical logistics and mobility applications. In our experiments, we consider multiple objectives that reflect both operational efficiency and service quality. Specifically, we aim to minimize overall routing cost, enhance route smoothness by reducing unnecessary detours and zigzag patterns, and improve stability by minimizing abrupt changes in planned routes caused by re-routing in highly dynamic environments. These objectives are crucial real-world deployments where balancing efficiency and robustness essential. More details about the DVRP problem setting can be found in (Jia et al., 2025). The implementations of our proposed
19

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
methods, MDTL-Avg and MDTL-Max, well the baseline algorithm Non-robust DR, are based (Jia et al., 2025).
The baseline Non-robust Single-learn is also adapted from (Jia et al., 2025).
Method
Route Distance +
Route Smoothness 1
Route Stability f
MDTTL-Avg (ours)
4.01 + 0.3
51.76 + 1.96
0.63 0.08
MDTL-Max (ours)
4.01 + 0.27
52.46 + 1.45
0.68 + 0.07
Non-robust DR
4.53 4: 0.52
50.32 + 1.35
0.55 + 0.08
Non-robust Single-learn
5.20 + 0.58
50.70 + 0.54
0.40 0.03
Table 3. DVRP: Results 4 Methods. J: smaller is better. f:: larger is better
Table 2 presents the performance of four methods evaluated on DVRP. Our proposed approaches, MDTL--Avg and MDTL- Max, consistently outperform the baselines across all considered objectives. terms of Route Distance, both MDTL-Avg and MDTL-Max achieve the lowest average values (4.01), indicating high efficiency in minimizing total travel cost. Regarding Route Smoothness, MDTL-Max achieves the best performance (52.46), suggesting it effectively reduce unnecessary detours and zigzag behaviors in dynamic environments. Finally, Route Stability, which measures the resilience of planned routes under dynamic re-routing, is significantly higher in both of our methods, with MDTL-Max achieving the highest score of 0.68, demonstrating its robustness to environmental changes.
B.4. Experiments on Biased Aggregation
The unbiased assumption made in MDTL-Max is to facilitate the convergence analysis. Nevertheless, our convergence results can still be extended to the case that the existing bias can be controlled, e.g., the bias introduced by the max aggregation can controlled through techniques like threshold-MI.MC (Wang et al., 2024g). On the other hand, even if there is bias, as long as the expected proxy still pessimistic, our transfer learning framework and pessimism guarantees still hold. This indicates that our methods are robust the bias.
To illustrate this, we develop an experiment on Cartpole to show the effect of bias. As shown in Figure 8, even if there is bias, our proxy is still conservative and our pessimism framework outperforms the baseline, and are hence robust.
Target Domain Performance
Average Accumulated Reward
Small Bias
Large Bias
175
Non-robust DR
150
125
100
75-
250
750
1000
1500
1750
Number of Episodes
Figure 8, Effect of Aggregation Bias for MDTL-Max
B.5. Additional Experiments on Negative Transfer
We further tested our MDTL-Avg and MDTL-Max algorithms two additional environments, aiming validate whether MDTL-MMxx can effectively mitigate negative transfer. For the CartPole Gym environment, we treat the default environment as the target domain and introduce Gaussian perturbations to the pole length in three source domains, with variances of 0.01, 0.02, and 0.03, respectively. For the recycling robot problem, we set most the source domains with a E (0, 0.1) and E (0, 0.1), in which the optimal policy should be waiting. We add another source domain with a = 3 = 0.9, in which the optimal policy should be searching. As shown in Figure and 10, MDTL--Max effectively leverages the most informative
20

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
domain, and hence avoids negative transfer.
200
MDTL-Avg MDTL-Max
Target Domain Performance
125
100
25
250
500
750
1000
1250
1500
1750
2000
Number of Episodes
Figure 9, Effect of Negative Transfer under CartPole Gym environment
250
MDTL-Avg MDTL-Max
Target Domain Performance
200
150
50
0
20
40
60
80
100
120
140
Time Step
Figure 10. Effect of Negative Transfer under Recycling Robot
B.6. Comparison to Related Robust RL Approaches
Note that our methods are based on distributionally robust RL, where our uncertainty set is constructed account for the potential distributional shift. We thus compare our proposed methods with other related robust RL approaches. We numerically verify that our methods enjoy both convergence and performance guarantees, and present the results under the recycling robot environment, where we use adversarial action-robust RI. (Tessler et al., 2019) and distributionally robust RL (Panaganti & Kalathil, 2022) as baselines.
As shown Table 4, although adversarial robust and distributionally robust RL outperforms non-robus. methods, their performance remain significantly inferior to ours.
21

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Method
Riest 0.01
Riest 0.03
Riest 0.05
Riest 0.07
Riest 0.1
MDTL-Avg (ours)
134.45
101.91
82.05
68.67
55.17
MDTL-Max (ours)
134,45
101.91
82.05
68.67
55.17
Adversarial Robust RL
61.06
58.10
55.39
52.92
49.58
Distributionally Robust RL
53.86
53.12
52.41
51.71
50.70
Table 4. Robot: Robustness comparison 4 Methods under Different Target Domain Uncertainty Levels
C. Theoretical Proof for Lemma 4.1
Proof. Denote the optimal policy of f by zy. Note that
V
Vi
Vi f((=)+++++)))+ +
(m')
+ f(T)) Vig
50,as n =arg f(n)
50, ((T)
VP. f(xx).
D. Theoretical Proofs for Averaged Operator Based Transfer
In this section, we provide the proofs the theoretical results for the averaged operator based transfer method.
D.1. Proof of Lemma 5.1
Recall that the robust and optimal robust Bellman operators for agent are defined as
Qs, a) r(s, a) (P)) (V),
(10)
where V(s) = Ene==Ene (als))((sss) Since TV is -contracctoon for any A E K., then for some Qı and Q2 we have
|T T21 S
Q T2Q2l S <11Q1 Q21l-
(11)
Hence T is also y-contraction and has unique fixed point QAo which completes the proof.
D.2. Proof of Theorem 5.2
In order to verify that Vïo is lower bound on VA, proceed as follows. Let (s) E.EA (lals)QRo(s, a). since
Po E Pk. we have
k Σ
a) = ?
- -VP)
K
k=1
3 (P)) -Voo) -Po) +7(P)) - V7,)
= 7
K
< + 7 Vi))
(12)
Consequently,
(s) (s) = Σ [(als) - Σ (s.
aEA
aEd
Σ T(als)(QTo0s..) Q7, (s,a))
AEA
$ (P(P)) VA,),
(13)
22

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
where (PT)s is the s-entry transition kemel induced by π under Po. Let Po ((P8)s,) (PT) matrix, and consider entry-wise relation, have
(PT))s)) m())) ) be transition
VÃO Vp, S <PPo(VAo V.)),
(14)
and hence
(I - -o)Po) - 12 0.
(15)
Note that (1- yPo + P + .... exists and has all positive entries, we thus obtain
Vão VĀ s ((I PP)) ' 00=00 0,
(16)
which completes the proof.
D.3. Proof of Proposition 5.4
For two distribution p. q E A(S), total variation is defined as D(p q) D(p, q) inf (B (B - ((-)))) where [p,, q) is the set of all couplings of p. g.
-11p- 1/11, and Wasserstein distance is defined as
We have
(5 a) (s)
(17)
K
-0pp (V
Σ
(pL) (Văo)
(18)
K
k=1
p (Vp) (pt)) (Văo))
(19)
ppp (V - [p*)) (V7) + (ppsss (VF)) app2) (Văo)).
(20)
For any vector and total variation distance, it holds that
Tp. (v)
1
Σ (v)))(() max Pia. ISp(ua))
1
max PaVa. [[Sp(va)}
K
K
kml
K pkv, TSp(va)
1
max max(Pea [Sp(va))
max
a
K
0:
(21)
Similarly, for Wasserstein distance, holds that
K
(v) 1
Σ (p))(()) = supp-uu
-A + Σ
P(s) inf in( + xd(s.))))
A2U
WES
#ES
supf-Ar Ar A20
Ps(s inf (v(r) Ad(s.))))
K
WES
< sup - X
Σ
P(s) inf (v(y) + Ad(s.)))) WES
supt -
Σ
(s)) inf (v(y) + Ad(s,)))) WES
A
SES
0.
(22)
The remaining proof follows similarly as Theorem 5.2.
23

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
D.4. Proof of Theorem 5.6
The contraction property of Too follows similarly as Lemma 5.1.
For policy (s) - arg m Qso a), we have
K
T () r()
(PW) max QAo(s, TooQAo(s, = Q4o(s, aEA
(23)
K
Hence QAO a) is also the unique fixed point of To moreover, VĀ = VAO. For any & E K, denote the worst-case kernel of VAO Pa by gic.e. ((P)): (PK) V(s,a) E
(24)
Since VAO = maX.ed QAo(s, (s, a), holds that
(s) - (so(s)= s Σ
r(ass) (Q - Qxo(s,a))
aEA
K
Σ
T(als) 7
Σ
(VXo) (V (VP)) (Vxo))
K
aEA
k=1
K
$ Σ T[als)) Σ
K ((P))VAO (AK)
(25)
aEA
k=1
Let k arg maxKeK EoEs ((ls))
VMP) and P. EoEA (P))) we obtain
(oo(s) - Vso(s) -
Σ T(als) Σ ((PP)VV (PP))VAO)
K
aEA
k=1
- (VT.VV VAo),
(26)
and hence by noting that (I (T has all positive entries we have
(I (V- VAO) 50 which implies that max, VIo I VAO- On the other hand, since VĀ = VAO and max, VIo 2 VT together with the upper bound it follows that max, Vão VAO. Note that (so = max.e Qso(s, a), hence arg max, VïO Te. which completes the proof.
(27)
D.5. Proof of Theorem D.1
We first show the convergence of Algorithm with the accurate operator to illustrate the efficiency of our algorithm design.
Theorem D.I. Let E - 1 < min $ 1
1 }. and X
(TK)
If
Tk. it holds that
T(I->)
Qao 2k Qk
EE 1D-
S TK
(28)
K
T
In this proof, we first assume that E 1 1-7 and X < to establish the general convergence rate, which is independent of K. Then we prove that carefully selected and À can balance each term in the convergence to achieve partial linear speedup.
Denote @++1 and @+++1 the values of Q and Qs at iteration t + 1. Since T = Tk. we have
K
Q++1= = K
Qt
km
Σ (11
U)Qk + (VP)
(29)
K
24

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
where (s) a (s, a). We also denote the value function associate with QAO by VAO max Q a). We define the iteration error as A 1+1 following form:
Qao Q++1, additionally, A = QAo Q". The iteration error then takes the
A r+|
QAO - QQ++1
1 K
Σ
(QAo ((1-))Qk + X(r + yop, (V21))
K
K
(11 [
1)(Q - Q1)+ + A - yopp (Vi)
K
= (1 A) + 7 K
Σ
Opp, (VA0) -p.. (VE))
K
1=1
(1 - x)
Σ (1 2)
Σ oo, (op, (VAo) -op, -Op. (V+)).
(30)
im0
By taking the -nom on both sides, we obtain
K
|| S (1 - 1 || + >> -
1 K
Σ
(op, (VAo) - Jp (Vi))
(31)
i=0
Since 0 S Q" S - the first term in (30) can be bounded as
21 1
- x) S (1
(32)
1-7
To bound the second term in (30), we divide the summation into two parts, Define the most recent aggregation step of as x(t) = 1 - (t mod E). Then for any BE < t s T, we have
K
Σ - (1-2)- x)-))-->> Ay
op, (VAo) - (v))
K 1=11
1=0
X(1)-BE
1 K
1 K (1 (1-2)
Σ
(1 (1-A)(-i>) xyy
Σ
(op, (opL oPp)
Σ
Σ (op, op. (vp.
K
K
x -E+1
K
< 7- (1 - x)(
Σ
(1-1)--i>7
oop (VAo)
(33)
1-7
k=1
Before proceeding with the proof, we introduce the following lemma to bound the second term in (33). The proof of this lemma can be found in Appendix D.6. We define A := QAO Qi the local iteration error of agent We also define
T(P)): (v) = K Σ ((P)); (v) for any vector v.
Lemma D.2. Ift mod 0, then
1/2 (op, (VAo) pp (VE)) < || Otherwise,
K
h-1
Σ
(op (VAo) (V
< Axx0)
+ 21
A
K
e' (t))
K
((t
+ ト
max ((Po).(Ao)) - 3(Pr)) (VAo)),
(34)
K
(s.a)
where we use the convention that Σ -(=x(()
|| 0.
25

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Then by Lemma D.2, (33) yields
K
Σ
(1- (1-A)-->> xy
I
Oop (Vxo) op (V2))
K
-BE-
k=
(lacto))
1 k
Σ
(1- i
+21
+
Σ ΣΣ
K
-
k=
x=1)=x(t)
t - 1 - 1x(
K
|T())) (o.))(Vax)) (VAo) - 8 (Pk))) (Vao))
max (a,0)88AA
K
Σ (1-
1
Σ Σ
K
|
+ 7AT 1- 1-7 7
+ 2x
(35)
K
-8+1
k=((=x(!
Furthermore, the local iteration error | can be bounded using the following lemma. The proof of this lemma can be found in Appendix D.7. Lemma D.3. If S E. then for any & E [0,T - 1] and k E K, | <
||ax + - 1)T.
Then by Lemma D.3, we have
K
Σ
(1 (1-
1
Σ
Σ
|
K
x(x 1111
1 x-((
Σ (|AX60)
i-1
2
Σ
(1-1)²=
+3 7
X(E 1)r)
1-7
i=x(t) 3E+1
j=x(i)
67212 1-7
< 22
max x(t) gg
| +
(E- 1
(36)
By combining both bounds (35) and (36), we obtain
Σ
by
op. -OPp
(x(1))8++1
6-22²
S7 x(1) BEESiE
max
4x00||
+ (F
max
Ax
+
11-
(E-
1-7
(1- A77 7d 7
(E (E-1)1)
9Et1
22
+ 2A(E - 1))
max -ESiSS
Ax(00
-DD
+ -
(37)
1-7
Plugging the bound (37) into (33) then yields
1) 'y Σ -Vx) (V,))
7 (1 1)^ -)0-X(0+ + 71 + 2X(E 1))
|x00|
max
1-7
x
12
+ 1-7
(6X2(E - 1)2 + XEE - 1))r.
(38)
26

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Consequently, the iteration error || A [+
be bounded using (32) and (38) for all rounds ( [0, T 1]:
A | 1 (1 - 1) -1
1-7 + 1 (1
x)=xx(0++
?2
+ + 2A(E - 1)) max
+
(6 - 1)2 + X(E - 1
1 1-7
S 7( 22(E 21(E-11) - 1)) max
Ax
2
(1- 2)
1-7
72
+ 1 1- (6 [ 1)2 + X(E - T))r.
(39)
Let p= P (1 2)(E
+ - (6>? 1)² + M(E - -))rr, then with the assumption S 7y(E-13 (39) can be written as
| || 5 2
1+7 max
A +
(40)
Unrolling the above recursion L times with L3E St S T - 1, we obtain
L-1
A
+7
Ax
7
''P
max
x(() x(t) LiE
2
+
1
2
P.
(41)
2
1- 7
1- 7
By choosing 3
1 V
(1-1T
L
AT
and t + 1 = T, (41) yields
1 1
+ 1-7 1- (1- 2) + 72
2
2
A
5 1-7 7
(6\2(E 1)2 XE 1))r)
2
1 1-7
<
1 - exp(
2V 5V(1 -)) + (1 7)-
exp - -v(11 - 7)17}
27
+ (1 7)² (6^2(E 1)2 + +(E - 1))r
2.2
< (1 (1- expl (1 (1-11110) - ))TT) + (1 7)² (6A2 - 1)2 + A(E 1))r.
(42)
(42) reveals the effect of E' and À on the convergence rate. Although E > 1 slows down the convergence, we can set (F - 1) S min 21 k and x 7(--))
(T to bring partial linear speedup with respect to the number of agents K.
Then (39) becomes
1
+ (E
1)r
I
(43)
TK
T
for T 2 E, which completes the proof.
D.6. Proof of Lemma D.2
When t mod E 0, i.e., t is aggregation step. Qj Q and the associate value functions V% Vi for any ls and
k' E IC, We denote 5 = 1 Ek Vi, then we have
1
Σ
(Vo)) - (CPP):(V1))
ā (V P9)
K
k=1
K
VAO - V
K
k=
||QAo Q'1l
||l|||
(44)
27

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
For general 4, we have
K
K
po (VAO) Op. (V2))
1
(pp (VAo)- (V6 (V2(l0) + op, (V2110) op, (V22)
K
K
1=1
K
opp (VAO) ope (12
Σ
Oo, (V - on. (12))
+
K
K
K
x
+
1
Σ
(op, (Vx
-op (vi))
K
K
<
A
v2((1) k
Via
(45)
K
For any state 8, we have
(s) - = -s)((())) -
-2
(s, xx)
(a)
Qi(s, af(s, (s.
Q (s, + Q2 (s.d(()))))) 2 (s, (())
Q (s. Q (s))
(46)
where (s) = arg m Q (s, a) and inequality (a) holds because Q (s, (x)) & Qx (s, a (s))
For each 1' such that x(1) 5 S i, holds that,
(s(()s((())) - (s, = (1 - 2) + (r(ss + ta (m - -v (x (s))
( (s, + (Q al(s)) (s)) - 70 (P.)) (VAo) (VAo)
+ r((,
(P 1 (VE
<
+ + (P (00 - ā (P2)2f()) (Vao)),
(47)
where equality (a) follows from the average optimal Bellman equation. Thus,
- Vx(0
(8) <
"s. - (s,
(()(()
t-1
= 21
Σ
x(t)((o (P)))(s(( - 7 (P.) (VAo)).
(48)
(1=
Similarly, we have
[-1
Vi(3) - (2) 2
Σ
(0t +1 (s, a (s)) - (s, a x(0 (s)))
(t)(
e-1
2 22
Σ
|
+ >2(t 1 x(t))(o
- (P.) (Vo))
(VAO)).
(49)
=(x((t)
28

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
By plugging the bounds in (48) and in (49) back into (45), we obtain
op. (VAO) op, (V:))
k
xx()
V/
KP
K
1-1
- x 2221
|
K
k=
(==x(1)
K
1
+ +1a(t - 1 11((-11x((t) x(t)
Σ
max a.EESxA
(VAo) - ((P)) (VAo)). (50)
K
This hence completes the proof.
D.7. Proof of Lemma D.3
When mod E = 0, then 4/ = A x(t), When mod E # O, we have
U (1 110k A(r + (V
= (1 - 1-10
top, - - 2 +
(51)
Thus
A = (1 - -)41 + Av(op, (VAo) op, (Vk
= - + + \T(ōp, (VAo) Jp, (VAo) + opx (Vxo) -o,, (V(-1))
= (1 x)
A x(1 - Σ (1- (0p, (Vao) -op (VAo))
(1=(())
+7X Σ
(1 X1- (op, (VAo) (V,)).
(52)
(=x(())
For any state-action pair (s, a) E S A
(11 - (x00) (1-2) -0----1-1 /xco/.
(53)
Note that
1-1
Pr Σ (1 - (VAo) oo, (VAO))
7 - Σ (1-2)
1- ('=((1)
('=x(!)
7 X(E 1LE 1)r,
(54)
1
1
alr (s E S x A,t e [0,T - 1] and E K. In addition, we have
[-1
1-1
Σ (1- (1- (op, -pp (VE')
Σ (1 (1-
(55)
ft=
(1=((1)
By combining the bounds in (53), (54), and (55), we obtain
| < (1 - xx(-
Xx(00
7 ME 1)r +
Σ (1- (1= 1) 1'1 A
1-
=(((()
< -1-11 (1 (1-7)
|x00
+ (1+ +1+
1 XEE - 1)1),
7
(56)
29

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
where the last inequality can be shown via inducting x(1) (
1--1. When À 1
(1 + - (1 1) Σ (1 + 1/E) s
(57)
Hence
| < |
+3 7 X(E 1)r,
(58)
-
which completes the proof.
D.8. Proof of Theorem 5.7
Since E[TT] = Ts and Too ΣK Tx it follows that E[ [o] = Σx E[Ta] TAo. Then by replacing Ta and TAo R
K
with their expectation versions, the proof follows naturally as that in Theorem D.I. and thus omitted.
E. Theoretical Proofs for Minimal Pessimism Principle
In this section, we provide the construction and proofs of the thcoretical results for the minimal pessimism principle.
E.1. Multi-level Monte Carlo Operator Construction
We first generate N according a geometric distribution with parameter V (0, 1). Then, we generate 2 unbiased estimator T 4 1, ..., 2N+1, We divide these estimators into two groups: estimators with odd indices, and estimators with even indices. We then individually calculate the maximal aggregation using the even-index estimators, odd-index ones, all of the estimators, and the first one:
T = max k,i
m 2,4 2,, NN+1j,
(59)
T = max k,i
2 1}.
(60)
Q^Q max - 1,2 2N+1J,
(61)
Q max (T))-
(62)
The multi-level estimator is then constructed as
TMLMcQ T A(Q)
(63)
pn
where p. = 41 - N and
T + T=Q A(Q) 2 T^QQ
(64)
2
E.2. Proof of Theorem 6.1
We first introduce some useful inequalities to extend the results in averaged operator based transfer to the max-aggregation version.
Lemma E.1. For any vector sequences {Xa} and (Yk} with & K. it holds that
1) maxKEK X maxtEk Yv 5 maxxex(Xx Yx),
2) MaxKEK X maxKEK Yall S maxtek: |X YEll-
3) maXKEK || maNGEK N < maxLek [all.
30

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
The proof of this lemma be found in Appendix E.3.
Since T is n-contraction for any & K. then for some Q1 and Q2. by Lemma E.1 we have
MP Q1 TMP Q211 = ll max
T
max T/ Q21
kEy
kek
max
| Q1 T2ll
kER
<1 Q2ll.
(65)
Hence TM is also y-contraction and has unique fixed point QMP- The claim for TMP can be similarly derived.
In order to verify that MP is a lower bound on VP, we proceed as follows. Let VMP = EaEA (aals)(7rp(ss a). since Po E Pk, we have
a) - Q7) ()) =y
max (Pt) (VMP)) - (Po) VNr + (Po)) - VF8))
REK
{0+ VA)).
(66)
Consequently,
V MP (8) (s) =
Σ Qhpps)) Σ T(als)Qpp (8,a)
aEA
aEA
Σ ((als) Q7, (s,a))
aEA
$ r (Po) (VMr Vi,)),
(67)
where (. the s-entry transition kernel induced by a under Po. Let Po (PT) T be a transition matrix, and consider entry-wise relation, have
MP VP, S S>Po(VM VA.),
(68)
and hence
(Po)P) - V2 < 0.
(69)
Note that (1->Po)-1 1 PPo PP? +... exists and has all positive entries, we thus obtain
MP VI, S ((I-7P))
(70)
The claim VØ, V can be derived similarly.
For policy T. (s) A arg maxae.A QMP(s, a), we have
T (MP = r(s
max(o((P)) max QMP a)) TMPQMP(S, a) QMP(s, kek
(71)
aEA
Hence QMP(s, a) is also the unique fixed point THp moreover, V = VMP. For any &s E K., denote the worst-case kernel of Var in Ps by Pk i.e.
(PWW)(MMB (PM)VMMP, ((s,a)
(72)
31

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Since VMP max4EA QMP a), by Lemma E.I it holds that
(s) (s)((s) I
<(lss)
(QMp(s, - -s,r(s,a)
aEA
Σ
x(als))
max (VTP)
max (PL)2 (VMP) MER
kEk
x(a|8)7
max(( - (Pa)( VMP)
KEK
KER
aEA
Σ {(als) keR
max (na)(( (Pk))VMP)
aEA
{(ass) max (PP) (VMP VMP)))
(73)
kek
aEA
Let P. arg maxkeK EaEd a(als)
VMP). obtain
(M VMM(() S S P& VMP),
(74)
and hence by noting that (I~~P) has all positive entries we have
(I - (V->P)(VMr VMP) s
(75)
which implies that max, VM MP S VMP. On the other hand, since VMP = VMr and max, MP 2
ViP together with the
upper bound it follows that max, VMP VMP. Note that (s) = max.e QMP (s, a), hence arg max, VMP Ta. which completes the proof.
E.3. Proof of Lemma E.1
The proof of the first two statements can be directly derived from reverse triangle inequality and Lipschitz continuity, and is thus omitted. We now consider the third inequality.
Denote X the i-th element of Xg, we have
X
max X S 0, Vi,
(76)
AEK
which indicates that Xx maxKeR X₃ is a vector with non-positive elements. Therefore
max REK.
[
max
X
= max max [
max X
REK
kek
REK.
max X3 -x²)
REY KER
maxxmmxx X min XT). kER
(77)
kEK
We also have
max AEK
|
max max X > max min X
(78)
Hence for each i,
max X
min
X
< max X + min
X & 2 max X
(79)
&ER
kek
kek
kek
kek
By taking the maximum over i, we thus obtain
max Xg
N4ll = (max X
min X
max
|| l1
(80)
AEK
Ke
keK
REK
kEK
which completes the proof.
32

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
E.4. Proof of Theorem F.2 Theorem E.2. Let E - 2 min H hh and X
1log((TK)
we set Max-Aggregation of Qu(s,a) be T(1-7)
max; Qs(s.a). If Tg, holds that
1 TK
E
1
QMP
max Qk KEX
(81)
T
In this proof, follow the same road map with Theorem D.1 based Lemma E.1. first assume that F: 1 1 1-7 and < E to establish the general convergence rate, which is independent of K. Then we prove that carefully selected E and A can balance each term in the convergence rate to achieve partial linear speedup.
Denote Q++1 and 2++1 the values of Q and Qu at iteration + Since Ta = Tu. we have
+++1
max Q++1
REA
max((1 YIYK + A(r++ (V))) REK
(82)
where (8) = a. (s, a). We also denote the value function associate with QP by VмP = maxac QMP (s, a). We define the iteration error as A [+1 QMP - QQ+1, additionally, A : =QM Q". The iteration error then takes the following form:
1+1 Qmr Q
mmax(QMP ((1 - -jQ4 + + NOP, T0P, (V KEK
max((1 2)(QMP O1) + A(QMP - yap, (V2)
kEK.
=(1 -))A + max (VMP) JP Opx (Vz)) REA
= - () Σ (1- 1 max((o, Coo,,,(aa (VxP) op, (V2)).
(83)
ReK.
0m0
By taking the Too -nom both sides, obtain
| s (1 - x) || +
71 (1 -
1)
max REK
(VM (VMP) - -Vp
(84)
1=0
1_ Since 0 S <Q0 < , the first term in (30) can be bounded as
A) 1
(1 (1- x S (1
(85)
To bound the second term in (83), divide the summation into two parts. Define the most recent aggregation step of x(t = t (t mod E3). Then for any 3E < < T, we have
(1 by
max REK
(V2))
1=1
-(1)-$E 9E Σ
(1- Xy
max (KWP) RER
Σ
(1 (1-2)--i>)
by
Imax(op, (kaP) (KMP)-op, (V2))
KER
1=0
-x(0)-8++1
7 1-7
(1 -1))))² -x00++++
Σ
(1 (1->)~->>7
m KER
(VMP) (AMax) (V2))
(86)
Before proceeding with the proof, we introduce the following lemma to bound the second term in (86). The lemma can be seen as a variant of Lemma D.2, and the proof can be found in Appendix E.5. We define A = QMP - Q the local iteration error of agent We also define (t) = maxkek (PP)) (v) for any vector v.
33

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Lemma E.3. Ift mod E- 0, then |maxxex((Pp) (VMP) pr (VE)) < |l Otherwise,
max ( - op. (VE)) - 4 KEK.
Σ
|
2A max
KER. -=x(()
+7 x(1))
max (K.x,a)EXXxxx
((Po::(ap)) E(PA): (VMP)), (87)
where we the convention that Σ -(=x(t) | O.
Then by Lemma E.3, (86) yields
Σ
(1- 1--))t->7
map (VAr) op, op. (V2))
AEK
(AAXiO))
Σ (1 >)
+ 2d max
Σ
A
KER
==
x=x(#)
+nt(t 1- x(i)) 1xx(())
(((W):VMP)) - (( (Vaar)))
max (kss,)EXXSxx
|AXo0||
Σ
(1- x)
+ 2d +2 max
Σ
A
E 8-1 1 7AT
(88)
RER j=x(!)
1 1-7
-=x())-B++1
Furthermore, the local iteration error A be bounded using the following lemma. This lemma can also be seen as variant of Lemma D.3 and the proof of can be found in Appendix E.6.
Lemma E.4. If S E. then for any t [O,T 1] and K. | S | + ME 1)r.
Then by Lemma E.4, we have
Σ
(1-
2 max
X72A
Σ
|
EER
)=x(!)
Σ (|AXiO)
1-1
<2x
Σ
1-)1)*
+3 2
A(E 11)r
1)1))
1-
-=x(
(=x(1)
6²2x²
< <2>> 1)
max
Ax
+ 1 1-
(E - 1)
(89)
x(t) Aisis
By combining both bounds (88) and (89), we obtain
Σ (1- by
max(op, max(op,(VM)) 0p,
Op, (Vi))
KER
-88+1
6-
5
max
4xc00|| + +227(E-1)
max
Ax(i)
(E
1)-r
x(t) -8PSis
x(t) (88)s
1 1-
+ Σ
(1- '27( 71 (E 1)r)
i=x 9E+1
22
+1(++ 21(E -1))
max -BSisi
Ax(0)
(90)
I-
34

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Plugging the bound (90) into (86) then yields
Σ
A) 1- X> (o -op
KEX
KEX
$ 7 (1- - 11)-)-xx0+0D + + 2X(E - 1))
|
max x(f)-ESiSi
1 - 7
72
+ 1 1- (6X2(E - 1)2 + - -
(91)
Consequently, the iteration error
|| | can be bounded using (85) and (91) for all rounds & E [0, 7 1]:
A ++ S (1-A) 1 y + 1-7 (1 A) -x0x+
?2
+ + 22(E - 1)) max
A2(00
+
(6A> - + X(E - (
1 1-7
$ 71 22 - -1)) x(t)-SEgist
Ax(1)
2
2)8E
1-9 7
72
+ 1-7
(E 1)3 + X(E I))r.
(92)
pe (1- 8) + - (6 1)2 1))f, then with the assumption
2
1-y T<(-11)
(92) can be written as
A |
1+7
max
A +p.
(93)
2
x(t)
-ESSS!
Unrolling the above recursion L times with L3E St
1, we obtain
L-1
A 4+1
S
1+
max X(t)-LEEsist
Ax
2
d'P
2
2
1=0
7
L
1- 7 1- p.
2
(94)
2
By choosing 3 = L V
(1
L
XT
and L + 1 = T, (94) yields
| ||
5
1 +
1-
AT
+
2
2
(1 (1 +
72 1- (6>2 - 1)2 - X(E - -1))T)
?
2
1-p 1
1
4
<
expl-
v x) +
exp( - / - 7)\T}
7
(1 11-7) 7)
27²
(1 7)² (6A²(E 1) + +(E - 1))r
1 ((1 (1-+))T]) (1 (1- (6A + X(E 1))r.
2y2
$ (1 7)2 expl
(95)
(95) reveals the effect of E' and À on the convergence rate. Although E > 1 slows down the convergence, can set (E- - 1) < min 1 2
and À (TK to bring partial linear speedup with respect to the number of agents K.
T(--))
Then (92) becomes
1
+ (E 1)r
AT
(96)
TK
T
for T 2 F., which completes the proof.
35

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
E.5. Proof of Lemma E.3
When mod E 0, i.e., / is an aggregation step, U Q and the associate value functions Vi V for any &s and K' E KC We denote 5t = maxkex Vi then we have
max (PPP): - -(P)) (V2)) s max (Vaar) ((p):(()) REK
kEK
Kup - v
s max keR
< Qua - @'11
= |||||-
(97)
For general I, we have
max(op, (VMP) pp. (V(() REX:
max (Tp, (VMP) - Tp. (V2(1)) + op, (V2 - op -px (V2)) kER
max(Op, (p, (VMP) JP, (P. (V2(10)) /// + Imax(ors (V2c(t)) - - (Vx
KER
KEE
Ax(1)
+ max(op, (V2((0)
(VE))
KEK:
<
A x(0)
+ max REX
12((0) V/d
(98)
For any state 8, we have
(i(()) - (s) (s) (x)
(x,
(0)
< Qh(s, at (s)) 22
(s (s,,((()))) + (s, of (s,
Q (s. o (s.
(99)
where (s) arg A A Q1 (s, a) and inequality (a) holds because Qx (s, (s)) s (s
(s, at (s)).
For each t' such that xt) t' Σ t, it holds that,
+x+t(s, - Q (s, (s)) = (1 x)Q + + (rtsss + 70 ( - Q% (s))
(a) (s, + A(QMp(s, (t))) r r(s) - -70 (P2) (1) (VmP)
=s))) 7o (1
s <2>||>i|| + + (p (pestt - x (PA) (VNP)),
(100)
where equality (a) follows from the average optimal Bellman equation. Thus,
Vi((s) - v2x00 (s) = Σ
( + (s. - (s,
)=x(()
t-1
2Λ
Σ
+ -1 -x(1)(( (P.)) (VxP) - (Pu (101)
(t)
36

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Similarly, we have
- -12
Σ
Q 1 a
(s,
Qs)
(s, (
(t=
f-1
Σ 22
Σ
- x(x))
(PA )2 (VMP) - (PW) (VMP)).
(102)
(=x(t)
By plugging the bounds in (101) and in (102) back into (98), we obtain
max (VMP) - op.
- ||
+
max
Vid
KER
RER
< || xtol| + 2A max
Σ
|
KER
('=((()
+ + - -x(!))
max kx
P(PW):(Vnn - ((p)) (Vur)).
(103)
This hence completes the proof.
E.6. Proof of Lemma E.4
When mod E = 0, then 4 = A x(t). When t mod E + O, we have
O = (1 - 1 1 + (V (V
(1 - 1)Qk 11 +AQMM - - (VMP) + yo (o (V)
(104)
Thus
4 (1 1)- + Ao(o (KMP)
op. (V (V--1))
(1 1-1)) + Ar(ōp, (VMP) op, op. ( + op, (VMP) -p., -V
1-1
(1 X( x(0) +7
Σ
(1 (1- (op, op, (VMP))
('=((()
t-1
+ y4
Σ (1- (1-1) (0 (==(()
-07
(105)
For any state-action pair (s, a) E S x A.
(1 - x) (x00)) < <1- Ax00|
(106)
Note that
Pr
Σ
(1 - (Op, op, (VMP)) _ 7 A Σ (1-
1 -- -'=(())
e' (==
< 7 X(E - 1)r,
(107)
1 -
for (8, S A, [0, - 1] and E K. In addition, we have
+
Σ (1- x)- (VMr) op, (V2'))
Σ (1- 1)------11 |
(108)
(9=(()
==x(()
37

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
By combining the bounds in (106), (107), and (108), we obtain
A 2 (1 x(
A Ax(+)
7 X(E +) +
Σ (1 1) 1) 11 A
1-9
(=x((1)
S (1 (1 -1-
|
+1+ (1+7
x((0
1 X(E 1)r),
7
(109)
where the last inequality can be shown via inducting on x(t) (0....,.. -
1}. When À 1
< (1 {\+ (1+1/B)
(1+
Se53.
(110)
Hence
| s A
+3 1 -
X(E 1)r,
(111)
which completes the proof.
E.7. Proof of Theorem 6.4
Since E T ] = Ts, by Lemma 6.2 it follows that E[TMLM = TMP. Then by replacing Ts and TMP with their expectation versions, the proof follows naturally as that in Theorem E.2, and is thus omitted.
38

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
F. Proof of Proposition 7.1
Proof. We show that n Ps satisfies the condition. We denote the robust value function w.r.t. n Pk by VR. then we have that
V
MP ~00p, (VF) 7 max op.. (VMP)
= = -Norp, (Vir) - +O0P, (VMP) - max o7, (VMP)
2 po (V2) yoRp, (VMP),
(112)
which is due to nPk S Pk. Thus
V MP 2 70hp, (Vn) - To,., (VMP) 2 >P'(V - -MP),,
(113)
and thus
V
V
MP.
(114)
G. Model-Free Algorithm Design and Analysis
We then develop studies for model-free setting. We will focus on ln-norm defined uncertainty set (Kumar et al., 2023b; Derman et al., 2021):
Pe.a (Ps. +9: 21
q q R
Σ9(0) 11q11a S {Taa.
(115)
where r is small enough so that Ps.a + q E ASS), 4q E Q. For these uncertainty sets, their robust Bellman operator can be estimated through a model-free estimator. Specifically, for any p. set
<(V)
A Rmn le Vllg:
(116)
wER
with --Σ. For popular choices of p, the optimization problem in (116) has a closed-form solution, specified in Table 5 (Kumar et al., 2023b). It then holds that
k(U)
max, min,
x
2
Σ. v(s) E, x(s) 2
2
Σ
[[8 13/+1 v(s;) i=1
E ( (ss)))
Table Penalty term for ly- no-m uncertainty set
Lemma G.I. (Theorem 1 in (Kumar al., 2023b)) Let Pa.a be the uncertainty defined using the In-norm. For any vector V, the following relationship holds:
PP., (V) = Ps.a V (V),
(117)
where k is defined as in (116).
With these results, we use the model-free estimator V(s') l'x(V) to estimate the robust Bellman operator and design model-free MTDL-Avg algorithm. The model-free variant of MDTL-Max can be designed similarly.
We then study its convergence.
39

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Algorithm 3 Model-Free MDTL-Avg
1: Initialization: Q*
2: for = =0,. -..... T - do
3: for k = I,..., K do
4: for All s do
5: (*) <- max, (x)
6:
for All a do
7:
Take action and observe the next state (s)
8:
(s) (1 (s) + (r(((s,)) - -V xIx(Vh))
9:
(*(,aa) -- m0, (s.
10: end for
11: end for
12:
end for
if = O(modE) then
((s,a)+ (x))(a)) Vs,a
15:
(s, +
16: end if
17:
end for
Similarly, define
A = Q' -QQ411 and Ag = Q" Qo:
(118)
where denote Qi by Q*
Lemma G.2 (Error iteration). For any & 2 0,
1 K
A S (1 - 1) Ao 7r
- - -v()))
K
+ Y
2)
Σ
(p* P.V.
(119)
K
2=
km
40

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Proof. The update of A++1 is as follows:
1411 = Q" Q++1
Σ (Q- - 2cti)
K
k=1
1
Σ (Q -11- N6r - - (x(()))))
K
Σ (1- (1--)\QQ - -Q-) + xQQ0 -r- PP + (x((,+)))
K 1=
K
(I 7r
1
+ -
E(o
- - + vx(,))
K
K
7d
(oV()) (V,+)) + ?1
-1-A))4
Σ
Σ
pkyyz P!VVA))
K
K
k=1
k=1
ri 1 K
(1- A)
Do + 7d
E x)=
Σ
- -
K
1=0
k=1
+ 7> Σ (1
2)
Σ
(p - Pt.
K
k=
k=1
Lemma G.3. It holds that
0 (8) 1-
(120)
IIQ - 211 <
(121)
1-
I v/ll < 1-
1A 2 0, and &s [K].
(122)
Proof. Due to the update rule, Q 2 0. On the other hand, if = >[<(V²1)) then
1 (s, < (1 - A.) a(1(5.a) S
1-
(123)
which is from introduction assumption on V S -
(122) then directly follows.
Lemma G.4. Ift mod = 0, then
1/4
< |A.]]. Otherwise,
K
(V*)
-v
-
K
+ 41
1
Σ
Σ
|
7X
1
max
Σ (o (V')) -V=)
K
K
3,4
(t)
(()
where we use the convention that -(=x(!)
|
||

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Proof. When L mod E 0, i.e., i is synchronization round, Qf = for any pair of agents k. K [K]. Hence,
0 (V*) 02 (V,)))
(124)
K
K
(o 5,0 (V*) - thh (Vi))
(125)
K
K
IV Vell
K
k=1
S lI Qil
|A1l|
(126)
For general t, we have
K
(o")V-
-v()))))
K
k=1
K
1
(V*) o² (Vx(e)) + (v;"))]
(127)
K
k=1
o (V*) (+ x(o))) + K Σ (o* x(1) oz
K
k=1
||00م)||| +
Σ
0 (V22.) (V("))]
K
k=1
||4x0))||
+
1
Σ
I x(e) V.*1l
(128)
K
k=1
For any state s, we have
(s(() 1b (s)
(((, a where (s) argmax... (s,a')
- Qt Q of - Qi-2(s, (()))
e (s.
(s)).
(129)
where inequality (a) holds because Q% (s. (s)) < U (1) (s)).
Now consider each El such that x(!) < 1' S 1. If Q (8. at (8)) = 0. then
Q241(s, (5,
(s, s $21]A
42

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
On the other hand, if (8, + 0, it holds that
/I((V)) (s, = =s + ( r(s, 70 -0 ( r+(,,
+
+n(Pt) (a) - -Fx(())))))
-1 + A( 20 (.) + 1(o (o))
- )r (V") + (P( (e (V/) - -[x((,;)))
S 21|1 + + (
J ())())
where the last inequality is from the fact that is 1 1-Lipschitz (Kumar al., 2023b).
Thus,
(s)
V (x)
1-1
< Σ
(8, -
(1=((()
1-1
2A Σ
|ل + 7\
Σ
1(o
(s)) - o (o)))))(
(130)
(1=x(1)
('=x(()
Similarly, we have
(s) (s)
(-1
2 22
Σ
| ||
ト
Σ
(1o
sat
- oaa at)) (o) (V))
(131)
(=x(()
(==((()
Plugging the bounds in (130) and in (131) back into (127), get
(v*)
- -
K
1 K
<
+|Axxoll+
+
Σ
|
- V
K
x(t)
Am
[-1
< ||xx))|||
+ 21
Σ |
K
k= k= (=x(()))
K
1
+ 7>
max Σ
- (V*) (
K
k=1
t' ('=((t)
This hence completes the proof.
Lemma G.5. Choose +.
|A S ||xx + 3y
67
SATK
X(E
1)r
V
AAlog
(132)
I-7
1 7
Proof. When i mod E. 0, then A Ax(1)
43

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
ihen mod E # 0,
A Q" O
(133)
S Q" - ((1 -)Q + -or
(134)
= (1 - \ + A()ō(V") 70,-
(135)
(1 A)4 + - -V
(136)
= (1 - - + Ax(o(v")-
- -10)
1-1
= (1 - 1) X A x()) + Σ (1- ((M))
j=x(()
+ +4 Σ
(1 - (1- 1V)) of-1(V,)))
(137)
(=x(1)
We then bound the term 7 Σ
j=x(1) (1 - (B(V-) - 1
Note that
a(V") (V*)
= =((V)) - (V")) + (V*) - (V*)
< 1-7 + ptv* ph 1
SAKT
log 5
(138)
where the last inequality due Hoeffding's inequality. For any state-action pair (x, a),
[11 - x(1- x(0)
(x)) 51 -x)=-((()) Axtol.
(139)
In addition, we have
1-1
i-1
Σ j=x()
<n Σ (1 (1-
(140)
j=x(1)
Combining the bounds in (139), (138), and (140), we get
A
[-1
< (1 - x)0
IA
x
2 A(E
-(1 1)T + 7\
Σ
(1- (1--)-1-11A
1-
f=x()
SAKT 5
+
Aog
(141)
The remaining proofs similarly follows the one for Lemma D.3.
Theorem G.6 (Convergence). Choose <
1-7 37X and S 1. It holds that
4 1
2,2
| < (1-7)2
exp(
V1 x)XT) + (1 71- (6X2 1)2 + A(E - 1))r.
2
44

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
Proof. By Lemma G.2
K
|Atil (1-1) + 7111
Σ (1
1)-=
Σ
i=0
K
+ 1 Σ (1-
(p v PV
K
Since 0 S Qu(s,a) a) $ 1- the first term can be bounded
(1 (1= s 5 (1--))tt
(142)
1-7
To bound the term 1 Eio
1)--1
(ot)) - (V/+)),, we divide the summation into two parts as follows.
R
For any BE S t S T, we have
k
21
Xx1l
1
Σ
(V+) (V
K
1=0
k=1
x(t) BE
K
Σ
(1- - X
* (V+) - o² (V,2))) |||l
=
K
1=0
k=1
1 K
Σ
(1- -1-1)(m xy
o+ - (((b)))]
K
-xE+1
<
7 -2)=xx(0+88
Σ
(1- a) 2 K -
1 7-7
x -
By Lemma G.4,
1 K
Σ
(V*) - (V())))
K
k=1
< Axool 22 1
Σ
Σ
|
71
1
max
Σ
(V (V*),
K
K
9,
x(=
hence
Σ
(1 2) -
Xy11 Σ (V*) (V
xx(1)-8E++
k
Σ
(1-1)' 1)
(I4x +
Σ
Σ x=(()x((t)
|لحم|
K
-B
K
1-1 Σ
(o
- (
max
5.a
1=1
==x(1)
<
Σ
(1 2)*
(IAxcoll + -21
Σ
K
1-1
|
+ 7A4
E-1
+
1->V
1 V
(E
1)
SAKT
K
1-7
1-7V
8
-
k=1+==((t)
45

---

Pessimism Principle can Be Effective: Towards Framework for Zero-Shot Transfer Reinforcement Learning
By Lemma G.5, we have that
Σ
t
(1-2)
1
K
Σ
1-1
K
i=x(t)-BE++
1= x=x(()
$2127
1) i 1
K i-1
Σ
||xxs)]|
37
X(F - 1)T +
67
Alog
SATK
K
1-7
1 1-9
5
(-x(t)-BE++
k= j (i)
12X
(E 1)// x log SATK
S 22 1) 1)
max x(0)BBSisi
||x(ح +
1-7 (E - 1) + 1-7 7
8
Similarly, we get the following recursion holds for all rounds T:
|1+1l
2
72
S 71 + 21(E - 1])
max
1A x(1) + 1-7 7 (1 111(1-
(1-1)35
1-9 1 (622 - 11)² + -EE
x(t)-BESiS)
AVE-1 -
1 1)
1
A
SATK
C
log
1-7
1-9
1 7
K
Similarly, choosing B = 1 (1-))T
-= IV
AT
1. 1+11 1 = T, we get
AT||
242
S (1 7)² expl - 2 ((1 - )) +
(1 (1- 7)2 (6A2 - 1)) + +AEE - T))
(E- 1)
VE 18-11 11
+ V1
SATK
+
log 8
(1-
(1 - 7}²
(1 (1-))/k))
Corollary G.7. Choose (E (B-1) 1) S min X 11 } and x 4log² I(1-+)
Let T 2 E. Then with probability at least 1 8
holds that
C1
C2 log" (TK)) E 1
C3 log (TK)
SATK log
|r S (1- (1->)2T
(1 7)3
T
(1 73)
VTK
46


Reviews:
[{"review_id": "bzKy8Q5Tq5", "reviewer": "Reviewer_rfmi", "summary": "The paper introduces a novel framework for zero-shot transfer reinforcement learning (RL) based on the pessimism principle. The key idea is to construct a conservative proxy for the target domain's performance, ensuring that the transferred policy achieves a robust lower bound on performance while avoiding negative transfer. The authors propose two types of conservative proxies: \n(1) an averaged operator-based proxy and (2) a minimal pessimism proxy. They develop distributed algorithms for optimizing these proxies, with convergence guarantees. The framework is shown to be robust against model uncertainties and scalable to large-scale problems. Experiments on the recycling robot and HPC cluster management problems demonstrate that the proposed methods outperform non-robust baselines, especially in scenarios with domain shifts and model uncertainty.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and feedback, and appreciate the reviewer identifying our contribution and novelty. Please refer to the link https://anonymous.4open.science/r/ICML-2663/README.md for our newly conducted experiments.\n\n**Other relevant robust RL approaches** \nWe thank the reviewer for pointing these methods out. We first want to highlight that our method is indeed based on distributionally robust RL, where our uncertainty set is constructed to account for the potential distributional shift. The adversarial robust RL, although commonly studied in experiments, generally lacks theoretical guarantees on, e.g., convergence, and tends to be overly conservative. Our methods, however, enjoy both convergence and performance guarantees. We will include more discussion in our final version. \n\nWe then numerically verify that our method is effective, and present the results for the recycling robot problem in Table 2 in the above link. We use adversarial action-robust RL [1] and distributionally robust RL [2] as new baselines. As the results show, although adversarially robust and distributionally robust RL outperform\nnon-robust methods, their performance remain significantly inferior to ours. \n\n[1] Tessler et al., 2019. \"Action Robust Reinforcement Learning and Applications in Continuous\nControl.\"  \n [2] Panaganti \\& Kalathil, 2022. \"Sample Complexity of Robust RL with a Generative Model.\"\n\n**Cost of MLMC**\nMLMC is primarily designed for proof under the tabular settings, and is essential for obtaining an unbiased update. As we discussed, the computational cost of MLMC can be viewed as a trade-off of transfer performance, thus such a high cost may be acceptable in safety-critical applications such as robotics and autonomous driving. \n\nIts cost can be reduced along several potential directions: (1) The level number $N$ of MLMC can be controlled through techniques like threshold-MLMC [1], which applies an upper bound on $N$. Although it results in a biased estimation, the bias can be controlled and hence still implies convergence (numerically verified in Fig 3 in the link). (2) Techniques to reduce the computational cost of the robust Bellman operator can also be applied. One potential approach is to relax the uncertainty set constraint, which would result in a conservative and efficient solution [2]. (3) Reducing the aggregation frequency by increasing $E$ is also another way to reduce the computational cost, but introduces a trade-off in the convergence rate, as we have shown in our theoretical results (further numerically shown in Fig 2 of the link). \n        \n[1] Wang, Y. et al. Model-Free Robust Reinforcement Learning with Sample Complexity Analysis, 2024  \n[2] Kumar, N.et al. Policy gradient for rectangular robust Markov decision processes, 2023\n \t\n**No robustness tests** We developed some robustness testing experiments in Appendix, where our methods are shown to be more robust to target domain uncertainty. We also conduct additional experiments to verify robustness. Firstly, as the results in Table 2 in the link show, our methods are more robust to uncertainty in the target domain. Under different levels of uncertainty, our methods outperform baselines and is hence robust. Secondly, in Fig 3, we showed that even with noise or bias in the update, our algorithms still outperform the baseline and are also robust.\n    \nWe will include more robustness tests in the final version.\n\t\n**Assuming prior knowledge of domain similarity** Generally, such knowledge can be obtained through domain experts, or estimated from a small amount of target data. \n\nMoreover, our method remains applicable even without prior knowledge, by setting $\\Gamma$ to a known upper bound on distributional distance (e.g., total variation between any two distributions is at most 1). While this yields an over-conservative proxy, it still helps prevent significant drops in transfer performance due to our built-in pessimism principle. We believe this is the most reliable approach in settings where no similarity information is available.\n\nWe also note that this limitation applies to other methods as well. For instance, domain randomization (DR) assumes prior access to a set of environments that includes the target domain (Chen et al., 2021). Without such prior knowledge, our pessimism principle helps prevent undesirable outcomes, whereas DR provides no such guarantees.\n     \n**Can ($\\zeta$) be adaptively tuned?**    \nWe first note that $\\zeta$ depends on $\\Gamma$ in our algorithm. Since we consider a zero-shot setting, $\\Gamma$ is pre-set and fixed, and it cannot be tuned without any additional information. However, if any additional knowledge is available, e.g., a small amount of target domain data, or we are allowed to fine-tune via exploration, we can use it to shrink the uncertainty radius and enhance the effectiveness.\n\n**Negative transfer** We sincerely refer the reviewer to our response to Reviewer QZdX. The results are in Fig 1.", "claims_and_evidence": "- **Claim 1**: The pessimism principle ensures a robust lower bound on target domain performance and avoids negative transfer.\n  - **Evidence**: Theoretical analysis (**Lemma 4.1**) shows that the sub-optimality gap depends on the level of pessimism\\( ||\\zeta|| \\), and experiments consistently outperform non-robust DR baselines (Figure 3).\n- **Claim 2**: The proposed algorithms are efficient and scalable., enabling decentralized learning across multiple source domains (Partially supported).\n  - Theorems 5.5 & 6.4: Provide convergence guarantees for both MDTL-Avg (Averaged Proxy) and MDTL-Max (Minimal Pessimism Proxy).  \n  - Algorithm 1 is designed for distributed training, ensuring privacy-preserving updates.\n  - **Concern**: The multi-level Monte Carlo (MLMC) method is computationally expensive, which may limit scalability in high-dimensional RL tasks.\n- **Claim 3**: The framework is robust to model uncertainty.\n  - Proposition 7.1 suggests that pessimistic proxies provide robustness against perturbations in the target domain, implying distributional robustness.  \n  - **Concern**: No explicit robustness tests (e.g., varying noise levels, adversarial perturbations) are conducted.", "methods_and_evaluation_criteria": "The authors propose two robust transfer proxies:  \n- Averaged Operator-Based Proxy (AO): Uses a weighted combination of source domains' robust value functions to construct a conservative but smooth estimate.  \n- Minimal Pessimism Proxy (MP): Uses a max-aggregation approach to prioritize similar domains while maintaining pessimistic guarantees.  \nThe algorithms are federated-style, designed for distributed policy learning without direct data sharing, making them privacy-friendly and applicable in decentralized systems.  \n\nThe experiments are conducted on two specific problems:  \n- **Recycling Robot Problem**: A classic RL problem where a robot with a rechargeable battery must decide whether to search for cans or wait for someone to bring them. This is a relatively simple environment with discrete states and actions. \n- **HPC Cluster Management Problem**: A task where a cluster manager must decide whether to allocate incoming tasks immediately or enqueue them for later processing. This is also a discrete environment.\n\nThe evaluation focuses on performance in the target domain and robustness to model uncertainty.\n  \nThe paper compares against domain randomization (DR) methods, which is a reasonable baseline for zero-shot transfer RL. However, it ignores other relevant robust RL approaches, such as:\n- Adversarial robust RL (e.g., worst-case minimax RL), which directly optimizes for robustness against worst-case scenarios.\n- Distributionally robust RL (DRO) approaches, which handle uncertainty by optimizing policies under distributional shifts.", "theoretical_claims": "I verified the key theoretical claims, particularly:\n\n**Lemma 4.1** (Pessimism Gap & Sub-Optimality Bound)  \n   - Correctly characterizes the effect of pessimism on target policy performance.  \n   - Uses standard value function contraction properties to establish robustness guarantees.  \n\n**Theorem 5.2** (Averaged Proxy is Conservative)  \n   - Proves that the AO proxy remains a valid lower bound to the target domain’s value function.  \n\n**Theorem 6.1** (Minimal Pessimism Proxy is More Effective)  \n   - Assumes prior knowledge of domain similarity, which is non-trivial in real-world applications.", "experimental_designs_or_analyses": "The experiments are conducted on two specific problems:  \n- **Recycling Robot Problem**: A classic RL problem where a robot with a rechargeable battery must decide whether to search for cans or wait for someone to bring them. This is a relatively simple environment with discrete states and actions. \n- **HPC Cluster Management Problem**: A task where a cluster manager must decide whether to allocate incoming tasks immediately or enqueue them for later processing. This is also a discrete environment.\n\n### **Strengths**  \n- Well-structured comparison against proximal domain randomization (DR) baselines.  \n- Clear ablation study on uncertainty levels (Table 1, Figure 2).  \n\n### **Concerns**  \n- No explicit negative transfer tests: Does not include cases where misleading source domains degrade performance.", "relation_to_broader_scientific_literature": "This work connects to several research directions:\n\n1. The framework extends **robust RL** (Iyengar, 2005; Nilim & El Ghaoui, 2004) and **transfer RL** (Chen et al., 2021) by incorporating pessimism-based transfer guarantees.  \n2. The **federated learning** approach aligns with distributed RL (Jin et al., 2022; Wang et al., 2024).", "essential_references_not_discussed": "1. **Adversarial Robust RL:**  \n   - Rajeswaran et al., 2017. \"EPOpt: Learning Robust Neural Network Policies Using Model Ensembles.\"  \n   - Tessler et al., 2019. \"Action Robust Reinforcement Learning.\"  \n\n2. **Distributionally Robust RL:**  \n   - Wiesemann et al., 2014. \"Distributionally Robust Convex Optimization.\"  \n   - Panaganti & Kalathil, 2022. \"Sample Complexity of Robust RL with a Generative Model.\"", "other_strengths_and_weaknesses": "**Strengths:**  \n- Novel use of pessimism in zero-shot transfer RL, the pessimism-based approach is well-motivated  \n- Strong theoretical contributions with formal performance guarantees.  \n- Federated-style implementation ensures privacy in multi-domain learning.  \n\n**Weaknesses:**  \n- Computational cost of MLMC aggregation not addressed  \n- Assumes Prior Knowledge of Domain Similarity(which may not hold in real-world applications).  \n- No explicit negative transfer experiments.", "other_comments_or_suggestions": "The paper covers the most relevant experiment but could benefit from performing negative transfer experiments to validate the robustness of pessimistic aggregation.", "questions_for_authors": "1. How does the MLMC-based pessimistic aggregation scale to large state-action spaces?\n2. Can the pessimism level \\( \\zeta \\) be adaptively tuned instead of being fixed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qLwDbH583x", "reviewer": "Reviewer_QBMR", "summary": "This paper studies zero-shot transfer reinforcement learning. The authors incorporate a pessimism principle into transfer learning to serve as a lower bound to conservatively estimate the target domain’s performance. The authors propose and analyze two types of conservative estimates, rigorously characterizing their effectiveness, and develop distributed, convergent algorithms to optimize them.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and feedback, and appreciate the reviewer identifying our contribution. Please refer to the link https://anonymous.4open.science/r/ICML-2663/README.md for our newly conducted experiments.\n\n**Upper bound on $\\max_\\pi\\zeta^\\pi$.**\n\nWe first clarify that $\\|\\zeta\\|$ has an upper bound and can be controlled via the construction of the uncertainty sets. With our proxies, it holds that $\\|\\zeta\\|\\leq \\mathcal{O}((1-\\gamma)^{-2}\\Gamma)$. Moreover, when adapting our Max proxy with different radii $\\Gamma_i$, it holds that $\\|\\zeta\\|\\leq \\mathcal{O}((1-\\gamma)^{-2}\\min\\{\\Gamma_i\\})$. Hence the suboptimality gap can be upper bounded by the domain similarities, which is reasonable and subject to the nature of the problem. This dependence also motivates one of the potential extensions of our framework -- a safe-to-improve principle: once a conservative proxy is constructed, any improvement to it directly improves target performance, without the loss of theoretical performance guarantees. For instance, if additional information is available (e.g., a small set of target domain data or allowance of explorations), we can further reduce radii $\\Gamma$ to ensure a better transfer performance. \n    \n**Formally define the problem setting. It would be better to specify the setting and the assumptions needed explicitly in this section.**\n\nOur formulation is as follows. Consider the target domain $\\mathcal{M}\\_0 = (\\mathcal{S},\\mathcal{A},P\\_0,r,\\gamma)$ with the unknown target kernel $P_0$. We have no data from it, but instead have access to multiple source domains $\\mathcal{M}\\_k = (\\mathcal{S},\\mathcal{A},P\\_k,r,\\gamma)$. Our goal in zero-shot multi-domain transfer learning is to optimize the performance $V_{P_0}^\\pi$ under $\\mathcal{M}_0$. For simplicity, we consider the case of identical reward (please see our response to Reviewer QZdx for extensions). \n \nThe only assumption is that, there exists an upper bound $\\Gamma\\geq D(P_0||P_k)$, that is known by the learner. This assumption is reasonable, as such information is essential to obtain performance guarantees. Even if in the worst-case, we can set $\\Gamma=1$ (in total variation) to construct an (overly) conservative proxy, which yet still avoids decisions with severe consequences in the target domain, preferred in transfer learning settings.   \n    \n \n**Theorem 7.2. seems to be too abrupt to appear, the $\\Delta_T$ is used without definition.**\n\nWe apologize for missing the definition of $\\Delta_T$. It is the difference between the algorithm output and the fixed points of the aggregated operators: $\\Delta_T := Q_{\\rm AO} - Q^T$ for MDTL-Avg, and $\\Delta_T := Q_{\\rm MP} - Q^T$ for MDTL-Max. We will clarify these.  \n \n\t\n**The proposed methods borrow some ideas from federated learning, it would be better to briefly introduce some related work in federated learning in the Related Work section (in appendix A. Additional Related Works).**\n\nWe thank the reviewer for the helpful suggestion. We will include related work on federated reinforcement learning (FRL). We note, however, that directly extending FRL to our setting poses non-trivial challenges: (1) FRL has linear update rules (non-robust operators), while ours involve non-linear robust operators; (2) FRL focuses on optimizing average performance across local environments, whereas we address the more challenging max-based multi-domain transfer; and (3) more importantly, FRL is not designed for transfer learning, while our pessimism principle enables transfer with theoretical guarantees.\n \n\n    \n**In Proposition 5.4., you prove the results under total variation or Wasserstein distance, do similar results hold under other metrics?**\n\nAccording to our proof, the result holds as long as $\\sigma\\_{\\bar{\\mathcal{P}}}(V) - \\frac{1}{K}\\sum_{k=1}^K\\sigma\\_{\\mathcal{P}\\_k}(V) \\leq 0$. We note that additional for $l_p$-norm, the support function has a duality (Clavier et al., 2024. Near-Optimal Distributionally Robust Reinforcement Learning with General $L_p$ Norms.): $\\sigma(V)=\\max_{\\alpha} ( PV\\_{\\alpha}+f(\\Gamma,V\\_{\\alpha}))$ for some function $f$. Thus $\\sigma\\_{\\bar{\\mathcal{P}}}(V) - \\frac{1}{K}\\sum_{k=1}^K\\sigma\\_{\\mathcal{P}\\_k}(V)=\\max\\_{\\alpha} ( \\bar{P}V\\_{\\alpha}+f(\\Gamma,V\\_{\\alpha}))-\\frac{1}{K}\\sum_i \\max_{\\alpha} ( P_iV\\_{\\alpha}+f(\\Gamma,V\\_{\\alpha}) )\\leq 0$. It is our future interest to extend to other uncertainty sets.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "No.", "relation_to_broader_scientific_literature": "This paper is related to transfer learning and federated learning. The authors propose to optimize a lower bound of the objective in the transfer reinforcement learning, and utilize some ideas from the federated learning literature.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper proposes a new framework based on the pessimism principle, which constructs and optimizes a conservative estimation of the target domain’s performance.\n2. The paper is well-written and easy-to-follow.\n3. The authors provide some upper bounds for the performance of their proposed algorithms. They also provide experimental results to support the theoretical findings.\n\nWeaknesses:\n1. The main concern is that the suboptimality gap is indeed dependent on the $\\max_{\\pi}\\zeta^\\pi $. Though optimizing the lower bound is a good solution and the authors provide the convergence results to the lower bounds, the gap $\\max_{\\pi}\\zeta^\\pi $ seems not controllable, meaning that there is no theoretical upper bound on the actual suboptimality gap. \n2. The section 2.1 can not formally define the problem setting. It would be better to specify the setting and the assumptions needed explicitly in this section.\n3. Theorem 7.2. seems to be too abrupt to appear, the $\\Delta_T$ is used without definition.\n4. The proposed methods borrow some ideas from federated learning, it would be better to briefly introduce some related work in federated learning in the Related Work section (in appendix A. Additional Related Works).", "other_comments_or_suggestions": "See the weaknesses above.", "questions_for_authors": "1. Regarding my concern in the weaknesses 1. above, could you please give some explanations?\n2. In Proposition 5.4., you prove the results under total variation or Wasserstein distance, do similar results hold under other metrics?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rOiQim613l", "reviewer": "Reviewer_tyX8", "summary": "The paper introduces a novel pessimism-based transfer learning framework to address critical challenges in zero-shot transfer RL. The authors propose constructing conservative proxies—via robust Bellman operators and novel aggregation schemes (both averaged and minimal pessimism operators)—that yield lower bounds on the target performance. In addition, they develop distributed algorithms with convergence guarantees.\n\n**Update after rebuttal**--I appreciate the authors’ thorough revision and the additional experiments addressing my concerns. I hope those will be implemented in the revised draft. I have decided to keep my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and feedback, and appreciate the reviewer identifying our contribution. Please refer to the link https://anonymous.4open.science/r/ICML-2663/README.md for our newly conducted experiments.\n\n**Hyperparameter sensitivity; Benchmark like contextual MDPs.**\nWe appreciate the reviewer’s suggestion. \n\nTwo hyperparameters are included: E and $\\lambda$. \n1. We implement algorithms with different E in Fig 2, Table 1. A larger E results in slower convergence but less computation, matching our theoretical results.\n2. We run experiments with $\\lambda=0.2$ in Table 2. Together with results in our paper for 0.1, it shows our methods are uniformly better and robust to it.\n\nWe present 2 more experiments. (1). Cartpole Experiment (Fig. 3): We apply our methods to the Cartpole environment, a common example of a contextual MDP. Results show that even when updates introduce bias, our methods outperform the baseline, confirming robustness. (2). Dynamic Vehicle Routing Problem (DVRP): We design a real-world scenario—DVRP [Iklassov, Z, et al., Reinforcement Learning for Solving Stochastic Vehicle Routing Problem, 2024]—which extends the classic vehicle routing problem with real-time requests and environmental uncertainty. Three important objectives are considered: Minimizing overall routing cost, Enhancing route smoothness, and Improving route stability. As shown in Table 3, our methods consistently outperform baselines across all criteria, and are more effective.\n\nWe will add more experiments in the final version.\n \n**Additional references.**\nWe thank the reviewer for recognizing our contributions and providing constructive suggestions.\n\nWe briefly discussed contextual RL (cRL) for zero-shot generalization in Appendix A. cRL seeks to optimize performance under a contextual distribution to enable transfer without fine-tuning. However, theoretical guarantees remain limited. For instance, [1] provides a suboptimality bound, but it requires samples from the contextual distribution, potentially including target domain data. Furthermore, it remains unclear whether cRL can mitigate negative transfer, especially when the context distribution is assumed to be uniform.\nWe also appreciate the reviewer’s mention of safe and risk-sensitive RL, which aim to optimize reward while maintaining low cost under specific criteria [2]. However, most of these works are developed for single-environment settings, and do not address transfer scenarios. We will incorporate a more thorough discussion in the final version.\n\n[1] Wang, Z., et al., Towards Zero-Shot Generalization in Offline Reinforcement Learning, 2025\n[2] García, J., et al., A Comprehensive Survey on Safe Reinforcement Learning, 2015\n\n**Intuition or summarizing remarks.**\nThank you for the suggestion. Our algorithm consists of two main parts: local updates and global aggregation. Each agent independently updates its local Q-table using its data, and after every E steps, a global aggregation step unifies the local Q-tables. Our convergence proof mirrors this structure: it is decomposed into a local part (governed by the local Bellman operator) and a global part (governed by the aggregation mechanism). We will include further discussion in the final version.\n    \n**Unbiased Estimation Assumption.**\nThe unbiased estimation assumption is introduced to simplify convergence analysis. However, our convergence results can be extended to settings with controlled bias, such as the bias introduced by max aggregation. This bias can be mitigated using techniques like threshold-MLMC [1].\nImportantly, even in the presence of bias, if the expected proxy remains pessimistic, our transfer learning framework and theoretical guarantees still hold. We thus argue that our method is robust to such biases.\nTo support this, we include an experiment on Cartpole (Fig. 3) analyzing the impact of bias. Results show that our proxy remains conservative, and the pessimism-based framework continues to outperform the baseline, confirming robustness.\n\n[1] Wang, Y., et al., Model-Free Robust Reinforcement Learning with Sample Complexity Analysis, 2024\n\n**Comparisons with baselines.**\nAs noted in the paper, our primary advantage lies in theoretical guarantees: we optimize a lower bound on the target domain’s performance, avoiding overly optimistic decisions that may fail under domain shift. In contrast, domain randomization (DR) lacks such guarantees and can be overly optimistic, especially when the randomized training environments fail to capture true uncertainty.\n\nWe acknowledge that when the uncertainty set is constructed using inaccurate prior knowledge, our method may become overly conservative, potentially leading to suboptimal performance compared to DR. Nonetheless, the ability to guarantee robust performance across all scenarios is a central contribution—especially important for safety-critical or high-stakes applications, where conservativeness is not only expected but necessary.", "claims_and_evidence": "The paper claims that incorporating a pessimism principle can yield a conservative proxy and the proposed distributed algorithms are computationally efficient and scalable. These are supported by theoretical proofs of contraction properties. However, empirical evidence is deficient to some extent.", "methods_and_evaluation_criteria": "This paper will largely benefit from widely used benchmark datasets. One example could be contextual MDP benchmarks.", "theoretical_claims": "I haven’t checked all the details in proof, but the theoretical contributions are both novel and rigorous. Some proofs could benefit from additional intuition or summary comments to aid understanding.", "experimental_designs_or_analyses": "The experimental design is sound; however, a discussion on hyperparameter sensitivity and empirical analysis in many different benchmarks could further strengthen the work.", "relation_to_broader_scientific_literature": "The paper is well situated within the broader literature:\n- It builds on established ideas in robust RL, domain randomization, and multi-task learning.\n- It addresses limitations in existing transfer RL methods (no guarantee in safety and performance when transferred) by providing performance guarantees and methods to avoid negative transfer.", "essential_references_not_discussed": "There are some papers discussing zero-shot transfer in the context of contextual reinforcement learning. I highly recommend the authors to include them in related works. Furthermore, a brief discussion of recent advances in safe RL and risk-sensitive RL (which also aim to avoid over-optimistic policies) could further contextualize the pessimism principle. However, the omission does not detract significantly from the paper’s contributions.", "other_strengths_and_weaknesses": "- The paper offers a thorough theoretical justification with detailed proofs. However, some proofs and algorithmic descriptions could benefit from additional intuition or summarizing remarks.\n- The distributed algorithm design and privacy-preserving updates are well aligned with modern large-scale and decentralized applications. While simulations are convincing, experiments on real-world tasks would further strengthen the empirical claims.", "other_comments_or_suggestions": "- I didn’t find any significant typos.\n- Font size in figure is too small.", "questions_for_authors": "- In Table 2, it’s hard to see difference between MDTL-Avg and MDTL-Max. Please provide other examples that could distinguish these two.\n- Actually, more experiments should be done to support this.\n- I’m curious how critical unbiased estimation assumption is and how robust is the method to the assumption.\n- I’m curious about the comparison with other methods that authors mentioned throughout the papers. How does this approach perform better than others both in your theory and  experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3HpSIKSYV", "reviewer": "Reviewer_QZdX", "summary": "This paper studies transfer learning where one aims to learn a good policy for a target domain with data collected from multiple source domains. And they consider the distributed and decentralized setting, where one central server can only access partial data from source domains. The authors apply the principle of pessimism in this problem. Specifically, their algorithm estimates a pessimistic evaluator which lowers bounds the performance of the policy in the target domain by utilizing an average of robust Bellman operators. With this conservative estimator, they can provide performance guarantees of their learned policy. What's more, they provide a minimal pessimism operator with which they can mitigate the issue of negative learning which previous methods may suffer in transfer learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and feedback. Please refer to the link https://anonymous.4open.science/r/ICML-2663/README.md for our newly conducted experiments.\n\n**Negative transfer**\nOur initial environments were relatively simple, so negative transfer was not evident. To support our claims, we ran a new Cartpole experiment (Fig 1). The source domains are with randomly generated parameters (length of pole), and the target is the default one. As shown, MDTL-Avg suffers from negative transfer, while MDTL-Max successfully mitigates it. We will elaborate further in the final version.\n\n**Experiments on E and robust baselines** \nWe validate the trade-off in Fig. 2 and Table 1. As E increases, convergence slows (under fixed step size) and performance declines. However, computational time also decreases, indicating reduced communication. \n\nWe also add two robust baselines (see Table 2 and reply to R rfmi).\n\n**Identical reward assumption**\nWe made this assumption to simplify presentation and analysis, but our framework can extend to settings with differing rewards.\n\n1. If the (unknown) target domain reward differs from the sources, additional knowledge on its relation to the source reward should be known, e.g., $|r_t(s,a)-r_s(s,a)|\\leq R(s,a)$, otherwise the target may have different goals with source domains, making transfer impossible. With such knowledge, we additionally define a **reward** uncertainty set for source: $\\mathcal{R}^k_{s,a}=\\{r': |r'-r(s,a)|\\leq R(s,a) \\}$ and modify the robust Bellman operator to $\\mathbf{T}^k(Q)=(r-R)+\\sigma(V)$, preserving pessimism and effectiveness.\n\n2. If source rewards also differ from each, we can still apply different local operator $\\mathbf{T}^k(Q)=(r^k-R^k)+\\sigma(V)$ to similarly ensure conservative.\n\nWe will discuss more in the final version.\n\n**Extended to the case with different similarities parameter in source domains**\nWe thank the reviewer for this insightful point. While most proofs are independent of the radius, some technical adjustments are needed. For example, in extending Thm. 6.4, we can adjust (89) to yield a bound $\\gamma\\lambda\\frac{E-1}{1-\\gamma}\\max_i\\{\\Gamma_i\\}$. We will discuss this further in the final version.\n\n**Claim on the scalability**\nThank you for this valuable comment. We will revise the claim. By scalability, we refer to the local updates, which can leverage scalable, model-free methods. While global aggregation is necessary in tabular Q-learning, our approach is not limited to this setting. Our method can be paired with function approximation or policy gradient approaches to improve scalability. For instance:\n1. With function approximation for robust RL [1], or\n2. Robust policy gradient [2], the global step only requires parameter aggregation (e.g., Jin et al., 2022), making it scalable in practice. \nWe further numerically verify this by develop experiments under Cartpole and DVRP (please see our response to #tyX8). We will revise the statement and leave further extensions for future work.\n\n[1] Zhou, R. et al., Natural actor-critic for robust reinforcement learning with function approximation, 2023\n[2] Wang, Y. & Zou, S., Policy gradient method for robust reinforcement learning, 2022\n\n \n**Proxies in Line 220**\nOur proxies outperform options (1)-(3) due to reduced conservativeness. Though proxy (4) is less conservative, it presents practical challenges:\n1. Its uncertainty set lacks a ball structure, making standard robust RL methods inapplicable.\n2. Intersection-based methods require accurate knowledge of each local model and set, which is not available via raw data communication alone.\nThus, our method has two advantages over (4): lower communication requirements and easier implementation.\n\n**E=1 in Theorem 5.6/6.4**\nWhen E = 1, the global Q-table is updated every step using $Q \\leftarrow \\frac{1}{K} \\sum_k \\mathbf{T}^k(V)$. This operator is a $\\gamma$-contraction, so convergence rate is independent of $\\Gamma$. Even in the presence of unbiased noise, the expected update preserves this rate. Technically, Eq. (34) bounds $\\|\\sigma(V) - \\sigma(V')\\| \\leq \\|V - V'\\| \\leq \\frac{1}{1 - \\gamma}$, independent of $\\Gamma$. However, $\\Gamma$ still affects the pessimism level $\\zeta$, which impacts the **effectiveness**, though not the **convergence rate**.\n\nWhen E > 1, agents perform multiple local updates with their own robust operators. The global update then reflects local differences, making it depend on $\\Gamma$. A similar effect is observed in Wang et al., 2024 (Theorem 1), where heterogeneity does not affect convergence when  E = 1.\n\n**Learn locally and average**\nWe apologize for the possible misunderstandings here. $Q_{AO}$ is the intended solution, and $Q_k$ is the algorithm’s output. The reviewer may have confused $Q_k$ with $Q^*\\_k$, the robust value function for local domain k. Importantly, $Q_{AO}$ is different from $\\frac{\\sum Q^*\\_k}{K}$, so simply learning locally and averaging does not yield the correct solution.", "claims_and_evidence": "One of the claims seems not clearly supported.\n\nThe authors made the following claims:\n\nThey introduce pessimism into transfer learning and based on that principle, they can construct the performance estimator of the learned policy which is a lower bound of the performance in the target domain. With this conservative estimator, they solve two problems existing in transfer learning, first, there is no performance guarantee of the learned policy in the target domain, second, the existing of source domains which are very distinct from the target domain may lead to negative transfer learning.\n\nThe claim on avoiding negative transfer is not supported clearly. They do show their algorithm can treat source domains differently, but I am expecting a theorem that can quantify how much performance gained or an example, that can show by treating all source domains equally, we suffer negative transfer but with the proposed algorithm, it is avoided.", "methods_and_evaluation_criteria": "Yes, it makes sense. The method proposed in this paper primarily builds on the idea of robust RL and q-learning. With robust RL, the authors build pessimistic performance estimator of the learned policy in the target domain which can lower bound the true performance. And with Q-learning, they propose practical algorithms to learn that pessimistic estimator.", "theoretical_claims": "I checked the correctness of Lemma 4.1, Lemma 5.1, Theorem 5.2 and Theorem 5.5. They seem correct to me.", "experimental_designs_or_analyses": "No, I cannot evaluate the soundness of the experimental designs. The authors implemented experiments on two tasks with different target domain uncertainty parameters and compared their algorithms with two baselines which are non-robust algorithms.", "relation_to_broader_scientific_literature": "The contributions in this paper are related to two main areas which are offline RL and robust RL.\n\nFirst, the key contribution is the application of the principle of pessimism to the problem of transfer learning which is related to offline RL where pessimism is widely used to provide high-probability lower bound estimator of Q-functions. In offline RL, the estimator is constructed using some concentration inequality. \n\nIn this paper, the authors construct the lower bound estimator via robust RL. They assume an uncertainty model set for each source domain where the target domain lies in. By finding the optimal policy in this uncertainty, they can lower bound the performance in the target domain.\n\nAnother broad related area is Q-learning. The algorithm proposed in this paper is a modified version of Q-learning in the framework of distributed transfer learning.", "essential_references_not_discussed": "I don't see essential related works that are not discussed. The discussion on robust RL which severs the base stone in this paper is placed in Appendix.", "other_strengths_and_weaknesses": "Strengths:\n1. The presentation is very clear which makes the paper easy to follow.\n\nWeaknesses:\n1. The results seem build on the assumption that the reward function is identical across all domains including source domains and the target domain. It restricts the scenarios where the algorithm is applicable. Correct me if my understanding is wrong.\n2. The authors comment in the main paper that their results can be directly extended to the case with different similarities parameter in source domains (which I agree), but it would be better to at least mention and supplement it in Appendix which I didn't find.\n3. The results are built on the tabular settings because of the average step of Q-values from source domains. Then the claim in line 384 on the scalability should be more conservative, though the authors specify the local update step (not global step) can be implemented by any scalable robust RL algorithm.", "other_comments_or_suggestions": "1. The authors specify a parameter $E$ in their algorithm which trades off the convergence rate and communication cost, it would be interesting to see the experimental results on varying $E$.", "questions_for_authors": "1. Is it true that the rewards of all source domains and the target domain are assumed to be same? If it is true, is there any way to get rid of it?\n2. I can see the advantage of the proposed algorithms in the distributed setting since only Q-values are communicated. What if the communication of any raw data between these source domains is safe and possible, do the proposed algorithms still outweigh those estimators defined in line 220-224?\n3. In Theorem 5.6 and 6.4, when $E=1$, on the right hand side, the similarity parameter $\\Gamma$ does not affect the convergence rate, why communicating at each round can get rid of the $\\Gamma$, it sounds unreasonable to me, is there any explanation?\n4. According to Theorem 5.6, the final $Q_{AO}$ is approaching $\\frac{\\sum_{k=1}^KQ _k}{K}$, why doesn't one directly learn a good $Q_k$ in each source domain since we have lots of data in each domain, and then take the average instead of doing it in an iterative way?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers appreciate the contribution of their paper. I also think it has nice theoretical contribution.
Decision: Accept (poster)