Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics
Authors: Ming-Hong Chen
Kuan-Chen Pan
You-De Huang
Xi Liu
Ping-Chun Hsieh
Venue: ICLR 2026 Poster (2026)
Keywords: Cross-domain reinforcement learning; transfer learning; Bellman consistency
Primary area: reinforcement learning

Abstract: Cross-domain reinforcement learning (CDRL) is meant to improve the data efficiency of RL by leveraging the data samples collected from a source domain to facilitate the learning in a similar target domain. Despite its potential, cross-domain transfer in RL is known to have two fundamental and intertwined challenges: (i) The source and target domains can have distinct state space or action space, and this makes direct transfer infeasible and thereby requires more sophisticated inter-domain mappings; (ii) The transferability of a source-domain model in RL is not easily identifiable a priori, and hence CDRL can be prone to negative effect during transfer. In this paper, we propose to jointly tackle these two challenges through the lens of \textit{cross-domain Bellman consistency} and \textit{hybrid critic}. Specifically, we first introduce the notion of cross-domain Bellman consistency as a way to measure transferability of a source-domain model. Then, we propose $Q$Avatar, which combines the Q functions from both the source and target domains with an adaptive hyperparameter-free weight function. Through this design, we characterize the convergence behavior of $Q$Avatar and show that $Q$Avatar achieves reliable transfer in the sense that it effectively leverages a source-domain Q function for knowledge transfer to the target domain. Through experiments, we demonstrate that $Q$Avatar achieves favorable transferability across various RL benchmark tasks, including locomotion and robot arm manipulation. Our code is available at https://rl-bandits-lab.github.io/Cross-Domain-RL/.

Published as a conference paper at ICLR 2026
CROSS-DOMAIN POLICY OPTIMIZATION VIA BELLMAN CONSISTENCY AND HYBRID CRITICS
Ming-Hong Chen Kuan-Chen Pan You-De Huang' Xi Liu²
Ping-Chun Hsieh
National Yang Ming Chiao Tung University, Hsinchu, Taiwan 2 "Applied Machine Learning, Meta AI, Menlo Park, CA, USA (mhchen1224. cs12, pinghsieh] @nycu. edu. tw
ABSTRACT
Cross-domain reinforcement learning (CDRL) is meant to improve the data effi- ciency of RL by leveraging the data samples collected from source domain to facilitate the learning in similar target domain. Despite its potential, cross-domain transfer in RL is known to have two fundamental and intertwined challenges: (i) The source and target domains can have distinct state space or action space, and this makes direct transfer infeasible and thereby requires more sophisticated inter-domain mappings; (ii) The transferability of a source-domain model in RL is not casily identifiable priori, and hence CDRL can be prone to negative effect during transfer. In this paper, we propose to jointly tackle these two challenges through the lens of cross-domain Bellman consistency and hybrid critic. Specifi- cally, we first introduce the notion of cross-domain Bellman consistency as a way measure transferability of source-domain model. Then, we propose QAvatar, which combines the Q functions from both the source and target domains with an adaptive hyperparameter free weight function. Through this design, we char- acterize the convergence behavior of QAvatar and show that QAvatar achieves reliable transfer in the sense that it effectively leverages source-domain Q function for knowledge transfer to the target domain. Through experiments, we demon- strate that Qvaatar achieves favorable transferability across various RL benchmark tasks, including locomotion and robot arm manipulation. Our code is available at https: h/tp://n////bbdiiiiiee www.bbannnsslllldeee lab. github. io/ Cross- Doma innnnnn-- R-RL/.
INTRODUCTION
Cross-domain reinforcement learning (CDRL) serves practical framework improve the sample efficiency of RL from the perspective of transfer learning, which leverages the pre-trained models from source domain enable knowledge transfer to the target domain, under the presumption that the data collection and model training are much less costly in the source domain (e.g., simulators). A plethora of the existing CDRL methods focuses knowledge transfer across environments that share the same state-action spaces but with different transition dynamics. This setting has been extensively studied from a variety of perspectives, such as reward augmentation (Eysenbach al., 2021; Liu et al., 2022), data filtering (Xu et al., 2023), and latent representations (Lyu et al., 2024). Despite the above progress, to fully realize the promise of CDRL, there are two fundamental challenges to tackle: (i) Distinct state and/or action spaces between domains: To support flexible transfer across wide variety of domains, the generic CDRL required to address the discrepancies in the state and action spaces between source and target domains. Take robot control as example. One common scenario is to apply direct policy transfer between robot agents of different morphologies (Zhang al., 2021), which naturally leads to a discrepancy in representations. This discrepancy significantly complicates the transfer of either data samples or learned source-domain models. (ii) Unknown transferability of source-dor -omain model the target domain: CDRL conventionally presumes that the source-domain model can achieve effective transfer under a properly learned cross-domain correspondence. However, in practice, given that the data budget of the target domain is limited, it is rather difficult to determine a priori the transferability of a source-domain model. Indeed, it has
"Equal contribution.

---

Published as a conference paper at ICLR 2026
been widely observed that transfer learning from the source domain can have negative impact on the target domain (Weiss et al., 2016; Pan & Yang, 2009).
As consequence, despite that CDRL has been shown to succeed in various scenarios, without proper design, the performance of CDRL could actually be much worse than the vanilla target-domain model learned without using any source knowledge. Notably, to tackle (i), several approaches have been proposed to address such representation discrepancy by learning state-action correspondence, either in the typical RL (You al., 2022) unsupervised settings (Zhang al., 2021; Gui al., 2023). However, existing solutions are all oblivious to the issues of model transferability between the domains. Hence, one fundamental research question about CDRL remains largely open:
How to achieve effective transfer in CDRL under distinct state-action spaces without the knowledge of the transferability of the pre-trained source-domain model?
In this paper, we affirmativeyy address the above question by revisiting cross- coss-domain state-action correspondence through the lens of cross-domain Bellman consistency, which quantifies the transfer- ability of a source-domain model. To enable reliable transfer across varying levels of source-model transferability, we introduce novel CDRL framework, QAvatar, which integrates source-domain and target-domaii critics. Drawing an analogy from the movie Avatar, where humans remotely control genetically engineered bodies to adapt to alien environments, QAvatar updates the target-domain policy via a weighted combination of the target- and source-domain Q functions, while learning the state-action correspondence by minimizing cross- domain Bellman loss.
To validate this idea, we first present a tabular prototype of QAaatar and show that it attains a tight sub-optimality bound under adaptive, hyperparameter-fiee weight function, regardless of source model transferability. This ensures improved sample efficiency while avoiding poor transfer. Building on this, we develop a practical version by combining QAvatar with normalizing flow-based mapping for learning state-action correspondence.
The main contributions of this paper can be summarized as follows: We propose the QAvatar framework that achieves knowledge transfer between two domains with distinet state and action spaces for improving sample efficiency. We then present a prototypical QAvatar algorithm and establish its convergence property. 2) We further substantiate the Q Qvatar framework by proposing practical implementation with a normalizing-flow -lowbbb-ed state-action mapping. This further demonstrates the compatibility of Qvaata with off-the-shelf methods for learning state-action correspondence. 3) Through experiments and an ablation study, we show that QAvatar outperforms the CDRL benchmark algorithms various RL benchmark tasks.
RELATED WORK
CDRL across domains with distinct state and action spaces. The existing approaches can be divided into two main categories: (i) Manually designed latent mapping: In (Ammar & Taylor, 2012; Gupta et al., 2017; Ammar et al., 2012), the trajectories are mapped manually from the source domain and the target domain then be calculated to find the correspondence of the states from the different domains. (ii) Learned inter-domain mapping: (Taylor et al., 2008; Zhang et al., 2021: You et al., 2022: Gui al., 2023; Zhu et al., 2024), the inter-domain mapping is mainly learned by enforcing dynamics alignment (or termed dynamics cycle consistency in (Zhang al., 2021)). Additional properties have also been incorporated as auxiliary loss functions in learning the inter-domain mapping, including domain cycle consistency (Zhang et al., 2021), effect cycle consistency (Zhu al., 2024). maximizing mutual information between states and embeddings (You et al., 2022) However, the existing approaches all presume that the domains are sufficiently similar and do not have any performance guarantees. By contrast, we propose reliable CDRL method that can achieve transfer regardless of source-domain model quality or domain similarity with guarantees.
common latent space. The distance between latent states can
CDRL across domains with identical state and action spaces. Various methods have been proposed for the case where source and target domains share the same state and action spaces but are subject to dynamics mismatch. Existing methods include (i) using the samples from both source and target domains jointly for learning (Eysenbach et al., 2021; Liu et al., 2022; Xu et al., 2023; Lyu et al., 2024), (ii) explicit characterization of domain similarity (Behboudian et al., 2022; Sreenivasan et al.,
2

---

Published as a conference paper at ICLR 2026
2023), and (iii) using both Q-functions for Q-learning updates (Wang et al., 2020). However, given the assumption on identical acttinaction spaces, they are not readily applicable to our CDRL setting.
3 PRELIMINARIES
In this section, we provide the problem statement and basic building blocks of CDRL, as well as the useful notation needed by subsequent sections. For set X, we let ((Y) denote the set of probability distributions over X. As in typical RL, we model each environment as an infinite-horizon discounted Markov decision process (MDP) denoted by M == (S, 7, M), where (i) S and A represent the state space and action space, (ii) P: S x A + (S) denotes the transition function, (iii) r S A - [0, 1] is the reward function (without loss of generality, lie in the [o. 1J interval), (iv) E [0, 1) is the discounted factor, and (v) Ja E A(S x A) denotes the initial state-action distribution. Notably, the use of an initial distribution over states and actions is standard setting in the literature of natural policy gradient (NPG) (Agarwal et al., 2021a; Ding et al., 2020; Yuan al., 2022; Agarwal al., 2020; Zhou et al., 2024). Given any policy
presume the rewards
S A(A). let T = (So, op, r,,-..) denote a (random) trajectory generated under x in M, and the expected total discounted reward under 'T is (M) == E E y'r(s,, [ IT; Su, ag ~ pl.. We use QM a)
and VM to denote the Q function and value function of policy x. We also define the state- action visitation distribution (also known as the occupancy measure in the MDP literature) of n as d' (8. (1 - 7) (#((,)) E P
5, a2 a; T. [)), for each a).
Problem Statement of MDPs, namely the source-domain MDP Mse := : Psa:: T's Y Mate) and the darget-domain MDP Mar := (Str: Acar, Par- Tuar: Y Maar). Notably, in addition to distinct state and action spaces, the two domains can have different reward functions, transition dynamics, and initial distributions. We assume that the two MDPs share the same discounted factor y which is rather mild. Moreover, the trajectories of the two domains are completely unpaired. Let I be the set of all stationary Markov policies for Mar-
RI., typical CDRL, the knowledge transfer involves two
The goal of the RL agent is to learn a policy x" in the target domain such that the expected total discounted reward is maximized, i.e., n" == arg max Vin,, (Har)) )- To improve sample efficiency learning from scratch), in CDRL, the target-domain agent is granted access to (Tare, Que, Vsc). which denotes a policy and the corresponding Q and value functions pre- pre-trained in Mare. Notably, we make no assumption on the quality of Tare (and hence Tue may not be optimal to .Max), despite that Tue shall exhibit acceptable performance in practice.
via knowledge transfer (compared
In this paper, we focus on designing a reliable CDRL algorithm in that it effectively leverages a source-domain Q function Q.n. for knowledge transfer to the target domain, regardless of the quality of Que and domain similarity.
Inter-Domain Mapping Functions. To address the discrepancy in state-action spaces in CDRL, learning an inter-domain mapping is one common block of many CDRL algorithms. Specifically, there are a variety of ways to construct the mapping functions, such handcrafted functions (Ammar & Taylor, 2012), encoders and decoders trained by cycle consistency (You et al., 2022) like cycle- GAN (Zhu et al., 2017), neural networks trained by dynamics alignment of the MDPs (Gui et al., 2023). Moreover, mapping functions have various candidate target spaces, such as a latent space, state or action spaces of the target domain (i.e., from Saie Aste to S Aua)) and state or action spaces of the source domain (i.e., from Sur. Aua to Sac. Ase).
For example, Gui et al. (2023) proposed learning two mappings, G Star + Suc and G2 Asre + Aur. via dynamics alignment, which infers the unknown mapping between unpaired trajectories of Max and Mur by aligning one-step state transitions. However, this unsupervised approach provides no performance guarantee and can suffer from identification issues. By contrast, propose learning inter-domain state and action mappings, Φ Stn -> Sare and 1/s Auar - Awe. using a cross-domain Bellman-like loss with guarantees (Section 4). Appendix D.1 shows toy example where eycle consistency fails, but the Bellman-like loss leverages target rewards to learn a better mapping.
'Throughout this paper, we use the subscripts "sre" and "tar" to represent the objects in the source and target domains, respectively.
3

---

Published as a conference paper at ICLR 2026
Tabular Approximate Q-Natural Policy Gradient. Natural Policy Gradient (NPG) (Kakade, 2001; Agarwal et al., 2019) is a classical RL algorithm. In this paper, we adopt NPG under two assumptions to analyze CDRL: (i) Tabular setting: finite state and action spaces, with independent parameters for each state-action pair (x, a); (ii) Approximate Q-function: the true Q" is inaccessible due to limited data, data D by executing TT(),, then obtain Q(t) by minimizing the standard TD loss for least- least-squares policy evaluation (LSPE) (Lagoudakis & Parr, 2001; Yu & Bertsekas, 2009; Lazaric et al., 2012)²
empirical approximation from samples. At iteration i, we first collect
(ID(Q()) (()) == (s.a,,)))) |r + N 1Q10 -("' (s.
(I)
Finally, we perform a one-step policy improvement: x (als) & T (als) exp (s, where 1) is the learning rate. This update improves the policy while staying close to the original.
Notation. Throughout this paper, for any policy x and any real-valued function h S A + R. we use h(s, π) and h(s, т) as the shorthand for (a-=(- [hss, a)] and h(s,a) [s) a)], respectively. For any real vector: and 2 1, we let ||=||p be the f--norm of z. For any real-valued function f S:S x A + R, we use If | as the shorthand for E (a.a)aa [f(s,a].
4 METHODOLOGY
Algorithm 1 Direct Q Transfer (DQT)
In this section, we first describe the concept of cross-domain Bellman consistency and accordingly propose the QAvatar framework in the tabular set- ting (i.e., Sur and Aur are finite). We then extend this framework to a practical deep RL implemen- tation.
Require: Source-domain Q function Quee total
iterations T, and rj = (1 7) /1/T.
1: Initialize π 7(1) as a uniformly random policy.
2: for iteration f. 1.... T do
3: Select (6()) and wh())
4: Update target-domain policy in (3).
5: end for
4.1 CROSS-DOMAIN BELLMAN CONSISTENCY
6: Return Flar
(T) Uniform (")) ()))
To motivate Source domain Q-function transfer, we present the sub-optimal gap of traditional NPG. First, we describe the definitions of state-action distribution coverage and TD error.
Definition (Coverage). Given a target-domate policy πी in Marr we say that n² has coverage Ca* for any policy E Thar, we have ||d" I S Cat. Assumption 1. The initial distribution is exploratory, i.e., (s, a) > O, for all s.a.
Notably, Cx is finite if ||d" //tarlls is finite (since ||Mtrr/d Iloo S 1/(1 7) for all i by the definition of d which holds under an exploratory initial distribution with (sr a) > for all (s, a)-a standard assumption in the NPG literature (Agarwal et al., 2021a; Ding et al., 2020; Yuan et al., 2022; Agarwal et al., 2020; Zhou et al., 2024). Intuitively, coverage enables direct comparison of Bellman errors between policies. We also Mrar,mnn shorthand for ming, (s,
(t) Definition 2 (TD Error). For each state-action pair (s, a) and t E N, the TD error edd (s, a) is defined as od (s, a) := Q/00 (s, - Tia (8, - - (t 1 (00 ,')'
(11
Proposition 1. Under the tabular and approximate- Q settings, and Assumption 1, the average sub-optimality of Q-NPG over T iterations is upper bounded by
Egrpper V (8)
(T(1) (s)
llog
A
++
Co
(
Q-10111 (m(())
log Aner + 1]
C1
(1)
(2)
vT(1
7)
T
VT(1
7) T
(n)
(b)
(a)
(c)
where Co = 2Cx ((1 - 7) and Ci == 2C+- /([1 7)8 pan min).
2I,PP under linear function approximation includes the tahular case via one-hot features:
4

---

Published as a conference paper at ICLR 2026
The detailed proof of Proposition 1 is provided in Appendix B. The upper bound of the sub-optimality gap has two parts. Term (a) characterizes Q-NPG learning and converges O(L/vT), while term (b) (or cquivalently term (c)) accounts for approximation error at each iteration, which can be made arbitrarily small with enough samples (Agarwal et al., 2021a). In CDRL, limited data amplifies term (b), potentially preventing convergence to the optimal policy. To mitigate this issue, instead of learning Q from scratch to approximate Q= we leverage a pre-trained source-domain Q- function (or)( (a)) with inter-domain mapping gi and (s(() to approximate Q= Here, the inter-domain mappings (5(+) and wh are introduced to address the state-action representation mismatch. For more specifically, we present Direct Q Transfer (DQT) method, in each iteration t, DQT proceeds in two steps: (i) It first updates gi(t) and v((²), e.g. by gradient descent on some loss function. (ii) The policy is updated by an NPG policy improvement step based on the pre-trained source-domain Que and inter-domain mappings e,(1), v6(())
x (1+22) (als) x π (9)) (als) exp (1)(m(((()) (s), (a)))
where rj is the step size. The pseudo code is in Algorithm 1. Before characterizing the convergence behavior, we describe the cross-domain Bellman error used in Proposition 2.
Definition (Cross-Domain Bellman Error). Given a pre-trained source-domain Quee inter-domain correspondences o, bb and target-domain policy T, for each state-action pair (s, a), the cross- domain Bellmun error is defined as 6,a(8, a; Φ, V,, Que, T) == (m))))())) (a)) - Tir a) yB Parl-) (-1)))))) (0 ()), 1/(1)))]|
Proposition 2. Under the DQT method in Algorithm I and Assumption 1, the average sub-optimality over T iterations is upper bounded as
Regnepao [VT (s) vx(() (s) S [log| A + 1]
Co T
(00))
Q
VT(1- - 7)
(a)
(6)
logg
Auar
+1]
Ci
Eaal Qun se ((()) eb((t)
)1 Maar,
(4)
VT(1 7)
T
(a)
(c)
where Co := 1==00,-(11- 7) and C1 == 20xx /((1 - -7)² Hlar min).
The detailed proof of Proposition 2 is in Appendix B. The main insights are: (i) Similar to Propo- sition 1, the upper bound has two terms. Term (a) characterizes Q-NPG learning, while the sub- optimality gap is mainly determined by the approximation error from Qaee- equivalent to the cross- domain Bellman error (term (c)). (ii) Minimizing this error requires & and 1o that reduce term (c). Motivated by Equation (4), we define cross-doman Bellman consistency.
Definition& 4 be 8- (d, 3) such that Eeal( Que @, 1) V)lld= la is no more than 8.
Bellman Consistency). Let 8 2 0. A source-domain critic Que. is said to under target domain policy x if there exist pair of inter-domain mapping
Transferability of a Source-Domain Model. Given a source-domain critic Qse. for any iteration t there exist inter-domain mappings ( and s)(0) such that Q.e is B-Bellman-consitter under (+)) then term (c) in (4) is bounded by C1 Thus, the transferability of a source-domain model is captured by d. In the perfect transfer scenario, where source and target domains are identical and Quee is optimal, setting á and s as identity mappings ensures small 8 for all yielding small sub-optimality gap for sufficiently large T.
Limitations of DQT. By Proposition (2), limitation of DQT is that with poorly transferable source critic, the cross-domain Bellman error at each iteration t is large, so term (c) in (4) dominates the bound and prevents effective cross-domain transfer.
4.2 THE QAVATAR ALGORITHM
To address DQT's limitation, we propose QAvatar, which uses a hybrid critic consisting of weighted combination of learned target-domain Q function and given source-domain Q function to enable
5

---

Published as a conference paper at ICLR 2026
Algorithm 2 QAvatar
Require:
Source-domain Q function Que-
I: Initialize the state mapping function e0 the action mapping function vi number on-policy samples
per iteration Nor. the target-domain policy x x()) weight decay function (0) 0. and 1l = (1 7) 1/T.
2: for iteration 1.... do
3: Sample D a, a of N on-policy samples using x in the target domain.
4: Update Que by minimizing the TD loss in (1), i.e. Q <- arg minqur LTD x (+)) D
5: Update á and e by minimizing (5). i.e., o01), , +- arg minu,, (e dh; Que, x D
6: Defined weight parameter att) = || pep (allaaaaeaa o00), (l + |1 p(o)
(t)
7: Update the target-domain policy by adapting NPG to CDRL in (6).
8: end for
9: Return Target-domain policy Tur Uniform ") (T)).
reliable cross-domain knowledge transfer. This design allows QAvatar improve sample efficiency
in favorable scenarios while avoiding reliance on poorly transferable source models. Specifically, Avatar comprises three major components:
Inter-domain mapping: Under QAvatar, we propose to learn the inter-domain mappings Φ Spar Suc and vi Aaar Are by minimizing the cross-domain Bellman loss
Le 10; Qsees mar D ((s |Iaar ".B'' 1 (o(s'))()))))))))))))))))))))( (baa))) Que (o(s), (()))
(5)
where Qsee is the pre-trained source-domain function and Diar = {(, a, Tur: 8')} denotes set of target-domain samples drawn under Tur- Intuitively, the loss in (5) looks for a pair of mapping functions Φ, such that Qaee aligns much with the target-domain transitions possible.
Target-domain Q function: To implement the hybrid critic, Qvaatar maintains karget-domand Q function Qie- serving as the critic of the current target-domain policy. At each iteration t, Qui is obtained via policy evaluation by minimizing the TD loss LTD LTD(Quri Tiar; Diar), where Dur = (s,a, 8') are target-domain samples (Equation 1).
NPG-like policy update with a weighted Q-function combination: QAvatar leverages both Qu and Qui for policy updates. At each iteration t,
(t++)( x (alss) exp (7((1 (((1)a(t))Qiil)
a(t)) (s, (o (00) (a))))
(6)
where a 10, 1] is weight function (see Section 4.3).
The pseudo code of QAvataris provided in Algorithm 2.
Remark 1. In line 6 of Algorithm I and line 8 of Algorithm 2, DQT and QAvatar output the final policy by selecting uniformly from all intermediate policies which is a standard procedure linking average sub-optimality to policy performance. In experiments, the last-iterate policy suffices and performs well.
4.3 THEORETICAL JUSTIFICATION OF QAVATAR
In this section, we present the theoretical result of Qvaata and thereby describe how to choose the proper decay parameter ax-))
Definition 5 Mooss-DDmminn Action Value Function). For each state-action pair (s, a) and e N, the cross-domain action value function f(t) (s, a) is defined as f(t) a) := == (1))(t))) .n. (olt)) (s), (5t)) (a)).
We are ready to present the main theoretical result, and the detailed proof is provided Appendix B.

---

Published as a conference paper at ICLR 2026
Proposition 3. (Average Sub-Optimalit) Under the QAvatar in Algorithm 2 and Assumption 1, the average sub-optimality over T iterations can be upper bounded as
T Ex Ex-par
V (s)
(8)
[log |uar
+1
Co
B
(7)
vT(1 - 7)
T
(a)
Dlg Aur
+
C1 T
00 w0)))) + + a(t))ll (aato)
(8)
VT(1 -1
(0)
(e)
where Co == 202x'xx(( (11-)) and C1 3C2+/((11 7)8 Paar mm)
Notably, the term (a) in (8) reflects the learning progress of NPG, and term (c) reflects the trans- ferability of a source-domain critic Que and the error of policy evaluation for the target-domain policy.
Hyperparameter-FFee Design of a(t). Based on (8), for each iteration t, term (c) can be minimized by choosing att) as an indicator function, i.c., set to I when (Q (((),, (((tt) < I (I and O otherwise. In practice, estimating the two error terms is noisy, so using an indicator can cause large fluctuations in (tt) and unstable training. To address this, we propose a smoother variant: att) [|<20] ( <all (unee o''), w:(+))) ( g-10 + [| (| ). Notably, this design hyperparameterr-fie and incurs minimal deployment overhead.
(+)
Key Implications of Proposition 3: (1) Effective transfer lowers the upper bound of average sub- optimality: In an ideal case with perfect mappings o², x such that "ø" bb;; Quee War, D = 0 for any Tur, obtain [Q<ca o², [later O. Then (t) = 1 at all making term (c) in (8) vanish. The bound thus reduces to term (a). which becomes negligible as T' grows. (2) QAvatar avoids being trapped by low-transfer critics. For source critic only Beelmman-cosisteee with large 5. cl (Quc: o, 2)) (l remains large, so (t) F 0. Consequently, term (c) reduces to the standard TD error.
4.4 PRACTICAL IMPLEMENTATION OF QAVATAR
We extend the QAvatar framework in Algorithm to a practical deep RL implementation. The pseudo code is provided in Algorithm 3 in Appendix.
Learning the target-domain policy and the Q function. To beyond the tabular setting, we extend Qvaatar by connecting NPG with soft policy iteration (SPI) (Haarnoja et al., 2018). In the entropy-regularized RL setting, SPI is known to be special case of NPG (Cen et al., 2022).
Based on this connection, we choose to integrate QAvatar with soft actor-critic (SAC) (Haarnoja et al., 2018), i.e., updating the target-domain critic Qua by the critic loss of SAC and updating the target- policy (+) by the SAC policy loss with the weighted combination of Qua and Que of Avatar.
Learning the inter-domain mapping functions with an augmented flow model. Similar to the tabular setting, we learn inter-domain mappings by minimizing the cross-domain Bellman loss. In practical RL problems, the state and action spaces are usually bounded, so the outputs of Φ Star + Sen and Aur + Ase must lie within feasible regions. As discussed in Section 2, adversarial learning is commonly used to address this (Taylor et al., 2008; Zhang et al., 2021: Gui al., 2023; Zhu et al., 2024), but it can lead to unstable training. Therefore, of (Brahmanage et al., 2023) in the normalizing flow map the outputs of the mapping functions into the feasible regions.
adopt the method RL literature (Hung et al., 2025), training a
7

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1 SETUP
Benchmark CDRL Methods. We compare QAvatar with recent CDRL benchmarks under different state-action spaces, including Cross-Morphology-DDe Domain Policy Adaptation (CMD) (Gui et al., 2023), Cross-domain Adaptive Transfer (CAT) (You et al., 2022), and Policy Adaptation by Representation mismatch (PAR) (Lyu et al., 2024). For fair comparison, all methods use the same source-domain models, including policy and corresponding Q-networks, pre-trained with SAC. We also evaluate both PPO-base based CAT, the original version in (You et al., 2022), and SAC-based CAT. Notably, CMD is an enhanced version of (Zhang et al., 2021) that integrates dynamics cycle consistency to learn state-action correspondences.
To demonstrate sample efficiency, we also compare Qvaatar with standard SAC (Haamnja et al., 2018), which learns from scratch in the target domain, and with direct fine-tuning (FT) of the source models (Ha et al., 2024), equivalent to SAC with source feature initialization. Both serve as competitive baselines. Hyperparameters are provided in Appendix F.
Evaluation Environments.
Locomotion: We use the standard MuJoCo environments, including HalfCheetah-v3 and Ant-v3, the source domains and follow the same procedure in (Zhang al., 2021; Xu et al., 2023) to modify them for the target domains. The detailed morphologies are in Appendix F.
Robot arm manipulation: We leverage Rohosuite, a popular package for robot learning released by (Zhu al., 2020) and evaluate algorithm door opening and table wiping. For each task, we use the Panda robot arm as the source domain and set the URSe rohot arm as the target domain.
Goal Navigation: A natural transfer scenario occurs when the source and target domains share the same goal but differ in robot type. We the Safety-Gym benchmark (Ray al., 2019) and evaluate transfer from Car to Doggo, keeping the goal unchanged, specifically using CarGoalO as the source and DoggoGoalO the target domain.
The dimensions of the state and action spaces of all the source-target pairs are in Table 3 in Appendix F. All the results reported below are averaged over 5 random seeds.
Exwluation Hewwrds
100co
Exllation Bowards
Foslarton Rewerds
Exoluation Nowardss
200
2000
150
53C0
Exoluation
1000
100
20
Timme
165
Time
1e5
Time Steps
165
Time Stejs
365
Time Stapo
105
(a) HalfCheetah
(b) Ant
(c) Door Opening
(d) Table Wiping
(e) Navigation
QAvatar (Ours)
SAC
FT
PAR
CAT-SAC
CAT
CMD
Figure 1: Training curves of Qvaatar and benchmark methods: (a)-(b) Locomotion tasks; (d)-(e) Robot arm manipulation tasks in Robosuite; (f) Navigation task from CarGoalO to DoggoGoal0.
5.2 EXPERIMENTAL RESULTS
Does QAvatar improve data efficiency?
Learning curves: As shown by Figure 1, we observe that QAvatar achieves improved data efficiency via cross-domain transfer than SAC throughout the training process in all the tasks, despite that these tasks have rather different dimensions as shown in Table 3.
CAT-SAC achieves moderate results on MuloCo but transfers slowly to other tasks, as CAT-like methods lack guarantees and depend on parameter-based transfer, i.e., weighted combinations of source and target policy layers. Such methods assume shared feature representations (Zhuang et al., 2020), which often fails when domains differ. FT improves data efficiency over SAC on MuJoCo but learns slowly in Robosuite due to dissimilar state action representations from different robot arms. CMD generally performs poorly and can be unstable (e.g., in Ant) owing its adversarial mapping module. We attribute CMD's 's weakness to its unsupervised design, which ignores target-domain rewards.
8

---

Published as a conference paper at ICLR 2026
Time to threshold: We provide Table I to mark the time to threshold. It shows that QAvatar requires only about 44% of the environment steps to achieve the threshold than SAC does in the best case.
Aggregated performance: To ensure a reliable comparison, follow the guidelines of (Agarwal et al., 202 and calculate the interquartile mean (IQM) using rliable, which enables evaluation at an aggregated level. Figure 2 shows that Qvatarindeee achieves significantly better performance than all baselines.
IQM
Table I: Time to threshold of Qvatar and SAC.
QAvatar CAT-SAC
SAC CAT
Environment
Threshold
Qvaata
SAC
QAvatar SAC
CMD PAR
HalfChectah
6000
126K
176K
0.71
Ant
1600
206K
346K
0.59
Door Opening
90
48K
98K
0.49
0.75 Human Normalized Score
0.25 0,50
Table Wiping
45
72K
98K
0.73
Navigation
20
218K
490K
0.44
Figure 2:
Aggregated IQMs (with 95%
stratified bootstrap CIs) across tasks.
How does Qvatar perform under strong positive and negative transfer? We consider a task where the source domain standard 'Antv33' and the target changes the goal move backward, with all else unchanged. Here, Sue and Que are adversarial due to opposite goals. We evaluate QAvatar in two scenarios: (a) Learning state/action mapping: strong transferability exists, as Ant is symmetric along the tront-back axis, allowing a perfect mapping. (b) Fixing mapping as identity: strong negative transfer case, since Sre provides adversarial reward signals. As shown Figure 3, QAvatar captures both positive transfer (high a<))) and negative transfer (low a())), demonstrating that o<()) reflects transferability.
We evaluate this scenario in the Cheetah environment (Section 5.1) using a low-quality source model with a total return of 1000 (vs. 7000 for the expert). Figure 4 illustrates the learning process and ait) of QAvatar. Results show that when the source model is of low quality, (t) decreases to a small value by the end of training, mitigating the effect of negative transfer.
Performance of Qvaatar with low-quality source domain:
12000
0.7
Extlattoon Rewands
0.5
Resisalio famanis
0.6
2000
8000
1500
60.3
6000
3.05
0.2
4000
0.1
2000
500
1.5
0.5 3.0 1.5 20
Time Stecs
165
Time Stops
165
Time Stops
365
Time Stops
165
(a) Evaluation Rewards
(b) (()
(a) Evaluation Rewards
(b) a(l)
scenario (a)
scenatio (b)
- SAC
High Quality
Low Quality
SAC
Figure 3: The training curve and the values of att for QAvatar under strongly positive and strongly negative transfer scenarios.
Figure The training curve and the values of a(t) in the Cheetah environment with low-quality source model.
Does QAvatar still perform reliably well when the source and target with two unrelated transfer scenario? We evaluate transfer from original H3pper-v3 in MuloCo to the table-wiping task in Robosuite. The configurations of these environments are provided in Section 5.1. Figure 9a shows that even when the source and target domains share no structural similarity, QAvatar still performs reliably and does not suffer from negative transfer.
How QAvatar perform on non non-stationary environment? We use the Ant environment and introduce stochasticity by adding (0, 0.1) noise to rewards and A (O, 0.05) to actions, following (Tessler et al., 2019). As shown in Figure 9b, despite stochastic rewards and transitions, the inter-domain mapping is effectively learned, enabling positive transfer and faster learning in the target domain.
QAvatar image-based experiment. We additionally evaluate QAvatar control tasks from the DeepMind Control Suite (DMC) (Tassa et al., 2018). In DMC, each observation consists of stack of three 84x84 RGB frames, and we apply an action repeat of 4. The SAC setup,
image-based continuous

---

Published as a conference paper at ICLR 2026
the flow-model training, and the cross-domain transfer protocol describe in Appendix D.4.
400
Evaliatoon newards
200
Eveluation newards
300
For cross-domain experiments, consider two transfer scenarios: The source model is trained
200
100
with SAC the walker_wall task, and the
100
larget tasks are walker_ run and cheetah_ run, respectively, Figure 5 indicate that QAvatar achieves substantially higher performance than SAC trained from scratch on both target tasks, and notably, Qvaatar succeeds even when SAC struggles to learn effectively on cheetah_ run.
0-
2
Time Steps
165
Time Steps
165
(a) cheetah_run
(b) walker_run
Figure 5: Performance comparison on DMC tasks.
Sensitivity test on the choice of N.. In Qvaatar, is updated periodically, and N₂ determines only the frequency of this closed-form update. As long as N₂ not too large (which would delay updates) or too small (which may cause to fluctuate too rapidly and introduce instability). the overall learning behavior remains largely unaffected. We evaluate three update intervals, Na 300, 1000, 3000, in the Cheetah and Table Wiping environments, whose configurations are provided in Section 5.1. As shown in Figure 6 and 7, the results indicate that (1) the learned a trajectories are highly similar across settings, and (2) performance exhibits only mild sensitivity. None of the choices lead to degradation or instability, suggesting that the method is reasonably robust to the selection of N within this range.
1.0
1.0
Evoluat Rewards
00
Ex Rewards
0.8
0.6
2 0.6
0.2
0.2
0.00
0.0
Time
Time
1e5
Time
(a) Evaluation Rewards
(b) a(t)
(a) Evaluation Rewards
(b) x(t)
Ng 300
N, 1000
No 3000
N₂ 300
1000
Ny - 3000
Figure 6: The training curves and the values of att) for QAvatar under three settings of the Na value in Cheetah environment.
Figure 7: The training curves and the values of (t) for QAvatar under three settings of the N., value in Table Wiping environment.
Extension: QAvatar with more than one source model. Qvaatar can be readily extended for transfer from multiple source model. Similar the idea of one source critic transfer, the weight az(t) for the i-th source critic Qsee,s (t) = (1/llca(Qxe.n ol"), v/ (1"))l ( ///////1111//)) "1 ||
Σj=1 1/llca(Quae..../ (0)) wi((t) )gg=())) Consider a two-source to one- target transfer scenario: (i) Source domain 1 (denoted by "sre1") is Ant-v3 with the both front legs disabled: (ii) Source domain (denoted by "src2") is Ant-v3 with the both back legs disabled. (iii) Target domain (denoted by "tar") the original Ant-v3 with no modifications. Figure 9c shows Avatar in multi-source cross-domain transfer achieve higher transferability by leveraging the knowledge from two source domains.
CONCLUDING REMARKS
We propose cross-domain Bellman consistency as a measure of source-model transferability, and introduce QAvatar, the first CDRL method that reliably handles distinct state-action representations with performance guarantees. Using a hybrid critic and a hyperparameter-Ffn Q Avatar achieves robust knowledge transfer even with weak source models. Experiments confirm its effectiveness for cross-domain RL. The general ideas of Bellman consistency and hybrid critics can be extended to cross-domain transfer in other learning settings, such as preference-bai and imitation learning (Fickinger et al., 2022; Chu et al., 2026). A limitation of our formulation is the assumption that torget-domain data collection is costlier than training compute. Since QAvatar takes about twice the training time of SAC due to acceleration would be needed when training efficiency is critical.
weighting scheme,
RL
mappings and the flow model, further
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was partially supported by the National Science and Technology Council (NSTC) of Taiwan under Grant Numbers J14-2628-EAA49-022 and 114-2634-F-A49-000-M1-
This work was also partially supported by the Center for Intelligent Team Robotics and Human-Robo Collaboration under the "Top Research Centers Taiwan Key Fields Program" of the Ministry Education (MOE), Taiwan. The authors also thank the National Center for High-performance Computing (NCHC) for providing computational and storage resources.
ETHICS STATEMENT
We conduct our research entirely in simulated environments, using no human participants or sensitive data. This work fully complies with the code of ethics.
REPRODUCIBILITY STATEMENT
The code for our experiments is provided in the supplementary material, along with a README file detailing the commands required to run the experiments. Furthermore, a comprehensive list of package dependencies is included to facilitate the recreation of the experimental environment.
USE OF LARGE LANGUAGE MODELS (LLMS)
Large language models (LLMs) were used solely for linguistic editing of the manuscript and had no involvement in the study design, methodology, experiments, or interpretation of the results.
BIBLIOGRAPHY
Joshua Achiam. Spinning Up in Deep Reinforvement Learning, 2018.
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. Pc-pg: Policy cover directed exploration for provable policy gradient learning. In Procersing in Advances in Neural Information Processing Systems, 2020.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 2021a.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021b.
Haitham B Ammar, Karl Tuyls, Matthew E Taylor, Kurt Driessens, and Gerhard Weiss. Reinforcement learning transfer via sparse coding. International Conference on Autonomous Agents and Multiagent Systems, 2012.
Haitham Bou Ammar and Matthew E. Taylo. Reinforcement learning transfer via common subspaces. In Adaptive and Learning Agents: International Workshop. 2012.
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Unsupervised cross-domain
transfer policy gradient reinforcement learning via manifold alignment. In AAAI Conference on Artificial Intelligence, 2015.
Paniz Behboudian, Yash Satsangi, Matthew E Taylor, Anna Harutyunyan, and Michael Bowling. Policy invariant explicit shaping: an efficient alternative to reward shaping. Neural Computing and Applications,
34(3):1673-1686, 34(3))1633333)333333313333333333
2022.
11

---

Published as a conference paper at ICLR 2026
Janaka Brahmanage, Jiajing Ling, and Akshat Kumar. Flowpg: Action-constrained policy gradient with normalizing flows. In Advances in Neural Information Processing Systems, 2023.
Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of
natural policy gradient methods with entropy regularization. Operations Research, 2022.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In JEEE International Conference on Robotics and Automation, 2019.
Li-Min Chu, Kai-Siang Ma, Ming-Hong Chen, and Ping-Chun Hsich. Semi-supervised cross-domain imitation learning, Transactions on Machine Learning Research, 2026.
Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovie. Natural policy gradient primal-dual method for constrained markov decision processes. In Advances in Neural Information Processing Systems, 2020.
Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to-real transfer. In IEEE International Conference Robotics and Automation, 2021.
Mahidhar Dwarampudi and NV Reddy. Effects of padding on Istms and cnns. arXiv preprint arXiv:1903.07288, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, 2018
Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations, 2021.
Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation learning via optimal transport. In International Conference on Learning Representations, 2022.
Haiyuan Gui, Shanchen Pang. Shihang Yu, Sibo Qiao, Yufeng Qi. Xiao Hc, Min Wang, and Xue Zhai. Cross-domain policy adaptation with dynamics alignment. Neural Networks, 2023.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. In International Conference on Learning Representations, 2017.
Scokhyeon Ha, Sunbeom Jeong, and Jungwoo Lee. Domain-awart fine-tuning: Enhancing neural network adaptability. In Association for the Advancement of Artificial Intelligence, 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
Wei Hung, Shao-Hua Sun, and Ping-Chun Hsich. Efficient action-constrained reinforcement learning via acceptance-rejeecion method and augmented MDPs. In International Conference Learning Representations, 2025.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002.
Sham M Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, 2001.
Michail G. Lagoudakis and Ronald Parr. Model-free least-squares policy iteration. In Advances in Neural Information Processing Systems, 2001.
Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Finite- sante- analysis of least- squares policy iteration. Journal of Machine Learning Research, 2012.
12

---

Published as a conference paper at ICLR 2026
Jinxin Liu, Zhang Hongyin, and Donglin Wang. DARA: Dynamics-Aware Reward Augmentation in
Offline Reinforcement Learning. In International Conference on Learning Representations, 2022.
Jiafei Lyu, Chenjia Bai, Jing-Wen Yang, Zongqing Lu, and Xiu Li. Cross-domain policy adaptation by capturing representation mismatch. In International Conference Machine Learning, 2024.
Steven Morad, Chris Lu, Ryan Kortvelesy, Stephan Liwicki, Jakob Foerster, and Amanda Prorok. Recurrent reinforcement learning with memoroids. In Advances in Neural Information Processing Systems, 2024.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions Knowledge and Data Engineering. 2009.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation, 2018.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-basel :aselesssslines: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
Aravind Rajeswaran, Sarvject Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. In International Conference on Learning Representations, 2016.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint
7(1):2, 2019.
Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL. efficient. In International Conference on Learning Representations, 2023.
Ram Ananth Sreenivasan, Hyun-Rok Lee, Yeonjeong Jeong, Jongseong Jang, Dongsub Shim, and Chi-Guhn Lee. A learnable similarity metric for transfer learning with dynamics mismatch. In PRL Workshop Series Bridging the Gap Between Al Planning and Reinforcement Learning, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 00690, 2018
Matthew E Taylor, Gregory Kuhlmann, and Peter Stone. Autonomous transfer for reinforcement learning. Autonomous Agents and Multiagent Systems, 2008
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. In International Conference on Muchine Learning, 2019.
Chang Wang and Sridhar Mahadevan. Manifold alignment without correspondence. In International Joint Conference on Artificial Intelligence, 2009.
Yue Wang, Yuting Liu, Wei Chen, Zhi-Ming Ma, and Tie- Yan Liu. Target transfer q-learning and its convergence analysis. Neurocomputing, 2020.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data, 2016.
Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuclong Li, and Wei Li. Cross-domain policy adaptation via value-guided data filtering. In Advances in Neural Information Processing Systems, 2023.
Heng You, Tianpei Yang, Yan Zheng, Jianye Hao, and E. Taylor, Matthew. transfer reinforcement learning based on state-action correspondence. In Uncertainty in Artificial Intelligence, 2022.
adaptive
13

---

Published as a conference paper at ICLR 2026
Huizhen Yu and Dimitri P Bertsekas. Convergence results for some temporal difference methods based on least squares. IEEE Transactions on Automatic Control, 2009.
Rui Yuan, Simon S Du, Robert M Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. In International Conference on Learning Representations, 2022.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Learn what not to learn: Action elimination with deep reinforcement learning. In Advances Neural Information Processing Systems, 2018.
Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Learning cross-domain correspondence for control with dynamics cycle-consistency. In International Conference on Learning Representations, 2021.
Yifei Zhou, Ayush Sekhari, Yuda Song, and Wen Sun. Offline data enhanced on- on-policy policy gradient with provable guarantees. In International Conference on Learning Representations, 2024.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International Conference on Computer Vision, 2017.
Ruiqi Zhu, Tianhong Dai, and Oya Celiktutan. Cross domain policy transfer with effect cycle- consistency. In IEEE International Conference on Robotics and Automation, 2024.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. Robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint 2rXiv:2009.122900 2020.
14

---

Published as a conference paper at ICLR 2026
APPENDICES
A SUPPORTING LEMMAS
Lemma (Performance difference lemma). For any two policies and x', we have
1- V" (1) - (=( - -_ s (s)
where Q= - (s) is the advantage function.
Proof. This can be directly obtained from Lemma 6.1 in (Kakade & Langford, 2002).
Lemma 2 ((Agarwal al., 2019), Chapter 4). Ler T = (8o, dos 81, ..,...)) denote the (random) trajectory generated under a policy æ in an infinite-horizon MDP M. For any function f S x A I. we have
E, Exf(ss - - E(s.a)aar
1
[1s.))aar((f(ss [f(s
(9)
Lemma (Importance Ratio). Given a fixed policy and fixed state-action pair (s, a). let pa (s.a) a)
denote the probability of reaching (3, a) under an initial distribution d= and policy π. after k time steps. Then, for any N, we have
Pa(5, a)
S
(10)
d'((s, a) (1 (1=))
Proof. To begin with, recall the definition of dt
d" (s) (1- Σ s, -.; =
P'Pss 5, 0; a, (T) 11)
t=0
(11)
Let Speat,k and & denote the state and action after / time steps. Then, we can write down py (x(88a): a):
pu (s, a) = Σ
P(Snext, = 8, Onext,k = aasss'''s, d'; (s' (s',a')
(12)
(d'ra))S x A
Σ
P(Smeat,k = Omext, =
(1 7)
7 P(s: =''a d';TT1))
(d'a')ESxA
I=0
(13)
8
= =(1 -1--7) 7)
[S 8, Anext, als', a'; nT, PP(s s',a, a, a'; n p))
1=0
8' s',a'ESx X
(14)
8
= (1 7
7' *P( 5,0 = a; T,
(15)
Then, we have
Ph (s a)
(1 (1-7) Ei-o 7tt P((++ 8; 0t+k = a: n.
(16)
d= (s,a)
(1 7) Ei 71 P(s 8, 8, (,
1==0 7' PP(se+ = 8, 0t+k = 0, T.
(17)
-t=0 76 Pl(tt = 8, Gt =.
>10 7d
(18)
1= 2d PP(s: = 3; p
1
(19)
1- 21 7tt IP(se = 5; 3T
15

---

Published as a conference paper at ICLR 2026
where (18) holds by P(S s, a+tk IT, 11) S 1 and (19) holds by taking the sum of an infinite
geometric sequence. By the fact that Lt= ' ((s = s, ag = a: क. 11) = (lss + Σi=1 Y P(s: = a+ a: I 11), we have
1 T-7 7 Ei=0 P(s = s, ar = ,;
1
7 p(s,a) + > 72 P(s = s, ae = a; n, 11
(20)
S (1 a))
(21)
where (21) holds by Ei=1 7t PP(2 = a, = a; n, 11) 2 0.
Lemma 4. Let vlt): S x A - R and (t) denote any tabular function used in the policy update and the policy at iteration t. That is,
t (1+1((a 8) x π (a 8) exp (s.
Then, we assume that |1 S 1/(1-7) and setting learning rate 1j (1-7) V and optimal policy n², we have
Σ
T
E (s.a))dd
(s,0)s s
VT [log Auar +1]
1 1-7
Proof. Let (s, a) == ul0) (s. - (5, According to the policy update rule, at iteration I, the policy x (((+)) for the next iteration is updated by the formula:
(a s) exp (n,( a))
7(0 (a 8) exp (nol*) (s,
x (((+1)(a 8) Ee a(t) (a' 8) exp (l (s,a")
(22)
E. ((()) (al s) exp (mD(t) (s, a')
Let 2== = Ea 70(0) (a' 8) exp (nb()) (s, a'). By multiplying both sides of (22) by Zr, taking the logarithm, and then taking the expectation on both sides w.r.t (s, a) ~ d= we obtain
E (ssa))aa (p, a) = =E ((aa)ad
[log Ze + loog π (++1 s) - log (a s)
(23)
< Next, bound the term log Ze. Note that (s, 1/T < and the fact that exp(x:
1 + x + x² for any S 1, we have
Σ (1)) (a' exp exp (mD()) (s.
log Z. log
(24)
a'EAA
Σ 1(1) (a' s) 1 (npll) (s, + (1p((((,,)))
< log
(25)
a'EA
1/2
< log
1+
(26)
(1 (1-7)²
1
< (1 7)
(27)
where (26) is because Evee E the fact that log(1 + x) for any 2 0. Then, we have
nt (al 8) (lt) 0 and || I 1/(11 - -)), (27) is follow
E B(saa)mar- (s,))(ss s B (eee))a
log (++)) 1 (8) - log (0 s) + (1 7)
r7²
(28)
16

---

Published as a conference paper at ICLR 2026
By taking the summation iterations both sides of (28), we have
(x.a)ddd (s,
< (1 7)2 + ((aa))a
Tr2
[log x (T 8) log (a s)]
Using the fact that log(z((a {)) S 0 and N (a
[.xe [1an we have
E (ssa)aar (s,)
Tn + log A (1 71-
By setting (1 1/T, we have
T
(saayarr (s,)), s
vT (log Aur
Let (() S A + R and (!)) denote value function used in the policy update and the Lemma 5.
policy iteration I. That is,
a (1+1( 18) x (t 8) exp (19())(s.a))
(8,
(29)
Then, by assuming that (| 10x0 - 1/(11 7) and setting the learning rate 1j = (1 7) V/TT and optimal policy n²,, we have
T
V (8) VT(() -s)-v+() (s)
E Egogiar
T
log Auar +1]
+
2C
Σ
T
E
(1.0)~=7())
(s) -Q- (s,a)
VT(1 -7)
1 T
Proof.
VF (Ptrr) ((( (Mar)
1
m0) (8.0)
E
Beeea
(30)
1
(8, - -100 + -r (s,
(31)
1
y E (e.e)~as
(s,
+ '-
1
E Flaa)aat
| (s,a+ - (s,a)
(32)
5 B(e.))as
1
(10 b00 (s, a))
+
1
E E
-D10 (s,0) + TT 50) (s, a))
(33)
where (30) holds by the performance difference lemma (ef. Lemma 1), (31) is obtained by adding D'(s, a) -'(ss, a), (32) is obtained by rearranging the terms in (31), and (33) holds by x A |z]. for all E R.. By the fact that
d=
C. we have
do
1
[200 (s,a)
1
E Eaand [ iz((t + A +A='"(saa)) (s.a)]
Y E B(aa))an
(s,
C.B E. s.andr(t)) (|- AT (s.
< B(oa))at
+ 1 1-7
(34)
(35)
17

---

Published as a conference paper at ICLR 2026
Recall the definitions that (x, a) = (1) (1, (x, and A (8,a)== (s, (s)
QT (s, (s)). Then, we have
E (e.0)=a-(()
| (s.a) A (s.a))
(s.a) (s, (t) (s))
Q (s, + - (s, (s, (s))
IF (s.)))dn(t))
(36)
[w() e [ (s. a) + | " (s. (s (s))
S E (s()))dn(t)
(37)
where (37) holds by the fact that |x ty I |2||| + y for any I, E R. Then, by linearity of expectation, we obtain
E (aa))aa(())
[
as)
(s, (s,
(0 (x),
(s)) -Q="0 (s.a) (s,
E Bandlt)) (s)) (5, (0()))
=E (s,a)varr()))
(38)
-B(saa)arta 2 (,) - @m (s. (..a)]
(39)
where (39) holds by Jensen's inequality. Then, substituting the result from (39) into (34), we have
1
(s))
+
1
C.
(,a)d=(1))
[1-00) + A (s, (.
(40)
1 -
1-7
<
1
E
(x,
+ 1-7
2C
-R
- (s)
(s, (s,
(41)
1
Next, summing over all iterations and combining with Lemma 4, we have
T
Eg Egneter
p" (T((s)
(s) - V+(1)) (s)
T
<
[log] Aua +1 + 1 T
20
Σ
T
E
(sa)marlt)
- as)
(42)
VT(1 - 7)
1=1
Recall that for any policy w, we use dd to denote the discounted state-action visitation distribution under policy π in the target domain.
Lemma 6. Under Algorithm for any t E N, we have
E (ea))aar1)) [[((s,a)) - Q= -
(s, (sxx
1 [11 (1-a(t))) (s.))ao (s.a)aa)) [ (s,
+ +a(t) E(t)) (aa))atni (ccals, a; Quee 6l1,, w010) x((11)
]]
I (1 (aacin
(43)
Proof. Recall the definition of f(()) = (1 - (1- (s, act)) (a)), we have
E, (x.a)aat
(()) [s, a) - Q (1) (s,a)
= E (0.0)ao00
(s. (((
(l(( - (s,
(44)
(1-a a())) (s, a) (s, a) + + Fer (8,
(45)
+ aa))) (o)()))))))( (a)) - (a a) + rear (s, Q- (s,a)
18

---

Published as a conference paper at ICLR 2026
= E (ssa))ddi [11 (1)) - (s, (ra())))) + trr((, - -7B, -B, 1
(d aa))
(')
+ BB BBy 1 (s', d'))) + att) Que (0)0) (s), (a)) - (8, + (1)
a' (-1o')
( Qure 1/ (a'))] + T Pun( 1 (ol (1)) d' (1)1't))-1
a' d''n (-1e))
- (s (s.4)
(46)
-E -B(sa)adat)) | (1-a(())Q! (s) - -rar(3, - -R a' d'va'
10 (f,
+a(t) (2(c(())))) (1(0 ()) - 1q (s'), 100 (a')))
(11)))
(47)
+ (at) /B 1 (s', +
Qare (6( 2)( (a'))
d' ('1x'")(113")
+ +nmr(s, - - (r" (6:
= -Kiaa))dnit) [11 [1- a(t) a(t)) (Qhi (s, - -Tar(s; P 1u0 (s',
+ a(t) (0))) w/ (a)) (ur(s) a) - -yE, ( Qu (5)) (48)
(113))
+ / Po (s a')] + (s) Q ( (x,
a' ('xx())(-11o))
where we obtain (45) by adding the dummy terms (1 -a(()) ( (s) + (s) and a(t)( ()) + as) to the inner part of (44), (46) is obtained by adding (1
a(I) ( (1-)(1)
:Bo-Ry(aaal (oll (s',a)) + P 1 (5', (s',d))) and a(t)
('~ y ( ( (a))) + ('1s')
a' ('
Quee (d(0 (s'), (a'))) to the inner part a' d'm (115)
of (45), (47) holds by rearranging the terms in (46), and (48) holds by the definition of f()), Then, by adding 3' (Qx(") (s", aa") - 7F, "">PPal-llaa (Qm(( aa") to the inner part of (48), we a" (n (-1o")
""~
can rewrite (48) as
B Blosasarri (1-a((1(( (Q - (8)))) ()) - RBanea(lea W (6', (s','))
a' ('
+ a(t)) ((n((())())))s))) (a)) - (s, a) -TB (B (ol( (s'), 2000 a ('mx'l)((("))
(49)
+ >,, B. (F(0)ss a')] + Fun (s, pm( (s) ('
(6" a")] + "E "E 1Q=10((0 (s" (s", - "T (Q=(0))
a" (1"
19

---

Published as a conference paper at ICLR 2026
( S B (esesmartes 111- a(t) (Qll) - [8) - PBonpalllaa Q (s', a''))]
+ |a(()) (an (oo( (a)) (8 (1 lune (ol 2b (0()
a' (')o'"
(50)
B (B [ (5', ,a)] ()) Q (s, ('x1")
+ BB BB 1Q (s", a")] - -,,,"" (l 12 (o",a"))
a" a'wx"(((1s")
a"~x""((1s")
$ E (sx))aal)))
(1 (l-a(t))
Q (s, - -nrr(s a) - yB s/a P (11s,a) 1 (s', a']l
a' a'"x'1))11s
(
+ a()) (oo(( ()))))))))))) - () - 7 ". (100)
a' (' (1 (1/5)
(51)
=reuls,a (0 ,000),
+ /B (s (s') yB," -,, 1Q (s" "'''"]]]] ('xx(')(1115)
d" (1115")
+ (m) - Q²(( (8,a) + 7B, 3" P a (1>""
(6" """""
< E (sa))arlt))
(1 -1-afhe af) (s) + a(t)rau(s, a: Que, Φ00) eb(0) 100)
(52)
+ > /Bs
(f)
(113")
where (50) holds by triangle inequality, (51) holds by the facts that 0 S a(l) S
1 and S att) S 1, (52) holds by coupling (s',a') and
(s" aa")
and
applying Bellman expectation equation as well as the definitions that enl (s, a)
(8)
(s.a) - (s, a) - BB PB (s', a')ll and coa(s, a; Quee 6(1), (0()) x (0) = (m()))))) (a)) (s) d's''"((s')
((1s))
(mm (o(10 (1)) By recursively ap- plying the procedure from (44) to (52) (s',''))) Qrt))((( (8/, a'),, we obtain bound on E ((.a))ar) (1) [(F()(s.) QT" (s.)))))))) (s))
as follows:
E ((ee)marll))
(s) ~-th - (s,a)
| ((-a(())e(t (1f (x, a) + a(l)eca(s a; Que, 60)
< E (o
2610) ,00, x110)
(53)
[[(0)(s a²) Ωt (s',
+ ((BnPPa((aaa) a'mx(t)(116')
20

---

Published as a conference paper at ICLR 2026
1 (1)) (s, + aa(t) a: Qwc, 600 sp10) xz(1))
E (ss))aaa)))
(1 ( 10 (s),a)) + a(l)cal(s' d' Que, bi x010) x110
+ (P((laa a' a'mx'))(116"))
(54)
+ 7' ("
-rr ("""
(10 (1-a (s. + a(t) a: Qse, @(()) 2b(t) x(())
I E (aa))arlt))
(1(1 ( (8) (s, a) va(t) a Que, 2(0, 1:(0) x0))) x0()
+ (1 7))
(55)
(e, (s, (1-a(tt)cd + (t) cod a: Que, (0 20(0)) , (1)) + +...)]]
+ 72 - +1
1
[ - a(l)) (s, al)eaa( a Que, @i0, 16(1)) , m/ (56)
<
E (s.))~a)())
7)2 Marammn
(1 (1-a(tt)BB (ma))art))))) (s.aa + a(t) (ma)marll) (0)
[Eco(s, a: Que. d( 26(0) x x(0)]
(1 Mea.min
(57)
where (54) follows by applying the procedure from (44) to (( - (d,a') aa), (55)
follows by applying the same procedure to subsequent time steps with importance sampling using the ratio bound in Lemma 3 and the same dummy variables (s, a) for all subsequent state-action pairs, and (57) follows from summing an infinite geometric series.
B PROOFS OF THE PROPOSITIONS
We first present the proof of Proposition 3 in Appendix B.1 and then establish Proposition 2 and 1 by a similar argument Appendix B.2.
B.1 PROOF OF PROPOSITION
Proposition 3. (Average Sub-Optimality) Under the QAvatar in Algorithm 2 and Assumption 1, the average sub-optimality over T iterutions can be upper bounded as
Fe Ra-par [
(s)
vzlh (4)
[log] Aur
+1
Co
- -
+
E
(e
(7)
VT(1 7)
T
(a)
(6)
[log Amar +1
C
(0)) 16)00) M + (1 -a(()))) (xxc)) )
<
(8)
VT(I 7)
T
(0)
(e)
where Co = 20x'/ /(1 - and Ci = 2C2./(1 7)8 Peur min).
Proof. Using Lemma 5 and setting (1) = ((()) we have
T
B
Berpair [=((s)
(s) v*(1) (1)
(s)
T
1=1
[og| Aus +1
20 1
Σ
E
(s.))aar(0)
(s) - (s,a)
(58)
VT(1 7)
1 7 T
1=1
21

---

Published as a conference paper at ICLR 2026
This establishes the first inequality. (8, Que. Φ wh, T) as
Furthermore, recall the definitions of (s,a) and
Ed (s.a) = Qw2 (5, a) (Ta((s,)) yB (11s,)) Q (s', d')]l, a'm=')))111)
(59)
(s, Que v. r) = (gc(((() ((()) (sr(s,,)
(oc (baa))))
(60)
We also define the weighted & norm under state-action distribution induced by any policy π as
|ldd (t) Ila= ((s,a))dd
(id (s. (s.a)
(+)
(61)
|/ (QuaQQua Quees (1)) 2b( Iaa- B(saa)ar Foals, Que, (5(1) ,z6(1) [T)
(62)
For the second inequality, by Lemma 6, we have
1
V (s) / (s)
I Resroper
T
[log Atar ++11
+ 20 1
Σ
T
E
-
-0x (s,a))
(63)
VT(1
7
7 T
1=1
log
AAar ++]
2C
T
(1 (1-a(1) (()))ll) (1) || aa(()]leaa (Qwe, 6i0) 26(0)) 11
+
11
VT(1 -17
(1 Mur,minn
T
(64)
This completes the proof of Proposition 3.
Additionally, by choosing a(t)
1| (())
(as discussed in Section 4), we have
Ioaa Q 201), ,(0) (m(()) + 11. 2m10))
T
V= (s) - (T(1) (s)
B Ea-per
T
t=1
2
log(Arar
4/2C
1
lleaa Quee, 310,,(())l -le 1aar (65)
+
(1
7)2
T
Mear,min T
|EEa ditt) p(11)1 20 in + 1 1
(1)
B.2 PROOF OF PROPOSITION
Proposition 2. Under the DQT method in Algorithm I and Assumption the average sub-optimality over T iterations is upper bounded as
E Baguupar
VTT((S (s) (-(1) (8) S
log Aua + 1]
Co
T
(me( (b(t)) -
2- 11
T
VT(1
- 7)
T
1=1
t=1
(a)
(b)
[log]
Auar
+
C
T
caal Q Qwx, (000) 25(0))
(4)
VT(1 7)
T
(a)
(c)
where Co := =0,== 7) and C == 2C /((1 9 Prar, min)
Proof. Notably, since the Proposition 2 is a special case of Proposition 3, we can simply follow all the steps taken for Proposition and (<()) = 1 for all / establish Proposition 2. More specifically,
22

---

Published as a conference paper at ICLR 2026
can replace (s, a) with Que (511))) (a)). Accordingly, under a(t) for all Lemma can be simply rewritten as
(ar( (((((a)) e (s.
(s.
E (a.a)arr10)
(66)
(1 7) Purmin (s.a)arelt) [ca((,, a; Que, @10), 2i(0) x0(1)
(67)
Similarly, Lemma 5 can be rewritten as
E Berestor V (s) / (s)
T
<
[log
VT(1- 7)) 1 - Σ
Aur
+ 1]
2C
T
R (sa))ar()))
(s) (a))) (s,a)
From the combination of the two results,
R Bewppa par
V- (s) va( (5) S
[log Amr +1]]
+
20
T
Eoal EQa See 200) (1)0))))
T
VT(I (11-7) 7)
(1 7)3
Man.min
T
1=1
(68)
L
B.3 PROOF OF PROPOSITION
Proposition 1. Under the tabular and approximate- Q settings, and Assumption I, the average sub-optimality of Q-NPG over T iterations upper bounded by
T
Egroparr
V
vT(() (s)
T
1=1
llog
Asar
+1
CD
e-l01 (m(())
log Ami
+ 1
C1
(1
(2)
vT(1
7)
T
VT(1
7)
T
(a)
(b)
(a)
(c)
where Co := 222-.'//O
- /1--))11)))))0))) and C1 == 20' /(1 7) Maas min).
Notably, since the Proposition is special case of Proposition 3, simply follow all the steps taken for Proposition and set ox(4) = 0 for all / to establish Proposition More specifically,
Proof.
replace (0) (s, a) with Qlill (s, a). Accordingly, under a<l) 0 for all Lemma 6 can be simply rewritten as
[1o (s, Qm (s.
E (sa))ar(t))
(69)
E (1 2- (maamaoll) (<a(),))) a))
(70)
Similarly, Lemma 5 can be be rewritten as
Exrpaex V*" (3)
pal (s)
Aur 1] 2C 1 log
T Σ
[100 (s, -QxT"
- (s.a)
+
VT(1 7)
TT T
From the combination of the two results,
E Egrpprr
V (T((s) (s) v*(1) (s)
(=()(s)
log Aur +1] 2C VT(1 - 7)
G (71)
T t=1
(1 7)3 3 Ptar.min T
23

---

Published as a conference paper at ICLR 2026
B.4 SAMPLE COMPLEXITY BOUND
To convert the convergence result in Proposition 3 into sample complexity guarantees, we adopt the standard technique of the least squares generalization bound for sequential function estimation (Agar- wal et al., 2019; Song et al., 2023) as follows.
Lemma 7 (Least squares generalization bound, Lemma 3 in (Song al., 2023)). Consider sequential function estimation setting with an instance space X and target space y. Let R > 0, ò E (0, 1). Let H. X [-R, R) be a class of real-valued fiuntioon Let D {(11 ........ (I FM) {M be dataset of M points where F'm. ~ pm == Pm (11mm (1.m-1), and ym is sumpled via the conditional probability p(-(mm)) ym ~ p(- (m)) hold:
(Fm) + Em- Suppose the following conditions
( s 1. h satisfies approximate realizability, i.e, infReR M Em=i Rezpp (x)))(()))))))))))))))))))))))() (xx)))
2. { JM are independent random variables such that Elyyn [2m] htt (xm)-
3. max 1yml R. and max,, (h" S R.
Then, the least-squares solution h := arg minneR ΣM (h(mm) gmn? satisfies that with proba- bility at least 1 8,
E Em (h()) s 3xM + 256R log 2|H1
(72)
We define
Huar
(1) inf E
E (s.a)~az(1)
(1, a) +BBg' Por,a'
(73)
where Q denotes a (finite) class of possible action-valle functions. For ease of exposition, suppose (t)
Kigr < max. for all t. Note that this can be achieved since Ksar,aax can be configured by choosing the function class Q. We also let F denote the product of the (finite) classes of possible inter-domain mappings o and sb.
Definition 6 (Cross-Domain Realizability). A source-domain critic Quee is said to satisfy the cross- domain realizability under a target-domain policy x if there exists a pair of inter-domain mappings (ò, ) in FF such that [cca(Qune me Φ, V)]le- 0.
Corollary 1. Consider the setting of Proposition 3 and assume a source-domain critic with cross- domain realizability for all In order to obtain e-optimal policy Maar with probability least 1 - 8, the number of target-domate samples needed under QAvatar is
(( [og Auar A 111)
CfCCd 2
Cua
-))
-min
(74)
(1-7)
3Kaar max
Moreover to obtain an e-optimal policy in Miar with probability at least 1 - 8, the number of target-domain samples needed under Q-NPG is
where Our == 1024 (I-7)? log 40 and Cot = (1-7)² log 117
((( [log] Auar + 111211
Cjur
(75)
2
3ax
Proof. To establish the sample complexity bound, we connect the sub-optimality gap in Proposition with the number of samples needed in learning the Q function and the inter-domain mappings. To begin with, we bound the ||e |_--o as follows: Let Q Sur x Asa - IR denote an action-value function in the target domain. Recall that use Ttar and Par to denote the reward function and the transition kernel of the target domain, respectively. For case of exposition, define two helper functions Ç : Sur x Aur R. and 5* Surr x Aur IR. as
((s, a: Q, T)) = Tar (s, a) t. Pan a' ~et IQ(s', a()] ((s.0), (" (s, aT) Tun a) + TE Pan Nu Q2" (s', (s',a')) (s.a).
(76)
(77)
24

---

Published as a conference paper at ICLR 2026
By the Bellman expectation equations, know (* a; m) O, for any (s, (s,a) a) and any target-domain policy и. Recall from Algorithm 2, in each iteration we sample batch 2itt) of Nuar target-domain samples to obtain the Qui by minimizing the empirical TD loss, ie.
Que arg min QUEED
Σ
[Ir | Bi 18 (5, Q (8)
(78)
Now we are ready to reinterpret (76)-(78) through the lens of Lemma 7. Let ((8, ²x and (* a; r) play the roles of h(xx) and hi" (xx). For each data sample (s, a, r,,)), by treating (C(s, a; .+) (r + " [
(s', a')] (s))) as the noise term 6m in Lemma 7, we
know Que actually plays the role of the least-squares solution (i.e. h in Lemma 7). Through this interpretation, we know that the three conditions in Lemma 7 are satisfied with A = Kur and R = 21 7). By applying Lemma and Jensen's 's inequality, the result in (72) implies that with probability at least 1 - 5/2,
Cur
11 1l (=(e)) I 3
(79)
Nus"
1024 log - Similarly, we proceed to bound the ||<ca (10) (5(0)) (aar as where Cur (1~7)²
follows. Define an additional helper function Sod Sua x A + IR as
Sea(s, a; Quee: n, Φ 1/= (s) + Pun ous (Q(c(((s')) ab(a'))) (o)() v())) (80)
Recall that in each iteration t. we also use the batch ( of Naa target-domain samples to obtain the (iit,, ah(() by minimizing the empirical cross-domain Bellman loss, i.e.,
6(1) w0l0) + arg min LCD(Ø, wh; Que, n (1),, ( 0.v
(81)
In each iteration t, we let o! (Ieca(Qure: oh ((("))) |ga0 We know Coa(s, a: Que, T(1), oz wb(h) O, for all (s,a).
(0) and e0 denote the inter-domain mappings that yield
Now we are ready to reinterpret (80) through the lens of Lemma 7. Let Sea(s, a; Que: n, Φ. 4) and (aa(s, a: Quee (()), oo")) ) play the roles of h(x) and h (z), respectively. For each data sample (s, (s,a.r, by treating (S, a; Que n. è, Φ) (r + ylE 1212 (Ø(s"), +aaa)) Qlil (Ø(s), (())) the noise term Gn in Lemma 7, know that (1() and eb(t) actually play the role of the least-squares solution (i.e,, h in Lemma 7). Again, through the above interpretation, we know that the three conditions in Lemma 7 hold with x 0 and R = 2/(1 7). By applying Lemma 7 and Jensen's 's inequality, the result in (72) implies that with probability at least 1 - 8/2,
Cet Nur
(Qarer (00), (100)
(0 S
(82)
where Cet sub-optimality gap in Proposition 3 as follows. With probability at least -, 8,
1024 log 4 ). We are ready to put everything together. We can rewrite the (1-
0
T Eannii | (s) (s) v= (s)
(s)
[log|
A
+ 1]
C
Σ
T
(att)llssd (Qre, (((), , + +(1 (1- (1-a(()) a(t))ll (2)
+
(83)
VT(1 7)
T
[og Atar| + 1]
+
C1
Σ
T
(a(0)/ / Ces +(1-0(())) a(l))/ 3x +
Ciar
(84)
VT(1 - -7)
T
Nur
Nur
t=1
[log] Atar |1]]
C
T
Ced
33 (c)
Otar Nuw
min (V
+
(85)
VT(1
T
Nuw
where (84) follows from (79) and (82), and (85) holds by choosing att) as an indicator function as described in Section 4.3. Accordingly, can convert this into sample complexity bound. We use
25

---

Published as a conference paper at ICLR 2026
[a]+ as the shorthand for max(0, =). Moreover, suppose Kiur < Kour maxx for all l. Note that Max,
(t)
can be configured by choosing the function class Q. Then, given any > O, for any 8 E (0, 1), we
have that for any T
C/ [log | 11 द T(c) and Nur min (T-PPE=>
C
(1-7)3
0--8222
3Barrmax
of
the average sub-optimality gap is no more than Hence, by the fact that the final target-domain
policy Tar Uniform(1 (T((), ()) we have that Benqie
V= (s) V
(T)
(s) S for any
log Aur |+]| (1-T)Ø
and Nar 2 min {
C. (1-3)22
Cu
}.
This implies a total
number of target-domain samples
( [lo Aur )
2 { CfC e
Cua
(1 (1-7)
min
(86)
3tsx
is needed to achieve an c-optimal target-domain policy under QAvatar. Moreover, recall that one can recover the vanilla Q-NPG by setting (t) for all Hence, by setting (tt) in (83)-(85), we can also obtain that a total number of target-domain samples
(( log Aur (1-7)
Cur
(87)
2
3rtaar
needed achieve an e-optimal target-domain policy under Q-NPG.
C A DETAILED DESCRIPTION OF RELATED WORK
CDRL across domains with distinct state and action spaces. The existing approaches can divided into the following main categories:
(i) Manually designed latent mapping: In (Ammar & Taylor, 2012) and (Ammar et al., 2012), the trajectories are mapped manually and by sparse coding from the source domain and the target domain to a common latent space, respectively. The distance between latent states can then be calculated domains. In (Gupta et al., 2017), the correspondence of the states is found by dynamic time warping and the mapping function which can map the states from two domains to the latent space is found by the correspondence.
find the correspondence of the states from the different
(ii) Learned inter-domain mapping: In the literature (Taylor et al., 2008; Zhang et al., 2021; You et al., 2022; Gui et al., 2023; Zhu et al., 2024), the inter-domain mapping is mainly learned by enforcing dynamics alignment (or termed dynamics cycle consistency in (Zhang et al., 2021)), i.e., aligning the one-step transitions of the two domains. Additional properties have also been incorporated as auxiliary loss functions learning the inter-domain mapping in the prior works, including domain cycle consistency (Zhang et al., 2021; You et al., 2022), effect cycle consistency (Zhu al., 2024), maximizing mutual information between states and embeddings (You et al., 2022), and alignment of target-domain rewards with the embeddings (You et al., 2022). Moreover, as the state and action spaces are typically bounded sets and these methods directly map the data samples between the two domains, adversarial learning has been used to restrict the output range of the mapping functions (Zhang et al., 2021: Gui et al., 2023). On the other hand, in (Ammar et al., 2015), the state mapping function is found by Unsupervised Manifold Alignment (Wang & Mahadevan, 2009).
Despite the above progress, the existing approaches all presume that the domains are sufficiently similar and do not have any performance guarantees (and hence can suffer from negative transfer in bad-case scenarios). By contrast, this paper proposes robust CDRL method that can achieve transfer regardless of source-domain model quality or domain similarity with guarantees.
CDRL across domains with identical state and action spaces. In CDRL, variety of methods have been proposed for the case where source and target domains share the same state and action spaces but are subject to dynamics mismatch.
26

---

Published as a conference paper at ICLR 2026
(i) Using the data samples from both source and target domains for policy learning: One popular approach is to use the data from both domains for model updates (Eysenbach et al., 2021; Liu et al., 2022; Xu et al., 2023). For example. for compensating the discrepancy between domains in transition dynamics, (Eysenbach et al., 2021) proposes to modify the reward function, which is learned by auxiliary domain classifier that distinguishes between the source-domain and target-domain transitions. (Liu et al., 2022) handles the dynamics shift problem in offline RL by augmenting rewards in the source-domain dataset. (Xu et al., 2023) proposes to address dynamics mismatch by value value-guided data filtering scheme, which ensures selective sharing of the nource-domant transitions based on the proximity of paired value targets.
(ii) Explicit domain similarity: (Sreenivasan al., 2023) proposes to selectively apply direct transfer of the source- source-domain policy to the target domain based on a learnable similarity metric, which is essentially the TD error of target domain trajectories with source Q function. Moreover, based on the policy invariant explicit shaping (Behboudian et al., 2022), (Sreenivasan et al., 2023) further uses the potential function a bias term for selecting actions.
(iii) Using both Q-functions for the Q-learning updates: Target Transfer Q-Learning (Wang et al., 2020) calculates the TD error by the source and target domains Q functions order to select the TD target from the two Q functions.
(iv) Domain randomization: To tackle sim-to-real transfer with dynamics mismatch, domain randomization (Rajeswaran et al., 2016; Peng et al., 2018; Chebotar et al., 2019; Du et al., 2021) and (Du et al., 2021) collects data from multiple similar source domains with different configurations to learn a high-quality policy that can work robustly in a possibly unseen but similar target domain.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 A TOY EXAMPLES FOR MOTIVATING THE BENEFIT OF CROSS-DOMAIN BELLMAN Loss
We consider the 3-by-3 grid navigation problem, as shown in Figure 8. In both domains, there only two actions: 'going top' and gooing right.' The state of the source domain is described in decimal coordinates, while the state of the target domain is described in binary coordinates. The white squares represent obstacles that cannot be traversed. There are three special states: (i) Start state: The episode always begins at this state, (ii) End state: The episode will only end at this state, and the agent will receive ending reward of +1. (iii) Treasure state: When the agent first navigates to this state, it will receive +0.5 rewards. In other states or at other times navigating the treasure state, the agent will not receive any reward. In the source domain, the start state, end state, and treasure state are set to (0, 0). (0, 2). and (2, 2), respectively. In the target domain, the start state, end state, and treasure state are set to (0, O, 0, 0), (0, 0, 1, 1), and (1, 1, 1, 1). respectively. We assume that the source Q-function Quee is optimal in the source domain and the environment discount factor 7 set to 0.99. is easy to verify that the optimal trajectory of the source domain is (0 + (0, 1) + (0, optimal trajectory of the target domain (0, 0, 0, 0) - 0,
(0,2)
(1, 2)
[2, 2)
10. 0,
10
(1. 1,1.
10, 1)
11,
12, 1)
10. 0..0...11
10.1.0.11
(1. 1..1)
10. 0)
12. 0)
10.0,0.0;
(,
(a) Source Domain
(b) Target Domain
Figure 8: Source and target domains of the grid navigation example.
(2, and the - 00, ~ (0, 1, (1, 1, 1, 1). Consider two trajectories in the source domain: Traj-A, which is the optimal trajectory, and Traj-B, defined as (0, 0) + (0, - 1) + (1,2) - 22, When we map the optimal trajectory of the target domain to Traj-A and the optimal trajectory of the target domain to Traj-B, BB, both mappings result in cycle consistency loss. This suggests that the cycle consistency cannot determine which mapping is superior. This phenomenon results from the unsupervised nature of dynamics cycle consistency. In contrast, when we mapping the optimal trajectory the target domain to Traj-A yields cross-domain Bellman-like loss of O, while mapping the optimal trajectory of
(1,2)
27

---

Published as a conference paper at ICLR 2026
the target domain to Traj-B results in cross-domain Bellman-like loss of 1. Thus, achieve optimal mapping results based on the cross-domain Bellman error, while the cycle consistency loss provides sub-optimal mapping results.
D.2 FINAL REWARDS
In this section, evaluate the asymptotic performance of all baselines and our algorithm. In the experiments, all targer-domain models are trained for 500k steps in MuJoCo and 100k steps in Robosuite. The results summarized in Table 2.
Table 2: Final rewards of QAaatarand all baselines in the experiments.
Algorithm
HalfCheetah
Ant
Door Opening
Table Wiping
Navigation
QAvatar
11586.0 + 1224,4
2858.8 1 848.0
216.6 + 131.3
76.6
13.5
38.5 : 13.2
SAC
10986.0 + 1821.8
1620.0 + 527.2
94.8 + 23.9
47.6
+ 11.0
19.7 + 13.6
FT
10756.8 + 1070.8
1644.3 + 748.2
129.9 + 34.6
42.1
15.4
12.5 9.0
PAR
8097.4 + 3962.0
737.6 +: 45.3
33.7 + 18.6
17.9 + 11.8
0.0
CAT-SAC
8756.5 + 1264.3
1628.9 + 200.6
23.7 - 10.7
2.7 2.4
CAT
46.1 149,9
17.1 27.3
34.7 8.4
55.5 : 29.7
-0.1
CMD
223.11 + 344.1
777.5 + 144.1
7.8 1 6.4
0.8 1 0.4
0.0:000
D.3 ABLATION STUDY: EXPERIMENTAL RESULTS
80
2500
3000
2500
Eva saattioon Rewards
Evaluation Rewards
2000
Evaluation Rewards
2000
1500
40
1500
1000
20
500
500
0
0.2
Time
0.4
Steps
0.6
0.8
le5
1.0
0
0.5 1.0
1.5 2.0
3.0 Steps le5
Time Steps
2
4
le5
Hepper
CAaster
(a) Unrelated transfer scenario
(b) Non-stationary environment
(c) Multiple source model
Figure 9: Training curves of ablation experiments. (a) Unrelated transfer scenario where the target domain is Table Wiping with URSe. (b) Training curves in non-stationary Ant-v3. (c) Transfer from two source domains a target domain.
D.4 ADDITIONAL EXPERIMENT AND EXPLANATION DURING REBUTTAL.
Q Avatar on image-based experiment.
We additionally evaluate QAvatar on image-based based continuous control tasks from the DeepMind Control Suite (DMC) (Tassa et al., 2018). DMC, each observation consists of a stack three 84x84 RGB frames, and an action repeat of is applied. The protocol details are described as follows:
SAC. For SAC, both the actor and critic use hidden layers with 1024 units each. The image encoder follows the IMPALA (Espcholt et al., 2018) architecture extract now-dimensional visual features. All remaining hyperparameters are the same as those used in Stable- Babble-BBselines3.
Flow model training. Training flow model directly on the raw high-dimensional image observations challenging. Therefore, we first pass each image stack through the source encoder to obtain its feature representation. and train the flow model match the distribution of these extracted features rather than the raw images. Notably, this modification does not alter the QAvatar framework, since the source model remains fixed during target-domain transfer.
28

---

Published as a conference paper at ICLR 2026
Cross-domain transfer To leverage the source critic for transfer, each target image observation is first passed through the target encoder to obtain its feature representation, which is then used as the input to the state decoder; the rest of the procedure follows the standard QAvatar framework.
Comparison Effective Cycle Consistency (Zhu et al., 2024).
Effective cycle consistency (ECC) (Zhu al., 2024) operates under the same unsupervised cross- domain assumption as DCC and CMD, where the agent has no access to target-domain rewards. This makes the problem fundamentally more challenging than the supervised CDRL setting (i.e., with target-domain reward signal) considered in our work. Building on the DCC objective, ECC further introduces effect cycle-consistency learn the mapping functions. We evaluate on Cheetah, Ant, Door Opening, and Table Wiping in Section 5.1. Figure 10 demostrate that although ECC produces more stable alignment than CMD, its overall performance remains significantly below SAC and other supervised CDRL baselines, which is consistent with the inherent limitations of the unsupervised CDRL methods.
3000
250
00
Exaluation Rowards
10000
Evaltuatio Rewards
Evalustion Rowards
Eralbation Bowards
2000
200
60
5000
40
1000
200
50
20
0.5
2.0
2.255.3
0.4
0.8
0.4
Steps
Steps
(a) HalfCheetah
Ant
(c) Door Opening
Table Wiping
QAvatar
SAC
ECC
CMD
Figure 10: Performance evaluation of ECC.
29

---

Published as a conference paper at ICLR 2026
E IMPLEMENTATION DETAILS OF QAVATAR
E. PSEUDO CODE OF THE PRACTICAL IMPLEMENTATION OF QAVATAR
In this section, we provide the pseudo code of the practical version of QAvatarin Algorithm 3.
Algorithm 3 Practical Implementation of QAvatar
1: Require: Source-domain Q-network Quee update frequency No. batch size N.
2: Initialize the state mapping function o, the action mapping function v,, the initial target-domain policy network x (1), entropy coefficient (3, replay buffer D, and a = 0. do
3: for iteration f 1, .....
4: Interact with the environment and store the transition (8t, Gr. 8141) the replay buffer D.
5: Sample two sets of N transitions, denoted as BSac and BMap. from the replay buffer D.
6: Update the target-domain [Quar,, Quar2) by SAC's critic loss:
arg min B(e,,,r,e') |r- ( (Qurl (6'66''" a') (og((<(d()<)))
(s
Que
(88)
7: Update the state mapping function & and action mapping function & by minimizing
8: the following loss:
(1t)
(t((t) arg min E (s Batop
BB ( t (xx=)))) Qxo( (sc()(s). vaa))
0.
(89)
9:
if t mod Na = then
Define 1| 110 = B(san [r + B
- (1 mmnnn=112 [minn Qmn (5', a')] minj-1.2 2 (s,a)
10:
11:
Ieaa )(1), ,(1))) Baannse |r TB (010)))))))) whit (a'))] Qua (s), (/1,
d(t))
12:
Update the weight a = || /1.// (Ileaa (Qares 6(11, eb(())) + 1 Ilo).
13:
end if
14:
Update the target-domain policy m:
n (0+1) = arg min (ssa,ns')BBa [llog ((0 (()d))))((((ss
(90)
x
a' a'>x'"((11))
(s, = (1 a) min (c) (s, + a 2/:00) (1)
(91)
end for
E.2
SOURCE-DOMAIN MODELS AND THEIR PERFORMANCE
For the locomotion tasks including HalfCheetah and Ant, train each source model for 1M steps. The average performance of the 5 source-domain models (under 5 distinct random seeds) in HalfCheetah and Ant are 7355 +: 2892 and 3689 1013, respectively. For the Robosuite tasks including Door Opening and Table Wiping, we train each source-domain model for 500K steps. The average performance of 5 random seed is 383 + 139 and 94 = 16. respectively. For the navigation environment, we train the model for 500K steps, and the average performance is 39.85.
E.3 INTER-DOMAIN MAPPING NETWORK AUGMENTED WITH A NORMALIZING FLOW MODEL
As discussed in Section 4, a flow-based generative model is employed to transform the outputs of the mapping functions into their corresponding feasible regions. Therefore, there are two architectural paradigms of the flow model can be considered. In the first paradigm, the state and action concatenated and jointly treated as the codomain of the flow model. This joint formulation adopted in Cheetah, Ant environment. In the second paradigm, the state and action modeled separately,
30

---

Published as a conference paper at ICLR 2026
with two independent flow models trained respectively for the state and the action. This decoupled formulation applied in Hopper-v3, Table Wiping, and Door Opening tasks.
E.4 QUALITY AND STABILITY OF LEARNED STATE/ACTION MAPPING FUNCTIONS & AND vb.
In QAvatar, a key diagmostic for assessing alignment quality and stability is the cross-domain Bellman error. When this approaches zero, it implies that Sre 6- Bellman--oosistern with sufficiently small ó under the learned mappings, indicating that & and v/ are well aligned. Figure 11 demostrate the curves of cross-domain Bellman experiments in Section 5. The results consistently show that the cross cross-domain Bellman error remains low relative the TD error when the learned mappings align well with the target domain.
versus the TD
of the target critic across all main
60
1.5
40
20
20
0.0
0.00
0.5 3.0
2.0.2. 3.0
0.2
0.44
0.5
0.2
1.0
022
0.4
3.6
0.5
1.0
HalfCheetah
(b) Ant
Door Opening
(d) Table Wiping
Navigation
Cross-Domain Bellman Error
TD Error Target Q-function
Figure 11: Cross-domain Bellman error and TD error of the target Q-function during training across all main experiments.
CONFIGURATION DETAILS OF THE EXPERIMENTS
F.1 STATE AND ACTION DIMENSIONS OF BENCHMARK ENVIRONMENTS
We summarize the state and action dimensions of each pair of source-domain and target-domain benchmark tasks in the following Table 3.
Table 3: Dimensions of the source and target domains ("Sre" and "Tar" represent the source domain and the target domain.)
State
Action
Environment
Src
Tar
Src
Tar
HalfCheetah
17
23
Ant
111
133
8
10
Door Opening
46
51
8
7
Table Wiping
37
34
7
6
Goal Navigation
40
72
2
F.2 MUJoCo AND ROBOSUITE ENVIRONMENTS
As mentioned in Section 5, We evaluate QAvatarin both MuloCo and Robosuite environments. In the MuJoCo environments, the source domains of our experiments are the original MuJoCo environments such as HalfCheetah-v3 and Ant-v3. The target domains are the modified MuJoCo environments such as HalfCheetah with three legs and Ant with five legs. In Robosuite environments, We evaluate QAvataron two tasks, including door opening and table wiping. For each task, consider cross- domain transfer from controlling a Panda robot arm to controlling a UR5e robot arm. These four tasks illustrated in Figure 12 and 13.
31

---

Published as a conference paper at ICLR 2026
(a) HalfCheetah
(b) Three-leg HalfChectah
(c) Ant
(d) Five-leg Ant
Figure 12: The environments of the source domains and the target domains. (a),(c): Source domains
Original MuloCo environments. (b).d)): Target domains Modified MuloCo environments.
(a) Door Opening: Panda
(b) Table Wiping: Panda
(c) Navigation: CarGoalO
(d) Door Opening: UR5c
(e) Table Wiping: UR5e
Navigation: DoggoGoal0
Figure 13: The environments of the source and target domains. (a) (c): Source domains. (d)-(f):
Target domains.
F.3 THE IMPLEMENTATION DETAILS OF BASELINES
SAC. The implementation of SAC used in our experiments is released by al., 2021). The settings of all hyperparameters except for the discouted factor 7 follows the default settings of SAC in the documentation of KaalleBBaeeline33. The discouted factor set 0.99 in all other MuJoCo environments. In Robosuite environments, we set the discouted factor to 0.9.
(Raffin
CMD. Since there is no publicly available implementation CMD, we leverage and adapt the codebase of DCC (Zhang al., 2021) (nttps: g thub. com s jtzzq//u/ae/ Cycle Saalee_DDnamlll cs) and reproduce CMD by following the pseudo code of CMD in its original paper (Gui et al., 2023). We follow the setting of the hyperparameters which is revealed in its original paper. Additionally, we change CMD from collecting the fixed amount of data to collecting data continuously for fair comparison. As for the source model, we use the same model used our algorithm. Moreover, we observe that the original setting could suffer because the collected trajectories mostly have low returns due to random behavior policy. Therefore, we consider a stronger version of CMD with target-domain data collected under the target-domain policy, which is induced by the source-domain pre-trained policy and the current dnter-domain mappings.
FT can be modify the input and output layers of the source policy to match the target domain's 's state and action dimensions, using random initialization, while keeping the middle layers with the same weights the
FT.
standard SAC algorithm with source feature initialization. Specifically,
32

---

Published as a conference paper at ICLR 2026
source model. Similarly, for the source Q function, adjust the input layer to fit the target domain's state and action dimensions with random initialization, while the remaining layers retain the source model's 's weights. After initialization, the agent is fine-tuned using the standard SAC algorithm.
CAT. We use the authors' implementation transfer-ande andffeeranndmlllle metl:l--tattt-eeinooreeeeeeett-- Single-agent #20Trans f http:///rrrooosoooaooaateaeeeeee PPO as the target-domain base algorithm following the original paper. For fair comparison, we use the same source model used in QAvatar.
and
CAT-SAC. As CAT can be integrated with any off-the-shelf RL method, adapt the original PPO-based CAT to CAT-SAC by using the SAC implementation in Spinning Up (Achiam, 2018) the backbone of CAT-SAC. All the SAC-related hyperparameters are the same those used by SAC and the CAT-related parameters are configured as in the original implementation. For a fair comparison, we use the same source model used by Qvaatar.
PAR. We use the authors' implementation (https: /tttps://gihub.... hoom/dnksjjNl jE1/PAR. qit) and consider the offline to online version of PAR, which is more compatible with the CDRL setting our paper. For the source-domain data required by PAR, we use the samples in the buffer collected during the training of the source-domain policies (shared by Q Avatarand other baselines). As result, to adapt PAR to the more general CDRL setting in our paper, similar to the data pre-processing methods used in handling sequences (Zahavy et al., 2018; Dwarampudi & Reddy, 2019; Morad et al., 2024; Wu & Hu, 2018), we use padding and truncation to handle the differences in state and action dimensions. More specifically,
Padding: If the target domain has more dimensions than the source, we append zeros to the end of each source sample.
Truncation: the target domain has n fewer dimensions than the source, we discard the last n from each source sample.
Note that this design is reasonable, as neither the baselines nor QAvatarhave any knowl- edge about the physical meaning of each entry in the state or action representations. For the hyperparameters, to ensure a fair comparison with QAvataras well as the baselines CAT- SAC and SAC, configg' ("tir_rrtrrree _nv interact freq') in their original code). Other parameters (e.g., beta, weight, etc.) and network architecture follow the recommendations provided the original PAR pa- per. In addition, we observe that in some environments, temperature tuning can improve performance. Therefore, we apply temperature tuning during the training process (as adopted by PAR's original code), and select the better one between using and not using temperature tuning as the final result.
the ratio between environment interaction and agent training I (i.e.,
F.4 HYPERPARAMTER TUNING OF THE BASELINES
In this section, we provide the value of hyperparamter tuning detail of in the Section For fairness, all SAC-based methods (QAvatar, SAC, FT, CAT-SAC, and PAR) exactly the same SAC-related hyperparameters. This ensures that any performance differences arise solely from whether transfer is applied and how it is implemented. For all locomotion tasks, we follow the recommended SAC hyperparameters from Stable- Baselines (Raffin et al., 2021). Thus, there are no hyperparameter need to tune in SAC and FT. For the other baselines.
CAT and CAT-SAC. We conducted tuning for the schedule of p(t), which is the weight of the linear combination of hidden layer parameters: p(t) = means that only the source-domain parameters used; p(t) means that only the target-domain parameters are used). Specifically, You et al.
33

---

Published as a conference paper at ICLR 2026
(2022) sets p(t) to be piecewise linear follows:
Let T be the total training steps.
0,
t [0, cT),
t -- cIT
p(()= =
1 (cT, c27):
(c2
)7
1,
LE Ce2T, T}.
The official CAT chooses 01 0.45, 02 = 0.9. For each environment, we choose the best among the following candidate choices: (c1, E {(0.15, 0.4), (0.4, 0.7), (.945,
PAR. We performed grid search over the penalty coefficient 3 {0.1, 0.5, 1.0, 2.0). which are the values suggested by the ablation study in the original paper. We also searched over the policy objective normalization coefficient // {2.5, 5.0}. as recommended by the official code and Appendix E.1 of the original paper. In addition, we evaluated both configurations that enable temperature tuning during training. which is used in their official implementation, and configurations that exclude temperature tuning, which the default setting. We selected the best-performing configuration the Cheetah environment and applied this configuration to all remaining environments.
CMD. We conduct a grid search over the loss weights (po. P1, P2) in
L2nd Pol-gan source: G1) + Pt Lyga (uugett G2) p2L3(G1,G.))
The official CMD setting is (po- P1; (2) (1, 1,3,, fix P and conduct the grid search on P2 (0.3, 1.0, 3.0), P2 = (1.0, 3.0, 10.0}. Across all combinations, the performance similar and consistently much lower than that of SAC. Based this observation, we adopt the original recommended CMD setting for all remaining environments.
F.5 DETAILED CONFIGURATION OF QAVATAR
The base algorithm, Soft Actor- Critic (SAC), implemented using Stall-Basselines (Raffin et al., 2021). 1). All experiments were conducted on a computing server equipped with dual Intel Xeon Gold 6154 CPUs (36 cores in total) and an NVIDIA Tesla V100-SXM2 GPU with 32 GB GPU memory.
The hyperparameters of Qvattr are summarized in the following table. Unless otherwise specified, the hyperparameter settings, including the learning rates of the actor and critic networks, batch size, replay buffer size, and discount factor, follow the default configurations of SAC.
Table 4: A list of hyperparameters of Qvaatar.
Parameter
Value
critic/actor learning rate
0.0003
mapping function learning rate batch size
0.0001
256
replay buffer size optimizer for Actor/critic optimizer for mapping function
10c
Adam
AdamW
hidden layer size
256
update a frequency Na
1000
34

---

Published as a conference paper at ICLR 2026
G ANALYSIS OF THE OFF-POLICY VARIANT OF PROPOSITION 3
In Proposition 3, we provide an upper bound on the average sub-optimality of the on-policy version of QAvatar. In this section, we derive the corresponding upper bound for the off-policy variant of QAvatar. The primary difference between the on-policy and off-policy settings lies in the data collection policy. The on-policy approach collects data using the learned policy T(", while the off- variant collects data using behavior policy "g"
(t) Based notation use in the main paper,
we provide the average sub-optimality of the off-policy version of QAvatar as following:
Corollary 2. Under the QAvatarin Algorithm 2, but with the data collection policy T( replaced by (B) and under Assumption the average sub-optimality over iterations can upper bounded as follows:
Egnepar V (s) (s)
(6))~v(0(0) (0)
T
[log]
A
+
Cu T
(sra)aa') 10 (s) "
(92)
VT(1 -11-))
T
1=1
(a)
(6)
[log]
A
+ 1]
Ci T
aalllle Qwe, (5()) (5(0)))1
+ - (())ll 21 )
(93)
vT(1 7))
(a)
(e)
where Co := 20m- 8/(1 - 7) and C1 := 2C e/ 7)9 Msar, min).
Proof. The proof follows exactly the steps as those used for Proposition 3. The only difference lies in the proof of Lemma 5, specifically in Equation (34). There, instead of assuming the learned C, we replace it with the behavior policy
policy (1) together with the corresponding
3 together with the corresponding
S C3. Notably, both C3 and C are bounded constants
given the condition of exploratory initial distribution in Assumption 1. For the subsequent derivations
in Lemma 5 as well those in Lemma 6, we may directly replace d= with d= This substitution preserves all arguments, because the relevant manipulations are carried out entirely inside expectations
with respect distribution, and the structure of the inequalities does not depend on which particular distribution the expectation is taken over.
Lemma 4 does not require any modification, because is designed to control the learning process of the NPG-style policy update. This part of the analysis does not depend on whether the data are may directly use the modified versions of Lemma Lemma 6, and Lemma 4. Following the same sequence of steps as in Equations (58) to
(t) collected by (*) or by "g With the above substitutions,
(64), we can then complete the proof of this corollary.
In our practical SAC-based implementation of QAvatar, set the behavior policy मे match the buffer distribution, which can be viewed mixture of all past learned policies. Consequently, the value of (t) can naturally be evaluated over the buffer distribution.
(t)
35


Reviews:
[{"review_id": "ATGExrkTDq", "reviewer": "Reviewer_hvoW", "summary": "The paper introduces QAvatar, a hybrid-critic framework for cross-domain reinforcement learning (CDRL), which targets transfer scenarios where source and target domains possess distinct state and/or action spaces mismatch and the transferability of source models is hard to measure. Experiments demonstrate substantial empirical improvements of QAvatar on diverse RL benchmarks with both positive and negative transfer cases.", "questions": "In Figure 1, while QAvatar outperforms in terms of sample efficiency, the gap with FT or CAT-SAC is at times moderate in locomotion (HalfCheetah task). A deeper dissection of where relative benefits are highest would be useful.\n\nCan the authors provide empirical results or qualitative diagnostics assessing when the learned mappings $\\phi$ and $\\psi$ are well aligned?  What if the mapping is misleading, would the proposed method avoid collapse?\n\nHave the authors tested QAvatar’s hybrid critic and mapping approach on image-based RL or other high-dimensional observation settings? If not, do they foresee major obstacles?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes the formal definition of cross-domain Bellman consistency and its incorporation into the analysis of transferability represents a strong conceptual advance over prior methods. \n\nThe paper presents a detailed theoretical analysis with proofs supporting the soundness of QAvatar’s adaptive weighting scheme.", "weaknesses": "Although the hybrid critic can theoretically downweight poor source Q-functions, the effectiveness of knowledge transfer fundamentally depends on the quality of the state/action mapping functions $\\phi$ and $\\psi$. When mapping is highly ambiguous or misaligned, both the source-supplied and hybrid Q-values may be misleading.\n\nWhile this paper classifies major approaches and discusses adversarial alignment and cycle consistency, the comparison or discussion with alternative alignment strategies (e.g., [1-3]) is lacking. \n\nIt is hard to tell how the proposed method would perform in high-dimensional practical tasks. \n\nThere is little discussion regarding the challenges posed by partial observability or scaling to dozens of source-target pairs.\n\n[1] Cross Domain Policy Transfer with Effect Cycle-Consistency.\n\n[2] Cross-Domain Policy Transfer by Representation Alignment via Multi-Domain Behavioral Cloning.\n\n[3] Cross-Domain Knowledge Transfer for RL via Preference Consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X02ceOF5Zs", "reviewer": "Reviewer_dbj9", "summary": "This paper addresses the problem of cross-domain reinforcement learning (CDRL) where the source and target domains have different state and action spaces. The authors identify two key challenges: (1) learning the mapping between these distinct spaces, and (2) avoiding \"negative transfer\" when the source-domain model is not actually beneficial for the target task. The core ideas include cross-domain Bellman Consistency, hybrid critic, hyperparameter-free weighting, and mapping via Bellman Loss.\n\nThe authors provide a theoretical analysis in the tabular NPG setting to show that QAvatar achieves a good sub-optimality bound that gracefully degrades to the standard NPG bound in cases of negative transfer. They then present a practical implementation using SAC and normalizing flows. Experiments on MuJoCo, Robosuite, and Safety-Gym tasks show that QAvatar significantly outperforms other CDRL baselines.", "questions": "- The computational cost of doubling training time (per Section 6) is a significant drawback. Could you discuss the trade-off? At what point does the 2x computational cost per step outweigh the ~2x sample efficiency gain?\n- The mapping functions are trained to minimize a Bellman loss dependent on $Q_{\\text {src }}$. What happens if $Q_{s r c}$ is sub-optimal? Does the loss find the \"correct\" physical mapping, or does it find a distorted mapping that simply makes the sub-optimal $Q_{\\text {src }}$ look consistent with the target dynamics?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The central concept of using a hybrid critic weighted by a measure of \"transferability\" is very clever. The automatic, hyperparameter-free weighting scheme, which pits the target TD error against the cross-domain Bellman error, is an elegant solution to the problem of negative transfer.\n- Learning the state-action mappings by minimizing the cross-domain Bellman loss (a reward- and dynamics-aware objective) is a strong alternative to unsupervised, dynamics-agnostic methods like cycle-consistency. The toy example in Appendix D.1 provides a good motivation for this.\n- The method shows consistent and significant sample efficiency improvements over all baselines (SAC, FT, PAR, CAT, CMD) across a good variety of cross-domain tasks (locomotion, manipulation, navigation) with differing state/action spaces.", "weaknesses": "- The practical implementation of QAvatar is very complex. It requires running a full SAC algorithm while also training two normalizing flow models for the state and action mappings, and also calculating two separate Bellman errors (target and cross-domain) at every step (or every $N_\\alpha$ steps) to compute the weight. The paper notes this results in \"about twice the training time of SAC,\" which is a major practical limitation and cost.\n- The paper uses normalizing flows for the mappings, which works for the low-dimensional state/action spaces in MuJoCo and Robosuite. It is not clear how this approach would scale to high-dimensional state spaces, such as images, where learning a normalizing flow is significantly more challenging.\n- Stability of Mapping Loss: The mapping functions $(\\phi, \\psi)$ are trained to minimize a crossdomain Bellman loss (Eq. 5) that depends on both the fixed $Q_{s r c}$ and the changing target policy $\\pi^{(t)}$. This creates a complex, non-stationary optimization problem for the mappings. It's unclear how stable this is, especially if $Q_{s r c}$ is sub-optimal or the policy is exploring.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LrRTMvRJm1", "reviewer": "Reviewer_5Ycb", "summary": "The paper proposes an interesting and novel approach with solid mathematical proof and promising empirical performance.\nThe paper proposes cross‑domain Bellman consistency $\\delta$ and QAvatar method. By learning a internal mapping,  the $Q_{src}$ aligns with the target-domain transitions and use a hybrid critic that interpolates target and mapped‑source Q values with a closed‑form weight $\\alpha$. The authors prove an average sub‑optimality bound and show empirical improvements on tasks with different state/action dimensionalities, including locomotion, manipulation, and navigation.", "questions": "Same as weakness\n\nadditional questions:\n\nCan you add ablation test regards to the update coefficient $N_\\alpha$? Is the performance sensitive to the udpate frequency?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clean problem framing for CDRL with mismatched spaces; explicit transferability notion via a Bellman‑style residual\n2. Hybrid critic that can down‑weight a poor source, limiting negative transfer\n3. A formal tabular NPG analysis that separates learning progress and approximation error\n4. The experimental evaluation is comprehensive and convincing, covering diverse tasks and challenging transfer scenarios", "weaknesses": "1. The theory-practice gap is substantial: the convergence analysis assumes on-policy, tabular NPG, while the implementation is off-policy, deep RL with replay buffers and twin Q-networks. The critical assumptions of the bound (e.g., on-policy error norms) do not hold in the implemented algorithm.\n\n2. The claim of being “hyperparameter-free” is overstated: the update frequency $N_\\alpha$ for the adaptive weight acts as a hidden hyperparameter, and no ablation study is provided.\n\n3. The loss to update the mapping $\\phi, \\psi$ Eqn 5 (unquared) and Eqn 73 (squared) are mismatch\n\n4. mior format issue: The title of tables houle be ahead of the table", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mZI8E7KHxL", "reviewer": "Reviewer_bUDR", "summary": "The paper studies cross-domain transfer RL: the source and the target MDPs could differ in both state and action spaces. The transfer could be infeasible in that good inter-domain mappings may not exist. It introduces QAvatar, an algorithm that uses a convex combination of the Q-values from the source and the Q-values from the learned critic in the target to update the target policy. They show the efficacy of their method through locomotion, manipulation, and goal navigation tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is mostly well-written.\n- Cross-domain transfer in RL is an interesting problem, and the authors tackle it with a novel approach.\n- The algorithm, QAvatar, is principled and is motivated by theoretical insights.", "weaknesses": "- There are discrepancies between what the theory suggests and what the practical version of QAvatar does: while $\\alpha_t$ suggested by the theory involves norm defined w.r.t. visitation distributions of the policy at time t (line 332), empirically, the authors use the entire replay buffer for it (lines 1315 - 1318). \n- In the experiments, other CDRL baselines considered by the authors perform worse than learning from scratch (SAC) and direct finetuning (FT).  Did the authors optimize the hyperparameters for these baselines?\n- While QAvatar is theoretically motivated, the method as such does not come with any guarantees. In this setting, one would be interested in sample complexity guarantees: the proposed algorithm is provably more sample efficient than learning from scratch. The authors do not provide guarantees of this form. But I understand that this might be beyond the scope of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)