Cross-Domain Reinforcement Learning Under Distinct State-Action Spaces Via Hybrid Q Functions

Authors: Kuan-Chen Pan, MingHong Chen, You-De Huang, Xi Liu, Ping-Chun Hsieh
Primary area: reinforcement learning
Keywords: Cross-domain transfer; Transfer learning; Reinforcement learning

Abstract:
Cross-domain reinforcement learning (CDRL) is meant to improve the data efficiency of RL by leveraging the data samples collected from a source domain to facilitate the learning in a similar target domain. Despite its potential, cross-domain transfer in RL is known to have two fundamental and intertwined challenges: (i) The source and target domains can have distinct state space or action space, and
this makes direct transfer infeasible and thereby requires more sophisticated interdomain mappings; (ii) The domain similarity in RL is not easily identifiable a priori, and hence CDRL can be prone to negative transfer. In this paper, we propose to jointly tackle these two challenges through the lens of hybrid Q functions. Specifically, we propose QAvatar, which combines the Q functions from both the source and target domains with a proper weight decay function. Through this design, we characterize the convergence behavior of QAvatar and thereby show that QAvatar achieves reliable transfer in the sense that it effectively leverages a source-domain Q function for knowledge transfer to the target domain. Through extensive experiments, we demonstrate that QAvatar achieves superior transferability across domains on a variety of RL benchmark tasks, such as locomotion and robot arm manipulation, even in the scenarios of potential negative transfer.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729942166741

thread_type:
official_review

<thread_object>
creator: reviewer_qmj1

<content>
summary:
This paper addresses cross-domain transfer in reinforcement learning (RL), particularly when the source and target domains have distinct state and action spaces. The authors propose CDRL, which combines Q-functions from the source and target domains, demonstrating its benefits in various RL tasks, including locomotion, robotic manipulation, and autonomous driving.

strengths:
- The paper is generally well-written, with clear definitions and proofs.
- The authors provide sufficient experiments on diverse tasks, including locomotion, robotic manipulation, and autonomous driving.

weaknesses:
- QAvatar shows a slight improvement in data efficiency (as shown in Figure 1); however, it still requires a similar amount of training time as SAC, which learns from scratch. Since QAvatar involves training in both the source and target domains, it doubles the training time compared to SAC, which may achieve similar performance without requiring as much training time. This makes the claim of improved data efficiency questionable.
- In a similar context to the previous question, it seems that QAvatar requires full training in the source domain. According to Figure 4, if the source-domain model is not fully trained (e.g., of low quality), QAvatar provides no benefits in such cases; instead, it is more practical to train the model directly in the target doamin from scratch. Then, what is the benifict of the cross domain transfer in this context? Could the authors provide additional experiments with other baselines, such as FT or CAT, under the same settings?”
- Can the authors provide experiments on negative transfer in other locomotion tasks, such as Hopper, HalfCheetah, and Ant? Are there specific scenarios where QAvatar may fail to transfer effectively to the target domain?
- Can the authors provide more explanation for why the baseline methods, such as CAT, DCC, and CMD, fail to transfer? It appears that CMD use the same locomotion tasks in their original papers, but no performance improvement is observed in Figure 1.

questions:
- Can the authors provide comparison with other RL algorithm as baseline such as TD3 or PPO?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628439242

thread_type:
official_review

<thread_object>
creator: reviewer_y8ga

<content>
summary:
This paper proposed a new framework for Cross-Domain Reinforcement Learning (CDRL) named QAvatar, which provides reliable knowledge transfer from a source-domain to a target domain via a weighted combination of the given source-domain Q function $Q_{src}$ and the learned target-domain Q function $Q_{tar}$. The proposed method can handle distinct state and action space between the source and target domains with performance guarantee. Additionally, the paper presents a practical implementation of QAvatar using a normalizing-flow-based state-action mapping and demonstrates its superior performance over a range of baseline methods through extensive experiments across various environments.

strengths:
- The paper is well-written and easy to follow. The notations are well-explained, visualizations are clear and helpful in supporting the conclusions.
- The motivation and main idea of the proposed method is clear and practical.
- The proposed method can improve sample efficiency and avoid negative transfer, the authors provide both theoretical analysis and empirical experiments across a rich set of environments with various state and action dimensions. The conclusion from the empirical experiments are well aligned with the theoretical analysis.
- The proposed method is robust to low quality source-domain policy as well as negative transfer scenarios.
- Authors provide reproduction and sanity checks for their baseline methods.

weaknesses:
- The performance margin between QAvatar and some other baselines various dramatically across different environments. A more thorough explanation will be helpful.
- The proposed method still relies on the assumption that the target-domain agent has access to $Q_{src}$ and $V_{src}$ in addition to $\pi_{src}$. What if we only have access to a pre-trained policy from the source-domain and nothing else (that is more practical in most real-world application scenarios).
- The proposed framework can only handle knowledge transfer from one source-domain and one target-domain, and the inter-domain mapping is required to be trained from scratch for every source-target pair.

questions:
- Why CAT outperformed QAvatar in Table wiping task, while QAvatar has significant advantages over CAT in other two Robosuite environments block lifting and door opening? block stacking and door opening should be harder than table wiping and they have higher state and action dimensions as well.
- What will happen if an adversarial source-domain policy is given ($Q_{src}$ is "opposite" to $Q_{tar}$ in some sense), will it harm the training process via the weighted sum mechanism between the $Q_{src}$ and $Q_{tar}$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730785130987

thread_type:
official_review

<thread_object>
creator: reviewer_dfa3

<content>
summary:
The paper proposes QAvatar, a novel CDRL framework that improves data efficiency by combining Q functions from both source and target domains with weight decay functions. Theoretical analysis shows that QAvatar effectively mitigates negative transfer while ensuring reliable transfer. Extensive experiments demonstrate significant improvements in data efficiency and robustness against negative transfer in various RL benchmark tasks.

strengths:
1. QAvatar is a good cross-domain reinforcement learning framework that effectively improves data efficiency , showing great potential for application.

2. The cross-domain Bellman loss function proposed in the paper provides better performance guarantees.

3. The authors conducted extensive experiments, demonstrating that QAvatar achieves excellent results in various environments, indicating high reliability.

weaknesses:
The experimental analysis for the lower quality source section is quite brief. Could you explain why the method performs so well on lower quality sources?

questions:
See questions in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730809089170

thread_type:
official_review

<thread_object>
creator: reviewer_dfmq

<content>
summary:
This paper focuses on cross-domain reinforcement learning and proposes QAvatar to solve transfer across tasks with different state-action spaces. QAvatar includes inter-task mapping learning and combines Q functions from source and target tasks with a proper weight decay function. Experiments are conducted on robotics tasks, compared with several previous transfer methods.

strengths:
1. Learning the inter-task mapping via a Bellman-like loss function looks novel;
2. Both theoretical and empirical analysis are provided.

weaknesses:
1. Although the paper provides theoretical analysis, it heavily depends on previous work.
2. The key implication 2 of Theorem 1 doesn't look original, as most transfer methods include a decay function, that can prevent negative transfer with a large training step t.
3. Some related work should be at least discussed [1-3] in this paper. At least one method with a theoretical guarantee should be compared, for example [3,4] to support the claim in the experiment section.

[1] Measuring the distance between finite markov decision processes. 2016.

[2] A Transfer Approach Using Graph Neural Networks in Deep Reinforcement Learning. 2024.

[3] Cross-Domain Policy Adaptation by Capturing Representation Mismatch. 2024.

[4] Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers. 2021.

questions:
In Fig 1, it looks like SAC learn from scratch outperforms all other transfer baselines in most of the tasks. The proposed QAvatar is only slightly better than SAC learn from scratch (some of the results are not statistically significant as the errorbars are overlapped). is there any reason for that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732391224055

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response (Part 1)

comment:
We greatly thank all the reviewers for the valuable feedback and insightful suggestions. Here we first provide our response to the questions asked by multiple reviewers, and the responses to the rest of the questions are posted below each individual review.

**Q1: Explain why most of the baselines are poor compared to SAC.**

A1: **Regarding DCC and CMD**: The scores reported in the original papers are indeed much lower than those of SAC, as summarized in the sanity check for reproduction in Section C.2 of Appendix C of the original manuscript (also see the table below). 
The main reason for the poor performance of DCC and CMD is that they learn in an unsupervised manner and hence do not utilize any information about the target rewards during training. 
|             | DCC (original) | DCC (reproduced) | CMD (original) | CMD (reproduced) | SAC             | QAvatar (Ours)  |
| ----------- | -------------- | ---------------- | -------------- | ---------------- | --------------- | --------------- |
| Swimmer     | 204 $\pm$ 56   | 132 $\pm$ 47     | 44 $\pm$ 38    | 41 $\pm$ 14      | 235 $\pm$ 141   | 316 $\pm$ 139   |
| Halfcheetah | 2471 $\pm$ 382 | 1360 $\pm$ 729   | 2114 $\pm$ 332 | 303 $\pm$ 75     | 11445 $\pm$ 1897 | 12819 $\pm$ 679 |
| Ant         | N/A             | 973 $\pm$ 501    | 649 $\pm$ 347  | 882 $\pm$ 52     | 2290 $\pm$ 785  | 2840 $\pm$ 1532 |

Remark: Regarding DCC, we directly use the official implementation provided by the original DCC paper. Moreover, as there is no CMD implementation available, we reproduced CMD by referring to the source code of DCC as these two algorithms are similar. We also follow the same hyperparameter configurations as listed in the CMD paper. With that said, we have tried our best in reproducing the baseline results. 

**Regarding CAT**: The issues with its performance are mainly three-fold: 

(1) **Parameter-based transfer**: CAT can be viewed as a parameter-based RL transfer method as the source-domain policy is transferred by taking a weighted linear combination of the hidden layer parameters of the source-domain and target-domain policies. In the general context of transfer learning, parameter-based transfer is known to rely on the assumption that the hidden layers of the source model can capture feature representations that are good to both the source and target domains (e.g., see [Zhuang et al., 2021]). However, there is no guarantee that this assumption would hold, especially when the domains are fairly dissimilar. Moreover, such direct neural network parameter sharing can be rather sensitive to the magnitude of the hidden layer parameters across the source and target models.

(2) **No theoretical guarantees**:  Recall that CAT replies on two major loss terms for cross-domain transfer, namely the Mutual Information loss $L\_{\text{MI}}$ and the Cycle Consistency loss $L\_{\text{cyc}}$. However, there is no theoretical guarantee on whether small $L\_{\text{MI}}$ and $L_{\text{cyc}}$ is a sufficient condition for effective cross-domain transfer. Moreover, based on (1), without additional assumptions, there is no clear theoretical connection between the performance of the source-domain policy and that of the weighted linear combination in the parameter space. Therefore, this also motivates the need for an algorithm with theoretical grounding like QAvatar. 

(3) **Potential effect of the base algorithm**: Given that CAT is fairly sophisticated and involves many components (cf. Figure 1 of the CAT paper), we use the official implementation provided by the CAT paper to ensure the correctness and reproducibility. CAT is by default a PPO-based algorithm, and we conjecture that its performance can be slightly correlated with the performance of vanilla PPO.  We provide the learning curves for four environments (i.e., HalfCheetah, Hopper, Door Opening, and Table Wiping) at https://imgur.com/a/347qFho and also in Figure F.6 in Appendix F. Specifically:
* Door Opening: Both CAT and PPO perform poorly.
* Hopper and Halfcheetah: While PPO performs reasonably well, CAT still suffers.
* Table Wiping: Both CAT and PPO perform quite well.
This corroborates the slight correlation between CAT and its base algorithm PPO.

[Zhuang et al., 2021] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, abd Hengshu Zhu, “A Comprehensive Survey on Transfer Learning,” Proceedings of the IEEE, 2021.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732391275705

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response (Part 2)

comment:
**Q2: Discuss the statistical significance of QAvatar’s performance.**

A2:  The overall advantage of QAvatar is indeed statistically significant compared to SAC: To provide a more reliable statistical analysis of QAvatar’s performance, we follow the general methodology proposed by [Agarwal et al., 2021] and calculate the Interquartile Mean (IQM) using rliable (https://github.com/google-research/rliable). This approach enables better evaluation of results at an aggregated level. Due to the varying reward ranges in different environments, as suggested by [Agarwal et al., 2021], we normalized the total reward to the range [0,1] in each environment to allow comparisons across environments. 

IQM: https://imgur.com/a/65TrGGM (also in Figure F.2 in Appendix F)

The above results show the aggregated IQMs for all environments (including Locomotion, Robot Arm Manipulation, and Autonomous Driving), along with the 95% stratified bootstrap confidence intervals. We observe that QAvatar indeed achieves significantly better performance than all the baselines.

[Agarwal et al., 2021] R. Agarwal, R., M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” NeurIPS 2021.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) This paper proposes QAvatar, a cross-domain reinforcement learning framework that addresses the challenge of knowledge transfer between domains with distinct state and action spaces. QAvatar leverages a weighted combination of a learned target-domain Q function and a pre-trained source-domain Q function to improve sample efficiency while mitigating negative transfer. The proposed hybrid Q function approach combines information from both domains and uses a decay function to adjust the reliance on the source domain over time. The paper provides theoretical analysis establishing the convergence properties of QAvatar. The experimental results showcase QAvatar’s ability to accelerate learning in the target domain, outperform baseline methods (including those learning from scratch), and maintain robustness even when the source domain model is of low quality or introduces negative transfer.

(b) Strengths:
- The paper provides both theoretical guarantees for the convergence of QAvatar and extensive empirical validation across diverse environments, demonstrating the practical benefits of the proposed approach.
- Robustness and mitigation of negative transfer: QAvatar incorporates a decay function that allows it to adapt to scenarios where the source domain model is of low quality or might lead to negative transfer, enhancing its reliability and practical applicability.
- The paper provides clear explanations of the methodology, theoretical analysis, and experimental setup. 

(c) Weaknesses:
- The current study focuses on transfer from a single source domain, instead of a generalist generalization setting. 

(d) The decision is to reject. There is no obvious strengths of the paper. All the reviewers are on the borderline (5, 6, 6, 6). I think the impact of transferring from single source domain to single target domain with limited evaluation in sim is perhaps too little impact to the current community -- which seeks for impacts that are realistic, scalable, and generalizable.