Title: Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: World models

reinforcement learning

visual control
Abstract: Training offline RL models using visual inputs poses two significant challenges, *i.e.*, the overfitting problem in representation learning and the overestimation bias for expected future rewards. Recent work has attempted to alleviate the overestimation bias by encouraging conservative behaviors. This paper, in contrast, tries to build more flexible constraints for value estimation without impeding the exploration of potential advantages. The key idea is to leverage off-the-shelf RL simulators, which can be easily interacted with in an online manner, as the “*test bed*” for offline policies. To enable effective online-to-offline knowledge transfer, we introduce CoWorld, a model-based RL approach that mitigates cross-domain discrepancies in state and reward spaces. Experimental results demonstrate the effectiveness of CoWorld, outperforming existing RL approaches by large margins.

Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning

Abstract

Training offline RL models using visual inputs poses two significant challenges, i.e. , the overfitting problem in representation learning and the overestimation bias for expected future rewards. Recent work has attempted to alleviate the overestimation bias by encouraging conservative behaviors. This paper, in contrast, tries to build more flexible constraints for value estimation without impeding the exploration of potential advantages. The key idea is to leverage off-the-shelf RL simulators, which can be easily interacted with in an online manner, as the ' test bed ' for offline policies. To enable effective online-to-offline knowledge transfer, we introduce CoWorld, a model-based RL approach that mitigates cross-domain discrepancies in state and reward spaces. Experimental results demonstrate the effectiveness of CoWorld, outperforming existing RL approaches by large margins.

1 Introduction

Learning control policies with visual observations can be challenging due to high interaction costs with the physical world. Offline reinforcement learning (RL) is a promising approach to address this challenge [11, 21, 37, 3, 65]. However, the direct use of current offline RL algorithms in visual control tasks presents two primary difficulties. Initially, offline visual RL is more prone to overfitting issues during representation learning, as it involves extracting hidden states from the limited, highdimensional visual inputs. Moreover, like its state-space counterpart, offline visual RL is susceptible to the challenge of value overestimation, as we observe from existing methods [22, 16].

Improving offline visual RL remains an underexplored research area. We aim to balance between overestimating and over-conservatism of the value function to avoid excessively penalizing the estimated values beyond the offline data distribution. Intuitively, we should not overly constrain the exploration with potential advantages. Our basic idea, as illustrated in Figure 1, is to leverage readily available online simulators for related (not necessarily identical) visual control tasks as auxiliary source domains, so that we can frame offline visual RL as an offline-online-offline transfer learning problem to learn mildly conservative policies.

Offline

dataset

State

alignment

Source	agent

Rich	interac1ons

Online

environment

Source	world	model

Reward

alignment

Target	world	model

Target	agent

Value

constraint

State

explora8on

1

OOD	data

Overes&ma&on

Target	true	value

(a)

Value

In-domain

OOD	data

Target-informed

source	cri&c

(c)

Value

In-domain

OOD	data

(b)

Value

In-domain

Over-conserva&sm

OOD	data

Mild-conserva&sm

by	CoWorld

(d)

Value

In-domain

We present a novel model-based transfer RL approach called Collaborative World Models (CoWorld). Specifically, we train separate world models and RL agents for source and target domains, each with domain-specific parameters. To mitigate discrepancies between the world models, we introduce a novel representation learning scheme comprising two iterative training stages. These stages, as shown in Figure 1, facilitate the alignment of latent state distributions ( offline to online ) and reward functions ( online to offline ), respectively. By doing so, the source domain critic can serve as an online 'test bed' for assessing the target offline policy. It is also more 'knowledgeable' as it can actively interact with the online environment and gather rich information. Another benefit of the domain-collaborative world models is the ability to alleviate overfitting issues of offline representation learning, leading to more generalizable latent states derived from limited offline visual data.

For behavior learning in the offline dataset, we exploit the knowledge from the source model and introduce a mild regularization term to the training objective of the target domain critic model. This regularization term encourages the source critic to reevaluate the target policy . As illustrated in Figure 2, it allows for flexible constraint on overestimated values of trajectories that receive low values from the 'knowledgeable' source critic. Conversely, if a policy yields high values from the source critic, we prefer to retain the original estimation by the offline agent. This approach is feasible because the source critic has been aligned with the target domain during world model learning.

We showcase the effectiveness of CoWorld in offline visual control tasks across the Meta-World, RoboDesk, and DeepMind Control benchmarks. Our approach is shown to be readily extendable to scenarios with multiple source domains. It effectively addresses value overestimation by transferring knowledge from auxiliary domains, even in the presence of diverse physical dynamics, action spaces, reward scales, and visual appearances. In summary, our work brings the following contributions:

We innovatively frame offline visual RL as a domain transfer problem. The fundamental idea is to harness cross-domain knowledge to tackle representation overfitting and value overestimation in offline visual control tasks.

We present CoWorld, a method that follows the offline-online-offline paradigm, incorporating specific techniques of world model alignment and flexible value constraints.

2 Problem Setup

We consider offline visual reinforcement learning as a partially observable Markov decision process (POMDP) that aims to maximize the cumulative reward in a fixed target dataset B ( T ) . We specifically focus on scenarios where auxiliary environments are accessible, enabling rich interactions and efficient online data collection. The goal is to improve the offline performance of the target POMDP 〈 O ( T ) , A ( T ) , T ( T ) , R ( T ) , γ ( T ) 〉 through knowledge transfer from the source POMDPs 〈 O ( S ) , A ( S ) , T ( S ) , R ( S ) , γ ( S ) 〉 . These notations respectively denote the space of visual observations, the space of actions, the state transition probabilities, the reward function, and the discount factor.

For example, in one of our experiments, we employ RoboDesk as the offline target domain and various tasks from Meta-World as the source domains. As illustrated in Table 1, these two environments present notable distinctions in physical dynamics, action spaces, reward definitions, and visual

appearances as the observed images are from different camera views. Our priority is to address domain discrepancies to enable cross-domain behavior learning.

3 Method

In this section, we present the technical details of CoWorld, which consists of a pair of world models {M ϕ ′ , M ϕ } , actor networks { π ψ ′ , π ψ } , and critic networks { v ξ ′ , v ξ } , where { ϕ, ψ, ξ } and { ϕ ′ , ψ ′ , ξ ′ } are respectively target and source domain parameters. As potential cross-domain discrepancies may exist in all elements of {O , A , T , R} , the entire training process is organized into three iterative stages, following an offline-online-offline transfer learning framework:

Offline-to-online state alignment : Train the offline world model M ϕ by aligning its state space with that of the source world model M ϕ ′ .

Online-to-offline reward alignment : Train M ϕ ′ and { π ψ ′ , v ξ ′ } in the online environment by incorporating the target reward information.

Online-to-offline value constraint : Train the target offline-domain agent { π ψ , v ξ } with value constraints provided by the source critic v ξ ′ .

3.1 Offline-to-Online State Alignment

Source model pretraining. We start with a source domain warm-up phase employing a modelbased actor-critic method known as DreamerV2 [16]. To facilitate cross-domain knowledge transfer, we additionally introduce a state alignment module, which is denoted as g ( · ) and implemented using the softmax operation. The world model M ϕ ′ consists of the following components:

Recurrent transition: h ( S ) t = f ϕ ′ ( h ( S ) t -1 , z ( S ) t -1 , a ( S ) t -1 ) Image encoding: e ( S ) t = e ϕ ′ ( o ( S ) t ) Posterior state: z ( S ) t ∼ q ϕ ′ ( h ( S ) t , e ( S ) t ) Prior state: ˆ z ( S ) t ∼ p ϕ ′ ( h ( S ) t ) Reconstruction: ˆ o ( S ) t ∼ p ϕ ′ ( h ( S ) t , z ( S ) t ) Reward prediction: ˆ r ( S ) t ∼ r ϕ ′ ( h ( S ) t , z ( S ) t ) Discount factor: ˆ γ ( S ) t ∼ p ϕ ′ ( h ( S ) t , z ( S ) t ) State alignment target: s ( S ) t = g ( e ( S ) t ) , (1)

where ϕ ′ represents the combined parameters of the world model. We train M ϕ ′ on the dynamically expanded source domain experience replay buffer B ( S ) by minimizing

We train the source actor π ψ ′ (ˆ z t ) and critic v ξ ′ (ˆ z t ) with the respective objectives of maximizing and estimating the expected future rewards E p ϕ ′ ,p ψ ′ [ ∑ τ ≥ t ˆ γ τ -t ˆ r τ ] generated by M ϕ ′ . Please refer to Appendix A.3 for more details. We deploy π ψ ′ to interact with the auxiliary environment and collect new data for further world model training.

State alignment. A straightforward transfer learning solution is to train the target agent in the offline dataset upon the checkpoints of the source agent. However, it may suffer from a potential mismatch issue due to the discrepancy in tasks, visual observations, physical dynamics, and action spaces across various domains. This becomes more severe when the online data is collected from

environments that differ from the offline dataset ( e.g. , Meta-World → RoboDesk). We tackle this issue by separating the parameters of the source and the target agents while explicitly aligning their latent state spaces. Concretely, the target world model M ϕ has an identical network architecture to the source model M ϕ ′ . We feed the same target domain observations sampled from B ( T ) into these models and close the distance of e ϕ ′ ( o ( T ) t ) and e ϕ ( o ( T ) t ) . We optimize M ϕ by minimizing

where sg ( · ) indicates gradient stopping and we use the encoding from the source model as the state alignment target. As the source world model can actively interact with the online environment and gather rich information, it keeps the target world model from overfitting the offline data. The importance of this loss term is governed by β 2 . We examine its sensitivity in the experiments.

3.2 Online-to-Offline Reward Alignment

To enable the source agent to value the target policy, it is essential to provide it with prior knowledge of the offline task. To achieve this, we train the source reward predictor r ϕ ′ ( · ) using mixed data from both of the replay buffers B ( S ) and B ( T ) . Through the behavior learning on source domain imaginations, the target-informed reward predictor enables the source RL agent to assess the imagined states produced by the target model and provide a flexible constraint to target value estimation (as we will discuss in Section 3.3).

Specifically, we first sample a target domain data trajectory { ( o ( T ) t , a ( T ) t , r ( T ) t ) } T t =1 from B ( T ) ( Line 14 in Alg. 1). We then use the source world model parametrized by ϕ ′ to extract corresponding latent states and relabel the target-informed source reward ( Line 15 in Alg. 1):

where k is the target-informed reward factor, which acts as a balance between the true target reward r ( T ) t and the output of the source reward predictor r ϕ ′ ( · ) provided with target states. It is crucial to

emphasize that using the target data as inputs to compute r ϕ ′ ( · ) is feasible due to the alignment of the target state space with the source state space.

We jointly use the relabeled reward ˜ r ( S ) t and the original source domain reward r ( S ) t sampled from B ( S ) to train the source reward predictor. This training is achieved by minimizing a maximum likelihood estimation (MLE) loss:

where the second term measures the negative log-likelihood of observing the relabelled source reward ˜ r ( S ) t . η represents a hyperparameter that gradually decreases from 1 to 0 . 1 throughout this training stage. Intuitively, η controls the progressive adaptation of the well-trained source reward predictor to the target domain with limited target reward supervision. We integrate Eq. (5) into Eq. (2) to train the entire world model M ϕ ′ for the source domain agent ( Line 16 in Alg. 1) and subsequently perform behavior learning to enable the source critic to assess the target policy ( Lines 17-19 in Alg. 1).

3.3 Min-Max Value Constraint

In the behavior learning phase of the target agent ( Lines 8-10 of Alg. 1), we mitigate value overestimation in the offline dataset by introducing a min-max regularization term to the objective function of the target critic model v ξ . Initially, we use the auxiliary source critic v ξ ′ to estimate the value function of the imagined target states. Following that, we train v ξ by additionally minimizing the maximum value among the estimates provided by source and target critics:

where V ( T ) t incorporates a weighted average of reward information over an n -step future horizon. The first term in the provided loss function fits cumulative value estimates (whose specific formulation can be located in Appendix A.3), while the second term regularizes the overestimated values for outof-distribution data in a mildly conservative way. The hyperparameter α represents the importance of the value constraint. The sg ( · ) operator indicates that we stop the gradient to keep the source critic from being influenced by the regularization term.

This approach provides flexibly conservative value estimations, finding a balance between mitigating overestimation and avoiding excessive conservatism in the value function. When the target critic overestimates the value function, the source critic is less vulnerable to the value overestimation problem as it is trained with rich interaction data. Thus, it is possible to observe v ξ (ˆ z ( T ) t ) > v ξ ′ (ˆ z ( T ) t ) , and our approach is designed to decrease the output of v ξ to the output of v ξ ′ . This prevents the target critic from overestimating the true value. Conversely, when the source critic produces greater values in v ξ ′ (ˆ z ( T ) t ) , the min-max regularization term does not contribute to the training of the target critic v ξ . This encourages the exploration of potentially advantageous states within the imaginations of the target world model. In line with DreamerV2 [16], we train the target actor π ψ by maximizing a REINFORCE objective function with entropy regularization, allowing the gradients to backpropagate directly through the learned dynamics:

As previously mentioned, V ( T ) t involves a weighted average of reward information over an n -step future horizon, with detailed formulation provided in Appendix A.3.

Furthermore, it is crucial to note that CoWorld can readily be extended to scenarios with multiple source domains by adaptively selecting a useful task as the auxiliary domain. This extension is easily achieved by measuring the distance of the latent states between the target domain and each source domain. For technical details of the adaptive source domain selection, please refer to Appendix C.

4 Experiments

4.1 Experimental Setups

Datasets. We evaluate CoWorld across three visual control environments, i.e. , Meta-World [54], RoboDesk [18], and DeepMind Control Suite (DMC) [47], including both cross-task and crossenvironment setups (Meta-World → RoboDesk). Inspired by D4RL [9], we build offline datasets of medium-replay quality using DreamerV2 [16]. The datasets comprise all the samples in the replay buffer collected during the training process until the policy attains medium-level performance, defined as achieving 1 / 3 of the maximum score that the DreamerV2 agent can achieve. Please refer to Appendix B.2 for further results of CoWorld trained with medium-expert offline data.

Compared methods. We compare CoWorld with both model-based and model-free RL approaches, including Offline DV2 [25], DrQ+BC [25], CQL [25], CURL [22], and LOMPO [39]. In addition, we introduce the DV2 Finetune method, which involves taking a DreamerV2 [16] model pretrained in the online source domain and subsequently finetuning it in the offline target dataset. Furthermore, DV2 Finetune can be integrated with the continual learning method, Elastic Weight Consolidation (EWC) [19], to regularize the model for preserving source domain knowledge, i.e. , Finetune+EWC . Please refer to Appendix E for more details.

4.2 Cross-Task Experiments on Meta-World

Meta-World is an open-source simulated benchmark designed for solving a wide range of robot manipulation tasks. We select 6 tasks as either the offline dataset or potential candidates for the online auxiliary domain. These tasks include: Door Close ( DC ∗ ), Button Press ( BP ), Window Close ( WC ), Handle Press ( HP ), Drawer Close ( DC ), Button Topdown ( BT ).

Main results. As shown in Table 2, we compare the results of CoWorld with other models on Meta-World. CoWorld achieves the best performance in all 6 tasks. Notably, it outperforms Offline DV2 [25], a method also built upon DreamerV2 and specifically designed for offline visual RL. For the online-to-offline finetuning models, DV2 Finetune achieves the second-best results by leveraging transferred knowledge from the auxiliary source domain. However, we observe that its performance experiences a notable decline in scenarios ( e.g. , Meta-World → RoboDesk) involving significant data distribution shifts between the source and the target domains in visual observation, physical dynamics, reward definition, or even the action space of the robots. Another important baseline model is DV2 Finetune+EWC , which focuses on mitigating the catastrophic forgetting of the knowledge obtained in source domain pretraining. Nevertheless, without additional model designs for domain adaptation, retaining source domain knowledge may eventually lead to a decrease in performance in the target domain. The LOMPO model suffers from the negative transfer effect when incorporating a source pretraining stage. It achieves an average return of 1 , 712 when it is trained from scratch in the offline domain while achieving an average return of 792 for online-to-offline finetuning. It implies that a naïve transfer learning method may degenerate the target performance due to unexpected bias.

Results with a random source domain. Given that we present the best-source results in Table 2, where we manually select one source task from Meta-World, one may cast doubt on the influence of

Multi-

Source

Source	Task

Target	Task

CoWorld	/

Offline	DV2

Source	Task

Episode	Return

Multi-

Source

DC

HP

BP

WC

BT

DC*

HP

BP

WC

BT

DC

MulB-

Source

DC*

HP

BP

WC

BT

DC

Source Task

DC

HP

BP

WC

BT

1.85

1.81

1.40

1.81

1.61

1.80

1.5

1.4

0.94

1.03

1.14

1.15

1.03

1.14

1.3

1.15

1.11

1.13

1.11

1.15

1.15

1.2

1.1

13.5216.4414.23

15.5813.16

16.04

1.0

1.24

1.14

1.20

0.96

1.00

1.20

0.9

0.8

1.23

1.24

1.06

1.31

1.28

1.21

0.7

(a)

Button Press

Push Button

(b)

Window Close

Open Slide

(c)

Drawer Close

Drawer Open

(d)

Handle Press

Upright Block off Table

fIterations

fIterations

ofIteratior

mberofIte

domain discrepancies between the auxiliary environment and the target offline dataset. In Figure 3 (Left), the transfer matrix of CoWorld among the 6 tasks of Meta-World is presented, where values greater than 1 indicate positive domain transfer effects. Notably, there are challenging cases with weakly related source and target tasks. In the majority of cases ( 26 out of 30 ), CoWorld outperforms Offline DV2 , as illustrated in the heatmap.

Results with multiple source domains. It is crucial to note that CoWorld can be easily extended to scenarios with multiple source domains by adaptively selecting a useful task as the auxiliary domain. From Table 2, we can see that the multi-source CoWorld achieves comparable results to the models trained with manually designated online simulators. In Figure 3 (Left), multi-source CoWorld achieves positive improvements over Offline DV2 in all cases, approaching the best results of models using each source task as the auxiliary domain. In Figure 3 (Right), it also consistently outperforms the DV2 Finetune baseline model. These results demonstrate our approach's ability to execute without strict assumptions about domain similarity and its ability to automatically identify a useful online simulator from a set of both related and less related source domains.

4.3 Cross-Environments: Meta-World to RoboDesk

To explore cross-environment transfer with more significant domain gaps, we employ four tasks from RoboDesk to construct individual offline datasets, i.e., Push Button , Open Slide , Drawer Open , Upright Block off Table . These tasks require handling randomly positioned objects with image inputs. Table 1 presents the differences between the two environments in physical dynamics, action space, reward definitions, and visual appearances.

Figure 4 presents quantitative comparisons, where CoWorld outperforms Offline DV2 and DV2 Finetune by large margins. For the best-source experiments, we manually select one source domain from Meta-World. For the multi-source experiments, we jointly use all Meta-World tasks as the source domains. In contrast to prior findings, directly finetuning the source world model in this cross-environment setup, where there are more pronounced domain discrepancies, does not result in significant improvements in the final performance. In comparison, CoWorld more successfully

















×



4

































(a) Meta-World

→

RoboDesk (

Push Green Button

)

-300

-100

100

300

500

700

Door

Close

Button

Press

Handle

Press

Window

Close

Button

Topdown

Drawer

Close

Offline DV2

DrQ-v2

CQL

CoWorld w/o Max

CoWorld

(b) Y-axis

=

Estimated value

-

True value

addresses these challenges by leveraging domain-specific world models and RL agents, and explicitly aligning the state and reward spaces across domains. We also showcase the performance of multisource CoWorld, which achieves comparable results to the best-source model that exclusively uses our designated source domain.

4.4 Cross-Dynamics Experiments on DMC

DMC is a widely explored benchmark for continuous control. We use the Walker and Cheetah as the base agents and make modifications to the environment to create a set of 8 distinct tasks, i.e. , Walker Walk ( WW ), Walker Downhill ( WD ), Walker Uphill ( WU ), Walker Nofoot ( WN ), Cheetah Run ( CR ), Cheetah Downhill ( CD ), Cheetah Uphill ( CU ), Cheetah Nopaw ( CN ). Particularly, Walker Nofoot is a task in which we cannot control the right foot of the Walker agent. Cheetah Nopaw is a task in which we cannot control the front paw of the Cheetah agent.

We apply the proposed multi-source domain selection method to build the domain transfer settings shown in Table 3. It is worth noting that CoWorld outperforms the other compared models in 5 out of 6 DMC offline datasets, and achieves the second-best performance in the remaining task. On average, it outperforms Offline DV2 by 169 . 6% and outperforms DrQ+BC by 37 . 5% . Corresponding qualitative comparisons can be found in Appendix B.1.

4.5 Further Analyses

Ablation studies. We conduct a series of ablation studies to validate the effectiveness of state space alignment (Stage A), reward alignment (Stage B), and min-max value constraint (Stage C). We show corresponding results on the offline Push Green Button dataset from RoboDesk in Figure 5(a). The performance experiences a significant decline when we abandon each training stage in CoWorld.

Can CoWorld address value overestimation? We evaluate the values estimated by the critic network of CoWorld on the offline Meta-World datasets when the training process is finished. In Figure 5(b), we compute the cumulative value predictions throughout 500 steps. The true value

is determined by calculating the discounted sum of the actual rewards obtained by the actor in the same 500 -steps period. We observe that existing approaches, including Offline DV2 and CQL , often overestimate the value functions in the offline setup. The baseline model ' CoWorld w/o Max ' is a variant of CoWorld that incorporates a brute-force constraint on the critic loss. It reformulates Eq. (6) as ∑ H -1 t =1 1 2 ( v ξ (ˆ z t ) -sg ( V t )) 2 + αv ξ (ˆ z t ) . As observed, this model tends to underestimate the true value function, which can potentially result in overly conservative policies as a consequence. In contrast, the values estimated by CoWorld are notably more accurate and more akin to the true values.

Dependence of CoWorld to domain similarities. We further investigate the dependence of CoWorld on domain similarity from the perspectives of different observation spaces and reward spaces. We first explore how CoWorld performs when we only have source domains with significantly distinct observation spaces from the target domain. As illustrated in Table 4, the agent receives low-dimensional state inputs in the source domain (Meta-World) and high-dimensional images in the target domain (RoboDesk). We can see that CoWorld outperforms Offline DV2 by 13 . 3% and 34 . 0% due to the ability to leverage low-dimensional source data effectively. Notably, the finetuning method ( DV2 Finetune ) is not applicable in this scenario. In Table 5, we also observe that CoWorld benefits from a source domain, even with a significantly different reward signal. Unlike previous experiments, we use a sparse reward function for the source Meta-World tasks. It is set to 500 only upon task completion and remains 0 before that. The experimental results demonstrate that although excessively sparse rewards can hinder the training process, CoWorld still achieves an average performance gain of 6 . 6% compared to DV2 Finetune under the same setting.

Comparison to jointly training one world model across domains. Notably, CoWorld is implemented with separate world models for the source and target domains. Alternatively, we can employ a jointly trained world model across various domains for more efficient memory usage. In Table 6, we compare the results from the original CoWorld and ' Multi-Task DV2 '. Multi-Task DV2 involves training DreamerV2 on both offline and online data with a joint world model and separate actor-critic models. CoWorld consistently performs better. Intuitively, using separate world models allows the source and target domains to have different physical dynamics, observation spaces, or reward formations, as the scenarios shown in Table 4 and Table 5.

Hyperparameter sensitivity. We conduct sensitivity analyses on Meta-World ( DC → BP ). From Figure 6, we observe that when β 2 for the domain KL loss is too small, the state alignment between the source and target encoders becomes insufficient, hampering the transfer learning process. Conversely, if β 2 is too large, the target encoder becomes excessively influenced by the source encoder, resulting in a decline in performance. We also find that the target-informed reward factor k plays a crucial role in balancing the influence of source data and target reward information, which achieves a consistent

improvement over DV2 Finetune ( 2456 ± 661 ) in the range of [0 . 1 , 0 . 7] . Moreover, we discover that the hyperparameter α for the target value constraint performs well within [1 , 3] , while an excessively larger α may result in value over-conservatism in the target critic.

5 Related Work

Learning control policies from images is critical in real-world applications. Existing approaches can be grouped by the use of model-free [22, 41, 44, 48, 36] or model-based [15, 14, 16, 43, 35, 13, 28, 29, 61, 51] RL algorithms. In offline RL, agents leverage pre-collected offline data to optimize policies and encounter challenges associated with value overestimation [23]. Previous methods mainly suggest taking actions that were previously present in the offline dataset or learning conservative value estimations [11, 21, 4, 55, 53, 40]. Recent approaches have introduced specific techniques to address the challenges associated with offline visual RL [27, 7, 23, 2, 39, 52, 43, 57, 5, 25]. Rafailov et al. [39] proposed to handle high-dimensional observations with latent dynamics models and uncertainty quantification. Cho et al. [5] proposed synthesizing the raw observation data to append the training buffer, aiming to mitigate the issue of overfitting. In a related study, Lu et al. [25] established a competitive offline visual RL model based on DreamerV2 [16], so that we use it as a significant baseline of our approach.

Our work is also related to transfer RL, which is known as to utilize the knowledge learned in past tasks to facilitate learning in unseen tasks [64, 42, 58, 45, 59, 8, 49, 46, 12, 20, 38, 24, 33]. Most existing approaches related to offline dataset + simulator focus on the offline-to-online setup, where the policy is initially pretrained on the offline dataset and then finetuned and deployed on an interactive environment [33, 60, 56, 63]. These methods aim to bridge the gap between offline and online learning and facilitate fast adaptation of the model to the online environment. In contrast, we explore the online-to-offline setup, which provides a new remedy for the value over-estimation problem. Additionally, Niu et al. [34] introduces a dynamics-aware hybrid offline-and-online framework to integrate offline datasets and online simulators for policy optimization. Unlike CoWorld, this method primarily focuses on low-dimensional MDPs and cannot be directly used in visual control tasks. In the context of visual RL, CtrlFormer [31] learns a transferable state representation via a sample-efficient vision Transformer. APV [43] executes action-free world model pretraining on source-domain videos and finetunes the model on downstream tasks. Choreographer [28] builds a model-based agent that exploits its world model to learn and adapt skills in imaginations, the learned skills are adapted to new domains using a meta-controller. VIP [26] presents a self-supervised, goal-conditioned value-function objective, which enables the use of unlabeled video data for model pertaining. Unlike previous methods, we handle offline visual RL using auxiliary simulators, mitigating the value overestimation issues with co-trained world models.

6 Conclusions and Limitations

In this paper, we proposed a transfer RL method named CoWorld, which mainly tackles the difficulty in representation learning and value estimation in offline visual RL. The key idea is to exploit accessible online environments to train an auxiliary RL agent to offer additional value assessment. To address the domain discrepancies and to improve the offline policy, we present specific technical contributions of cross-domain state alignment , reward alignment , and min-max value constraint . CoWorld demonstrates competitive results across three RL benchmarks. An unsolved problem of CoWorld is the increased computational complexity associated with the training phase in auxiliary domains (see Appendix B.7). It is valuable to improve the training efficiency in future research.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. 62250062, 62106144, 62302246), the Shanghai Municipal Science and Technology Major Project (No. 2021SHZDZX0102), the Fundamental Research Funds for the Central Universities, and the CCFTencent Rhino-Bird Open Research Fund. This work was also supported by the Natural Science Foundation of Zhejiang Province, China (No. LQ23F010008), the High-Performance Computing Center at Eastern Institute of Technology (Ningbo), and the Ningbo Institute of Digital Twin.

References

Brian M Adams, Harvey T Banks, Hee-Dae Kwon, and Hien T Tran. Dynamic multidrug therapies for hiv: Optimal and sti control approaches. Mathematical Biosciences & Engineering , 1(2):223-241, 2004.

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In ICML , pages 104-114, 2020.

Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In ICLR , 2023.

Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning. In NeurIPS , volume 35, pages 36902-36913, 2022.

Daesol Cho, Dongseok Shim, and H Jin Kim. S2p: State-conditioned image synthesis for data augmentation in offline reinforcement learning. In NeurIPS , 2022.

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In CoRL , 2019.

Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In ICLR , 2021.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In NeurIPS , 2021.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML , pages 2052-2062, 2019.

Dibya Ghosh, Chethan Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. arXiv preprint arXiv:2304.04782 , 2023.

Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. arXiv preprint arXiv:2206.04114 , 2022.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR , 2020.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML , pages 2555-2565, 2019.

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR , 2021.

William Hua, Hongyuan Mei, Sarah Zohar, Magali Giral, and Yanxun Xu. Personalized dynamic treatment regimes in continuous time: a bayesian approach for optimizing clinical decisions with timing. Bayesian Analysis , 17(3):849-878, 2022.

Harini Kannan, Danijar Hafner, Chelsea Finn, and Dumitru Erhan. Robodesk: A multi-task reinforcement learning benchmark. https://github.com/google-research/robodesk , 2021.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. In ICLR , 2023.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurIPS , volume 33, pages 1179-1191, 2020.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In ICML , pages 5639-5650, 2020.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Xin Liu, Yaran Chen, Haoran Li, Boyu Li, and Dongbin Zhao. Cross-domain random pre-training with prototypes for reinforcement learning. arXiv preprint arXiv:2302.05614 , 2023.

Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. Transactions on Machine Learning Research , 2023.

Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In ICLR , 2023.

Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In IROS , pages 1048-1055, 2019.

Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreographer: Learning and adapting skills in imagination. In ICLR , 2023.

Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. In ICLR , 2023.

Zhiyu Mou, Yusen Huo, Rongquan Bai, Mingzhou Xie, Chuan Yu, Jian Xu, and Bo Zheng. Sustainable online reinforcement learning for auto-bidding. In NeurIPS , volume 35, pages 2651-2663, 2022.

Yao Mark Mu, Shoufa Chen, Mingyu Ding, Jianyu Chen, Runjian Chen, and Ping Luo. Ctrlformer: Learning transferable state representation for visual control via transformer. In ICML , pages 16043-16061, 2022.

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 , 2022.

Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint arXiv:2303.05479 , 2023.

Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, Xianyuan Zhan, et al. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. In NeurIPS , volume 35, pages 36599-36612, 2022.

Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models. In NeurIPS , volume 35, pages 23178-23191, 2022.

Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In ICML , pages 17359-17371, 2022.

Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven offline decision-making via invariant representation learning. In NeurIPS , 2022.

Rafael Rafailov, Kyle Beltran Hatch, Victor Kolev, John D Martin, Mariano Phielipp, and Chelsea Finn. MOTO: Offline pre-training to online fine-tuning for model-based robot learning. In 7th Annual Conference on Robot Learning , 2023.

Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Proceedings of Machine Learning Research , pages 1154-1168, 2021.

Marc Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. In NeurIPS , 2022.

Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R Devon Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. In NeurIPS , volume 34, pages 12686-12699, 2021.

Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML , pages 8583-8592, 2020.

Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In ICML , pages 19561-19579, 2022.

Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In ICML , pages 9870-9879, 2021.

Yanchao Sun, Xiangyu Yin, and Furong Huang. Temple: Learning template of transitions for sample efficient multi-task rl. In AAAI , volume 35, pages 9765-9773, 2021.

Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew Cohen, and Furong Huang. Transfer rl across observation feature spaces via model-based regularization. In ICLR , 2022.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173 , 2022.

Mengjiao Yang and Ofir Nachum. Representation matters: offline pretraining for sequential decision making. In ICML , pages 11784-11794, 2021.

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 , 2021.

Chengyang Ying, Zhongkai Hao, Xinning Zhou, Hang Su, Songming Liu, Jialian Li, Dong Yan, and Jun Zhu. Reward informed dreamer for task generalization in reinforcement learning. arXiv preprint arXiv:2303.05092 , 2023.

Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In ICML , pages 25611-25635, 2022.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In NeurIPS , volume 34, pages 28954-28967, 2021.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL , 2019.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In NeurIPS , volume 33, pages 14129-14142, 2020.

Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. In ICML , pages 40452-40474, 2023.

Hongyu Zang, Xin Li, Jie Yu, Chen Liu, Riashat Islam, Remi Tachet Des Combes, and Romain Laroche. Behavior prior representation learning for offline reinforcement learning. In ICLR , 2023.

Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. In ICML , pages 11214-11224, 2020.

Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In ICLR , 2021.

Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. In ICLR , 2023.

Wendong Zhang, Geng Chen, Xiangming Zhu, Siyu Gao, Yunbo Wang, and Xiaokang Yang. Predictive experience replay for continual visual control and forecasting. arXiv preprint arXiv:2303.06572 , 2023.

Zhiyue Zhang, Hongyuan Mei, and Yanxun Xu. Continuous-time decision transformer for healthcare applications. In International Conference on Artificial Intelligence and Statistics , pages 6245-6262. PMLR, 2023.

Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, and Jing Jiang. Adaptive policy learning for offline-to-online reinforcement learning. In AAAI , volume 37, pages 11372-11380, 2023.

Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.07888 , 2020.

Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. In ICLR , 2023.

Appendix

In this appendix, we provide the following supplementary materials: (A) Details of the proposed model, including further descriptions of the learning schemes, the notations, the world model architecture, the behavior learning objective functions, and hyperparameters. (B) Additional experimental results, including visualization of the learned policy, quantitative results on offline datasets with mixed data quality, comparison to using a pre-trained foundation model such as R3M, and computational efficiency. (C) Implementation details of the multi-source CoWorld model and further empirical analysis on the selected source domain. (D) Detailed setups of the source and target domains. (E) Details of the compared methods. (F) Potential social impacts of the proposed method.

A Model Details

A.1 Framework of CoWorld

As illustrated in Figure 7, the entire training process of CoWorld comprises three iterative stages: offline-to-online state alignment (Stage A), online-to-offline reward alignment (Stage B), and onlineto-offline value constraint (Stage C). First, we feed the same target domain observations sampled from B ( T ) into the encoders and close the distance of e ϕ ′ ( o ( T ) t ) and e ϕ ( o ( T ) t ) in Stage A. Second, in Stage B, the source reward predictor r ϕ ′ ( · ) is trained with mixed data from both of the replay buffers B ( S ) and B ( T ) . Notably, when we sample data from B ( T ) , the reward will be relabelled as the target-informed source reward. Finally, we introduce a min-max value constraint using the source critic to the target critic in Stage C.

Offline-to-OnlineStateAlignment

Online-to-OfflineRewardAlignment

Online-to-OfflineValueConstraint

TargetobservationOt

(T)

Targetobservation

0

(T)

Tt

(T)

Target observation ot

(T)

mixup

Target critic

Guide

Source critic

Encoder e

Encoder eΦ

Source reward

Target-informed

UE

predictor rΦ

DKL

Target actor T

mixup

Take an action at

(T)

Predictreward

Latent space g(es(o(m),g(ea(()

(htt)

A.2 World Model

We adopt the framework of the world model used in [16]. The image encoder is a Convolutional Neural Network (CNN). The image predictor is a transposed CNN and the transition, reward, and discount factor predictors are Multi-Layer Perceptrons (MLPs). The discount factor predictor serves as an estimate of the probability that an episode will conclude while learning behavior based on model predictions. The encoder and decoder take 64 × 64 images as inputs.

A.3 Behavior Learning

For the behavior learning of CoWorld, we use the actor-critic method from DreamerV2 [16]. The λ -target V ( T ) t in Eq. (6) is defined as follows:

where λ is set to 0 . 95 for considering more on long horizon targets. The actor and critic are both MLPs with ELU activations [6]. The target actor and critic are trained with guidance from the source critic and regress the λ -return with a squared loss. The world model is fixed during behavior learning. The source actor and critic are:

We train the source actor π ψ ′ by maximizing

The source critic v ξ ′ is optimized by minimizing

A.4 Hyperparameters

The hyperparameters of CoWorld are shown in Table 8.

B Additional Quantitative and Qualitative Results

B.1 Visualizations on Policy Evaluation

We evaluate the trained agent of different models on the Meta-World and DMC tasks and select the first 45 frames for comparison. Figure 8 and Figure 9 present examples of performing the learned policies of different models on DMC and Meta-World respectively.

t

= 1

t

= 5

t

= 10

t

= 15

t

= 20

t

= 25

t

= 30

t

= 35

t

= 40

t

= 45

CoWorld

CURL

Offline DV2

LOMPO

7人之

N

YK

以人人K

K

(a) Policy evaluation on the DMC

t

= 5

t

= 10

t

= 15

t

= 20

t

= 25

t

task

= 30

t

= 35

t

= 40

t

= 45

AAYYT

AAYYYAAAY

(b) Policy evaluation on the DMC

t

= 1

t

= 5

= 10

t

= 15

一人

t

= 20

1>

(c) Policy evaluation on the DMC

t

= 25

task

t

= 30

>

Walker Nofoot task

t

CURL

LOMPO

Offline DV2

CoWorld

CURL

LOMPO

Offline DV2

CoWorld

t

= 1

t

= 35

t

= 40

√

N

t

= 45

CoWorld

Offline DV2

CURL

LOMPO

t

= 1

t

= 5

t

= 10

t

= 15

t

= 20

t

= 25

t

= 30

t

= 35

t

= 40

t

= 45

t

= 50

B.2 Quantitative Results on DMC Medium-Expert Dataset

Similar to the data collection strategy of the medium-replay dataset, we build offline datasets with medium-expert quality using a DreamerV2 agent. The medium-expert dataset comprises all the samples in the replay buffer during the training process until the policy attains expert-level performance, defined as achieving the maximum score that the DreamerV2 agent can achieve. As shown in Table 9, CoWorld outperforms other baselines on the DMC medium-expert dataset in most tasks.

B.3 Quantitative Results on Meta-World

Figure 10(a) compares the performance of different models on Meta-World. DV2 Finetune demonstrates better performance in the initial training phase, thanks to its direct access to the source environment. Instead, CoWorld introduces auxiliary source value guidance to assist the training of the target agent. In the final phase of training, the source value guidance is more effective, and then CoWorld outperforms DV2 Finetune . Figure 10(b) presents the ablation studies of CoWorld conducted on Meta-World, highlighting the effectiveness and necessity of each training stage.

















×



4





























(a) Comparison with other baselines

















×



4































(b) Ablation studies of CoWorld

B.4 Effect of Latent Space Alignment

We feed the same observations into the source and target encoder of CoWorld and then use the tdistributed stochastic neighbor embedding (t-SNE) method to visualize the latent states. As shown in Figure 11, the representation learning alignment bridges the gap between the hidden state distributions of the source encoder and target encoder.

B.5 Additional Results on the Realistic Sim2Real Setup

Due to the limitation in experimental resources, we are unable to conduct experiments with real robots. We make efforts to construct a more realistic sim2real setup. The experiment is conducted with the identical robot control task for both the source and target domains. We manually introduce two types of noise into the visual observation and action space of the target domain, trying to mimic the complex and noisy real-world scenes.

Visual noise : We modify the original DeepMind Control environment by replacing the static background with dynamic backgrounds of random real-world videos.

Action noise : We add Gaussian noises n t sampled from N (0 , 1) to every dimension of the action in Meta-World and RoboDesk, which originally ranges in (-1,1). This mimics scenarios where the offline dataset is collected using a low-cost (less inaccurate) real robot.





























(a) Latent space before alignment





























(b) Latent space after alignment

As shown in Table 10, we compare CoWorld with the finetuned DreamerV2 model on this new setup. We apply noise of three magnitudes, w ∼ { 0 . 1 , 1 , 5 } , in the Meta-World Button Topdown task, leading to noisy actions of a real t = a t + w · n t .

From the above results, it is evident that CoWorld consistently outperforms the naive finetuning method in this 'sim2real' setup. Importantly, we assess the model under more challenging setups, with more significant domain gaps, as illustrated in Table 1.

B.6 Comparison with Pre-trained Foundation Model R3M

R3M[32] is pretrained on Ego4D human video dataset and facilitates efficient learning of downstream robotic tasks. R3M is shown to be a competitive model, particularly in its ability to transfer representations across domains with diverse visual inputs. For model comparison, we leverage the pre-trained weights of R3M from the official repository to initialize the representation model. We then perform policy optimization based on it for downstream tasks in the Meta-World environment. We respectively employ expert data, sourced from the official repository, alongside our own data, which is collected from scratch with mixed data quality. And DV2 Finetune is also pretrained in a related task and finetuned on the offline dataset. As demonstrated in Table 11, our approach outperforms the R3M / DV2 fine-tuning model.

It is important to note that:

Despite its generalizable representations, R3M is NOT specifically designed to solve the value overestimation problem, which is fundamental in offline RL. In contrast, our approach not only aligns state representations across domains but also effectively tackles the issue of value overestimation, and therefore achieves better performance.

The fine-tuning process of R3M necessitates expert demonstrations for high-quality imitation learning. However, its performance empirically deteriorates when applied to the offline dataset of the medium-replay data.

The pre-training process of R3M typically takes around 5 days on a V100 GPU, while the entire training procedure of our approach takes only about 2 days on a 3090 GPU.

B.7 Training Efficiency

As shown in Table 12, we evaluate the training/inference time on Meta-World (Handle Press → Button Topdown) using a single RTX 3090 GPU. Empirically, CoWorld achieves convergence ( 90% of the highest returns) in approximately 14 hours; while it costs DV2 Finetune about 13 hours. These results indicate that CoWorld requires a comparable training wall-clock time to DV2 Finetune , while consistently maintaining better performance in terms of returns after model convergence.

C Multi-Source CoWorld

The key idea of multi-source CoWorld is to allocate a set of one-hot weights ω i =1: M t to candidate source domains by calculating their KL divergence in the latent state space to the target domain, where i ∈ [1 , M ] is the index of each source domain. This procedure includes the following steps:

World models pretraining: We pretrain a world model for each source domain and target domain individually.

Domain distance measurement: At each training step in the target domain, we measure the KL divergence between the latent states of the target domain, produced by e ϕ ( o ( T ) t ) , and corresponding states in each source domain, produced by e ϕ ′ i ( o ( T ) t ) . Here, e ( T ) ϕ is the encoder of the target world model, and e ϕ ′ i is the encoder of the world model for the source domain i .

Auxiliary domain identification: We dynamically identify the closest source domain with the smallest KL divergence. We set ω i =1: M t as a one-hot vector, where ω i t = 1 indicates the selected auxiliary domain.

Rest of training: With the one-hot weights, we continue the rest of the proposed online-to-offline training approach. During representation learning, we adaptively align the target state space to the selected online simulator by rewriting the domain alignment loss term in Eq. (3) as

To evaluate the effectiveness of the multi-source adaptive selection algorithm, we conducted experiments on Meta-World and RoboDesk Benchmark. For each target task, two source tasks are used, including the CoWorld best-performing task and the CoWorld worst-performing task. Additionally, the sub-optimal source task is added for some target tasks.

As shown in Table 13, multi-source CoWorld can adaptively select the best source task for most multisource problems to ensure adequate knowledge transfer. The performance of multi-source CoWorld is reported in Table 2. CoWorld flexibly adapts to the transfer learning scenarios with multiple source domains, achieving comparable results to the model that exclusively uses our manually designated auxiliary simulator as the source domain (best-source). This study significantly improves the applicability of CoWorld in broader scenarios.

D Source and Target Domains

Meta-World. For the Meta-World environment, we adopt robotic control tasks with complex visual dynamics. For instance, the Door Close task requires the agent to close a door with a revolving joint while randomizing the door positions, and the Handle Press task involves pressing a handle down while randomizing the handle positions. To evaluate the performance of CoWorld on these tasks, we compare it with several baselines in six visual RL transfer tasks.

RoboDesk. We select Meta-World as the source domain and RoboDesk as the target domain. Notably, there exists a significant domain gap between these two environments. The visual observations, physical dynamics, and action spaces of the two environments are different. First, Meta-World adopts a side viewpoint, while RoboDesk uses a top viewpoint. Further, the action space of Meta-World is 4 dimensional, while that in RoboDesk is 5 -dimensional. Considering these differences, the Meta-World → RoboDesk benchmark presents a challenging transfer learning problem.

DeepMind Control. We train the source agents in standard DMC environments and train the target agents in modified DMC environments. Walker Uphill and Cheetah Uphill represent tasks in which the ground has a 15 ◦ uphill slope. Walker Downhill and Cheetah Downhill represent the tasks in which the plane has a 15 ◦ downhill slope. We evaluate the model in six tasks with different source domains and target domains.

We assume that there exist notable distinctions between the source and target domains (see Table 1). This assumption can be softened by our proposed approach that mitigates domain discrepancies between distinct source and target MDPs. Our experiments reveal that the CoWorld method exhibits a notable tolerance to inter-domain differences in visual observation, physical dynamics, reward definition, or even the action space of the robots. This characteristic makes it more convenient to choose an auxiliary simulator based on the type of robot. For example:

When the target domain involves a robotic arm ( e.g. , RoboDesk), an existing robotic arm simulation environment ( e.g. , Meta-World) can be leveraged as the source domain.

In scenarios with legged robots, environments like DeepMind Control with Humanoid tasks can serve as suitable auxiliary simulators.

For target domains related to autonomous driving, simulators like CARLA can be used.

E Compared Methods

We compare CoWorld with several widely used model-based and model-free offline methods.

Offline DV2 [25]: A model-based RL method that modifies DreamerV2 [16] to offline setting, and adds a reward penalty corresponding to the mean disagreement of the dynamics ensemble.

DrQ+BC [25]: It modifies the policy loss term in DrQ-v2 [50] to match the loss given in [10].

CQL [25]: It is a framework for offline RL that learns a Q-function that guarantees a lower bound for the expected policy value than the actual policy value. We add the CQL regularizers to the Q-function update of DrQ-v2 [21].

CURL [22]: It is a model-free RL approach that extracts high-level features from raw pixels utilizing contrastive learning.

LOMPO [39]: An offline model-based RL algorithm that handles high-dimensional observations with latent dynamics models and uncertainty quantification.

LOMPO Finetune : It pretrains a LOMPO agent [39] with source domain data and subsequently finetunes the pretrained agent in the offline target domain.

DV2 Finetune : It pretrains a DreamerV2 agent [16] in the online source domain and subsequently finetunes the pretrained agent in the offline target domain. Notably, Meta-World → RoboDesk tasks' action space is inconsistent, and we can't finetune directly. Instead, we use the maximum action space of both environments as the shared policy output dimension. For Meta-World and Meta-World → RoboDesk transfer tasks, we pretrain the agent for 160 k steps and finetune it 300 k steps. For DMC transfer tasks, we pretrain the agent for 600 k steps and finetune it for 600 k steps.

DV2 Finetune+EWC : It modifies the DV2 Finetune method with EWC [19] to regularize the model for retaining knowledge from the online source domain. The steps of pretraining and finetuning are consistent with DV2 Finetune .

F Broader Impacts

CoWorld is a transfer learning method that may benefit future research in the field of offline RL, model-based RL, and visual RL. Beyond the realm of reinforcement learning, this approach holds great potential to contribute to various domains such as robotics and autonomous driving.

In real-world scenarios of healthcare applications, Zhang et al. [62] employed offline RL algorithms to train policies using a large amount of historical dataset, determining the follow-up schedules and tacrolimus dosages in Kidney Transplantation and HIV. There are also corresponding simulators [17, 1] designed by medical domain experts, with parameters learned from real-world data.

Another practical use of the proposed setup is advertising bidding, where direct interactions with real online advertising systems for training are challenging. A recent solution involves constructing a simulated bidding environment based on historical bidding logs for interactive training, such as [30], and mitigating the inherent differences between the virtual advertising environment and real-world advertising systems. Therefore, in many real-world scenarios, it is possible to optimize the policies learned from offline datasets with simulators.

A potential negative social impact of our method is the introduction of existing biases from the additional domain. If the training data used to develop our algorithm contains biases, the model may learn those biases, leading to unfair outcomes in decision-making processes. It's crucial to carefully address biases in both data and algorithmic design to mitigate these negative social impacts.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: It is difficult to provide a theoretical explanation for CoWorld mentioned in the article.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Code is available at https://github.com/qiwang067/CoWorld .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide data and code at https://github.com/qiwang067/CoWorld . Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 4.1 and Appendix A.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Table 2 and Table 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section B.7.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics in all aspects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not involve the responsible release of data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We include the licenses at https://github.com/qiwang067/CoWorld .

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We include the documentation at https://github.com/qiwang067/ CoWorld .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing experiments and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Collaborative World Models (CoWorld), a novel approach to offline reinforcement learning (RL) with visual inputs. CoWorld conceptualizes offline RL as an online-to-offline transfer learning problem, utilizing an auxiliary online simulator to address the challenges of overfitting in representation learning and value function overestimation. The method involves training separate source and target world models and actor-critic agents, aligning their latent spaces, and using a three-stage learning procedure to improve performance on the target task.\n\nSTRENGTHS:\n- The idea is novel: CoWorld presents an interesting perspective on offline RL by framing it as an online-to-offline transfer learning problem.\n- The problem is important: The method aims to tackle both overfitting in representation learning and the trade-off between value function overestimation and over-conservatism.\n- Ablation studies and Analysis: The paper includes comprehensive ablation studies that validate the contribution of each component in the CoWorld framework.\n\nWEAKNESSES:\n1. Dependence on auxiliary online environments: CoWorld heavily relies on the availability of an online environment in a sufficiently similar domain, which may limit its applicability in real-world scenarios.\n2. Unclear rationale for separate world models: The decision to maintain separate source and target world models, rather than a single jointly trained model, is not adequately explained or supported.\n3. Marginal performance improvement: The performance gains of CoWorld over simpler baselines (e.g., fine-tuning) are sometimes marginal and highly sensitive to hyperparameters.\n4. Computational complexity: The alternation between online and offline agents until convergence may be computationally expensive and potentially unstable.\n5. Strong assumptions: The requirement of a high-quality simulator for a similar source domain contradicts the motivation for offline RL, which assumes online data collection is costly or dangerous.\n6. Insufficient clarity in the presentation: The necessity and function of individual components—such as world model learning, state alignment, and reward alignment—are not clearly delineated, leading to potential confusion.\n\nQUESTIONS:\n1. How dependent is CoWorld's performance on the quality and similarity of the online simulator to the target task?\n2. Have you considered training on multiple source domains to improve robustness and generalization?\n3. How does the computational complexity of CoWorld compare to baseline methods in terms of training time and resource requirements?\n4. Can you provide more insight into the process of selecting an appropriate source domain, and how this might be done without leaking test-time information?\n5. How would CoWorld perform if the source and target domains have significantly different observation spaces?\n\nLIMITATIONS:\nYes, the authors have discussed the limitations in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies model-based offline reinforcement learning with visual observations. This work focuses on the overfitting problem in state representation and the overestimation bias for the value function and introduces the environment model from related tasks to present an offline-online-offline solution. The proposed solution contains state alignment, reward alignment, and value alignment. The empirical studies conduct the experiments with transfer across heterogeneous tasks.\n\nSTRENGTHS:\n1. The proposed setting is very interesting to me. The author provides a new perspective on reinforcement learning cross-task transfer from the angle of transfer learning, which is challenging.\n\n2. The proposed solution is reasonable. When online interaction in the current task is unavailable, leveraging online interaction from related tasks is similar to traditional transfer learning. Although the offline-online-offline process appears somewhat complex, it indeed provides an idea that may promote related research.\n\nWEAKNESSES:\n1. The provided results have significant standard deviations. I am not sure if the authors taking 10 episodes over 3 seeds provided a sufficient evaluation. This issue can be referred to: [Deep reinforcement learning at the edge of the statistical precipice. NeurIPS’21]. \n2. In fact, I highly appreciate the contributions made by the authors, if the statistical results of the experiment are reliable (see Weakness 1). This work successfully achieved transfer in multiple environments, which is rare for me, and makes a significant contribution. But where does the success of the transfer come from? The success of state alignment is relatively easy to understand, similar to traditional visual domain adaptation. But what does the success of reward alignment come from? From information like \"objects should be picked\" in the reward signal? This seems to impose relatively strict limitations on the relevance of the task, which is lacking in discussion in the current version. \n3. Similarly, what are the conditions for mitigating value overestimation? Figure 2 is well visualized, but it seems to implicitly assume that the source critic and the ideal target critic perform similarly on the broad support. If based on such an assumption, cross-environment transfer does not seem so difficult. I still affirm the contribution of this work, but can the authors provide a formal analysis of this and determine the applicable conditions for the proposed approach?\n\nQUESTIONS:\n1. How to ensure that source critic is beneficial to the target task, and is it possible to have a negative transfer when facing conflicting tasks?  \n2. See the weakness 3.\n\n--- \nAfter rebuttal, my concerns have been addressed well. I decide to raise my score to 6.\n\nLIMITATIONS:\nThe authors have provided a discussion about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies CoWorld, a model-based RL method that learns the policy through an offline-online-offline framework. Specifically, CoWorld assumes that, in addition to the offline dataset, there is another (simulated) environment for online interaction. Through the offline dataset and interactive environment, CoWorld constructed two world models to train the separate policies. Next, this paper introduces three components to enhance the policy performance, including offline-to-online state alignment, online-to-offline reward alignment, and min-max value constraint. The proposed CoWorld and several baselines are evaluated on Meta-World, RoboDesk, and DMC benchmarks. From the results presented, this paper suggests that CoWorld outperforms existing RL methods by large margins.\n\nSTRENGTHS:\n- This paper proposes an interesting method for training an RL policy by co-learning from both the offline dataset and the interactive environment.\n- This work presents a reasonable number of ablation studies and experiments to verify the proposed method. These experiments cover various directions, including environmental settings, component contributions, and hyperparameter sensitivity.\n- This paper is generally well-written and easy to follow. The depicted figures and tables are well-articulated.\n\nWEAKNESSES:\n- While I agree the idea of ​​CoWorld is interesting, I feel that it may not be easy to find suitable practical applications. After all, a low-cost, high-efficiency interactive environment is itself a strong assumption. It would be more convincing if the author could describe practical situations where the proposed method is applicable.\n- Related to the first point, CoWorld learns the separate world models from both the offline dataset and the online environment. I'm curious about how significant the difference between these two sources can be. The setting used in this paper is only a slight difference (4 vs. 5 in the action space, or scale in reward space). But when the two resources are significantly different, can CoWorld still work? For example, (1) one environment with dense reward and another one with sparse reward (not just scale), or (2) the observation and action spaces are obviously different from the two sources.\n- From Table 2, I am concerned about the claim that the proposed CoWorld significantly outperforms other baselines because (1) in several cases, the CoWorld's performance overlaps with other methods when considering its standard deviations, and (2) the results are obtained from only three random seeds. I encourage the authors to conduct more experimental runs and/or provide statistical analysis to eliminate this concern.\n- A minor point: while this paper is generally well-written, I found some typos that should be easily fixed:\n 1. The notation T represents both the target environment and the maximum number of time steps. (line 14 in Algorithm 1)\n 2. Target/Offline (**S**) $\\rightarrow$ should be T? (Table 4, page 13)\n 3. ... on DMC **Meidum**-Expert Dataset $\\rightarrow$ Medium (line 481, page 14)\n 4. ... as illustrated in **Table 1** $\\rightarrow$ should be Table 6? (line 516, page 17)\n\nBased on the observed strengths and weaknesses, I will set my initial recommendation as \"Borderline accept\".\n\nI will be happy to raise my rating if the authors appropriately address my concerns or point out my misunderstandings.\n\nQUESTIONS:\nPlease consider addressing my concerns and questions in Weaknesses; I will list some key ones:\n1. In what practical scenarios is the proposed CoWorld method applicable? Can you provide more examples and details?\n2. How does CoWorld operate when the two world model architectures have greater differences? Can it still effectively transfer information between these two resources?\n3. How to prove that CoWorld is really significantly better than other methods? Can you provide the results of more experimental runs or statistical analysis?\n\nLIMITATIONS:\nA major limitation is that the proposed CoWorld method is applicable to only a limited number of practical application scenarios. Other limitations are adequately discussed in the Limitation Section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This work proposes a model-based algorithm for offline visual reinforcement learning using domain transfer learning. The proposed method demonstrates superior performance over existing approaches in several baselines.

STRENGTHS:
S1. The paper presents the necessity of proposed modules in the algorithms through ablations and validation in multiple setups.

S2. The proposed method outperforms baselines in different benchmarks.

WEAKNESSES:
W1. The cost of training time is higher in comparison to baselines.

QUESTIONS:
Q1. How does the performance of DV2 Finetune, DV2 Finetune+EWC, and CoWorld look when the x-axis is training wall-clock time instead of the number of training iterations?

LIMITATIONS:
Yes, the authors have addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13