Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
Authors: Qi Wang
Junming Yang
Yunbo Wang
Xin Jin
Wenjun Zeng
Xiaokang Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: World models
reinforcement learning
visual control
Primary area: reinforcement_learning

Abstract: Training offline RL models using visual inputs poses two significant challenges, *i.e.*, the overfitting problem in representation learning and the overestimation bias for expected future rewards. Recent work has attempted to alleviate the overestimation bias by encouraging conservative behaviors. This paper, in contrast, tries to build more flexible constraints for value estimation without impeding the exploration of potential advantages. The key idea is to leverage off-the-shelf RL simulators, which can be easily interacted with in an online manner, as the “*test bed*” for offline policies. To enable effective online-to-offline knowledge transfer, we introduce CoWorld, a model-based RL approach that mitigates cross-domain discrepancies in state and reward spaces. Experimental results demonstrate the effectiveness of CoWorld, outperforming existing RL approaches by large margins.

Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
Qi Wang 4.2 Junming Yang Yunbo Wang't Xin Jin Wenjun Zeng² Xiaokang Yang'
MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University, China
2 Ningbo Institute of Digital Twin, Eastern Institute of Technology. China
School of Computer Science and Engineering, Southeast University, China
https://qiwangOG6 bripgg0067gggtttth github. io/coworld
Abstract
Training offline RL models using visual inputs poses two significant challenges, i.e. the overfitting problem in representation learning and the overestimation bias for expected future rewards. Recent work has attempted to alleviate the overestimation bias by encouraging conservative behaviors. This paper, contrast, tries to build more flexible constraints for value estimation without impeding the exploration potential advantages. The key idea is leverage off-the-shelf RL simulators, which can be easily interacted with in an online manner, as the "test bed" for offline policies. To enable effective online-to-offline knowledge transfer, we introduce CoWorld, model-basee RL approach that mitigates cross-domain discrepancies in state and reward spaces. Experimental results demonstrate the effectiveness of CoWorld, outperforming existing RL approaches by large margins.
Introduction
Learning control policies with visual observations be challenging due high interaction costs with the physical world. Offline reinforcement learning (RL) is a promising approach to address this challenge [1] 21 37 65). However, the direct of current offline RL algorithms in visual control tasks presents two primary difficulties. Initially, offline visual RL is more prone to overfitting issues during representation learning, as involves extracting hidden states from the limited, high- dimensional visual inputs. Moreover, like its state-space counterpart, offline visual RL is susceptible the challenge of value overestimation, as we observe from existing methods
Improving offline visual RL remains an under- explored research area. We aim balance be- tween overestimating and over-conservatism of the value function avoid excessively pe- nalizing the estimated values beyond the of- fline data distribution. Intuitively, we should not overly constrain the exploration with po- tential advantages. Our basic idea, as illus- trated in Figure] [ is to leverage readily avail- able online simulators for related (not neces- sarily identical) visual control tasks as aux- iliary source domains, so that we can frame offline visual RL as an coffine-online-oflie transfer learning problem to learn mildly con- servative policies.
Interactions
Value
constraint
Online
Source world model
Source agent
environment
State exploration
State alignment
Reward Valggmmet
Offline dataset
Target world model
Target agont
Figure 1: Our approach for offline visual RL.
"Equal contribution.
Cooresponding author: Yunbo Wang Stunbow@sstu..du.ccc
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Value
Value
Value
Value
Overestimation
Target true value
Target-informed source critic
Mild-conservatismm by CoWorld
Over-conservatism
In-domain
OOD data
In-domain
OO0 data
In-domain
OOD data
In-domain
0OD data
(n)
(c)
(d)
Figure 2: To address value overestimation in offline RL (a), we can directly penalize the estimated
values beyond the distribution of offline data, which may hinder the agent's exploration of potential states with high rewards (b). Unlike existing methods, CoWorld trains a cross-domain critic model in an online auxiliary domain to reassess the offline policy (c), and regularizes the target values with flexible constraints (d). The feasibility of this approach lies in the domain alignment techniques during the world model learning stage.
We present novel model-based transfer RL approach called Collaborative World Models (CoWorld), Specifically, we train separate world models and RL agents for source and target domains, each with domain- -omain--pecific parameters. To mitigate discrepancies between the world models, we introduce novel representation learning scheme comprising two iterative training stages. These stages, as shown in Figure II facilitate the alignment of latent state distributions (offline online) and reward functions (online to offline), respectively. By doing so, the source domain critic can serve as an online "test bed" for assessing the target offline policy. It is also more "knowledgeable" with the online environment and gather rich information. Another benefit of the camain-colllaboaan world models is the ability alleviate overfitting issues of offline representation learning, leading to more generalizable latent states derived from limited offline visual data.
it can actively interact
For behavior learning in the offline dataset, we exploit the knowledge from the source model and introduce a mild regularization term to the training objective of the target domain critic model. This regularization term encourages the source critic reevaluate the target policy. As illustrated in Figure 2 allows for ficxible constraint on overestimated values trajectories that receive low values from the "knowledgeable" source critic. Conversely, if policy yields high values from the source critic, we prefer to retain the original estimation by the offline agent. This approach is feasible because the source critic has been aligned with the target domain during world model learning.
We showcase the effectiveness of CoWorld offline visual control tasks across the Meta-Worlf RoboDesk, and DeepMind Control benchmarks. Our approach is shown to be readily extendable to scenarios with multiple source domains. effectively addresses value overestimation by transferring knowledge from auxiliary domains, even in the presence of diverse physical dynamics, action spaces, reward scales, and visual appearances. In summary, our work brings the following contributions:
We innovatively frame offline visual RL as domain transfer problem. The fundamental idea is to harness cross-domain knowledge to tackle representation overfitting and value overestimation in offline visual control tasks.
We present Corrrdd, method that follows the offline-online-ofline paradigm, incorporating specific techniques of world model alignment and flexible value constraints.
2 Problem Setup
We consider offline visual reinforcement learning as a partially observable Markov decision process (POMDP) that aims to maximize the cumulative reward in a fixed target dataset B(T), We specif- ically focus on scenarios where auxiliary environments are accessible, enabling rich interactions and efficient online data collection. The goal is to improve the offline performance of the target POMDP (O(T), (T), (T), R(T), (TT) through knowledge transfer from the source POMDPs ((S), (()) ((5). These notations respectively denote the space of visual observations, the space actions, the state transition probabilities, the reward function, and the discount factor.
For example, in one of our experiments, we employ RoboDesk as the offline target domain and various tasks from Meta- World as the source domains. As illustrated Table these two environments present notable distinctions in physical dynamics, action spaces, reward definitions, and visual

---

Table 1: RoboDesk (target domain) vs. Meta- Woold (auxiliaryy source domain).
Source: Meta- World
Target: RoboDesk
Similarity Difference
Task
Window Close
Open Slide
Related manipulation tasks Different
Dynamics Action space
Simulated Sawyer robot Simulated Franka rohot Box(-1, 1, (5,), float32)
Box(-1, 1, (4.), float64)
Different
Reward scale
[0, I]
[0, 10] Top-view images
Different
Observation
Right-view images
Different view points
appearances as the observed images are from different camera views. Our priority is to address domain discrepancies enable cross-domain behavior learning.
3 Method
In this section, we present the technical details of CoWorld, which consists of pair of world models (My, actor networks {Ter, Joh. and critic networks (Ugr, Ug], where {ø, v. { and {Ø', " )'} are respectively target and source domain parameters. As potential cross-domain discrepancies may exist all elements of {O, T, RJ, the entire training process is organized into three iterative stages, following an offline-ooninneofflinng transfer learning framework:
A) Offline-to-onliee state alignment: Train the offline world model M& by aligning its state space with that the source world model Ma'
B) Online-to-ofliine reward alignment: Train Ma and (mo,, Wer in the online environment by incorporating the target reward information.
C) Online-to-offinee value constraint: Train the target dofmine-domain agent { 0g) with value constraints provided by the source critic Ee'.
3.1 Offine-to-Online State Alignment
Source model pretraining. We start with source domain warm-up phase employing model- based actor-critic method known as DreamerV2 [16]. To facilitate cross-domain knowledge transfer, additionally introduce state alignment module, which is denoted as g(-) and implemented using the softmax operation. The world model Ma consists of the following components:
Recurrent transition:
h = for (N,
(8) a11) Image encoding:
(5) = (0(
Posterior state:
+ (8) ~ (1b(0) (
Prior state:
x p (1))))
(1)
Reconstruction:
0(5) par (11 (11,9) (3))
Reward prediction:
215) ror (1(3) (5))
Discount factor:
(1 ~ pa (11 (8))
State alignment target:
where ø' represents the combined parameters of the world model. We train My on the dynamically expanded source domain experience replay buffer (5() by minimizing
C(o') Egge
lop (5) 11 ( (5)) In Ir (r! 1
(8))
Iape
(9)
2t
1
image
prodiction
discount prediction
(2)
+ KL
4o (2,
(1) (9) (2 14 [
KL divergence
We train the source actor Th (2x) and critic (ge(()) with the respective objectives of maximizing and estimating the expected future rewards Epgr 1Er2e fr- generated by Ma Please refer to Appendix A.3 for more details. We deploy T&y interact with the auxiliary environment and collect new data for further world model training.
straightforward transfer learning solution to train the target agent in offline dataset upon the checkpoints of the source agent. However, it may suffer from potential mismatch issue due to the discrepancy in tasks, visual observations, physical dynamics, and action spaces across various domains. This becomes more severe when the online data is collected from
State alignment.

---

Algorithm The training scheme of CoWorld.
1: Require: Offline dataset B(+)
2: Initialize: Parameters of the source model {ø', v/, 5'' } and the target model {Ø, v E)-
3: Pretrain the source agent and collect replay buffer 18())
4: while not converged do
5: for each step in {1 K1 do
In the offiine domain
6: Sample {(o (m) (?T) rr. ) ~ (B
7:
Train the target world model M₁ using Eq. +
state alignment
8:
Generate {(=]') .a ) using my and Ma-
Behavior learning with constraint
9:
Train the critic ve using Eq. (6 over [(2 dz Tes
(0)
10:
Train the actor no using Eq. over ((.))) 5a Fam end for
12:
for each step in {1 : K2} do
D In the online domain
13:
re Sample ((o) (2) (2)
Sample {(0, (8) a (8)
1 B(S).
14:
de
Mi-1 ~ B
> Online-to-oftline reward alignment
Relabel the source rewards (f/5) using Eq. 44.
15:
16:
Train My using Eq.
( combined with Eq. (5)
17:
Generate {(=] aa.")
(8) (1) Tim= using and Mar.
Source domain behavior learning
18:
Train Ter and eyr over the imagined [(=( a. (8) get Use Tay to collect new source data and append (S))
19:
20:
end for end while
21:
environments that differ from the offline dataset (e.g., Meta- World - RoboDesk). We tackle this issue by separating the parameters of the source and the target agents while explicitly aligning their latent state spaces. Concretely, the target world model Ma has an identical network architecture to the source model Mar. We feed the same target domain observations sampled from (T() into these models and close the distance of car(o and (oT))) We optimize Ma by minimizing
(()) = Eqo
In 166 (((T)) In (T (T))
Impo(te (T (T))
image reccnsttrncee ucciion
rewnrd prodiction
discount prediction
(3)
+ 81 (z)
(T) I of (l (T()) + 32KL
(o(ole (T) oo(es(!"))
KT. divereeee
domain aligoment loss
where sg(:) indicates gradient stopping and use the encoding from the source model the state alignment target. As the source world model can actively interact with the online environment and gather rich information, it keeps the target world model from overfitting the offline data. The importance of this loss term is governed by 32. We examine its sensitivity in the experiments.
3.2 Online-to-Offline Reward Alignment
To enable the source agent to value the target policy, it is essential to provide it with prior knowledge of the offline task. To achieve this, we train the source reward predictor Tyr using mixed data from both of the replay buffers BB()) and B(T). Through the behavior learning on source domain imaginations, the target-informed reward predictor enables the source RL agent assess the imagined states produced by the target model and provide flexible constraint to target value estimation (as we will discuss in Section 3.3).
Specifically, we first sample target domain data trajectory {(o . (T) D from B(T) (Line 14 in Alg. [.. We then use the source world model parametrized by of to extract corresponding latent states and relabel the target-informed source reward (Line 15 in Alg. I
h (he-1, 1
ē = 60 (2)
(4)
&, ~ yo (Fe,et ē.)
p(() = (1 -1-k) k) ray (he, &e) + k (T))
where & is the target-informed reward factor, which acts as a balance between the true target reward (T)) and the output of the source reward predictor For (-) provided with target states. It is crucial to

---

emphasize that using the target data as inputs to compute (gr()) is feasible due to the alignment of the target state space with the source state space.
We jointly use the relabeled reward F/ and the original source domain reward r sampled from B(S) to train the source reward predictor. This training is achieved by minimizing a maximum likelihood estimation (MLE) loss:
(o(d') 17 (g))
Σ In In (M 101)] + Egtr)
Σ
(f! (
}. (5)
where the second term measures the negative log-likelihood of observing the relabelled source reward ((8) represents a hyperparameter that gradually decreases from 1 0.1 throughout this training stage. Intuitively, 77 controls the progressive adaptation of the well-trained source reward predictor to the target domain with limited target reward supervision. We integrate Eq. (5) into Eq. (2) to train the entire world model Mar for the source domain agent (Line 16 in Alg. (1) and subsequently perform behavior learning to enable the source critic to assess the target policy (Lines 17-19 in Alg. I
3.3 Min-Max Value Constraint
In the behavior learning phase of the target agent (Lines 8-10 Alg. D. we mitigate value overestimation in the offline dataset by introducing min-max regularization term to the objective function the target critic model Vi.. Initially, we use the auxiliary source critic Ve to estimate the value function of the imagined target states. Following that, train ve by additionally minimizing the maximun value among the estimates provided by source and target crities:
((E) P
Σ
(2((2 ag =g((;"T))* max (ve(El (E(
(6)
value regression
value constraint
where Vi(T) incorporates weighted average of reward information over an n-step future horizon. The first term in the provided loss function fits cumulative value estimates (whose specific formulation can be located in Appendix A.3). while the second term regularizes the overestimated values for out- of-distribution data mildly conservative way. The hyperparameter represents the importance of the value constraint. The sg(-) operator indicates that we stop the gradient to keep the source critic from being influenced by the regularization term.
This approach provides flexibly conservative value estimations, finding a balance between mitigating overestimation and avoiding excessive conservatism in the value function. When the target critic overestimates the value function, the source critic is less vulnerable to the value overestimation problem as it is trained with rich interaction data. Thus, it is possible to observe vg (2(")) > ver (z(*)), and our approach is designed to decrease the output dg to the output of vgr. This prevents the target critic from overestimating the true value. Conversely, when the source critic produces greater values
in ugr (2, (2(T)) ). the min-mat max regularization term does not contribute the training of the target critic .g.. This encourages the exploration of potentially advantageous states within the imaginations of the target world model. In line with DreamerV2 [16), we train the target actor Fe by maximizing REINFORCE objective function with entropy regularization, allowing the gradients to backpropagate directly through the learned dynamics:
[(()) Eps
(HH[a
+ -
In n. (a. 8 ((T) (s((,
(2(2)))
entropy regolarization
dynamics backprop
REINFORCE
(7)
As previously mentioned, Vy(T) involves a weighted average of reward information over an n-step future horizon, with detailed formulation provided in Appendix A.3
Furthermore, crucial to note that CoWorld can readily be extended to scenarios with multiple source domains by adaptively selecting useful task as the auxiliary domain. This extension is easily achieved by measuring the distance of the latent states between the target domain and each source domain. For technical details of the adaptive source domain selection, please refer to Appendix C
5

---

Table Mean episode returns and standard deviations of 10 episodes over 3 seeds on World.
Model
BP- DC
DC + BP
BT WC
BP-+ HP
WC- DC
HP-+ BT
Avg.
Offline DV2
2143+579
3142+-533
3921+752
278+128
3899+6679
3002-:346
2730
DrQ + BC
567+11
587+668
623+
1203++234
134+66
642-999
626
CQL.
1984+13
867+:330
6834-268
988+:39
577+121
462-+67
927
CURL
1972:111
51:17
2811733
9861147
3661:52
189110
641
LOMPO
2883-183
4461:458
2983:1569
2230:1:233
2756:1331
1961:1287
1712
DV2 Finetune
3500
2456.1661
346711101
3702-1451
4273
3499
3781
DV2 Finetune + EWC
15661123
167
978.1772
528.1334
2048.1
224-1147
918
LOMPO Finetune
2591191
95:153
142.170
332.1452
3698.11615
224.888
792
CoWorld (Best-Source)
3967
3623:1543
4521-1367
477
48
38891115
4241
CoWorld (Multi-Sourcee
3864-352
35734:541
4507-159
4460-1783
137
3626:1275
4094
Experiments
4.1 Experimental Setups
Datasets. We evaluate CoWorld across three visual control environments, i.e., Meta- World [54). RoboDesk [18], and DeepMind Control Suite (DMC) (77. including both cross-task and cross- environment setups (Meta-World
RoboDesk). Inspired by D4RL. [9], we build offline datasets of medium-replay quality using DreamerV2 [16]. The datasets comprise all the samples in the replay buffer collected during the training process until the policy attains medium-level performance, defined as achieving 1/3 of the maximum score that the DreamerV2 agent can achieve. Please refer Appendix B.2 for further results of CoWorld trained with medium-exper offline data.
Compared methods. We compare CoWorld with both model-based and model-free RL approaches, including Offline DV2 [25). DrQ+BC (25 CQL 225 CURL [22], and LOMPO [3) In addi-
[16] model pretrained in the online source domain and subsequently finetuning it in the offline target dataset. Furthermore, DV2 Finetune can be integrated with the continual learning method, Elastic Weight Consolidation (EWC) [99, to regularize the model for preserving source domain knowledge, i.e. Finetune+EWC. Please refer to Appendix for more details.
tion, we introduce the DV2 Finetune method, which involves taking a Dreamer V2
4.2 Cross-Task Experiments on World
Meta-World is open-source simulated benchmark designed for solving wide range of robot manipulation tasks. We select tasks as either the offline dataset or potential candidates for the online auxiliary domain. These tasks include: Door Close (DC*), Button Press (BP), Window Close (WC), Handle Press (HP), Drawer Close (DC), Button Topdown (BT).
Main results. As shown in Table 2 we compare the results of CoWorld with other models on Meta- World. CoWorld achieves the best performance in all tasks. Notably, outperforms Offline DV2 255 method also built upon DreamerV2 and specifically designed for offline visual RI.. For the tolinc-to-offline finetuning models, DV2 Finetune achieves the second-best results by leveraging transferred knowledge from the auxiliary source domain. However, observe that its performance experiences a notable decline scenarios (e.g., Meta-World - RoboDesk) involving significant data distribution shifts between the source and the target domains in visual observation, physical dynamics, reward definition, or even the action space of the robots. Another important baseline model is DV2 Finetune+ which focuses mitigating the catastrophic forgetting of the knowledge obtained in source domain pretraining. Nevertheless, without additional model designs for domain adaptation, retaining source domain Enowledge may eventually lead decrease in performance in the target domain. The LOMPO model suffers from the negative transfer effect when incorporating a source pretraining stage. It achieves average return of 1,712 when it is trained from scratch in the offline domain while achieving an average return of 792 for saline-to-offlnee finetuning. It implies that a naïve transfer learning method may degenerate the target performance due unexpected bias.
Results with a random source domain. Given that we present the best-source results in Table where we manually select source task from Meta-World, one may cast doubt on the influence of

---

Source Task
Source Task
CoWorld /
DC
BP
WC
HP
DC
BT
Source
Offine DV2
DC
HP
BP
WC
BT
DC
5000
1.85
1.81
1.40
1.81
1.61
1.80
1.4
0.94
1.03
1.14
1.15
1.03
1.14
1.3
Episode Return
Target Task
HP WC
1.15
1.11
1.13
1.11
1:15
1.15
3500
1.1
13.52
16.44
14.29
03.5833.100
16.04
3000
1.0
A
Offiine DV2
1.24
0.96
1.20
0.9
DV2 Finetune
CoWorld (Best-Source)
2000
BT
CoWorld (Multi- Source)
0.7
1500
Figure 3: Left: The value each grid indicates the ratio of returns achieved by CoWorld compared to Offline DV2. Highlighted grids represent the top-performing source domain. Right: Returns on Drawer Close (DC*) with different source domains, where the multi-source CoWorld (yellow line) is shown to automatically discover (i.e., Door Close) the source domain and achieve comparable results with the popperrfrnning single-source CoWorld (red line).
Offiine DV2
DV2 Finetune
CoWorld [Best Source)
CoWorid (Multi-Source) Source)
A20
X20
150
340
1:0
100
Epiodde
170
280
570
120
330
10 15
20
25
10
36
20
25
5
30
13
20
Number of Iteratioss |x10
Number of Iterations 1K10"
Number Ieerations 1x110*)
Number Ioraations (x110")
Dutton Press
Button
Wildow Close
Open Stide
Close Drawer dpps (d) Mandlo Ujriihh Block Tuble
Figure 4: Quantitative results in domain transfer scenarios of Meta-World + RoboDesk.
domain discrepancies between the auxiliary environment and the target offline dataset. In Figure (Left), the transfer matrix of CoWorld among the tasks of Meta-World is presented, where values greater than indicate positive domain transfer effects. Notably, there are challenging cases with weakly related source and target tasks. In the majority of cases (26 out of 30), CoWorld outperforms Offtine DV2, illustrated in the heatmap.
Results with multiple source domains. is crucial note that CoWorld can be easily extended to scenarios with multiple source domains by adaptively selecting useful task as the auxiliary domain. From Table we can see that the multi-source CoWorld achieves comparable results to the models trained with manually designated online simulators. In Figure 3 (Left), multi- CoWorld achieves positive improvements over Offline DV2 in all cases, approaching the best results of models using each source task as the auxiliary domain. In Figure ( (Right), it also consistently outperforms the DV2 Finetune baseline model. These results demonstrate our approach's 's ability to execute without strict assumptions about domain similarity and its ability to automatically identify useful online simulator from a set of both related and less related source domains.
4.3 Frrss-Envirooments:: Meta- World to RoboDesk
To explore cross-environmen transfer with more significant domain gaps, we employ four tasks from RoboDesk to construct individual offline datasets, i.e., Push Button, Open Slide, Drawer Open, Upright Block off Table. These tasks require handling randomly positioned objects with image inputs. Table I
presents the differences between the two environments in physical dynamics, action space, reward definitions, and visual appearances.
presents quantitative comparisons, where CoWorld outperforms Offline DV2 and DV2 Finetune by large margins. For the best-source experiments, we manually select one source domain from Meta-War World. For the mulli-source experiments, we jointly use Meta-World tasks as the source domains. In contrast to prior findings, directly finetuning the source world model in this cross-enviromment setup, where there are more pronounced domain discrepancies, does not result in significant improvements in the final performance. In comparison, CoWorld more successfully
Figure 4
7

---

400
Offiine
CoWorld w/o Max
CoWorld
700
pblsode Return
375
350
500
325
300
300
CoWorld
w/o Stage
CoWorid Stage
250
CoWorld w/o Stage
-100
CoWorld w/o Max
-300
200
Handle
Window
Button
Drawer
Number Iterations (x10*)
Close
Press
Press
Close
Topdown
Close
(a) Meta-World -> RoboDesk (Push Green Button)
(b) Y-axis Estimated value True value
Figure 5: (a) Ablation studies on state alignment, reward alignment, and min-max value constraint,
(b) The disparities between the estimated value by various models and the true value. Please see the text in Section 4.5 for the implementation of CoWorld w/o Max.
Table 3: Mean rewards and standard deviations of 10 episodes in offline DMC over seeds.
Model
wW + WD wW - WU
WW - WN CR + CD
CR - CU
CR + CN
Avg.
Offline DV2
435:-22
13914
21414
243-77
3+1
5114
181
DrQ+BC
291 10
299+11
318+:40
663+15
202+
132+:33
355
CQL
46+19
64=32
29+2
2+11
52 +57
111+157
51
CURL
43+5
21+3
23:13
26+77
4+2
11+4
LOMPO
462.187
260.121
460.19
395:152
46.119
120-4
DV2 Finetune
379+23
354::29
407=337
702:+41
208
454
417
LOMPO Finetune
209+21
141+27
212419
142:229
17+11
105:112
137
CoWorld
629.19
407-111
426.132
745
225:1:00
493.110
488
addresses these challenges by leveraging domain-specific world models and RL agents, and explicitly aligning the state and reward spaces across domains. We also showcase the performance of multi- source CoWorld, which achieves comparable results to the best-source model that exclusively uses our designated source domain.
4.4 Cross-Dynamics Experiments on DMC
DMC is widely explored benchmark for continuous control. We use the Walker and Cheetah the base agents and make modifications to the environment to create a set distinct tasks, i.e. Walker Walk (WW), Walker Downhill (WD), Walker Uphill (WU), Walker Nofoot (WN), Cheetah Run (CR), Cheetah Downhill (CD), Cheetah Uphill (CU), Cheetah Nopaw (CN). Particularly, Walker Nofoot task in which cannot control the right foot of the Walker agent. Cheetah Nopaw is task in which we cannot control the front paw of the Cheetah agent.
We apply the proposed multi-source domain selection method to build the domain transfer settings shown in Table is worth noting that CoWorld outperforms the other compared models out of 6 DMC offline datasets, and achieves the second-best performance in the remaining task. On average, outperforms Offline DV2 by 169.6% and outperforms DrQ+BC by 37.5%. Corresponding qualitative comparisons can be found in Appendix B.I
4.5 Further Analyses
Ablation studies. We conduct series of ablation studies validate the effectiveness of state space alignment (Stage A), reward alignment (Stage B), and min-max value constraint (Stage C). We show corresponding results the offline Push Green Button dataset from RoboDesk in Figure 5(a) The performance experiences a significant decline when we abandon each training stage in CoWordd.
Can CoWorld address value overestimation? We evaluate the values estimated by the critic network CoWld on the offline Meta-Worke World datasets when the training process is finished. In Figure 5(b) compute the cumulative value predictions throughout 500 steps. The true value

---

Table 4: Experiments with significantly distinct observation spaces across domains. We use low- dimensional state data as inputs for the RL agents in the source domain and high-dimensional image observations in the target domain. MW represents Meta-Woo World and RD stands for RoboDesk.
Method
MW: Button Press > RD: Push Button
MW: Window Close + RD: Open Slide
Offline DV2 CoWorid
347 = 24
156 + 46
393 64
209 + 43
Table 5: Experiments with significantly distinct reward formations across domains. We use sparse rewards in the source domain while maintaining the dense rewards in the target domain.
Method
MW: Button Press - RD: Push Button
MW: Window Close + RD: Open Slide
DV2 Finetune CoWorld
314 + 51
173 + 39
335 + 28
184 + 32
is determined by calculating the discounted sum of the actual rewards obtained by the actor in the same 500-steps period. We observe that existing approaches, including Offline DV2 and CQI. often overestimate the value functions in the offline setup. The baseline model "CoWorld w/o Max" is variant of CoWorld that incorporates a brute-force constraint on the critic loss. It reformulates Eq. (6) as Σ } (1ee (&+) - =g(V)))² + crug (&x). As observed, this model tends to underestimate the true value function, which can potentially result in overly conservative policies a consequence. In contrast, the values estimated by CoWorld are notably more accurate and more akin to the true values.
Dependence of Corlr to domain similarities. We further investigate the dependence of CoWorld on domain similarity from the perspectives of different observation spaces and reward spaces. We first explore how CoWorld performs when we only have source domains with significantly distinct observation spaces from the target domain. As illustrated in Table 4 the agent receives low-dimensional state inputs in the source domain (Meta- World) and high-dimensional images in the target domain (RoboDesk). We can see that CoWorld outperforms Offline DV2 by 13.3% and 34.0% due to the ability to leverage low-dimensional source data effectively. Notably, the finetuning method (DV2 Finetune) is not applicable in this scenario. In Table we also observe that CoWorld benefits from a source domain, even with significantly different reward signal. Unlike previous experiments, we use a sparse reward function for the source Meta- World tasks. It is set to 500 only upon task completion and remains O before that. The experimental results demonstrate that although excessively sparse rewards can hinder the training process, CoWorld still achieves an average performance gain of 6.6% compared to DV2 Finelune under the same setting.
Comparison to jointly training one world model across domains. Notably, CoWorld is imple- mented with separate world models for the source and target domains. Alternatively, we can employ a jointly trained world model across various domains for more efficient memory usage. In Table we compare the results from the original CoWorld and "Multi-Task DV2". Multi-Task DV2 involves training DreamerV2 on both offline and online data with a joint world model and separate actor-critic models. CoWorld consistently performs better. Intuitively, using separate world models allows the source and target domains to have different physical dynamics, observation spaces, or reward formations, as the scenarios shown in Table 4 and Table
Table 6: Comparison to jointly training one world model across domains (Multi-Task DV2).
Method
MW: Button Press + RD: Push Button
MW: Window Close - RD: Open Slide
Multi-Task DV2 CoWorld
342 + 29
173 +: 22
428 + 42
202 + 19
Hyperparameter sensitivity. We conduct sensitivity analyses on Meta- Bera- (DC + BP). From 6 we observe that when 3, for the domain KL loss is too small, the state alignment between the source and target encoders becomes insufficient, hampering the transfer learning process. Conversely, if 33 is too large, the target encoder becomes excessively influenced by the source encoder, resulting decline in performance. We also find that the target-informed reward factor k plays crucial role in balancing the influence of source data and target reward information, which achieves a consistent

---

4000
4000
Episodo Return
8500
5750
3800
3600
Episode Return
3800
3250
3400
$400
8000
8
3200
5200
2750
3000
2500
LI 2600
2800
2800
2250
2400
2400
2200
0.6
Domain KL loss scale Bg
Target- ingettinclliee reward factor k
Target critic value loss scale a
Figure 6: Sensitivity analysis of the hyperparameters on Meta- Woold (DC + BP).
improvement over DV2 Finetune (2456 + 661) in the range of [0.1, 0.7]. Moreover, discover that the hyperparameter a for the target value constraint performs well within [1, 3], while an excessively larger may result in value over-conservatism in the target critic.
5 Related Work
Learning control policies from images is critical in real-world applications. Existing approaches can be grouped by the use of model-free [22]
48 36) or model-based [15]
14
16
43
35 13
28
29
61
51 RL algorithms. In offline RL, agents leverage pre-collected offline data to optimize
policies and encounter challenges associated with value overestimation 23 Previous methods mainly
suggest taking actions that were previously present in the offline dataset or learning conservative value estimations
53 40. Recent approaches have introduced specific techniques to address the challenges associated with offline visual RL (27 7 [23 33 [52 43 57 [5]. Rafailov et al. 39 proposed handle high nigh-dimensional observations with latent dynamics models and uncertainty
11 21 55
quantification. Cho ef al. [5] proposed synthesizing the raw observation data to append the training buffer, aiming to mitigate the issue of overfitting. In a related study, Lu et al. (25) established competitive offline visual RL model based on DreamerV2 [16], so that we use it as a significant baseline of our approach.
Our work is also related to transfer RL, which is known as to utilize the knowledge learned in past tasks
facilitate learning in unseen tasks (64 42 58
45 59
46
20
38
33 Most existing
approaches related to offline dataset + simulator focus on the offline- fefinee-o-oniie tonoinne setup, where the
policy is initially pretrained on the offline dataset and then finetuned and deployed on an interactive environment [33 60 56 63). These methods aim to bridge the gap between offline and online learning and facilitate fast adaptation of the model to the online environment. contrast, we explore the mnline-to-offlin setup, which provides a new remedy for the value over-estimation problem. hybrid affline-and-onling framework to integrate offline datasets and online simulators for policy optimization. Unlike CoWorld, this method primarily focuses on low-dimensional MDPs and cannot be directly used visual control tasks. the context of visual RL, CtrlFormer [31 learns transferable state representation via sample-efficient vision Transformer. APV [43] executes action-free world model pretraining on source-domain videos and finetunes the model on downstream tasks. Choreographer (28) builds model- based agent that exploits its world model to learn and adapt skills in imaginations, the learned skills are adapted to new domains using a meta-controller. VIP [26] presents self-supervised, goal-conditioned value-function objective, which enables the use of unlabeled video data for model pertaining. Unlike previous methods, we handle offline visual RL using auxiliary simulators, mitigating the value overestimation issues with co-trained world models.
Additionally, Niu et al. [34] introduces a dynamics-
Conclusions and Limitations
In this paper, we proposed transfer RL method named CoWorld, which mainly tackles the difficulty in representation learning and value estimation in offline visual RL. The key idea is to exploit accessible online environments to train an auxiliary RL agent to offer additional value assessment. To address the domain discrepancies and to improve the offline policy, we present specific technical contributions of cross-domain state alignment, reward alignment, and min-max value constraint. CoWorld demonstrates competitive results across three RL benchmarks. An unsolved problem of CoWorld is the increased computational complexity associated with the training phase in auxiliary domains (see Appendix B.7). It valuable to improve the training efficiency in future research.
10

---

Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant No. 62250062, 62106144, 62302246), the Shanghai Municipal Science and Technology Major Project (No. 2021SHZDZZ01102) the Fundamental Research Funds for the Central Universities, and the CCF- Tencent Rhino-Bird Open Research Fund. This work was also supported by the Natural Science Foundation of Zhejiang Province, China (No. LQ23F01000S) the High-PPeformance Computing Center at Eastern Institute of Technology (Ningbo), and the Ningbo Institute of Digital Twin.
References
Brian M Adams, Harvey Banks, Hee-Dae Kwon, and Hien T Tran. Dynamic multidrug therapies for hiv: Optimal and control approaches. Mathematical Biosciences Engineering, 122):223-24, 2004.
[2] Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In ICMI, pages 104-114, 2020.
[3] Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In ICLR, 2023.
[4] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li. Liang Bin, Chelsea Finn, and Chongjic Zhang. Lapo: vatent-variable weighte policy optimization for offline reinforcement learning. In NeurIPS, volume 35, pages 36902-36913, 2022.
[5] Daesol Cho, Dongseok Shim, and H Jin Kim. S2p: State-conditioned image synthesis for data augmentation in offline reinforcement learning. In NeurlPS, 2022.
[6] Djé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network lcarning by exponential linear units (clus), arXiv preprint 11X.22151,.07289, 2015.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In CoRL. 2019.
Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In ICLR, 2021.
[9] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint 2rXiv:2004.07219, 2020.
[10] Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement Icarning. In NewrIPS, 2021.
[11] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. In ICML, pages
2019.
[12] Dibya Ghosh, Chethan Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. arXiv preprint 2.X.:22304.04882 2023.
[13] Damijar Hafner, Kuang-Huei 1 Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. arXiv preprint S.Xiv:2206.041222 2022. 2022.
[14] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. /CLR, 2020.
[15] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In ICML, pages 2555-2565, 2019.
[16] Danijar Hatner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In /CLR, 2021.
[17] William Hua, Hongyuan Mei, Sarah Zohar, Magali Giral, and Yanxun Xu. Personalized dynamic treatment regimes in continuous time: a bayesian approach for optimizing clinical decisions with timing. Bayesian Analysis, 17(3):449-878 2022.
[18] Harini Kannan, Danijar Hafner, Chelsea Finn, and Dumitru Erhan. Robodesk: A multi-task reinforcement learning benchmark. htttps://giiaiii wtw.://ww..ooooooooeeeeeeerr///e
2021.

---

[19] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barriiska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351133556, 2017.
[20] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. In ICLR, 2023.
[21] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurlPS, volume 33, pages 11 117911911 2020.
[22] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. ICML, pages 5639-5650, 2020.
[23] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 2020.
[24] Xin Liu, Yaran Chen, Haoran Li, Boyu Li, and Dongbin Zhao. Cross-domain random pre-training with prototypes for reinforcement learning. arXiv preprint arXiv:2302.06614 2023.
[25] Cong Lu, Philip Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Oshome,, and Yee whye Teh. Challenges and opportunities offline reinforcement learning from visual observations. Transactions on Machine Learning Research, 2023.
[26] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In ICLR, 2023.
[27] Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In IROS, pages 1048-1055, 2019.
[28] Pietro Mazzaglia, Tim Verhelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreographer: Learning and adapting skills in imagination. In ICLR, 2023.
[29] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. In ICLR, 2023.
[30] Zhiyu Mou, Yusen Huo, Rongquan Bai, Mingzhou Xie, Chuan Yu, Jian Xu, and Bo Zheng. Sustainable online reinforcement lcarning for auto-bidding. In NeurlPS, volume 35, pages 2651-2663, 2022.
[31] Yao Mark Mu, Shoufa Chen, Mingyu Ding, Jianyu Chen, Runjian Chen, and Ping Luo. Ctrlformer: Learning transferable state representation for visual control via transformer. In ICML. pages 16043-16061, 2022.
[32] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
[33] Mitsuhiko Nakamoto. Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma. Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-qi: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint arXiv:2303.05479, 2023.
[34] Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, Xianyuan Zhan, al. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. In NewrlPS, volume 35, pages 36599-36612,, 2022.
[35] Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models. In NeurIPS, volume 35, pages 23178-23199 2022.
[36] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In ICMI,, pages 17359-17371, 2022.
[37] Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven offfine decision decision-making via invariant representation learning. In NeurIPS, 2022.
[38] Rafael Rafailov, Kyle Beltran Hatch, Victor Kolev, John D Martin, Mariano Phielipp, and Chelsea Finn. MOTO: Offline pre-training to online finc-tuning for model-based robot learning. In 7th Anmual Conference on Robot Learning, 2023.
[39] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforvement learning from images with latent space models. In Proceedings of Machine Learning Research, pages 1154- 1168, 2021.
12

---

[40] Mare Rigter. Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. In NeurlPS, 2022.
[41] Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R Devon Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. In NeurIPS, volume 34, pages 12699, 2021.
[42] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. ICML, pages 8583-8592, 2020.
[43] Younggyo Sen, Kimin Lee, Stephen I. James, and Picter Abbeel. Reinforcement learning with action-free pre-training from videos. In ICML, pages 19561-19579, 2022.
[44] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In ICML, pages 19700-8799 2021.
[45] Yanchao Sun, Xiangyu Yin, and Furong Huang. Temple: Learning template transitions for sample efficient multi-task rl. In AAAI, volume 35, pages 9765-9773, 2021.
[46] Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew Cohen, and Furong Huang. Transfer rl across observation feature spaces via model-based regularization. In ICLR, 2022.
[47] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801 000900, 2018.
[48] Tete Xiao, Ilia Radosavovic, Trevor Darrell, and Jitendra Malik, Masked visual pre-training for motor control. arXiv preprint 2rXiv:2203.06/73, 2022. 2022.
[49] Mengjiao Yang and Ofir Nachum. Representation matters: offline pretraining for sequential decision making. In ICML, pages 1 11784-11794 2021.
[50] Denis Yarats. Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved augaeneeeeete reinforcement learning. arXiv preprint wwxiv:2007.09645 2021.
[51] Chengyang Ying. Zhongkai Hao, Xinning Zhou, Hang Su, Songming Liu. Jialian Li. Dong Yan, and Jun Zhu. Reward informed dreamer for task generalization in reinforcement learning. arXiv preprint arXiv:2303.0m 055092, 2023.
[52] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How leverage unlabeled data in offline reinforcement learning. ICML, pages 2561 1-25635, 2022.
[53] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model- policy optimization. In NeurlPS, volume 34, pages 28954-28967, 2021.
[54] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. CoRL. 2019.
[55] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsca Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In NewrlPS, volume 33, pages 14129-14142, 2020.
[56] Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-one to-online reinforcement learning. In ICMI., pages 40452-40474 2023.
[57] Hongyu Zang, Xin Li, Jie Yu, Chen Liu, Riashat Islam, Remi Tachet Des Combes, and Romain Laroche. Behavior prior representation learning for offline reinforcement learning. In ICLR, 2023.
[58] Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. In ICML, pages 11214-11224, 2020.
[59] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In ICLR, 2021.
[60] Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offlinc-to-on reinforcement leamning. In ICLR, 2023.
[61] Wendong Zhang, Geng Chen, Xiangming Zhu, Siyu Gao. Yunbo Wang, and Xiaokang Yang. Predictive experience replay for continual visual control and forecasting, arXiv preprint 26310 2023.
13

---

[62] Zhiyue Zhang, Hongyuan Mei, and Yanxun Xu. Continuous-tine decision transformer for healthcare applications. In International Conference Artificial Intelligence and Statistics, pages 6245-6262. PMLR, 2023.
[63] Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, and Jing Jiang. Adaptive policy learning for offine-to-onlinc reinforcement learning. In AAAI, volume 37, pages 11372-11388 2023.
[64] Zhuangdi Zhu, Kaixiang Lin, Anil Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprimt arXiv:2009.0755 07888, 2020.
[65] Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimiza- tion. In ICIR, 2023.
Appendix
In this appendix, we provide the following supplementary materials: (A) Details of the proposed model, including further descriptions of the learning schemes, the notations, the world model archi- tecture, the behavior learning objective functions, and hyperparameters. (B) Additional experimental results, including visualization of the learned policy, quantitative results offline datasets with mixed data quality, comparison to using a pre-trained foundation model such as R3M, and computational efficiency. (C) Implementation details of the multi-source CoWorld model and further empirical analysis on the selected source domain. (D) Detailed setups of the source and target domains. (E) Details of the compared methods. Potential social impacts of the proposed method.
A Model Details
A.1 Framework of CoWorld
As illustrated in Figure the entire training process of Co comprises three iterative stages: offline-to-onlie state alignment (Stage A), online-to-offlne reward alignment (Stage B), and online- to-offline value constraint (Stage C). First, we feed the same target domain observations sampled from B(T) into the encoders and close the distance e (of (ofT)) and (o, (T) in Stage A. Second, in Stage B, the source reward predictor (.) is trained with mixed data from both of the replay
buffers B(S) and ((T), Notably, when sample data from B(T), the reward will be relabelled the
target-informed source reward. Finally, we introduce a min-max value constraint using the source critic to the target critic in Stage C.
Offline-to-Online State Alignment
Online-to-Offline Reward Alignment
Online-tor Pallne-to-OOfflne Value Constraint
Target observation ?
Target observation (2)
Target reward (X)
Target observation
!
mixup
Target critic 1/2
Guide
Source critic f
Encoder esy
Encoder Es
Source reward predictor TV
Target-informed Source reward !!
Target actor my
D KL
mixup
action
Predict reward he, 2,)
Latent space (c (0 ) of (TT))
Figure 7: CoWorld uses auxiliary online environment to build policy "test bed" that is aware of offline domain information. This, in turn, can guide the visual RL agent in the offline domain to learn colilly-conservativee policy, striking balance between value overestimation and over-conservatism. For notations, we use the superscript S and T to represent data from the source and target domains. Additionally, subscripts (ø', V') and (d, vh. E) are employed distinguish model parameters for different domains. The notations of source and target domains summarised in Table 7
14

---

Table Notations of the source and target domains.
Domains
Model Parameters
Data
Source/Online (S)
World model 6', Actor v Critic g
Raw data (o, (8) . (5)), Relabeled reward
with infomation from both domains F(()
Target/Offline (T)
World model 6, Actor v Critic {
Raw data (o (T) ", ..(
A.2
World Model
We adopt the framework of the world model used in 16. The image encoder Convolutional Neural Network (CNN). The image predictor is transposed CNN and the transition, reward, and discount factor predictors are Multi-Layer Perceptrons (MLPs). The discount factor predictor serves as an estimate of the probability that an episode will conclude while learning behavior based on model predictions. The encoder and decoder take 64 x 64 images as inputs.
A.3 Behavior Learning
For the behavior learning of CoWorld, we use the actor-critic method from DreamerV2 [16. The X-target VI(T) in Eq. (6) is defined as follows:
((T) +
(1
Aloc
[+1
+
ift <
(8)
if i H
where À is set to 0.95 for considering more on long horizon targets. The actor and critic are both MLPs with ELU activations [6]. The target actor and critic are trained with guidance from the source critic and regress the A-return with squared loss. The world model is fixed during behavior learning. The source actor and critic are:
a(5)) TT (a. (1(5) (1())
Source Actor:
Source Critic:
(E)(2) Eap Erz
(8)
We train the source actor Twr by maximizing
H
[(b')= = Ep Pay
Σ
(8H aa
pV(5) (8)
entropy rogularizaaion
dynamics backprop
(10)
(1 In Tep (a)
(5)
ss((v((5) ver (2(
REINFORCE
The source critic ver is optimized by minimizing
C(E') = Epp sPo'
H
(5)
sg
V. (S) 2
)²]
(11)
A.4
Hyperparameters
The hyperparameters of CoWorld are shown in Table
B Additional Quantitative and Qualitative Results
B.1 Visualizations on Policy Evaluation
We evaluate the trained agent of different models on the Meta-World and DMC tasks and select the first 45 frames for comparison. Figure X and Figure present examples of performing the learned policies of different models DMC and Meta- World respectively.
15

---

1=1 1=5
t= 10
[= 15
1= 20
= 25
[= 30
[= 35
[= 40
[= 45
CURL
LOMPO
Offline DV2
CoWorld
(a) Policy evaluation on the DMC Walker Downhill task
t=1
f=5
f=
t=
t=
CURL
LOMPO
Offline DV2
CoWorld
(b) Policy evaluation the DMC Walker Uphill task
t=1
t=
t=
10
t= 15
= 20
t= 25
t= 30
t= 35
t= 40
t= 45
CURL
LOMPO
Offline DV2
CoWorld
(c) Policy evaluation the DMC Walker Nofoot task
Figure 8: Additional qualitative results of policy evaluation on the DMC tasks.
16

---

Table 8: Hyperparameters of CoWorld.
Name
Notation
Value
Co-training
Meta- World / RoboDesk
DMC
Domain KL loss scale
82
1.5
Target-informed reward factor
k
0.3
0.9
Target critic value loss scale Source domain update iterations
a
2
K1
2 10
2 104
Target domain update iterations
K2
5 104
2 10a
World Model
Dataset size
106
Batch size
B
50
Sequence length
L
50
KL loss scale
8
1
Discrete latent dimensions
32
Discrete latent classes
32
RSSM number units
600
World model learning
2 10-1
Behavior Learning
Imagination horizon
H
15
Discount
X
0.995
X-target
0.95
Actor learning rate
4 10
Critic learning rate
1 10-1
1=
t=
=
25
30
35
40
50
CURL
LOMPO
Offline DV2
CoWorld
Figure 9: Policy evaluation the Meta- World Button Topdown task. The model-free method CURL cannot complete the task (green box). CoWorld achieves better performance and finishes the task in fewer steps (red box) than Offline DV2 (blue box).
B.2 Quantitative Results on DMC Medium-Expert Dataset
Similar to the data collection strategy of the medium-replay dataset, we build offline datasets with medium-experl quality using DreamerV2 agent. The medium-experi dataset comprises the sam- ples in the replay buffer during the training process until the policy attains expert-level performance, defined as achieving the maximum score that the DreamerV2 agent can achieve. As shown in Table 9 World outperforms other baselines on the DMC mediun-exper dataset in most tasks.
17

---

Table Performance on DMC medium-experi dataset. We report the mean rewards and standard deviations of 10 episodes over 3 seeds.
Model
WW WD WW WU
ww
WN
CR
- CD
CR
CU
CR
CN
Avg.
Offline DV2
450 + 24
141
8
248 +
3::0
48 +
184
DrQ+BC
808:1477
762:161
808
862+13
454
730
737
LOMPO
548:224
449=117
688:99
174429
19:10
113#35
332
Finetune
7844-46
671--65
8514-91
858
428:149
833417
738
CoWorld
848+-9
774+-29
919417
871+-13
475+116
844+1
789
B.3
Quantitative Results Meta-World
Figure TO(a) compares the performance of different models on Meta-World. DV2 Finetune demon- strates better performance in the initial training phase, thanks to its direct access the source environment. Instead, CoWorld introduces auxiliary source value guidance to assist the training of the target agent. the final phase of training, the source value guidance is more effective, and then CoWorld outperforms DV2 Finetune. Figure To(b) presents the ablation studies of Co conducted on Meta- World, highlighting the effectiveness and necessity of each training stage.
4000
4000
3500
3500
Episode Return
Eoisode Return
3000
3000
2500
2500
2000
2000
CoWorld
1500
CoWorld (B (Best-Source)
1500
CoWorld w/o Stage
CoWorld (Multi Source)
CoWorld w/o Stage
1000
1000
CoWorld w/c Stape
500
Offinne
500
5
15
25
0
10
15
25
30
Number of Iterations (x10*)
Number of Iterations (x10*)
(a) Comparison with other baselines
(b) Ablation studies of CoWorld
Figure 10: (a) Comparison with various approaches on the Meta-World Button Press task. (b) Meta-World
Ablation studies the Meta- World Button Press task that show the effect of state alignment (green), reward alignment (purple), and min-max value constraint (orange).
B.4 Effect of Latent Space Alignment
We feed the same observations into the source and target encoder of CoWorld and then use the t- distributed stochastic neighbor emhedding (t-SNE) method to visualize the latent states. shown in Figure II] the representation learning alignment bridges the gap between the hidden state distributions of the source encoder and target encoder.
B.5 Additional Results on the Realistic Sim2Real Setup
Due to the limitation experimental resources, we are unable to conduct experiments with real robots. We make efforts to construct more realistic sim2real setup. The experiment is conducted with the identical robot control task for both the source and target domains. manually introduce two types of noise into the visual observation and action space of the target domain, trying to mimic complex and noisy real-world scenes.
Visual noise: We modify the original DeepMind Control environment by replacing the static background with dynamic backgrounds of random real-world videos.
Action noise: We add Gaussian noises n, sampled from (/00, 1) to every dimension of the action MorldWorld and RoboDesk, which originally ranges (-1,1). This mimics scenarios where the offline dataset is collected using a low-cost (less inaccurate) real robot.
18

---

100
50
50
0-
50
-50
Source encoder
Source encoder
-100
Target
-100
Target
-150
-100
-50
50
100
150
-150
-100
-50
50
100
150
(a) Latent space before alignment
(b) Latent space after alignment
Visualization of the latent space alignment on Meta-Wort Handle Press + Button Press task by the t-SNE method. (a) Latent space of CoWorld before alignment. (b) Latent space of CoWorld after alignment.
Figure 1I:
Table 10: Results with more significant domain gaps.
Target Domain
CoWorld
DV2 Finetune
DMC Walker Walk
544
457
DMC Cheetah Run
296
220
RoboDesk Push Green 1)
406
358
Meta-World Button Topdown (w = 0.1)
3752
2693
Meta World Button Topdown (w 1)
3567
3104
Meta World Button Topdown (w = 5)
951
670
As shown in Table 10 we compare CoWorld with the finetuned DreamerV2 model on this new setup. We apply noise of three magnitudes, ~ (0.1,1,5), in the Meta-Worla Button Topdown task, leading to noisy actions of at ag w flg-
From the above results, it is evident that CoWorld consistently outperforms the naive finetuning method in this 'sim2real' setup. Importantly, we assess the model under more challenging setups. with more significant domain gaps, as illustrated in Table
B.6
Comparison with Pre-trained Foundation Model R3M
R3M 32 is pretrained Ego4D human video dataset and facilitates efficient learning of downstream robotic tasks. R3M is shown to be a competitive model, particularly in its ability to transfer representations across domains with diverse visual inputs. For model comparison, the pre-trained weights of R3M from the official repository to initialize the representation model. We then perform policy optimization based on it for downstream tasks the Meta-World environment. We respectively employ expert data, sourced from the official repository, alongside our own data, which is collected from scratch with mixed data quality. And DV2 Finetune is also pretrained in related task and finetuned on the offline dataset. As demonstrated in Table 11 our approach outperforms the R3M DV2 fine-tuning model.
leverage
is important to note that:
Despite its generalizable representations, R3M is NOT specifically designed to solve the value over- estimation problem, which is fundamental in offline RL. In contrast, our approach not only aligns state representations across domains but also effectively tackles the issue of value overestimation. and therefore achieves better performance.
The fine-tuning process of R3M necessitates expert demonstrations for high-quality imitation learning. However, its performance empirically deteriorates when applied to the offline dataset of the medium-replay data.
The pre-training process of R3M typically takes around 5 days on V100 GPU, while the entire training procedure of our approach takes only about days 3090 GPU.
19

---

Table 11: Comparison of CoWorld with using pre-trained foundation model, R3M.
CoWorld
R3M (expert data)
R3M (our data)
DV2 Finetume
Button Press Topdown
3889
1609
311
3499
Drawer Close
4845
N/A
4616
4273
Handle Press
4570
N/A
1603
3702
Table 12: Runtime comparisons evaluated on Meta- World (HP - BT).
Model
Training iterations
Training time
Inference time per episode
Offline DV2
300k
2054 min
2.95 sec
DrQ+BC
300k
200 min
2.28 sec
CQL
300k
405 min
1.88 sec
CURL
300k
434 min
2.99 sec
LOMPO
100k
1626 min
4.98 sec
DV2 Finetune
460k
1933 min
6.63 sec
DV2 Finetune+EWC
460k
1533 min
5.58 sec
CoWorld
460k
3346 min
4.47 sec
B.7 Training Efficiency
As shown in Table 12 we evaluate the training/inference time on Meta-World (Handle Press Button Topdown) using single RTX 3090 GPU. Empirically, CoWorld achieves convergence (90% of the highest returns) in approximately 14 hours; while it costs DV2 Finetune about 13 hours. These results indicate that CoWorld requires comparable training wall-clock time to DV2 Finetune, while consistently maintaining better performance in terms returns after model convergence.
C Multi-Source CoWorld
The key idea of multi-source CoWorld is to allocate a set of one-hot weights wf=1:M to candidate source domains by calculating their KL divergence in the latent state space the target domain, where E [1, M] is the index of each source domain. This procedure includes the following steps:
World models pretraining: We pretrain world model for each source domain and target domain individually.
2. Domain distance measurement: At each training step in the target domain, measure the KL divergence between the latent states of the target domain, produced by (s( (T))), and corresponding states in each source domain, produced by est (T) Here, </2
(T) is the encoder of the target world
model, and eøy is the encoder of the world model for the source domain i.
3. Auxiliary domain identification; We dynamically identify the closest source domain with the smallest KL. divergence. We set w wi=1:M as a one-hot vector, where wi indicates the selected auxiliary domain.
4. Rest of training:
With the one-hot weights, we continue the rest of the proposed online-to-offliee training approach. During representation learning, we adaptively align the target state space to the selected online simulator by rewriting the domain alignment loss term in Eq. (3) as
Σ KL sglu(ew (of (T)) g(( }
LMS 32
(12)
To evaluate the effectiveness the multi-source adaptive selection algorithm, we conducted experi- ments on Meta-World and RoboDesk Benchmark. For each target task, two source tasks are used, including the CoWorld berf-pern task and the CoWordd worst-performing task. Additionally, the sub-optimal source task is added for some target tasks.
20

---

Table 13: The source domain automatically selected by Multi-Souree CoWorld. MW represents
Meta- World and RD stands for RoboDesk.
Target domain
Selected source domain
MW: Door Close
MW: Drawer Close
MW: Button Press
MW: Handle Press
MW: Window Close
MW: Button Topdown
MW: Handle Press
MW: Button Press
MW: Button Topdown
MW: Handle Press
MW: Drawer Close
MW: Door Close
RD: Push Button
MW: Button Press
RD: Open Slide
MW: Window Close
RD: Drawer Open
MW: Drawer Close
RD: Upright Block off Table
MW: Handle Press
As shown in Table 13 multi-source CoWorld can adaptively select the best source task for most multi- source problems to ensure adequate knowledge transfer. The performance of multi-source CoWorld reported in Table CoWorld flexibly adapts to the transfer learning scenarios with multiple source domains, achieving comparable results to the model that exclusively uses our manually designated auxiliary simulator as the source domain (best-source). This study significantly improves the applicability of CoWorld in broader scenarios.
D Source and Target Domains
Wera-World. For the Meta Woold environment, we adopt robotic control tasks with complex visual dynamics. For instance, the Door Close task requires the agent to close door with revolving joint while randomizing the door positions, and the Handle Press task involves pressing a handle down while randomizing the handle positions. To evaluate the performance of CoWorld on these tasks, we compare it with several baselines in six visual RL transfer tasks.
RoboDesk. We select Meta- World as the source domain and RoboDesk as the target domain. No- tably, there exists significant domain gap between these two environments. The visual observations, physical dynamics, and action spaces the two environments are different. First, Meta-World adopts side viewpoint, while Robolesk uses top viewpoint. Further, the action space of Meta- World is dimensional, while that RoboDesk is 5-dimensional. Considering these differences, the Seta-World-- RoboDesk benchmark presents challenging transfer learning problem.
Deep Control. We train the source agents in standard DMC environments and train the target agents in modified DMC environments. Walker Uphill and Cheetah Uphill represent tasks in which the ground has 15t uphill slope. Walker Downhill and Cheetah Downhill represent the tasks in which the plane has 15 downhill slope. We evaluate the model in six tasks with different source domains and target domains.
We assume that there exist notable distinctions between the source and target domains (see Table T This assumption
be softened by our proposed approach that mitigates domain discrepancies between distinct source and target MDPs. Our experiments reveal that the CoWorld method exhibits notable tolerance to inter-domain differences in visual observation, physical dynamics, reward definition, or even the action space of the robots. This characteristic makes it more convenient to choose auxiliary simulator based on the type of robot. For example:
When the target domain involves a robotic arm (e.g., RoboDesk), an existing robotic arm simulation environment (e.g., Meta-Worde can be leveraged as the source domain.
In scenarios with legged robots, environments like DeepMind Control with Humanoid tasks can serve as suitable auxiliary simulators.
For target domains related to autonomous driving, simulators like CARLA can be used.
21

---

E
Compared Methods
We compare CoWorld with several widely used model-based and model-free offline methods.
Offline DV2 25 A model-based RI. method that modifies DreamerV2 V2 [16 to offline setting, and adds reward penalty corresponding to the mean disagreement the dynamics ensemble.
DrQ+BC [25]: It modifies the policy loss term in DrQ-v2 (50) to match the loss given in [T]]
CQL 25 It is framework for offline RL that learns Q-function that guarantees a lower bound for the expected policy value than the actual policy value. We add the CQL regularizers to the Q-function update of DrQ-v2 (21).
It is model-free RL approach that extracts high-level features from raw pixels utilizing contrastive learning.
CURL 22
LOMPO [3 An offline model-based RL algorithm that handles high-dimensional observations with latent dynamics models and uncertainty quantification.
LOMPO Finetune: It pretrains a LOMPO agent 39 with source domain data and subsequently
finetunes the pretrained agent in the offline target domain.
It pretrains Dreamer V2 agent [16] in the online source domain and subsequently finetunes the pretrained agent in the offline target domain. Notably, Meta- World -+ RoboDesk tasks' action space is inconsistent, and we can't finetune directly. Instead, use the maximum action space of both environments as the shared policy output dimension. For Meta-World and Meta-World - RoboDesk transfer tasks, we pretrain the agent for 160k steps and finetune it 300k steps. For DMC transfer tasks, we pretrain the agent for 600k steps and finetune it for 600k steps.
DV2 Finetune:
DV2 Finetune+EWC: modifies the DV2 Finetune method with EWC 19 to regularize the model for retaining knowledge from the online source domain. The steps of pretraining and finetuning are consistent with DV2 Finetune.
F Broader Impacts
CoWorld is transfer learning method that may benefit future research in the field of offline RL, model--ased RL, and visual RL. Beyond the realm of reinforcement learning, this approach holds great potential to contribute to various domains such as robotics and autonomous driving.
In real-world scenarios of healthcare applications, Zhang el to train policies using a large amount of historical dataset, determining the follow-up schedule les and tacrolimus dosages in Kidney Transplantation and HIV. There are also corresponding simulators [1 designed by medical domain experts, with parameters learned from real-world data.
62 employed offline RI. algorithms
Another practical use of the proposed setup is advertising bidding, where direct interactions with real online advertising systems for training are challenging. A recent solution involves constructing a simulated bidding environment based on historical bidding logs for interactive training, such [30], and mitigating the inherent differences between the virtual advertising environment and real-world advertising systems. Therefore, in many real-world scenarios, it is possible optimize the policies learned from offline datasets with simulators.
A potential negative social impact of our method is the introduction of existing biases from the additional domain. the training data used to develop our algorithm contains biases, the model may learn those biases, leading to unfair outcomes in decision-making processes. It's crucial to carefully address biases in both data and algorithmic design to mitigate these negative social impacts.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 0
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification: It is difficult to provide theoretical explanation for CoWorld mentioned in the article.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Code is available
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
We provide data and code at httppp///ggthbbb...oo//aaaaaaggn
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ipp/.cc/pullic/uuiii wwwiicccguie//eeeeeeeeeis CoodeSubeiiie ssi asionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4.1 and Appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Table 2aa Table
3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section B.7
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wrrrr..cooooolcc/ccccaaaa
Answer: [Yes]
Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This paper does not involve the responsible release of data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We include the licenses htttpss//ggttubb.cmm///vaaaagg/p htwp:ttww...ccmqiiigtaaet
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We include the documentation at https: com qiwang067
Coworld
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "OJx20zvRd7", "reviewer": "Reviewer_oydA", "summary": "This work proposes a model-based algorithm for offline visual reinforcement learning using domain transfer learning. The proposed method demonstrates superior performance over existing approaches in several baselines.", "questions": "Q1. How does the performance of DV2 Finetune, DV2 Finetune+EWC, and CoWorld look when the x-axis is training wall-clock time instead of the number of training iterations?", "limitations": "Yes, the authors have addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper presents the necessity of proposed modules in the algorithms through ablations and validation in multiple setups.\n\nS2. The proposed method outperforms baselines in different benchmarks.", "weaknesses": "W1. The cost of training time is higher in comparison to baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We have conducted new experiments to discuss the training cost of our model. Please refer to the results presented in the **Rebuttal PDF**. We hope our responses will address your specific concerns regarding training efficiency.\n\n> Q1: How does the performance look when the x-axis is training wall-clock time instead of the number of training iterations?\n\nPer the reviewer's request, we present the results on the Meta-World benchmark (Button Press $\\rightarrow$ Handle Press) in **Figure 1 in the Rebuttal PDF**. As shown, CoWorld achieves convergence (90% of the highest returns) in approximately 14 hours; while it costs DV2 Finetune (DV2-FT) about 13 hours. These results indicate that CoWorld requires a comparable training wall-clock time to DV2-FT, while consistently maintaining better performance in terms of returns after model convergence.\n\nWe will include these results in the revised paper.\n\n> Q2: The cost of training time is higher in comparison to baselines.\n\nWe further illustrate the efficiency of CoWorld from two aspects:\n- Convergence time: From the results above, CoWorld and DV2-FT achieve comparable convergence speeds in wall-clock time. Thereafter, our approach consistently outperforms DV2-FT.\n- Improving efficiency: In **Table 6 in the Rebuttal PDF**, we add new experiments demonstrating that efficiency can be further optimized by flexibly reducing the frequency of source training steps during the co-training phase. The overall performance still surpasses that of DV2-FT (3702 $\\pm$ 451) and DV2-FT-EWC (528 $\\pm$ 334).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4JVvVJcxsV", "reviewer": "Reviewer_DaxB", "summary": "This paper studies CoWorld, a model-based RL method that learns the policy through an offline-online-offline framework. Specifically, CoWorld assumes that, in addition to the offline dataset, there is another (simulated) environment for online interaction. Through the offline dataset and interactive environment, CoWorld constructed two world models to train the separate policies. Next, this paper introduces three components to enhance the policy performance, including offline-to-online state alignment, online-to-offline reward alignment, and min-max value constraint. The proposed CoWorld and several baselines are evaluated on Meta-World, RoboDesk, and DMC benchmarks. From the results presented, this paper suggests that CoWorld outperforms existing RL methods by large margins.", "questions": "Please consider addressing my concerns and questions in Weaknesses; I will list some key ones:\n1. In what practical scenarios is the proposed CoWorld method applicable? Can you provide more examples and details?\n2. How does CoWorld operate when the two world model architectures have greater differences? Can it still effectively transfer information between these two resources?\n3. How to prove that CoWorld is really significantly better than other methods? Can you provide the results of more experimental runs or statistical analysis?", "limitations": "A major limitation is that the proposed CoWorld method is applicable to only a limited number of practical application scenarios. Other limitations are adequately discussed in the Limitation Section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes an interesting method for training an RL policy by co-learning from both the offline dataset and the interactive environment.\n- This work presents a reasonable number of ablation studies and experiments to verify the proposed method. These experiments cover various directions, including environmental settings, component contributions, and hyperparameter sensitivity.\n- This paper is generally well-written and easy to follow. The depicted figures and tables are well-articulated.", "weaknesses": "- While I agree the idea of ​​CoWorld is interesting, I feel that it may not be easy to find suitable practical applications. After all, a low-cost, high-efficiency interactive environment is itself a strong assumption. It would be more convincing if the author could describe practical situations where the proposed method is applicable.\n- Related to the first point, CoWorld learns the separate world models from both the offline dataset and the online environment. I'm curious about how significant the difference between these two sources can be. The setting used in this paper is only a slight difference (4 vs. 5 in the action space, or scale in reward space). But when the two resources are significantly different, can CoWorld still work? For example, (1) one environment with dense reward and another one with sparse reward (not just scale), or (2) the observation and action spaces are obviously different from the two sources.\n- From Table 2, I am concerned about the claim that the proposed CoWorld significantly outperforms other baselines because (1) in several cases, the CoWorld's performance overlaps with other methods when considering its standard deviations, and (2) the results are obtained from only three random seeds. I encourage the authors to conduct more experimental runs and/or provide statistical analysis to eliminate this concern.\n- A minor point: while this paper is generally well-written, I found some typos that should be easily fixed:\n 1. The notation T represents both the target environment and the maximum number of time steps. (line 14 in Algorithm 1)\n 2. Target/Offline (**S**) $\\rightarrow$ should be T? (Table 4, page 13)\n 3. ... on DMC **Meidum**-Expert Dataset $\\rightarrow$ Medium (line 481, page 14)\n 4. ... as illustrated in **Table 1** $\\rightarrow$ should be Table 6? (line 516, page 17)\n\nBased on the observed strengths and weaknesses, I will set my initial recommendation as \"Borderline accept\".\n\nI will be happy to raise my rating if the authors appropriately address my concerns or point out my misunderstandings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your great efforts in reviewing our paper and hope that the following responses can address all of your concerns.\n\n> Q1: Practical situations where the proposed method is applicable. \n\nGood question! CoWorld can be applied in various real-world scenarios such as robot control, autonomous driving, healthcare, advertising bidding, and power systems. For instance, it can leverage established simulators like Meta-World and CARLA for robotics and autonomous driving, or use medical simulators to evaluate clinical treatments in healthcare. Similarly, it can enhance advertising systems through simulated bidding environments and improve power system operations using electrical system simulators. Please refer to our **General Response (Q1)** for further details.\n\n\n>Q2: How does CoWorld operate when the two world model architectures have greater differences? Can it still effectively transfer information between these two resources?\n\nTo investigate the effectiveness of CoWorld under significantly different world model architectures, extended experiments were conducted where the source domain observations were low-dimensional states and the target domain observations were high-dimensional images. CoWorld achieves competitive performance against strong baselines. Please refer to **General Response (Q2, Point(1))** for more details.\n\n>Q3: How to prove that CoWorld is really significantly better than other methods? Can you provide the results of more experimental runs or statistical analysis?\n\nWe extended the results from 3 random seeds to 5 random seeds and tested the performance variations with 3, 5, and 10 random seeds, as well as 10, 20, 50, and 100 episodes. The experiments demonstrate that CoWorld consistently outperforms the baseline model DV2 Finetune (DV2-FT) in terms of performance stability across various tasks and different numbers of random seeds. Please refer to **General Response (Q3)** for more detail.\n\n> Q4: Line 14 in Algorithm 1: The notation $T$ represents both the target environment and the maximum number of time steps.\n\nSorry for the misleading notations. We will use $N$ to represent the maximum number of time steps instead of $T$ in the revised paper. For example: $\\{(o_{t}^{(T)}, a_{t}^{(T)}, r_{t}^{(T)})\\}_{t=1:N} \\sim \\mathcal{B}^{(T)}$. \n\n> Q5: Typos in Table 4 (Page 13) and Line 481 (Page 14).\n\nWe will correct these typos in the revised paper.\n\n> Q6: Line 516 (Page 17): ...as illustrated in Table 1 $\\rightarrow$ should be Table 6?\n\nIn Line 516, we wanted to emphasize that the cross-environment setup from Table 1 is even more challenging than the adding-noise setup shown in Table 6, due to more significant domain gaps. Corresponding results for the cross-environment setup are presented in Section 4.3 of the main text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KKvmAM8ylf", "reviewer": "Reviewer_Bpkq", "summary": "This paper studies model-based offline reinforcement learning with visual observations. This work focuses on the overfitting problem in state representation and the overestimation bias for the value function and introduces the environment model from related tasks to present an offline-online-offline solution. The proposed solution contains state alignment, reward alignment, and value alignment. The empirical studies conduct the experiments with transfer across heterogeneous tasks.", "questions": "1. How to ensure that source critic is beneficial to the target task, and is it possible to have a negative transfer when facing conflicting tasks?  \n2. See the weakness 3.\n\n--- \nAfter rebuttal, my concerns have been addressed well. I decide to raise my score to 6.", "limitations": "The authors have provided a discussion about the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed setting is very interesting to me. The author provides a new perspective on reinforcement learning cross-task transfer from the angle of transfer learning, which is challenging.\n\n2. The proposed solution is reasonable. When online interaction in the current task is unavailable, leveraging online interaction from related tasks is similar to traditional transfer learning. Although the offline-online-offline process appears somewhat complex, it indeed provides an idea that may promote related research.", "weaknesses": "1. The provided results have significant standard deviations. I am not sure if the authors taking 10 episodes over 3 seeds provided a sufficient evaluation. This issue can be referred to: [Deep reinforcement learning at the edge of the statistical precipice. NeurIPS’21]. \n2. In fact, I highly appreciate the contributions made by the authors, if the statistical results of the experiment are reliable (see Weakness 1). This work successfully achieved transfer in multiple environments, which is rare for me, and makes a significant contribution. But where does the success of the transfer come from? The success of state alignment is relatively easy to understand, similar to traditional visual domain adaptation. But what does the success of reward alignment come from? From information like \"objects should be picked\" in the reward signal? This seems to impose relatively strict limitations on the relevance of the task, which is lacking in discussion in the current version. \n3. Similarly, what are the conditions for mitigating value overestimation? Figure 2 is well visualized, but it seems to implicitly assume that the source critic and the ideal target critic perform similarly on the broad support. If based on such an assumption, cross-environment transfer does not seem so difficult. I still affirm the contribution of this work, but can the authors provide a formal analysis of this and determine the applicable conditions for the proposed approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. Our responses are provided below. If you have any further questions, please feel free to let us know.\n\n> Q1: Not sure if taking 10 episodes over 3 seeds is sufficient.\n\nWe extended the results from 3 random seeds to 5 random seeds and tested the performance variations with 3, 5, and 10 random seeds, as well as 10, 20, 50, and 100 episodes. The experiments demonstrate that CoWorld consistently outperforms the baseline model DV2 Finetune (DV2-FT) in terms of performance stability across various tasks and different numbers of random seeds. Please refer to **General Response (Q3)** for more details.\n\n>Q2: The source of success in transfer, particularly in reward alignment, is unclear and seems to impose strict limitations on task relevance, which is not sufficiently discussed. \n\n**(1) On the reasoning behind 'Online-to-Offline Reward Alignment'**\n\nFirst, let us review the training details in the reward alignment process: \n- Step 1: We sample $\\\\{o^{(T)}, a^{(T)}, r^{(T)}\\\\}$ from the target offline dataset;\n- Step 2: In Eq. (4), we use the source reward predictor to predict $R_{\\phi^\\prime}(o^{(T)}, a^{(T)})$ based on the target inputs;\n- Step 3 (**Important!**): Also in Eq. (4), we mix the previous output and the true target reward to generate $\\tilde{r}^{(S)}=(1-k) \\cdot R_{\\phi^\\prime}(o^{(T)}, a^{(T)}) + k \\cdot r^{(T)}$;\n- Step 4 (**Important!**): In Eq. (5), we use $\\tilde{r}^{(S)}$ to supervise the training of $R_{\\phi^\\prime}(o^{(T)}, a^{(T)})$ based on the target inputs;\n- Step 5: Also in Eq. (5), we remain the training term of $R_{\\phi^\\prime}(o^{(S)}, a^{(S)})$ based on the source inputs to approach the raw source reward data $r^{(S)}$.\n\nClearly, due to the aforementioned **Steps 3-4**, the reward model $R_{\\phi^\\prime}(\\cdot)$ learns to approximate the reward combined with the true target reward $r^{(T)}$ conditioned on the target inputs $o^{(T)}$ and $a^{(T)}$, where $\\{o^{(T)}, a^{(T)}, r^{(T)}\\}$ are from the same target tuple. In this way, the reward alignment process integrates target information into the learned reward function of the source domain. Consequently, it enables the subsequent learning process of the source critic to be informed with rich target domain knowledge.\n\n**(2) Can CoWorld benefit from a source domain with a significantly different reward signal?**\n\nYes, CoWorld can benefit from a source domain with a significantly different reward signal. In our experiments, the source rewards are sparse and the target rewards are dense. The experiment results demonstrate that although excessively sparse rewards can hinder the training process, CoWorld still outperforms DV2-FT with a more balanced reward structure. For detailed results and analysis, please refer to **General Response (Q2, Point(2))**.\n\n>Q3: What are the conditions for mitigating value overestimation? \n\nGood question! Regarding the conditions for mitigating value overestimation, Figure 2 illustrates our approach effectively. However, it does indeed assume that the source critic and the ideal target critic perform similarly across a broad range of states. This assumption can be softened by our proposed approaches, namely, 1) state alignment and 2) reward alignment. Through these approaches, we aim to mitigate domain discrepancies between distinct source and target MDPs.\n\n\nAccording to Eq. (9) in Appendix A.4, the source critic $v_{\\xi^\\prime}(\\cdot)$ is trained to estimate the expected future rewards $\\mathbb E[\\sum\\_{\\tau \\ge t\\}({\\gamma}\\_{\\tau-t} \\cdot \\hat{r}\\_\\tau)]$ given the source latent at timestamp $t$. There are two critical points here:\n- For the training supervisions, during the imagination phase, $v_{\\xi^\\prime}(\\cdot)$ is optimized **ONLY** based on the n-step predicted rewards (Algorithm 1, Lines 22-24), **rather than the ground-truth source rewards**. As previously demonstrated, the predicted rewards contain a mix of information from both the source and target domains.\n- At the input end, $v_{\\xi^\\prime}(\\cdot)$ is indeed trained on the imagined source latents. However, due to the state alignment process, the source latents share close distributions with the latents conditioned on target inputs. \n\nThus, it is not entirely accurate to suggest that $v_{\\xi^\\prime}(\\cdot)$ is trained to '*maximize the expected sum of source rewards in a range of source latents.*' In fact, it learns to maximize the expected sum of **target-informed rewards** across latents that are **closely distributed to the target latents**. Consequently, the source critic can be effectively applied to the target domain to guide the training of the target agent, using the target latents as inputs. \n\n\n> Q4: How to ensure that source critic is beneficial to the target task, and is it possible to have a negative transfer when facing conflicting tasks? \n\nIn Figure 3 (left) of the original manuscript, we evaluate our model's performance across each pair of source and target domains. We observe positive transfer effects (indicated by the red blocks) in most cases (18/20 single-domain transfer scenarios). And indeed, we also observe negative transfers (indicated by the blue blocks) in 2/20 scenarios (e.g., DC$^*$ $\\rightarrow$ BP). \n\nTo ensure an effective source critic, as described in Lines 159-162, we first manually select several candidates for the source domain that are potentially relevant to the target task. We then propose a method for multi-source training, with technical details provided in Appendix C. By introducing the multi-source CoWorld, we can automatically discover a beneficial task from multiple source candidates. As shown in Figure 3 (left), the multi-source CoWorld consistently achieves positive transfer (indicated by the results inside the dashed blue box). \n\nOn the right side of Figure 3, we further demonstrate the effectiveness of the multi-source domain selection method. It achieves comparable results to that of the top-performing single-source CoWorld.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dj79gx2GNZ", "reviewer": "Reviewer_vWw1", "summary": "This paper introduces Collaborative World Models (CoWorld), a novel approach to offline reinforcement learning (RL) with visual inputs. CoWorld conceptualizes offline RL as an online-to-offline transfer learning problem, utilizing an auxiliary online simulator to address the challenges of overfitting in representation learning and value function overestimation. The method involves training separate source and target world models and actor-critic agents, aligning their latent spaces, and using a three-stage learning procedure to improve performance on the target task.", "questions": "1. How dependent is CoWorld's performance on the quality and similarity of the online simulator to the target task?\n2. Have you considered training on multiple source domains to improve robustness and generalization?\n3. How does the computational complexity of CoWorld compare to baseline methods in terms of training time and resource requirements?\n4. Can you provide more insight into the process of selecting an appropriate source domain, and how this might be done without leaking test-time information?\n5. How would CoWorld perform if the source and target domains have significantly different observation spaces?", "limitations": "Yes, the authors have discussed the limitations in the submission.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The idea is novel: CoWorld presents an interesting perspective on offline RL by framing it as an online-to-offline transfer learning problem.\n- The problem is important: The method aims to tackle both overfitting in representation learning and the trade-off between value function overestimation and over-conservatism.\n- Ablation studies and Analysis: The paper includes comprehensive ablation studies that validate the contribution of each component in the CoWorld framework.", "weaknesses": "1. Dependence on auxiliary online environments: CoWorld heavily relies on the availability of an online environment in a sufficiently similar domain, which may limit its applicability in real-world scenarios.\n2. Unclear rationale for separate world models: The decision to maintain separate source and target world models, rather than a single jointly trained model, is not adequately explained or supported.\n3. Marginal performance improvement: The performance gains of CoWorld over simpler baselines (e.g., fine-tuning) are sometimes marginal and highly sensitive to hyperparameters.\n4. Computational complexity: The alternation between online and offline agents until convergence may be computationally expensive and potentially unstable.\n5. Strong assumptions: The requirement of a high-quality simulator for a similar source domain contradicts the motivation for offline RL, which assumes online data collection is costly or dangerous.\n6. Insufficient clarity in the presentation: The necessity and function of individual components—such as world model learning, state alignment, and reward alignment—are not clearly delineated, leading to potential confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The availability of an online environment.\n\nThere are many real-world scenarios (such as robot control, autonomous driving, and healthcare) where existing simulators can be employed as source domains. See **General Response (Q1)** for details.\n\nFurther, our method is specifically designed to address cross-domain inconsistencies in dynamics, visual appearances, and action space (Table 1), thereby reducing the necessity for a very similar online environment.\n\n> Q2: How dependent is CoWorld on the quality of the online simulator?\n\nIn Fig 3 (left) of the original paper, we evaluate our model's performance across each pair of source and target domains. We observe positive transfer effects (indicated by the red blocks) in 18/20 single-domain transfer scenarios. Indeed, we also observe negative transfers (blue blocks) in 2/20 scenarios. \n\nIn Lines 159-162 and Appendix C, we present the multi-source CoWorld that can automatically discover a beneficial task from multiple source candidates. In Fig 3 (left), the multi-source CoWorld consistently achieves positive transfer (highlighted by the dashed blue box). The right side of Fig 3 demonstrates the effectiveness of the multi-source CoWorld. It yields results comparable to those of the top-performing single-source CoWorld.\n    \n> Q3: Why maintain separate source and target world models?\n\nFirst, using separate world models allows the source and target domains to have significantly different observations. See **General Response (Q2, Point(1))** for additional empirical results where the source domain has a low-dim observation space.\n\nAdditionally, we compare CoWorld (w/ separate models) and Multi-Task DV2 (w/ one model) in **Table 7 in the Rebuttal PDF**. The latter involves training on offline and online data with a joint world model and separate actor-critic models. CoWorld consistently performs better in terms of the cross-environment results (Meta-World -> RoboDesk). \n    \n> Q4: Training on multiple source domains.\n    \nAs suggested, we design two baseline models trained with multiple source domains and compare them with our proposed multi-source CoWorld:\n- DV2-FT (multi-source pre-train), which involves pre-training on DC, WC, and DC* as sources for 40K steps on each task and then finetuning on BP for 300K steps.\n- CoWorld (multi-source co-train), which involves co-training with DC, WC, and DC*, as well as the target (BP), achieving a return of 3012.\n\nAs shown in **Table 8 in the Rebuttal PDF**, training on multiple sources can lead to negative transfer. In contrast, CoWorld can identify appropriate auxiliary tasks, enabling it to better manage the potential negative impacts of less relevant source tasks\n\n> Q5: Computational complexity.\n\nWe present the results on Meta-World (BP -> HP) in **Fig 1 in the Rebuttal PDF**. CoWorld achieves convergence (90% of the highest returns) in approximately 14 hours; while it costs DV2 Finetune (DV2-FT) about 13 hours. These results indicate that CoWorld requires a comparable training wall-clock time to DV2-FT, while consistently maintaining better performance in terms of returns after model convergence.\n\nWe further illustrate the efficiency of CoWorld from two aspects:\n- Convergence time: From the results above, CoWorld and DV2-FT achieve comparable convergence speeds in wall-clock time. Thereafter, our approach consistently outperforms DV2-FT.\n- Improving efficiency: In **Table 6 in the Rebuttal PDF**, we add new experiments demonstrating that efficiency can be further optimized by flexibly reducing the frequency of source training steps during the co-training phase. The overall performance still surpasses that of DV2-FT (3702 $\\pm$ 451) and DV2-FT-EWC (528 $\\pm$ 334). \n\nCoWorld, like the baseline models, can be trained using a single 3090 GPU.\n\n> Q6: More insight into selecting an appropriate source domain.\n\nRecognizing the challenges of manually selecting an appropriate source domain, we propose a multi-source CoWorld approach with an adaptive domain selection method.\n\nFirst, we typically have some information about the target task, based on which we can initially construct a set of source domain candidates (e.g., if the target domain is a robotic arm visual control task, we can randomly select several tasks from Meta-World or other environments as potential source domains), and we allow some of these source domains to differ significantly from the target task.\n\nNext, we propose a method for adaptive source domain selection. This is achieved by measuring the distance of the latent states between the offline target dataset and each source domain provided by different world models. We have included technical details in Appendix C and corresponding results in Fig 3.\n\n> Q7: The performance gains over simpler baselines (fine-tuning).\n\nNotably, finetuning requires a very high similarity between the source and target domains, resulting in poor performance in the following three scenarios: \n1. Cross-environment experiments (Fig 4)\n2. Adding-noise experiments (Table 6)\n3. Scenarios with sparse rewards in the source domain (**General Response Q2, Point(2)**).\n\nAs shown in **Table 9 in the Rebuttal PDF**, CoWorld demonstrates significant improvement over the fine-tuning method, particularly in cases where there are more pronounced domain discrepancies.\n\nFurthermore, another disadvantage of the finetuning method is that it cannot be applied in scenarios where source and target domains have observation spaces of different dimensions. See **General Response (Q2, Point(1))**. \n\n> Q8: What if the source and target domains have significantly different observation spaces?\n\nWe have conducted new experiments using a source task with low-dimensional states (Meta-World) and a target domain characterized by high-dimensional images (RoboDesk). Please refer to **General Response (Q2, Point(1))** for the results.\n\n> Q9: Clarity in presentation.\n\nWe hope **Table 10 in the Rebuttal PDF** can make the paper easier to understand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Responses to All Reviewers
In addition to the specific responses below, we here reply to the general questions raised by the reviewers.

>Q1: Practical situations where the proposed method is applicable.

As demonstrated in Table 1 of our manuscript, our method can be used in scenarios despite notable domain gaps in terms of dynamics, visual appearances, and action space. Typical real-world applications include robot control, autonomous driving, healthcare, advertising bidding, and power systems. 
- **Robotics and autonomous driving:** For the first two scenarios, established simulators such as Meta-World and CARLA [1] are readily available, and numerous existing studies have explored the transfer learning problem using these simulators.
- **Healthcare applications:** Existing RL models in this field [2] are learned on large volumes of offline clinical histories. Additionally, we have medical domain simulators [3-4] designed by experts, which allow us to evaluate the learned clinical treatments. Exploring how to use these simulators to improve RL therapies in real-world data would be a promising research avenue. 
- **Advertising bidding:** Since direct online interactions with real advertising systems are impractical, recent work has constructed a simulated bidding environment based on historical bidding logs to facilitate interactive training [5]. In this context, the next research focus may be on how to effectively use such simulators to enhance real-world advertising systems.
- **Power system:** In the realm of power systems, we can use the power system simulator as the source domain, with real power system data serving as the target domain. Recent RL-related studies in power systems [6-7] demonstrate the effectiveness of training RL controllers on high-fidelity simulations or offline data before deploying them on real-world systems, such as service restoration in microgrids and demand response in electric grids.

In summary, there are many real-world scenarios where existing simulators can be employed to improve offline RL policies. As more and more simulators are developed, the research on online-to-offline RL transfer will become increasingly practical.

References:

[1] Dosovitskiy et al. "CARLA: An Open Urban Driving Simulator", CoRL, 2017. 

[2] Zhang et al. "Continuous-Time Decision Transformer for Healthcare Applications", AISTATS, 2023.

[3] Hua et al. "Personalized Dynamic Treatment Regimes in Continuous Time: A Bayesian Approach for Optimizing Clinical Decisions with Timing", Bayesian Analysis, 2022.

[4] Adams et al. "Dynamic Multidrug Therapies for HIV: Optimal and STI Control Approaches", Math Biosci Eng, 2004.

[5] Mou et al. "Sustainable Online Reinforcement Learning for Auto-bidding", NeurIPS, 2022.

[6] Du et al. "Deep reinforcement learning from demonstrations to assist service restoration in islanded microgrids", IEEE Trans. Sustain. Energy, 2022.

[7] Lesage-Landry et al. "Batch reinforcement learning for network-safe demand response in unknown electric grids", Electric Power Systems Research, 2022.

>Q2: Dependence of CoWorld to the source domain.

We further elaborate on CoWorld’s dependence on the similarity of the source domain from the perspectives of different observation spaces and reward spaces.

**（1）How would CoWorld perform if the source and target domains have significantly different observation spaces?**

To investigate the effectiveness of CoWorld under significantly different observation spaces, we conduct experiments on MetaWorld $\rightarrow$ RoboDesk tasks, where the source domain observations consist of low-dimensional states, and the target domain observations are high-dimensional images. As illustrated in **Table 1 in Rebuttal PDF**, CoWorld outperforms Offline DV2 by **13.3%** and **34.0%** due to the ability to effectively leverag low-dimensional source data.

NOTE: the finetuning baseline method (DV2-FT) is not applicable is this scenario.

**（2）Can CoWorld benefit from a source domain with a signifcantly different reward signal?**

In the Meta-World $\rightarrow$ RoboDesk, we modify the rewards of the source domain (Meta-World) to be sparse in order to investigate the impact of the source task's reward on target training. The reward in the source domain is set to 500 only upon task completion and remains 0 before that. Despite the significant differences in rewards, CoWorld achieves an average performance gain of 6.6% compared to DV2-FT. Please refer to **Table 2 in the Rebuttal PDF**.

> Q3: The performance stability of CoWorld.

(1) We extend the 5-seeds experiments to the four tasks and observe consistent results with those obtained using 3 random seeds. This indicates that using 3 random seeds is sufficient to reveal the differences in average performance between the compared models. Please refer to **Table 3 in the Rebuttal PDF**.
    
(2) We compare the performance of CoWorld and DV2 Finetune (DV2-FT) using 3, 5, and 10 random seeds, respectively. The results show that CoWorld achieves stable performance across different numbers of training seeds and consistently outperforms the baseline models. Please refer to **Table 4 in Rebuttal PDF**.

(3) To assess the stability of the model during testing, we conduct further experiments running 10, 20, 50, and 100 episodes in the Meta-World Button Press $\rightarrow$ Handle Press task. We calculate the mean and standard deviations for each case. The results below indicate that testing 10 episodes under each random seed is sufficient to demonstrate the improvement of our model over the baseline model in terms of average performance. Please refer to **Table 5 in the Rebuttal PDF**.
 

**NOTE:** In RoboDesk, OS is short for Open Slide and PB is short for Push Button.

Meta-review: ### Summary

This paper studies offline reinforcement learning (RL) challenges with visual inputs, specifically focusing on overfitting in representation learning and overestimation bias in future reward predictions. The authors propose CoWorld, a model-based RL approach that integrates offline and online learning by utilizing existing RL simulators for policy testing. CoWorld aims to align state and reward spaces across domains to facilitate effective knowledge transfer between online and offline environments. Experimental results show that CoWorld significantly improves performance compared to existing methods.

### Decision

The paper is generally well-written and easy to understand. The depicted figures and tables are well explained. The topic the paper is studying is relevant to the neurips community and essential to improving the performance of offline RL algorithms. As the authors noted, these findings can potentially have implications for robotic control, autonomous driving, and healthcare applications. The paper demonstrates competitive results across three RL benchmarks. 

The reviewers pointed out several concerns during the rebuttal phase, which the authors addressed successfully. However, I would recommend that the authors incorporate those changes into the camera-ready version of the paper. Most of these changes are minor, like highlighting some of the potential applications and experiments with more seeds.
Decision: Accept (poster)