MoVie: Visual Model-Based Policy Adaptation for View Generalization
Authors: Sizhe Yang
Yanjie Ze
Huazhe Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: visual reinforcement learning
visual generalization

Abstract: Visual Reinforcement Learning (RL) agents trained on limited views face significant challenges in generalizing their learned abilities to unseen views. This inherent difficulty is known as the problem of $\textit{view generalization}$. In this work, we systematically categorize this fundamental problem into four distinct and highly challenging scenarios that closely resemble real-world situations. Subsequently, we propose a straightforward yet effective approach to enable successful adaptation of visual $\textbf{Mo}$del-based policies for $\textbf{Vie}$w generalization ($\textbf{MoVie}$) during test time, without any need for explicit reward signals and any modification during training time. Our method demonstrates substantial advancements across all four scenarios encompassing a total of $\textbf{18}$ tasks sourced from DMControl, xArm, and Adroit, with a relative improvement of $\mathbf{33}$%, $\mathbf{86}$%, and $\mathbf{152}$% respectively. The superior results highlight the immense potential of our approach for real-world robotics applications. Code and videos are available at https://yangsizhe.github.io/MoVie/.

MoVie: Visual Model-Based Policy Adaptation for View Generalization
Sizhe Yang Yanjie Ze 1e, Huazhe Xu
*Shanghai Qi Zhi Institute, UUniversity of Electronic Science and Technology of China, Shanghai Jiao Tong University, "Tsinghua University, "Shanghai Al Lab Equal contribution yangsizhe.gihuub.o//oooo
Abstract
Visual Reinforcement Learning (RL) agents trained on limited views face sig- nificant challenges in generalizing their learned abilities to unseen views. This inherent difficulty is known as the problem of view generalization. In this work, we systematically categorize this fundamental problem into four distinct and highly challenging scenarios that closely resemble real-world situations. Subsequently, we propose straightforward yet effective approach to enable successful adaptation of visual Model-based policies for View generalization (MoVie) during test time, without any need for explicit reward signals and any modification during training time. Our method demonstrates substantial advancements across all four scenarios encompassing total of 18 tasks sourced from DMControl, xArm, and Adroit, with relative improvement of 33%, 86%, and 152% respectively. The superior results highlight the immense potential of our approach for real-world robotics applications. Code and videos are available at wawggiizee.iihub.io/MM...... gotooubioo/ooiii
DMControl
xArm
Adroit
600
493
427
40
400
200
20
TD-MPC
DM
IDM+STN
MoVie
Figure 1: View generalization results across 3 domains. Our method Mo largely improves the test-time view generalization ability, especially in robotic manipulation domains.
Introduction
Visual Reinforcement Learning (RL) has achieved great success in various applications such as video games [17, 18], robotic manipulation [22], and robotic locomotion [34]. However, one significant challenge for real-world deployment of visual RL agents remains: policy trained with very limited views (commonly one single fixed view) might not generalize to unseen views. This challenge is especially pronounced robotics, where few fixed views may not adequately capture the variability of the environment. For instance, the RoboNet dataset [3] provides diverse views across range of manipulation tasks, but training on such large-scale data only yields moderate success rate (10% 20%) for unseen views [3].
Recent efforts have focused improving the visual generalization of RL agents [1.7,9,36] However, these efforts have mainly concentrated on generalizing to different appearances and backgrounds. In contrast, view generalization presents unique challenge as the deployment view is unknown and may move freely in the 3D space. This might weaken the weapons such as data augmentations [12, 14, 15,33] that are widely used in appearance generalization methods. Meanwhile, scaling up domain
37th Conference on Neural Information Processing Systems (NcurIPS 2023), New Orleans.

---

Training Time
Test Time
(Seen View)
(Policy Adapation Unseen
Spatial Adaptive Encoder
Oy
Encoder
Latent Dynamics
O2
Latent Dynamics
h
d
h
d
Action
Transfoom Param
Action
CASE
h
O++1
Conzatenaiio
Encoder
Or+1
2D Transfommaton
Spatial Adaptive Encoder
Figure 2: Overview of MoVie. During training (left), the agent is trained with the latent dynamics loss. At test time (right), we freeze the dynamics model and modify the encoder as a spatial adaptive encoder to adapt the agents novel views.
randomization [19,21, 19]31,31] to all possible views usually unrealistic because the large cost and offline nature of existing robot data. With these perspectives combined, it is difficult to apply those common approaches to address view generalization.
In this work, we commence by explicitly formulating the test-time view generalization problem into four challenging settings: a) novel view. where the camera changes to a fixed novel position and orientation: b) moving view, where the camera moves continuously around the scene, c) shaking view, where the camera experiences constant shaking, and d) novel FOV, where the field of view (FOV) of the camera is altered initially. These settings cover wide spectrum of scenarios when visual RL agents are deployed the real world. By introducing these formulations, we aim to advance research in addressing view generalization challenges and facilitate the utilization of robot datasets [3] and deployment on physical robotic platforms.
To address the view generalization problem, we argue that adaptation to novel views during test time crucial, rather than aiming for view-invariant policies. We propose Mo MoVie, simple yet effective method for adapting visual Model-based policies to generalize to unseen Views. Mo leverages collected transitions from interactions and incorporates spatial transformer networks (STN [13]) in shallow layers, using the learning objective of the dynamics model (DM). Notably, Mo Vie requires no modifications during training and compatible with various visual model-based RL algorithms. It only necessitates small- scale interactions for adaptation to the deployment view.
We perform extensive experiments on 7 robotic manipulation tasks (Adroit hand [20] and xArm [7]) and 11 locomotion tasks (DMControl suite [30]), across the proposed 4 view generalization settings, totaling 18 X configurations. MoVie improves the view generalization ability substantially, com- pared to strong baselines including the inverse dynamics model (IDM [7]) and the dynamics model (DM). Remarkably, MoVie attains a relative improvement of 86% xArm and 152% in Adroit, underscoring the potential of our method in robotics. We are committed to releasing our code and testing platforms. To conclude, our contributions are three-fold:
We formulate the problem of view generalization in visual reinforcement learning with wide range of tasks and settings that mimic real-world scenarios.
We propose a simple model-based policy adaptation method for view generalization (MoVie) which incorporates STN into the shallow layers of the visual representation with self-supervised dynamics prediction objective.
We successfully showcase the effectiveness of our method through extensive experiments. The results serve as a testament its capability and underscore its potential for practical deployment in robotic systems, particularly with complex camera views.
Related Work
Visual generalization in reinforcement learning. Agents trained by reinforcement learning (RL. from visual observations are prone to overfitting the training scenes, making it hard to generalize to

---

unseen environments with appearance differences. A large corpus of recent works has focused on addressing this issue [7,9, 10, 14, 19,1, 36]. Notably, SODA [10] provides a visual generalization benchmark to better evaluate the generalizability of policies, while they only consider appearance changes of agents and backgrounds. Distracting control suite [26] adds both appearance changes and camera view changes into DMControl [30], where the task diversity is limited.
View generalization in robotics. The field of robot learning has long grappled with the challenge of training models on limited views and achieving generalization to unseen views. Previous studies, such as RoboNet [3], have collected extensive video data encompassing various manipulation tasks. However, even with pre-training on such large-scale datasets, success rates unseen views have only reached approximately 10% 20% [3]. In recent efforts tackle this challenge, researchers have primarily focused on third-person imitation learning [23-25] and view-invariant visual representations [2,4, 14,,16,22, (2,3,, but these approaches are constrained by the number of available camera views. In contrast, our work addresses more demanding scenario where agents trained on single fixed view are expected to generalize to diverse unseen views and dynamic camera settings.
Test-time training. There is line of works that train neural networks at test-time with self-supervised
learning in computer vision [5, 28,28.22 29,, robotics [27], and visual RL [7]. Specifically, PAD is the closest to our work [7], which adds an inverse dynamics model (IDM) objective into model-free policies for both training time and test time and gains better appearance generalization. In contrast, we (i) we focus on visual model- based policies, (ii) we require no modification
differ in a lot of aspects:
in training time, and (iit) our method designed for view generalization specifically.
3 Preliminaries
Formulation. We model the problem as Partially Observable Markov Decision Process (POMDP)
M = (O,A, R. 7). where E O are observations, E A are actions, F O A O is a transition function (called dynamies as well), r E are rewards, and y E [0, 1) is discount factor. During training time, the agent's 's goal is to learn a policy и that maximizes discounted cumulative rewards on M, i.e., max E, Σ ?'re). During test time, the reward signal from the environment is not accessible to agents and only observations are available, which are possible to experience subtle changes such as appearance changes and camera view changes.
Model-based reinforcement learning. TD-MPC [11] is a model-based RI. algorithm that combines model predictive control and temporal difference learning. TD-MPC learns a visual representation z h(o) that maps the high- -ieeenimeniin observation o E O into latent state z E Z and latent dynamics model d Z X A + Z that predicts the future latent state z' = d(z, a) based on the current latent state and the action a. MoDem [8] accelerates TD-MPC with efficient utilization of expert demonstrations D {D,, D2... ... DN} } to solve challenging tasks such as dexterous manipulation [20]. In this work, we select TD-MPC and MoDem as the backbone algorithm to train model- based agents, while our algorithm could be easily extended to most model-based RL
algorithms such as Dreamer [6].
4 Method
We propose simple yet effective method, visual Model-based policy adaptation for View generaliza- tion Vie), which can accommodate visual RI. agents to novel camera views at test time.
Learning objective for the test time. Given a tuple (o., a,, 0 the original latent state dynamics prediction objective can be written as
Ldynamies [ld(h(o.), a.) h( 1)
(1)
where h is an image encoder that projects a high-dimensional observation from space O into latent space 2 and d is a latent dynamics model d x A Z.
In test time, the observations under unseen views lie in different space O', so that their corresponding latent space also changes to 2'. However, the projection h learned training time can only map
O - 2 while the policy æ only learns the mapping Z + A, thus making the policy hard to generalize to the correct mapping function 2' + A. Our proposal thus to adapt the projection
from mapping function h O + Z to more useful mapping function h O' + Z so that the
3

---

Baseline
Ours
A
A
Training Time
h
π
ASAE
Test
Time
Figure An illustration of the reason why MoVie is effective. We treat the frozen dynamics model as the source of supervision to adapt the latent space of h SAS to that of the training views.
Transform Param
Transform Param
Localisation
Localisation Net
Net
conv
ReLU
conv
ReLU
Zz
Sampler
Sampler
Or
MLP
Figure 4: The architecture of our spatial adaptive encoder (SAE). We incorporate STNs before the first two convolutional layers of the encoder for better adaptation of the representation h.
policy would execute the correct mapping + A without training. A vivid illustration is provided in Figure 3.
We freeze the latent dynamics model d, denoted as d", so that the latent dynamics model is not training target but supervision. We also insert STN blocks [13] into the shallow layers of h to better adapt the projection h; so that we write h as h SAS (SAE denotes spatial adaptive encoder). Though the objective is still the latent state dynamics prediction loss, the supervision here is superficially identical but fundamentally different from training time. The formal objective is written as
||d5 (h SAE (o), a) SA (o(-))))
Spatial adaptive encoder. We now describe more details about our modified encoder architecture during test time, referred to as spatial adaptive encoder (SAF). To keep our method simple and fast to adapt, we only insert two different STNs into the original encoder, as shown in Figure 4. We observe in our experiments that transforming the low-level features (i.e., RGB features and shallow layer features) is most critical for adaptation, while the benefit of adding more STNs is limited (see Table 6). An STN block consists two parts: (i) a localisation net that predicts an affine transformation with 6 parameters and (ii) grid sampler that generates affined grid and samples features from the original feature map. The point-wise affine transformation written as
xta y²
x
To) As
di
212
113
yr
d223
where G is the sampling grid, (af, vf) are the target coordinates of the regular grid in the output feature map, (x), yf) are the source coordinates in the input feature map that define the sample points, and A is the affine transformation matrix.
Training strategy for SAE. We use learning rate 1 10-5 for STN layers and x 10-7 for the encoder. We utilize replay buffer with size 256 to store history observations and update 32 times for each time step heavily utilize the online data. Implementation details remain in Appendix A.

---

Figure 5: Visualization of three task domains, including DMControl [30], xArm [7], Adroit [20].
Training View
Novel View
Moving View
Shaking View
Novel FOV
Figure 6: Visualization of the training view and four view generalization settings. Trajectory website yangsizheegithub.io/Moo
visualization for all tasks is available
5 Experiments
In this section, we investigate how well an agent trained on a single fixed view generalizes to unseen views during test time. During the evaluation, agents have access reward signals, presenting significant challenge for agents to self-supervise using online data.
5.1 Experiment Setup
Formulation of camera view variations: a) novel view, where we maintain a fixed camera target while adjusting the camera position in both horizontal and vertical directions by certain margin, b) moving view, where we establish predefined trajectory encompassing follows this trajectory, moving back and forth for each time step while focusing on the center of scene, c) shaking view, where we add Gaussian noise onto original camera position each time step, and d) novel FOV, where the FOV of the camera is altered once, different from the training phase. A visualization of four settings is provided Figure and videos are available in yangsizhe. muugsizhe.gihub.i//MooY. github.io//ooiiiee for a better understanding of our configurations. Details remain in Appendix B.
scene and the camera
Tasks. Our test platform consists of 18 tasks from 3 domains: 11 tasks from DMControl [30], 3 tasks from Adroit [20], and 4 tasks from xArm [7]. A visualization of the three domains in Figure Although the primary motivation for this study is for addressing view generalization in real-world robot learning, which has not yet been conducted, we contend that the extensive range tasks tackled in our research effectively illustrates the potential of Mo Vo for real-worle application. We run seeds per experiment with seed numbers 0, 1, and run 20 episodes per seed. During these 20 episodes, the models could store the history transitions. We report cumulative rewards for DMControl tasks and success rates for robotic manipulation tasks, averaging over episodes.
Baselines. We first train visual model--asee policies with TD-MPC [11] on DMControl and xArm environments and MoDem [8] on Adroit environments under the default configurations and then test these policies in the view generalization setting. Since MoVie consists of two components mainly (i.e., DM and STN), we build two test-time adaptation algorithms by replacing each module: a) DM, which removes STN from Mo Vie and b) IDM+STN, which replaces DM in MoVie with IDM [7]. IDM+STN is very close to PAD [7], and we add STN for fair comparison. We keep training settings the same.

---

Table 1: Experiment conclusion across all domains and all settings. The best method on each setting is in hold and the relative improvement over TD-MPC is also reported.
Setting
Domain
TD-MPC
DM
IDM+STN
MoVie
DMControl
395.61
508.89 (-29%)
378.80 (144%)
623. 19 (157%)
Novel view
XArm
16
36 (*125%)
18 (113%)
46 (1188%)
Adroit
8
(178%)
12 (150%)
(1325%)
DMControl
605.98
611.87 (11%)
502.34 (J17%)
673. 74 (111%)
Moving view
xArm
20
31 (156%)
24 (120%)
42 (1110%)
Adroit
15
20 (133%)
15 (10%)
45 (1200%)
DMControl
441.79
348.26 (L21%)
291 19..4 (.26%)
558.2 238.23 (126%)
Shaking view
xArm
42
44 (15%)
44 (15%)
45 (17%)
Adroit
30
(117%)
(113%)
(1110%)
DMControl
527.47
613.74 (716%)
542.43 (13%)
770.56 (1465))
Novel FOV
xArm
34
47 (138%)
40 (118%)
75 (1121%)
Adroit
31
51 (165%)
31 (10%)
68 (1119%)
DMControl
492.71
520.68 (16%)
426.76 (L13%)
656.43 (133%)
All settings
xArm
28
40 (143%)
32 (114%)
52 (186%)
Adroit 21
(143%)
21 (10%)
(h152%)
Table 2:
Results in novel view. The best method on each task is in bold.
Novel view
TD-MPC
DM
IDM+STN
MoVic
Cheetah, run
00.13120.33
254.43
127. 16
342.39 39 15..95
Walker, walk
149.78
622
215. 115.90+22.30
512. 410444
Walker, stand
568.01 16.81
635.64111.770
508. 157
679. 12..2.0333
Walker, run
127.07
131.65=22 20
94.49+18.72
94.49+18.87
Cup. catch
922.83 133.77
949.3611.220
932. 25.23344
961.98 12.68
Finger, spin
137.30+4.44
750.65+9.90
141.03+37.55
892.01+1.00
Finger, turn-easy
180.80 54
528.91 160..22
348 1488000_123.68
705. 705.36171.99
Finger, turn-hard
261.43=72.88
312.36=134.22 28
306.76
331 831.83118
Pendulum, swingup
40.65
64.51
82.46.8.19
528. 728.76.17.55
Reacher, casy
981.25++44
982.30+00.8 83
875.31
984.6642.
Reacher, hard
592.91
724.91
533.50
821 221..011: 03
DMControl
395.61
508.85
378.80
623. 19 (157%)
xArm, reach
3+2
85413
1+2
95:00
xArm, push
4625
4518
60
4817
XAmm, peg in box xArm, hammer xArm
5+0
5415
3+2
617
10:10
10:10
845
36::22
16
36
18
46 (1188%)
Adroit, door Adroit, hammer
0z0
1045
0
6647
6#2
11:10
21
Adroit, pen
18=5
2048
1545
25418
Adroit
8
14
12
34 (1325%)
5.2 Main Experiment Results
Considering the extensive scale of our conducted experiments, we present an overview of our findings in Table and Figure 1. The detailed results for four configurations provided in Table Table 3, Table 4, and Table respectively. We then detail our findings below.
Superiority of MoVie across domains, especially in robotic manipulation. It is observed that irrespective of the domain or configuration, MoVie consistently outperforms TD-MPC without any adaptation and other methods that apply DM and IDM. This large improvement highlights the fundamental role of our straightforward approach in addressing view generalization. Additionally, we observe that MoVie exhibits greater suitability for robotic manipulation tasks. We observe significant relative improvement of 86% in xArm tasks and 152% in Adroit tasks, as opposed to comparatively modest improvement of only 33% in DMControl tasks; see Table 1. We attribute this

---

Table 3: Results moving view. The best method on each task in bold.
Moving view
TD-MPC
DM
IDM+STN
MoVie
Cheetah, run
235.37=63.38
344.70=7.55
199.484228 78
365.22: 66
Walker, walk
632.77 13.62
707 777.60.26.77
373.44 133..8
810. 110.19.1.777
Walker, stand
803.97=15.75
576. 77..87741.69
712.48+11.
Walker, run
295.5414..0 1.39
250.8417.02
190.021..00
183.43
Cup, catch
$87.20++3.7
910.51+4.12
915. 16=7.31
251.26610..
Finger, spin
336.9111399
19..11
532.90 13.25
596.00.211.
Finger, turn-casy
115.45=19.23
128.93+180.22
683.85=112.00 64
744.45+16..
Finger, turn-hard
593.46 144.222
454.58
559. 100.2001 121335
558.261.25.4
Pendulum, swingup
26.23.11.88
83.88 10.55
47.20
236.81 186.81150..2 1.50.33
Reacher, easy
184.10+4.00 88
981.78+0.60
695.03+236..9 91
982.58#1.39
Reacher, hard
854.761311
864.55.1556.37
752.55.13.55
872.46.51.88
DMControl
605.98
611.87
502.34
673.74 74 (111%)
XArm, reach xArm, push
15=5
71+2
21
80+21
5815
52110
5515
73.5
XAm,, peg in box xArm, hammer xArm
0+0
0+0
1=2
5+8
817
01.
10
81.7
20
31
24
42 (1110%)
Adroit, door Adroit, hammer
010
10:5
040
66:17
25
31
28=02
43+18
Adroit, pen Adroit
20
20
18:14
26:12
15
20
45 (1200%)
Table 4: Results in shaking view. The best method on each task is in bold.
Shaking view
TD-MPC
DM
IDM+STN
MoVie
Cheetah, run
381.04+39.3
317 317.66=9.57
212.31+19.74
493.54
Walker, walk
662.13
627.77 122144
340.481366 136.92
835.99 14.99
Walker, stand
834.70+1.77
604.55+16:20
471.30=#69.19
687.96=9.11
Walker, run
251.58+13.00
186.83+3.75
128.31++9.0
291.39+7.0
Cup, catch
752.861141.81
835. 133.91
648.75.115
951. 20 12 01
Finger, spin
89.5511.772
88. 88.56_5.86 5.86
145.66 145.688122 6:2.93
284.00 05 1473.73
Finger, tum-casy
717.60=-40.8 933
255.23350.35
694. 1144.00-91.00 .2.0
Finger, turn-hard
411.41+44.25
383.96+32.00
132.40434.22
629. 204110..7
Pendulum, swingup
23.73:19.24
4..50
27.30
48.23:330.25
Reacher, easy
529.58:165.
225. 0155.08:31000 022
299. 11459.00
771. 11..381150..
Reacher, hard
205.53425.11
54.80433.44
1 60
453. $53.488381..
DMControl
441.79
348.26
291.45
558. 23 (126%)
xArm, reach
7642
7647
8322
88
xArm, push
6413
55410
61
57+13
xArm, peg in box xAmm, hammer xArm
3#2
15:13
3=2
5+5
25:5
28+7
28+12
30=13
42
44
44
45 (17%)
Adroit, door Adroit, hammer Adroit, pen Adroit
1#2
10+0
5=5
83=2
58112
65118
55
70
30.5
30.18
1815
35.00
30
35
26
63 (1110%)
disparity to two factors: a) the inherent complexity of robotic manipulation tasks and b) the pressing need for effective view generalization in this domain.
Challenges in handling shaking view. Despite improvements of MoVie in various settings, we have identified relative weakness in addressing the shaking view scenario. For instance, xArm tasks, the success rate of MoVie is only 45%. which is close to the 42% success rate of TD-MPC without adaptation. Other baselines such as DM and IDM+STN also experience performance drops. We acknowledge the inherent difficulty of the shaking view scenario, while it is worth noting that in real-world robotic manipulation applications, cameras often exhibit smoother movements or are positioned in fixed views, partially mitigating the impact of shaking view.
7

---

Table 5: Novel FOV. The best method on each task is in bold.
Novel FOV
TD-MPC
DM
IDM+STN
MoVie
Cheetah, run
128.5516.57
379.01
299.02.s8.44
532. $32.944119 19.774
Walker, walk
239. 199.544129.1
182.21=12.75
579.58=47..8
920. 1.0.188=.70
Walker, stand
611.70121.21
732.07
475.466555.2
785.52
Walker, run
184.28=1.888
337 137.28+11.10
295.99=10.00
Cup, catch
940.201224
912.701:34.
16418331
973.60 60
973.
Finger, spin
675.81 #4.99
656.63=1.54
414.83=79.00
917.21+1.71
Finger, turn-easy
792.26.144 2412261115..0
690.61 11.0..88
805.631154.0
845.35 135.52
Finger, turn-hard
591.43
393.18#: 193.18#118..
449 18:1181.99
640. 21
Pendulum, swingup
216.90 216.90177.55
184.33 183.99
253.36177...
699 10.66
Reacher, casy
929.40++2.33
931.56+12.00 69
937.20==45.00 97
971.48=11.00
Reacher, hard
424.11
421.56194.08
287.96
707.75 18..05
DMControl
527.47
613.74
542.43
770.56 (716%)
xArm, reach
53=12
98+2
8545
100=0
xArm, push
60
7115
6015
81.22
xArm, peg box XArm, hammer xArm
20+5
13=10
6=11
63+11
312
6,15
812
55.15
34
47
40
75 (+121%)
Adroit, door
1+2
58=10
3+5
81:12
Adroit, hammer
6617
75118
76110
81.10
Adroit, pen Adroit
26=10
20+8
15#5
41+7
31
51
31
68 (1119%)
Effective adaptation in novel view, moving view, and novel FOV scenarios. In addition to the shaking view setting, MoVie consistently outperforms TD-MPC without adaptation by 4x in robotic manipulation tasks across the other three settings. It is worth noting that these three settings are among the most common scenarios encountered in real-world applications.
Real-world implications. Our findings have important implications for real-world deployment of robots. Previous methods, relying on domain randomization and extensive data augmentation during training. often hinder the learning process. Our proposed method enables direct deployment of offline or simulation- traineed agents, improving success rates with minimal interactions.
5.3 Ablations
To validate the rationale behind several choices of MoVie and our test platform, we perform comprehensive set of ablation experiments.
Integration of STN with low-level featuues. To enhance view generalization, we incorporate two STN blocks [13], following the image observation and the initial convolutional layer of the feature encoder. This integration is intended align the low-level features with the training view, thereby preserving the similarity of deep semantic features to the training view. As shown in Table 6, by progressively adding more layers to the feature encoder, we observe that deeper layers not provide significant additional benefits, supporting our intuition for view generalization.
Table 6: Ablation on applying different numbers of STNs from shallow to deep. The best method on each setting is in bold.
Cheetah-run
2 (MoVie)
3
4
Novel View
254.424 13.89
259.641132.01
342.39 :54.95
309.25
133.19
327.30 1
Moving View 344.7017.54
360.29413.66
365.22 225.222:42.6
361.28
357 $57.29441.22
Shaking View 317.66=9.57
491.96+22007
493.54=55.50
454.82+19.57
472.5748.10
Novel FOV
379.01
512.69
5.32.94:5 19.74
505.65
492.79
Different novel views. We classify novel views into three levels of difficulty, with our main experiments employing the medium difficulty level by default. Table 7 presents additional results for

---

the easy and hard difficulty levels. As the difficulty level increases, we observe consistent decrease in the performance of all the methods.
Table Ablation on different novel views. The best method on each setting is in bold.
Cheetah-run
TD-MPC
DM
IDM+STN
MoVie
Easy Medium
265.90
141.61+1100
349.86
466.67 =58.99
90.13+20.38
254.43+13.8 89
127. 16
342. 554.99951..8
Hard
48.64+13.22 20
112.39=8.36
59.6445.35
215.36:162.0 08
Table 8: Ablation on IDM with and without STN. The best method is in bold.
Task
Setting
IDM
IDM+STN
Novel view
40+5
60+5
Moving view
4642
5515
xArm, push
Shaking view
60+10
61:5
Novel FOV
5812
6015
All settings
51
59
Novel view
615
8
Moving view
510
1015
xArm, hammer
Shaking view
35+8
28+12
Novel FOV
6+7
8+2
All settings
13
14
Novel view
867.87+3.14
932. 25-2.34
Moving view
912.85+10.22
915.1617.31
Cup, catch
Shaking view
435.40+87.32
648. 71.77777
Novel FOV
815.27134.61
964.833 10..45
All settings
757.84
865.37
Novel view
177. 177.26=2.49
141.03
Moving view
332.90+0.2
Finger, spin
Shaking view
71
145.68 +.5.6882.93
Novel FOV
311.66118.5
414.83.17..2
All settings
232.13
301.86
Novel view
111.73+18.99
127.76:11.16
Moving view
205.19125.0
199.48
Cheetah, run
Shaking view
210.06+27.99
212.32+19.74
Novel FOV
262.76:12.117
389.03318844
All settings
197.43
209.45
The efficacy of STN in conjunction with IDM. Curious readers might be concerned about our direct utilization of IDM+STN in the main experiments. suggesting that STN could potentially be detrimental to IDM. However, Table shows that STN not only benefits our method but also improves the performance of IDM, thereby demonstrating effectiveness of SAE for adaptation of the representation and validating our baseline selection.
Finetune or freeze the DM. In our approach, we employ the frozen DM form of supervision to guide the adaptation process of encoder. However, it remains unverified the readers whether end-to-end finetuning of both the DM and the encoder would yield similar benefits. The results presented in Table demonstrate that simplistic end-to-end finetuning does not outperform MoVie, thereby reinforcing the positive results obtained by MoVic.

---

Table 9: Ablation on whether to finetune the DM. The best method is in bold.
Task
Setting
Finetune DM
MoVie
Novel view
51+2
48:7
Moving view
48+7
73:5
xArm, push
Shaking view
4545
57:13
Novel FOV
68+5
81+2
All settings
53
64
Novel view
1015
16112
Moving view
0+0
8
xArm, hammer
Shaking view
545
30+13
Novel FOV
11:2
5515
All settings
6
32
Novel view
648.90425.666
961.98
Moving view
676.05
951.26
Cup, catch
Shaking view
228.20+7..8
951.20
Novel FOV
658.50
073.60#1.94
Al1 settings
552.91
959.51
Novel view
278.57+26.34
892.01#1.20
Moving view
192.95472.76
896.00
Finger, spin
Shaking view
1.50+0.70
284.05
Novel FOV
372.82+51663
917.21
All settings
211.46
747.31
Novel view
273.01+11.2
342.39
Moving view
331.55+:22.22
365 665.22+42.66
Cheetah, run
Shaking view
476.25133025
496.54
Novel FOV All settings
561.94437.78
532.94419.74
410.68
433.52
Conclusion
In this study, we present MoVie, method for adapting visual model-based policies to achieve view generalization. MoVic mainly finetunes spatial adaptive image encoder using the objective of the latent state dynamics model during test time. Notably, we maintain the dynamics model in frozen state, allowing it to function as form of self-supervision. Furthermore, we categorize the view generalization problem into four distinct settings: novel view, moving view, shaking view, and novel FOV. Through systematic evaluation of Mo on 18 tasks across these four settings, totaling 64 different configurations, we demonstrate its general and remarkable effectiveness.
One limitation of our work is the lack real robot experiments, while our focus is on addressing view generalization in robot datasets and deploying visual reinforcement learning agents in real-world scenarios. In our future work, we would evaluate Mo Vie on real-world robotic tasks.
Acknowledgment
This work supported by National Key R&D Program of China (2022/D0161700),
10

---

References
[1] David Bertoin, Adil Zouitine, Mehdi Zouitine, and Emmanuel Rachelson. Look where you look! auliency-guided q q-networks for generalization in visual reinforcement learning. NeurIPS, 2022. 1, 17
[2] Boyuan Chen, Pieter Abbeel, and Deepak Pathak. Unsupervised learning of visual 3d keypoints for control. In ICML, 2021. 3
[3] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. CoRI., 2019. 2,
[4] Danny Driess, Ingmar Schubert, Pete Florence, Yunzhu Li, and Marc Toussaint. Reinforcement learning with neural radiance fields. arXiv, 2022. 3
[5] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. NeurIPS, 2022. 3
[6] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv, 2020. 3
[7] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. ICLR, 2021.
1, 2, 3, 13, 17
[8] Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations. ICLR,
2023. 3,5, 13
[9] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep g-learning with convnets and vision transformers under data augmentation. NewrlPS, 2021. 1, 17
[10] Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In International Conference on Robotics and Automation, 2021. 3
[11] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive
control. ICML, 2022. 3, 5,5,11 13
[12] Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning- from-scratch baseline. ICML, 2023.
[13] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. NeurlPS, 2015. 2, 4,
[14] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. NeurIPS, 2020. 1,3
[15] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: simple technique for generalization in deep reinforcement learning. arXiv preprint arXiv: 1910.0536. 2019. 3,3
[16] Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In CoRL, 2022. 3
[17] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv, 2013. I
[18] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
[19] Xue Bin Peng, Marcin Andrychowiez, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In ICRA, 2018. 2,3
[20] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. RSS, 2018. 2, 3, 5, 13
[21] Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain random- ization via probabilistic inference for robotics simulators. arXiv, 2019. 2,3
11

---

[22] Rutav Shah and Vikash Kumar. Rrl: Resnet representation for reinforcement learning. ICML,
2021.
[23] Jinghuan Shang and Michael S Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In IROS, 2021. 3
[24] Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third-person visual imitation learning via decoupled hierarchical controller. NeurlPS, 2019. 3
[25] Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. ICLR, 2017. 3
[26] Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite-a challenging benchmark for reinforcement learning from pixels. arXiv, 2021.
[27] Yu Sun, Wyatt Ubellacker, Wen-Loong Ma, Xiang Zhang, Changhao Wang, Noel Csomay- Shanklin, Masayoshi Tomizuka, Koushil Sreenath, and Aaron D Ames. Online learning of unknown dynamics for model-based controllers in legged locomotion. RA-L, 2021.
[28] Yu Sun, Xiaolong Wang. Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In JCMI,, 2020. 3
[29] Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In JCML, 2020.
[30] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yuzhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv, 2018 3, 13
[31] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization transferring deep neural networks from simulation the real world.
In IROS, 2017. 3,3
[32] Hsiao-Yo Yu Fish Tung, Zhou Xian, Mihir Prabhudesai, Shamit Lal, and Katerina Fragkiadaki. 3d-oes: Viewpoint-invann innvariatt object-factorized environment simulators. arXiv, 2020. 3
[33] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. NewrIPS, 2020. 1,3
[34] Ruihan Yang, Minghao Zhang, Nicklas Hansen, Huazhe Xu, and Xiaolong Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. ICLR, 2021.
[35] Denis Yarats, Rob Fergus. Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv, 2021. 17
[36] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. NeurlPS, 2022. 3,
17
[37] Yanjie Ze, Nicklas Hansen, Yinbo Chen, Mohit Jain, and Xiaolong Wang. Visual reinforcement learning with self-supervised 3d representations. RA-L, 2023.
12

---

Appendix
A Implementation Details
In this section, we describe the implementation details of our algorithm for training on the training view and test time training in view generalization settings on the DMControl [30], xArm [7], and
Adroit [20] environments. We utilize the official implementation of TD-MPC [11] and MoDem [8] which are available at cattuub.com/nicllsaasseeen
as the model-based reinforcement learning codebase. During training time, we use the default hyperpa- rameters in official implementation of TD-MPC and MoDem. We present relevant hyperparameters during both training and test time in Table 10 and Table 11. One seed of our experiments could be run on single 3090 GPU with fewer than 2GB and it takes 1 hours for test-time training.
and github.com//ie wwoo//aceboooeeeerccc///ccccccee
Training time setup. We train visual model-based policies with TD-MPC DMControl and xArm environments, and MoDem on Adroit environments, We employ identical network architecture and hyperparameters as original TD-MPC and MoDem during training time.
The network architecture of the encoder in original TD-MPC is composed of a stack of 4 convolutional layers, each with 32 filters, no padding, stride of 7 x 7 kernels for the first one, 5 x 5 kernels for the second one and x kernels for all others, yielding a final feature map of dimension 3 x 3 32 (inputs whose framestack is have dimension 84 84 x 9). After the convolutional layers, fully connected layer with an input size of 288 performs a linear transformation on the input and generates 50-dimensional vector the final output.
The network architecture of the encoder in original Modem is composed of a stack of convolutional layers, each with 32 filters, no padding, stride of 2, 7 x 7 kernels for the first one, 5 x 5 kernels for the second one and x kernels for all others, yielding a final feature map of dimension 2 2 32 (inputs whose framestack is have dimension 224 x 224 6). After the convolutional layers, a fully connected layer with an input size of 128 performs a linear transformation on the input and generates 50-dimensional vector the final output.
Test time training setup. During test time, we train spatial adaptive encoder (SAE) to adapt to view changes. We insert STN blocks before and after the first convolutional layer of the original encoders TD-MPC and MoDem. The original encoders are augmented by inserting STN blocks, resulting in the formation of SAF. Particularly, for the STN block inserted before the first convolutional layer, the input is a single frame This means that when the frame stack size is N, N individual frames are fed into this STN block. This is done to apply different transformations to different frames in cases of moving and shaking view.
To update the SAE, we collect online data using buffer with size of 256. For each update, we randomly sample (observation, action, next observation) tuples from the buffer as a batch. The optimization objective is to minimize the loss in predicting the dynamics of the latent states, as defined in Equation
During testing on each task, we run 20 consecutive episodes, although typically only a few or even less than one episode is needed for the test-time training to converge. To make efficient use of the data collected with minimal interactions, we employ a multi-update strategy. After each interaction with the environment, the SAE is updated 32 times.
The following is the network architecture of the first STN block inserted into the encoder of TD-MPC
STN_Block_0.o Booock_O__DMPCCCCC___
(localization): Sequential
By default, each image consists of three channels. Each frame in the
+ observation is treated an independent input to the STN.
(0): : Conv2d (in_chhnnees=3, out_ out_chaanels=,, kernel_size-7, stride=1)
(1) : (aaPooll((Kermee_ssize
stride=4, padding-0)
(2) : ReLU()
(3) : Conv2d (in_cchnnelss8, mtt_channels=10, kernel_size=5, stride=1)
(4) : MaxPool2d (kereelkkeeeeeeeeeeeeeee size=4, stride=4, padding=0)
(5): : ()
(fc_loc): Sequential(
(0): : Linear (inearrin_dee (in_dim=90, out_ 3im=
13

---

(1): ReLU()
(2) Linear Linear(in (in_dim=32, out_dim=6)
The following is the network architecture of the second STN block inserted into the encoder of TD-MPC.
TTM_Block_1_IDHHE
(localization): Sequential(
(0) : (in, channelssa,,,, out out_ channeel-n8, kernel_size=7, stride=1)
(1): : (xPool12(kormee_sizee stride=3, padding-O)
(2) ReLU()
: Conv2d (in_chhneelss out out_ ch_.chaanels-10, kernel_size-6, stride-1)
(4) MaxPool2d (kermelleeeeeeeeeeeee stride=2, padding=0)
(5) : ReLU()
(fc_loc) Sequential
(0) : Linear (in_dim=90, out_dim=32)
(1): : ReLU()
(2): Linear Linear(in_din (in_dim=32, out_dim-6)
The following is the network architecture of the first STN block inserted into the encoder of MoDem. Hr__Blcck_O_Moeeee
(localization): Sequential(
By default, each image consists of three channels. Each frame in the independent input to the STN. (0) : (in2dddin_caaneelsse out pat_channe,s=5, kernel_size=7, stride=2)
+ observation is treated
(1): : BaxPool2a(termee_szzee
stride=4, padding=0)
(2) : ReLU()
(3) : Conv2d (in_ chainneleesss5.. out chtt_caanees-1,,, kernel_size=6, stride-2)
(4) (axPool2dkkerrel__c.o
stride-4, padding-0)
(5): : ()LU()
(fc_loc) Sequential(
(0) Linear (in_ out out_dim=32)
(1): ReLU()
(2): Linear dim out_dim=6)
The following is the network architecture of the second STN block inserted into the encoder of MoDem.
STN_Block_1_MoDer
(Iocalization): Sequential
(0): Conv2d (it_tiiinnnnnnseeeeeeeeeea out nut__chaneel=8, kernel_size=7, stride=2)
(1) (kxermll size=3,, stride=3, padding=0)
: ReLU()
(3) : Conv2d(in (an22e(in_chmmmee-g out_ catt_chanees-100 kernel_size-5, stride=2)
(4) : MaxPool2d (korrel_ieeeeeeeeeeeeeee stride-2, padding-0)
(5) ReLU()
(fc_loc): Sequential(
(0) : (in_dimin out out_dim=32)
(1): ReLU()
(2) Linear iinear(in (in_dim=32, out_dim=6)
B Environment Details
categorize the view generalization problem into four distinct settings: novel view, moving view, shaking view, and novel FOV. In this section, we provide descriptions of the implementation details for
14

---

Table 10: Hyperparameters for training time.
Hyperparameter
Value
Discount factor
0.99
Image size
x 84 (TD-MPC)
224 x 224 (MoDem)
Frame stack
(TD-MPC)
2 (MoDem)
Action repeat
(xArm)
2 (Adroit, Finger, and Walker in DMControl)
4 (otherwise)
Data augmentation
+4 pixel image shifts (TD-MPC)
100 pixel image shifts (MoDem)
Seed steps Replay bulfer size
5000
Unlimited
Sampling technique
PER (a 0.6, 3 0.4)
Planning horizon
Latent dimension
50
Learning rate
le-3 (TD-MPC)
3e-4 (MoDem)
Optimizer (0)
Adam (3, 0.9, 32 0.999)
Batch size
256
Number of demos
(MoDem only)
Table 11: Hyperparameters for test time training.
Hyperparameter
Value
Buffer size
256
Batch size
32
Multi-update times
32
Learning rate for encoder
1c-6 (xAr))
le-7 (otherwise)
Learning rate for STN blocks
le-5
each setting. The detailed camera settings can be referred to in the code of the environments that we are committed to releasing or in the visualization available on our website yangsizhe.githu..o/Me. github.io/Mooi Vic.
Novel view. In this setting, for locomotion tasks (cheetah-run, walker-stand, walker-walk, and walker-run), the camera always faces the moving agent, while for other tasks, the camera always faces a fixed point in the environment. Therefore, as we change the camera position, the camera orientation also changes accordingly.
Moving view. Similar to the previous setting, the camera also always faces the moving agent or fixed point in the environment. The camera position varies continuously.
Shaking view. To simulate camera shake, we applied Gaussian noise to the camera position (XYZ. coordinates in meters) at each time step. For DMControl and Adroit, the mean of the distribution O, the standard deviation is 0.04, and we constrain the noise within the range of -0.07 to +0.07. For xArm, the mean of the distribution is 0, the standard deviation is 0.4, and we constrain the noise within the range -0.07 to +0.07.
Novel FOV. We experiment with larger FOV. For DMControl, we modify the FOV from 45 to 53. For Arm, we modify the FOV from 50 60. For Adroit, we modify the FOV from 45 to 50. We also experiment with a smaller FOV and results are presented in Appendix F.
C Visualization of Feature Maps from Shallow to Deep Layers
We incorporate STN in the first two layers of the visual encoder for all tasks. After visualizing the features of different layers (as shown in Figure 7), found that the features of shallow layers contain
15

---

more information about the spatial relationship. Therefore transforming the features of shallow layers for view generalization is reasonable.
Figure 7: Visualization of features from shallow to deep layers. The features shallow layers contain more information about the spatial relationship.
D Visualization of Feature Map Transformation
We visualize the first layer feature map of the image encoder from TD-MPC and MoVie in Figure 8. It is observed that the feature map from Mo the novel view exhibits closer resemblance that on the training view.
Training View
TD-MPC
Ours
Novel View
Figure 8: Visualization of the first layer feature maps of the original encoder on the training view and on the novel view, and the learned SAE the novel view.
E Extended Description of Baselines
TD-MPC. We test the agent trained training view without any adaptation in the view generalization
settings.
DM. This is derived from MoVie by removing STN blocks, which just adapts encoder during test time.
IDM+STN. This is derived from MoVie by replacing the dynamics model with the inverse dynamics model which predicts the action in between based on the latent states before and after transition. The inverse dynamics model is finctuned together with the encoder and STN blocks during testing.
F Ablation on Different FOVs
In our main experiments, we consider the novel FOV as a FOV larger than the original. In Table 12, present results for both smaller and larger FOV scenarios. Our method demonstrates the successful handling of both cases.
Table 12: Ablation on different FOVs. The best method on each setting is in bold.
Cheetah-run
TD-MPC
DM
IDM+STN
Mo
Small FOV
104.8554..99 128.55=6.67
398.75-17.52 379.01=10.90
75.9218.76 299.02+88.47
530.37
Large FOV
532.94+19.7
16

---

G Comparison with Other Visual RL Generalization Algorithms
In addition to baselines in the main paper, we also compare Mo with state-of-the- methods that focus on visual generalization or data augmentation in visual RL., i.e., DrQ-v2 [35], SVEA [9], PIE-G [36], SGQN [1] and PAD [7]. Results on DMControl environments are shown in Table 13. And the training performance is shown in Table 14. It is observed that MoVie still outperforms these two methods that do not adapt in test time, while PIE-G could also achieve reasonable returns under the view perturbation. This may indicate that fusing the pre-trained networks from PIE-G into MoVie might lead to better results, which we leave as future work. Note that unlike MoVie, other methods need strong data augmentation or modification during training time.
Table 13: Comparison with other visual RL generalization algorithms. The best method is in bold.
Task
Setting
DrQ-22
SVEA
SGQN
PIE-G
PAD
MoVic
Novel
857.05.22.3
743.50170.am
776.26:122.44
834.10.23.20 20
854.51
961.98
Cup, catch
Moving view
911 10:00:1100 6as
343.80
971.082n.30
800.43.10.12
099.34.24.33 31
951. 261.:66
Shaking view
638. 53..9
501 56.66666600 76
943.08= 23
869.80 16.4.66
870.06::21.. 72
951. 20=21.0000
Novel FOV FOV
919.83=12.76
534.36i261.00
803.00+55
927. 16.330
928.8 28.77
973.60 :199
All settings
831.64
635.80
873.35
860.16
838.19
959.51
Novel
518.75
312.38
383.78:6..8
680.10:37.44
233.60::46
01:.01::1
Finger, spin
Moving view
706.91 .9
506-56:1156 156 oo
543.2311..0 .00
628.60.16.80
547 147.98813.2 22
$96.00.21 4s
Shaking view
39. 13
101 67..6666..00 oo
6..40024.. 26
551 83 .2
30 3o
284 1:4.05=z4r5 473.70
Novel FOV
793. 7.
005.03=278.00
553. 253.26=1138
755.86=48
#2.08=2234
917.21
Al1 Bettings
514.77
378.88
412.21
704.09
215.91
747.31
Table 14: The performance of DrQ-v2, SVEA, SGQN, PIE-G, PAD and MoVie under the training view on DMControl tasks.
Task
DrQ-v2
SVEA
SGQN
PIE-G
PAD
MoVic
Cup, catch
153.61+187
971 971.55
970 57
=8.99 881 281.9644.30
974.75 110.26=10.92
980.56
Finger, spin
822. 1=1.02
840.333 33032.s
103.10
985,20=2.25
H Results of Original Models on Training View
The performance of the original agents without any adaptation under the training view is reported in Table 15, 16, and 17 for reference. In the context of view generalization, it is evident that the performance of agents without adaptation significantly deteriorates.
Table 15: Training Result on DMControl.
Task
Cheetah, run
Walker, walk
Walker, stand
Walker, run
Cup, catch
Finger, spin
Reward
658. 10
944.99 Finger, tarnhard
983. 5315544
197.75211.3
980.56=4.39
985.20 2..125
Task
Finger, turn-casy
Pendulum, swingup
Reacher, easy
Reacher, hard
Reward
756. 1661166
616.96=110.4
827.26+61.72
983.8=0.34
937. 937.433=5..59
Table 16: Training Result on xArm.
Task
Reach
Push
Peg in box
Hammer
Success rate (%)
9645
90+17
80+10
83+20
Table 17: Training Result on Adroit.
Task
Door
Hammer
Pen
Success rate (%)
9641
78
48415
17


Reviews:
[{"review_id": "tPkwj1nGoB", "reviewer": "Reviewer_CADg", "summary": "Adapting policy into new view setup is an important task in RL. This paper presents MoVie, Model-based policies for View generalization to achieve the fast view adaptation of model-based policy. Specifically, they combine spatial transformer networks into the encoder models and train it during test-time using dynamics prediction loss. Despite the simplicity, the experimental results show that the strong performance of MoVie over several benchmarks on various test set. ", "questions": "See weakness section. ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is simple yet provide stable performance gain over various methods. Experimental validation is extensive enough. \n2. The paper is well written and easy to follow. ", "weaknesses": "1. Technical novelty and technical depth is limited. I might decrease scores if I found works that combine STN with test-time training. \n\nBesides, few discussions are made on the choice of test-time loss and entire architecture. For example, the experimental results suggest that the DM is better than IDM especially when combining test-time training, but the paper lacks discussion on why it is. Besides, I agree prediction loss is one of the natural loss to train STN, but I think it is also possible to extend the method to model-free methods by using some self-supervised loss function (like contrastive learning). \n\n2. I think the experimental validation is already comprehensive, but it lacks qualitative analysis of their representation before/after adaptation. \n\n3. No theoretical justification for the choice of test-time loss and or entire architecture. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions. We address each of your comments in the following.\n\n**Q1:** The proposed method is simple yet provide stable performance gain over various methods. Experimental validation is extensive enough. But Technical novelty and technical depth is limited. I might decrease scores if I found works that combine STN with test-time training.\n\n**A1:** We thank the reviewer for acknowledging the effectiveness of our proposed approach. We emphasize that besides the method we propose, one main contribution in our work is the **formulation** of the view generalization problem and the resulting test platform, across locomotion tasks and robotic manipulation tasks.  \n\n**Q2:**  Few discussions are made on the choice of test-time loss and entire architecture. For example, the experimental results suggest that the DM is better than IDM especially when combining test-time training, but the paper lacks discussion on why it is.\n\n**A2:**  The main reason we use the dynamic loss during test time is because such a loss is common in model-based RL. Hence, using the same objective during test time leads to consistency between training and test. One might naturally think that IDM could be similar to our DM objective, while there are two reasons that make IDM loss worse than DM under our setting:\n\n1. Model-based RL methods such as TD-MPC and MoDem do not introduce IDM during training originally, which is mainly because introducing IDM loss is not generally helping as shown in Figure 4 of the rebuttal file. This is because additional optimization objectives could heavily affect optimization outcomes.\n\n2. Directly using IDM during test time (as our baseline) results in inconsistency between training and test, which thus leads to suboptimal performance. As shown in Table 1, Table2, Table 3, Table 4 and Table 5 in our main paper, despite being slightly better than MoVie on a few tasks, IDM+STN underperforms MoVie on most tasks.\n\n**Q3:** I agree prediction loss is one of the natural loss to train STN, but I think it is also possible to extend the method to model-free methods by using some self-supervised loss function (like contrastive learning) \n\n**A3:** We believe this is an exciting direction and we made initial attempt toward this direction. Some previous works attempted to include auxiliary tasks during training, such as PAD [1], but it was observed that these auxiliary tasks could negatively impact the training performance of the RL algorithm. We attempted to apply our adaptation method in DrQ-v2 [2] while it is not able to achieve reasonable results as shown in Table 5 of the rebuttal file. In our initial attempt, the view generalization ability has not been greatly improved, and the performance gets even worse after adaptation on some tasks. We believe it requires non-trivial effort to explore how to use more self-supervised losses for model-free RL methods, but we want to point out that this is not trivial to find the suitable loss that does not hurt the training but also helps during test time.\n\n**Q4:** I think the experimental validation is already comprehensive, but it lacks qualitative analysis of their representation before/after adaptation\n\n**A4:** In Appendix C (also in Figure 2 of the rebuttal file), we visualize the features of shallow layers before and after adaptation, and it is observed that features are transformed after adaptation, closer to the training view. This interprets why our method could adapt to different views. \n\n**Q5:** No theoretical justification for the choice of test-time loss and or entire architecture\n\n**A5:**\nThe main focus of our work is to study and solve the view generalization problem in RL, more from an empirical perspective. We also agree about the importance of rigorous theoretical sketch, which could be our future exploration direction.\n\n[1] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. ICLR, 2021. \n\n[2] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.\n"}, {"review_id": "I18XKKMbIV", "reviewer": "Reviewer_CdSm", "summary": "This paper focuses on improving the generalization ability of visual DRL to adapt to unseen views. The authors propose a model-based policy adaptation approach that combines spatial transformer networks with a self-supervised dynamics prediction objective to address this problem. The effectiveness of the approach is evaluated through experiments on three commonly used benchmarks.", "questions": "- The experiments need to be more comprehensive. The comparison baselines used are more like ablation studies rather than state-of-the-art (SOTA) methods, such as TD-MPC (the backbone of MoVie), DM (MoVie without STN), and IDM+STN (MoVie replacing DM with IDM). The authors should consider incorporating more commonly-used or SOTA data augmentation algorithms for visual DRL, such as DrQ-v2[1] and SEVA[2].\n- The authors mention multiple times that only shallow STNs can improve performance, but they fail to provide deeper analysis, such as visual illustrations. Is the number of layers set for all tasks, or do different tasks require their own suitable layers? This information would be valuable.\n- The authors claim that \"Our proposed method enables direct deployment of offline or simulation-trained agents ...\", However, the lack of practical experimental demonstrations, such as robot manipulation, makes their claim unconvincing. I strongly suggest that the authors conduct real-world robot experiments to assess the applicability of their proposed method.\n\nReferences:\n- [1] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.\n- [2] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision 362 transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.", "limitations": "Not Applicable.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and easy to follow. \n- The proposed method is well-motivated. \n- The authors try to solve a realistic and essential problem, i.e., view generalization, especially in real-world or sim-to-real robotic scenarios.", "weaknesses": "- While the problem setting is attractive, the technical contribution is insufficient as a full NeurIPS paper. It is a straightforward combination of multiple existing works, like STN. \n- There needs to be more than the current evaluation to demonstrate the superiority over existing works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions. We address each of your comments in the following.\n\n**Q1:** While the problem setting is attractive, the technical contribution is insufficient as a full NeurIPS paper. It is a straightforward combination of multiple existing works, like STN.\n\n**A1:** We would like to respectfully point out that our method is not just combining simple techniques straightforwardly. Instead, it is well-motivated, aiming to solve a fundamental and realistic problem: *generalize by transforming the unseen test view to the training view*. Though such motivation is simple, it is not trivial to do this since we do not store the training views, and thus we do not have paired images to align the latent space between training views and test views. To solve this, we devise to use a dynamic loss combined with STN, utilizing the consistency in dynamics prediction to align the latent space. \n\nMoreover, one main contribution of our paper is to formulate and study the view generalization problem in RL systematically, across locomotion and manipulation tasks, covering a broad range of tasks. Our proposed method also achieves non-trivial success across all these tasks over strong baselines. \n\nWe hope the reviewer acknowledges the efforts we made in addressing the realistic view generalization problem.\n\n\n**Q2:** There needs to be more than the current evaluation to demonstrate the superiority over existing works.\n\n**A2:** Please see A3 for details.\n\n**Q3:** The experiments need to be more comprehensive. The comparison baselines used are more like ablation studies rather than state-of-the-art (SOTA) methods, such as TD-MPC (the backbone of MoVie), DM (MoVie without STN), and IDM+STN (MoVie replacing DM with IDM). The authors should consider incorporating more commonly-used or SOTA data augmentation algorithms for visual DRL, such as DrQ-v2 [1] and SVEA [2]. \n\n**A3:** Thank you for your suggestion. To further show the advantage of MoVie, we compare MoVie with DrQ-v2 [1], SVEA [2] and PAD [3] on 2 tasks across 4 settings. As shown in Table 1 of the rebuttal file, MoVie outperforms other methods across all these settings. By the way, the training performance of PAD limited the scale of the experiments. We trained these baselines on several tasks, but the performance of PAD can compete with others on only 2 tasks.\n\n**Q4:** The authors mention multiple times that only shallow STNs can improve performance, but they fail to provide deeper analysis, such as visual illustrations. Is the number of layers set for all tasks, or do different tasks require their own suitable layers? This information would be valuable. \n\n**A4:** We incorporate STN in the first two layers of the visual encoder for all tasks. After visualizing the features of different layers (as shown in Figure 1 of the rebuttal file), we found that the features of shallow layers contain more information about the spatial relationship; therefore transforming the features of shallow layers for view generalization is reasonable. \n\n**Q5:** The authors claim that \"Our proposed method enables direct deployment of offline or simulation-trained agents ...\", However, the lack of practical experimental demonstrations, such as robot manipulation, makes their claim unconvincing. I strongly suggest that the authors conduct real-world robot experiments to assess the applicability of their proposed method.\n\n**A5:** We agree that real-world evaluation is valuable. However, simulation remains critical to the community for a number of reasons: (i) it provides researchers with common benchmarks that accurately measure progress in the area, (ii) it facilitates reproducibility and statistically significant results, and (iii) improves equity in the area of  machine learning by removing barriers of entry for researchers to contribute to our collective knowledge. While we agree that reliable real-world benchmarks and evaluations would be valuable and very welcome, it should not be treated as a prerequisite for research on visual generalization due to it still being an open and underexplored problem.\n\n\n\n[1] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.\n\n[2] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision 362 transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.\n\n[3] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. ICLR, 2021. \n"}, {"review_id": "mS9Rg0i4ex", "reviewer": "Reviewer_N1Ux", "summary": "The paper addresses the novel problem of view generalization in reinforcement learning, where an RL agent is trained on an environment with a fixed view and then evaluated on a test environment having the exact same dynamics but observed from a different perspective. In order to address this issue, the authors introduce an innovative method that permits test-time adaptation to the new view. \nThe authors integrate a learnable spatial transformer network (STN) into the feature extraction component of the agent. This augmented encoder is then fine-tuned at test time to generate features that enable a frozen latent dynamic model to predict future state representations. An essential aspect of the authors' approach is that this adaptation does not necessitate any reward signal, which is a key advantage.\nExperimental results demonstrate a significant reduction in the generalization gap presented by a new view across several heterogeneous benchmarks.\n\n\n", "questions": "1. To better understand the importance of the generalization gap, could you provide performance data related to the original view? This information would help quantify the performance decrease when the agent is switched to a different view.\n\n2. The use of inverse dynamics models with spatial transformer networks in your method isn't entirely clear. \nDoes IDM completely replace the DM in the TD-MPC algorithm?\nOr does it just used as a side network during TD-MPC training and used for test adaptation only?\n", "limitations": "The authors mentioned a limitation of this work: the proposed method has not been tested in real-world scenarios, such as with physical robots, which may limit its immediate applicability.\nAnother potential limitation could be that the method has only been tested on model-based reinforcement learning (RL), whereas it would be relatively straightforward to evaluate its effectiveness in model-free RL.\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well-structured and easy to follow, providing clear and detailed explanations that make  believe that I could readily reproduce the method based on the descriptions provided.  \nThe authors introduce a new problem in the field that could stimulate further research.  \nThe proposed method's strength lies in its test-time adaptation capability, which eliminates the need for any reward signal, making it highly applicable to real-world scenarios. This characteristic also suggests that the approach could be integrated into any model-based method, enhancing its universality.  \nAnother advantage is the apparent minimal interaction required with the testing environment, promoting efficiency.  \nThe method is evaluated across various environments, with the results showcasing impressive improvements compared to non-adaptive methods.  \nThe paper's ablation study effectively illustrates the contribution of each component of the method, highlighting the importance of every aspect in achieving the observed results.  Additionally, the annex offers excellent visualizations demonstrating the impact of the spatial transformer on the feature map, providing intuitive understanding of the method's mechanics.\n\n", "weaknesses": "Despite the many strengths of the paper, a few areas could benefit from further development and clarification.\n\n1. It would be beneficial to see a comparison of the performance decrease relative to the original view. The lack of this data makes it challenging to truly gauge the significance of the generalization gap and the efficacy of the proposed method.\n\n2. The scope of comparative studies is somewhat limited. The paper primarily compares with methods not designed for visual adaptation, which may not provide the most insightful comparison. It would have been advantageous to see how the proposed approach stacks up against other strategies specifically intended for visual adaptation.\n\n3. An exploration of whether the proposed method can also benefit model-free algorithms such as Soft Actor-Critic (SAC) is missing. This could broaden the applicability of the findings and provide additional insights into the method's versatility.\n\n4. It remains unclear why the proposed method underperforms with the Inverse Dynamics Model (IDM). A deeper exploration of this anomaly could bolster the robustness and reliability of the approach.\n\n5. Finally, the similarity between the proposed method and the PAD method, which performs test-time domain adaptation in model-free RL, raises questions. The key differences are the use of a Spatial Transformer Network and the leveraging of the latent dynamics model of model-based methods instead of adding it as an auxiliary component. However, these differences, while noteworthy, do not necessarily represent a substantial departure from the PAD method. This similarity begs the question of whether the presented method offers significant novelty or if it is essentially an adjustment of existing approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions. We address each of your comments in the following.\n\n**Q1:** It would be beneficial to see a comparison of the performance decrease relative to the original view. The lack of this data makes it challenging to truly gauge the significance of the generalization gap and the efficacy of the proposed method.  \n\n**A1:** The performance of the original agents without any adaptation under the training view has been reported in Table 13, 14, and 15 in appendix of the original paper. When tested on our view generalization testing platform, the performance of the agent without adaptation experiences a significant decrease.\n\n**Q2:** It would have been advantageous to see how the proposed approach stacks up against other strategies specifically intended for visual adaptation. \n\n**A2:** We appreciate your suggestion on including other visual adaptation methods in baselines. And we compare MoVie with PAD [1] which also adapts visual encoder at test time on 2 tasks across 4 settings (We choose tasks on which PAD performs well at training time, and the addition of auxiliary tasks negatively affects PAD's performance on some other tasks.). The results are provided in Table 1 of the rebuttal file. It is observed that MoVie outperforms PAD significantly across all 8 settings.\n\n**Q3:** An exploration of whether the proposed method can also benefit model-free algorithms such as Soft Actor-Critic (SAC) is missing. This could broaden the applicability of the findings and provide additional insights into the method's versatility.  \n\n**A3:** We believe this is an exciting direction and we made initial attempt toward this direction.  We attempted to apply our adaptation method to DrQ-v2 [2] by integrating a dynamics model and SAE (spatial adaptive encoder) during testing while it is not able to achieve reasonable results. In our initial attempt, the view generalization ability has not been greatly improved, and the performance gets even worse after adaptation on some tasks as shown in Table 5 of the rebuttal file. We believe it requires non-trivial effort to explore how to use more self-supervised losses for model-free RL methods, but we want to point out that this is not trivial to find the suitable loss that does not hurt the training but also helps during test time.\n\n**Q4:** It remains unclear why the proposed method underperforms with the Inverse Dynamics Model (IDM). A deeper exploration of this anomaly could bolster the robustness and reliability of the approach.\n\n**A4:** Intuitively, IDM and DM are somewhat similar, while there are two reasons that make IDM loss worse than DM under our setting:\n\n1. Model-based RL methods such as TD-MPC and MoDem do not introduce IDM during training originally, which is mainly because introducing IDM loss is not generally helping as shown in Figure 4 of the rebuttal file. This is because additional optimization objectives could heavily affect optimization outcomes.\n\n2. Directly using IDM during test time (as our baseline) results in inconsistency between training and test, which thus leads to suboptimal performance. As shown in Table 1, Table2, Table 3, Table 4 and Table 5 in our main paper, despite being slightly better than MoVie on a few tasks, IDM+STN underperforms MoVie on most tasks.\n\n**Q5:** The similarity between the proposed method and the PAD method, which performs test-time domain adaptation in model-free RL, raises questions. This similarity begs the question of whether the presented method offers significant novelty or if it is essentially an adjustment of existing approaches.\n\n**A5:** Although one component of our method is similar to PAD [1], we have made significant improvements: \n\n1. As shown in Figure 4 of the rebuttal file, auxiliary tasks added at training time like PAD could negatively impact the performance of the algorithm. And our method does not require any modification at training time. \n\n2. And we made an effort to explore the prediction loss in model-based RL which can serve as a self-supervised task. We found that the dynamics model in model-based RL is a stable supervision for adaptation. \n\n3. Our method is well-motivated, aiming to solve a fundamental and realistic challenge: *generalize by transforming the unseen test view to the training view*. Since we do not have paired images to align the latent space between training views and test views, we devise to use a dynamic loss combined with STN, utilizing the consistency in dynamics prediction to align the latent space.\n\n4. Furthermore, as you said, we propose spatial adaptive encode (SAE) in order to better adapt to both static and dynamic changes in view.\n\n**Q6:** To better understand the importance of the generalization gap, could you provide performance data related to the original view?  \n\n**A6:** Please see A1 for details.\n\n**Q7:** The use of inverse dynamics models with spatial transformer networks in your method isn't entirely clear. Does IDM completely replace the DM in the TD-MPC algorithm? Or does it just used as a side network during TD-MPC training and used for test adaptation only?\n\n**A7:** The IDM is added and trained at test time and the loss of inverse dynamics is used to optimize IDM and SAE (spatial adaptive encoder) simultaneously. The reason we didn't add IDM as auxiliary tasks during training is that it would negatively impact the training performance as shown in Figure 4 of the rebuttal file. Thanks for your thoughtful question and we will add more details about IDM in the final version.\n\n[1] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. ICLR, 2021. \n\n[2] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.\n\n"}, {"review_id": "TpLzTBDmP8", "reviewer": "Reviewer_E44Y", "summary": "This paper mainly provides a training paradigm. Utilizing the dynamic transition model of the environment during the testing phase as a supervisory signal, STN is used to quickly finetune the mapping of observed potential states, resulting in better performance of the strategy mapping trained on a single view task on unseen test views. This work conducted comparative experiments on three types of tasks and four challenging views, achieving results that surpass existing methods. Ablation experiments were conducted to prove the effectiveness of each module setting.", "questions": "1. Have the training methods of the three baselines in the comparative experiment also been finetuned? How much data was used during finetuning, and is this amount of data acceptable for practical applications?\n2. Can the network of h ^ SAE be directly used in training to achieve learning from multiple perspectives of tasks? Directly generalize from an unseen perspective without finetuning.\n3. The reason why Movie is better than IDM+STN should be explained.", "limitations": "The authors have discussed the limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The training paradigm provided in this article can effectively handle the generalization problem when view changes occur, and a thorough comparative experiment and ablation experiment have been conducted on the proposed model. The core idea of reconstructing mapping h using the environmental dynamic transfer model in the paper is effective.", "weaknesses": "1. The subscripts for o and a in Formula 2 are missing.\n2. There are issues with the baseline selection of IDM+STN. Cheetah run alone cannot prove that IDM+STN is generally better than IDM. It is not a problem to conduct ablation experiments on the Cheetah run model alone, but this cannot be used as sufficient evidence for the superiority of IDM+STN over IDM and thus selecting it as the baseline.\n3. Chapter 3 emphasizes the differences between Formula 2 and Formula 1, on one hand, due to the fixed parameters of Network d, and on the other hand due to the use of SAE in Network h. One of the core methods is to fix the parameters of d, which is intuitive to avoid changes in the target domain of h during the update process, resulting in the failure of the policy pi. However, the ablation experiment showed that the setting of d^* did not achieve a consistently effective effect.\n4. The main approach is to use the dynamic transfer model of the environment under the new view as supervision to quickly train the changed h '. Can you obtain a generalized h directly based on o'?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions. We address each of your comments in the following.\n\n**Q1:** The subscripts for o and a in Formula 2 are missing.\n\n**A1:** Thank you for catching this. We will fix this in the final version.\n\n**Q2:** There are issues with the baseline selection of IDM+STN. Cheetah run alone cannot prove that IDM+STN is generally better than IDM. It is not a problem to conduct ablation experiments on the Cheetah run model alone, but this cannot be used as sufficient evidence for the superiority of IDM+STN over IDM and thus selecting it as the baseline. \n\n**A2:** We appreciate your feedback and acknowledge your concerns regarding our choice of IDM+STN as the baseline. We broaden our experimental setup to include 5 tasks across 4 settings. The results provided in Table 3 of the rebuttal file shows that IDM+STN is generally better than IDM.\n\n**Q3:** Chapter 3 emphasizes the differences between Formula 2 and Formula 1, on one hand, due to the fixed parameters of Network d, and on the other hand due to the use of SAE in Network h. One of the core methods is to fix the parameters of d, which is intuitive to avoid changes in the target domain of h during the update process, resulting in the failure of the policy pi. However, the ablation experiment showed that the setting of d^* did not achieve a consistently effective effect. \n\n**A3:** We agree with the reviewer that the pursuit of a universally optimal representation is critical. But it is inherently challenging for a single approach to excel across all tasks in the field of reinforcement learning. In our experiments, although finetuning the dynamics model slightly outperforms our method in the novel FOV scenario, it underperforms in others. Furthermore, we conducted experiments on more tasks and settings, and the results presented in Table 4 of the rebuttal file demonstrate fixing the dynamics model during adaptation is generally better than finetuning it. \n\n**Q4:** The main approach is to use the dynamic transfer model of the environment under the new view as supervision to quickly train the changed h '. Can you obtain a generalized h directly based on o'?  \n\n**A4:** In order to learn a view invariant representation and directly generalize to an unseen view without finetuning, images from different views are needed. However, as shown in Figure 3 of the rebuttal file, training with multi-view images leads to poor training performance in our attempt. Additionally, accessing images from different views is not easy, especially in the real world. \n\n**Q5:** Have the training methods of the three baselines in the comparative experiment also been finetuned?  How much data was used during finetuning, and is this amount of data acceptable for practical applications?  \n\n**A5:** TD-MPC in the baselines has not been finetuned at test time. DM and IDM+STN have been finetuned for fair comparison. We ran 20 episodes on each of the settings and we found that the performance has been greatly improved after adaptation of one episode in most tasks. An episode consists of tens or hundreds of steps in our robotic tasks, which is quite acceptable for practical applications.\n\n**Q6:** Can the network of h ^ SAE be directly used in training to achieve learning from multiple perspectives of tasks? Directly generalize from an unseen perspective without finetuning.\n\n**A6:** Please see A4 for details.\n\n**Q7:** The reason why Movie is better than IDM+STN should be explained.\n\n**A7:** Intuitively, IDM and DM are somewhat similar, while there are two reasons that make IDM loss worse than DM under our setting:\n\n1. Model-based RL methods such as TD-MPC and MoDem do not introduce IDM during training originally, which is mainly because introducing IDM loss is not generally helping as shown in Figure 4 of the rebuttal file. This is because additional optimization objectives could heavily affect optimization outcomes.\n\n2. Directly using IDM during test time (as our baseline) results in inconsistency between training and test, which thus leads to suboptimal performance. As shown in Table 1, Table2, Table 3, Table 4 and Table 5 in our main paper, despite being slightly better than MoVie on a few tasks, IDM+STN underperforms MoVie on most tasks.\n"}, {"review_id": "FguKP7gipG", "reviewer": "Reviewer_BF5c", "summary": "This work presents an approach to train model-based RL methods such that it generalize to novel views on multiple RL benchmarks. The method leverages classical STN and frozen encoder to benefit the generalization performance.", "questions": "In the experiments, does the baseline methods see all various views in the training sets as well? I.e. is the baseline method in figure 3 also seeing different views?\n\nThere are many other choices to solve this problem. Is contrastive learning / or data augmentations on different views also a way to map different views to the same representations?\n\nI am not sure the application on real-world robot experiments, although some visualizations are presented.", "limitations": "See above.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The method is sound and simple with barely hyperparameters tuning\n\nThe design choices study such as different moving views and the insight are helpful for research community.", "weaknesses": "The writing can be largely improved on section 2 and 3. It is unclear what the model-based RL problem is, and how to define the view generalization, how to map the model to actual control/policy for evaluations, for audience that are not familiar with this line of work.\n\nThe problem is specific and the method novelty are limited.\n\nComparisons with baselines are unclear. At least some comparison with model-free RL and recent work [1] would be helpful.\n[1] Multi-View Masked World Models for Visual Robotic Manipulation, Seo et al., 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions. We address each of your comments in the following.\n\n**Q1:** The writing can be largely improved on section 2 and 3. It is unclear what the model-based RL problem is, and how to define the view generalization, how to map the model to actual control/policy for evaluations, for audience that are not familiar with this line of work.\n\n**A1:** Thank you for your suggestion. We will add more preliminaries about model-based RL in the final version. Model-based reinforcement learning is an approach to solving RL problems where the agent tries to learn a model of the environment it interacts with. Once the agent has a model of the environment, it can utilize optimal control methods, such as Model Predictive Control (MPC) and Monte Carlo Tree Search (MCTS), for planning and control. As for the definition of view generalization, we consider it as the out-of-distribution generalization about camera intrinsic and extrinsic parameters. The visual RL agent is trained on one camera parameter setting and tested on different camera parameter settings.\n\n**Q2:** The method is sound and simple with barely hyperparameters tuning. The problem is specific and the method novelty are limited.\n\n**A2:** We thank the reviewer for acknowledging the conciseness and usability of our proposed approach. We greatly value your feedback.\n\nWe would like to highlight some technical contributions in our method. We made an effort to explore the prediction loss in model-based RL which can serve as a self-supervised task. We found that the dynamics model in model-based RL is a stable supervision for adaptation. We also propose spatial adaptive encode (SAE) in order to better adapt to both static and dynamic changes in view. Furthermore, our method is well-motivated, aiming to solve a fundamental and realistic question: *transforming the unseen test view to the training view*. Since we do not have paired images to align the latent space between training views and test views, we devise to use a dynamic loss combined with STN, utilizing the consistency in dynamics prediction to align the latent space.\n\nWe would also like to emphasize that besides the method we propose, one main contribution in our work is the formulation of the view generalization problem and the resulting test platform, across locomotion tasks and robotic manipulation tasks.\n\n**Q3:** Comparisons with baselines are unclear. At least some comparison with model-free RL and recent work [1] would be helpful. [1] Multi-View Masked World Models for Visual Robotic Manipulation, Seo et al., 2023  \n\n**A3:** Thank you for your suggestion. More comprehensive experiments which incorporate comparison with model-free RL and recent work on visual generalization can better reflect the generalization ability of our method. We compare MoVie with DrQ-v2 [1], SVEA [2] and PAD [3] on 2 tasks across 4 settings. As shown in Table 1 of the rebuttal file, MoVie outperforms other methods across all the settings. As for the recent work [4] that you mentioned, we think that its problem setup is different from ours. It uses multi-view images during training, whereas we focus on out-of-distribution generalization and use only images from single view during training.\n\n**Q4:** In the experiments, does the baseline methods see all various views in the training sets as well? I.e. is the baseline method in figure 3 also seeing different views? \n\n**A4:** No. Our method also does not see various views during training time. All the methods (ours and baselines) are trained with the single fixed view.\n\n**Q5:** There are many other choices to solve this problem. Is contrastive learning / or data augmentations on different views also a way to map different views to the same representations?\n\n**A5:** The solution to view generalization we initially thought of was contrastive learning and data augmentation. In order to learn a view invariant representation, images from different views or strong data augmentation are needed. However, using images from different views (as shown in Figure 3 of the rebuttal file) or strong data augmentation negatively impacts the performance of the RL algorithm [5]. And this is the reason why we abandoned contrastive learning and data augmentation.\n\n**Q6:** I am not sure the application on real-world robot experiments, although some visualizations are presented.\n\n**A6:**  We believe that our manuscript clearly demonstrates that online adaptation to view changes in practical robotic scenarios, covering an extensive range of robotic manipulation tasks to validate its applicability. Our setting is under a practical setting, suitable for both simulation and real world (single view training and any view adaptation). In our future work, we would like to conduct experiments in real world.\n\n[1] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.\n\n[2] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision 362 transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.\n\n[3] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. ICLR, 2021. \t\n\n[4] Multi-View Masked World Models for Visual Robotic Manipulation, Seo et al., 2023  \n\n[5] Kostrikov, Ilya, Denis Yarats, and Rob Fergus. \"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.\" arXiv preprint arXiv:2004.13649 (2020).\n"}]

Author rebuttal:
We thank all the reviewers for their insightful comments and suggestions. 

We are delighted to receive your recognition of the strengths in our work, including but not limited to the meaningful problem formulation, well-motivated and effective method, extensive experimental validation and good writing – “stable performance gain over various methods” “experimental validation is extensive enough” (CADg), “solve a realistic and essential problem” “the paper is well written and easy to follow” (CdSm), “introduce a new problem in the field that could stimulate further research” (N1Ux), “effectively handle the generalization problem when view changes occur” (E44Y), “the method is sound and simple” (BF5c). 

Your suggestions and concerns are also valuable. We have replied separately and conducted extensive additional experiments as support. The experiment results are given in the PDF file. 

**EXP1: Comparison with other generalization or data augmentation algorithms for visual RL** in reply to Reviewer CdSm, Reviewer N1Ux and Reviewer BF5c. Results are given in Table 1. MoVie outperforms other methods including DrQ-v2 [1], SVEA [2] and PAD [3] across all the settings. 

**EXP2: Ablation for baseline selection** in reply to Reviewer E44Y. Results are given in Table 3 and Table 4. We broaden our experimental setup to include 5 tasks across 4 settings. The results show that IDM+STN is generally better than IDM and fixing the dynamics model during adaptation is generally better than finetuning it.

**EXP3: Visualizations of features from shallow to deep layers** in reply to Reviewer CdSm. Results are given in Figure 1. We found that the features of shallow layers contain more information about the spatial relationship, so transforming the features of shallow layers for view generalization is reasonable.

**EXP4: Visualizations of features before and after adaptation** in reply to Reviewer CADg. Results are given in Figure 2. It is observed that features are transformed after adaptation closer to the training view. This interprets why our method could adapt to different views.

**EXP5: Comparison of DrQ-v2 and DrQ-v2 with our adaptation method** in reply to Reviewer CADg and Reviewer N1Ux. Results are given in Table 5. While extending our method to model-free RL is exciting, the view generalization ability has not been greatly improved in our initial attempt. We want to point out that it is not trivial to find the suitable loss that does not hurt the training but also helps during test time.

**EXP6: Training performance of DrQ-v2 using single-view and multi-view images during training** in reply to Reviewer N1Ux and Reviewer E44Y. Results are given in Figure 3. It is observed that training with multi-view images leads to poor training performance.

**EXP7: Training performance of TD-MPC with and without IDM as auxiliary task** in reply to Reviewer N1Ux and Reviewer E44Y. Results are given in Figure 4. It is observed that auxiliary tasks added at training time could negatively impact the performance.

[1] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.

[2] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision 362 transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.

[3] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. ICLR, 2021. 


Meta-review: This submission deals with the view generalization problem in image-based reinforcement leanring. While the overall rating of this submission was fair, it had a few critical feedbacks such as its limited technical contribution and experimental justimation. Due to these reasons, a few reviewers and the AC had a discussion. The conlcusion is that positives, like the simplicity of the method without much hyperparameter tuning and also its interesting problem setup, are worth enough to be shared with other audience in the community. We however strongly suggest the authors to update the submission with more evaluations and results to solidify the paper.
Decision: Accept (poster)