SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Novel FOV. The best method on each task is in bold.
Novel FOV	TD-MPC	DM	IDM+STN	MoVie
Cheetah, run	128.55±6.57	379.01±10.90	299.02±88.47	532.94±19.74
Walker, walk	239.54±129.47	882.21±12.75	579.58±47.87	920.18±8.76
Walker, stand	679.70±21.21	732.07±14.77	678.46±55.32	785.52±24.45
Walker, run	184.28±1.39	337.28±11.40	295.99±10.00	482.03±13.84
Cup, catch	940.20±28.21	912.70±34.54	964.83±10.45	973.60±1.94
Finger, spin	675.81±4.99	886.63±4.54	414.83±79.20	917.21±1.71
Finger, turn-easy	792.26±145.24	690.61±140.88	805.63±54.95	845.35±35.52
Finger, turn-hard	591.43±109.84	393.18±118.38	449.88±121.99	640.21±183.23
Pendulum, swingup	216.90±78.56	184.33±83.99	253.36±78.26	699.90±50.64
Reacher, easy	929.40±12.35	931.56±12.69	937.20±45.97	971.48±11.07
Reacher, hard	424.11±83.74	421.56±94.93	287.96±6.33	707.75±86.05
DMControl	527.47	613.74	542.43	770.56 (↑46%)
xArm, reach	53±12	98±2	85±5	100±0
xArm, push	60±13	71±5	60±5	81±2
xArm, peg in box	20±5	13±10	6±11	63±11
xArm, hammer	3±2	6±5	8±2	55±5
xArm	34	47	40	75 (↑121%)
Adroit, door	1±2	58±10	3±5	81±12
Adroit, hammer	66±7	75±18	76±10	81±10
Adroit, pen	26±10	20±8	15±5	41±7
Adroit	31	51	31	68 (↑119%)
Effective adaptation in novel view, moving view, and novel FOV scenarios. In addition to the shaking view setting, MoVie consistently outperforms TD-MPC without adaptation by  $2 \times 4 \times$  in robotic manipulation tasks across the other three settings. It is worth noting that these three settings are among the most common scenarios encountered in real-world applications.
Real-world implications. Our findings have important implications for real-world deployment of robots. Previous methods, relying on domain randomization and extensive data augmentation during training, often hinder the learning process. Our proposed method enables direct deployment of offline or simulation-trained agents, improving success rates with minimal interactions.
To validate the rationale behind several choices of MoVie and our test platform, we perform a comprehensive set of ablation experiments.
Integration of STN with low-level features. To enhance view generalization, we incorporate two STN blocks [13], following the image observation and the initial convolutional layer of the feature encoder. This integration is intended to align the low-level features with the training view, thereby preserving the similarity of deep semantic features to the training view. As shown in Table 6, by progressively adding more layers to the feature encoder, we observe that deeper layers do not provide significant additional benefits, supporting our intuition for view generalization.
Table 6: Ablation on applying different numbers of STNs from shallow to deep. The best method on each setting is in bold.
Cheetah-run	0	1	2 (MoVie)	3	4
Novel View	254.42±13.89	259.64±32.01	342.39±54.95	309.25±33.19	327.39±10.44
Moving View	344.70±7.54	360.29±13.64	365.22±42.66	361.28±6.78	357.29±31.24
Shaking View	317.66±9.57	491.96±20.07	493.54±56.80	454.82±19.57	472.57±8.49
Novel FOV	379.01±10.90	512.69±17.67	532.94±19.74	505.65±12.02	492.79±21.66
Different novel views. We classify novel views into three levels of difficulty, with our main experiments employing the medium difficulty level by default. Table 7 presents additional results for
the easy and hard difficulty levels. As the difficulty level increases, we observe a consistent decrease in the performance of all the methods.
Table 7: Ablation on different novel views. The best method on each setting is in bold.
Cheetah-run	TD-MPC	DM	IDM+STN	MoVie
Easy	265.90±7.94	441.61±14.00	349.86±55.82	466.67±58.94
Medium	90.13±20.38	254.43±13.89	127.76±16.16	342.39±54.95
Hard	48.64±13.20	112.39±8.36	59.64±5.35	215.36±62.03
Table 8: Ablation on IDM with and without STN. The best method is in bold.
Task	Setting	IDM	IDM+STN
xArm, push	Novel view	40±5	60±5
Moving view	46±2	55±5
Shaking view	60±10	61±5
Novel FOV	58±2	60±5
All settings	51	59
xArm, hammer	Novel view	6±5	8±5
Moving view	5±0	10±5
Shaking view	35±8	28±12
Novel FOV	6±7	8±2
All settings	13	14
Cup, catch	Novel view	867.87±3.14	932.75±23.34
Moving view	912.85±10.22	915.16±7.31
Shaking view	435.40±87.32	648.75±12.77
Novel FOV	815.27±34.61	964.83±10.45
All settings	757.84	865.37
Finger, spin	Novel view	177.26±2.49	141.03±37.52
Moving view	332.90±0.28	532.90±3.25
Shaking view	106.72±7.11	145.68±2.93
Novel FOV	311.66±8.55	414.83±79.20
All settings	232.13	301.86
Cheetah, run	Novel view	111.73±18.98	127.76±16.16
Moving view	205.19±25.45	199.48±25.78
Shaking view	210.06±27.99	212.32±19.74
Novel FOV	262.76±62.17	299.03±88.47
All settings	197.43	209.43
The efficacy of STN in conjunction with IDM. Curious readers might be concerned about our direct utilization of  $\mathrm{IDM + STN}$  in the main experiments, suggesting that STN could potentially be detrimental to IDM. However, Table 8 shows that STN not only benefits our method but also improves the performance of IDM, thereby demonstrating effectiveness of SAE for adaptation of the representation and validating our baseline selection.
Finetune or freeze the DM. In our approach, we employ the frozen DM as a form of supervision to guide the adaptation process of the encoder. However, it remains unverified for the readers whether end-to-end finetuning of both the DM and the encoder would yield similar benefits. The results presented in Table 9 demonstrate that simplistic end-to-end finetuning does not outperform MoVie, thereby reinforcing the positive results obtained by MoVie.
Table 9: Ablation on whether to finetune the DM. The best method is in bold.
Task	Setting	Finetune DM	MoVie
xArm, push	Novel view	51±2	48±7
Moving view	48±7	73±5
Shaking view	45±5	57±13
Novel FOV	68±5	81±2
All settings	53	64
xArm, hammer	Novel view	10±5	36±12
Moving view	0±0	8±7
Shaking view	5±5	30±13
Novel FOV	11±2	55±5
All settings	6	32
Cup, catch	Novel view	648.90±25.668	961.98±2.68
Moving view	676.05±79.69	951.26±10.68
Shaking view	228.20±7.84	951.20±21.01
Novel FOV	658.50±8.48	973.60±1.94
All settings	552.91	959.51
Finger, spin	Novel view	278.57±26.34	892.01±1.20
Moving view	192.95±72.76	896.00±21.65
Shaking view	1.50±0.70	284.05±473.73
Novel FOV	372.82±51.63	917.21±1.71
All settings	211.46	747.31
Cheetah, run	Novel view	273.01±11.29	342.39±54.95
Moving view	331.55±22.22	365.22±42.66
Shaking view	476.25±30.25	493.54±56.80
Novel FOV	561.94±37.73	532.94±19.74
All settings	410.68	433.52
In this study, we present MoVie, a method for adapting visual model-based policies to achieve view generalization. MoVie mainly finetunes a spatial adaptive image encoder using the objective of the latent state dynamics model during test time. Notably, we maintain the dynamics model in a frozen state, allowing it to function as a form of self-supervision. Furthermore, we categorize the view generalization problem into four distinct settings: novel view, moving view, shaking view, and novel FOV. Through a systematic evaluation of MoVie on 18 tasks across these four settings, totaling 64 different configurations, we demonstrate its general and remarkable effectiveness.
One limitation of our work is the lack of real robot experiments, while our focus is on addressing view generalization in robot datasets and deploying visual reinforcement learning agents in real-world scenarios. In our future work, we would evaluate MoVie on real-world robotic tasks.
[22] Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. ICML, 2021. 1
[23] Jinghuan Shang and Michael S Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In IROS, 2021. 3
[24] Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third-person visual imitation learning via decoupled hierarchical controller. NeurIPS, 2019. 3
[25] Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. *ICLR*, 2017. 3
[26] Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite-a challenging benchmark for reinforcement learning from pixels. arXiv, 2021. 3
[27] Yu Sun, Wyatt L Ubellacker, Wen-Loong Ma, Xiang Zhang, Changhao Wang, Noel V Csomay-Shanklin, Masayoshi Tomizuka, Koushil Sreenath, and Aaron D Ames. Online learning of unknown dynamics for model-based controllers in legged locomotion. RA-L, 2021. 3
[28] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020. 3
[29] Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020. 3
[30] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdelmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv, 2018. 2, 3, 5, 13
[31] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017. 2, 3
[32] Hsiao-Yu Fish Tung, Zhou Xian, Mihir Prabhudesai, Shamit Lal, and Katerina Fragkiadaki. 3d-oes: Viewpoint-invariant object-factorized environment simulators. arXiv, 2020. 3
[33] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. NeurIPS, 2020. 1, 3
[34] Ruihan Yang, Minghao Zhang, Nicklas Hansen, Huazhe Xu, and Xiaolong Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. *ICLR*, 2021. 1
[35] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv, 2021. 3, 17
[36] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. NeurIPS, 2022. 1, 3, 17
[37] Yanjie Ze, Nicklas Hansen, Yinbo Chen, Mohit Jain, and Xiaolong Wang. Visual reinforcement learning with self-supervised 3d representations. RA-L, 2023. 3
In this section, we describe the implementation details of our algorithm for training on the training view and test time training in view generalization settings on the DMControl [30], xArm [7], and Adroit [20] environments. We utilize the official implementation of TD-MPC [11] and MoDem [8] which are available at github.com/nicklashansen/tdmpc and github.com/facebookresearch/modem as the model-based reinforcement learning codebase. During training time, we use the default hyperparameters in official implementation of TD-MPC and MoDem. We present relevant hyperparameters during both training and test time in Table 10 and Table 11. One seed of our experiments could be run on a single 3090 GPU with fewer than 2GB and it takes  $\sim 1$  hours for test-time training.
Training time setup. We train visual model-based policies with TD-MPC on DMControl and xArm environments, and MoDem on Adroit environments, We employ identical network architecture and hyperparameters as original TD-MPC and MoDem during training time.
The network architecture of the encoder in original TD-MPC is composed of a stack of 4 convolutional layers, each with 32 filters, no padding, stride of 2,  $7 \times 7$  kernels for the first one,  $5 \times 5$  kernels for the second one and  $3 \times 3$  kernels for all others, yielding a final feature map of dimension  $3 \times 3 \times 32$  (inputs whose framestack is 3 have dimension  $84 \times 84 \times 9$ ). After the convolutional layers, a fully connected layer with an input size of 288 performs a linear transformation on the input and generates a 50-dimensional vector as the final output.
The network architecture of the encoder in original Modem is composed of a stack of 6 convolutional layers, each with 32 filters, no padding, stride of 2,  $7 \times 7$  kernels for the first one,  $5 \times 5$  kernels for the second one and  $3 \times 3$  kernels for all others, yielding a final feature map of dimension  $2 \times 2 \times 32$  (inputs whose framestack is 2 have dimension  $224 \times 224 \times 6$ ). After the convolutional layers, a fully connected layer with an input size of 128 performs a linear transformation on the input and generates a 50-dimensional vector as the final output.
Test time training setup. During test time, we train spatial adaptive encoder (SAE) to adapt to view changes. We insert STN blocks before and after the first convolutional layer of the original encoders in TD-MPC and MoDem. The original encoders are augmented by inserting STN blocks, resulting in the formation of SAE. Particularly, for the STN block inserted before the first convolutional layer, the input is a single frame. This means that when the frame stack size is N, N individual frames are fed into this STN block. This is done to apply different transformations to different frames in cases of moving and shaking view.
To update the SAE, we collect online data using a buffer with a size of 256. For each update, we randomly sample 32 (observation, action, next Observation) tuples from the buffer as a batch. The optimization objective is to minimize the loss in predicting the dynamics of the latent states, as defined in Equation 2.
During testing on each task, we run 20 consecutive episodes, although typically only a few or even less than one episode is needed for the test-time training to converge. To make efficient use of the data collected with minimal interactions, we employ a multi-update strategy. After each interaction with the environment, the SAE is updated 32 times.
The following is the network architecture of the first STN block inserted into the encoder of TD-MPC.
STN_DBlock_0_TDMPC(
(localization): Sequential(
# By default, each image consists of three channels. Each frame in the
 $\leftrightarrow$  observation is treated as an independent input to the STN.
(0): Conv2d(in_channels=3, out_channels=8, kernel_size=7, stride=1)
(1): MaxPool2d(kernel_size=4, stride=4, padding=0)
(2): ReLU()
(3): Conv2d(in_channels=8, out_channels=10, kernel_size=5, stride=1)
(4): MaxPool2d(kernel_size=4, stride=4, padding=0)
(5): ReLU()
) (fc_location): Sequential(
(0): Linear(in_dim=90, out_dim=32)
(1):ReLU() (2):Linear(in_dim=32,out_dim=6) ）
The following is the network architecture of the second STN block inserted into the encoder of TD-MPC.
STN_Block_1_TDMPC(
(localization): Sequential(
(0): Conv2d(in_channels=32, out_channels=8, kernel_size=7, stride=1)
(1): MaxPool2d(kernel_size=3, stride=3, padding=0)
(2):ReLU()
(3): Conv2d(in_channels=8, out_channels=10, kernel_size=5, stride=1)
(4): MaxPool2d(kernel_size=2, stride=2, padding=0)
(5):ReLU()
)
(fc_location): Sequential(
(0): Linear(in_dim=90, out_dim=32)
(1):ReLU()
(2): Linear(in_dim=32, out_dim=6)
)
The following is the network architecture of the first STN block inserted into the encoder of MoDem.
STN_Dem(
(localization): Sequential(
# By default, each image consists of three channels. Each frame in the
 $\rightarrow$  observation is treated as an independent input to the STN.
(0): Conv2d(in_channels=3, out_channels=5, kernel_size=7, stride=2)
(1): MaxPool2d(kernel_size=4, stride=4, padding=0)
(2):ReLU()
(3): Conv2d(in_channels=5, out_channels=10, kernel_size=5, stride=2)
(4): MaxPool2d(kernel_size=4, stride=4, padding=0)
(5):ReLU()
) (fc_location): Sequential(
(0): Linear(in_dim=90, out_dim=32)
(1):ReLU()
(2): Linear(in_dim=32, out_dim=6)
)
The following is the network architecture of the second STN block inserted into the encoder of MoDem.
STN_Block_1_MoDem(
(localization): Sequential(
(0): Conv2d(in_channels=32, out_channels=8, kernel_size=7, stride=2)
(1): MaxPool2d(kernel_size=3, stride=3, padding=0)
(2):ReLU()
(3): Conv2d(in_channels=8, out_channels=10, kernel_size=5, stride=2)
(4): MaxPool2d(kernel_size=2, stride=2, padding=0)
(5):ReLU()
)
(fc_location): Sequential(
(0): Linear(in_dim=90, out_dim=32)
(1):ReLU()
(2): Linear(in_dim=32, out_dim=6)
)
We categorize the view generalization problem into four distinct settings: novel view, moving view, shaking view, and novel FOV. In this section, we provide descriptions of the implementation details for
Table 10: Hyperparameters for training time.
Hyperparameter	Value
Discount factor	0.99
Image size	84 × 84 (TD-MPC)
224 × 224 (MoDem)
Frame stack	3 (TD-MPC)
2 (MoDem)
Action repeat	1 (xArm)
2 (Adroit, Finger, and Walker in DMControl)
4 (otherwise)
Data augmentation	±4 pixel image shifts (TD-MPC)
±10 pixel image shifts (MoDem)
Seed steps	5000
Replay buffer size	Unlimited
Sampling technique	PER (α = 0.6, β = 0.4)
Planning horizon	5
Latent dimension	50
Learning rate	1e-3 (TD-MPC)
3e-4 (MoDem)
Optimizer (θ)	Adam (β1 = 0.9, β2 = 0.999)
Batch size	256
Number of demos	5 (MoDem only)
Table 11: Hyperparameters for test time training.
Hyperparameter	Value
Buffer size	256
Batch size	32
Multi-update times	32
Learning rate for encoder	1e-6 (xArm)
1e-7 (otherwise)
Learning rate for STN blocks	1e-5
each setting. The detailed camera settings can be referred to in the code of the environments that we are committed to releasing or in the visualization available on our website yangsizhe.github.io/MoVie.
Novel view. In this setting, for locomotion tasks (cheetah-run, walker-stand, walker-walk, and walker-run), the camera always faces the moving agent, while for other tasks, the camera always faces a fixed point in the environment. Therefore, as we change the camera position, the camera orientation also changes accordingly.
Moving view. Similar to the previous setting, the camera also always faces the moving agent or a fixed point in the environment. The camera position varies continuously.
Shaking view. To simulate camera shake, we applied Gaussian noise to the camera position (XYZ coordinates in meters) at each time step. For DMControl and Adroit, the mean of the distribution is 0, the standard deviation is 0.04, and we constrain the noise within the range of -0.07 to +0.07. For xArm, the mean of the distribution is 0, the standard deviation is 0.4, and we constrain the noise within the range of -0.07 to +0.07.
Novel FOV. We experiment with a larger FOV. For DMControl, we modify the FOV from 45 to 53. For xArm, we modify the FOV from 50 to 60. For Adroit, we modify the FOV from 45 to 50. We also experiment with a smaller FOV and results are presented in Appendix F.
C Visualization of Feature Maps from Shallow to Deep Layers
We incorporate STN in the first two layers of the visual encoder for all tasks. After visualizing the features of different layers (as shown in Figure 7), we found that the features of shallow layers contain
more information about the spatial relationship. Therefore transforming the features of shallow layers for view generalization is reasonable.
Figure 7: Visualization of features from shallow to deep layers. The features of shallow layers contain more information about the spatial relationship.
D Visualization of Feature Map Transformation
We visualize the first layer feature map of the image encoder from TD-MPC and MoVie in Figure 8. It is observed that the feature map from MoVie on the novel view exhibits a closer resemblance to that on the training view.
Figure 8: Visualization of the first layer feature maps of the original encoder on the training view and on the novel view, and the learned SAE on the novel view.
E Extended Description of Baselines
TD-MPC. We test the agent trained on training view without any adaptation in the view generalization settings.
DM. This is derived from MoVie by removing STN blocks, which just adapts encoder during test time.
IDM+STN. This is derived from MoVie by replacing the dynamics model with the inverse dynamics model which predicts the action in between based on the latent states before and after transition. The inverse dynamics model is finetuned together with the encoder and STN blocks during testing.
F Ablation on Different FOVs
In our main experiments, we consider the novel FOV as a FOV larger than the original. In Table 12, we present results for both smaller and larger FOV scenarios. Our method demonstrates the successful handling of both cases.
Table 12: Ablation on different FOVs. The best method on each setting is in bold.
Cheetah-run	TD-MPC	DM	IDM+STN	MoVie
Small FOV	104.85±4.59	398.75±17.52	75.92±8.76	530.37±12.84
Large FOV	128.55±6.57	379.01±10.90	299.02±88.47	532.94±19.74
G Comparison with Other Visual RL Generalization Algorithms
In addition to baselines in the main paper, we also compare MoVie with state-of-the-art methods that focus on visual generalization or data augmentation in visual RL, i.e., DrQ-v2 [35], SVEA [9], PIE-G [36], SGQN [1] and PAD [7]. Results on DMControl environments are shown in Table 13. And the training performance is shown in Table 14. It is observed that MoVie still outperforms these two methods that do not adapt in test time, while PIE-G could also achieve reasonable returns under the view perturbation. This may indicate that fusing the pre-trained networks from PIE-G into MoVie might lead to better results, which we leave as future work. Note that unlike MoVie, other methods need strong data augmentation or modification during training time.
Table 13: Comparison with other visual RL generalization algorithms. The best method is in bold.
Task	Setting	DrQ-v2	SVEA	SGQN	PIE-G	PAD	MoVie
Cup, catch	Novel view	857.05±27.27	743.50±70.973	776.26±123.63	834.10±23.20	854.51±150.78	961.98±2.68
Moving view	911.20±1.626	843.80±28.17	971.08±9.30	809.43±19.12	699.34±24.31	951.26±10.68
Shaking view	638.51±39.98	501.56±66.76	943.08±7.23	869.86±14.93	870.06±21.72	951.20±21.01
Novel FOV	919.83±12.76	534.36±61.09	803.00±55.15	927.26±15.36	928.87±28.75	973.60±1.94
All settings	831.64	655.80	873.35	860.16	838.19	959.51
Finger, spin	Novel view	518.75±14.63	312.38±117.76	383.78±6.82	680.10±37.47	233.60±48.93	892.01±1.20
Moving view	706.91±7.69	596.56±156.00	543.23±1.00	828.60±16.47	547.98±3.22	896.00±21.65
Shaking view	39.13±6.37	101.56±67.03	168.60±4.26	551.83±7.52	209.30±7.30	284.05±473.73
Novel FOV	793.70±0.65	505.03±278.99	553.26±1.38	755.86±48.25	82.08±23.4	917.21±1.71
All settings	514.77	378.88	412.21	704.09	215.91	747.31
Table 14: The performance of DrQ-v2, SVEA, SGQN, PIE-G, PAD and MoVie under the training view on 2 DMControl tasks.
Task	DrQ-v2	SVEA	SGQN	PIE-G	PAD	MoVie
Cup, catch	953.61±1.57	971.55±0.06	970.58±3.72	959.98±8.92	974.75±5.33	980.56±4.33
Finger, spin	822.4±1.02	840.33±32.08	603.10±1.17	884.96±4.30	710.26±10.92	985.20±2.25
H Results of Original Models on Training View
The performance of the original agents without any adaptation under the training view is reported in Table 15, 16, and 17 for reference. In the context of view generalization, it is evident that the performance of agents without adaptation significantly deteriorates.
Table 15: Training Result on DMControl.
Task	Cheetah, run	Walker, walk	Walker, stand	Walker, run	Cup, catch	Finger, spin
Reward	658.10±9.98	944.99±21.71	983.53±5.34	697.75±11.35	980.56±4.33	985.20±2.25
Task	Finger, turn-easy	Finger, turn-hard	Pendulum, swingup	Reacher, easy	Reacher, hard
Reward	756.16±150.76	616.96±149.44	827.26±61.72	983.8±0.34	937.43±54.59
Table 16: Training Result on xArm.
Task	Reach	Push	Peg in box	Hammer
Success rate (%)	96±5	90±17	80±10	83±20
Table 17: Training Result on Adroit.
Task	Door	Hammer	Pen
Success rate (%)	96±3	78±7	48±15