SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) Ablation on momentum  $m \in \{0, 0.1, 0.01, 0.001\}$ .
(c) Compare our minimal adaptation in Stage 2 and the full adaptation.
Figure 6: Ablation experiments. We ablate each component of H-InDex and show that each individual part effectively combines to contribute to the overall effectiveness of H-InDex.
Stage 2: adapt  $0.18\%$  params (ours)
Hours) Stage 2: adapt  $100\%$  parameters
Figure 7: Ablation on Stage 2 (adapting  $0.18\%$  parameters or adapting  $100\%$  parameters). We observe that simply finetuning the entire visual representation would lead to sub-optimal results. Instead, H-InDex only adapts the parameters in BatchNorm layers and effectively solves all the tasks.
Figure 9 provides more ablation results on Stage 3, further supporting the necessity of updating the BatchNorm layers during the training of RL agents.
Adapting  $100\%$  parameters vs. adapting  $0.18\%$  parameters in Stage 2. In Stage 2 of H-InDex, we intentionally chose to adapt only  $0.18\%$  of the parameters (the affine transformations in BatchNorm layers) in the pre-trained representation. This decision was made to address a specific concern, as depicted in Figure 6c and Figure 7. By altering only the setting for Stage 2 while keeping all other factors constant, we observed that across all tasks, adapting all parameters is not more advantageous. This phenomenon may be attributed to the fact that freezing and finetuning partial parameters help mitigate catastrophic forgetting, which is often caused by full finetuning [1]. In Figure 8, we also show the necessity of our Stage 2. We could conclude from results that correctly finetuning the visual representation is one key to the stable convergence, and not correctly finetuning the model, such as finetuning all the parameters, could be even worse than the frozen model.
Robust visual generalization. One concern of our hand representation is its generalization ability, compared to the vision model pre-trained on large-scale datasets, such as VC-1 [17]. Therefore, we change the background of the training scene to various novel backgrounds, as shown in Figure 12 (see Appendix C) and evaluate VC-1 and H-InDex on the task relocate potted meat can. The
Figure 8: Ablation on Stage 2 (with or without adaptation). We also conduct more experiments to show the necessity of Stage 2. We could observe a consistent improvement across these tasks by applying Stage 2, which only adapts  $0.18\%$  parameters of the visual representation.
Stage 2: adapt  $0.18\%$  params (ours) w/o. Stage 2
Stage 3:  $m > 0$  Stage 3:  $m = 0$
Figure 9: Ablation on Stage 3 (momentum  $\mathbf{m} > 0$  or  $\mathbf{m} = 0$ ). We observe that our Stage 3 contributes greatly to some specific tasks, such as relocate mustard bottle, and for some tasks like hammer, tuning this parameter only results in a slightly faster convergence. For tasks not shown here, we all use  $m = 0$ , since H-InDex with Stage 1 and Stage 2 has been strong enough.
results given in Table 3 (see Appendix C) show that H-InDex could handle the changed background better than VC-1. We also see the consistent performance drop across all scenes, emphasizing the importance of visual generalization.
Visualization of self-supervised keypoint detection in Stage 2. In Stage 2, a self-supervised keypoint detection objective is employed to fine-tune a minimal percentage  $(0.18\%)$  of parameters in the pre-trained model. The visualization results, as shown in Figure 10, demonstrate the successful detection of keypoints. This observation highlights the pre-trained model's capability to effectively allocate attention to the hand and objects depicted in the images, even with the adaptation of only a small subset of parameters.
Visualization of affine transformations adaptation in Stage 2. The significance of Stage 2 in H-InDex is evident from Figure 6a. To gain deeper insights into this phenomenon, we visualize the distribution of the adapted parameters, specifically the affine transformations in BatchNorm layers. We accomplish this by fitting a Gaussian distribution. Figure 11 presents the visualization results, highlighting an interesting trend. For the shallow layers, the distributions of the adapted models closely resemble those of the pre-trained models. However, as we move deeper into the layers, noticeable differences emerge. We attribute this disparity to the fact that dissimilarities between human hands and robot hands extend beyond low-level features like color and texture. Instead,
Figure 10: Visualization of our self-supervised keypoint detection. We select four tasks here and mark the detected keypoints on images with  $\star$ red stars. We observe that these keypoints consistently mark the dynamic regions of images. Full videos are available on yanjieze.com/H-InDex.
Figure 11: Visualization of affine transformation adaptation in Stage 2. We fit a Gaussian distribution for parameters of affine transformations. We omit the X-axis and Y-axis here for simplicity, Human represents the original pre-trained model and robot represents the adapted representation in different tasks. It is observed that deep layers are subjected to large distribution shifts.
they encompass higher-level features such as dynamics and structure [2, 31, 34]. This observation underscores the importance of our adaptation approach, as it effectively addresses the variations in both low-level and high-level features, facilitating the success of H-InDex.
In this study, we introduce H-InDex, a visual reinforcement learning framework that leverages hand-informed visual representations to tackle complex dexterous manipulation tasks effectively. H-InDex outperforms other recent state-of-the-art representations in a range of 12 tasks, including six kinds of manipulation skills. The effectiveness of H-InDex can be attributed to its three-stage approach, wherein Stage 1 incorporates a pre-trained 3D human hand representation and Stage 2 and Stage 3 focus on careful in-domain adaptation with only  $0.36\%$  parameters updated. These stages collectively contribute to the successful preservation and utilization of the human hand prior knowledge.
It is also important to acknowledge some limitations of our work. We did not investigate the generalization capabilities of H-InDex, particularly in scenarios involving the grasping of novel objects. Our future work aims to address this limitation and enhance H-InDex's generalization capabilities for real-world applications. In addition, we find that our Stage 3 is surprisingly effective in some specific tasks like relocate mustard bottle, while we have not given a theoretical understanding of such phenomena. We consider this problem as a possible future direction.
This work is supported by National Key R&D Program of China (2022ZD0161700).
[21] Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep reinforcement learning for dexterous manipulation. arXiv, 2017. 2
[22] Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexamv: Imitation learning for dexterous manipulation from human videos. In ECCV, 2022. 2, 3, 6, 13
[23] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. RSS, 2018. 2, 3, 6, 13
[24] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv, 2020. 1, 3, 4, 7
[25] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In ICRA, 2018. 2, 7
[26] Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. ICML, 2021. 1, 2, 3, 6, 7, 13, 15, 16
[27] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. CoRL, 2022. 3
[28] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: learning a robotic hand imitator by watching humans on youtube. RSS, 2022. 3
[29] Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vrl3: A data-driven framework for visual deep reinforcement learning. arXiv, 2022. 2, 3
[30] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv, 2022. 1, 2, 3, 4, 6, 15
[31] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015. 10
[32] Yanjie Ze, Nicklas Hansen, Yinbo Chen, Mohit Jain, and Xiaolong Wang. Visual reinforcement learning with self-supervised 3d representations. RA-L, 2023. 2
[33] Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. CoRL, 2023. 2
[34] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818-833. Springer, 2014. 10
Codebase. Our major codebase is built upon the official implementation of RRL [26], which is publicly available on https://github.com/facebookresearch/RRL and includes the Adroit manipulation tasks [23]. The DexMV [22] tasks are from the official code https://github.com/yzqin/dexmv-sim. All the visual representations in our work are also available online, including RRL (pre-trained ResNet-50, provided in PyTorch officially), R3M (https://github.com/facebookresearch/r3m), MVP (https://github.com/ir413/mvp), VC-1 (https://github.com/facebookresearch/eai-vc), and FrankMocap (https://github.com/facebookresearch/frankmocap). This ensures the good reproducibility of our work. We are also committed to releasing the code.
Network architecture for H-InDex. The architecture employed by H-InDex is based on ResNet-50 [10], referred to as  $h_{\theta}$ . In the initial stage (Stage 1),  $h_{\theta}$  takes as input a  $224 \times 224$  RGB image and processes it to generate a compact vector of size 2048. In Stage 2, we modify  $h_{\theta}$  by removing the average pooling layer in the final layer, resulting in the image being decoded into a feature map with dimensions  $7 \times 7 \times 2048$ . Moving on to Stage 3,  $h_{\theta}$  once again produces a compact vector of size 2048, while simultaneously updating the statistics within the BatchNorm layers using the exponential moving average operation.
Implementation details for Stage 2. Our implementation strictly follows the previous work that also uses the self-supervised keypoint detection as objective [11, 14, 15]. We give a PyTorch-style overview of the learning pipeline below and refer to [11] for more implementation details. Notably, the visual representation  $h_{\theta}$  (24M) contains the majority of parameters, while all other modules in the pipeline maintain a parameter count ranging from 1M to 3M. We use 50 demonstration videos as training data for each task and train 100k iterations to ensure convergence with learning rate  $1 \times 10^{-4}$ . One of our core contributions is to only adapt the parameters in BatchNorm layers in  $h_{\theta}$ , and we emphasize that the learning objective is not our contribution, as it has been well explored in [11, 14, 15].
for_in range(num_iters):
    # sample data
    source_view, target_view = next(data_view) # 3x224x224
    # self-supervised keypoint-based reconstruction
    # h_theta is our visual representation
    feature_map = h_theta(target_view) # -> 7x7x2048
    keypoint_feat = keypointEncoder(keypoint_map) # -> 30x56x56
    keypoint_feat = upSampler(keypoint_feature) # -> 256x28x28
    appearance_feat = appearanceEncoder.source_view) # -> 256x28x28
    target_view_recon = imageDecoder([keypoint_feat, appearance_feat]) # -> 3x224x224
    # compute loss
    loss = perceptual_loss(target_view, target_view_recon)
    # compute gradient and update model
    optimizer.zero_grad()
    loss_backward()
    optimizer.step()
B Task Descriptions
In this section, we briefly introduce our tasks. We use an Adroit dexterous hand for manipulation tasks. The task design follows Adroit [23] and DexMV [22]. Visualizations of task trajectories are available at h-index-rl.github.io.
Hammer (Adroit). It requires the robot hand to pick up the hammer on the table and use the hammer to hit the nail.
Door (Adroit). It requires the robot hand to open the door on the table.
Pen (Adroit). It requires the robot hand to orient the pen to the target orientation.
Pour (DexMV). It requires the robot hand to reach the mug and pour the particles inside into a container.
Place inside (DexMV). It requires the robot hand to place the object on the table into the mug.
Relocate YCB objects [3] (DexMV). It requires the robot hand to pick up the object on the table to the target location. The objects in our tasks include foam brick, box, mug, mustard bottle, tomato soup can, and potted meat can.
C Visual Generalization
One concern of our hand representation is its generalization ability, compared to the vision model pre-trained on large-scale datasets, such as VC-1 [17]. Therefore, we change the background of the training scene to various novel backgrounds, as shown in Figure 12 and evaluate VC-1 and H-InDex on the task relocate potted meat can. The results given in Table 3 show that H-InDex could handle the changed background better than VC-1. We also see the consistent performance drop across all scenes, emphasizing the importance of visual generalization.
Figure 12: Various backgrounds for visual generalization. The first image shows the training scene and the rest 9 images show the novel scene.
Table 1: Scores for generalization to unseen backgrounds on relocate potted meat can task. We evaluate VC-1 and H-InDex with 20 episodes for each seed.
Scene ID / Method	VC-1 [17]	H-InDex
Origin	2391.74±602.83	2240.37±85.45
1	896.28±1006.55	915.95±922.65
2	603.26±920.48	771.28±793.51
3	451.36±839.45	578.42±764.09
4	360.21±772.64	472.66±715.03
5	300.02±718.05	393.32±676.41
6	256.80±673.16	340.07±639.68
7	224.21±635.56	298.20±608.54
8	226.59±610.58	265.82±581.03
9	214.30±581.76	239.60±556.80
Average	392.56	475.04
D Main Experiments (ConvNets Only)
In our primary experimental analysis, we conduct a comprehensive comparison of five visual representations, with three of them being ConvNets, including our method. Figure 13 presents an isolated
demonstration of the comparison among the ConvNets. Notably, our method H-InDex exhibits superior performance in comparison to the other ConvNets.
Figure 13: Episode return for 12 challenging dexterous manipulation tasks. Mean of 3 seeds with seed number 0, 1, 2. Shaded area indicates  $95\%$  CIs.
E Success Rates in Main Experiments
We present the success rates of our six task categories as in Table 3. Regarding the hammer task, it is evident that both H-InDex and VC-1 exhibit success rates near  $100\%$ . However, a notable disparity arises when considering episode returns, indicating the varying degrees of task execution proficiency even among successful agents.
Table 2: Success rates for main experiments. Highest success rates for each task are marked with bold fonts. The success rates only reflect whether the task is finished but do not reflect how fast the task is finished. H-InDex still dominates other methods.
Task name / Method	RRL [26]	R3M [18]	MVP [30]	VC-1 [17]	H-InDex
Hammer (2M)	89±15	24±21	83±11	97±3	100±0
Door (4M)	92±1	99±2	100±0	99±2	96±5
Pen (8M)	78±4	58±6	80±4	81±2	90±2
Pour (8M)	38±33	0±0	23±38	67±29	99±2
Place inside (6M)	70±50	1±2	99±2	98±4	97±6
Relocate large clamp (8M)	33±31	0±0	47±27	23±21	50±45
Relocate foam brick (2M)	87±11	42±37	48±46	44±49	86±10
Relocate box (6M)	94±5	45±24	48±50	49±50	85±14
Relocate mug (2M)	100±0	82±2	54±51	74±44	100±0
Relocate mustard bottle (2M)	100±0	82±8	100±0	99±2	99±2
Relocate tomato soup can (2M)	97±3	18±31	6±10	30±52	99±2
Relocate potted meat can (2M)	97±6	56±15	69±48	88±21	94±5
Average	81.3	42.3	63.1	70.8	91.3
Table 3: Success rates and scores for generalization to unseen backgrounds on relocate potted meat can task. We evaluate VC-1 and H-InDex with 20 episodes for each seed.
Scene ID / Method	VC-1 [17] (success rate)	VC-1 [17] (score)	H-InDex (success rate)	H-InDex (score)
1	38±43	896.28±1006.55	48±48	915.95±922.65
2	26±39	603.26±920.48	32±46	771.28±793.51
3	19±36	451.36±839.45	24±42	578.42±764.09
4	15±33	360.21±772.64	19±39	472.66±715.03
5	13±31	300.02±718.05	16±36	393.32±676.41
6	11±29	256.80±673.16	14±34	340.07±639.68
7	10±27	224.21±635.56	12±32	298.20±608.54
8	10±26	226.59±610.58	11±30	265.82±581.03
9	9±25	214.30±581.76	10±29	239.60±556.80
Average	16.8	392.56	20.7	475.04
We categorize hyperparameters into task-specific ones (Table 4) and task-agnostic ones (Table 5). Across all baselines, all the hyperparameters are shared except the momentum  $m$ , which is only used in our algorithm. All the hyperparameters for policy learning are the same as RRL [26]. This ensures the comparison between different representations is fair.
Our exploration of the momentum  $m$  in Table 4 has been limited to a specific set of values, namely  $\{0,0.1,0.01,0.001\}$ , through the use of a grid search technique, due to the limitation on computation resources. It is observed that carefully tuning  $m$  could take more benefits.
Table 4: Task-specific hyperparameters.
Task name / Variable	Momentum m	Demonstrations	Training steps (M)	Episode length
Hammer	0.1	25	2	200
Door	0.0	25	4	200
Pen	0.0	25	6	100
Pour	0.0	50	8	200
Place inside	0.001	50	8	200
Relocate large clamp	0.01	50	8	100
Relocate foam brick	0.01	25	2	100
Relocate box	0.001	25	6	100
Relocate mug	0.0	25	8	100
Relocate mustard bottle	0.001	25	6	100
Relocate tomato soup can	0.01	25	8	100
Relocate potted meat can	0.0	25	4	100
Table 5: Task-agnostic hyperparameters.
Variable	Value
Dimension of image observations	224 × 224 × 3
Dimension of robot states	30
Dimension of actions	30
Hidden dimensions of policy π	256, 256
BC learning rate	0.001
BC epochs	5
BC batch size	32
RL learning rate	0.001
Number of trajectories for one step	100
VF batch size	64
VF epochs	2
RL step size	0.05
RL gamma	0.995
RL gae	0.97