H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation
Authors: Yanjie Ze
Yuyao Liu
Ruizhe Shi
Jiaxin Qin
Zhecheng Yuan
Jiashun Wang
Huazhe Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Reinforcement Learning
Representation Learning
Dexterous Manipulation
TL;DR: We propose H-InDex, a hand-informed visual representation for dexterous manipulation with reinforcement learning.

Abstract: Human hands possess remarkable dexterity and have long served as a source of inspiration for robotic manipulation. In this work, we propose a human $\textbf{H}$and-$\textbf{In}$formed visual representation learning framework to solve difficult $\textbf{Dex}$terous manipulation tasks ($\textbf{H-InDex}$) with reinforcement learning. Our framework consists of three stages: $\textit{(i)}$ pre-training representations with 3D human hand pose estimation, $\textit{(ii)}$ offline adapting representations with self-supervised keypoint detection, and $\textit{(iii)}$ reinforcement learning with exponential moving average BatchNorm. The last two stages only modify $0.36$% parameters of the pre-trained representation in total, ensuring the knowledge from pre-training is maintained to the full extent. We empirically study $\textbf{12}$ challenging dexterous manipulation tasks and find that $\textbf{H-InDex}$ largely surpasses strong baseline methods and the recent visual foundation models for motor control. Code and videos are available at https://yanjieze.com/H-InDex .

H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation
Yanjie Ze
Yuyao Liu
Ruizhe Shi
Jiaxin Qin Huazhe Xu
Zhecheng Yuan
Jiashun Wang
Shanghai Qi Zhi Institute
TTsinghua University, III *Renmin University of China Carnegie Mellon University "Shangaai Al Lab Equal contribution. Order decided by coin flip.
Shanghai Jiao Tong University
yanjieze.com//-InDDe
Abstract
Human hands possess remarkable dexterity and have long served as a source of inspiration for robotic manipulation. In this work, we propose human Hand- Informed visual representation learning framework to solve difficult Dexterous manipulation tasks (H-InDex) with reinforcement learning. Our framework con- sists of three stages: (i) pre-training representations with human hand pose estimation, (ii) offline adapting representations with self-supervised keypoint detec- tion, and (iii) reinforcement learning with exponential moving average BatchNorm. The last two stages only modify 0.36% parameters of the pre- pre-trained representation in total, ensuring the knowledge from pre-training is maintained to the full extent. We empirically study 12 challenging dexterous manipulation tasks and find that H- InDex largely surpasses strong baseline methods and the recent visual foundation models for motor control. Code is available at yanjieze. Haatiizze.ommH--InDeee
Introduction
Humans can adeptly tackle intricate and novel dexterous manipulation tasks. However, multi- fingered robotic hands still struggle to achieve such dexterity efficiently. Recent progress in representation learning for visuomotor tasks has proved that pre-trained universal represen- tations may accelerate robot learning manipula- tion tasks [17, 18, 26, 30]. In light of previous success, the similar morphology between human hands and robotic hands begs the question: can robotic hands leverage representations learned from human hands for achieving dexterity?
Normalized Average Score
100
98.2
81.4
80.1
75
72.8
50
35.7
25
0
H-InDex (ResNet-50)
VC-1 (VIT-B)
RRL (ResNet-50)
MVP (VIT-S)
R3M (ResNet-50)
ResNus-50
VIT.S
In this paper, we propose Hand- Informed visual reinforcement learning framework for Dexterous manipulation (H-InDex) that uses and adapts visual representations from human hands to boost robotic hand dexterity. Our framework consists of three stages:
Figure 1: Normalized average score for our al-
gorithm H-InDex and the baselines (VC-1 [17], MVP [30], R3M [18], and RRL [26]).
Stage 1: Pre-training representations with 3D human hand pose estimation, where we adopt the
feature encoder from an off-the-shelf 3D hand pose estimator FrankMocap [24].
Offline adapting representations with self-supervised keypoint detection. where freeze the convolutional layers in the pre-trained representation and only finetune the affine
Stage 2:
37th Conference on Neural Information Processing Systems (NcurIPS 2023), New Orleans.

---

transformations in BatchNorm layers (0.18% parameters of the entire model). Such minimal modification of the pre-trained representations ensures that human dexterity is retained to the maximum extent and adapts the human hand representations into the target robotic domain.
Stage 3: Reinforcement learning with exponential moving average (EMA) BatchNorm and the adapted representations. EMA operates to dynamically update the mean and variance in BatchNorm layers, to further make the model adapt to the progressive learning stages.
In contrast to previous works that also learn representations from human videos [17, 18,308, 30], there are two major benefits of our framework: i) H-InDex explicitly learn human dexterity by foreing the model to predict the 3D hand pose instead of predicting or discriminating pixels unsupervisedly using masked auto-encoding [9] or time contrastive learning [25]; ii) H-InDex directly adopts the off-the-shelf visual model that is designed to capture human hands rather than training large models on large-scale datasets for specific robotic tasks. These two points combined demonstrate new cost-effective way to solve robotic tasks such as dexterous manipulation by leveraging existing visual models that are originally and only designed for human understanding.
To show the effectiveness of H-InDex, we experiment on 12 challenging visual dexterous manipulation tasks from Adroit [23] and DexMV [22]. We mainly report episode returns instead of success rates to better show how well the robots solve the tasks. In comparison with several strong visual foundation models for motor control, H-InDex largely surpasses all of them as shown in Figure 1.
To summarize, our contributions are three-fold:
We propose novel visual reinforcement learning framework called H-InDex utilize rich human hand information efficiently for dexterous manipulation.
We show the effectiveness of our framework on 12 challenging visual dexterous manipulation tasks, comparing with recent strong foundation models such as VC-1 [17].
Our study has offered valuable insights into the application of pre-trained models for dexterous manipulation, by exploring the direct application of 3D human hand pose estimation model, originating from the vision community.
2 Related Work
Visual reinforcement learning for dexterous manipulation. Recent research has explored the
use of deep reinforcement learning (RL) for solving dexterous manipulation tasks 17,21,23, 26, 29]. For example, Rajeswaran et al. [23] investigated the use of vector state information as input to the RL algorithm. Despite the success, assuming access to the ground-truth state limits its possibility to be deployed in the real world. RRL [26] finds that ImageNet pre-trained ResNets [10] are surprisingly effective in achieving dexterity with visual observations. Under the umbrella of visual RL, MoDem [7] leverages learned dynamics model to solve the tasks with good utilization of demonstrations. Furthermore, VRL3 [29] utilizes offline RL to pre-train the visual representations and the policies in an end-to-end manner. In this work, H-InDex is designed to focus on visual representations while leaving the policy, training framework, and reward signals unchanged. As a result, H-InDex offers an orthogonal and complementary approach to prior efforts in this area.
Foundation models for visuo-motor control. Given the diversity of robotic tasks and computational constraints, there is a growing interest in developing a single visual foundation model that can serve as general feature extractor. Such model would enable the processing of high-dimensional visual observations into compact vectors, providing a promising approach for efficient and effective control of a wide range of robotic systems [8, 17,19,26, 30, 33. Among them, R3M [18] pre-trains a ResNet-50 on Ego4D [6] dataset and evaluates on several robotic manipulation tasks with imitation learning. MVP [30] pre-trains vision transformers [4] with Masked AutoEncoder (MAE) [9] on internet-scale data, achieving strong results on dexterous manipulation tasks. Similarly, a very recent foundation model VC-1 [17] explores the scaling up of MAE for motor control and achieves consistently strong results across wide range of benchmarks. However, it should be noted that VC-1 and R3M only employ IL to solve dexterous manipulation tasks, making it unclear whether these models are suitable for the setting of reinforcement learning, where agents need to trade off between exploration and exploitation. GNFactor [33] distills the 2D foundation models into 3D space, but their agents are limited to address the gripper- based manipulation problems.

---

Learning dexterity from videos. A growing body of recent research aims to leverage human manipulation videos for improving visuomotor control tasks 20, 28. 303. line of works focuses on directly extracting human hand poses from videos and employing RI/II. to train on the
retargeted robot, joint positions, such as DexMV [22], Robotic telekinesis [28], VideoDex [27], and Imitate Video [20]. In contrast to these approaches, our work explores representation learning approach for leveraging online human-object interaction videos without explicit pose estimation to improve dexterity in robotic manipulation, sharing similar motivation MVP [30] and VC-1 [17].
3 Preliminaries
Formulation. We model the problem as a Markov Decision Process (MDP) M = (S.A.T, where s é are states, a A are actions, T: : S + S is a transition function, r E R are rewards, and y [0,1) is discount factor. The agent's goal is to learn policy Ty that maximizes discounted cumulative rewards on M. i.e., max, R.,, [Z=-0"r:), while using as few interactions with the environment as possible, referred as sample efficiency.
In this work, we focus on visual RL for dexterous manipulation tasks, where actions are high- dimensional (a e aA and ground-truth states are generally unknown, approximated by image observations o O together with robot proprioceptive sensory information q Q, i.e. 8 (o, q). To better address the hard exploration problem in high-dimensional control (7, 26, 29], we assume access to limited number of expert demonstrations Dexpen
D2. D
Demo Augmented Policy Gradient (DAPG) [23] is model-free policy gradient method that utilizes given demonstrations to augment the policy and trains the policy with natural policy gradient (NPG) [13]. DAPG mainly consists of two stages:
(1) Pre-training the policy with behavior cloning, which is to solve the following maximum likellhood problem:
maximize
Σ
In (a) (s).
(1)
(s.a)EDexpet
(2) RL finetuning with the demo augmented loss, which is to add an additional term to the gradient,
9aug Σ
Velnno(a (s,
Σ
Volu (a s)w(s,
(2)
(s,a)ED,
(s.a)e
original gradient
demo augmented gradient
where AT (s,aa) a) is the advantage function, D., represents the dataset obtained by executing policy my on the MDP, and w(s, a) is weighting function.
4 Method
In this work, our goal is to achieve sample efficient visual reinforcement learning agents in dexterous manipulation tasks by incorporating human hand dexterity into visual representations. To this end, we propose Hand-Informed visual reinforcement learning for Dexterous manipulation (H-InDex), simple yet effective learning framework to address the contact-rich dexterous manipulation problems effectively in limited interactions. The overview of our method is provided in Figure 2. H-InDex consists of three stages: 1) a representation pre-training stage where we pre-train the visual rep- resentations with the 3D human hand pose estimation task, aiming to make visual representations understand human hand dexterity from diverse natural videos; 2) a representation offline adaptation stage where we adapt only 0.18% parameters in the pre-trained representation with the sulf-supervieed keypoint objective with in-domain data; 3) a reinforcement learning stage where the visual repre- sentation is frozen and we utilize the exponential moving average operation to update the mean and variance in BatchNorm the visual representations.
Stage 1: Representation pre-training. We start by pre- ve-traiining visual representations with monocular 3D human hand pose estimation, which well-established human hand understanding task in the computer vision community with large-scale annotated datasets available. Together with the datasets, there are a plethora of open-sourced models from which we use an off-the-shell model FrankMocap [24]. FrankMocap is whole-body pose estimation system with the hand module trained
3

---

Pre-train representation with 3D human hand pose estimation
image encoder
COIV
1000 BAA
compact repr.
hand decoder
(2) Offline adapt representation with self-supervised keypoint detection
image encoder
keypoint decoder
ConV
B
feature mappa
src view
reconstrct tgt view
Adapt affine transformation human - robot
Adapt running mean and var low return high return
(3) Reinforcement learning with compact representation and EMA BatchNorm
image encoder
conV
B
compact repr.
policy
action
N
food
Figure 2: The overview of H-InDex. H-InDex consists of three stages: 7) representation pre-training, representation offline adaptation, and 3) reinforcement learning.
on diverse hand datasets, totaling 400k samples. We adopt the ResNet-50 [10] feature encoder in the hand module to extract visual representations.
The use pre-trained model from the hand pose estimation task [24] shares the intuition with recent works on foundation models for motor control [16-19, [0]] learning representations from human manipulation vidcos. However, the use of the pre-trained hand model offers two distinct advantages that are not typically found in other approaches: i) the model explicitly predicts the hand-related information from diverse videos, forcing to learn the interaction and the movement of human hands; ii) the model can be borrowed from vision community without any extra cost to re-train foundation model.
Stage 2: Representation offline adaptation. In the previous stage, we only pre-train visual repre- sentations that are suitable human-centric images, neglecting the morphology and structure gap between robot hands and human hands. To bridge the gap without losing the information learned in the pre-training stage, we adopt self-supervised keypoint detection objective [1 14, 15] to only finetune the affine transformations in the BatchNorm layers of the pre-trained model, which occupy only 0.18% of the entire model parameters. While finetuning only a small portion of parameters, empirically outperforms both frozen model and fully finetuned model. We hypothesize that this is because the BatchNorm finetuning bridges the gap and mitigates catastrophic forgetting caused by finetuning [[].
now describe the self-supervised keypoint objective. Given a target image I and a source image T. sampled from a video, we aim to reconstruct / with the appearance feature of I, and the keypoint feature of Tt. Denote our pre-trained visual representation as hgg the keypoint feature extractor K the appearance feature extractor Fs. and the image decoder G... First, we extract semantic

---

Hammer
Door
Pen
Pour
Place Inside
Relocate Objects
Start
End
Figure 3: Visualization of our six kinds of dexterous manipulation tasks and one sampled trajectory. We depict both the initial confi anfiiuration and the goal. Videos of trajectories for tasks are available on our website yanjieze.com/H-InDDD.
feature map (J)) from the target image I, and then get the keypoint feature Kgp (ha((h)))))))) (T)). At the same time, we extract the appearance feature (1,))) from the source image L.- We then try to reconstruct the target image Le by decoding the concatenated keypoint feature and the appearance feature 1/ Gw (KCr (1)()))), F (1.)). Our final supervision is the perceptual loss [12] Epeccop.
Lkeypoint pereep (4, 12) [A(4) AGG, where A is the semantic feature prediction function in [12].
Stage Reinforcement learning. During the reinforcement learning stage, the distribution of observations is continually changing. For example, in the early learning stage, the observations are usually random explorations, while at the end the learning stage, most of the observations are converged trajectories. Such property of reinforcement learning requires the internal statistics of neural networks to move slowly towards the current observation distribution. Therefore, we utilize the exponential moving average (EMA) operation to dynamically update the statistics (i.e., the running mean and the running variance) in BatchNorm layers.
Formally, for the input that has dimensions, iee., x= (a(11, , (1)), we update the running mean ((() and the running variance )2 in BatchNorm layers with the following equation,
1 (0) < (1 - m) (1((0 + m []
2 - (1- mm-m) (0)2 + m Var Var[x (").
(5) for = 11...,, k. where m is the momentum. When is set to 0, our EMA BatchNorm layers revert back to the original layers, ensuring that the modification does not have negative impacts at the very least. Finally, all these three stages collectively contribute to our final method H-InDex. We remain implementation details in Appendix A.
Experiments
In this work, we delve into the application of visual reinforcement learning to address dexterous manipulation tasks, with a particular emphasis on visual representation aspect. We evaluate effectiveness of our proposed framework, H-InDex, across various tasks and elucidate the significance of each component in achieving the final results. Of partieular importance the integration of prior knowledge pertaining human hand dexterity into framework.
5

---

19303
Henmme
3000
1000
Pinn
20000
Four
3000
2003
1992
10000
2000
2002
2000
0
1/0
1500
Insido
Rebocol: Lange
292
5248d6 16322
202
1000
300
1250
1400
750
402
000
012
1.0
22.0
2000
1500
1500
Ephade
1750
1500
1290
200
1000
Envronment Steps (X1.0)
Enopomment Stepe
Broitommen Stepe (x1)P)
Endiroment Stepe (K10f)
1-teOeee (oun))
MVP
R23M
Figure 4: Episode return for challenging dexterous manipulation tasks. We compare H-InDex with four strong visual representations for motor control, i.e., [C]1 MVP [30], R3M [18], and RRL [26]. Mean of seeds with seed number 0, 1, Shaded area indicates 95% confidence intervals.
5.1 Experiment Setup
We evaluate H-InDex on 12 challenging visual dexterous manipulation tasks from Adroit [23] and DexMV [22] respectively, including Hammer, Door, Pen, Pour, Place Inside, and Relocate YCB Objects (7 different objects [3]). Visualization of each task is given in Figure 3 and detailed descriptions are provided in Appendix B. This selection of tasks the most extensive compared to previous works e.g.. DAPG (4 tasks) and DexMV (7 tasks), thereby showcasing the robustness and versatility of H-Ineex. The tasks were performed with varying numbers of steps based on their level of complexity. We mainly report the cumulative rewards to show the speed of task completion. The dimension image observations o
x 224 x 224 across all methods. We run experiments on an RTX 3090 GPU; each seed takes roughly 12 hours. Due to the limitation on computation resources, we choose run seeds for each group of experiments and consistently use seeds with seed numbers 0, 1, to ensure reproducibility. We also observe that H-Inexx enjoys slight variance between seeds, while baselines tend to have a larger variance.
5.2 Main Experiments
To demonstrate the effectiveness of H-InDex, we evaluate diverse recent strong visual representations
for motor control, including (i) VC-1 [17], which trains masked auto-encoders over 5.6M images with over 10,000 GPU-hours and we use the ViT-B [4] model (86M parameters): (ii) MVP [30]. which also uses masked auto-encoders for pre-training and we use the VIT-S [4] model (22M parameters); (iii) R3M [18], which pre-trains ResNet-50 (22M parameters) with time contrastive learning and video-language alignment (iv) RRL [26], which uses the ResNet-50 pre-trained on the ImageNet classification task directly. Due to task differences,
normalize the cumulative rewards based on the highest rewards achieved and present the average scores in Figure 1. We also report the learning curves in Figure 4. We then detail our observations below.
H-InDex emerges as the dominant representation. Across 12 tasks, H-InDex outperforms recent state-of-the the--ar representation VC-1 by 16.8% absolute improvement. Furthermore, H- InDex surpasses RRL, the original state-of-the-art representation Adroit, by 25.4%. Analyzing the learning curves, H-InDex demonstrates superior sample efficiency in 10 out of the 12 tasks. In

---

only two tasks, namely relocate mug and relocate mustard bottle, VC-1 exhibits slight advantage over H-InDex.
ConvNets v.s. ViTs, Among representations utilizing the ResNet-50 architecture (i.e, H-InDex, R3M, RRI., only H-InDex showcases obvious advantages over ViT-based representations. This suggests that with appropriate domain knowledge, ConvNets can still outperform ViTs. Additionally, we notice that ConvNets and ViTs excel in different tasks. For instance, in relocate tomato soup can, VC-I and MVP achieve returns that are only half of what H-InDex and RRL accomplish. However, in relocate mug, VC-1 performs well. These observations highlight the task-dependent nature of the strengths exhibited by ConvNets and ViTs.
5.3 The Effectiveness of 3D Human Hand Prior
The significance of transferring the 3D human hand knowledge into dexterous manipulation is non-negligible. To demonstrate the utility of such 3D human hand prior, we compare our vanilla pre-trained representation i.e., the feature extractor from the FrankMocap hand mod- ule [24] (denoted as FrankMoccp Hand) with other 4 representative pre-trained models: (i) Fraak Body, which is the body estimation module from FrankMocap [24], pre-trained with 3D body pose estimation, (ii) AlphaPose [5], which is widely-used robust 2D human pose estimation algorithm, (iii) R3M [18], which is pre-trained with time contrastive learn- ing [25] and language-video alignment on Ego4D [6], and (iv) RRL [26]. which directly uses the ResNet-50 pre-trained on the ImageNet classification task. All the models use a ResNet- 50 architecture and do not any adaptation, ensuring the faimess of our comparison. We also put H-InDex as the best results achieved for comparison. Results are shown in Figure 5. We now detail our observations below:
Hammer
Moaak hand v.s. RRL/R3M. Our vanilla represen- tation has significantly outperformed both RRL and R3M without the need for any adaptation.
15000
Episode Return
10000
A
FrankMocap hand v.s. 3D/2D body-centric represen- fations. Our vanilla 3D hand representation, FrankMocap Hand, demonstrates superior sample efficiency compared to FrankMocap Body and significantly outperforms Al- phaPose. This confirms our hypothesis that the 3D human hand prior more advantageous than both the 3D and 2D human body prior. It is worth noting that AlphaPose, being a whole-body 2D pose estimation model, also capable of estimating 2D hand poses. The fact that our 3D hand prior outperforms the 2D hand prior in this context further supports its effectiveness. We hypothesize this is because 2D pose estimation does not require deep spatial reasoning compared to 3D pose estimation.
0
0.0
0.5
1.0
1.5
2.0
Environment Steps (x10f)
H-inDex
AlphaPose
FrankMocap Hand
R3M
FrankMocap Body
Figure 5: Compare vanilla pre-trained
representations with H-InDex.
H-InDex v.s. FrankMocap hand. Our vanilla hand representation already surpasses all other pre-trained ConvNets performance. However, by applying our adaptation technique, we can further enhance the sample efficiency of the hand representation, underscoring the significance of adapting the model with in-domain data in a proper way.
5.4 Ablations
To validate the rationale behind the design choices of H-InDex, we performed comprehensive set of ablation experiments.
Effects of each stage. Figure 6a provides insights into the contributions of each stage towards the overall efficiency of H-InDex. We refer to RRL as w/o Stage 1,2,3. Significantly. Stage exhibits the most notable enhancement, underscoring the efficacy of human dexterity. Moreover, Stage 2 and Stage 3 also contribute appreciable advancements. In addition, the value of momentum (m) in Stage 3 has a significant influence, as illustrated in Figure 6b. To determine the optimal value, we performed a grid search over m {0, 0.1, 0.01, 0.001). This analysis highlights the importance of selecting an appropriate momentum value for achieving optimal performance in Stage 3 of H-nxDex.
7

---

Hammer
Hammer
Hammer
15000
15000
15000
Episade Return
12500
Episoris Return
12500
Episade Return
12000
10000
10000
7500
7500
7500
Seage
5000
Stage 1-2
5000
1s0011
5000
2500
Stage 1-2:3
2500
m=0
2500
Stage
0.0
0.5
1,0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Environment Steps (x10f)
Environment Steps (x11*)
Environment Steps (x10f)
(a) Ablation on the effectiveness of
(b) Ablation on momentum m
(c) Compare our minimal adaptation in Stage 2 and the full adaptation.
our three stages.
{0. 0.1, 0.01, 0.001).
Figure Ablation experiments. We ablate each component of H-InDex and show that each individual part effectively combines to contribute to the overall effectiveness of H-InDex.
4000
Pen
20000
12300
3000
15000
93390
2092
1500
2000
10000
3000
1300
200
1000
0.0
1.8
Piace Inside
1583
Rebocato Longo oom
1600
Rcbectte Foorr BlEK
1500
Relocote Box
1400
E24odd 1Mmm
1200
100
760
1002
750
000
500
250
2000
Nalocate Mag
2000
Perocata Marcard Sctie
Rid ocata Torrato Soup Can
2500
Ralocala Potaec Must Can
2750
2000
2000
2000
2000
1753
Epaede
1000
3000
500
1000
202
Envrooment (X10*)
Enviroment Swipe (x110))
Ensiroment (K10))
adapt 100%
Figure 7: Ablation on Stage 2 (adapting 0.18% parameters or adapting 100% parameters). We observe that simply finetuning the entire visual representation would lead sub-optimal results. Instead, H-InDex only adapts the parameters in BatchNorm layers and effectively solves all the tasks.
Figure provides more ablation results on Stage 3, further supporting the necessity of updating the BatchNorm layers during the training of RL agents.
Adapting 100% parameters v.s. adapting 0.18% parameters in Stage In Stage of HnDDex we intentionally chose to adapt only 0.18% of the parameters (the affine transformations in BatchNorm layers) the pre-trained representation. This decision was made to address a specific concern, as depicted in Figure 6c and Figure 7. By altering only the setting for Stage 2 while keeping all other factors constant, we observed that across all tasks, adapting all parameters is not more advantageous. This phenomenon may be attributed to the fact that freezing and finetuning partial parameters help mitigate catastrophic forgetting, which is often caused by full finetuning [1]. In Figure 8, we also show the necessity of our Stage 2. We could conclude from results that correctly finetuning the visual representation is one key to the stable convergence, and not correctly finetuning the model, such as finetuning all the parameters, could be worse than the frozen model.
Robust visual generalization. One concern of our hand representation is its generalization ability, compared to the vision model pre-trained on large-scale datasets, such as VC-1 [17]. Therefore, we change the background of the training scene various novel backgrounds, as shown Figure 12 (see Appendix C) and evaluate VC-1 and H-InDex on the task relocate potted meat can. The

---

15002
Hemmme
3305
1005
Penn
20000
Fourr
12200
2503
3002
2303
2002
200
0.0
1.5
Placs Inside
1000
Relecole Foum Brick
1500
Relocate Box
2500
Relocns Mlyg
2250
Estazee ba0u
202
1000
3000
150
500
000
402
50
0.5
1.5
200
Envesmmnt Steps (X10%)
Envirnmmnt Sapp (xL09)
Stveonment Stops (X.10))
Envirnment Stape (K10f)
Figure 8: Ablation on Stage 2 (with or without adaptation). We also conduct more experiments to show the necessity of Stage We could observe a consistent improvement across these tasks by applying Stage 2, which only adapts 0. 18% parameters of the visual representation.
Lammer
Inside
ooste Larpe Clamp
Relecatee Foom
15000
252
1250
1403
Rebmm
125.0
200
1253
10000
2500
100
200
2200
Box
Rolocato Soup
1200
2002
Epaode Ratom
090
1500
1002
1000
502
500
Stage 3, mm0
Stage m-d
Figure 9: Ablation on Stage 3 (momentum m 0 or m= 0). We observe that our Stage contributes greatly to some specific tasks, such as relocate mustard bottle, and for some tasks like hammer, tuning this parameter only results in slightly faster convergence. For tasks not shown here, all use m 0, since H-InDex with Stage and Stage has been strong enough.
results given in Table (see Appendix C) show that H-InDex could handle the changed background better than VC-1. We also see the consistent performance drop across all scenes, emphasizing the importance of visual generalization.
Visualization of self-supervised keypoint detection in Stage In Stage self-supervised keypoint detection objective is employed to fine-tune minimal percentage (0.18%) of parameters in the pre-trained model. The visualization results, as shown Figure 10, demonstrate the successful detection of keypoints. This observation highlights the pre-trained model's capability to effectively allocate attention to the hand and objects depicted in the images, even with the adaptation of only a small subset of parameters.
Visualization of afline transformations adaptation in Stage The significance of Stage 2 in H-Inee is evident from Figure 6a. To gain deeper insights into this phenomenon, we visualize the distribution of the adapted parameters, specifically the affine transformations in BatchNorm layers. We accomplish this by fitting Gaussian distribution. Figure 11 presents the visualization results, the adapted models
highlighting an interesting trend. For the shallow layers, the distributions closely resemble those of the pre-trained models. However, as we move deeper into the layers, noticeable differences emerge. We attribute this disparity to the fact that dissimilarities between human hands and robot hands extend beyond low-level features like color and texture. Instead,

---

Hammer
Pen
Pour
Relocate Objects
Figure 10: Visualization of our self-supervised keypoint detection. We select four tasks here and mark the detected keypoints on images with wrdd stars. We observe that these keypoints consistently mark the dynamic regions of images. Full videos are available on yanjieze.com/H-InDee
Layert
Laywe
Rio D000
7.5
0.0
Layara Bha
Layors 8%a
Layera BNS
Laye44BL BNL
Layore Bhi2
Layora Bh@
10
75
10
50
25
0.
0.2
5/
0.0
115
811
03
04
Figure 11: Visualization of affine transformation adaptation in Stage 2. We fit Gaussian distribution for parameters of affine transformations. We omit the X-axis and Y-axis here for simplicity, Human represents the original pre- preitreined model and robot represents the adapted representation in different tasks. is observed that deep layers are subjected large distribution shifts.
they encompass higher-level features such as dynamics and structure [2,31,34] This observation underscores the importance of our adaptation approach, as it effectively addresses the variations in both low-level and high-level features, facilitating the success of H-InDex.
Conclusion
In this study, introduce H-InDex, visual reinforcement learning framework that leverages hand- informed visual representations to tackle complex dexterous manipulation tasks effectively. H-InDex range of 12 tasks, including six kinds
outperforms other recent state-of-the-art representations of manipulation skills. The effectiveness of H-InDex can be attributed to its three-stage approach, wherein Stage incorporates pre-trained 3D) human hand representation and Stage and Stage focus on careful in-domain adaptation with only 0.36% parameters updated. These stages collectively contribute the successful preservation and utilization of the human hand prior knowledge.
is also important to acknowledge some limitations of our work. We did not investigate the generalization capabilities of H-InDex, particularly in scenarios involving the grasping of novel objects. Our future work aims to address this limitation and enhance H-InDex's 's generalization capabilities for real-world applications. In addition, we find that Stage 3 is surprisingly effective in some specific tasks like relocate mustard bottle, while we have not given a theoretical understanding of such phenomena. We consider this problem possible future direction.
Acknowledgment
This work is supported by National Key R&D Program of China (2022DD0161700)
10

---

References
[1] Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. rseeasa::
Achieving deep reinforcement learning without catastrophic forgetting. Neurocomputing. 2021.
4, 8
[2] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations, In Proceedings of the IEEE conference computer vision and patlern recognition, pages 6541-6549, 2017. 10
[3] Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. Benchmarking in manipulation research: The ycb object and model set and benchmark-
ing protocols. arXiv, 2015. 6, 14
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv, 2020. 2,
[5] Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. PAMI, 2022. 7
[6] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. 2,7
[7] Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelcrating visual model-based reinforcement learning with demonstrations. ICLR, 2023. 3
[8] Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning- from-scratch baseline. ICML, 2023. 2
[9] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 2
[10] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 2, 4, 13
[11] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. NeurlPS, 2018. 4, 13
[12] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 5
[13] Sham M Kakade. A natural policy gradient. NewrIPS, 2001. 3
[14] Tejas D Kulkarni, Ankush Gupta, Catalin lonescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. NeurlPS, 2019. 4, 13
[15] Yizhuo Li, Miao Hao, Zonglin Di, Nitesh Bharadwaj Gundavarapu, and Xiaolong Wang. Test-time personalization with transformer for human pose estimation. NeurIPS, 2021. 4, 13
[16] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. ICLR, 2023. 4
[17] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the search for artificial visual cortex for embodied intelligence? 2023. 1, 4, 6, 8, 14, 15, 16
[18] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv, 2022. 1, 2, 4, 6, 7, 15
[19] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurpris- ing effectiveness of pre-trained vision models for control. ICMI., 2022. 2, 4.4
[20] Austin Patel, Andrew Wang, Ilija Radosavovic, and Jitendra Malik, Learning to imitate object interactions from internet videos. arXiv, 2022. 3

---

[21]
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth- Matej Vecerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep reinforcement learning for dexterous manipulation. arXiv, 2017. 2
[22] Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In ECCV, 2022. 2, 3, 6, 13
[23] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. RSS, 2018. 2, 3, 6, 13
[24] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv, 2020. 1, 3, 7,
[25] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang. Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In ICRA, 2018. 2,7 7
[26] Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. ICML,
2021. 1, 2, 3, 6, 7, 13, 15, 16
[27] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet
videos. CoRL, 2022. 3
[28] Aravind Sivakumar. Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: learning a robotic
hand imitator by watching humans on youtube. RSS, 2022. 3
[29] Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vr13: A data-driven framework for visual deep reinforcement learning. arXiv, 2022. 2,
[30] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for
motor control. arXiv, 2022. 1, 2, 3, 4, 6, 15
[31] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.000 2015. 10
[32] Yanjie Ze, Nicklas Hansen, Yinbo Chen, Mohit Jain, and Xiaolong Wang. Visual reinforcement
learning with self-supervised 3d representations. RA-I., 2023. 2
[33] Yanjie Ze, Ge Yan, Yueh-Hua Wu, Annabella Macaluso, Yuying Ge, Jianglong Ye, Nicklas Hansen, Li Erran Li, and Xiaolong Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. CoRL, 2023. 2
[34] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Fision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 818-833. Springer, 2014. 10
12

---

Appendix
A Implementation Details
Codebase. Our major codebase is built upon the official implementation of RRL [26], which is publicly available sttps://ee /ttpt///ghhhbbbb cowm/facboolressrrcc/etttt
and includes the Adroit manipulation tasks [23]. The DexMV [22] tasks are from the official code https://gi con/yyggi/ddxmy- sim. All the visual representations in work also available online, including RRL (pre-trained ResNet-50, provided in PyTTrch officially), MVP (attps://github com/ir413/ wow//faceeoooooeeeeaaaaaaaaaaaaa wwt.e//rreeeeerereeeeeeaeaeaaeee eai-vc), and FrankMocap (https: This ensures the good reproducibility of our work. We are also committed to releasing the code.
R3M (tttps://github.. wtt.:au..oooooooaaaaaaaaea wttp://aaccboorreerraaaaaaa//aae mvp), VC-1 //github.
Network architecture for H-InDex. The architecture employed by H-InDex is based on ResNet- 50 [10], referred to as hg. In the initial stage (Stage 1). ha takes as input a 224 x 224 RGB image and processes it to generate compact vector of size 2048. In Stage 2, we modify he by removing the average pooling layer in the final layer, resulting in the image being decoded into a feature map with dimensions 7 7 2048. Moving to Stage 3, he once again produces compact vector of size 2048, while simultaneously updating the statisties within the BatchNorm layers using the exponential moving average operation.
Implementation details for Stage 2. Our implementation strictly follows the previous work that also uses the self-supervised keypoint detection as objective 14, 15]. We give Py PyTochh-ttyl overview of the learning pipeline below and refer to [11] for more implementation details. Notably, the visual representation ha (24M) contains the majority of parameters, while all other modules in the pipeline maintain a parameter count ranging from 1M to 3M. We use 50 demonstration videos as training data for each task and train 100k iterations to ensure convergence with learning rate 1 x 10"4, One of our core contributions is to only adapt the parameters in BatchNorm layers hgg and we emphasize that the learning objective is not our contribution, it has been well explored in [11, 14, 15].
for in range (num num_iters):
# sample data sierce_view target_vies next (extt___itter) 3x224x224
self- keypoint-based reconstruction
# h_ ahtheta is our visual representation
feature_mas (
7x7x2048
keypoint_feaa up_eamplerrkeyypinttteeee
30x56x56
keypoint_feat' = up up_sampler bedoypointffetturee # -> 256x28x28
apperamce_featt apperance_ enperance_eecoder (source_ # -> 256x28x28
porget_view_rees = image_ decoder [[keppoint_ feat,
aeceppoont_etttapppaccceeeeeee aeparance_featt) # 3x224x224
compute loss loss perceptual lacppptaallllssssseeee (target_vier, carget_view_ren
compute gradient and update model
optimizer.zen meriiieerzeerogeee (g
l.ss. backward
B
Task Descriptions
In this section, we briefly introduce our tasks. We use an Adroit dexterous hand for manipulation tasks. The task design follows Adroit [23] and DexMV [22]. Visualizations of task trajectories are available at sindex--l.githuub..
Hammer (Adroit). It requires the robot hand to pick up the hammer the table and the hammer to hit the nail.
Door (Adroit). It requires the robot hand to open the door on the table.
13

---

Pen (Adroit), It requires the robot hand to orient the pen to the target orientation.
Pour (DexMV). It requires the robot hand reach the mug and pour the partieles inside into container.
Place inside (DexMV). It requires the robot hand place the object on the table into the mug.
Relocate YCB objects [3] (DexMV). It requires the robot hand pick up the object on the table the target location. The objects in our tasks include foam brick, box, mug, mustard bottle, tomato soup can, and potted meat can.
C Visual Generalization
One concemn of our hand representation is its generalization ability, compared the vision model pre-trane tre-trained on large-scale datasets, such as VC-1 [17]. Therefore, we change the background of the training scene various novel backgrounds, shown in Figure 12 and evaluate VC-1 and H-InDex on the task relocate potted meat can. The results given Table show that H-InDex could handle the changed background better than VC-1. We also the consistent performance drop across all scenes, emphasizing the importance of visual generalization.
Figure 12:
Various backgrounds for visual generalization. The first image shows the training scene and the rest 9 images show the novel scene.
Table I: Scores for generalization to unseen backgrounds on relocate potted meat can task. We evaluate VC-1 and H-InDex with 20 episodes for each seed.
Scene Method
VC-1 [17]
H-InDex
Origin
2391.74 74.602.8 83
2240.37=s5.d
896.28. 28. 1006.56
915.95.922.dd
2
(08.262920.44 48
771. 28-783. 51
3
431 36..m6 45
578. 42-764..000
4
360.21 1 64
472 66 05
5
300.02. 0n
393. 32..337.666
256. 804670 16
340.07 640.077639..6 68
224.21 +335
298. 20-2200-6..00
8
226.59ta10. 58
265. 00...82.81..0
9
214.30a581.00
239 60-556-566600
Average
392.56
475.04
D Main Experiments (ConyNets Only)
In our primary experimental analysis, we conduct a comprehensive comparison of five visual repre- sentations, with three of them being ConvNets, including our method. Figure 13 presents an isolated
14

---

demonstration of the comparison among the ConvNets. Notably, our method H-InDex exhibits superior performance in comparison the other ConvNets.
19303
"errme"
3302
Doo
4000
Pen
20000
Pour
3502
2000
2000
1992
2500
10000
8090
2000
300
2000
0-
1902
traids
Large Cherp
1900
222
200
1300
1400
1200
1220
750
400
500
600
012
1.0
22:0
2002
1000
1500
12.00
1583
1900
1252
502
1000
Envrenment Stape (X1.0))
Dnwinomee Stepo
Dnvitommnt Stepe (x10f)
Enstroment Stepe (X115)
HirDDx (ours)
R3M
RRL
Figure 13: Episode return for 12 challenging dexterous manipulation tasks. Mean of 3 seeds with seed number 1, Shaded area indicates 95% Cls.
E Success Rates in Main Experiments
We present the success rates of our six task categories in Table 3. Regarding the hammer task, it is evident that both H-InDex and VC-1 exhibit success rates near 100%. However, a notable disparity arises when considering episode returns, indicating the varying degrees of task execution proficiency even among successful agents.
Table 2: Success rates for main experiments. Highest success rates for each task are marked with bold fonts. The success rates only reflect whether the task is finished but do not reflect how fast the task is finished. HnDex still dominates other methods.
Task name/ Method
RRL [26]
R3M [18]
MVP [30]
VC-1 [17]
H-InDex
Hammer (2M)
80 15
24+21
83:11
97.3
100.0
Door (4K)
92+1
99+2
100=0
99s2
96+5
Pen (8M)
78+1
5846
80+1
81+2
90=2
Pour (8M)
38a33
0.0
23:38
67.29
99.2
Place inside (6M)
70.50
1:2
99.2
98.4
97.6
Relocate large clamp (8M)
33=31
47+27
23+21
50+45
Relocate foam brick (2M)
87.11
42:37
48:46
44149
86.
Relocate box (6M)
94.5
45:24
48.50
49:50
85:
Relocate mug (2M)
100+0
82+2
54+51
74+11
100+0
Relocate mustard bottle (2M)
100.0
82.8
100.0
9042
99x2
Relocate tomato soup can (2M)
9743
18:31
6:10
30:52
99.2
Relocate potted meat (2M)
97:6
56+15
69+48
88+21
94=5
Average
81.3
42.3
63.1
70.8
91.3
15

---

Table 3: Success rates and scores for generalization unseen backgrounds on relocate potted meat can task. We evaluate VC-1 and H-InDex with 20 episodes for each seed.
Scene ID / Method
VC-1 (success rate)
VC-I [17] (score)
H-InDex (success rate)
H-InDex (score)
38+43
896. 28 1006.55
48+48
915.95=922.6
26=39
603. 265.266920.46
32+46
771.288773.55
19:36
451.36e839.0
24+42
478.422764.00
15x33
360. 260.214772.61
19x39
472.66=715.08
13x81
300.02+718.05
16x36
393.32st176.c
11a29
256.80a67s.10
14x34
340.07 a739939.6
10x27
224.21 663.56
12x32
298.20.808.cr
10.26
226.59.610.36
11,30
265.82.5a
9.25
21 4.1.00588.77
10+29
239.60. 556.65
Average
16.8
392.56
20.7
475.04
F
Hyperparameters
We categorize hyperparameters into task-specific ones (Table 4) and task-agnostic ones (Table 5), Across all baselines, all the hyperparameters shared except the momentum m, which is only used in our algorithm. All the hyperparameters for policy learning are the same as RRL [26]. This ensures the comparison between different representations fair.
Our exploration of the momentum m Table has been limited to a specific set of values, namely {0,0.) 0.1.0, 0.01, 0.001}. through the use of grid search technique, due to the limitation on computation resources. is observed that carefully tuning m could take more benefits.
Table 4: Task-specific hyperparameters.
Task name / Variable
Momentum
Demonstrations
Training steps (M)
Episode length
0.1
25
2
200
Door
0.0
25
4
200
Pen
0.0
25
6
100
Pour
0.0
50
8
200
Place inside
0.001
50
8
200
Relocate large clamp
0.01
50
8
100
Relocate foam brick
0.01
25
2
Relocate box
0.001
6
100
Relocate mug
0.0
25
8
100
Relocate mustard bottle
0.001
25
6
100
Relocate tomato soup can
0.01
25
8
100
Relocate potted meat can
0.0
25
4
100
Table 5: Task-agnostic hyperparameters.
Vuriable
Value
Dimension of image observations
x21x 221
Dimension robot states
30
Dimension of actions
30
Hidden dimensions policy
256,
BC learning rate
0.001
BC epochs
BC batch size
32
learning rate
0.001
Number trajectories for step
100
VF hatch size
64
VF epochs
RL step size
0.05
gammaa
0.995
RI. gne
0.97
16


Reviews:
[{"review_id": "NXolzYpZyp", "reviewer": "Reviewer_GX9c", "summary": "This paper introduces a framework called H-InDex that uses human hand-inspired visual representation learning to solve complex manipulation tasks with reinforcement learning. The framework consists of three stages and outperforms other methods in challenging dexterous manipulation tasks.", "questions": "* Are the proprioceptive states of the robotic hand (e.g. joint states) considered in stage 3?\n\n* How does this work encode the object information in the compact representations?", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* A novel and effective method to perform dexterous hand manipulation using visual RL. \n\n* Extensive experiments and ablation study on various tasks.\n\n* A convincing framework exploiting the pre-trained vision models for robot learning.", "weaknesses": "* The method seems only to be able to perform on a single object on each task, while the vision model should be able to generalize on category-level objects.\n\n* It is unclear of the RL action space, reward design, robotic control parameters, and the coordinate system used in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions. We address each of your comments in the following.\n\n**Q1:** The method seems only to be able to perform on a single object on each task, while the vision model should be able to generalize on category-level objects.\n\n**A1:** Achieving generalization across objects is not trivial, especially when the training objects are fixed and not diverse. In this work, all our tasks only have one single object and we do not put our focus on generalizable manipulation (clearly stated in the limitation part). Instead of generalizable manipulation, one good property of the pre-trained visual presentation is visual generalization, such as generalizing to unseen scenes. To show our model’s visual generalization ability, we change the visual backgrounds and evaluate our H-InDex and VC-1 on Relocate Potted Meat Can task. The results are given in Table 2 of the rebuttal file and the visualizations are also given in Figure 6. We could observe that, though VC-1 has a slightly better training performance on this task, our method presents better robustness on these new scenes, indicating the strong visual generalization ability of H-InDex.\n\n**Q2:** It is unclear of the RL action space, reward design, robotic control parameters, and the coordinate system used in the paper.\n\n**A2:** In this work, all our tasks follow prior works (Adroit [1] and DexMV [2]) and we do not make specific changes to the tasks, such as action space and reward design. The action space is 30-dim and the action controls the joints of a five-finger robot hand, i.e., Adroit hand. We use a dense reward for RL and the reward design is task-specific. We will add more task details in our revised version.\n\n**Q3:** Are the proprioceptive states of the robotic hand (e.g. joint states) considered in stage 3?\n\n**A3:** Yes, and the robot hand states are only used in Stage 3 (RL stage). In our paper, we also mention the usage of robot hand states in Section 3 (L102). In addition, the usage of robot hand states is a common practice in visual-based robot learning [3, 4, 5], and all our baseline methods also use robot states for fair comparison.\n\n**Q4:** How does this work encode the object information in the compact representations?\n\n**A4:** The object information is directly encoded from the visual observations. We do not use any other information such as the object bounding box, the object coordinates, and so on. This makes the learning process extremely challenging, especially for dexterous manipulation.\n\n**Please do not hesitate to let us know if you have any additional comments.**\n\n[1] Rajeswaran, Aravind, et al. \"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.\" arXiv preprint arXiv:1709.10087 (2017).\n\n[2] Qin, Yuzhe, et al. \"Dexmv: Imitation learning for dexterous manipulation from human videos.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[3] Shah, Rutav, and Vikash Kumar. \"Rrl: Resnet as representation for reinforcement learning.\" arXiv preprint arXiv:2107.03380 (2021).\n\n[4] Xiao, Tete, et al. \"Masked visual pre-training for motor control.\" arXiv preprint arXiv:2203.06173 (2022).\n\n[5] Radosavovic, Ilija, et al. \"Real-world robot learning with masked visual pre-training.\" Conference on Robot Learning. PMLR, 2023.\n\n\n"}, {"review_id": "lzl31jDRrP", "reviewer": "Reviewer_s3to", "summary": "This paper studies a three stage framework for adapting pre-trained visual representations for dexterous manipulation tasks. Specifically, this work proposes only adapting a subset of network parameters in stages 2 and 3. The approach is tested on 12 manipulation tasks from Adroit and DexMV. These experiments demonstrate that the proposed techniques lead to more sample efficient learning. Furthermore, the overall approach outperforms alternative pre-training methods from recent work.", "questions": "- How does asymptotic performance compare in the ablations in Figure 6? \n- Do the sample efficiency improvements generalize to other tasks?", "limitations": "Yes, the authors discuss limitations of the work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- This paper demonstrates that using a visual backbone pretrained for (human) hand-pose estimation leads to improvements in learning downstream dexterous manipulation tasks over strong baselines from recent work. These results suggest that this inductive bias might be missing from recently proposed methods.\n- This work proposes adapting a specific subset of parameters during stages 2 and 3 of training, and demonstrates that this leads to more efficient learning.\n- The paper is clearly written and most of the details of the proposed approach are clear.", "weaknesses": "Large parts of the proposed approach are directly applied from prior work including the pretrained visual representation (stage 1) and the learning objective from stage 2. This is clearly stated in the manuscript and appendix, and is not in itself an issue. However, this means the contributions of this work are primarily in the parameter efficient techniques used in stage 2 and 3.\n\n- Unfortunately, the results in Figure 6 indicate that stage 2 may improve sample efficiency, but may not improve asymptotic performance. Thus, it is unclear if this is a significant contribution.\n- Furthermore, the sample efficiency gains (Figure 6) are only demonstrated on two tasks (Hammer and Place Inside), so it is unclear how general these findings are.\n- Similarly, when compared to full finetuning (Figure 6c) it is not clear if asymptotic performance is improved and the sample efficiency gains are minimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions. We address each of your comments in the following.\n\n**Q1:** Large parts of the proposed approach are directly applied from prior work including the pre-trained visual representation (stage 1) and the learning objective from stage 2. This is clearly stated in the manuscript and appendix, and is not in itself an issue. However, this means the contributions of this work are primarily in the parameter efficient techniques used in stage 2 and 3.\n\n**A1:** We thank the reviewer for insightful comments and for acknowledging our contributions in stages 2 and 3 of the proposed approach. We greatly value your feedback.\n\nHowever, we would like to respectfully point out that Stage 1, the usage of a pre-trained representation using 3D human hand pose estimation is also one of our main contributions. While it is true that Stage 1 simply employs an existing model, **the inductive bias, introduced by us is not trivial and still intuitive**. A lot of recent works stick to using unsupervised learning on egocentric data to pretrain a new task-agnostic model, such as MVP [1,4], R3M [2], VIP [3], and RPT  [5]. However, we show that with the proper usage, the existing pre-trained model from other domains could be enough. **Moreover, it's not straightforward to adapt a pre-trained model from different domains.** As shown in Figure 5 of the paper, some other models that also share a similar intuition, such as AlphaPose, could not gain a reasonable result. And as shown in our new ablation results (Figure 2, Figure 3, and Figure 4 in the rebuttal file), wrongly using the pre-trained model would not lead to reasonable performance. **Our framework, combining all three stages together and correctly adapting the pre-trained model, unleashes the potential of the pre-trained encoder.**\n\nWe believe our Stage 1 and the efforts in combining all three stages effectively are also the core contributions of our work.\n\n[1] Xiao, Tete, et al. \"Masked visual pre-training for motor control.\" arXiv preprint arXiv:2203.06173 (2022).\n\n[2] Nair, Suraj, et al. \"R3m: A universal visual representation for robot manipulation.\" arXiv preprint arXiv:2203.12601 (2022).\n\n[3] Ma, Yecheng Jason, et al. \"Vip: Towards universal visual reward and representation via value-implicit pre-training.\" arXiv preprint arXiv:2210.00030 (2022).\n\n[4] Radosavovic, Ilija, et al. \"Real-world robot learning with masked visual pre-training.\" Conference on Robot Learning. PMLR, 2023.\n\n[5] Radosavovic, Ilija, et al. \"Robot Learning with Sensorimotor Pre-training.\" arXiv preprint arXiv:2306.10007 (2023).\n\n\n**Q2:** Unfortunately, the results in Figure 6 indicate that stage 2 may improve sample efficiency, but may not improve asymptotic performance. Thus, it is unclear if this is a significant contribution. Furthermore, the sample efficiency gains (Figure 6) are only demonstrated on two tasks (Hammer and Place Inside), so it is unclear how general these findings are. Similarly, when compared to full finetuning (Figure 6c) it is not clear if asymptotic performance is improved and the sample efficiency gains are minimal.\nHow does asymptotic performance compare in the ablations in Figure 6? Do the sample efficiency improvements generalize to other tasks?\n\n**A2:** During the rebuttal phase, we have conducted a more comprehensive evaluation of our proposed Stage 2 and Stage 3, across almost all tasks, as shown in our rebuttal PDF file (Figure 2, Figure 3, and Figure 4). \n\nIn Figure 2, we conduct experiments across all 12 tasks and show that adapting all parameters in Stage 2 is not better than our proposed method in Stage 2, which only adapts 0.18% parameters. It is also observed that on Relocate Box, adapting all parameters is better, but this phenomenon only appears in one task.\n\nIn Figure 3, we conduct experiments across all 7 tasks that use m>0 in Stage 3. It is shown that our adaption in Stage 3 also makes a solid contribution to the final results achieved.\n\nIn Figure 4, we show that directly removing Stage 2 instead of full finetuning (which is shown in Figure 2) is also not good. This observation is natural since we use in-domain data to finetune the pre-trained model, while it could be generally shown that wrongly finetuning the model (Figure 2) could possibly hurt more than not finetuning the model (Figure 4). \n\nIn response to the reviewer's comment on asymptotic performance, we present the convergence results in Figure 7. This indicates that our method reaches the peak performance at 2M steps, so we saw no need to continue further steps.\n\nOverall, we enhance the ablations during the rebuttal phase and we hope that these new results could show the advantages of our proposed techniques in improving the pre-trained vision model for downstream RL tasks.\n\n**Please do not hesitate to let us know if you have any additional comments.**\n"}, {"review_id": "QzMlHPKHKs", "reviewer": "Reviewer_rofX", "summary": "The paper proposes H-InDex for dexterous manipulation tasks with a 30 DoF robotic hand in simulation. The method consists of three stages: the first stage adopts a pre-trained network that is trained with a 3D human hand pose estimation dataset; the second stage adapts the representation with self-supervised keypoint detection, which updates only 0.18\\% parameters of the entire model; and the final stage learns an RL policy on top of the adapted module while also adapting representation by using the exponential moving average for changing distribution based on training state.\n\nThe method is extensively evaluated on 12 different visual dexterous manipulation tasks and is demonstrated it outperforms the current SoTA baselines by a 16.8\\% absolute improvement and also achieves superior sample efficiency in 10 out of the 12 tasks. The authors also tested the effectiveness of the proposed human hand prior by comparing the method with several different priors and showed the hand prior from human movies does improve the performance. Finally, ablations are conducted to show the effects of each state, how the adaptation works, qualitative results on keypoint detection, and visualization of the adaptation in BatchNorm layers.\n\nOverall, the paper is very well written and organized, and the reviewer enjoyed reading this work. However, there are some unclear descriptions and conclusions that cannot be drawn from the experiments. The reviewer would be happy to raise the score if the authors could answer the questions and/or clarify the reviewer's misunderstanding.", "questions": "- Fig.2: The data distribution changes between stages 2 and 3 are a bit confusing. The left human to robot should happen in stage 2 and the right one should happen in stage 3, correct? If it is, the review thinks it would be clearer to enclose the left one with blue dots and the right one with green dots. Otherwise, it looks like the adaptations are independent of the proposed three stages and are confusing.\n- Fig.2 and Sec.4: In stage 2, what do the dots between 'keypoint decoder' and 'reconstruct tgt view' mean? The reviewer read the implementation details shown in the appendix, but it does not explain the 'dots' part.\n- The ratio of parameter changes is written as 0.36\\% in the abstract and conclusion but written as 0.18\\% in the other parts. Which is true or the reviewer misunderstands something?\n- Fig.7: The reviewer cannot see the videos of how keypoints move in the video.\n- Sec.5.2: The reviewer is curious about the performance of the BC-only policy while adapting the domain gap using expert trajectories. Does it correspond to the performance at environment steps being zero in Figure 4?", "limitations": "- As mentioned in the conclusion section, the method is tested on the same objects in the training set.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The idea of using a prior from human hand pose estimation is interesting, and the experimental results support the soundness of the method.\n- Only adapting the limited parameters by changing the statistics of the BatchNorm layers is also interesting and could be useful for other tasks that involve domain transfer, such as sim2real.\n- The experiments are extensively conducted and they support the idea of the proposed method. However, there are some unclear descriptions and/or conclusions that are summarized in the Weaknesses section.", "weaknesses": "- The authors draw some conclusions that are not empirically supported or require more experiments. The reviewer thinks the authors need to tone down their claims and/or cite adequate prior works to justify the claims. For example,\n  - L.145-146: 'While finetuning only a small portion of parameters, it empirically outperforms both a frozen model and a fully finetuned model.' Please show the numbers probably in the appendix.\n  - L.209,212: 'with appropriate domain knowledge, ConvNets can still outperform ViT', 'these observations highlight the task-dependent nature of the strengths exhibited by ConvNets and ViTs.' The reviewer thinks these are rather the nature of algorithms, not feature extractors (which is also just a hypothesis, without any evidence). To draw this conclusion, the authors need to do experiments with the same backbone networks, the same algorithms, and large enough amounts of random seeds. Without those, the reviewer does not think these claims are valid.\n  - L.273-276: 'For the shallow layers, the distributions of the adapted models closely resemble those of the pre-trained models. However, as we move deeper into the layers, noticeable differences emerge. We attribute this disparity to the fact that dissimilarities between human hands and robot hands extend beyond low-level features like color and texture. Instead, they encompass higher-level features such as dynamics and structure.' At the very least, please cite relevant papers that analyze the adaptation between two domains. Especially, it would be very difficult to claim that 'they encompass higher-level features such as dynamics and structure.'. For what reasons did the authors draw this conclusion?\n- Fig.7: It is very hard to understand whether the keypoints are reasonable or not. For example, the bottom red star of the leftmost figure does not look meaningful.\n- Fig.6: Why do the authors specifically choose the three environments, specifically why only (c) is conducted in the 'Place Inside'? Since the differences between the proposed method and the others look minor for all three ablations, the readers can feel the environments can be cherry-picked. The reviewer thinks the authors need to show more results on different environments (can be in the appendix if space does not allow it).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions. We address each of your comments in the following. We omit the questions due to space limits.\n\n**Q1:** L.145-146 .....\n\n**A1:** We have conducted a more comprehensive evaluation on our Stage 2 and the results are given in Figure 2 and Figure 4 of the rebuttal file. It is clearly shown that finetuning all parameters trivially could lead to suboptimal performance (Figure 2) and so is without any finetuning (Figure 4).\n\n**Q2:** L.209,212:.....\n\n**A2:** We would like to clarify that the statements mentioned might be misunderstood a bit by the reviewer, and we wanted to explain more here. \n\n‘with appropriate domain knowledge, ConvNets can still outperform ViT’ With such statements, we are not trying to show ConvNets are better than ViT in general cases. *Instead, we want to show the advantage of our method*, that a feature representation with ConvNet architecture could beat other strong ViT-based methods including MVP and VC-1. \n\n‘these observations highlight the task-dependent nature of the strengths exhibited by ConvNets and ViTs.’ This statement is saying that on different tasks, different architectures show very different results. For example, on *Relocate Tomato Soup Can*, RRL is much better than VC-1, while on *Relocate Mug*, it is the opposite. We agree with the reviewer that the statements are made not clear to fully express our observations, and would be more careful on the claims. \n\n\n**Q3:** L.273-276:.....\n\n**A3:** Previous studies in interpretability have shown that the deeper layers in neural networks have more high-level information. For example, Bau et al [2] show that color and texture concepts dominate at lower layers while more object and part detectors emerge in deeper layers; The visualizations in [3, 4] show the increase in complexity and variation on higher layers, while having simpler components from lower layers. In our experiments, we observe that the adaptation transforms the deep layer features more, thus we make the hypothesis that our method adapts the distribution of human hand features to the distribution of robot hand features, *especially in the higher-level parts*.\n\n**Q4:** Fig.7: ....\n\n**A4:** We apologize for the unclear explanation in the paper and give more explanations here. The term *keypoints* in our study refers to landmarks as defined in [1], which identify the most critical pixels for image reconstruction. As a result, some of these points might be located in unexpected positions. Additionally, the quantity of keypoints is preset (we use n=30 in our paper), implying the potential for redundant or idle points.\n\n**Q5:** Fig.6: .....\n\n**A5:** We acknowledge the insufficiency of the ablations and we want to respectfully emphasize that our ablation results are not cherry-picked; instead, the task is randomly sampled. During the rebuttal phase, we give a much more comprehensive evaluation of our Stage 2 and Stage 3. The results are given in the PDF file and clearly show the advantage of our method across tasks. We are also committed to releasing our codes.\n\n**Q6:** Fig.2: .....\n\n**A6:**  Thank the reviewer for the detailed suggestion on paper improvements. We have updated Figure 2 as suggested. The updated version is also given in the PDF file.\n\n**Q7:** Fig.2 and Sec.4: .....\n\n**A7:** We omit the complex self-supervised learning process in Figure 2 due to space limitation and explain briefly in Section 4.  Here we would like to give more intuitive explanations. More details could be found in [1].\n\nFrom a high perspective, the self-supervised learning task used in our paper is to reconstruct the target image, given a source image and a target image. The target image provides the keypoints, and the source image provides the appearance. For example, if we have a figure with a robot hand and the background (this is the source image), we could provide the new positions of the robot hand (encoded from the target image), and the information in the source image would be sufficient to reconstruct the target image where the robot hand is in the new position. \n\n\n**Q8:** The ratio of parameter changes....\n\n**A8:**  In Stage 2, the learnable parameters (weight, bias) in BatchNorm Layers are updated, which is 0.18% of the entire network. In Stage 3, moving mean and average parameters in BatchNorm are updated, which is also 0.18% of the entire network. These two parts together are 0.36%.\n\n**Q9:** Fig.7: The reviewer cannot see the videos of how keypoints move in the video.\n\n**A9:** The videos are displayed on our anonymous website during the initial submission. It could be correctly played in the Chrome browser of one MacBook or iPhone.\n\n**Q10:** Sec.5.2....\n\n**A10:** Yes. The performance of BC-only policy corresponds to the environment steps being zero.\n\n**Please do not hesitate to let us know if you have any additional comments.**\n\n\n[1] Jakab, Tomas, et al. \"Unsupervised learning of object landmarks through conditional image generation.\" Advances in neural information processing systems 31 (2018).\n\n[2] Bau, David, et al. \"Network dissection: Quantifying interpretability of deep visual representations.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\n[3] Yosinski, Jason, et al. \"Understanding neural networks through deep visualization.\" arXiv preprint arXiv:1506.06579 (2015).\n\n[4] Zeiler, Matthew D., and Rob Fergus. \"Visualizing and understanding convolutional networks.\" Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. Springer International Publishing, 2014.\n"}, {"review_id": "sFQxTIDoUX", "reviewer": "Reviewer_di2q", "summary": "The paper works on the problem of visual RL dexterous manipulation with pre-trained representations. Specifically, the authors propose to fix the pre-trained Convolution layers and adapt only the BatchNorm layers on a small amount of in-domain data, and then learn the policy based on the pre-trained feature with EMA updates. Experiments are performed on 12 simulation tasks.   ", "questions": "In Fig. 6 (b), why would larger or smaller momentums lead to better performance than an intermediate one? Could authors give more theoretical analysis on this?  \n\nIt seems like the standard BatchNorm without EMA already achieves comparable performance. It would be great if more theoretical guidance on choosing the momentum value is provided instead of using empirical study; \n\nFurther clarification or analysis from the authors regarding the significance and contribution of stage 3 would be valuable to better understand its role in achieving improved results;\n\nIn Fig. 8, it would be better to also present the scales for X-axis and Y-axis.", "limitations": "No limitations or failure modes are presented in the paper. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of leveraging large-scale pretraining to boost the performance of visual RL is interesting and well-motivated; \n\nThe proposed method for adapting pre-trained feature for dexterous manipulation is simple and straightforward; \n\nThe paper is well-written and easily read. ", "weaknesses": "For some tasks, the proposed method seems on par or slightly worse than baselines (e.g., Door, Relocate Mug, Relocate Mustard Bottle); \n\nThe necessity of stage 3 in the pipeline can be called into question since the performance of the pipeline without stage 3 appears to be comparable to the performance of the full pipeline in the Hammer task.  \n\nNo limitations or failure modes are presented in the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions. We address each of your comments in the following.\n\n**Q1:** For some tasks, the proposed method seems on par or slightly worse than baselines (e.g., Door, Relocate Mug, Relocate Mustard Bottle);\n\n**A1:** In the field of deep reinforcement learning, it is inherently challenging for a single approach to excel across all tasks. In our experiments, although VC-1 slightly outperforms our method in tasks like Relocate Mug and Relocate Mustard Bottle, it significantly lags behind in others, including Relocate Tomato Soup Can and Relocate Foam Brick. We agree with the reviewer that the pursuit of a universally optimal representation is critical. Nevertheless, our method delivers robust performance across a spectrum of tasks, only equating to baseline levels in a few tasks. We believe the results demonstrate the advantage on both task quantities and average performance.\n\n**Q2:** The necessity of stage 3 in the pipeline can be called into question since the performance of the pipeline without stage 3 appears to be comparable to the performance of the full pipeline in the Hammer task.\n\n**A2:** To further show the functionality of our Stage 3, we conduct a more comprehensive evaluation during the rebuttal phase. Our experiments are conducted across all 7 tasks where the momentum m > 0. The results are given in Figure 3 of the rebuttal PDF file. It is observed that our simple adaptation in Stage 3 is helpful for all these tasks, especially for Relocate Mustard Bottle and Relocate Large Clamp. In our initial submission, due to the computation limit, we do not ablate on all the tasks and only put results on hammer. \n\n**Q3:** No limitations or failure modes are presented in the paper. \n\n**A3:** We have described the limitations in the conclusion section. Our framework also has failure modes. For instance, in Figure 2 of the rebuttal file, on Relocate Box task, removing Stage 2 proves to be more effective than incorporating it. This suggests that merely combining Stage 1 with Stage 3 might be a sufficiently effective approach.\n\n**Q4:** In Fig. 6 (b), why would larger or smaller momentums lead to better performance than an intermediate one? Could authors give more theoretical analysis on this? \n\n**A4:** Intuitively, the larger momentum means that the model values the current and recent data points more and assigns them a larger weight, and this could be suitable to the case where the policy learns the tasks very quickly and thus the incoming data is changing quickly. In our experiments, the methods converge fast on *hammer* compared to other tasks, and thus the momentum might be large (m=0.1). Following these intuitions, we set the momentum by additional considering empirical performance. It could be good if the rigorous theoretical analysis is built and the momentum could be selected based on the theory, which is not the focus of  work but could be an interesting problem for future research.\n\n**Q5:** It seems like the standard BatchNorm without EMA already achieves comparable performance. It would be great if more theoretical guidance on choosing the momentum value is provided instead of using empirical study;\n\n**A5:** See A2 and A4.\n\n**Q6:** Further clarification or analysis from the authors regarding the significance and contribution of stage 3 would be valuable to better understand its role in achieving improved results;\n\n**A6:** See A2.\n\n**Q7:** In Fig. 8, it would be better to also present the scales for X-axis and Y-axis.\n\n**A7:** We update Figure 8 with X and Y axis. The new figure is also given as Figure 5 of the rebuttal file. Thank the reviewer for this suggestion.\n\n**Please do not hesitate to let us know if you have any additional comments.**\n"}]

Author rebuttal:
We thank all the reviewers for their insightful comments. We have addressed all your individual comments. We wanted to thank the reviewer for acknowledging the novelty and the empirical evaluation of our work – “a novel and effective method” (GX9c), “experiments are extensively conducted” (rofX), “interesting and well-motivated” (di2q), “improvements over strong baselines” (s3to). More additional experiments are also conducted during the rebuttal phase to support our proposed method (given in the PDF file), as suggested by the reviewers. 

**EXP1: Ablation on Stage 2 (adapting 0.18% parameters or adapting 100% parameters)** in reply to Reviewer s3to and Reviewer rofX. Results are given in Figure 2. We compare *only adapting 0.18% parameters in Stage 2* (our proposed method) with *adapting all parameters*, across all 12 tasks. It is clearly shown that finetuning all parameters could lead to suboptimal performance and even make agents fail to learn on *Door, Relocate Large Clamp, and Relocate Potted Meat Can*. We also observe that on Relocate Box task, finetuning 100% parameters is surprisingly more effective, but considering the high variance and the low average scores of finetuning 100% parameters across 12 tasks, the advantage of our proposed adaptation is clearer.

**EXP2: Ablation on Stage 3 (m > 0 or m = 0)** in reply to Reviewer s3to and Reviewer rofX, Reviewer di2q. Results are given in Figure 3. We compare setting *m > 0* with *m=0* across all 7 tasks where m is set to be larger than 0. It is observed that adding this simple technique (updating statistics with momentum) during the RL phase could have a large impact. The empirical results emphasize the importance of correct adaptation when applying the pre-trained model. We do not conduct more ablations on the rest 5 tasks that use m=0 originally, since we currently do not find a suitable m\in {0.1, 0.01, 0.001} that could outperform m=0, which means the visual representation without adaption in the RL phase has been good enough.

**EXP3: Ablation on Stage 2 (with or without adaptation)** in reply to Reviewer s3to and Reviewer rofX. In addition to experiments on adapting 100% parameters, we also show more results about not adding Stage 2 in Figure 4. The experiments cover all 6 kinds of tasks in our paper, and we think it is extensive to show the necessity of our Stage 2. It is observed that except Door task (where results are similar), our method with Stage 2 outperforms that without Stage 2 across all other tasks.

**EXP4: Visual Generalization to unseen backgrounds** in reply to Reviewer GX9c. Results are given in Table 2 and visualizations are given in Figure 6. The experiments are conducted on Relocate Potted Meat Can task, where the best baseline VC-1 is slightly better than our H-InDex. However, we show that when the backgrounds change from seen to unseen, VC-1 is distracted more than H-InDex, and H-InDex achieves 475.04 average scores across 9 new scenes while VC-1 has only 392.56 scores. This experiment shows that even after adaptation, our visual representation does not overfit to training data simply and still keeps good generalization ability.

Again, we thank the reviewers for their constructive feedback. We believe that all individual comments have been addressed, but are happy to address any further comments from reviewers.


Meta-review: The paper proposes a three-stage framework for dexterous manipulation using human hand-inspired visual representation learning. The initial concerns from the reviewers include: 1) unclear details of the method and statements, 2) unclear contributions on the three stages, 3) missing ablations on multiple tasks, and 4) marginal improvements of stages 2&3 and inferior performance on certain tasks.
The rebuttal addressed most of these concerns by clarifying the method and providing additional results. In the end, there are two weak accept, 1 borderline accept and 1 borderline reject. The main concern from the two reviewers who gave borderline scores is the suboptimal performance.
The AC considers the proposed method provides some new insights on pretrained representations for dexterous manipulation and the experiments support the claims. 
Therefore, the AC recommends accepting the paper.
The authors should include the new results and discussions according to reviewers' comments in their final version.
Decision: Accept (poster)