SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yanjie Ze $^{12}$  Yuyao Liu $^{3*}$  Ruizhe Shi $^{3*}$  Jiaxin Qin $^{4}$  Zhecheng Yuan $^{31}$  Jiashun Wang $^{5}$  Huazhe Xu $^{316}$
$^{1}$ Shanghai Qi Zhi Institute  $^{2}$ Shanghai Jiao Tong University  $^{3}$ Tsinghua University, IIIS  $^{4}$ Renmin University of China  $^{5}$ Carnegie Mellon University  $^{6}$ Shanghai AI Lab *Equal contribution. Order is decided by coin flip.
yanjieze.com/H-InDex
Human hands possess remarkable dexterity and have long served as a source of inspiration for robotic manipulation. In this work, we propose a human Hand-Informed visual representation learning framework to solve difficult Dexterous manipulation tasks (H-InDex) with reinforcement learning. Our framework consists of three stages: (i) pre-training representations with 3D human hand pose estimation, (ii) offline adapting representations with self-supervised keypoint detection, and (iii) reinforcement learning with exponential moving average BatchNorm. The last two stages only modify  $0.36\%$  parameters of the pre-trained representation in total, ensuring the knowledge from pre-training is maintained to the full extent. We empirically study 12 challenging dexterous manipulation tasks and find that H-InDex largely surpasses strong baseline methods and the recent visual foundation models for motor control. Code is available at yanjieze.com/H-InDex.
Humans can adeptly tackle intricate and novel dexterous manipulation tasks. However, multifingered robotic hands still struggle to achieve such dexterity efficiently. Recent progress in representation learning for visuomotor tasks has proved that pre-trained universal representations may accelerate robot learning manipulation tasks [17, 18, 26, 30]. In light of previous success, the similar morphology between human hands and robotic hands begs the question: can robotic hands leverage representations learned from human hands for achieving dexterity?
In this paper, we propose Hand-Informed visual reinforcement learning framework for Dexterous manipulation (H-InDex) that uses
and adapts visual representations from human hands to boost robotic hand dexterity. Our framework consists of three stages:
Figure 1: Normalized average score for our algorithm H-InDex and the baselines (VC-1 [17], MVP [30], R3M [18], and RRL [26]).
- Stage 1: Pre-training representations with 3D human hand pose estimation, where we adopt the feature encoder from an off-the-shelf 3D hand pose estimator FrankMocap [24].
- Stage 2: Offline adapting representations with self-supervised keypoint detection, where we freeze the convolutional layers in the pre-trained representation and only finetune the affine
transformations in BatchNorm layers (0.18% parameters of the entire model). Such minimal modification of the pre-trained representations ensures that human dexterity is retained to the maximum extent and adapts the human hand representations into the target robotic domain.
- Stage 3: Reinforcement learning with exponential moving average (EMA) BatchNorm and the adapted representations. EMA operates to dynamically update the mean and variance in BatchNorm layers, to further make the model adapt to the progressive learning stages.
In contrast to previous works that also learn representations from human videos [17, 18, 30], there are two major benefits of our framework: i) H-InDex explicitly learn human dexterity by forcing the model to predict the 3D hand pose instead of predicting or discriminating pixels unsupervisedly using masked auto-encoding [9] or time contrastive learning [25]; ii) H-InDex directly adopts the off-the-shelf visual model that is designed to capture human hands rather than training large models on large-scale datasets for specific robotic tasks. These two points combined demonstrate a new cost-effective way to solve robotic tasks such as dexterous manipulation by leveraging existing visual models that are originally and only designed for human understanding.
To show the effectiveness of H-InDex, we experiment on 12 challenging visual dexterous manipulation tasks from Adroit [23] and DexMV [22]. We mainly report episode returns instead of success rates to better show how well the robots solve the tasks. In comparison with several strong visual foundation models for motor control, H-InDex largely surpasses all of them as shown in Figure 1.
To summarize, our contributions are three-fold:
- We propose a novel visual reinforcement learning framework called H-InDex to utilize rich human hand information efficiently for dexterous manipulation.
- We show the effectiveness of our framework on 12 challenging visual dexterous manipulation tasks, comparing with recent strong foundation models such as VC-1 [17].
- Our study has offered valuable insights into the application of pre-trained models for dexterous manipulation, by exploring the direct application of a 3D human hand pose estimation model, originating from the vision community.
Visual reinforcement learning for dexterous manipulation. Recent research has explored the use of deep reinforcement learning (RL) for solving dexterous manipulation tasks [7, 21, 23, 26, 29]. For example, Rajeswaran et al. [23] investigated the use of vector state information as input to the RL algorithm. Despite the success, assuming access to the ground-truth state limits its possibility to be deployed in the real world. RRL [26] finds that ImageNet pre-trained ResNets [10] are surprisingly effective in achieving dexterity with visual observations. Under the umbrella of visual RL, MoDem [7] leverages a learned dynamics model to solve the tasks with good utilization of demonstrations. Furthermore, VRL3 [29] utilizes offline RL to pre-train the visual representations and the policies in an end-to-end manner. In this work, H-InDex is designed to focus on visual representations while leaving the policy, training framework, and reward signals unchanged. As a result, H-InDex offers an orthogonal and complementary approach to prior efforts in this area.
Foundation models for visuo-motor control. Given the diversity of robotic tasks and computational constraints, there is a growing interest in developing a single visual foundation model that can serve as a general feature extractor. Such a model would enable the processing of high-dimensional visual observations into compact vectors, providing a promising approach for efficient and effective control of a wide range of robotic systems [8, 17-19, 26, 30, 32, 33]. Among them, R3M [18] pre-trains a ResNet-50 on Ego4D [6] dataset and evaluates on several robotic manipulation tasks with imitation learning. MVP [30] pre-trains vision transformers [4] with Masked AutoEncoder (MAE) [9] on internet-scale data, achieving strong results on dexterous manipulation tasks. Similarly, a very recent foundation model VC-1 [17] explores the scaling up of MAE for motor control and achieves consistently strong results across a wide range of benchmarks. However, it should be noted that VC-1 and R3M only employ IL to solve dexterous manipulation tasks, making it unclear whether these models are suitable for the setting of reinforcement learning, where agents need to trade off between exploration and exploitation. GNFactor [33] distills the 2D foundation models into 3D space, but their agents are limited to address the gripper-based manipulation problems.
Learning dexterity from videos. A growing body of recent research aims to leverage human manipulation videos for improving visuomotor control tasks [17,20,22,27,28,30]. A line of works focuses on directly extracting human hand poses from videos and employing RL/IL to train on the retargeted robot joint positions, such as DexMV [22], Robotic telekinesis [28], VideoDex [27], and Imitate Video [20]. In contrast to these approaches, our work explores a representation learning approach for leveraging online human-object interaction videos without explicit pose estimation to improve dexterity in robotic manipulation, sharing a similar motivation as MVP [30] and VC-1 [17].
**Formulation.** We model the problem as a Markov Decision Process (MDP)  $\mathcal{M} = \langle S, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$ , where  $\mathbf{s} \in S$  are states,  $\mathbf{a} \in \mathcal{A}$  are actions,  $\mathcal{T}: S \times \mathcal{A} \mapsto S$  is a transition function,  $r \in \mathcal{R}$  are rewards, and  $\gamma \in [0,1)$  is a discount factor. The agent's goal is to learn a policy  $\pi_{\theta}$  that maximizes discounted cumulative rewards on  $\mathcal{M}$ , i.e.,  $\max_{\theta} \mathbb{E}_{\pi_{\theta}}\left[\sum_{t=0}^{\infty} \gamma^{t} r_{t}\right]$ , while using as few interactions with the environment as possible, referred as sample efficiency.
In this work, we focus on visual RL for dexterous manipulation tasks, where actions are high-dimensional  $(\mathbf{a} \in \mathcal{A}^{30})$  and ground-truth states  $s$  are generally unknown, approximated by image observations  $\mathbf{o} \in \mathcal{O}$  together with robot proprioceptive sensory information  $\mathbf{q} \in \mathcal{Q}$ , i.e.,  $\mathbf{s} = (\mathbf{o}, \mathbf{q})$ . To better address the hard exploration problem in high-dimensional control [7,23,26,29], we assume access to a limited number of expert demonstrations  $\mathcal{D}_{\mathrm{expert}} = \{D_1, D_2, \dots, D_N\}$ .
**Demo Augmented Policy Gradient (DAPG) [23] is a model-free policy gradient method that utilizes given demonstrations to augment the policy and trains the policy with natural policy gradient (NPG) [13]. DAPG mainly consists of two stages:
(1) Pre-training the policy with behavior cloning, which is to solve the following maximum-likelihood problem:
$$
\underset {\theta} {\text {m a x i m i z e}} \sum_ {(s, a) \in \mathcal {D} _ {\text {e x p e r t}}} \ln \pi_ {\theta} (a \mid s). \tag {1}
$$
(2) RL finetuning with the demo augmented loss, which is to add an additional term to the gradient,
$$
g _ {\text {a u g}} = \underbrace {\sum_ {(s , a) \in \mathcal {D} _ {\pi}} \nabla_ {\theta} \ln \pi_ {\theta} (a \mid s) A ^ {\pi} (s , a)} _ {\text {o r i g i n a l g r a d i e n t}} + \underbrace {\sum_ {(s , a) \in \mathcal {D} _ {\text {e x p e r t}}} \nabla_ {\theta} \ln \pi_ {\theta} (a \mid s) w (s , a)} _ {\text {d e m o a u g m e n t e d g r a d i e n t}}, \tag {2}
$$
where  $A^{\pi}(s,a)$  is the advantage function,  $\mathcal{D}_{\pi}$  represents the dataset obtained by executing policy  $\pi_{\theta}$  on the MDP, and  $w(s,a)$  is a weighting function.
In this work, our goal is to achieve sample efficient visual reinforcement learning agents in dexterous manipulation tasks by incorporating human hand dexterity into visual representations. To this end, we propose Hand-Informed visual reinforcement learning for Dexterous manipulation (H-InDex), a simple yet effective learning framework to address the contact-rich dexterous manipulation problems effectively in limited interactions. The overview of our method is provided in Figure 2. H-InDex consists of three stages: 1) a representation pre-training stage where we pre-train the visual representations with the 3D human hand pose estimation task, aiming to make visual representations understand human hand dexterity from diverse natural videos; 2) a representation offline adaptation stage where we adapt only  $0.18\%$  parameters in the pre-trained representation with the self-supervised keypoint objective with in-domain data; 3) a reinforcement learning stage where the visual representation is frozen and we utilize the exponential moving average operation to update the mean and variance in BatchNorm of the visual representations.
Stage 1: Representation pre-training. We start by pre-training visual representations with monocular 3D human hand pose estimation, which is a well-established human hand understanding task in the computer vision community with large-scale annotated datasets available. Together with the datasets, there are a plethora of open-sourced models from which we use an off-the-shelf model FrankMocap [24]. FrankMocap is a whole-body pose estimation system with the hand module trained
(1) Pre-train representation with 3D human hand pose estimation
(2) Offline adapt representation with self-supervised keypoint detection
(3) Reinforcement learning with compact representation and EMA BatchNorm
Figure 2: The overview of H-InDex. H-InDex consists of three stages: 1) representation pre-training, 2) representation offline adaptation, and 3) reinforcement learning.
on 6 diverse hand datasets, totaling 400k samples. We adopt the ResNet-50 [10] feature encoder in the hand module to extract visual representations.
The use of a pre-trained model from the 3D hand pose estimation task [24] shares the intuition with recent works on foundation models for motor control [16-19, 30]: learning representations from human manipulation videos. However, the use of the pre-trained hand model offers two distinct advantages that are not typically found in other approaches:  $i$  ) the model explicitly predicts the hand-related information from diverse videos, forcing it to learn the interaction and the movement of human hands; ii) the model can be borrowed from vision community without any extra cost to re-train a foundation model.
Stage 2: Representation offline adaptation. In the previous stage, we only pre-train visual representations that are suitable for human-centric images, neglecting the morphology and structure gap between robot hands and human hands. To bridge the gap without losing the information learned in the pre-training stage, we adopt a self-supervised keypoint detection objective [11, 14, 15] to only finetune the affine transformations in the BatchNorm layers of the pre-trained model, which occupy only  $0.18\%$  of the entire model parameters. While finetuning only a small portion of parameters, it empirically outperforms both a frozen model and a fully finetuned model. We hypothesize that this is because the BatchNorm finetuning bridges the gap and mitigates catastrophic forgetting caused by finetuning [1].
We now describe the self-supervised keypoint objective. Given a target image  $I_{t}$  and a source image  $I_{s}$  sampled from a video, we aim to reconstruct  $I_{t}$  with the appearance feature of  $I_{s}$  and the keypoint feature of  $I_{t}$ . Denote our pre-trained visual representation as  $h_{\theta}$ , the keypoint feature extractor as  $\mathcal{K}_{\psi}$ , the appearance feature extractor as  $\mathcal{F}_{\phi}$ , and the image decoder as  $\mathcal{G}_{\omega}$ . First, we extract a semantic
Figure 3: Visualization of our six kinds of dexterous manipulation tasks and one sampled trajectory. We depict both the initial configuration and the goal. Videos of trajectories for all tasks are available on our website yanjieze.com/H-InDex.
feature map  $h_{\theta}(I_t)$  from the target image  $I_t$  and then get the keypoint feature  $\mathcal{K}_{\psi}(h_{\theta}(I_t))$ . At the same time, we extract the appearance feature  $\mathcal{F}_{\phi}(I_s)$  from the source image  $I_s$ . We then try to reconstruct the target image  $I_t$  by decoding the concatenated keypoint feature and the appearance feature as  $I_t' = \mathcal{G}_{\omega}(\mathcal{K}_{\psi}(h_{\theta}(I_t)), \mathcal{F}_{\phi}(I_s))$ . Our final supervision is the perceptual loss [12]  $\mathcal{L}_{\text{percep}}$ ,
$$
\mathcal {L} _ {\text {k e y p o i n t}} = \mathcal {L} _ {\text {p e r c e p}} \left(I _ {t}, I _ {t} ^ {\prime}\right) = \| \Lambda (I _ {t}) - \Lambda \left(\mathcal {G} _ {\omega} \left(\mathcal {K} _ {\psi} \left(h _ {\theta} (I _ {t})\right), \mathcal {F} _ {\phi} (I _ {s})\right)\right) \| _ {2} ^ {2}, \tag {3}
$$
where  $\Lambda$  is the semantic feature prediction function in [12].
Stage 3: Reinforcement learning. During the reinforcement learning stage, the distribution of observations is continually changing. For example, in the early learning stage, the observations are usually random explorations, while at the end of the learning stage, most of the observations are converged trajectories. Such a property of reinforcement learning requires the internal statistics of neural networks to move slowly towards the current observation distribution. Therefore, we utilize the exponential moving average (EMA) operation to dynamically update the statistics (i.e., the running mean and the running variance) in BatchNorm layers.
Formally, for the input  $x$  that has  $k$  dimensions, i.e.,  $x = \{x^{(1)},\dots ,x^{(k)}\}$ , we update the running mean  $\mu^{(i)}$  and the running variance  $(\sigma^{(i)})^2$  in BatchNorm layers with the following equation,
$$
\mu^ {(i)} \leftarrow (1 - m) \cdot \mu^ {(i)} + m \cdot \mathbb {E} \left[ x ^ {(i)} \right], \tag {4}
$$
$$
\left(\sigma^ {(i)}\right) ^ {2} \leftarrow (1 - m) \cdot \left(\sigma^ {(i)}\right) ^ {2} + m \cdot \operatorname {V a r} \left[ x ^ {(i)} \right], \tag {5}
$$
for  $i = 1, \dots, k$ , where  $m$  is the momentum. When  $m$  is set to 0, our EMA BatchNorm layers revert back to the original layers, ensuring that the modification does not have negative impacts at the very least. Finally, all these three stages collectively contribute to our final method H-InDex. We remain implementation details in Appendix A.
In this work, we delve into the application of visual reinforcement learning to address dexterous manipulation tasks, with a particular emphasis on the visual representation aspect. We evaluate the effectiveness of our proposed framework, H-InDex, across various tasks and elucidate the significance of each component in achieving the final results. Of particular importance is the integration of prior knowledge pertaining to human hand dexterity into our framework.
Figure 4: Episode return for 12 challenging dexterous manipulation tasks. We compare H-InDex with four strong visual representations for motor control, i.e., VC-1 [17], MVP [30], R3M [18], and RRL [26]. Mean of 3 seeds with seed number 0, 1, 2. Shaded area indicates  $95\%$  confidence intervals.
We evaluate H-InDex on 12 challenging visual dexterous manipulation tasks from Adroit [23] and DexMV [22] respectively, including Hammer, Door, Pen, Pour, Place Inside, and Relocate YCB Objects (7 different objects [3]). Visualization of each task is given in Figure 3 and detailed descriptions are provided in Appendix B. This selection of tasks is the most extensive compared to previous works e.g., DAPG (4 tasks) and DexMV (7 tasks), thereby showcasing the robustness and versatility of H-InDex. The tasks were performed with varying numbers of steps based on their level of complexity. We mainly report the cumulative rewards to show the speed of task completion. The dimension of image observations  $\mathbf{o} \in \mathcal{O}$  is  $3 \times 224 \times 224$  across all methods. We run experiments on an RTX 3090 GPU; each seed takes roughly 12 hours. Due to the limitation on computation resources, we choose to run 3 seeds for each group of experiments and consistently use 3 seeds with seed numbers 0, 1, 2 to ensure reproducibility. We also observe that H-InDex enjoys a slight variance between seeds, while baselines tend to have a larger variance.
5.2 Main Experiments
To demonstrate the effectiveness of H-InDex, we evaluate diverse recent strong visual representations for motor control, including (i) VC-1 [17], which trains masked auto-encoders over 5.6M images with over 10,000 GPU-hours and we use the ViT-B [4] model (86M parameters); (ii) MVP [30], which also uses masked auto-encoders for pre-training and we use the ViT-S [4] model (22M parameters); (iii) R3M [18], which pre-trains a ResNet-50 (22M parameters) with time contrastive learning and video-language alignment (iv) RRL [26], which uses the ResNet-50 pre-trained on the ImageNet classification task directly. Due to task differences, we normalize the cumulative rewards based on the highest rewards achieved and present the average scores in Figure 1. We also report the learning curves in Figure 4. We then detail our observations below.
H-InDex emerges as the dominant representation. Across 12 tasks, H-InDex outperforms the recent state-of-the-art representation VC-1 by a  $16.8\%$  absolute improvement. Furthermore, H-InDex surpasses RRL, the original state-of-the-art representation in Adroit, by  $25.4\%$ . Analyzing the learning curves, H-InDex demonstrates superior sample efficiency in 10 out of the 12 tasks. In
only two tasks, namely relocate mug and relocate mustard bottle, VC-1 exhibits a slight advantage over H-InDex.
ConvNets v.s. ViTs. Among representations utilizing the ResNet-50 architecture (i.e., H-InDex, R3M, RRL), only H-InDex showcases obvious advantages over ViT-based representations. This suggests that with appropriate domain knowledge, ConvNets can still outperform ViTs. Additionally, we notice that ConvNets and ViTs excel in different tasks. For instance, in relocate tomato soup can, VC-1 and MVP achieve returns that are only half of what H-InDex and RRL accomplish. However, in relocate mug, VC-1 performs well. These observations highlight the task-dependent nature of the strengths exhibited by ConvNets and ViTs.
5.3 The Effectiveness of 3D Human Hand Prior
The significance of transferring the 3D human hand knowledge into dexterous manipulation is non-negligible. To demonstrate the utility of such 3D human hand prior, we compare our vanilla pre-trained representation i.e., the feature extractor from the FrankMocap hand module [24] (denoted as FrankMocap Hand) with other 4 representative pre-trained models: (i) FrankMocap Body, which is the body estimation module from FrankMocap [24], pre-trained with 3D body pose estimation, (ii) AlphaPose [5], which is a widely-used robust 2D human pose estimation algorithm, (iii) R3M [18], which is pre-trained with time contrastive learning [25] and language-video alignment on Ego4D [6], and (iv) RRL [26], which directly uses the ResNet-50 pre-trained on the ImageNet classification task. All the models use a ResNet-50 architecture and do not use any adaptation, ensuring the fairness of our comparison. We also put H-InDex as the best results achieved for comparison. Results are shown in Figure 5.
We now detail our observations below:
FrankMocap hand v.s. RRL/R3M. Our vanilla representation has significantly outperformed both RRL and R3M without the need for any adaptation.
FrankMocap hand v.s. 3D/2D body-centric representations. Our vanilla 3D hand representation, FrankMocap Hand, demonstrates superior sample efficiency compared to FrankMocap Body and significantly outperforms AlphaPose. This confirms our hypothesis that the 3D human hand prior is more advantageous than both the 3D and 2D human body prior. It is worth noting that AlphaPose, being a whole-body 2D pose estimation model, is also capable of estimating 2D hand poses. The fact that our 3D hand prior outperforms the 2D hand prior in this context further supports its effectiveness. We hypothesize this is
because 2D pose estimation does not require deep spatial reasoning compared to 3D pose estimation.
Figure 5: Compare vanilla pre-trained representations with H-InDex.
H-InDex v.s. FrankMocap hand. Our vanilla hand representation already surpasses all other pre-trained ConvNets in performance. However, by applying our adaptation technique, we can further enhance the sample efficiency of the hand representation, underscoring the significance of adapting the model with in-domain data in a proper way.
To validate the rationale behind the design choices of H-InDex, we performed a comprehensive set of ablation experiments.
Effects of each stage. Figure 6a provides insights into the contributions of each stage towards the overall efficiency of H-InDex. We refer to RRL as w/o Stage 1,2,3. Significantly, Stage 1 exhibits the most notable enhancement, underscoring the efficacy of human dexterity. Moreover, Stage 2 and Stage 3 also contribute appreciable advancements. In addition, the value of momentum  $(m)$  in Stage 3 has a significant influence, as illustrated in Figure 6b. To determine the optimal value, we performed a grid search over  $m \in \{0,0.1,0.01,0.001\}$ . This analysis highlights the importance of selecting an appropriate momentum value for achieving optimal performance in Stage 3 of H-InDex.
(a) Ablation on the effectiveness of our three stages.