SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similar to  $g_{\theta}(\cdot), h_{\omega}(\cdot, \cdot)$  is composed of two sub-networks $^2$ $e_{\omega_1}(\cdot, \cdot)$  and  $t(\cdot)_{\omega_2}$  parameterized by the weights  $\omega = \{\omega_1, \omega_2\}$ . As shown in the lower branch of Fig. 2, our query encoder  $e_{\omega_1}(\cdot, \cdot): \mathbb{R}^{T \times W \times H \times 3} \times \mathbb{R}^3 \to \mathbb{R}^{T \times h'}$  extracts  $h'$ -dimensional query feature per time step. For each  $t$ , we create an initial query descriptor  $\mathbf{f}^t \in \mathbb{R}^h$  by concatenating (1) a canonical query position  $\mathbf{p}$ , (2) a query position after applying the pose flow  $\Phi_{\theta}^{pose}(\mathbf{p}, t)$  and its corresponding pixel-aligned feature [35], and (3) a skinning weight vector  $w(\mathbf{p})$ . This query descriptor is fed to an MLP-based query encoder to extract a query feature  $\mathbf{f}'^t \in \mathbb{R}^{h'}$ . Our temporal encoder  $t_{\omega_2}(\cdot): \mathbb{R}^{T \times h'} \to \mathbb{R}^{6N + 3}$  then applies temporal convolutions to the per-time query features to predict the Fourier coefficients  $\mathbf{c}^{\mathbf{p}}$  representing the displacement flow of query  $\mathbf{p}$ . Finally, our shape flow  $\Phi_{\omega}^{shape}(\cdot, \cdot): \mathbb{R}^3 \times [0,1) \to \mathbb{R}^3$  for a query point  $\mathbf{p}$  is modeled by Eq. (1) using the query-wise Fourier coefficients  $\mathbf{c}^{\mathbf{p}}$  predicted by  $h_{\omega}(\cdot, \cdot)$ .
Our final flow  $\Phi_{\theta, \omega}(\cdot, \cdot): \mathbb{R}^3 \times [0,1) \to \mathbb{R}^3$  for a query point  $\mathbf{p}$  is obtained as a sum of the learned pose and shape flows, where both of them are represented by the Fourier series to enforce smooth query dynamics along the temporal axis:
$$
\Phi_ {\theta , \omega} (\mathbf {p}, t) = \Phi_ {\theta} ^ {p o s e} (\mathbf {p}, t) + \Phi_ {\omega} ^ {s h a p e} (\mathbf {p}, t). \tag {5}
$$
<sup>1</sup>Since the initial per-frame joints are discretized in time, their Fourier series coefficients are approximated by the numerical integration as discussed in [9]. We use trapz function in PyTorch [31] library.
2 Although the input encoder and the temporal encoder in Eq. (2) and Eq. (4) have different network architectures, we use the same notations  $e(\cdot)$  and  $t(\cdot)$  for brevity.
Loss functions. Our network is trained using the (1) occupancy loss and (2) correspondence loss. First, the occupancy loss penalizes the deviation between the reconstructed and ground truth occupancy probabilities:
$$
\mathcal {L} _ {o c c} (\theta , \omega) = \frac {1}{| \mathcal {X} _ {o c c} |} \sum_ {\mathbf {p}, t, \mathcal {I} \in \mathcal {X} _ {o c c}} \| \hat {\partial} _ {\theta , \omega} (\mathbf {p}, t, \mathcal {I}) - o _ {g t} (\mathbf {p}, t, \mathcal {I}) \| _ {2}, \tag {6}
$$
where  $\mathcal{X}_{occ}$  is a set of training examples and  $\mathcal{I}$  is a sequence of RGB inputs.  $\hat{o}_{\theta,\omega}(\mathbf{p},t,\mathcal{I})$  and  $o_{gt}(\mathbf{p},t,\mathcal{I})$  denote the reconstructed and ground truth occupancy probabilities, respectively. Note that  $\hat{o}_{\theta,\omega}(\mathbf{p},t,\mathcal{I})$  is obtained by propagating the pre-trained canonical occupancy at  $\mathbf{p}$  to the corresponding query position at  $t$  using the query flow learned by our network. Since the canonical occupancy field is pre-trained and fixed, this occupancy loss enforces our flow network to learn query trajectories that better propagate the canonical occupancies  $s.t.$  the occupancies propagated to  $t$  better resemble the ground truth 3D shape at  $t$ .
Second, our correspondence loss penalizes the deviation between the reconstructed query position  $\hat{p}_{\theta,\omega}(\mathbf{p},t,\mathcal{I}) \coloneqq \Phi_{\theta,\omega}(\mathbf{p},t|\mathcal{I})$  and the ground truth query position  $p_{gt}(\mathbf{p},t,\mathcal{I})$  corresponding to  $\mathbf{p}$  given a set of training examples  $\mathcal{X}_{corr}$ :
$$
\mathcal {L} _ {\text {c o r r}} (\theta , \omega) = \frac {1}{| \mathcal {X} _ {\text {c o r r}} |} \sum_ {\mathbf {p}, t, \mathcal {I} \in \mathcal {X} _ {\text {c o r r}}} \| \hat {p} _ {\theta , \omega} (\mathbf {p}, t, \mathcal {I}) - p _ {g t} (\mathbf {p}, t, \mathcal {I}) \| _ {2}. \tag {7}
$$
Our final training objective is to minimize  $\mathcal{L}(\theta, \omega) = \mathcal{L}_{occ}(\theta, \omega) + \lambda \mathcal{L}_{corr}(\theta, \omega)$ , where  $\lambda$  is a hyper-parameter to balance the influence between the two loss terms. For more details on network training and architecture, please refer to the supplementary section.
Datasets and evaluation metrics. We use the two-hand (TH) and single-hand (SH) subsets of 30 FPS version InterHand2.6M [26] dataset, which contains diverse hand motions captured in RGB sequences with dense shape annotations. For each subset, we use samples annotated as valid hand type and follow the train/val/test splits of the original InterHand2.6M dataset. The resulting TH subset contains 477K training and 4K validation sequences, and SH subset contains 656K training and 5K validation sequences. To maintain consistency with the existing 4D continuous representations [29, 14, 15], we use sub-sequences of  $T = 17$  frames sampled from the original sequences as inputs to our method. For testing, we use 2K sub-sequences (34K frames) randomly sampled from the test sequences of each subset. For qualitative evaluation, we additionally show our results on RGB2Hands [41] real dataset, which contains sequences of RGB hand motions without shape annotations. For evaluation metrics on 4D reconstruction, we use mean Intersection over Union (IoU) and Chamfer L1-Distance (CD) computed w.r.t. the ground truth shapes of InterHand2.6M dataset. We also use L1 Correspondence Error (L1-Corr) for evaluating our learned shape correspondences.
Compared methods. As our method is the first RGB-based 4D continuous hand representation, there is no direct baseline. We thus compare ours to (1) RGB-based 3D hand reconstruction methods and (2) 4D implicit reconstruction methods modified to take RGB hands as inputs. For 3D hand reconstruction methods, we consider state-of-the-art methods on InterHand2.6M: ACR* [44], Two-Hand-Shape-Pose [45], IntagHand [19], and Im2Hands [18]. Note that ACR* [44] results are produced using the pre-trained ACR model officially released for in-the-wild demo. As these methods are proposed for two-hand reconstruction, we compare them on TH subset. For 4D representations, we consider (1) Occupancy Flow, which is the only 4D continuous representation that has shown results from RGB videos, and (2) LoRD, which is the most recent state-of-the-art 4D articulated implicit function proposed for human bodies. As LoRD originally takes SMPL [21] meshes fitted to 2.5D or 3D point cloud inputs, we have implemented a modified version  $(\mathrm{LoRD}^{\dagger})$  that uses MANO [34]-topology hand meshes reconstructed from RGB frames using IntagHand [19] to make direct comparisons. We compare these 4D implicit methods on both TH and SH subsets. For the off-the-shelf hand pose estimator  $\Psi(\cdot)$  in our method, we adopt the joint estimation module of IntagHand [19] as in [18].
Two-hand extension. Our hand representation can be naturally extended to estimate 4D two-hand shapes to make comparisons on InterHand2.6M [26] TH subset. In a nutshell, we use two joint
flow networks each trained for left and right hands, respectively. The shape flow network is shared among both side of hands to implicitly capture correlation between them. Note that left and right side conditioning for shape flow estimation is done by taking as inputs a query position after applying the pose flow w.r.t. the corresponding side of hand. Please find more details in the supplementary.
4.1 Video-Based 4D Hand Reconstruction
In Tab. 1, our method achieves state-of-the-art reconstruction results on InterHand2.6M [26]. The existing 3D reconstruction methods (3D Mesh and 3D Implicit categories) perform shape reconstruction for each frame independently. Thus, they often produce shapes with temporal jitters and do not reason about hand motions (e.g., motion inter- and extrapolation). In 4D Implicit category, Occupancy Flow [29] achieves sub-optimal reconstruction quality due to the lack of shape articulation prior. While  $\mathrm{LoRD}^{\dagger}$  [14] achieves better performance, it does not fully leverage pixel-aligned features for shape reconstruction. In Fig. 3, we also qualitatively show the reconstruction results, where ours produces the most plausible shapes that are also well-aligned to the input RGB observations.
Table 1: Quantitative comparisons of video-based 4D hand reconstruction on InterHand2.6M [26] dataset. Our method achieves state-of-the-art results on both TH and SH subsets.
Subset	Category	Method	IoU (%) ↑	CD (mm) ↓
TH	3D Mesh	ACR* [44]	45.2	7.89
Two-Hand-Shape-Pose [45]	48.7	6.58
IntagHand [19]	57.5	5.27
3D Implicit	Im2Hands [18]	61.8	4.59
4D Implicit	Occupancy Flow [29]	32.4	18.27
LoRD† [14]	58.0	4.89
FOURIERHANDFLOW (Ours)	62.8	4.46
SH	4D Implicit	Occupancy Flow [29]	44.7	13.21
LoRD† [14]	63.3	4.38
FOURIERHANDFLOW (Ours)	65.8	3.90
Figure 3: Qualitative comparisons of video-based 4D hand reconstruction on InterHand2.6M [26] TH subset. Please also refer to our supplementary for full-length video-based reconstruction comparisons, where ours achieves more favorable results (e.g. without temporal jitters).
In Tab. 2 (columns 3-4), we also compare training and inference time between the implicit representations. Note that we use InterHand2.6M TH subset hereinafter unless specified. Im2Hands [18] is slow in both training and inference due to the global shape encoding and decoding step for shape refinement. While  $\mathrm{LoRD}^{\dagger}$  [14] is the fastest in training, its inference is slow due to the test-time model optimization and per-time occupancy decoding. Occupancy Flow [29] achieves fast inference thanks to its simple network architecture, but its efficiency is still sub-optimal due to the use of an ODE solver [5]. Our method achieves the fastest inference time using the Fourier query flow representation. In Tab. 2 (column 5), we also compare the shape correspondence error between the implicit hand functions. Note that Im2Hands [18] and  $\mathrm{LoRD}^{\dagger}$  [14] do not capture temporal shape correspondences due to per-time occupancy decoding. Compared to Occupancy Flow [29], we learn more accurate shape correspondences by articulation-aware query flow modeling.
Table 2: Training / inference time and learned correspondence comparisons among implicit representations. Training time is measured for each training iteration with a batch size of 1 and inference time is measured for each frame to make comparisons with the frame-based method [18]. Each time result is obtained as an average of 1K measurements.
Category	Method	Training Time (sec.) ↓	Inference Time (sec.) ↓	L1-Corr (mm) ↓
3D Implicit	Im2Hands [18]	3.31	6.62	N/A
4D Implicit	Occupancy Flow [29]	6.15	0.26	16.8
LoRD† [14]	0.15	114.72	N/A
FOURIERHANDFLOW (Ours)	2.75	0.22	10.8
In addition, we compare our learned joint flow with the noisy per-frame joints estimated by  $\Psi(\cdot)$  [19]. In Tab. 3, our joint flow network is shown to be effective in refining the initial per-frame joint predictions using hints from the input RGB frames and hand skeleton graph structures.
Table 3: Joint refinement results on Mean Per Joint Position Error (MPJPE).
Method	MPJPE (mm) ↓
Before Ref.	13.3
After Ref.	11.0
4.2 Additional Experimental Results
Texture transfer. In Fig. 4, we show our results on texture transfer using the learned correspondences of implicit shapes. Given the reference texture defined in our canonical hand field (Sec. 3.1), we achieve high-quality texture transfer results on the hand reconstructions at randomly sampled sequences and time stamps. Note that, as our canonical hand field is shared among all sequences (cf. [29]), we can naturally obtain dense correspondences between inter-sequence hand reconstructions.
Figure 4: Texture transfer results using the learned correspondences between our hand reconstructions. Best viewed with  $300\%$  zoom-in.
Motion inter- and extrapolation. In Fig. 5, we show our motion inter- and extrapolation results. Given RGB frames observed at each time step, we sample hand shapes at inter- and extrapolated time values from the learned Fourier query flows. Our sampled shapes are shown to model smooth temporal evolution of hand shapes.
Figure 5: Our motion inter- and extrapolation results. Given RGB frames observed at each time step, our method can sample shapes at inter- and extrapolated time values.
Generalization test. In Fig. 6, we show additional qualitative results on RGB2Hands [41] real images using our model trained on InterHand2.6M [26] dataset. Our model is shown to produce plausible reconstructions even from unseen RGB2Hands images, demonstrating its generalization ability.
Figure 6: Generalization test on unseen RGB2Hands [41] real images.
In Tab. 4, we compare video-based 4D hand reconstruction results among the variations of our method. w/o Pose Flow and w/o Shape Flow (rows 2-3) denote our method variations in which pose or shape flow is not modeled, respectively. It is shown that each type of flow contributes to more accurate reconstruction, thus the best accuracy is achieved when both flows are used. Fourier  $\rightarrow$  ODE and Fourier  $\rightarrow$  InputCond (rows 4-5) are variations in which (1) query flows are modeled by solving ODE [5] and (2) query positions are directly estimated conditioned on a time value, respectively. Our method based on Fourier query flow achieves the best performance, showing the effectiveness of flow estimation in the frequency domain (cf. spatial domain).
Table 4: Quantitative comparisons of video-based 4D hand reconstruction among the variations of our method on InterHand2.6M [26] TH subset. Our proposed method achieves the best results.
Method	IoU (%) ↑	CD (mm) ↓	L1-Corr (mm) ↓
w/o Pose Flow	50.1	7.86	20.9
w/o Shape Flow	57.9	5.06	12.1
Fourier → ODE	60.4	4.93	11.5
Fourier → InputCond	61.0	4.87	11.0
FOURIERHANDFLOW (Ours)	62.8	4.46	10.8
We present FOURIERHANDFLOW, which is the first neural 4D continuous representation for human hands learned from RGB videos. We combine a continuous 3D hand occupancy field together with query flows learned as Fourier coefficients to model smooth and continuous temporal shape dynamics. To enable articulation-aware shape modeling, we introduce two types of Fourier query flow: (1) pose flow based on implicit linear blend skinning and (2) shape flow that models query-wise displacements. Our method achieves state-of-the-art results on video-based 4D hand reconstruction while being computationally more efficient than the existing implicit function-based methods.
Broader impact. Our method can be potentially used for video-based 4D reconstruction of articulated implicit shapes in general (e.g. human or animal bodies), which is useful for AR/VR applications.
Limitations. Although our method is more efficient than the state-of-the-art 3D and 4D implicit functions, its inference time is not yet sufficient to provide a real-time solution (cf. low-resolution mesh-based reconstruction methods [19, 44]). We plan to investigate ways to further improve our computational efficiency, e.g., by adapting frequency-domain representation additionally along the spatial domain. Also, while Fourier query flow naturally models smooth temporal deformations, it constrains our query flow estimation on the low-mid frequency subspace. Thus, it may not model hand motions with actual temporal jitters, although they do not commonly occur. In addition, our method currently takes a fixed length of  $T = 17$  sub-sequence as inputs following Occupancy Flow [29]. We will try to further address these limitations as our future work.
This work was in part sponsored by NST grant (CRC 21011, MSIT), KOCCA grant (R2022020028, MCST) and IITP grant (2019-0-00075, MSIT). Minhyuk Sung acknowledges support from the NRF grant (RS2023-00209723) and IITP grants (2022-0-00594, RS-2023-00227592) funded by the Korean government (MSIT), the Technology Innovation Program (20016615) funded by the Korean government (MOTIE), as well as grants from ETRI, KT, NCSOFT, and Samsung Electronics.
[39] Tuan-Anh Vu, Duc Thanh Nguyen, Binh-Son Hua, Quang-Hieu Pham, and Sai-Kit Yeung. Rfnet-4d: Joint object reconstruction and flow estimation from 4d point clouds. In ECCV, 2022.
[40] Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. Dual grid met: Hand mesh vertex regression from single depth maps. In ECCV, 2020.
[41] Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt. RGB2hands: Real-time tracking of 3d hand interactions from monocular rgb video. ACM TOG, 2020.
[42] John Yang, Hyung Jin Chang, Seungeui Lee, and Nojun Kwak. Seqhand: RGB-sequence-based 3d hand pose and shape estimation. In ECCV, 2020.
[43] Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In CVPR, 2022.
[44] Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P Breckon, and Jue Wang. Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In CVPR, 2023.
[45] Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3d pose and shape reconstruction from single color image. In ICCV, 2021.
[46] Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang. A hand pose tracking benchmark from stereo matching. In ICIP, 2017.
[47] Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In ICCV, 2019.