FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flow
Authors: Jihyun Lee
Junbong Jang
Donghwan Kim
Minhyuk Sung
Tae-Kyun Kim
Venue: NeurIPS 2023 poster (2023)
Keywords: 4D representation
hand reconstruction
implicit representation
TL;DR: We present FourierHandFlow, which is a spatio-temporally continuous representation for human hands that combines a 3D occupancy field with articulation-aware query flows represented as Fourier series.

Abstract: Recent 4D shape representations model continuous temporal evolution of implicit shapes by (1) learning query flows without leveraging shape and articulation priors or (2) decoding shape occupancies separately for each time value. Thus, they do not effectively capture implicit correspondences between articulated shapes or regularize jittery temporal deformations. In this work, we present FourierHandFlow, which is a spatio-temporally continuous representation for human hands that combines a 3D occupancy field with articulation-aware query flows represented as Fourier series. Given an input RGB sequence, we aim to learn a fixed number of Fourier coefficients for each query flow to guarantee smooth and continuous temporal shape dynamics. To effectively model spatio-temporal deformations of articulated hands, we compose our 4D representation based on two types of Fourier query flow: (1) pose flow that models query dynamics influenced by hand articulation changes via implicit linear blend skinning and (2) shape flow that models query-wise displacement flow. In the experiments, our method achieves state-of-the-art results on video-based 4D reconstruction while being computationally more efficient than the existing 3D/4D implicit shape representations. We additionally show our results on motion inter- and extrapolation and texture transfer using the learned correspondences of implicit shapes. To the best of our knowledge, FourierHandFlow is the first neural 4D continuous hand representation learned from RGB videos. The code will be publicly accessible.

FOURIERHANDFLOW: Neural 4D Hand Representation Using Fourier Query Flow
Jihyun Lee' Junbong Jang'
Donghwan Kim
Minhyuk Sung'
Tae-Kyun Kim 1.2
'KAIST 2 Imperial College London
Syynnlee, junbongjang, kdoh2522, mhsung, kimtaekyun) 0kaist. ac kr
Abstract
Recent 4D shape representations model continuous temporal evolution of implicit shapes by (1) learning query flows without leveraging shape and articulation priors or (2) decoding shape occupancies separately for each time value. Thus, they do not effectively capture implicit correspondences between articulated shapes or regularize jittery temporal deformations. In this work, we present FOURIER- HANDFLOW, which is tempoorllyp continuous representation for human hands that combines a 3D occupancy field with articulation-aware query flows represented Fourier series. Given input RGB sequence, aim to learn a fixed number of Fourier coefficients for each query flow to guarantee smooth and continuous temporal shape dynamics. To effectively model spatio-temporal deformations of articulated hands, we compose our 4D representation based on types of Fourier query flow: (1) pose flow that models query dynamics in- fluenced by hand articulation changes via implicit linear blend skinning and (2) shape flow that models query-wise displacement flow. In the experiments, our method achieves state-of-te of-thc--tttttt results on basee- 4D reconstruction while being computationally more efficient than the existing 3D/4D implicit shape rep- resentations. We additionally show our results on motion inter- and extrapolation and texture transfer using the learned correspondences of implicit shapes. To the best of our knowledge, FOURIER HANDFLO is the first neural 4D contin- uous hand representation learned from RGB videos. Our code available htops:///ithhbb cwt htt.://./mmmunaeee/oreeerer....e
Fourier Query Flow
Figure I: From monocular RGB sequence inputs, FOURIERHANDFLOW learns 4D hand shapes that continuous in both space and time. It models temporal shape evolutions with query flows learned as fixed number of coefficients for Fourier series to guarantee smooth temporal dynamics.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1 Introduction
27 have achieved appealing results in modeling resolution-free 3D articulated shapes, such as human bodies and hands. Motivated by their success in 3D domain, several recent methods [4 15| 20
Neural implicit representations (35 18
16 23
30 25 24
33 37 have proposed to model time- varying implicit shapes that are continuous in both space and time. Pioneering work in this direction, Occupancy Flow (29], learns occupancy field with free-form query flows (i.e. trajectories) over time using Neural ODE [5] to model continuous temporal deformations. However, it is prone to produce spatially over-smooth shapes (e.g. hands with fingers not separated) due the lack of shape category and articulation priors. Following works on modeling human bodies [14 [5] address this issue by disentangling pose and shape-dependent deformations in latent space and decoding an occupancy field separately for each time value. However, they (1) cannot easily regularize abrupt or jittery motions, (2) are not computationally efficient, and (3) do not capture temporal correspondences between shapes all due to occupancy decoding for each time sampling. Also, they are learned using together with sparse geometry inputs (e.g. RGB-D or point clouds), thus it is non-trivial to adapt most of them to learn directly from RGB inputs alone.
37
39
a parametric model [2] 32
In this paper, we aim to tackle less-explored problem of learning representation from monocular RGB sequences for human hands. To address the aforementioned limitations, we focus on learning a representation that exhibits the following desired properties:
pptii-temppoally continuous
1. Continuous and smooth 4D reconstruction: The learned temporal evolution of 3D implicit hands should be continuous and smooth (i.e. without temporal jitters or abrupt motions).
2. Computational efficiency: The representation should allow computationally efficient 4D reconstruction.
3. Articulated shape modeling with correspondences: The representation should effectively model articulated hand geometries while capturing implicit shape correspondences.
To this end, we present novel pixel-aligned 4D hand representation, namely FOURIERHANDFLLOWW To meet continuous and smooth 4D reconstruction and computational efficiency, we propose to combine a continuous 3D hand occupancy field with articulation-awe aware query flows represented as Fourier series along the temporal axis, which is parameterized by the coefficients learned from an input RGB sequence. In particular, we aim learn coefficients for a fixed number of low-mid frequency terms to naturally guarantee smooth temporal evolution of 3D hands. This effectively imposes frequency-don -omenin constraints on the learned motions to avoid jittery or abrupt temporal deformations, which are non-trivial to regularize using the existing 4D continuous representations 29 [5 [4].
Also, by representing a query flow as a sum of the continuous trigonometric basis functions, we can preserve temporal shape continuity while being more computationally efficient than the existing 4D continuous representations (29 [5] that require solving ODEs [5] 38 or occupancy decoding
separately each time sampling [15] [4] In the experiments (Sec. 4.3) we also show that learning
query flows in the frequency domain achieves better accuracy than that in the spatial domain.
For articulated shape modeling with correspondences, we compose our 4D hand representation based on two types of Fourier query flow: (1) pose flow and (2) shape flow. For the pose flow, we first estimate Fourier coefficients representing the flow of 3D hand joints, namely joint flow, to model the temporal change of hand articulation. For effective estimation of the joint flow, we take as input the noisy per-frame joint predictions obtained from an off-the-sheif hand pose estimator [19] and learn to denoise them in the frequency domain. The estimated joint flow is then propagated to each query point using linear blend skinning (LBS) to compute the per-query pose flow, for which we utilize a pre-trained implicit LBS weight field. Next, our shape flow is learned as the Fourier coefficients representing per-query displacement flow w.E.f the previously estimated pose flow. Shape flow aims to model spatio-temporal deformations (e.g. soft-tissue or cannot be expressed in the prior stage alone (i.e., linear deformations w.r.t. the joint articulation changes). Our final query flow is obtained as a sum of the learned pose and shape flows. Unlike the existing 4D continuous human representations [4] that disentangle pose and shape in feature space and decode 3D occupancies separately at each time step, our method captures correspondences between the implicit shapes by modeling query flows directly in the larget spatio-temporal space.
deformations) that
To the best of our knowledge, HOURIERHANDFLE continuous hand representation learned from RGB videos. In the experiments, validate the effec-
the first work proposed for spatio-temporally

---

tiveness of FOURIERHANDFLOW on video-based 4D hand reconstruction using InterHand2.66 6M [26] dataset, where we achieve state-of-the- art results in comparison to the existing (1) image-based 3D hand shape reconstruction methods and (2) 4D implicit shape reconstruction methods modified to take RGB hand sequences as inputs. Also, our inference speed is about x0x and 500x faster than the existing state-of Hate-of-theeart 3D and 4D implicit hand functions, respectively. We additionally show the effectiveness of our method on motion inter- and extrapolation and texture transfer using the learned correspondences of implicit shapes. Lastly, we examine the generalization ability of our method on unseen RGB2Hands 41 real images.
2 Related Work
4D representation for implicit shapes. Motivated by the success of 3D implicit representations [3
25 24 (7), which is resolution-fres (i.e. continuous in space), several methods have been proposed for 4D representation that models dynamic implicit 39
in shape modeling [35] 18
16,
30
shapes [14] 15 29,
37). These methods mainly combine the existing 3D implicit shape representation (e.g. occupancy field (23)) with additional mechanism to model continuous temporal evolution of shapes. Occupancy Flow (29] models such temporal dynamics by estimating free-form
37
33
query flows over time via Neural ODE [5] Other works [39 33 37 decode an occupancy field conditioned on a time value using lemporal-aware shape features. However, these methods often produce spatially over-smooth shapes due the lack of shape category and articulation priors, which is addressed by the following methods [14] [5] proposed for 4D implicit human representation. 4D- CR [T5] learns shape, initial state, and motion-disen disiiannddett latent features from training point cloud pairs of the same motion with different identities synthesized using SMPL (2] model. LoRD [14] learns 4D humans using local representations conditioned on SMPL shapes iteratively registered to point cloud or RGB-D inputs. While these methods show promising results, they are learned by leveraging a parametric model 21 32 together with sparse geometry input observations (e-g. 2.5D or 3D point clouds). Also, they cannot easily regularize abrupt or jittery motions and are not computationally efficient due to the use of an ODE solver [15) and/or occupancy decoding separately for each time value [4 15). To address these limitations, we propose to learn articulation-awanl query flows in the frequency domain to enable smooth and efficient temporal modeling of shapes.
Hand shape reconstruction. 3D hand shape reconstruction has been an active area of research. Most of existing methods model 3D hand shapes by predicting MANO 34 model parameters 45 10 2
12 47 44 or vertex positions of a template hand mesh 19 17
20 40 from an input observation (e.g. RGB, depth, sparse joint positions). However, their hand reconstructions are constrained to a discretized representation of shape, which usually is a low-resolution mesh with MANO topology have adopted implicit shape representation to model resolution-free hand shapes. These implicit representations are also shown to reconstruct hand shapes that are better aligned to input RGB observations
(V)) = 778). To address this issue, several recent methods [6
18
43
6
[8] 9.
The most related domain to our work, temporal- aware 3D hand shape reconstruction from RGB, had been however limited due to the lack of available datasets. While there are datasets and methods proposed for temporal-awart hand pose (i.e.. sparse joints) estimation [46]
3
g
28
3 22], there
had been no large-scale dataset that contains sequences of RGB observations with accurate dense shape annotations until the release of InterHand2. 6M 126. Thus, few existing methods had created synthetic RGB datasets 42 or perform self supervieee learning 4 for hand shape tracking model. However, they only show qualitative results of per-frame mesh estimation, where our goal is to learn 4D continuous hand shapes to allow arbitrary-resolution reconstruction and motion inter- and extrapolation. In this work, we train our 4D hand model on the single-hand and two-hand subsets of InterHand2,66 6M and compare our results to more recent state-of-the-as 3D hand reconstruction methods [8] 44 19 45) on InterHand2.66 M.6 and 4D implicit functions (29 [4] adapted to learn hand
shapes from RGB.
Shape modeling using frequency domain. We briefly discuss the existing work on frequency-domanl shape modeling. specifically focusing on deep learning- based methods for shape reconstruction and generation. Shen et al. [36] proposes to estimate Fourier-domain slices from input images to enable computationally efficient 3D shape reconstruction. Hui el al. [13] introduces a wavelet- domain diffusion model to allow fiffussionbbseed generative modeling directly on implicit shape representation. However, these methods mainly aim to model non-articulated objects (e.g. chairs, airplanes). Fourier Occupancy Field [9, which is more related work to ours, proposes to represent
3

---

an occupancy field with Fourier series along the z-axis to enable efficient human reconstruction. Similar [9), we adopt Fourier series (ef. discrete Fourier or wavelet transform) to preserve the continuity of our representation in the target spatio-temporal space. In contrast to [2] that learns Fourier coefficients to model static occupancy field along one spatial dimension, we learn Fourier series to model query flows over time to capture continuous 4D deformations of hands.
3 FOURIERHANDFLONE Pixel-Aligned 4D Hands with Fourier Query Flow
Our work aims to learn a 4D continuous hand representation from a sequence of single-view RGB frames (I') where I' R xxxxx33 is an RGB frame of size W x H observed at time To maintain consistency with the existing 4D continuous representations 14, 29 [3], our method takes as input sub-sequence of T 17 consecutive frames at once. In what follows, we first briefly explain our pre-trained occupancy and linear blend skinning weight fields (Sec. B.I), which are prepared prior to learning our query flows. We then explain our Fourier query flow representation (.22 and 4D hand representation using Fourier query flows to enable articulation-awe aware shape modeling (Sec. 3.3
3.1 Pre-Training Occupancy and Linear Blend Skinning Weight Fields
The main focus of our work is to learn query flows 2(-,)) : R x (0, + R$ that models 3D trajectory of query point p E R over the time span & (0, To perform 4D hand reconstruction, we propagate a pre-trained hand occupancy field in the canonical coordinate system to the coordinate system at target time using the learned query flows without requiring neural occupancy decoding separately for each t. To obtain our canonical occupancy field, we learn an occupancy function o(-) R + [0, 1] that maps query point p in the canonical space to the occupancy probability in [0, 1]. Along with o(-). we also learn an implicit linear blend skinning (LBS) function w<-) : R > [0, 1 that maps p to LBS weight vector for B hand bones, which will be later used to model query flows induced by the change of hand articulation (Sec (.3)). We learn both o()) and wx() prior learning our query flows, for which we train modified version of LEAP [25 network. For more details and backgrounds on LBS, please refer to the supplementary section.
Note that, while Occupancy Flow [29] takes similar approach and propagates occupancy field at = 0 learned for each sequence separately, we propagate a canonical hand occupancy field that is shared among all sequences to allow modeling implicit correspondences between hand shapes from different sequences. Also, learning occupancy in the canonical system (cf. inservation-specini system) is known to be more robust addressed in the recent implicit functions (25 16 [8]
3.2 Fourier Query Flow Representation
We now introduce our query flow representation based on Fourier series. The representation for query flow .)..) (1) should be easy to be learned for smooth query dynamics (e.g., without jitters and abrupt motions) and (2) should be computationally efficient. However, it is difficult to achieve both
merits using the existing 4D continuous representations 14 15 29] with an ODE solver [15] 29] or occupancy decoding for each time step 14. [5]. To address these limitations, we explore the Fourier-based representation for query flows. Specifically,
consider set of the sine-cosine form of Fourier series ()) d=)..,. where each (-)-) represents the flow an arbitrary query point in d E {x, y, >} dimension over the normalized time span E [0, 1):
N
(t) = ag 2
Σ ( cos(nt) + by sin(nt)), for d E {x,
(I)
In Eq. (1. ad E N+ and ba E R are coefficients of cosine and sine basis functions defined in d E {2,8,2) dimension, respectively. N denotes the number of each basis function, which would be theoretically an infinity to represent an arbitrary signal. In this work, as most of high-frequency hand motions are unnatural to occur in the real world, we aim to learn coefficients for a fixed number (N = 6) of basis functions. This eases the training of neural networks by constraining the number of Fourier coefficients to learn, while guaranteeing the reconstruction of smooth query dynamics along the temporal axis. Also, note that Eq. (I) naturally models continuous query trajectory over time by

---

Joint Flow Network go Used model query-wise pose flow LBS 3.3)
Sharred
Temperal Encoder
Y1
taz
Joint
shared
Encoder
Temporal Encoder
Fourler basis functions
Hand-Skeleton Graphs (5'H=1
Applied step along the spalial axis
Feature Graphs [+')
Applied per joint along the temporal axis
Fourier Coef. (el)
Joint Flow (2')
Shape Flow Network h
(x,yz)
abared
Temporal Encoder
Imagee Sequence (L')E-1
Query Encoder Parl
twe
c²
x
(p:)
Fourier Coef.
Shape Flow 0 (p.)
Query Descriptors [r')
Appliad per time step
Query Features [(')]==
Applled along temporal
c²
Query Point p
Fourier basis functions
Figure 2: We model query flow by estimating the joint flow (blue upper branch) which used to compute query-wise pose flow and the shape flow (pink lower branch). Our joint flow network first predicts the Fourier coefficients for the flows hand joint positions using the joint encoder and the temporal encoder. The predicted joint flow is propagated to query point via implicit linear blend skinning to compute the pose flow (see Sec. 3.3). Our shape flow network then predicts the Fourier coefficients for the query-wisc displacement flow w.r.t. the estimated pose flow using the query encoder and the temporal encoder. Our final query flow computed as a sum of the pose and shape flows.
representing it as sum of trigonometric functions. Once the coefficients for the basis functions are estimated per input sequence, sampling query positions along the temporal axis also does not require additional computations (e.g., solving ODEs [29] or neural occupancy decoding [14 [5] other than evaluating Eq. with different value of at 0(1) complexity.
3.3 Pixel-Aligned 4D Hand Representation Using Fourier Query Flow
We introduce our pixel-aligned 4D hand representation using the Fourier query flow representation discussed in Sec. 3.2 To enable articulation-awe shape modeling, representation is based on two types of Fourier query flow: (1) pose flow and (2) shape flow.
Pose flow. Our pose flow aims to model query flow influenced by the change of hand articulation. In this stage, instead of directly estimating flow for a query point, we first predict the Fourier query flow for 3D hand joint positions, namely joint flow. The learned joint flow is then propagated to an arbitrary query point using pre-trained LBS field (Sec. 3.1 to model query-wise pose flow.
Let с E R6N+3 be the concatenation of all Fourier coefficients (i.e., {ad1 1 and (bd) for d E {ar, =)) that determines the Fourier flow of one query point (Eq. ( In this stage, aim
learn the flows of J = 21 hand joint positions parameterized by {c> } Jj=1- where each c² e g6N-3
denotes the Fourier coefficients of j-th joint flow. To effectively learn {c') //-1 from input RGB sequence, we condition our prediction on the 3D joint positions obtained per frame using an off-the- shelf pose estimator V/ (-). Our joint flow network go (.) learns the mapping from (I') {cl // as follows:
go (IT)) = te, (co, (TI), ((TT)
In Eq. 22) above, (-) is composed two sub-networks ea, (5.) (
and bg (+) parameterized by the weights = {0,, 02). Our joint encoder eø, (:-) :.--)RRXXM XTXxxxxx3 x X xxxxxxx Txx first extracts f-dimensional per-joint features given an input RGB sequence and initial joints predicted by Vr(.). As shown in the upper branch of Fig. [ we represent the initial joints hand-skeleton graph g for each time step, where each node feature is set the concatenation of 3D positions and

---

pixel-aligned features [35 of the corresponding hand joint predicted by (-). Then, the joint encoder applies graph convolutions
extract feature graph F² with the updated per-joint features. Our temporal encoder te, : RTXXx + _ +3) then temporally aggregates the per-joint features using temporal convolutions, These semporal-waat joint features are concatenated with the Fourier coefficients of the initial per-frame joints and fed an MLP for the prediction of refined coefficients {e²))=1. Finally, the flow of j-th hand joint (-) (0,1) + R over the normalized time span [0, is modeled by Eq. [ with the estimated coefficients c². It is important to note that, although go(.) initially takes joints predicted by (.). to reduce search space, go(.) learns to refine the noisy per-frame joint predictions into continuous joint flow using hints from the input RGB and skeleton graph structures (please also refer to Table. 3 for experimental results).
Once the joint flow (+)()))=1 is obtained, we can propagate the learned joint flow to an arbitrary query point p via implicit LBS compute the per-query pose flow $0 (::-) R3 [0. 1) R3:
B
$ D0 (p,t)
= Σ b=1
we (p) TopP.
(3)
where The5 is the rigid bone transformation matrix for b-th hand bone computed from the joint positions (1)(1)) at time t using biomechanically valid conversion algorithm [16]. wr(-) R [0, 1] is function that returns pre-trained LBS weight for b-th hand bone given an input query point p. Note that, as p is sampled from the canonical space (Sec. 3.1.p our bone transformation matrices (T, b are computed w.rf. to the canonical hand pose [16] to model the query correspondences between the canonical system and the posed system at time t.
Shape flow. Our shape flow models query-wise displacement flow w.r.t. the estimated pose flow. It aims to model spatio-temporal deformations (e.g,, soft tissue or identity-dependen deformations) that cannot be solely expressed by linear deformations w.r.l hand articulation changes (Eq. (3). Formally, our shape flow network ha (::)): R W x x3 T - RNN learns a mapping from an input RGB sequence and query point p to Fourier coefficients cP representing the displacement flow of p as follows:
hw ((I') -wug (Ews ((T) p)).
Similar to ga (-).), (-)(--) -) is composed of two sub-networks e (s..) and t(-)w, parameterized by the weights (Wi, w2). As shown in the lower branch of Fig. 2 our query encoder twr (---)
RTxWxH*3 R$ + T extracts /// diddimensional query feature per time step. For each I, we create initial query descriptor f E RR*h by concatenating (1) canonical query position p. (2) a query position after applying the pose flow @g (p, t) and its corresponding pixel-aligned feature (35), and (3) skinning weight vector w wp)). This query descriptor is fed to an MLP-based query encoder to extract a query feature f" R² Our temporal encoder twz (-) : R'* + R then applies temporal convolutions to the per-time query features to predict the Fourier coefficients cP representing the displacement flow of query p. Finally, our shape flow & shape (.,-) R (0, - R for a query point p is modeled by Eq. I using the query-wise Fourier coefficients cP predicted by hww (..)..
Our final flow (::)) R x [0, 1) + R for query point p is obtained as a sum of the Iearned pose and shape flows, where both of them are represented by the Fourier series to enforce smooth query dynamics along the temporal axis:
D (p. t) = -0000 (p. t) + shape (p.t).
(5)
'Since the initial per-frame joints discretized in time, their Fourier series coefficients approximated by the numerical integration as discussed in (9]. We use trapz function PyTorch [31] library. Although the input encoder and the temporal encoder in Eq. 2 and Eq. d4) have different network architectures. we use the same notations e(.) and t(-) for brevity.

---

Loss functions. Our network is trained using the (I) occupancy loss and (2) correspondence loss. First, the occupancy loss penalizes the deviation between the reconstructed and ground truth occupancy probabilities:
Locc(0,w)
(p, - (2,
1,2,
(6)
[oooe
P. More
where Noce is a set of training examples and I is a sequence of RGB inputs. (p I) and Oge (p, 1, denote the reconstructed and ground truth occupancy probabilities, respectively. Note that (p, is obtained by propagating the pre-trained canonical occupancy at p to the corresponding query position at using the query flow learned by our network. Since the canonical occupancy field is pre-trained and fixed, this occupancy loss enforces our flow network to learn query trajectories that better propagate the canonical occupancies s.t. the occupancies propagated to better resemble the ground truth 3D shape at t.
Second, our correspondence loss penalizes the deviation between the reconstructed query position (P, t, T) := (p. I and the ground truth query position Pst t, corresponding to given a set of training examples Neorr:
Coor((0,w)
Σ
(p, - (p,
RooRT
P.C.IEX.orr
Our final training objective is to minimize [(),w) (0,w) + XCcorr (0, (0,w), w)) where is hyper-paramette parameter to balance the influence between the two loss terms. For more details on network training and architecture, please refer the supplementary section.
4 Experiments
Datasets and evaluation metries. We use the two-hand (TH) and single-hand (SH) subsets of 30 FPS version InterHand2.6M (26) dataset, which contains diverse hand motions captured RGB sequences with dense shape annotations. For each subset, we use samples annotated as valid hand type and follow the train/val/test splits of the original InterHand2.61 6M dataset. The resulting TH subset contains 477K training and 4K validation sequences, and SH subset contains 656K training and 5K validation sequences. To maintain consistency with the existing 4D continuous representations 22 14 15). we use sub-sequences of T = 17 frames sampled from the original sequences as inputs to our method. For testing, we use 2K sub-sequences (34K frames) randomly sampled from the test sequences of each subset. For qualitative evaluation, we additionally show our results on RGB2Hands [1] real dataset, which contains sequences of RGB hand motions without shape annotations. For evaluation metrics on 4D reconstruction, computed w.r.t. the ground truth shapes of InterHand2.66 MM dataset. We also use LI Correspondence Error (LI-Corr) for evaluating our learned shape correspondences.
mean Intersection over Union (IoU) and Chamfer LI-Distancs (CD)
Compared methods. As our method is the first RGB-based 4D continuous hand representation, there is no direct baseline. We thus compare ours to (1) RGB-basaee 3D hand reconstruction methods and (2) 4D implicit reconstruction methods modified take RGB hands as inputs. For 3D hand reconstruction methods, we consider state-of-the-as methods on InterHand2266
ACR* $ [44, Two-HanddShape
Pose 45 IntagHand [9), and Im2Hands 18. Note that ACR*
44 results are produced using
the pre-trained ACR model officially released for in-the-wild demo. As these methods are proposed for two-hand reconstruction, we compare them on TH subset. For 4D representations, we consider (1) Occupancy Flow, which is the only 4D continuous representation that has shown results from RGB videos, and (2) LoRD, which is the most recent state-of the-art 4D articulated implicit function proposed for human bodies. As LoRD originally takes SMPL 21 meshes fitted to 2.5D or 3D point cloud inputs, we have implemented a modified version (LoRDī) that uses MANO (34 topoology hand meshes reconstructed from RGB frames using IntagHand [19 to make direct comparisons. We compare these 4D implicit methods on both TH and SH subsets. For the off-the-shet hand pose estimator (-()) in method, adopt the joint estimation module of Hnta [19 in [8].
Two-hand extension. Our hand representation can be naturally extended to estimate 4D two-hand shapes make comparisons on InterHand2.6M. 26 TH subset. In a nutshell, we use two joint
7

---

flow networks each trained for left and right hands, respectively. The shape flow network is shared among both side hands to implicitly capture correlation between them. Note that left and right side conditioning for shape flow estimation is done by taking as inputs query position after applying the pose flow w.r.t. the corresponding side of hand. Please find more details in the supplementary.
4.1 Video-Based 4D Hand Reconstruction
In Tab. D our method achieves state-of-the-an oft----rttt-ttttt- reconstruction results InterHand2.66) [26). The existing 3D reconstruction methods (3D Mesh and 3D Implicit categories) perform shape recon- struction for each frame independently. Thus, they often produce shapes with temporal jitters and do not reason about hand motions (e.g., motion inter- and extrapolation). In 4D Implicit category, Occupancy Flow [29] achieves sub-optimal reconstruction quality due to the lack of shape articulation prior. While LoRD 14 achieves better performance, it does not fully leverage pixel-aligned features for shape reconstruction. In Fig. also qualitatively show the reconstruction results, where ours produces the most plausible shapes that are also well-aligned to the input RGB observations.
Table 1: Quantitative comparisons of iideo-based 4D hand reconstruction on InterHand2. 6M dataset. Our method achieves state-of-the-ar afrtt results on both TH and SH subsets.
Subset
Category
Method
IoU (%6 f
CD (mm)
ACR"
45.2
7,89
Mesh
Two-Hand-Shap--l Pose
48.7
6.58
Hand
19
57.5
5.27
TH
Implicit
1m22andds
Occupancy Flow
32.4
18.27
4D Implicit
LoRD
58.0
FOURIERHANDFL.om OW (Ours)
62.8
4.46
Occupancy Flow
44.7
13.21
SH
4D Implicit
LoRD'
63.1
4.38
FOERTERHANDPL.OM (Oars)
65.8
3,90
Input
Im2Hands [18]
OFlow [29]
LoRD [14]
Ours
Ore. Vlew
Org.
Org.
View
Org. Fiew
View
Figure 3: Qualitative comparisons of video-base 4D hand reconstruction on InterHand2. 6. [26] TH subset. Please also refer our supplementary for full-length video-based reconstruction comparisons, where ours achieves more favorable results (e.g. without temporal jitters).
In Tab. 2 (columms 3-4), we also compare training and inference time between the implicit representa- tions. Note that we use InterHand2.6MM TH subset hereinafter unless specified. Im2Hands [18) is slow in both training and inference due to the global shape encoding and decoding step for shape refinement. While LoRD 14 the fastest in training. its inference slow due the test-time model optimization and per-time occupancy decoding. Occupancy Flow (29 achieves fast inference thanks to its simple network architecture, but its efficiency is still sub-optimal due to the use of an ODE solver [5]. Our method achieves the fastest inference time using the Fourier query flow representation. In Tab. 2 (column 5), we also compare
shape correspondence error between implicit hand functions. Note that Im2Hands [18) and LoRD [4] do not capture temporal shape correspondences due to per-time occupancy decoding. Compared to Occupancy Flow [29], we learn more accurate shape correspondences by articulation-awee query flow modeling.

---

Table 2: Training inference time and learned correspondence comparisons among implicit represen-
tations. Training time measured for each training iteration with batch size of and inference time is measured for each frame to make comparisons with the Trame-based method [8] Each time result obtained as an average of KK measurements.
Category
Method
Training Time (sec.)
Inference Time (sec. 4
L.I-Cor (mm)
3D Implicit
Im2Hands
3.31
6.62
N/A
Occupancy Flow
0.26
4D Implicit
LoRD'
0.15
114.72
N/A
FOURIERlHANDFL.Ow (Ours)
2.75
0.22
10.8
In addition, we compare our learned joint flow with the noisy per-frame joints estimated by (-()) [19]. In Tab, our joint flow network shown to be effective refining the initial per-frame joint predictions using hints from the input RGB frames and hand skeleton graph structures.
Table Joint refinement results on Mean Per Joint Position Error (MPJPE).
Method
MPIPE (min)
Before Ref. After Ref.
13.3
11.0
4.2 Additional Experimental Results
Texture transfer. In Fig. we show our results on texture transfer using the learned correspondences of implicit shapes. Given the reference texture defined in our canonical hand field (Sec. [3.1]. we achieve high-quality texture transfer results on the hand reconstructions at randomly sampled sequences and time stamps. Note that, our canonical hand field is shared among all sequences (cf. 22),, we can naturally obtain dense correspondences between inter-sequence hand reconstructions.
Reference
Transferred
Reference
Transferred
Figure Texture transfer results using the learned correspondences between our hand reconstructions. Best viewed with 300% zoom-in.
Motion inter- and extrapolation. In Fig. we show our motion inter- and extrapolation results. Given RGB frames observed at each time step, we sample hand shapes at inter- and extrapolated time values from the learned Fourier query flows. Our sampled shapes shown model smooth temporal evolution of hand shapes.
Observed AEEEE
Observed at F=
Figure 5: Our motion inter- and extrapolation results. Given RGB frames observed at each time step, method can sample shapes at inter- and extrapolated time values.
Generalization test. In Fig. we show addi- tional qualitative results on RGB2Hands [1 real images using our model trained on Inter- Hand2.6n 6M (26 dataset. Our model is shown to produce plausible reconstructions even from unseen HaB2H images, demonstrating its generalization ability.
Figure 6: Generalization
test on unseen
RGB2Hands (11 real images.

---

4.3 Ablation Study
In Tab, we compare video-based based 4D hand reconstruction results among the variations of our method. w/o Pose Flow and w/o Shape Flow (rows 2-3) denote our method variations in which pose or shape flow is not modeled, respectively. It is shown that each type of flow contributes more accurate reconstruction, thus the best accuracy is achieved when both flows are used. Fourier - ODE and Fourier + InputCond (rows 4-5) are variations in which (1) query flows are modeled by solving ODE S and (2) query positions are directly estimated conditioned on time value, respectively. Our method based on Fourier query flow achieves the best performance, showing the effectiveness of flow estimation in the frequency domain (cf. spatial domain).
Table 4: Quantitative comparisons of video-based 4D hand reconstruction among the variations of our method InterHand2.6M (26) TH subset. Our proposed method achieves the best results.
Method
IoU (%) f
CD (mm)
LI-Cor (mm) 1
w/o Pose Flow
50.1
7.86
20,9
w/o Shupe Flow
57.9
5.06
12.1
Fourier ODE
60.4
4,93
11.5
Fourier - InputCond
61.0
4.87
11.0
FOURIERHANDFLON (Ours)
62.8
4.46
Conclusion
We present FOURIERHANDFLOW which is the first neural 4D continuous representation for human hands Iearned from RGB videos. We combine a continuous 3D hand occupancy field together with query flows learned as Fourier coefficients to model smooth and continuous temporal shape dynamics. To enable articulation-awve aware shape modeling, we introduce two types of Fourier query flow: (1) pose flow based implicit linear blend skinning and (2) shape flow that models query-wise displacements. Our method achieves state-of-the-art results on video-based 4D hand reconstruction while being computationally more efficient than the existing implicit function-based methods.
Broader impact. Our method can potentially used for video-based 4D reconstruction of articulated implicit shapes in general (e.g. human or animal bodies), which is useful for AR/VR applications.
Limitations. Although our method is more efficient than the state-of- of-te-of-thearr art 3D and 4D impleit functions, its inference time is not yet sufficient to provide real-time solution (cf. low-resolution mesh- reconstruction methods 19 44)). We plan to investigate ways further improve our computational efficiency, e.g., by adapting requeencydomaale representation additionally along the spatial domain. Also, while Fourier query flow naturally models smooth temporal deformations, constrains our query flow estimation on the low-mid frequency subspace. Thus, it may not model hand motions with actual temporal jitters, although they not commonly occur. addition, our method currently takes fixed length of T = 17 sub- sub-sequence as inputs following Occupancy Flow (29]. We will try to further address these limitations as our future work.
Acknowledgments and Disclosure of Funding
This work was in part sponsored by NST grant (CRC 21011. MSIT), KOCCA grant (R2022020028, MCST) and ITTP grant (2019-0-00075, MSTT). Minhyuk Sung acknowledges support from the NRF grant
and IITP grants (2022-0-00594, RS-2023-0022759) funded by the Korean government (MSTT), the Technology Innovation Program (20016615) funded by the Korean government (MOTIE), as well grants from ETRI, KT, NCSOFT, and Samsung Electronics.
10

---

References
[1] Seungryul Baek, Kwang Kim, and Tae-Kyun Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In CVPR, 2019.
[2] Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images the wild. CVPR, 2019.
[3] Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, and Nadia Mag- nenat Thalmann. Exploiting spatial-temporat temporal relationships for 3d pose estimation via graph convolutional networks. In ICCV, 2019.
[4] Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Yen-Y Yu Lin, and Xiaohui Xie. Temporal-aware self-super learning for 3d hand pose and mesh estimation in videos. In WACV, 2021.
[5] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. NeurIPS, 2018.
[6] Enric Corona, Tomas Hodan, Minh Vo, Francesc Moreno-Noguer, Chris Sweeney, Richard Newcombe, and Lingni Ma. Lisa: Learning implicit shape and appearance of hands. In CVPR, 2022.
[7] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. NewrIPS, 2016.
[8] Zhipeng Fan, Jun Liu, and Yao Wang. Adaptive computationally efficient network for monocular 3d hand pose estimation. In ECCV, 2020.
[9] Qiao Feng, Yebin Liu, Yu-Kun Lai, Jingyu Yang, and Kun Li. Fof: Learning fourier occupancy field for monocular real-time human reconstruction. In NeurIPS, 2022.
[10] Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann. Robust 3d hand pose estimation in single depth images: from single-view enn to multi-view cnns. In CVPR, 2016.
[11] Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, and Cordelia Schmid. Leveraging photometric consistency over time for sparsely supervised hand-ohject reconstruc- tion. In CVPR, 2020.
[12] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. CVPR, 2019.
[13] Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu. Neural wavelet-domain diffusion for 3d shape generation. SIGGRAPH Asia, 2022.
[14] Boyan Jiang, Xinlin Ren, Mingsong Dou, Xiangyang Xue, Yanwei Fu, and Yinda Zhang. Lord: Local 4d implicit representation for high-fidelity dynamic human modeling. In ECCV, 2022.
[15] Boyan Jiang, Yinda Zhang, Xingkui Wei, Xiangyang Xue, and Yanwei Fu. Learning composi- tional representation for 4d captures with neural ode. In CVPR, 2021.
[16] Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, and Siyu Tang. A skeleton- driven neural occupancy representation for articulated hands. In 3DV, 2021.
[17] Dominik Kulon, Riza Alp Guler, Iasonas Kokkinos, Michael M Bronstein, and Stefanos Zafeiriou. Weakly-supervised mesh-convolutionall hand reconstruction in the wild. In CVPR, 2020.
[18] Jihyun Lee, Minhyuk Sung. Honggyu Choi, and Tae-Kyun Kim. Im2hands: Learning attentive implicit representation of interacting two-hand shapes. In CVPR, 2023.
[19] Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, and Yebin Liu. Interacting attention graph for single image two-hand reconstruction. In CVPR, 2022.
[20] Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In CVPR, 2021.

---

[21]
Matthew Loper. Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. Smpl: A skinned multi-person linear model. ACM TOG, 2015.
[22] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja. Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. Mediapipe: A framework for perceiving and processing reality. In CVPR Workshops, 2019.
[23] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[24] Marko Mihajlovic, Shunsuke Saito, Aayush Bansal, Michael Zollhoefer, and Siyu Tang. Coap: Compositional articulated occupancy of people. CVPR, 2022.
[25] Marko Mihajlovic, Yan Zhang, Michael Black, and Siyu Tang. Leap: Learning articulated occupancy of people. In CVPR, 2021.
[26] Gyeongsik Moon, Shoou-1 Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.cm 6m: dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In ECCV, 2020.
[27] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representa- tion. In ICCV, 2021.
[28] Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. Ganerated hands for real-time 3d hand tracking from monocular rgb. CVPR, 2018
[29] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow:
4d reconstruction by learning partiele dynamics. In ICCV, 2019.
[30] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS. 2019.
[32] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR, 2019.
[33] Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojeic, Srinath Sridhar, and Leonidas J Guibas. Caspr: Learning canonical spatiotemporal point cloud representations. NeurIPS, 2020.
[34] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM TOG, 2017.
[35] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In CVPR, 2019.
[36] Weichao Shen, Yunde Jia, and Yuwei Wu. 3d shape reconstruction from images in the frequency domain. In CVPR, 2019.
[37] Jiapeng Tang, Dan Xu, Kui Jia, and Lei Zhang, Learning parallel dense correspondence from spatio-temporal descriptors for efficient and robust 4d reconstruction. In CVPR, 2021.
[38] Gerald Teschl. Ordinary differential equations and dynamical systems, volume 1 140. American Mathematical Soc., 2012.
12

---

[39] Tuan-Anh Vu, Due Thanh Nguyen, Binh-Son Hua, Quang- Hieu Pham, and Sai-Kit Yeung. Rfnet-4d: Joint object reconstruction and flow estimation from 4d point clouds. In ECCV, 2022.
[40] Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. Dual grid met: Hand mesh vertex regression from single depth maps. In ECCV, 2020.
[41] Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Somnychnkk,, Neng Qian, Miguel A Otaduy, Dan Casas, and Christian Theobalt. Rgb2hands: Real-time tracking of 3d hand interactions from monocular rgb video. ACM TOG, 2020.
[42] John Yang, Hyung Jin Chang, Seungeui Lee, and Nojun Kwak. Seqhand: Rgb-sequence-basee 3d hand pose and shape estimation. In ECCV, 2020.
[43] Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In CVPR, 2022.
[44] Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P Breckon, and Jue Wang. Acr: Attention collaboration-base regressor for arbitrary two-hand reconstruction. In CVPR. 2023.
[45] Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma, and Hongan Wang. Interacting two-hand 3d pose and shape reconstruction from single color image. In ICCV, 2021.
[46] Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang. A hand pose tracking benchmark from stereo matching. In ICIP, 2017.
[47] Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgh image. ICCV, 2019.
13


Reviews:
[{"review_id": "tNaNVRTdip", "reviewer": "Reviewer_g8GH", "summary": "This paper introduces FOURIERHANDFLOW, which is a spatio-temporal continuous representation for the human hands. It combines a continuous 3D hand occupancy field with articulation-aware query flows represented as Fourier series along the temporal axis. These query flows are parameterized by coefficients learned from an input RGB sequence. Specifically, two types of Fourier query flows, namely pose flow and shape flow, are used to address the challenges of continuous and smooth 4D reconstruction, computational efficiency, and articulated shape modeling with correspondences.", "questions": "- In the prediction of pose flow, I think that the essence of being able to perform articulated shape modeling is to represent the skeleton of the hand in the form of a graph, which can maintain the topological structure of the hand. Therefore, in this process, the overall skeleton of the hand is predicted frame by frame. But the subsequent prediction of Fourier coefficients is based on the trajectory of each joint, so how is the temporal smoothness of each joint constrained here?\n- For different input sequences, is it necessary to normalize them to the same sampling space? If so, what are the specific implementation details?\n- In the pose flow, the paper estimates the joint flow of the keypoints and then propagates it to arbitrary points based on LBS. Therefore, the estimation of the shape flow after arbitrary points heavily will rely on the accuracy of LBS. Does the paper have any special handling or explanation for this situation?\n- Recommendation: For the ODE based method, although its computational complexity is relatively high, it can maintain the property of Diffeomorphism. This aspect can be considered later.", "limitations": "The authors have partially addressed the limitations of their work, though there is space for improvement (see the section Strengths And Weaknesses).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper provides a well-articulated description of the problems with existing implicit methods. It presents a clear logical flow and addresses specific challenges effectively. The results show particularly noticeable improvements for abrupt or jittery motions correction.\n- Although Fourier series is not a novel concept, this paper extends it to the temporal dimension.", "weaknesses": "- It is undeniable that this work is closely related to the Fourier Occupancy Field [9]. The authors should consider adding comparative experiments with this work.\n- This paper focuses on describing the acquisition of Fourier Query Flow in the method section but lacks a direct description of how the final pipeline for generating the 4D hand is constructed (e.g., how the pre-trained canonical field occupancy is utilized). A more comprehensive overview of this aspect should ideally be included in the first paragraph of the methodology section for better clarity.\n- The pose flow and the shape flow are two important branches proposed in this paper, and the final flow is the sum of both. It would be helpful to provide clearer illustrations that show the trajectories generated by each branch separately, as well as the combined trajectory. This would enable readers to gain a clearer understanding of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and finding that the proposed method is “novel and interesting” [vgcD] and shows “large improvements” [rxcm, g8GH]. We also appreciate [rxcm] for finding that the “paper is clearly written and well-motivated.” In what follows, we address the concerns of the reviewers.\n\n---\n\n> **Weakness 1.** “It is undeniable that this work is closely related to the Fourier Occupancy Field [9]. The authors should consider adding comparative experiments with this work.”\n>\n\n**Reply:** Fourier Occpancy Field (FOF) [9] uses Fourier Series along one of the spatial axes (i.e., z-axis) to enable efficient 3D human reconstruction, while our method uses Fourier Series along the temporal axis (1) to regularize high-frequency (e.g., jittery) temporal shape change and (2) to enable efficient 4D hand reconstruction. Along with this difference, we believe that another important difference between FOF and our method lies in the characteristics of the shape category that each aims to reconstruct. FOF aims to reconstruct clothed humans, which have complex shape variations (e.g., cloth wrinkles) but with fewer self-occlusions caused by the underlying shape articulations. In contrast, our method aims to reconstruct human hands, which have more complex articulations leading to more severe self-occlusions. Thus, most of the existing hand implicit functions, e.g., [6, 16, 18], use articulated implicit representation to directly model pose-dependent deformations with respect to the learned canonical hand shape to incorporate strong pose prior. Similarly, one of our main contributions was also to propose articulation-aware query flows (i.e., pose and shape flows) to directly model pose-dependent 4D hand deformations. Since FOF is a non-articulated implicit function, we thought that the existing state-of-the-art methods specifically designed for the same hand reconstruction task would be more challenging baselines. In the table below, we show the experimental comparisons with FOF, where FOF yields similar results to our non-articulated implicit function baseline (Occupancy Flow [29]) in Table 1 in the paper. We will add this discussion and the experimental results in the revision.\n\n| | Shape accuracy (IoU $\\uparrow$) | Shape accuracy (CD $\\downarrow$) |\n| --- | --- | --- |\n| FOF [9] | 30.4 | 23.88 |\n| Ours | 62.8 | 4.46 |\n\n \n>**Weakness 2.** “A more comprehensive overview of this aspect should ideally be included in the first paragraph of the methodology section for better clarity.”\n>\n\n> **Weakness 3.** “It would be helpful to provide clearer illustrations that show the trajectories generated by each branch separately, as well as the combined trajectory. This would enable readers to gain a clearer understanding of the method.”\n>\n\n**Reply:** Thank you for your suggestion. We will make sure to (1) add the overview of our overall pipeline and (2) show the trajectories generated by each type of flow in the revision.\n\n>**Question 1.** “In the prediction of pose flow, I think that the essence of being able to perform articulated shape modeling is to represent the skeleton of the hand in the form of a graph, which can maintain the topological structure of the hand. [...] But the subsequent prediction of Fourier coefficients is based on the trajectory of each joint, so how is the temporal smoothness of each joint constrained here?”\n>\n\n**Reply:** The temporal smoothness of each joint is preserved from our trajectory prediction in the *frequency-domain subspace* (lines 49-50). For maintaining the topological structure in our temporal reconstruction, we feed the joint features extracted using a graph convolutional network (GCN) as inputs to our Fourier coefficients estimation module (lines 202-203). We observe that such structure-aware input feature conditioning is sufficient to be able to preserve the hand topological structure in our temporal reconstruction as shown in the supplementary video.\n\n>**Question 2.** “For different input sequences, is it necessary to normalize them to the same sampling space? If so, what are the specific implementation details?”\n>\n\n**Reply:** Along the spatial dimension, we normalized the 3D coordinate space of each input sequence by aligning the predicted hand root joint of the first frame to the origin point. Along the temporal dimension, we did not apply normalization. We will clarify this in the revision.\n\n>**Question 3.** “[...] the estimation of the shape flow after arbitrary points heavily will rely on the accuracy of LBS. Does the paper have any special handling or explanation for this situation?”\n>\n\n**Reply:** The accuracy of the estimated shapes after LBS (S.3.1 in the supplementary paper) is dependent on two factors: (1) the accuracy of the pre-trained skinning weights $\\mathbf{w}^{\\mathbf{p}}$ and (2) the accuracy of our joint flow estimation, which determines the rigid bone transformations {$\\mathbf{T}_b$} for $b=1, \\textit{...}, B$. To examine factor (1), we calculated the accuracy of the pre-trained $\\mathbf{w}^{\\mathbf{p}}$ by computing the shape IoU after applying LBS using the ground truth rigid bone transformations. As the resulting shape IoU was quite high (98.62% on InterHand2.6M [26] test dataset), our shape accuracy after LBS would mainly depend on the remaining factor (2), which is the accuracy of our joint flow estimation. Thus, we performed experiments to examine the robustness of our final shape accuracy with varying quality of the initial joint estimation and showed that our method is quite robust to it. We kindly refer you to our answer to Weakness 3 of the reviewer [Uqr4] for the experimental results.\n\n>**Question 4.** “Recommendation: For the ODE based method, although its computational complexity is relatively high, it can maintain the property of Diffeomorphism. This aspect can be considered later.”\n>\n\n**Reply:** Thank you very much for your valuable comment. We will consider this aspect in our future work."}, {"review_id": "7eQIYkrGkN", "reviewer": "Reviewer_rxcm", "summary": "This paper proposes FourierHandFlow, a 4d hand pose and shape representation that inherently uses Fourier series as query flow representation. Given RGB sequence, a fixed number of Fourier series are learned to represent hand pose and shape. The authors use two types of flows to decompose pose and shape: pose flow (joint flow) and shape flow. Such decomposition makes it more efficient to reconstruct 4d hands. Experiments on Interhand2.6M and RGB2Hands datasets demonstrate its superiority over existing two hand estimation methods.", "questions": "1. How to obtain the gt 3d shape of InterHand2.6M for training? Is that simply the MANO labeling? If true, what’s the advantage of using implicit representation instead of MANO for hand shape learning? \n\n2. L. 228, what does ^2 mean? Is that a mistake? \n", "limitations": "Yes. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is clearly written and well-motivated.  The compact Fourier series representation is proved to be effective for two hand 4d reconstruction in a smoother way. The video results are impressive. The quantitative results show large improvements. ", "weaknesses": "The main weakness is that, I am not very sure whether such representation could correctly model the surface details (or pose dependent deformation) of the hand because Fourier series based representation seem to generate over smoothed results due to its low dimension nature of shape flow. For example, at video 04:06 (frame 7101), the little finger of the left hand in “Alt. View 1” is too thin. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and finding that the proposed method is “novel and interesting” [vgcD] and shows “large improvements” [rxcm, g8GH]. We also appreciate [rxcm] for finding that the “paper is clearly written and well-motivated.” In what follows, we address the concerns of the reviewers.\n\n---\n\n> **Weakness 1.** “I am not very sure whether such representation could correctly model the surface details (or pose dependent deformation) of the hand because Fourier series based representation seem to generate over smoothed results due to its low dimension nature of shape flow.”\n> \n\n**Reply:** We want to clarify that we used Fourier series **along the temporal** **axis** (Eq. (1) in the paper) to model the smooth temporal change of hand shapes for 4D reconstruction. Although such representation may cause over-smoothed shape change **along the temporal (*t*) axis**, our shape prediction **along the spatial (*x, y, z*) axes** is not bounded by the frequency-domain subspace because we did not apply Fourier series along the spatial axes unlike in the existing methods [9, 13, 16]. We will make sure to clarify this point in the revision. \n\n> **Question 1.** “How to obtain the gt 3d shape of InterHand2.6M for training? Is that simply the MANO labeling? If true, what’s the advantage of using implicit representation instead of MANO for hand shape learning?”\n> \n\n**Reply:** The ground truth shapes of InterHand2.6M [26] are given as meshes with MANO [34] topology. Following the 3D/4D reconstruction methods [15, 16, 18, 24, 25, 29, 39] that similarly use the ground truth meshes with a fixed topology (e.g. MANO [34] or SMPL [21] meshes) for training implicit functions, we believe that using implicit representation introduces several advantages over mesh-based representation. Most importantly, implicit representation utilizes the pixel-aligned features directly corresponding to the **dense** query points, often leading to more accurate shape reconstructions that are better aligned to the input image [9, 18]. In Table 1 in the paper, we experimentally demonstrated that our method reconstructs more accurate shapes than the existing methods based on MANO parameter [44, 45] or MANO vertex regression [19] on InterHand2.6M. We also note that, as implicit representations learn continuous shape field, they can naturally reconstruct shapes in an arbitrary resolution (including the extrapolated resolution beyond the observed ground truth resolution). In Figure R1 (in PDF), we show the qualitative reconstruction examples of the current state-of-the-art mesh-based reconstruction method [19] and ours on InterHand2.6M dataset, where ours produces more plausible hand shapes. We will add this discussion in the revision.\n\n> Question 2. *“L. 228, what does ^2 mean? Is that a mistake?”*\n> \n\n**Reply:** ^2 denotes *footnote 2*. We will change the footnote symbol for better clarity in the revision. Thank you for pointing this out."}, {"review_id": "4hoTNyixUs", "reviewer": "Reviewer_Uqr4", "summary": "This paper introduces an implicit spatio-temporally continuous hand representation for RGB videos. Firstly, based on LEAP [25], the occupancy function and LBS weights are pretrained as priors for query points. Then two query flow representations are introduced to model the skeleton and the shape, respectively. The query flow representations are Fourier coefficients, which can be treated as a low-pass filter for 4D representations. Experiments on InterHand2.6M and RGB2Hands show the proposed method achieves accurate and efficient 4D predictions.", "questions": "See weaknesses.", "limitations": "Yeah.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed 4D implicit representation is highly efficient during both training and testing.\n- The Fourier coefficients are effective to get smooth and continuous temporal dynamics.", "weaknesses": "- [Balance] This paper proposes using Fourier coefficients to model 4D hand and emphasizes this will guarantee smooth and continuous temporal dynamics. Especially, it learns coefficients for N=6 basis functions (Line 175). I am wondering how Fourier coefficients and N balance the accuracy, the efficiency and the smoothness. For example, will N=6 lead to over smooth? why not use a larger N? is it because of efficiency?\n\n- [Optimisation] Unlike existing representations, the proposed representation is coefficients. I am wondering if this will raise any training or optimation issues. For example, will this representation require more epochs to converge or be harder to fit during training?\n\n- [$\\Psi$] For pose flow, there is an off-the-shelf pose estimator $\\Psi$. I'm curious about the role this pose estimator plays. Is it like a condition that directly determine the final performance or an initial values that only reduce search space (Line 209) or a refinement (Tab.3)? Would it possible to use different $\\Psi$ with varied performance and see the impact on the final results during training? Also, what will happen if $\\Psi$ provides bad predictions during testing? What if we do not have $\\Psi$?\n\n- [Metric] It would be better to compare SOTA methods like [19,44,45] wrt other metrics like MPJPE, even if they focus on a single frame or are based on MANO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and finding that the proposed method is “novel and interesting” [vgcD] and shows “large improvements” [rxcm, g8GH]. We also appreciate [rxcm] for finding that the “paper is clearly written and well-motivated.” In what follows, we address the concerns of the reviewers.\n\n---\n\n> **Weakness 1.** “[Balance] This paper proposes using Fourier coefficients to model 4D hand and emphasizes this will guarantee smooth and continuous temporal dynamics. [...] why not use a larger N? is it because of efficiency?”\n> \n\n**Reply:** In Table S1 in the supplementary, we provide our 4D hand reconstruction results with the varying value of $N =$ {$4, 6, 8$}. As discussed in lines 19-21 in the supplementary, our model performance is slightly decreased when $N$ is too small ($N = 4$), while it is not affected much by $N$ when $N ≥ 6$. Thus, we choose $N = 6$ in our main method to achieve a good balance between accuracy and computational efficiency. We did not consider a value beyond 8 for $N$, as it exceeds the Nyquist frequency given $T = 17$. \n\n> **Weakness 2.** “[Optimisation] Unlike existing representations, the proposed representation is coefficients. I am wondering if this will raise any training or optimization issues. For example, will this representation require more epochs to converge or be harder to fit during training?”\n> \n\n**Reply:** Although our model prediction is done in the coefficients space, the whole forward process is differentiable (note that Eq. (1) is defined by the linear combination of sine and cosine functions), and we have observed no particular difficulty during the model optimization. About the training convergence, in Figure R2 (in PDF), we show the validation loss graphs of our model and the variation of our model based on the direct shape-space prediction (*Fourier → InputCond* in Table 4 in the paper). There is no significant difference in the training convergence trend, and both models converge approximately after 100K steps in training.\n\n> **Weakness 3.** “[Ψ] For pose flow, there is an off-the-shelf pose estimator Ψ. I'm curious about the role this pose estimator plays. Is it like a condition that directly determine the final performance or an initial value that only reduce search space (Line 209) or a refinement (Tab.3)? Would it possible to use different Ψ with varied performance and see the impact on the final results during training? Also, what will happen if Ψ provides bad predictions during testing? What if we do not have Ψ?”\n> \n\n**Reply:** $\\Psi$ provides the initial (noisy) pose values to reduce the search space of our model prediction, and here we demonstrate that our final shape accuracy is quite robust to the quality of the initial pose predictions. In the table below, we evaluate our 4D reconstruction results using two different models for $\\Psi$ [19, R1]. While the initial pose accuracies have a considerably large gap (3.7mm in MPJPE), our final shape accuracies do not vary much, and **both settings achieve the state-of-the-art 4D reconstruction results** compared to the baseline methods in Table 1 in the paper. \n\n| $\\Psi$ | Initial joint error (MPJPE $\\downarrow$) | Shape accuracy (IoU $\\uparrow$) | Shape accuracy (CD $\\downarrow$) |\n| --- | --- | --- | --- |\n| **IntagHand [19]** | 13.3 | 62.8 | 4.46 |\n| **DIGIT [R1]** | 17.0 | 62.2 | 4.58 |\n\n[R1] Fan *et al.*, Learning to disambiguate strongly interacting hands via probabilistic per-pixel part segmentation. In 3DV, 2021.\n\nTo additionally examine “*what will happen if $\\Psi$ provides bad predictions during testing”*, we also measured our 4D reconstruction accuracy after injecting uniformly sampled noises with the amplitude [−$x$, +$x$] to every dimension of the initial pose prediction in the test time — following the exact ablation study setting in [16] (Section D in the supplementary). As shown in the table below, our model performance is quite robust to the injected noises (gently decreasing its accuracy for a higher level of noise), and again, **all settings achieve the state-of-the-art results** compared to the baselines in Table 1 in the paper.\n\n| Noise level ($x$)  | Shape accuracy (IoU $\\uparrow$) | Shape accuracy (CD $\\downarrow$) |\n| --- | --- | --- |\n| **0mm (No noise)** | 62.8 | 4.46 |\n| **1mm** | 62.7 | 4.48 |\n| **3mm** | 62.5 | 4.50 |\n| **5mm** | 62.2 | 4.53 |\n\nCurrently, our method does not assume a situation where $\\Psi$ does not exist. We will further investigate this direction in our future work.\n\n> **Weakness 4.** “[Metric] It would be better to compare SOTA methods like [19,44,45] wrt other metrics like MPJPE, even if they focus on a single frame or are based on MANO.”\n> \n\n**Reply:** Thank you for your suggestion. Although our primary goal of this research was accurate shape reconstruction rather than joint estimation, we additionally evaluated our results on MPJPE in comparison to [19, 44, 45], as shown in the table below. As [19] and [45] originally use the additional *ground truth bone length* for rescaling the joint predictions, we use the mean bone length of the hands in the training set of InterHand2.6M [29] following [18] to perform fair comparisons. Our method achieves a lower (better) MPJPE than the compared methods on InterHand2.6M dataset.\n\n| $\\Psi$ | Joint error (MPJPE $\\downarrow$) | \n| --- | --- |\n| **ACR\\*[44]** | 18.3 |\n| **Two-Hand-Shape-Pose [45]** | 15.9 |\n| **IntagHand [19]** | 13.3 |\n| **Ours** | 11.0 |"}, {"review_id": "wvWsdfFOlY", "reviewer": "Reviewer_s3gm", "summary": "This paper introduces FourierHandFlow - an implicit 4d representation for learning spatio-temporal hand shape deformations. \nThe core idea is to introduce a coarse-to-fine implicit deformation model parameterized with a fixed Fourier basis to ensure smoothness and efficient inference. The coarse (pose / joint flow) part models the dynamics of the joints conditioned on images and pose predictions, and the fine part models the dynamics of the per-query point deformations on top of the joint flow, conditioned on image features. \nExperimental evaluation is conducted on InterHands2.6M and indicates that the proposed method outperforms recent baselines in terms of quality of shape reconstruction.\n", "questions": "- Could you please clarify what exactly is the geometry you are using for training the occupancy/lbs models and for evaluation? If it is MANO fitted shapes, then, if your model does not have any appearance, I am confused how can it learn any more detailed reconstruction than the underlying low-resolution parametric model?\n- Why is there no comparison to a simple MANO model fitted to whatever InterHand2.6m constraints are available?\n", "limitations": "Authors discuss limitations and broader societal impact. ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Overall coarse-to-fine flow formulation generally makes sense, as most of the local per-point deformations would be strongly dependent on the joints. \n- Proposed formulation uses a fixed Fourier basis parameterisation of the flows is interesting, and bears multiple benefits:\n  a) it allows for more efficient inference, since re-sampling points over temporal domain can be computed in closed form for different timesteps.\n  b) it is guaranteed to lead to smooth trajectories by construction.\n- Implicit 4D formulation leads to automatic learning of correspondences, which allows texture transfer.\n- Quantitative results indicate that the proposed method outperforms multiple recent baselines (although there is not enough clarity on the evaluation protocol, see below).", "weaknesses": "1. The paper is at times hard to follow. \n- For example, when introducing the method (L136), authors do not really specify an exact form of the underlying representation, and vaguely refer to it taking a sequence of RGB frames as input. Does the overall method take any other sources of supervision or conditioning? \n- Similarly, from Section 3.1 it is not really possible to understand what is the ground truth used to pre-train the occupancy and LBS functions. Do you have geometry ground truth here? What is the resolution of this ground truth (a short look on InterHand2.6 suggests that geometry that is provided is coming from MANO)?\n- After going through the experimental section I am still not sure I understand what is the ground truth geometry that this paper is comparing against. \n\n2. Motivation / quality.\n- The quality of the resulting geometry seems to be of low resolution and does not contain any high-frequency details. \n- The main argument for using more complex machinery of implicit 4D representations over explicit mesh-based representations (L108) is the ability of those to capture high-resolution geometry details. \n- Yet, the resulting meshes seem to contain no high-resolution details, and examples e.g. in Figure 3 show that all the considered methods struggle at rough alignment with the images - which should be possible to solve with more robust keypoint / segmentation constraints.  \n- The question thus arises if there is enough motivation for the development of the complex implicit machinery. If the resolution is the only limiting factor, why not upsample the mesh of MANO and fit it to ground truth scans, followed by a simple image2shape regression? \n\n3. Evaluation / baselines.\n- There is no comparison to a parametric model MANO fitted to sparse constraints such as keypoints / segmentation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and finding that the proposed method is “novel and interesting” [vgcD] and shows “large improvements” [rxcm, g8GH]. We also appreciate [rxcm] for finding that the “paper is clearly written and well-motivated.” In what follows, we address the concerns of the reviewers.\n\n---\n\n> **Weakness 1-1.** “For example, when introducing the method (L136), authors do not really specify an exact form of the underlying representation, and vaguely refer to it taking a sequence of RGB frames as input. Does the overall method take any other sources of supervision or conditioning?”\n> \n\n**Reply:** During training, our method takes as input a sequence of monocular RGB frames (lines 136-137) and outputs 4D hand shapes in the form of an occupancy field, which is supervised by the ground truth 4D hand geometry (i.e., a temporal sequence of 3D hand geometries) of InterHand2.6M [26] dataset (lines 258-260). During testing, we only use a sequence of RGB frames as input to our method, as our main goal is to reconstruct 4D human hands from monocular RGB sequences (lines 19-20, 36-37).\n\n> **Weakness 1-2.** “Similarly, from Section 3.1 it is not really possible to understand what is the ground truth used to pre-train the occupancy and LBS functions. Do you have geometry ground truth here? What is the resolution of this ground truth?”\n> \n\n**Reply:** The same InterHand2.6M [26] hand geometries are used for pre-training the occupancy and LBS functions. These ground truth shapes are given as meshes with the MANO [34] topology.\n\n> **Weakness 1-3.** “After going through the experimental section I am still not sure I understand what is the ground truth geometry that this paper is comparing against.”\n> \n\n**Reply:** We used \"the ground truth shapes of InterHand2.6M dataset\" (line 268-269).\n\n> **Weaknesses 2-1 - 2.3.** “[...] The main argument for using more complex machinery of implicit 4D representations over explicit mesh-based representations (L108) is the ability of those to capture high-resolution geometry details. Yet, the resulting meshes seem to contain no high-resolution details. [...]\n> \n\n> **Weakness 2-4.** “The question thus arises if there is enough motivation for the development of the complex implicit machinery. [...]”\n> \n\n> **Question 1.** \"I am confused how can it learn any more detailed reconstruction than the underlying low-resolution parametric model?\"\n\n**Reply:** In Figure 3 in the paper, we showed the qualitative results of ours in comparison to the same implicit function-based methods only. In Figure R1 (in PDF), we additionally show the reconstruction examples of the current state-of-the-art mesh-based method [19] and ours on InterHand2.6M [26] dataset. As implicit representations learn continuous shape fields, they can naturally reconstruct shapes in an arbitrary resolution (including the extrapolated resolution beyond the observed ground truth resolution). Following the existing methods [15, 16, 18, 24, 25, 29, 39] that similarly use the ground truth MANO [34] of SMPL [21] meshes for training implicit functions, we believe that using implicit representation introduces several advantages including (1) the aforementioned arbitrary-resolution resolution reconstruction and (2) more accurate shape modeling due to the utilization of the pixel-aligned features directly corresponding to **dense** query points. In Table 1 in the paper, we also experimentally demonstrated that our method reconstructs more accurate shapes than the existing methods based on MANO parameter [44, 45] or MANO vertex regression [19] on InterHand2.6M. \n\nAbout (1) using \"keypoint / segmentation constraints\" and (2) \"upsampl[ing] the mesh of MANO and fit[ting] it to ground truth scans\" mentioned in the review, please refer to our answer to the next questions.\n\n> **Weakness 3-1.** “There is no comparison to a parametric model MANO fitted to sparse constraints such as keypoints / segmentation.”\n> \n\n> **Question 2.** *“Why is there no comparison to a simple MANO model fitted to whatever InterHand2.6m constraints are available?”*\n> \n\n**Reply:** We would like to kindly remind you that **we aimed to perform 4D hand reconstruction from monocular RGB sequences in this paper**. Thus, **only RGB observations are available as inputs to our method — making it non-trivial to directly perform MANO fitting to \"keypoints\", \"segmentation\", or \"ground truth scans\" as mentioned**. Thus, for experimental comparisons, we mainly considered the existing methods [18, 19, 44, 45] with the state-of-the-art *RGB-based* reconstruction results on InterHand2.6M [26]. Although it is possible to consider taking a multi-stage-based approach, in which we first predict the keypoints and segmentation masks from the input RGB frames and then perform optimization-based MANO fitting to those predictions, the accuracy of the fitted MANO would be directly bounded to the accuracy of the intermediate predictions. Also, considering that one of the objectives of this paper was the *\"computational efficiency of 4D reconstruction\"* (lines 41-42), a MANO fitting-based approach would lead to a less efficient solution due to the need for test-time optimization. In the table below, we show the experimental results of such MANO fitting using the state-of-the-art keypoint and segmentation predictor (IntagHand [19]) on InterHand2.6M [26] dataset, which yields less accurate and less efficient reconstruction results than ours. For the comparisons with the existing methods based on the feedforward MANO parameter prediction [45] or MANO vertex coordinates prediction [19] directly from images - which also intermediately utilize the *predicted* joints and segmentation priors - please refer to Table 1 in the paper.\n\n|  | Shape accuracy (IoU $\\uparrow$) | Shape accuracy (CD $\\downarrow$) | Inference time (sec. $\\downarrow$) |\n| --- | --- | --- | --- | \n| **MANO [34] fitting** | 43.7  | 6.48 | 28.4 |\n| **Ours** | 62.8 | 4.46 | 0.22 |"}, {"review_id": "kCT0iCKpTL", "reviewer": "Reviewer_vgcD", "summary": "The paper proposes a method to reconstruct 4D hand (3D hand sequence) from a short RGB sequence with two types of Fourier Query Flow (pose flow and shape flow). In the Fourier Query Flow, the 3D trajectory of each point is transformed into 3 Fourier series along the time dimension and represented with the first few coefficients (3*(2*6+1) = 39 in this paper). Pose flow is generated with joint flow via LBS. The geometry is represented in the canonical space with a pretrained occupancy network and warped into the real space with the pose flow. Then shape flow adds small displacements to it. Experiments demonstrate that the proposed method outperforms existing methods and can produce continuous and smooth results.", "questions": "1.\tIt seems that using the low frequency representation can prevent temporal jitters or abrupt motions. However, lack of high frequency parts may lead to Gibbs phenomenon. So, I want to see some discussion about this. Does your method still work in the swift movement situation?\n2.\tIt seems that the shape flow is working in the real space (not the canonical space), which is different from some methods, like MANO. So, I wonder if it works for different hand shapes, like longer finger or wider palm. If it works, I’d like to see some examples with different hand shapes.\n3.\tHow about applying Fourier Query Flow to MANO model directly but not a pretrained occupancy network? I don’t see why using occupancy network would be better in this scenario. I’d like to see more discussion on this. \n", "limitations": "I appreciate the authors’ discussions about limitations of the proposed approach in Sec. 5, which helps the understanding of the suitable scenarios.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: The idea of represent 3D flow with Fourier series is novel and interesting. \n\nQuality: The presented results is of high-quality.\n\nClarity: The paper is well-written and easy to follow.\n\nSignificance: It seems that this method prevents the results from temporal jitters or abrupt motions, which I’ll discuss in Questions. And it’s very computational efficient. \n", "weaknesses": "I do not see some major weakness. But I still have some questions about this method. I’ll leave them to the Question part. And it would be great if authors can add these discussions to the paper. \n\nA typo: line 228 t()^2 ?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and finding that the proposed method is “novel and interesting” [vgcD] and shows “large improvements” [rxcm, g8GH]. We also appreciate [rxcm] for finding that the “paper is clearly written and well-motivated.” In what follows, we address the concerns of the reviewers.\n\n---\n\n> **Weakness 1.** \"A typo: line 228 t()^2 ?\"\n\n**Reply:** ^2 denotes *footnote 2*. We will change the footnote symbol for better clarity in the revision. Thank you for pointing this out.\n\n> **Question 1.** “It seems that using the low frequency representation can prevent temporal jitters or abrupt motions. However, lack of high frequency parts may lead to Gibbs phenomenon. So, I want to see some discussion about this. Does your method still work in the swift movement situation?”\n\n**Reply:** While Gibbs phenomenon *theoretically* occurs when approximating a discontinuous function using a finite Fourier Series, we have observed that the function that we aim to reconstruct -- the hand motion captured in InterHand2.6M [26] -- is mostly smooth enough to be reconstructed with a number ($N=6$) of Fourier terms. In Table S1 in the supplementary, we quantitatively showed that our shape reconstruction accuracy does not noticeably increase when using a higher number of Fourier terms than $6$ on InterHand2.6M dataset. Also, *Sequences 1 and 2* of our supplementary video (0:13-0:42) qualitatively show one of the swiftest hand motions captured in the dataset, and we have not *noticeably* observed the Gibbs phenomenon in the reconstructed motion. However, if one has to reconstruct higher-frequency motions (other than the hand motions in InterHand2.6M), it would be desirable to use a higher number of Fourier terms ($N>6$) in our method. We will add this discussion in the revision.\n\n> **Question 2.** “It seems that the shape flow is working in the real space (not the canonical space), which is different from some methods, like MANO. So, I wonder if it works for different hand shapes, like longer finger or wider palm. If it works, I’d like to see some examples with different hand shapes.”\n\n**Reply:** As you mentioned, our hand shape reconstruction itself is not constrained by MANO [34] PCA subspace. However, we have found that the hands captured in InterHand2.6M [26] dataset do not contain significant shape variations across the identities. Although our implicit function-based method leads to better reconstruction results than the MANO-based prediction method [45] in Table 1 in the paper, we will try to show reconstruction examples with more diverse hand shapes using the *very recently-released* dataset [R1] containing more hand shape variations. Thank you for your valuable comment.\n\n[R1] Potamias *et al.*, Handy: Towards a high fidelity 3D hand shape and appearance model, In CVPR, 2023.\n\n> **Question 3.** “How about applying Fourier Query Flow to MANO model directly but not a pretrained occupancy network? I don’t see why using occupancy network would be better in this scenario. I’d like to see more discussion on this.”\n> \n\n**Reply:** We thought that one of the important advantages of Fourier Query Flow is its ability to inherently model the **continuous** shape deformations along the temporal axis, e.g., allowing temporal shape inter- and extrapolation (Figure 5 in the paper). Although it is still possible to apply Fourier Query Flow to the MANO model (which is spatially discretized) directly, we wanted to preserve the continuity in the representation along both the spatial and the temporal axes — to keep the consistency between the axes and also to follow the motivation of the existing 4D continuous representations [14, 15, 29]. For example, we have observed that our method can generate more plausible hand shapes due to the capability for reconstruction in an arbitrary spatial resolution -- as shown in Figure R1 (in PDF). We will make sure to add this discussion in the revision."}]

Author rebuttal:
We provide the figures referred to in our author response in the PDF file below.

Meta-review: This paper got mostly positive ratings  (A, WA, WA, BA, R) from reviewers. 

The major strength of this paper lies in their spatio-temporally continuous implicit representation based on as Fourier series, supported by the performance improvements shown in their experiment. 

The major concern particularly raised by the reviewer on the negative side is regarding the less clear presentations on some parts and the motivation of the approach in terms of applying implicit representation on the relatively spare MANO mesh GTs. 

The authors resolved most concerns by other reviewers in their rebuttal and discussion phase. 

The AC is inclined to the positive side, given the novel part in their representation and experimental demonstrations. The AC also found that the authors convincingly answered the questions and concerns raised by reviewers. Overall, the paper meets the bar of NeurIPS. 
Authors should very carefully check the reviewer’s feedback and improve the quality of the paper for the camera-ready version.
Decision: Accept (poster)