Pose Modulated Avatars from Video
Authors: Chunjin Song
Bastian Wandt
Helge Rhodin
Venue: ICLR 2024 poster (2024)
Keywords: NeRF
Neural Rendering
Dynamic Avatars
Frequency Modulation
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: It is now possible to reconstruct dynamic human motion and shape from a sparse set of cameras using Neural Radiance Fields (NeRF) driven by an underlying skeleton. However, a challenge remains to model the deformation of cloth and skin in relation to skeleton pose. Unlike existing avatar models that are learned implicitly or rely on a proxy surface, our approach is motivated by the observation that different poses necessitate unique frequency assignments. Neglecting this distinction yields noisy artifacts in smooth areas or blurs fine-grained texture and shape details in sharp regions. We develop a two-branch neural network that is adaptive and explicit in the frequency domain. The first branch is a graph neural network that models correlations among body parts locally, taking skeleton pose as input. The second branch combines these correlation features to a set of global frequencies and then modulates the feature encoding. Our experiments demonstrate that our network outperforms state-of-the-art methods in terms of preserving details and generalization capabilities. Our code is available at https://github.com/ChunjinSong/PM-Avatars.

Published as a conference paper at ICLR 2024
POSE MODULATED AVATARS FROM VIDEO
Chunjin Song', Bastian Wandt & Helge Rhodin 1.3
'Department of Computer Science. University of British Columbia
*Department of Electrical Engineer, Linköping University BBielefeld University
{chunjins rhodi n. ) css ubc. ca bastian wandt @ tandtt@ii... ailu.
ABSTRACT
is now possible to reconstruct dynamic human motion and shape from sparse set of cameras using Neural Radiance Fields (NeRF) driven by an underlying skeleton. However, challenge remains model the deformation of cloth and skin in relation to skeleton pose. Unlike existing avatar models that are learned implicitly or rely on a proxy surface, our approach is motivated by the observa- tion that different poses necessitate unique frequency assignments. Neglecting this distinction yields noisy artifacts in smooth areas or blurs fine-grained tex- ture and shape details in sharp regions. We develop two-branch neural net- work that adaptive and explicit in the frequency domain. The first branch a graph neural network that models correlations among body parts locally, taking skeleton pose as input. The second branch combines these correlation features set of global frequencies and then modulates the feature encoding. Our ex- periments demonstrate that our network outperforms state-of-the-ar methods terms of preserving details and generalization capabilities. Our code is available at htt com/ Cnnnnnnnnnnnnnnnnnnoo Avatars
INTRODUCTION
Human avatar modeling has garnered significant attention as enabling 3D telepresence and digitiza- tion with applications ranging from computer graphics (Wu et al.) 2019 |Baauttinnvv et al.) 2021 Peng et al., 2021a Lombardi al. 2021 medical diagnosis (Hu et 2022). To tackle this chal- lenge, the majority of approaches start from a skeleton structure that rigs a surface mesh equipped with neural texture (Bagautdinov et al., 2021 Liu al. 202T) or learnable vertex features (Kwon et al. 2021 Peng al. 2021 b). Although this enables reconstructing intricate details with high precision (Liu et al., 2021) et al. 2019) in controlled conditions, artifacts remain when learning the pose-dependent deformation from sparse examples. To counteract, existing methods typically rely on a parametric template obtained from a large number laser scans, which still limits the variety of the human shape and pose. Moreover, their explicit notion of vertices and faces is diffi cult to optimize and can easily lead to foldovers and artifacts from other degenerate configurations. Moreover, the processed meshes are usually sampled uniformly in one static pose thus not being adaptive to dynamic shape details like wrinkles.
Our objective is to directly reconstruct human models with intricate and dynamic details from given video sequences with a learned neural radiance field (NeRF) model (Mildenhall et al. 2020). Most related are surface-free approaches such A-NeRF (Su et al. 2021) and NARF (Noguchi et al. 202T) that directly transform the input query points into relative coordinates of skeletal joints and then predict density and color for volumetric rendering without an intermediate surface represen- tation. To further enhance the ability to synthesize fine details, (Su et al.) 2022, Li et al. 2023) explicitly decompose features into local part encodings before aggregating them to the final color. Closely related are also methods that learn neural radiance field of the person in canonical T- pose (Jiag et al. 2022 Li et al., 2022at Wang et al. 2022). Despite their empirical success, as depicted in Fig. D it is evident that single query point, when considered in different pose texts, difficult to be learned. Specifically, the T-shirt region appears flat in one pose (2nd row) but transforms into highly textured surface in another (1*t row). The commonly used positional

---

Published as a conference paper at ICLR 2024
Fina
Smooth
Reference
Close-up
A-NeRE
DANSO
Frequency
Figure 1: Motivation. Our frequency modulation approach enables pose-dependent detail by using explicit frequencies that depend on the pose context, varying over frames and across subjects. Our mapping mitigates artifacts in smooth regions well as synthesizes fine geometric details faithfully, e.g., when wrinkles are present (1 row). By contrast, existing surface-free representations struggle either with fine details (e.g. marker) or introduce noise artifacts in uniform regions (e.g. the black clouds, 2nd row). To quantify the difference in frequency deviation (STD) pixels within 5 patches of the input closeup and illustrate the frequency his- tograms of the reference. Even for the same subject in similar pose the frequency distributions are distinct, motivating our pose-dependent frequency formulation.
these cases, we calculate the standard
encoding MMildenhall al., 2020) maps points with fixed frequency transformations and is hence non-adaptive. Entirely implicit mappings from observation to canonical space are complex and con- tain poo-dd one to many settings. Thus they struggle explicitly correlate pose context with the matching feature frequency bands of query points. As a result, these methods either yield overly smoothed details or introduce noisy artifacts smooth regions.
In this paper, investigate new ways of mapping the skeleton input to frequencies of dynamic NeRF model to tackle the aforementioned challenges. Specifically, we design network with branch that is explicit in the frequency space and build multi-level representation that adapts to pose dependencies. To accomplish this, we utilize Sine functions as activations, leveraging its explicit notion of frequency and its capability to directly enforce high-frequency feature transfor- mations (Sitzmann et al., 2020; Mehta et al. 2021 Wu et al.) (023). The main challenge that we address here is on how control the frequency the Sine activation for deforming characters.
We first apply a graph neural network (GNN) (Su et al., 2022) on the input pose to extract corre- lations between skeleton joints, thereby encoding pose context. Given query point coordinates for NeRF rendering, the joint-specific correlation features are first combined with part-level feature aggregation function and then utilized to generate point-dependent frequency transformation coef- ficients. The frequency modulation process takes place within set of intermediate latent features, allowing us to optimize the modulation effects at different scales effectively. Lastly, similar to ex- isting NeRF methods, we output density and color to synthesize and render images. Across various scenarios, we consistently outperform state-of-the-an methods. Our core contributions are
We introduce a novel and efficient neural network with two branches, tailored generate high-fidelity functional neural representations of human videos via frequency modulation.
A simple part feature aggregation function that enables high-frequency detail synthesis in sharp regions and reduces artifacts near overlapping joints.
We conduct thorough evaluation and ablation studies, which delve into the importance of window functions and frequency modulations with ofate-of-the-ar results.
2 RELATED WORK
Our work is in line with those that apply neural fields model human avatar representations. Here we survey relevant approaches on neural field (Xie et al. 2022) and discuss the most related neural avatar modeling approaches.
2

---

Published as a conference paper at ICLR 2024
Neural Fields. As successful application, a breakthrough was brought by Neural Radiance Fields (NeRF) (Mildenhall et al., 2020). Recently, extending NeRF to dynamic scenes becomes more
and more popular and enables numerous downstream applications (Park et al., 2021a Pumarola eet al., 2021 Park et al.] 2021b] Cao & Johnson 2023). The key idea is to either extend NeRF with additional time dimension (T-NeRF) and additional latent code (Gao al. 2021: Li al.) 2022b 2021b), or to employ individual multi-layer perceptrons (MLPs) to represent a time-varying deformation field and static canonical field that represents shape details (Du al. 2021 Park al.,
2021a by Tretschk et al. 2021; Yuan et al. 202T). However, these general extensions from static to dynamic scenes only apply to small deformations and do not generalize to novel input poses.
Our method is also related to applying periodic functions for high frequency detail modeling within the context of neural fields. NeRF (Mildenhall et al., 2020) encodes the 3D positions into a high- dimensional latent space using a sequence of fixed periodic functions. Later on, Tanciker al. (Tancik) jet al., 2020) carefully learn the frequency coefficients of these periodic functions, but they are still shared for the entire scene. In parallel, Sitzmannet al. (Sitzmann et al. (2200) directly uses a Sine- function as the activation function for latent features, which makes frequency bands adaptive to the input. (Lindell et al., [022] Fathony et al.) 2021) further incorporate the multi-scale strategy of spectral domains to further advance the modeling of band-limited signals. Recently, (Hertz jet al., 2021; Mehta et al., 2021 Wu et al., (023) propose to modulate frequency features based on spatial patterns for better detail reconstructions. However, differing from these methods, we explicitly associate the desired frequency transformation coefficients with pose context, tailored to the dynamics in human avatar modeling.
Neural Fields for Avatar Modeling. Using neural networks model human avatars (Loper al., 2015) is widely explored problem (Deng et al., [2200] Saito et al. [021]] Chen et al. (2021). How- ever, learning personalized body models given only videos of single avatar is particularly chal- lenging which is our research scope in this paper.
In the pursuit of textured avatar modeling, the parametric SMPL body model is a common basis. For instance, (Zheng et al. 2022) (223) propose partitioning avatar representations into local radi- ance fields attached to sampled SMPI. nodes and learning the mapping from SMPL, pose vectors to varying details of human appearance. On the other hand, approaches without a surface prior, such as A-NeRF (Su al. (02T) and NARF (Noguchi et al.) 202T), directly transform the input query points into relative coordinates of skeletal joints. Later on, TAVA (Li et al. (202aa) jointly models non-rigid warping fields and shading effects conditioned on pose vectors. ARAH (Wang et al. 2022) explores ray intersection NeRF body model initialized using a pre-trained hypernetwork. Similar to for- mer disentanglement (Wu et al.] 2019 Song et al. 2019 Wu et al. 2020 (022). DANBO (Su et al.] 2022) applies a graph neural network to extract part features and decompose an independent part feature space for a scalable and customizable model. To reconstruct high-frequency details, Neural Actor (Liu et al. 202I) utilizes an image-to-image translation network learn texture mapping, with a constraint on performers wearing tight clothes for topological consistency. Further studies (Peng al. 2021b a Dong et al. 2022) suggest assigning global latent code for each training frame to compensate for dynamic appearance. Most recently, HumanNeRF (Weng et al. 2022) and its following works like Vid2Avatar (Guo et al., (023) and MonoHuman (Yu et al. 2003) show high-
fidelity avatar representations for realistic inverse rendering from monocular video. Despite the significant progress, none of them explicitly associate the pose context with frequency modeling. which we show is crucial for inereasing shape and texture detail.
3 METHOD
Our objective is to reconstruct 3D animatable avatar by leveraging collection of N images, along with the corresponding body pose represented as the sequence of joint angles [0=] Our key technical ingredient is pose-guided frequency modulation network and its integration for avatar reconstruction. Fig. 2 provides a method overview with three main components. First, we employ a Graph Neural Network (GNN) to estimate local relationships between different body parts. The GNN facilitates effective feature aggregation across body parts, enabling to learn the nearby pose contexts without relying surface priors. Then the aggregated GNN features are learned to mod- ulate the frequencies of input positions. Lastly, the resulting per-query feature vector is mapped to the corresponding density and radiance at that location as in the original NeRF framework.
3

---

Published as a conference paper at ICLR 2024
Sin
Funttion
Figure 2: Method overview. First, a graph neural network takes skeleton pose input encode correlations of joints. Together with the relative coordinates {ā;} } of query point æ, a window func- tion is learned to aggregate the features from all parts. Then the aggregated GNN features are used to compute frequency coefficients (orange) which later modulate the feature transformation of point (green). Finally, density a and appearance e predicted in NeRF.
3.1 PART-RELATIVE POSE ENCODING
Inspired by DANBO (Su et al. 2022). we adopt a graph representation for the human skeleton, where each node corresponds to joint that is linked to neighboring joints by bones. For given pose with Ng joints = [w1,w2 w2........ WNnl where w; E I (Zhou et al., 2019) is the rotation parameter of bone (1, 2, ...,, Nn), we regress feature vector for each bone part as:
[G1, G2...,, GN = GNN()),
where GNN represents a learnable graph neural network. To process the irregular human skeleton, we employ two graph convolutional layers, followed by per-node 2 2-layer Multi-Layer Perceptrons (MLPs). To account for the irregular nature human skeleton nodes, we learn individual MLP weights for each node; see (Su et al. 2022) for more comprehensive details.
Given a sample location a E R3 in global coor- dinates for which the NeRF should output color and density, we first map it to the i-th bone- relative space as
(w/
x
{w/""
fwe
(wx)
Wincow
Spmole
Cor muckut Layer
Min Pocling
where T(w;) denotes the world-to-bone coor- dinates transformation matrix computed by the rotation parameter w,. We first perform a valid- ness test for the scaled relative positions (#, 8e- -... where 8; is learnable scaling factor to control the size of the volume the i-th part con- tributes to. This facilitates the processing efficiency and concentrates the network on local patterns. no {a;} falls in [-1, 1], z is estimated locate far from the body surface and is discarded. Then these features are employed to drive the frequency modulation adaptively.
Figure 3: Learned window function. The query point location is processed with spatial and pose- dependent window to remove spurious correla- tions between distant joints.
3.2 FREQUENCY MODULATION
Two-stage Window Function. To decide which per-part point-wise features to pass on the down- stream network and address the feature aggregation issues in (Su et al., 2022). design a learnable two-stage window function. In Fig. the window function takes the per-part GNN feature set {G+} and the scaled relative position set { of valid point inputs. To facilitate learning volume dimensions {s;} that adapt to the body shape and to mitigate seam artifacts, we define
w exp(-a([{=.11))
where ||:1|, is the Lormo The w function attenuates the extracted feature based on the relative spatial distances to the bone centers such that multiple volumes separated via the spatial simi- larities between x and the given parts. Thus we name w as the spatial window function. We set and 6 in (Lombardi et al. (02T). However, it is possible that part might prioritize
4

---

Published as a conference paper at ICLR 2024
over other parts when multiple valid parts' feature space overlap. This motivates us to take the pose context into account and consider the point-wise feature of each part as the second stage. First, we perform 2; sin (W. -Z;) for each F, where W. indicates Gaussian initialized fully-connected layer as in FFN (Tancik et al., 2020). After concatenating { and the GNN features [G&} cor- respondingly, we apply full-connected layer regress { the point-wise feature of the i-th part. Then we attenuate the feature as ft = f wa to focus on spatially nearby parts.
To further decide which part x belongs to, we compute relative weights by aggregating all { through a max pooling for the holistic shape-level representation. The max-pooled feature is fed into sequence of fully-connected layers which are activated by a Sigmoid function to output per-part weight wy-. We call the feature window function wi to echo the spatial window function wo since it operates on pose features. Finally, we compute the per-part weight we = w wy and output modulation frequencies as
NB w; G., 10,.02,-- ... ,0,] MLP (ST)),
(4)
where ft" 9g and n represent the aggregated part feature, the modulation frequency coefficient at i-th layer and the layer number of the subsequent modulation module, respectively.
After the two-stage window function, the extracted fm sparsely correspond to a small part set. To echo the locality assumption throughout the entire network, we also perform the window function uy for the NB relative positions of query point x as {#; = T; w:). Note that, we aggregate the part features before MLP-based frequency modulation avoid time-consuming processing over all parts for all samples. Thus computation complexity reduces significantly.
Frequency Prediction. Inspired by pi-GAN (Chan et al. 2021). we build the backbone network for each x with series of Sine-activated fully-connected layers (Sitzmann et al.) 2020). To this end, we first concatenate all the re-weiheed positions { whole and input it into Sine-activated fully-connected layer (Sitzmann et al. 2020). Later on, each fully-connected layer is defined as
f sin(O; W b,),
(5)
where W; and b, are trainable weight and bias in the i-th layer L. Finally, we concatenate the Sine-activated features } as (zx) = [f, fz. ... f,] for further processing.
Design Discussions and relation to DANBO. Both our method and DANBO (Su et al. 2022) use the GNN features as a building block to measure bone correlations. By contrast to DANBO which directly estimates part-level feature spaces from GNN features, we leverage these aggregated GNN features to estimate the appropriate frequencies, driving the frequency modulation for the input positions. This enables the linked MLP networks to adaptively capture a wide spectrum of coarse and fine details with high variability, as illustrated in the visual comparisons to DANBO in Fig. [ 5 6 Moreover, although some work (Hertz et al., 2021] Wu et al.) (023) aims to modulate frequency features with locality, we make the first step for pose-dependent frequency modulation, which is critical in human avatar modeling. We start from existing conceptual components, analyze their limitations, and propose a novel solution. Specifically, we focus on how to connect GNN pose embeddings with frequency transformations. We then propose a simple window function to improve efficiency without losing accuracy which is also special and helpful in neural avatar modeling.
3.3 VOLUME RENDERING AND Loss FUNCTIONS
The output feature 5(x) can accurately capture the information of pose dependency and spatial positions, and thus enables adaptive pattern synthesis. To obtain high-quality human body, we learn a neural field to predict the color c and density a at position x as
(c,a)= = F(S(x),
(6)
where r E 2 indicates the given ray directions. Following the existing neural radiance rendering pipelines for human avatars (Su et al. 2021 2022 Wang et al. 2022), we output the image of the human subject as in the original NeRF:
(r) = Σ T1 exp(-o,,.)) c₁2 T, -xpl-
/=1
5

---

Published as a conference paper at ICLR 2024
GT
DANBO
Ours
Figure 4: Visual comparisons on MonoPerfCap. We can preserve better shape contours (1 row) and produce realistic cloth textures without artifacts (highlighted by the red arrow on 2nd row).
GT
A-NeRF
DANBO
ARAH
TAVA
Ours
Figure 5: Visual comparisons for novel view synthesis. Compared to baselines, can vividly reproduce the structured patterns.
Here, C' and 6; indicate the synthesized image and the distance between adjacent samples along given ray respectively. Finally, we compute the L1 loss ||-11, for training as
Cree
Σ - - rEDt
where the whole ray set and Cgt is the ground truth. The usage of L, loss is to enable more ro- bust network training. Following (Lombardi et al. 202T), add regularization loss the scaling factors to prevent the per-bone volumes from growing too large and taking over other volumes:
No
C.
$?),
where {sf 1,f s are the scaling factors along {z, y, 2} axes respectively. Hence, total loss with weight 7,
L = Crea + XaLa.
(10)
RESULTS
In this section, we compare our approach with several sate-of-the-an methods, including Neural- Body (Peng et al. 2021b), Anim-NeRF (Peng et al. 202 A-NeRF (Su et al.) 2021). TAVA (Li al. 2022a). DAN BO (Su et al. 2022), and ARAH Wang et al. 2022). These methods vary in their utilization of surface- free, template based, scan- based priors. We also conduct an ablation study to assess the improvement achieved by each network component. This study analyzes and discusses the effects of the learnable window function. Source code will be released with the publication.

---

Published as a conference paper at ICLR 2024
GT
A-NeRF
DANBO
ARAH
TAVA
Ours
Figure 6: Visual comparisons for novel pose rendering. For novel poses that are unseen during training. cloth wrinkles form chaotically. Hence, none of the methods is expected to match the folds. Ours yields the highest detail, including the highlighted marker texture.
Table I: (a) Unseen pose synthesis on MonoPerfCap Xu et al. (2018). Our full model shows better overall perceptual quality chosen models from monocular videos. (b) Ablation studies on Human3.6.1 S9. Our full model outperforms all ablated baselines across all metrics.
onlGNNN BoGNN onlyShy only of only or! no vindow Ours (Tall)
Novel view
Aug
PSNR
25.91
26.08
25.86
26.21
25.91
22.21
26.39
0,929
PSNR1
SSIM;
PSNR?
SSIM?
PSNR1
SSIM)
LPIPS]
0.108
0.118
0.112
0.124
0.438
0.100
oslyGNN
20.03
0.841
22.71
0.863
21.37
0.852
Novel pose
onlySom
DANBO
20.10
19.37
0.827
0.842
22,66
22.39
0.861
0.560
21.25
21.12
0.852
0.844
PSNR
SSIMI
0.991
24.75
0.899
24.45
0.901
24.82
0.904
25.01
0.900
24.72
01028
20.98
0.908
25.12
Ours
20.55
0,853
22.85
0.866
21.70
0,860
0.135
0.502
4.1 EXPERIMENTAL SETTINGS
We evaluate our method on widely recognized benchmarks for body modeling. Following the pro- tocol established by Anim-NeRE
perform comparisons the seven actors of the Human3.cm .6M dataset (Ionescu et al. 2011 2013] Peng et al.) (2021a) using the method described in (Gong et al, 2018 compute the foreground maps. Like DANBO, also apply MonoPerfCap (Xu al., 2018 as a high-resolution dataset to evaluate the robustness to unseen poses in monocular videos.
To ensure fair comparison, split and used metrics (Su et al. 2021 2022 Li et al. 2022a). Specifically, we utilize standard image metrics such as pixel-wise Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Metric (SSIM) (Wang et al.) 2004) to evaluate the quality of output images. Additionally, we employ perceptual metrics like the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al.) 2018) to assess the structural accuracy and textured details of the generated images. Since our primary focus on the foreground subjects rather than the background image, we report the scores based on tight bounding boxes, ensuring the evaluation to be focused on the relevant regions of interest.
follow the previous experimental settings including the dataset
4.2 NOVEL VIEW SYNTHESIS
To evaluate the generalization capability under different camera views, we utilize multi-view datasets where the body model is learned from subset of cameras. The remaining cameras then utilized as the test set, allowing us to render the same pose from unseen view directions.
We present the visual results in Fig. Comparing the selected baselines, method shows supe- rior performance in recovering fine-grad ine-grained details, as evident in the examples such as the stripe tex- ture depicted the first row. We attribute this to the explicit frequency modulation which mitigates grainy artifacts and overly smooth regions. Tab. 2 quantifies our method's ''s empirical advantages, supporting our previous findings.

---

Published as a conference paper at ICLR 2024
GT
Ours
window
uf"
[A]
[b]
Figure 7: Ablation studies on sub-branch networks (a) and on window functions (b), Only the
full model can faithfully synthesize the structured patterns (e.g. the strip textures) in (a) and avoid artifacts and contour distortions in (b).
Reference
DANBO
Figure 8:
Geometry reconstruction. Our method yields more precise, less noisy shape estimates. Some noise remains as no template mesh or other surface prior is used.
4.3 NOVEL POSE RENDERING
We follow prior work and measure the quality of novel pose synthesis by training on the first part of video and testing on the remaining frames. Only the corresponding 3D pose is used as input. This tests the generalization of the learned pose modulation strategy and applicability to animation.
We present the visual comparisons for the Human3. 6M dataset in Fig. O where our method shows superior results than baselines terms fine-grained and consistent renderings. Specifically, our method generates sharper details such those seen in wrinkles and better texture consistency, as exemplified by the clearer marker (highlighted by boxes). Tab. 2 further quantitatively verifies that method generalizes well to both held-out poses and out-of -if-o---iitiibttion poses across the entire test set. Note that no methods match wrinkle locations perfectly, as these form chaotically, dependent on the past motion, not just the single frame used by all methods for conditioning. Learning such motion dynamics remains an open problem that is orthogonal to learning pose- dependent detail.
Moreover, we provide the visual comparisons in Fig. 4 and the quantitative metrics in Tab. [ (a) for the high-resolution outdoor monocu- lar video sequences MonoPerICap (Xu et al. 2018). Being consistent with the results on the Human3.6M sequences, our method presents better capability in learning a gener- alized model from monocular videos.
Table 2: Novel-view and novel-pose synthesis results, averaged over the Human3.6M test set. Our method benefits from the explicit frequency modulations, yielding better perceptual quality, reaching the best overall score all metrics.
Novel view
Nuvel pose
PSNRT SSIMT LPIPS
PSNR1
SSIM
LPIPS]
Furthermore, we test the animation ability of our approach by driving models learned from the Human3.6k 6.6 dataset with extreme out-of- distribution poses from AIST (Li et al. 202 The qualitative results shown in Fig. validate ap-
Template/Sean-besee based prior
NeuralBody
23.36
0.905
0.140
22.81
0.888
0.157
Anim. NeRF
23.34
0.897
0.157
22.61
0.881
0.170
23.27
Template-free
A-NeRF
24.26
0.911
0.129
23.02
0.853
0.171
DANBO
24.69
0.917
0.116
23.74
0.901
0,131
that even under extreme pose variation
TAVA
24.72
0.919
0.124
23.52
0.899
0.14]
proach produces plausible body shapes with de-
Ours
25.06
0.921
0.110
24.15
0.906
0.124
sired texture details while the baseline show se-
using public release that differs to Wang et al.
2022
vere artifacts. Here no quantitative evaluations are performed since no ground truth data is available.
4.4 GEOMETRY VISUALIZATION
In Fig. we analyze the geometry reconstructed with our approach against reconstructions from the baseline. Our method captures better body shapes and per-part geometry. Specifically, our results present overall more complete body outline and a smoother surface. contrast, the baselines pre- dict more noisy blobs near the body surface. Together with the results of novel view and novel pose
8

---

Published as a conference paper at ICLR 2024
Target
DANBO
Target
Figure 9: Animation capability. Our method maintains the reconstructed high frequency details when retargeting and creates fewer artifacts, lending itself for retargeting.
synthesis, we also attribute our more consistent rendering results across novel views than the base- lines better geometry preservation. This might suggest that more faithful modeling of geometry is also beneficial for the visual fidelity as shown in the appended video.
4.5 ABLATION STUDY RESULTS
To test the significance of each component, we conduct the ablation study on all Human3. 6.6 se- quences with six ablated models: 1. Only preserve the upper branch network with GNN features as onlyGNN; Only preserve bottom branch network without frequency modulation as onlySyn; 3. For the two-stage window function, we only preserve w. to evaluate the effectiveness of the spatial similarities between æ and all bone parts as only w": 4. We only preserve w for the im- portance of max-pooled object-level feature as only w 5. We remove the whole window function no window. 6. We concatenate all pose parameters into vector and replace the GNN lay- ers with one single MLP to evaluate the effectiveness of per-part modeling as noGNN; see Fig. [5] for detailed architecture. In Fig. 7we present the qualitative ablation study results for S9, Specifi- cally, the oNlyGNN model is prone to produce blurry textures due to its low frequency bias while the onlySyn model introduces noisy artifacts near stripe textures. Similar to onlySyn, the noGNN model blurs the stripe patterns with noisy artifacts. Only the combination yields their full advantage and successfully synthesizes structured patterns, as shown in Fig. 7 (a). As mentioned method section, w and w cater for the relationships in position space and feature space, respectively. As shown by the results of novel pose rendering (Fig. (b)), neither using w. or w/ alone suffices to produce the image quality of the full model, demonstrating the necessity of all contributions. Tab. (b) lists our corresponding quantitative results of all sequences to further support our statement.
Tab. I (a) presents the quantitative scores from the MonoPerfCap dataset, showcasing the perfor- mance of both the onlyGNN and onlySyn models. Similar the result differences shown in Tab, (b), our full model consistently outperforms these two ablated models, which reveals our full model's adaptability to diverse in-the-will scenarios depicted in high-resolution images. With all these sults, we conclude that each component clearly contribute to the empirical success of the full model. More ablation studies the pose-dependent frequency modulation can be found in the appendix.
CONCLUSION
We introduce a novel, frequency-based framework based on NeRF (Mildenhall et al. 2020) that enables the accurate learning of human body representations from videos. The main contribution of our approach is the explicit integration of desired frequency modeling with pose context. When compared state-of-the-art algorithms, our method demonstrates improved synthesis quality and enhanced generalization capabilities, particularly when faced with unseen poses and camera views.
Acknowledgements This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant and by Advanced Research Computing at the Univer- sity of British Columbia. It was also partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation, Sweden. We thank Shih-Y Yang Su for his help and discussions. We also thank ARC UBC and Compute Canada for providing computational resources.

---

Published as a conference paper at ICLR 2024
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (relu). CoRR, 2018.
Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih. Driving-signal aware full-body avatars. ACM TOG, 2021.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In CVPR, 2023,
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
Liang- Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. 2017.
Xu Chen, Yufeng Zheng, Michael Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differen- tiable forward skinning for animating non-rigid neural implicit shapes. In ICCV, 2021.
Boyang Deng, John P Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Nasa neural articulated shape approximation. In ECCV, 2020.
Junting Dong, Qi Fang, Yudong Guo, Sida Peng, Qing Shuai, Xiaowei Zhou, and Hujun Bao. To- talselfscan: Learning full-body avatars from self-portrait videos of faces, hands, and bodies. In NeurlPS, 2022.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In ICCV, 2021.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In ICLR, 2021.
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021.
Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin. Instance-level human parsing via part grouping network. In ECCV, 2018.
Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges. Vid2avatar: 3d avatar recon- struction from videos in the wild via supe-vupe scene decomposition. In CVPR, 2023.
Amir Hertz, Or Perel, Raja Giryes, Olga workine-Hornung and Daniel Cohen-Or. Sape: Spatially- adaptive progressive encoding for neural optimization. NeurIPS, 2021.
Hao Hu, Dongsheng Xiao, Helge Rhodin, and Timothy H Murphy. Towards a visualizable, de- identified synthetic biomarker of human movement disorders. Journal of Parkinson's Disease, 2022.
Catalin Ionescu, Fuxin Li, and Cristian Sminchisescu. Latent structured models for human pose estimation. In ICCV, 2011.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing natural environments. IEEE TPAMI, 2013.
Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In ECCV, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.
Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to recon- struct 3d human pose and shape via model-fitting in the loop. In ICCV, 2019.
Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learn- ing generalizable radiance fields for human performance rendering. NeurlPS, 2021.
10

---

Published as a conference paper at ICLR 2024
Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music condi- tioned 3d dance generation with aist++. In ICCV, 2021a.
Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free animatable volumetric actors. In ECCV, 2022a.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In CVPR, 2022b.
Zhe Li, Zerong Zheng, Yuxiao Liu, Boyao Zhou, and Yebin Liu. Posevocab: Learning joint- structured pose embeddings for human avatar modeling, In SIGGRAPH, 2023.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021b.
David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. BACON: Band-limited Coordinate Networks for Multiscale Scene Representation. CVPR, 2022.
Lingjie Liu, Mare Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM TOG, 2021.
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM TOG, 2021.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM TOG, 2015.
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated Periodic Activations for Generalizable Local Functional Representations. In ICCV, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. 2021.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Gold- man, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representa- tion for topologically varying neural radiance fields. ACM TOG, 2021b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. 2019.
Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. ICCV, 2021a.
Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021b.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
Shunsuke Saito, Jinlong Yang, Qianli Ma. and Michael Black. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In CVPR, 2021.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. NeurIPS, 2020.

---

Published as a conference paper at ICLR 2024
Chunjin Song, Zhijie Wu, Yang Zhou, Minglun Gong, and Hui Huang. Etnet: Error transition network for arbitrary style transfer. In NeurlPS, 2019.
Shih-Yam Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. NeurIPS, 2021.
Shih-Yang Su, Timur Bagautdinov, and Helge Rhodin. Danbo: Disentangled articulated neural body representations via graph neural networks. In ECCV, 2022.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS, 2020.
Justus Thies, Michael Zollhöfer, and Matthias NieBner. Deferred neural rendering: Image synthesis using neural textures. ACM TOG, 2019.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In ICCV, 2021.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurlPS, 2021.
Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang. Arah: Animatable volume rendering of articulated human sdfs. In ECCV, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004.
Chung-Yi Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan Barron, and Ira Kemelmacher- Shlizerman. HumanNeRF: Free- viewpoitt Rendering of Moving People from Monocular Video. In CVPR, 2022.
Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Sagnet: Structure-aware generative network for 3d-shape modeling. ACM TOG, 2019.
Zhijie Wu, Chunjin Song. Yang Zhou, Minglun Gong, and Hui Huang. Efanet: Exchangeable feature alignment network for arbitrary style transfer. In AAAI, 2020.
Zhijie Wu, Chunjin Song, Guanxiong Chen, Sheng Guo, and Weilin Huang. Completeness and coherence learning for fast arbitrary style transfer. Transactions on Machine Learning Research, 2022.
Zhijie Wu, Yuhe Jin, and Kwang Moo Yi. Neural fourier filter bank. In CVPR, 2023.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural Fields in Visual Com- puting and Beyond. Computer Graphics Forum, 2022.
Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, and Christian Theobalt. Monoperfcap: Human performance capture from monocular video. ACM TOG, 2018.
Zhengming Yu, Wei Cheng, Xian Liu, Wayne Wu, and Kwan- Yee Lin. Monohuman: Animatable human neural field from monocular video. In CVPR, 2023.
Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In CVPR, 2021.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018.
Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, and Yebin Liu. Structured local radiance fields for human avatar modeling. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2024
Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu. Avatarrex: Real-time expressive full-body avatars. ACM TOG, 2023.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks, In CVPR, 2019.
13

---

Published as a conference paper at ICLR 2024
Appendices
In this part, we first present the details about method implementation, used dataset and data split. Then provide more comparison results on the frequency histograms, geometry visualization and motion retargeting. We also attach more ablation studies to emphasize the significance of pose- guided frequency modulation and window functions. More qualitative results for novel view syn- thesis and novel pose rendering are provided as well. Finally, we discuss the limitations and social impacts of this project. See the attached video for the animation and geometry visualization results.
A IMPLEMENTATION DETAILS
For consistency, we maintain the same hyper-parameter settings across various testing experiments, including the loss function with weight A. the number of training iterations, and the network capac- ity and learning rate. All the hyper-parametery are chosen depending on the final accuracy on chosen benchmarks. Our method is implemented using PyTorch (Paszke et al. 2019). We utilize the Adam optimizer (Kingma & Bal 2014) with default parameters 31 0.9 and B2 0.99, We employ the step decay schedule to adjust the learning rate, where the initial learning rate is set to 5 x 10 and we drop the learning rate to 10% every 500000 iterations. Like former methods (Su et al. 2021 2022), we set A 0.001 and NB 24 to accurately capture the topology variations and avoid introducing unnecessary training changes. The learnable parameters GNN, window function and the frequency modulation part activated by the Sine function while other parameters in the neural field F are activated by Relu (Agarap 2018). We train our network on a single NVidia RTX 3090 GPU for about 20 hours.
B MORE DETAILS ABOUT DATASETS
Follow the experimental settings of previous methods (Su et al. 2021 2022), we choose the Hu- man3.6M (Ionescu et al.) 2011) and MonoPeríCap (Xu et al. 2018) as the evaluation benchmarks. These datasets cover the indoor and outdoor scenes captured by monocular and multi-view videos. Specifically, we use a total of subjects for evaluation under the same evaluation protocol as in An- imNeRF (Peng et al.) 2021a). We compute the foreground images with (Gong et al. 2018) to focus on the target characters. Likewise, we adopt the identical pair of sequences and configuration as em- ployed in A-NeRF (Su al. (022]:: Weipeng and Nadia, consisting of 1151 and 1635 images each, with resolution of 1080 x 1920. We estimate the human and camera poses using SPIN (Kolotouros) al. 2019 and following pose refinement (Su al.) 2021). We apply the released DeepLabv3 model (Chen et al. 2017) to estimate the foreground masks. The data split also stays the same as the aforementioned methods for a fair comparison.
C MORE RESULTS
Histogram Comparisons. Showing the frequency histograms of different frames, like the Fig. D in the main text, appear to be a clear solution to demonstrate our motivation. Thus we provide more examples and corresponding analysis here. Using the two close-ups in Fig.
T of the main text, we present the corresponding frequency histograms for each method in Fig, 10 To test our effectiveness when training over long sequences, we provide the histogram results of two frames collected in the ZJU-MoCap (Peng et al.) (202Tb) dataset, in Fig. [I Compared to DANBO, our method can produce more similar contours to the ground truth histograms. Additionally, the matched histogram distances (shown below the histogram subfigures and denoted as F-Dist) further confirm our advantages of producing adaptive frequency distributions which is the key point of our method.
Besides measuring the holistic histogram similarities, we directly compute frequency map by regarding the standard deviation (STD) value at each pixel as gcay-scale value. As evidenced in Fig. II our method provides significantly improved results, represented by the error images between the output frequencies and the ground truth values. All these results reveal that our method can faithfully reconstruct the desired frequency distributions both locally and holistically.
14

---

Published as a conference paper at ICLR 2024
GT
DANBO
Ours
GT
DANBO
Ours
Figure 10: Motivation Demonstration on Human3.6M frames. Using the two frames in Fig. of the main text, we present the frequency histograms, compute three image quality metrics (e.g. PSNR, SSIM, LPIPS) and the distances between the output frequency map and ground truth values (F-Dist) to justify pose-guided frequency modulation. Compared to DANBO which modulates frequencies implicitly, our method can synthesize higher-quality images with more adaptive fre- quency distributions across different pose contexts.
GT
DANBO
Ours
GT
DANBO
Ours
Sanguan
2802 2011
Figure 11: Motivation Demonstration on ZJU-Mocap frames. To test our effectiveness over long video sequences, qualitatively and quantitatively evaluate our method with two ZJU-Mocap frames. Being consistent with the findings in Fig. I0 our method outperforms DANBO with more similar frequency histograms (3rd row) and better quantitative metrics on image quality and fre- quency modeling (last row). Additionally, we further illustrate the color-coded frequency eтror maps (2nd row) for both methods show that our method can faithfully reconstruct the desired frequency distributions both locally and holistically. For the left frame with smooth patterns, our method in- troduces slightly few frequency errors as easy to model www-ureyynccy variations. On the other hand, for the right frame with much more high-frequency wrinkles, the desired frequencies with significantly less errors, which clearly demonstrate the importance of pose-guided frequency modulation. Here red denotes positive and blue denotes negative errors.
method faithfully reproduces
15

---

Published as a conference paper at ICLR 2024
Only Syn
Our
Only
Our
100
Figure 12: Abaltion study the network components with frequency analysis. Our full model produces more adaptive frequency distributions and higher image quality than the ablated models.
0.039
0.045
0.028
0.401
0.028
0.928
GT
Ours
Humannerf
Menohuman
VidZAustar
Vid2/vatar-mask
Figure 13: Visual comparisons on ZJU-Mocap with modern baselines. Among all methods, only our model can successfully synthesize the avatar fist with accurate arm contours (top row) and faithfully reconstruct the desired wrinkles (bottom row). In contrast, all baselines either distort or blur the fist patterns (top row) and clothing textures (bottom row). Additionally, the state-of-the-a methods fail to capture the cloth colors. Similar to Fig, II illustrate the frequency error maps and compute the distances between the output frequency map and ground truth values (F-Dist). Being consistent with the findings in Fig. [00] and Fig. [1] the pose-guided frequency modulation can better reconstruct the desired frequency distributions both locally and holistically. Thus more adaptive patterns can be reproduced with reduced errors.
16

---

Published as a conference paper at ICLR 2024
Table Novel-view synthesis results on the ZJU-Mocap Peng et (2021b) test set. The pose- dependent frequency modulation facilitates generalizing to novel views with one single camera in- put. Here 'Vid2Avatar' indicates the original Vid2Avatar implementation while Vii2Avvttar-mask is the model only focusing the foreground human representations, thus obtaining better metrics than the original version.
S377
S387
S393
S394
Avg
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
NeuralBody
28.90
0.967
27.15 0.953
28.30 0.957
28.41 0.956
28.19 0.958
HumanNeRR
30.21 0.975
27.95 0.963
28.38 0.961
30.00 0.963
29.14 0.965
MonoHuman
30.23
0.976
28.20 0.962
28.40
0.962
30.02
0.963
29.21
0.966
Vid2Avatar
29.52
0.951
28.10
0.938
27.19
0.929
28.95
0.932
28.44
0.938
Avdztrrttt+mask
29.53
0.976
28.15
0.965
27.95
0.962
29.83
0.965
28.87
0.967
Ours
30.72
0.979 28.20
0.965
28.91 0.964 30.36
0.965
29.55 0.968
Table Novel pose rendering results on the ZJU-Mocap Peng et al. (2021b) test set. Similar to our method improves the quantitative results for novel pose rendering.
Tab.
S377
S387
S393
S394
Avg
PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR
SSIM
NeuralBody
29.18 0.969 26.43
0.951
28.15
0.956
28.14 0.956
27.98
0.958
HumanNeRF
30.18 0.975 27.60 0.962
28.39 0.961
28.78
0.960
28.74
0.965
MonoHuman
30.26 0.977 27.66
0.961
28.78 0.962
29.11 0.961
28.95
0.965
Vid2Avatar
29.86 0.954 27.42 0.938
27.20
0.929
28.55
0.932
28.26
0.938
Vid2Avatar+mask
29.85
0.977 27.38
0.964
28.09
0.962
29.25
0.963
28.64
0.967
Ours
30.95 0.979 27.71
0.964
28.96
0.964
29.60
0.963
29.31 0.968
Template-based
Template-free
Vid2Avatar
Vid2Avatar mask
Humannerf
Monohuman
Ours
Geometry reconstruction on ZJU-Mocap S377. Compared other template-free methods (HumanNeRF and MonoHuman), our method yields more precise and less noisy body shape. Some bumpy patterns remains as template mesh or other surface prior used, in contrast to Vid2Avatar.
Figure 14:
Modern baselines on ZJU-Mocap sequences. To show our advantages with better generality, we train our method and three baselines, including HumanNeRF Weng et al. (2022). MonoHuman Yu et al. (2023), and Vid2Avatar Guo et al.] (2023), over four ZJU-Mocap sequences (S377. S387, S393 and S394) following the experimental settings of HumanNeRF and MonoHuman. All re- sults achieved through their public codes with default ayper-parameters Note that, denote 'Vid2Avatar' as the original Vid2Avatar implementation and correspondingly 'Vid2Avatar-mask" the model only focusing the foreground human representations with better empirical results.
17

---

Published as a conference paper at ICLR 2024
Table 5: Geoemtry Reconstruction on the ZJU-Mocap Peng al. (0211) test The pose- dependent frequency modulation improves lemplate-free baselines with more faithful reconstruction and achieves similar results to template-basee method.
S377
S387
S393
S394
Avg
Template/Scan-assel prior
Vid2Avatar Vid2Avatar+mask
0.1180
0.4696
0.3330
0.2480
0.2921
0.0336
0.0283
0.0404
0.0410
0.0359
Template-free
HumanNeRF
0.0401
0.0777
0.0609
0.0543
0.0582
MonoHuman
0.0897
0.0801
0.0836
0.1006
0.0885
Ours
0.0337
0.0365
0.0455
0.0441
0.0400
Table Ablation studies on full Human3. 6M sequences. Our full model improves over ablated baselines, showing the necessity of all components.
onlyGNN noGNN onlySyn only w.
only w
no window Ours (full)
Novel view
PSNR
24.72
24.90
24.29
24.99
23.63
23.32
25.06
SSIM
0.914
0.918
0.911
0.920
0.835
0.842
0.921
LPIPS
0.125
0.112
0.124
0.110
0.272
0.305
0.110
Novel pose PSNR
23.72
23.64
23.74
24.00
22.97
22.42
24.15
SSIMT
0.898
0.900
0.900
0.905
0.824
0.832
0.906
LPIPS
0.140
0.129
0.138
0.125
0.283
0.310
0.124
(b)
m
X
Figure 15: Architecture of the ablated noGNN model. Compared full model, achieve noGNN by removing graph neural networks and directly feeding the joint parameters as a whole to single Multi-Layer Perceptron (MLP) for frequency modulation.
In Fig. we illustrate the visual comparisons for our method and chosen baselines. Both Human NeRF and MonoHuman blur the highlighted avatar fist while we can faithfully reproduce the realistic patterns on the top row. Additionally, on the bottom, the pose-based frequency modulation enables smoother body contours and synthesizes more adaptive wrinkles (e.g. the wrinkle directions) while the baselines all fail to produce accurate cloth colors. The provided error images also underline that our method can more successfully simulate the ground truth images. Similar to Fig. [I] compute the matched histogram distances (F-Dist) for the close-ups. The lowest frequency distance demon- strates that method can best generate adaptive textures with appropriate frequency distributions. Thus better dynamic wrinkles are enabled. Tab. 3 lists the metric comparisons for novel view synthesis and novel pose rendering, further championing former observations.
18

---

Published as a conference paper at ICLR 2024
GT
onlyGNN
onlySyn
Ours
Figure 16: Ablation study sub-branch networks with novel pose rendering.
GT
window
only w""
only w/
Ours
Figure 17: Ablation study on window functions with novel view synthesis.
Like ARAH Wang et al. (2022). we follow its settings and offer the geometry comparison results. Specifically, also apply NeuS Wang and compute quantitative metrics with L2 Chamfer Distance (CD). We show qualitative results in Fig. 14and quantitative results in Tab. 5 Our method possesses better geometry reconstruction with smoother body surfaces than the template-free methods (HumanNeRF and MonoHuman) which introduce obvious floating artifacts empty space. The geometry scores are slightly worse than Vid2Avatar which is template based baseline and relies on SMPL parameters as prior. The SDF-based surface rendering also helps Vid2Avatar achieve smoother shapes.
[2021) to obtain the pseudo geometry ground truth
Complete Metries on Human3.6N, sequences. Besides the overall average numbers in the Tab. of the main text, we also report a per-subject breakdown of the quantitative metrics against baseline methods. Specifically, Tab. 7 lists the scores for the novel view synthesis while Tab, 8 details each method's results in novel pose rendering. Being consistent with the visual results shown Fig. and of the main text, our method almost outperforms all baselines for all subjects.
GT
A-NeRF
DANBO
ARAH
TAVA
Ours
Figure 18: Failure case. How to generalize to challenging cases still an open problem, where all methods fail capture the stripe-wise textures under this pose.
19

---

Published as a conference paper at ICLR 2024
DANBO
Ours
Additional comparison results with DANBO for novel view synthesis. Due the adaptive frequency modulation, our method can better synthesize the shape contour (e.g. hands on 1st column), the sharp patterns (e.g. the marker on 2nd column), and high-frequency details (e.g. the wrinkles on 1s column and the pant textures on 3rd column). Otherwise, DANBO which achieves frequency learning implicitly, blurs the sharp patterns and smoothes the fine-grained details.
Figure 19:
DANBO
Ours
Figure 20: Additional comparison results with DANBO for novel pose rendering. Due to the adaptive frequency modulation, our method successfully reduces the noisy artifacts on 1s column, reproduces the white markers on 2nd column and the black marker on 3rd column, reconstructs the sharp shape contours (e.g. the hand) on both 4th and 5th columns. Otherwise, DANBO which achieves frequency learning implicitly. blurs the sharp contours and smoothes the significant patterns with fine-grained details.
20

---

Published as a conference paper at ICLR 2024
Only Syn
Ours
Figure 21: Abaltion study on the network components with time consistency analysis. Our full model can consistently produce more adaptive details (e.g. in the pant region), synthesize more structured textures (e.g. the stripes) and preserve more realistic contours (e.g. the leg shape). See texts for details.
Figure 22: Unseen pose renderings for the sequences from both Human3.66 3.M (top two rows) and MonoPerfCap (bottom two rows). Our network is robust to various poses across different datasets.
21

---

Published as a conference paper at ICLR 2024
A-NeRF
DANBO
ARAH
TAVA
Ours
Figure 23: Visual comparisons for novel view synthesis (1 row) and novel pose rendering (2nd row), Compared to baselines, we can faithfully reconstruct the shape boundaries (e.g. the hands on both ee and rows) and the high-frequency details (e.g. the dynamic wrinkles on nnd row).
22

---

Published as a conference paper at ICLR 2024
Table 7: Novel-view synthesis results on the Human3.6M Lonescu et al. (2011) test set. Our method benefits from the explicit frequency modulations, leading to better perceptual quality. It matches or outperforms all baselines across subjects, reaching the best overall score in all three metrics.
58
HSNMISSIMTLPSSS PSNR SSIM LIMPS
PINR SSIM LPWS
PENR SSIM LPIPS
PSNRISSIMI LIPISS
PSNK SSIM LIMES
PSNK
SSIM LPMPS
PSNR SSIM
LPUPS
TrmgtateeScan--ee prioe Neaniillats
ss 807
2461 0.917
22.53
0.155
0113 71.72 0.84 0.151 24.25 8911
23.00
0.96 166
13.16
6.160
Tenglate-fre AINERE
23
1191 0.118
2467 0.919 01114
25.7H
0.887
0.147 24.40
0111 0.125 22.70 0.907
0.130
0.916 0.126
38
0.916
11153
24.26
0.91 0.329
DANBO
01129
01EES 0.917
01016
25:15
0.925
0.108
0911
24.59
0.917
01166
25.25
Durs
24.53 11922 0.102 24.77 8,925
ILI02
24.55
0,985 0,124 24,55 1L923 0.187 24.01 0.921 0.108 26.39
IL529
10,109
25
20
00921
1.128
25.06
0,921 6,110
we ovalante tiing for ulficsaly nitsued ARAIL whnch Iso underganz relattecsstion rosling to slghtly dillcceii number the one4 Wary cl 2002
Table 8: Novel pose rendering results on the Human3.6M Ionescu et al. (2011) test set. Our pose guided frequency modulation pipeline generalizes better across unseen poses.
58
AVg
PSNRISSINTLPPRS LPPPS PSNR
SSIM LIMES
PENR
SSIM LPIPS
PENK SSIM LIIPS
PSNRISSINFLPIR
PSNK
SSIM LHPS
PSNK SSIM LIPSS
PSNR
SSIM LPIPS
Template/Scon-bbeee priue
Nearilliats
22.10
01875
0.165
70:94
0.816 0.172 23085
0.88
0.190
0,964
0.179
12.81
0.838
ARAAI
Tonglatneiee
A-NGR
22.77
(1881
0,888 0151
6.700
01182
0.846 0.170 24.16
0,880
13.00
0.841
DANBO
0.124
01130 22.50
28:74
23,33
Durs
2375 11901 0,114
73,55 8,905 0.17
26.77
8:988
6,117 25,99 I1M 0,135 23,16 03989 8:136 25.13
IUWN 11,13
25.0 00007 16343
36.15
0,986 6.124
we evalaate tiing laz olficialy relcased ARAH, wlnch Iso undergonz relaatreastion, coslinn 1s sighii diffoccal number the oscs in WWayg ct.al. 2022
Visual Comparisons with Baselines. Pose-modulated frequency learning plays a critical role in method. To demonstrate the importance of this concept, we present more comparisons with DANBO which performs frequency modeling implicitly. In Fig. 19 and Fig. 20 our method is better at preserving large-scale shape contours well fine-grained textures with high-frequency details. Besides the results in Fig. [ and o of the main text, we offer two more characters from Human3.6M 3.6 sequences to evaluate the results on novel view synthesis and novel pose rendering. As shown in Fig. 233 we can successfully reproduce the detailed shape structures (e.g. the hand on 1st row) and high-frequency wrinkles (e.g. 2nd row). These findings stay consistent with the discussions in main text and the quantitative results in Tab.
Ablation studies. We formulate our framework from connections between frequencies and pose contexts. To more comprehensively evaluate the effectiveness of our pose-guided frequency modu- lation concept, we provide one more visual comparison in Fig. [6] It clear that only our full model successfully synthesizes high frequency patterns, c.g., shown in the pant region.
To additionally showcase the capabilities in reproducing the frequency distributions, we illustrate the frequency histograms for the ground truth images and the network outputs in Fig. 12 Our full model remarkably reduces the gap to the ground truth histograms qualitatively and quantitatively.
Moreover, as shown in Fig. 21 our full model presents much better time consistency than ablated models. Specifically, the full model constantly preserves more adaptive details (c.g. the patterns in the pant region) than the onlyGN. model and synthesizes more structured stripe-wise patterns than the onlySyn model. Moreover, the onlySyn model distorts the leg shape on the last column.
To highlight the empirical importance of the window function wp and w.f, Fig. 17 depicts qualitative differences between the ablated baselines and our full model. It is clear that using w/' or w/ alone cannot produce the image quality of the full model, demonstrating the necessity of the window function design. Tab. 6 additionally presents the quantitative comparisons for all ablation models, further supporting the aforementioned discussions.
Geometry Visualization. The attached video visualizes two examples for the geometry reconstruc- tion comparison with DANBO. Like the discussions in the main text, we can present overall more complete body outline and smoother surface than the baseline. Please see video for details.
Motion Retargeting, Generality to unseen human poses is critical to a number of down-streamed applications, e.g. Virtual Reality. We provide two examples in the attached video. Although our
23

---

Published as a conference paper at ICLR 2024
Figure 24: Additional visual results for novel view synthesis. is clear that our method can faithfully reproduce a larger spectrum of details, from large-scale shape contours (e.g. 1s row) to fine-grained textures (e.g. 2nd row), across different scenes.
Figure 25: Additional visual results for novel pose rendering. clear that our method can gener- alize well to the unseen poses with different patterns.
model is trained on the Human3.66 3.6M sequences, consistently be adapted to the unseen poses with challenging movements. The desired time consistency convincingly demonstrates our strong generality th-ditrtthedisttiiuttinnn poses.
24

---

Published as a conference paper at ICLR 2024
More Visual Results. In order assess the performance of method in handling unseen camera views and human skeletons, we provide additional results showcasing novel view synthesis Fig. 24 and novel pose rendering in Fig. 25 We also illustrate the rendering results for different sequences of Human3.66 6M and MonoPertCap datasets for unseen poses in Fig. 22
From the results, it is evident that method, with its developed frequency modulation modeling. effectively captures diverse texture details and shape contours even when confronted with human poses that differ significantly from those in the training set. This empirical advantage be tributed to the adaptive detail modeling capabilities facilitated by pose-modulated frequency learn- ing strategy.
D LIMITATIONS AND DISCUSSIONS
Although our method is faster than other neural field approaches, computation time remains a con- straint for real-time use. Our method is also person-specific, demanding individual training for each person. And our method heavily relies on accurate camera parameters and lacks support for property editing like pattern transfer. Thus this approach shines with ample training time and available data. Additionally, shown in Fig. [8] under extreme challenges, our method cannot vividly reproduce the desired patterns but introduces blurry artifacts. However, we would like note that, how to advance the generalization to such cases still open since existing methods suffer from similar worse artefacts well.
Social Impacts. Our research holds the promise of greatly improving the efficiency of human avatar modeling pipelines. promoting inclusivity for underrepresented individuals and activities in supervised datasets. However, it's imperative to address the ethical aspects and potential risks of creating 3D models without consent. Users must rely datasets specifically collected for motion capture algorithm development, respecting proper consent and ethical considerations. Furthermore, in the final version, all identifiable faces will be blurred for anonymity.
25


Reviews:
[{"review_id": "dy4J7edLrm", "reviewer": "Reviewer_McVe", "summary": "This paper introduces a method for pose-modulated animatable human avatar creation using Nerfs. The key idea that the paper introduces is to modulate the frequencies of the sinusoidal embeddings used to encode input 3D locations, before they are input into the Nerf's MLP for inference, depending on their local position and nearest bones' pose. These are key to modeling fine-grained texture details of folds caused by bone deformation on clothing, etc. To achieve this the authors propose a module to that first encodes the poses of joints via a graph neural network structure and then predict the frequency modulation of a 3D location based on its local bone's encoded GNN features and position in 2D space relative to the skeletal structure. The authors compare the proposed method to several existing competing methods, both qualitatively and quantitatively, and observe superior performance for their method.", "questions": "I would like to see the authors' response to the questions I have posed in the \"weaknesses\" section of my review. \n\nOverall, I feel that while the idea of pose-conditioned frequency modulation to model wrinkles on clothing is interesting and worth sharing with the wider research community, the rendering quality of the proposed method is below that of the state-of-the-art methods. It would have been ideal if the authors had built their method on of the more recent high-quality nerf-based human avatar methods to achieve both overall high-quality and improvements in modeling of surface wrinkles.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "In terms of novelty, the problem of animatable neural human avatar creation is a widely studied one. However this paper proposes the original new idea of modulating the frequency bands used in Nerfs to correctly learn to model wrinkles on clothes based on the deforming pose. This idea is conceptually sound and provides an interesting novel insight to the problem of human avatar creation. Modeling the deformation of loose clothing is still a fairly unsolved problem within this domain and hence advances the field forward.\n\nThe proposed solution and experimental methodology are technically sound. The paper will well-written and structured. Many details are described in the supplement. The authors have promised to released the code.", "weaknesses": "The main weaknesses are in terms of the results and experiments. \n\n1. Overall the results in the supplementary videos are quite blurry. The effect of the improvement in texture quality of the wrinkles with the proposed method are also subtle and hard to really appreciate. The numerical results in Table 2 of the paper correlate with this fact and show marginal numerical improvement in the reported metrics. Do the authors believe these numerical improvements are statistically significant?\n\n2. For the novel view synthesis task, I am curious as to why the authors did not compared against the following several more recent state-of-the-art methods, which result in higher rendering quality.\n\na) Guo et al., Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition, CVPR 2023.\n\nb) Jiang et al.,  Neuman: Neural human radiance field from a single video, ECCV 2022.\n\nc) Weng et al., Humannerf: Free-viewpoint rendering of moving people from monocular video, CVPR 2022.\n\nd) Yu et al., MonoHuman: Animatable Human Neural Field from Monocular Video, CVPR 2023.\n\n3. Related also to question 2, is why did the authors choose to not report quantitative metrics for geomtric reconstruction quality and compare it to the existing state-of-the-art methods listed in question 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z6eeJOgHTM", "reviewer": "Reviewer_SAAP", "summary": "The authors tackle the problem of human avatar modeling from a monocular video sequence. The paper proposes a pose-driven frequency modulation approach for the underlying NeRF model to achieve a higher accuracy of the rendered images. This approach is shown to be effective for modeling people with tight closing compared to some of the baseline methods.", "questions": "- Please address my concerns in the weaknesses section", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problem of avatar modeling has high practical significance\n- The frequency modulation approach makes sense to introduce the details in cases where they are required, which can serve as a regularization measure\n- The paper is fairly well written", "weaknesses": "- The comparison lacks modern baselines, such as Vid2Avatar, MonoHuman, and HumanNeRF, which were referenced in the related work.\n- Video results have very low FPS, and therefore, the temporal smoothness of the proposed approach cannot be evaluated.\n- It is unclear whether or not GNNs are actually needed for this task, ex. Vid2Avatar uses pose conditioning without GNNs to directly produce the embeddings via an MLP\n- No experiments on loose clothing where the method's effectiveness for high-frequency clothing modeling can be asserted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XucHTCHPrb", "reviewer": "Reviewer_PkgA", "summary": "The paper presents a framework that learns an animatable human body model from video without relying on surface information. The approach builds on previous research by incorporating frequency modeling with pose context. The experiments conducted show that the proposed method is capable of generating higher-quality images and has improved generalization abilities when dealing with new poses and viewpoints.", "questions": "Please see the weakness section for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written. Technical details are also well-elaborated.\n* Based on the evaluation, the overall quality of the results seems to be satisfactory. Additionally, the quantitative results show better performance compared to DANBO.", "weaknesses": "Motivation:\n* I find the motivation in Figure 1 to be unclear. The two poses are quite different - the first contains wrinkles while the second doesn't - but their frequency distribution appears quite similar. Perhaps it would be better to choose a sample with poses that are closer to each other and have more significant differences in frequency.\n* The paper mentioned that even when a subject is in a similar pose, the frequency distributions can still be distinct. This seems contradictory to the motivation of pose-dependent frequency modulation, as one might wonder why pose-dependent frequency modulation would be beneficial.\n\nEvaluations:\n* Are the point locations clearly separated by the learned weights for each part, or are the parts mixed around the joints? It would be best to visualize the weights for the areas where the bones overlap.\n* The novel pose results on MonoPerfCap do not match the numbers reported in DANBO. Is there a setup difference?\n* Table 1 (b) caption seems to be incorrect. Is the ablation study tested on Human3.6M S9, or is it on MonoPerfCap? \n* The ablation should be performed on the complete setup to allow for better comparison with DANBO. Additionally, the current ablation study is unable to determine which specific design element is responsible for the model's improved performance compared to DANBO.\n* Is there a specific reason why NeuMan is not being compared with Template/Scan-based prior methods and HumanNeRF for the template-free method in Table 2? A discussion or quantitative comparison would be better.\n* It would be more convincing to evaluate the performance of the model on the entire ZJU-Mocap dataset rather than a few selected frames.\n\nMinor:\n* The code link seems to be missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)