Locality Sensitive Avatars From Video

Authors: Chunjin Song, Zhijie Wu, Shih-Yang Su, Bastian Wandt, Leonid Sigal, Helge Rhodin
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Computer Vision, Neural Rendering, Avatar Modeling

Abstract:
We present locality-sensitive avatar, a neural radiance field (NeRF) based network to learn human motions from monocular videos. To this end, we estimate a canonical representation between different frames of a video with a non-linear mapping from observation to canonical space, which we decompose into a skeletal rigid motion and a non-rigid counterpart. Our key contribution is to retain fine-grained details by modeling the non-rigid part with a graph neural network (GNN) that keeps the pose information local to neighboring body parts. Compared to former canonical representation based methods which solely operate on the coordinate space of a whole shape, our locality-sensitive motion modeling can reproduce both realistic shape contours and vivid fine-grained details. We evaluate on ZJU-MoCap, SynWild, ActorsHQ, MVHumanNet and various outdoor videos. The experiments reveal that with the locality sensitive deformation to canonical feature space, we are the first to achieve state-of-the-art results across novel view synthesis, novel pose animation and 3D shape reconstruction simultaneously. Our code is available at https://github.com/ChunjinSong/lsavatar.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730326536405

thread_type:
official_review

<thread_object>
creator: reviewer_cjiu

<content>
summary:
This paper studies the problem of learning 3D neural avatars from 2D videos. It is based on several prior works that decompose the dynamic neural human avatar into canonical space, rigid deformation, and non-rigid deformation. The major contribution lies in the modeling of the non-rigid deformation part. Specifically, it trains a pose-conditioned non-rigid deformation prediction network with a GNN architecture. Some experiments show improvement of the proposed method.

strengths:
- The studied problem in this paper is important. Reconstructing neural human avatars from videos has been a long-standing problem in the community, with several different applications in the movie industry, AR/VR, and gaming industry.
- The proposed method basically follows the framework set by the prior works, which is sound. Modeling the non-rigid deformation with a pose-conditioned network is also a sound and widely used method in the community.

weaknesses:
- The major concern is the generalizability of the pose-conditioned GNN. The multi-view human data is sparse in the community, therefore the GNN might overfit on those training and struggle to generalize to the in-the-wild dataset.
- Most results presented are with tight clothes, which might not fully demonstrate the effectiveness of the proposed method. It would be interesting to see more large-scale results on more challenging clothes, such as THUman2.1 dataset.
- The major contribution of this paper can be seen as incremental. The only major change of this paper's pipeline compared to prior works is to use GNN to model non-rigid deformation, which is a sound but not-so-exciting technical contribution.
- Some baselines are not compared, e.g. [Li et al. 2023], so it's hard to say whether the proposed method is truly the state-of-the-art.
- Some experiment settings are not clear. See questions for details.

[Li et al. 2023] PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling.

questions:
- For the experiments on the ActorsHQ dataset, is it rendered under a novel pose or training pose?
- For the videos rendered in the supplementary material, is it under novel pose or training pose?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730568661901

thread_type:
official_review

<thread_object>
creator: reviewer_nejy

<content>
summary:
This paper introduces Locality-Sensitive Avatar, a novel neural radiance field (NeRF) based network for learning human motions from monocular videos. The key innovation lies in using a graph neural network (GNN) to model non-rigid deformations while preserving fine-grained, locality-sensitive details.

strengths:
The model demonstrates superior rendering performance compared to other approaches, particularly evident through enhanced normal rendering results. It is also capable of learning from a small amount of data and remains effective even for novel poses, showcasing its robustness and versatility.

weaknesses:
The model struggles with accurate reconstruction of detailed areas such as the face or hands. Additionally, the visualization of non-rigid motion is insufficient, making it unclear what this component represents, and it is difficult to clearly understand the difference between considering and not considering non-rigid motion.

The core idea is to utilize non-rigid motion, which could effectively express the fluttering of loose-fit clothes. The current examples capture only person-specific details.

questions:
1. What are the differences between the proposed model and GauHuman or HUGS?
@inproceedings{hu2024gauhuman,
  title={Gauhuman: Articulated gaussian splatting from monocular human videos},
  author={Hu, Shoukang and Hu, Tao and Liu, Ziwei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={20418--20431},
  year={2024}
}

@inproceedings{kocabas2024hugs,
  title={Hugs: Human gaussian splats},
  author={Kocabas, Muhammed and Chang, Jen-Hao Rick and Gabriel, James and Tuzel, Oncel and Ranjan, Anurag},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={505--515},
  year={2024}
}

2. Does using SMPL pose parameters as LBS weights result in any artifacts?

3. What is the rendering speed of the proposed model?

4. Can you show the qualitative and quantitative results if the non-rigid Δx component is omitted?

5. Does considering non-rigid deformations help in accurately representing effects like the flapping of clothes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703768051

thread_type:
official_review

<thread_object>
creator: reviewer_p8gg

<content>
summary:
This work presents a neural radiance field (NeRF) based network to learn human motions from monocular videos. The key argument is its method that can improve the local details of humans. They evaluated on ZJU-MoCap, ActorsHQ, SynWild, and various outdoor videos.

strengths:
1. Human modeling from monocular video is an important task.
2. The method is clearly presented.
3. The code will be released.

weaknesses:
1. Visual Quality. (1) In the demo video,  the gap between this work and SOTAs can not be clearly seen, especially for HumanNeRF. Samples at least on HumanRF/Synwild are also expected in the demo to better evaluate the universality of the work.  In the paper, the visual compassion is done on cases with simple textures and topologies. Other publicly available in-door datasets (e.g., AIST++, DNA-Rendering, MVHumanNet) with rich texture, and diverse/complex clothes should be evaluated. Also, the reconstruction difference with Vid2Avatar is hard to differentiate, which is also demonstrated in Table G, page 19.  (2)Whats the setting of the demo video is also not clear. Are they nvs, np, or just seen view reconstruction? The setting should be clarified. 

2. Comparison to SOTAs. (1) Many state-of-the-art methods are missing in the comparison or discussion, such as MonoHuman in Tab.3 and the demo video, gaussian splatting based methods GaussianAvatar, 3DGS-Avatar (CVPR2024). (2) For the animation ability, it would be better to show driven poses that are out-of-distribution, like the MDM setting in MonoHuman. 

3. Motivation of the method. The pipeline looks like a combination of different modules.  As the key of this work, details are improved by GNN. However, the motivation is not clear. What are the challenges of other feature extraction paradigms to model in detail? Why does GNN work? Why are other methods not working? If you add GNN to their implementation, will they also work?

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732266049799

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Responses to Reviewers

comment:
We thank all reviewers for the detailed feedback and the time committed. The reviewers recognize our studied task in this submission as important (Reviewer p8GG) and a long-standing problem in the community with different applications (Reviewer cJiu). Additionally, the method is clearly presented (Reviewer p8GG), novel (Reviewer nejY) and technically sound (Reviewer cJiu) with superior rendering performance (Reviewer nejY), showcasing its robustness and versatility (Reviewer nejY).

In this submission, our primary technical contribution is the integration of locality encoding into canonical space rendering, enabling simultaneous improvements in generality and adaptive detail, which is an aspect significantly overlooked by previous methods. **Empirically, we emphasize that one of our key advantages is to outperform state-of-the-art methods in novel view synthesis, novel pose animation, and shape reconstruction simultaneously for monocular human modeling. By comparing with seven SoTA methods in our original paper and two additional baselines, we make thorough evaluations on totally twenty four sequences across six datasets to ensure statistical significance.** We further justify our experimental setting in the following **Justifying Experimental Protocals** part in details.

As demonstrated in the tables within our appendix, different baselines exhibit unique advantages across various scenarios. Specifically, as shown in Figure 3, Figure 4 and Figure F, Vid2Avatar produces most reasonable 3D shapes among chosen baselines but often over-smoothes necessary fine-grained details. In parallel, other methods (e.g. HumanNeRF and 3DGS-Avatar) provide better rendering images than Vid2Avatar but yield very noisy surfaces. With numerous results reported here and in our manuscript, we conclude that our method consistently improves former algorithms in both novel image rendering and 3D shape reconstruction. Specifically, our method surpasses all the baselines on LPIPS with **over 10%** relative improvement across all scenarios and presents more faithfully geometric patterns. Our ablation study results further validate that including locality encoding and canonical space rendering in one single framework can facilitate the reproduction of both realistic shape contours and vivid fine-grained details. Questions asked by multiple reviewers are addressed below.

We hope that the provided clarifications and results, which demonstrate that the claims made in the paper hold over additional baselines and new sequences, can alleviate the remaining concerns. To ensure reproducibility, we will make the code of our method available upon acceptance.


**Clarifications on the supplementary videos (Reviewer p8GG, Reviewer cJiu)**

Thank you for pointing this out. To demonstrate generalization to unseen poses, all results are obtained under the novel pose setting, except for the MVHumanNet sequences. We use poses extracted from the AIST++ dataset to animate our avatar representation learned on Youtube sequences. For the results trained on remaining datasets, driven poses are provided by the official datasets including ZJU-Mocap, ActorsHQ and MVHumanNet. We further detail the dataset settings in Section B and add descriptions for the demo video in the **Summary of Supplementary Video** part of Section D in the appendix.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732266351917

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Uploaded

comment:
We thank all of the reviewers and ACs for their time and effort in providing these helpful suggestions. Based on these reviews, we have revised our paper and uploaded the new version. Changes are highlighted in blue. To summarize, we have made the following changes to our paper and supplementary material:

1. Revising Table 1 and the “Modeling Articulated Avatars” part of Section 2 to summarize the difference between our method and newly added baselines, including GauHuman and GaussianAvatar.
2. Updating the manuscript to clarify the experiment comparisons on the ActorsHQ dataset (L418-L422) and discussions on out-of-distribution motions (L423-L426).
3. Adding two paragraphs in Section C to detail the advantages of using GNN to encode locality information.
4. Making a new paragraph (**Summary of Supplementary Video**) in Section D to summarize the contents of the supplementary video for better demonstration.
5. Adding one figure in Figure J to make visual comparisons across six consecutive frames from the demo video on the ZJU-Mocap dataset.
6. Adding one figure in Figure M to illustrate the non-rigid effects and validate the importance of locality encoding.
7. Adding one figure in Figure N to compare the shape meshes reconstructed from ActorsHQ dataset.
8. Adding results in Table G for quantitative comparisons and results in Figure F, Figure G, Figure H for qualitative comparisons on the MVHumanNet dataset.
9. Updating Tab. 3 (a) and adding Tab. E and Tab. F to incorporate one more sequence of ActorsHQ dataset and include more comparison baselines. The corresponding figure is shown in Figure R.
10. Adding chapters in the supplementary video to compare under the dataset setting of SynWild, ActorsHQ, MVHumanNet.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732519233185

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for the reviews

comment:
Dear reviewers,
﻿

As the discussion period is close to end, we thank you for your time and efforts on reviewing our paper with numerous constructive suggestions, from detailed expositions to comparisons with more recent methods on the latest dataset. We really appreciate them! During the author-reviewer discussion period, the authors have tried to address reviewers’ every comment with great efforts. In particular, we significantly consolidated our experiment part by adding comparison results suggested by all reviewers. With the additional experiments, we can confidently conclude the empirical superiority of our method over state-of-the-art baselines. Lastly, the authors commit to releasing the code upon acceptance.
﻿

Based on our updated manuscript, could the reviewers re-evaluate our submission? We are also open to new discussions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732664060317

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updating the "Comparison on SynWild" part of video

comment:
Dear reviewers,

To further demonstrate our advancements in producing shape geometry, we add a comparison with GoMAvatar, the 3DGS-based state-of-the-art method in shape reconstruction, alongside the NeRF-based Vid2Avatar method. Please refer to the **"Comparison on SynWild"** section of the supplementary video, where we compare the normal map sequences of our method with both approaches on the SynWild dataset. The results clearly show that our method outperforms both NeRF-based and 3DGS-based state-of-the-art methods, delivering sharper structured patterns and finer details.

We would greatly appreciate any additional feedback or insights you may have regarding the updates. We look forward to your review and continued discussions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733272230800

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you for your thorough reviews

comment:
Dear Reviewers and ACs,

First, thank you again for your time and efforts to conduct such thorough and helpful discussions. To echo your professional comments, we  were diligently replying reviewers's questions and have significantly improved our submission in aspect of providing more comprehensive evaluation results and offering further clarifications  as summarized before. With these new updates, we are more confident about our scientific conclusion that the unification of the canonical space rendering and local deformation can significantly improve baselines in novel view synthesis, novel pose animation and shape reconstruction simultaneously. While this technical upgrade is simple and straightforward, we would emphasize that it represents a significant oversight by the research community and we make the first step to unify them together which is a clear conceptual technical contribution. Also our complete experiments with notable improvements further demonstrate our empirical advantages widely. 

To reveal our effectiveness for the clothing modeling with real-world, complex textures, we provide quantitative comparisons here and will attach visual presentation in the revision. Following the setting of MonoHuman and PM-Avatar which are best baselines equipped with canonical space rendering and locality encoding respectively, we pick one sequence ("Weipeng") from MonoPerfCap dataset and two online Youtube sequences ("Way2sexy" and "Invisible") for comparisons. The exemplified figures are listed below:
1. Weipeng: The second row of Figure 4 in PM-Avatar (https://openreview.net/pdf?id=5t44vPlv9x).
2. Way2sexy: https://www.youtube.com/watch?v=gEpJDE8ZbhU.
3. Invisible: https://www.youtube.com/watch?v=ANwEiICt7BM.

Below, the comparison results on these three subjects wearing clothes with intricate textures or structured patterns furthermore highlight our robustness to complex garments.

|           | LPIPS |  FID  |  KID  |
|:---------:|:-----:|:-----:|:-----:|
| MonoHuman | 37.05 | 65.14 | 24.29 |
| PM-Avatar | 47.36 | 93.67 | 64.19 |
|    **Ours**   | **32.48** | **59.22** | **22.76** |


Additionally, please note that we generally follow the settings of latest baselines (MonoHuman, Vid2Avatar, 3DGS-Avatar and GoMAvatar) to perform fair comparisons including the definition of novel pose rendering. Except the video results on Youtube sequences, the video on ActorsHQ sequences provides another representative demo for the out-of-distribution animation because there is no squatting action in the training data. As discussed at Section E,  while no current methods can synthesize good results for loose clothing and extremely out-of-distribution poses, we can produce more visually appealing results than baselines.

Lastly, we sincerely thank Reviewer p8GG for giving detailed suggestions to enhance our experimental assessment. We also deeply appreciate Reviewer cJiu for the further suggestions on exposition and recognizing **the great potential of our method**. We would incorporate your comments into our revision and future research directions. It will also be great to hear more feedbacks from Reviewer nejY in the updated official comments afterwards.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces locality sensitive avatars which learns human motions from monocular videos. It decompose motion into canonical space and corresponding offset to non-rigid effects. A GNN is applied to model the non-rigid effects so as to keep fine-grained, locality-sensitive details. The proposed methods are tested on several datasets and achieved state-of-the-art results.

The proposed method studies an important problem as acknowledged by the reviewers. The proposed method is tested on several benchmarks and achieved state-of-the-art results. The original submission lacked experiments on some challenging dataset and comparisons with some existing works, but the authors addressed these concerns during the discussion. The paper is overall not so well written as pointed out by reviewer cJiu which makes it challenging to follow.

After discussion, all reviewers agree that this paper is above borderline, so I tend to accept this paper.