Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zhenhui Ye
authors: Weichuang Li
authors: Jiawei Huang
authors: Ziyue Jiang
authors: Jinzheng He
authors: Jinglin Liu
authors: Chen Zhang
authors: Xiang Yin
authors: Zejun MA
reviews: 
Comments: The paper proposes a method for realistic one-shot 3D talking face generation driven by videos and audio. Given an input source image, the method first reconstructs a canonical 3D face in the tri-plane representation with an Image-to-Plane (I2P) model. The I2P comprises 1. a ViT branch with a stack of SegFormer blocks to handle the canonicalization and a VGG branch to capture the high-frequency appearance features. The driving motion is represented by the projected normalized coordinate code (PNCC) [Zhu et al. 2016]. Instead of using the deformation field to animate as in previous papers, the method proposes to have a motion adapter composed of a shallow SegFormer to output the residual motion diff-plane given the PNCC. The sum of the canonical tri-plane and the motion diff-plane are volume-rendered and upsampled by a super-resolution module to end up with the source image driven by the given motion. Lastly, the background and the torso are composited with the Head-Torso-Background supersampling model. The driving PNCC can come from audio using an audio-to-motion model using a flow-enhanced VAE or video by fitting 3DMM to the reference video. The method is compared against Face-vid2vid [Wang et al. 2021], OT-Avatar [Ma et al. 2023], HiDe-NeRF [Li et al. 2023a] for the video-driven animation, and MakeItTalk [Zhou et al. 2020], PC-AVS [Zhou et al. 2021], and RAD-NeRF [Tang et al. 2022] for the audio-driven animation.
Strengths: Out of the contributions that the authors claim, the motion adapter generating the residual motion diff-plane to animate the canonical tri-plane given the target PNCC seems to be the most interesting part of this work.
Weaknesses: Some design decisions may not be well-supported. See my questions.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper proposes a method to create a reanimatable avatar from a single image. Leveraging the latent space of EG3D, the method learns to generate a canonical tri-plane from a given input image. This canonical triplane is then deformed using motion module conditioned on projected normalized co-ordinate code (PNCC). Finally, the deformed triplane is volume rendered to generate the final image. The motion conditioned tri-plane representation generates good results and outperforms some prior art, but the overall architecture and design is very similar to HiDe-NeRF.
Strengths: 1) The paper is well written.

2) The qualitative results on reanimation are good, even when driven by audio.

3) The quantitative results show that the proposed method out-performs prior art.

4) The background is rendered well and merges seamlessly with the foreground.
Weaknesses: 1) Given that the overall architecture is very similar to HiDe-NeRF, it is unclear where the improvement of the proposed method is coming from. Is it because a pretrained Tri-plane is a better representation than the multi-resolution tri-plane features of HiDe-NeRF? It would be great if the authors could clarify this

2) The addiction of background and torso modelling, while important, is relatively incremental.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 3 good