Gaussian Head & Shoulders: High Fidelity Neural Upper Body Avatars with Anchor Gaussian Guided Texture Warping
Authors: Tianhao Walter Wu
Jing Yang
Zhilin Guo
Jingyi Wan
Fangcheng Zhong
Cengiz Oztireli
Venue: ICLR 2025 Poster (2025)
Keywords: Neural Radiance Field
Gaussian Splatting
Neural Head Avatar
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The ability to reconstruct realistic and controllable upper body avatars from casual monocular videos is critical for various applications in communication and entertainment. By equipping the most recent 3D Gaussian Splatting representation with head 3D morphable models (3DMM), existing methods manage to create head avatars with high fidelity. However, most existing methods only reconstruct a head without the body, substantially limiting their application scenarios. We found that naively applying Gaussians to model the clothed chest and shoulders tends to result in blurry reconstruction and noisy floaters under novel poses. This is because of the fundamental limitation of Gaussians and point clouds -- each Gaussian or point can only have a single directional radiance without spatial variance, therefore an unnecessarily large number of them is required to represent complicated spatially varying texture, even for simple geometry. In contrast, we propose to model the body part with a neural texture that consists of coarse and pose-dependent fine colors. To properly render the body texture for each view and pose without accurate geometry nor UV mapping, we optimize another sparse set of Gaussians as anchors that constrain the neural warping field that maps image plane coordinates to the texture space. We demonstrate that Gaussian Head & Shoulders can fit the high-frequency details on the clothed upper body with high fidelity and potentially improve the accuracy and fidelity of the head region. We evaluate our method with casual phone-captured and internet videos and show our method archives superior reconstruction quality and robustness in both self and cross reenactment tasks. To fully utilize the efficient rendering speed of Gaussian splatting, we additionally propose an accelerated inference method of our trained model without Multi-Layer Perceptron (MLP) queries and reach a stable rendering speed of around 130 FPS for any subjects.

Published as a conference paper at ICLR 2025
GAUSSIAN HEAD & SHOULDERS: HIGH FIDELITY NEURAL UPPER BODY AVATARS WITH ANCHOR GAUSSIAN GUIDED TEXTURE WARPING
Tianhao Wu University of Cambridge
Jing Yang University of Cambridge
Zhilin Guo
University of Cambridge
Jingyi Wan University of Cambridge
Fangcheng Zhong University of Cambridge
Cengiz Oztireli
University of Cambridge
ABSTRACT
The ability to reconstruct realistic and controllable upper body avatars from ca- sual monocular videos is critical for various applications in communication and entertainment. By equipping the most recent 3D Gaussian Splatting representa- tion with head 3D morphable models (3DMM), existing methods manage to create head avatars with high fidelity. However, most existing methods only reconstruct head without the body, substantially limiting their application scenarios. We found that naively applying Gaussians to model the clothed chest and shoulders tends to result in blurry reconstruction and noisy floaters under novel poses. This is because of the fundamental limitation of Gaussians and point clouds each Gaussian or point can only have a single directional radiance without spatial vari- ance, therefore unnecessarily large number of them is required to represent complicated spatially varying texture, even for simple geometry. In contrast, we propose model the body part with neural texture that consists of coarse and pose-dependent fine colors. To properly render the body texture for each view and pose without accurate geometry nor UV mapping. optimize another sparse set of Gaussians as anchors that constrain the neural warping field that maps image plane coordinates the texture space. We demonstrate that Gaussian Head & Shoulders can fit the high-frequency details on the clothed upper body with high fidelity and potentially improve the accuracy and fidelity of the head region. We evaluate our method with casual phone-captured and internet videos and show our method archives superior reconstruction quality and robustness in both self and cross reenactment tasks. To fully utilize the efficient rendering speed of Gaussian splatting, we additionally propose an accelerated inference method our trained model without Multi-Layer Perceptron (MLP) queries and reach stable render- ing speed around 130 FPS for any subjects.
INTRODUCTION
Personalized and controllable 3D head avatar is a crucial asset for interactive Mixed Reality and metaverse applications. Recent developments in the 3D representations such as 3DMM (Li et al. Gerig et al. 2017). Neural Radiance Field (Mildenhall et al.) 2020), Instant Neural Primi- tives Müller et al. 2022), and other implicit representations (Mescheder al., 2019 have brought rapid advancements in the reconstruction of vivid and controllable neural avatars (Zhngg et al.)
2017
2022
Grassal al., 202T Zielonka et 2022 Gao al. 2022). With the most recent 3D Gaus- sian Splatting representation (Kerbl et al. 2023). neural avatars can be convincingly reconstructed
from a monocular video with impressive fidelity. However, most current methods for creating head avatars concentrate solely on the face and head, discarding other visible parts of the body by using semantic mask during the training process. Consequently, this results in avatars that appear as heads without bodies, which is not sufficient for many immersive applications. including video conferenc- ing, where a more complete avatar is needed (Shao et al.] 2024; Xiang et al.) 2024; Zielonka et al.] 2022 Gao et al. 2022). Recent techniques aim create complete avatars by including visible

---

Published as a conference paper at ICLR 2025
parts of the body, like shoulders and chest (Zhegg al. 2023 Zhao et al., 2024 Wang al. 2024
Zheng et al. 2022). However, they are limited to simplified settings where the subject dresses in plain clothing without detailed textures and is instructed to restrict upper body movement. On the other hand, existing full-body avatar methods typically focus on the overall quality of the limbs and torso and fail to faithfully capture the fine details such as high-frequency texture clothes (Ko-) cabas et al. [023] Hu et al. [2024b] Li et al.) 2024] Lei et al. (023). Applications of neural avatars that require detailed reconstruction of the upper body area often encounter significant challenges in capturing faithful and intricate details. Overall, current methods still fall short of delivering the level of performance needed for practical, real-world use.
The Gaussian Splatting representation, while being efficient and effective in certain aspects, strug- gles with accurate modeling of clothed upper bodies. As one of its fundamental limitations, each Gaussian can represent only one color from a specific viewing angle. This heavily limits its capa- bility to handle dynamic objects that have complex textures, such as clothing with intricate patterns. To capture the detailed appearance of such objects, an excessively large number of Gaussians would be needed, increasing memory requirement and slowing down the rendering speed. addition, the complicated vose-dependent appearances such brightness changes and cloth wrinkles further increase the difficulty of modeling them with plain Gaussians alone. As a result, when the recon- structed avatar is driven novel poses, the Gaussians tend to produce several undesirable artifacts such as blurred texture, incorrect colors or floating ellipsoid; see Fig 1
To address the limitations of existing Gaussian-based avatar methods clothed upper-body, we gue that the chest and shoulders are expected to have relatively simpler geometry and more intricate deformation compared the head. Therefore, modeling them with regular and 3DMM-driven Gaus- sians would be unsuitable and is an over-complication of the problem. Instead, a more appropriate and standard approach would be representing their appearance with a frequeer texture.
In traditional based rendering pipeline, the texture is first mapped to mesh geometry in the 3D world space via UV mapping. and then rasterized to the 2D image plane in the view space to obtain the pixel color. However, this approach requires a well-defined UV mapping and accurate mesh geometry, which is challenging obtain from monocular videos alone due to the lack of multi-view correspondences. Besides, compared to faces that share more common characteristics and stronger priors, the appearance of upper bodies can vary dramatically depending on the texture and tightness of the clothes and they hence contain fewer detectable landmarks. Consequently, body 3DMMs such as SMPL (Loper et al. (2115) fail to provide geometry accurate enough for this purpose.
Hence, we propose to bypass the mapping from texture space to world space, and instead use a sparse set of Gaussians "anchors" to define direct neural warping field from canonical 2D texture space, which consists of a coarse RGB texture and a fine neural texture, to the image plane. As the tracking of body 3DMM tends be inaccurate due to the lack of landmarks, we only trans- form anchor Gaussians together with the head Gaussians via a head FLAME 3DMM (Li et al., 2017) through Linear Blend Skinning (LBS). The transformed anchor Gaussians are used as soft constraints of the texture warping represented by a coordinate-basee gether with the neural texture, regular Gaussians, and the anchor Gaussians. As the resolution of the neural texture is not limited by the number of Gaussians or the density control scheme, we can easily learn the exhibited in Gaussian rendering under novel poses; see Fig I
MLP, which is optimized to-
textures with sharp details on the clothes and avoid the common artifacts
To maintain a competitive rendering speed with Gaussian Splatting and enable real-time interactive applications, we additionally propose method to remove the neural warping field and neural texture in the model and allow inference of reconstructed avatars at novel poses without any MLP querics. This accelerated inference effectively increases the rendering speed from 70 FPS to around 130 FPS, which surpasses the rendering speed of plain Gaussian Splatting avatars for subjects with high- frequency clothes.
We evaluate the proposed method with various casual monocular videos collected using smartphones or from the Internet. Compared to state-of-the-ar methods which incorporate different representa- tions including neural radiance field, Gaussian Splatting, and point clouds, we show that our ap- proach achieves better performance and robustness for both self-reenactment and rensareenntt tasks. In summary, our contributions are:
2

---

Published as a conference paper at ICLR 2025
Anchor Constrained
Texture Warping
3DMM-Driivn
Gaussions Amciion
Body Texture
Cwesd Monucular Video
Couseian FPS: 96
Ouss FPS: 132
FPS: 72
Figure 1: Gaussian Head & Shoulders reconstructs 3DMM-driven upper body avatars from casual monocular videos. By utilizing a high-frequency body neural texture which is warped using a neural texture warping field constrained by set of sparse anchor Gaussians, we can learn sharp details of the cloth texture with highly efficient rendering speed.
We propose a novel approach that maps intricate texture to the image plane via sparse set of anchor Gaussians driven by LBS with 3DMM. This allows accurate and robust modeling of high-fidelity clothed chest and shoulders with less number of Gaussians.
We propose a method to remove the MLP in our method at inference time to prevent any costly queries when rendering with novel poses and expressions and reach rendering speed of around 130 FPS.
2 RELATED WORKS
Neural Head Avatars The recent advancement in neural 3D implicit and explicit representations has sparked a surge methodologies within the field of controllable 3D head avatars. Among these approaches, a prominent family of methods involves the reconstruction of a 5D neural radiance
field, manifested through various forms such as pure MLP (Gafni et al., 2021 Wang et al.
I021
Kirschstein et al. 2023). hash grid latents (Xu et al.
2023
Gao et al. 2022 Ziclonka et al.
2022
Xu et al. [2023] Dhamo et al.) 2023; Xiang et al.
2024
Saito et al. 20244] Chen et al.) 2023)
and 3D Gaussians (Wang al.
2024 Zhao et al.,
2024).
Another
of methods utilizes more
2021 Zheng explicit representations such as deformable meshes with neural textures (Grassal et al.
al., 2022 Buehler et al. 2021
Gropp al., 2020 Khakhulin al., [022]] Kim | 2018)) and
point clouds (Zheng et al. 2023). With the most recent Gaussian Splatting techniques, the head
avatars reconstructed from monocular videos have already reached high fidelities. However, many methods simplify the problem by reconstructing only the head and neck part, resulting in head- only reconstruction that not suitable for many applications. Several methods have attempted to also model the chest and shoulders to provide a more immersive user experience (Zheng et al. 2022 Zhao et al. 2024 Wang al.) 2024 [Zhnng et al.) 2023). but they are limited to simple clothes with plain colors, and cannot handle the movements in the upper body in the video.
Neural Full-Body Avatars Several works have tried to reconstruct a controllable full-body neural
avatar from multi-view or monocular videos (Liu et al. 2024 (hao et 2024 Svitov al. 2024]
Li et al. 2024 Lei et al., 2023 Kocabas et al. 2023 Hu et al. 2024b Jiang et al., 2022) Due the highly articulated nature of human bodies, they tightly rely on body 3DMMs to deform the neural body representation via LBS and hence fail to faithfully capture subjects with complicated or loose clothing as those cannot be modeled with existing body 3DMMs. Besides, they typically focus on the overall quality of the torso and limbs, and hence tend to present non-trivial artifacts when reconstructing and re-animating an avatar that has a tight focus around the head and shoulder regions.
METHOD
Given a monocular video featuring a talking subject with various expressions and head poses, our goal is to reconstruct high-fidelity and animatable avatar including the head and clothed upper body. As illustrated in 2 our method jointly optimizes 1) a set of standard 3D Gaussians (Kerbl) al. 2023) which tightly follow the transformation of 3DMM via LBS represent the head region,
3

---

Published as a conference paper at ICLR 2025
Begale Garziins
Andlox Gansiine
Body
Tinal Readering
+
+
Garatton Recterization
T, (x)
MLN
8, t, Nibnh
LBS E, P.P. W
(d) Neural Texture
Xp
Anchor Conetracee Beguarisatin
Nidmk
(b) Compponndonee
(c) Warping
Figure 2: Method. (a) We utilize a of standard head Gaussians and anchor Gaussians driven by LBS with the FLAME model. target coordinates in the texture space. This 3D-2D correspondence is used to constrain (c) neural texture warping field that maps each pixel on the image plane x, to pixel in the texture space (d) We then sample in the texture space to fetch the coarse texture T, and latent texture T,, which parsed by an MLP to obtain pose-dependent fine texture C₁ Both coarse and fine textures are then combined to form a body texture, which is blended with other Gaussians through alpha compositing form the final rendering.
(b) Anchor Gaussians are initialized with set of corresponding
2) set of sparse anchor Gaussians spawning the clothed body, and 3) learnable neural texture with pose-dependent neural texture warping field constrained by the anchor Gaussians to represent the clothed body with sharp details and high robustness.
3.1 PRELIMINARY- GAUSSIAN SPLATTING
3D Gaussian Splatting is volumetric representation that utilizes dense set of anisotropic Gaus- sians with varying opacity and view-dependent radiance to represent 3D geometry and appearance. Each Gaussian is described with four parameters: position (Gaussian mean) μ1, 3D covariance ma- trix E, opacity a and Spherical Harmonic (SH) coefficients SH for computing view-dependent RGB color. For matrix S, stored as a scaling vector s, and a rotation matrix R, stored as quaternion vector q. The covariance matrix is obtained as: Σ RSS R..
of optimization, the covariance matrix further decomposed into scaling
To render 3D Gaussians to RGB images, their means are projected onto 2D image plane with standard projective transformation, while the projected covariance matrix is obtained by Σ' SWEWTJ where W is the world to camera transformation and J is the Jacobian approxi- mating the projective transformation (Zwicker al. 200T). The rendered RGB color at each pixel is then obtained through:
i-1
C(x)
Σ C
II az(x)),
IEN
j=1
(x = O; exp
(x (-1xx-1) (()= (1()
where x is the 2D pixel coordinate, c, is the riew-dependen RGB radiance of i-th Gaussian on the ray obtained from SH function, a; and μ¹ are the opacity and projected 2D mean of the i-th Gaussian respectively.
4

---

Published as a conference paper at ICLR 2025
3.2 TLAME-DRIVED HEAD GAUSSIANS
As the face region contains highly distinguishable characteristics and can be described accurately with parametric head 3DMM such as FLAME (Li at. 2017), we directly utilize standard 3D Gaus- sians that are deformed with parametric 3DMM via neural LBS to represent the head part ZZhnng al. 2023; Zhao et al. 2024). Specifically, we learn personalized FLAME expression and pose blendshapes and LBS weights through a small 3D coordinate- based MLP for each Gaussian: E, P.W MLPP a()),, where & E RR are the expression blendshapes, P E x9x3 are the pose blendshapes, W E IR' are the LBS weights corresponding to each of the nj bones. Fol- lowing (Hu & Liu| 2023). we use the standard skinning function LBS to obtain the rotation and translation T for each Gaussian, and apply them to get the Gaussian mean je and covariance Σ in the 3D view space:
R. T = LBS (p: P) (e( J(),
(3)
1 = Ru + T, Ed = RER",
(4)
where J is the joint regressor in FLAME, and Bp and Bg are linear combination of blendshapes based on per-frame coefficients 0 and v that control the head animation. They can then be rendered with a standard Gaussian rasterization pipeline in Eq 1
3.3 3D-2D CORRESPONDENCE VIA ANCHOR GAUSSIANS
3D Gaussian Splatting has shown promising performance and robustness in reconstructing 3D ge- ometry and appearance from RGB images. However, they suffer from a significant constraint each individual Gaussian can only represent a spatially invariant color under fixed viewing direction, hence vast number of Gaussians is required to represent objects with detailed textures, regardless of the actual complexity of the geometry. naive application of Gaussian Splatting therefore fails to capture the fine details of the upper body with complex textures and intricate deformation, and results in blurry details and floating artifacts under challenging poses.
We hence propose to learn high-frequency texture in canonical texture space, and use sparse set of Gaussians as anchors to guide the warping between texture space and image plane. As such, we only need a small number of Gaussians and a texture with per-pose warping represent a clothed body with arbitrarily complicated textures. Since anchor Gaussians themselves do not need to exactly represent the high-frequence appearance, we model them simplified version of regular Gaussians: they only use view-independent RGB colors, are isotropic Gaussians with quaternion fixed at (1, 0, O, 0), and are excluded from the density control and therefore are not split, cloned, or pruned. To prevent them from becoming trivial in rendering, their opacity and size are clamped to be no smaller than 0.05 and 0.0001 respectively.
The anchor Gaussians are initialized as follows: after a short warm-up period that only trains plain Gaussian, we first reproject all Gaussian means onto the image plane of canonical training frame, and filter out Gaussians that are located around the head region based on semantic masks. We then use farthest point sampling (Qi et al. 2017) to select N, 1024 Gaussians as anchor Gaussians. The first SH basis is converted to RGB values and the anchor scales in three directions are averaged to form single scale for the anchor Gaussians. We then obtain a sparse set of anchor Gaussians, as well as their projected 2D means 8.¹ on the image plane (2D view space) of the canonical frame:
-P(A))
where P is the camera projective transformation, Íu is the 3D Gaussian mean of the i-th anchor Gaussian transformed to 3D view space with LBS. To build the correspondence between anchor Gaussians and texture space coordinates, we assume that the mapping between the 2D image plane of the canonical frame and the texture space is an identity mapping. We can hence define a function fanchor (i) as fixed correspondence between the i-th 3D anchor Gaussian mean and its target 2D pixel coordinate in texture space:
(6) where I is the identity function to map 2D image plane coordinates to texture space. Note that fanchor (i) is fixed after initialization and does not update with further optimization of fię Such cor- respondences will later be used to constrain the pose-dependent texture warping, as will be detailed in Sec 3.6
fanchor (i) = I(x)),
5

---

Published as a conference paper at ICLR 2025
3.4 NEURAL TEXTURE AND TEXTURE WARPING
We use a trainable neural texture in canonical space with a pose-dependent neural texture warping field represent the part of the avatar with relatively simple overall geometry and complicated appearances, i.e., the clothed shoulder and chest. In a traditional textured mesh rendering pipeline, the texture is first mapped to the mesh triangles through pre-defined UV mapping, and the meshes are then rasterized to find the first intersections with the camera rays. Those first intersections therefore establish a mapping between texture space and image plane. However, this approach is not applicable without accurate surfaces and well-defined UV mapping. We instead propose to bypass the intermediate step and learn a per-pose warping that directly maps pixel coordinates on image plane x, to the texture coordinates x, for texture fetching. Specifically, the warping field is represented using a coordinate- based MLP:
Ax MLP v (v(x)), ((xx)),)(0)),tt)))xxxxx.... (xxddm)))
where is the positional encoding (Mildenhall et al., 2020). 0 is the FLAME pose parameters containing head and neck rotations, t is the camera position, Xidmk is 2D body landmarks for neck, left and right shoulders. The corresponding texture coordinate is obtained as X X, + Ax.
Our optimizable texture includes coarse texture Tr with channels and a latent texture Ty with Dj channels. Both textures have sizes of [H + 2P, W + 2P), where H, W are the image height and width, P is the padding size which we empirically set 50 to account for body parts that move in and out in the video sequence. The latent texture Ty is passed to an MLP to obtain pose-dependent appearances such as brightness changes on the clothes:
(x,) = MLP, (Ty (x,), 7(0). (()., (
(8)
where Tx(x)). (x(x) are coarse and latent texture sampled at 2D coordinate x, via bilinear in- terpolation. The textured pixel color at the coordinate X2 is therefore obtained as Cx(x,) Tx(x)) Cx.)).
By constraining with the correspondences between deformable anchor Gaussians and their fixed projections on 2D texture space. we can learn accurate and effective texture warping for various body movements including translation, rotation, and depth-based (in-and-out) motions; see Fig 3
3.5 RENDERING
To this end, we have a hybrid representation that includes 3D regular Gaussians that represent the head of the avatar, 3D anchor Gaussians that sparsely span over the body region, and 2D neural texture for the body. To render all of them together for joint optimization, we simply use alpha blending:
C*(x,)=
C(x,)
+(1 +1-6((x,)) C(x,) + (1-
axxx))
Anchor Gaussians
Hend Gaussians
Body Texture
where C(x,), C(x,) are the rendered RGB color of anchor Gaussian and regular Gaussian, á(xx), (x.) are the total alpha of anchor Gaussian and regular Gaussian pixel x, respectively.
Note that our rendering process always renders anchor Gaussians front of the regular Gaussians regardless of their actual positions. Though not physically realistic, designed this rendering order so anchor Gaussians are always non-trivial and never occluded by regular Gaussians.
3.6 OPTIMIZATION
The optimization is split into three different stages: anchor warm-up stage, main optimization stage, and texture refinement stage. In the anchor warm-up stage, neither anchor Gaussians nor body texture is applied, only the regular Gaussians are rendered and optimized. The purpose of this stage is to move Gaussians to roughly spawn over the area of interest including both head and body. At the end of this stage, we initialize anchor Gaussians from regular Gaussians using the method described in Sec 3.3 In the second stage, render all of the regular Gaussians, anchor Gaussians, and the textured body with alpha compositing described in Eq 9 stage, to recover faithful appearance for the body texture and enhance its robustness under novel
and jointly optimize them together. In the

---

Published as a conference paper at ICLR 2025
Anchor Grussion Projections Texture Spuce
Deformed Anchors Pur Space
Canmical Body Texture
Rendered Head Guzzsians & Body Texture Warpped with Anthor Conetrains
Figure 3: Anchor Warping. The anchors are initialized with corresponding projections on canonical texture space. When anchors are deformed via LBS model the per-frame body movement, they map to the same projections in texture space and hence establish correspondences for body texture warping.
poses, we remove anchor Gaussians from the rendering pipeline, i.e. set C and & to 0 in Eq 9 and freeze everything else except for the neural texture, texture warping field, and opacity and SH of regular Gaussians.
Following Zheng et al., 2023 2022), the training losses include standard MSE RGB loss Le = M MSE(C- -COT), and FLAME regularization that encourages the FLAME blendshapes and LBS weights predicted for each Gaussian stay close the pseudo ground truth & P., W, obtained from the nearest FLAME vertex:
1 N+NN
filum
Σ
(AElE: - Exl2 + Appp: Pele Aw/ Wale).
(10)
N
During main optimization stage, we additionally include VGG feature loss (Johnson et al. 2016 Simonyan & Zisserman 2015) LVGG F ugg (C) F vog (CGT)) ). and head mask regularization
encourage regular Gaussians to stay only within the head region and allow the body texture be trained properly without being occluded:
Chead = MSEE(mz(aa - ateead 0)),
(1I)
where O'ead is the alpha mask of the head region obtained with matting pre-processing and semantic mask. We also include L1 regularization background to be learned in the neural texture, as well as an LI loss to slowly decrease the opacity of anchor Gaussians allow the body texture be trained properly:
the 2D neural warping field encourage clean
HW La
N₂
Ewarp
|a.).
(12)
HW
N.
Finally, we include an anchor loss as a soft constraint of the per-pose texture warping:
1 No
Lanchor
"
(fanchor (i) (Φ) + 22;))²
(13)
No 1=1
ie., for each anchor Gaussian, we first transform it to 3D view space via LBS, and then project onto the image plane to obtain its 2D mean á via Eq 2 is then warped by the neural warping field MLP to obtain the corresponding coordinate in the texture space, which is optimized to match the anchor correspondence defined during initialization.
In the third stage, we remove the regularization losses including Lhead, Lwarp and La-
The total training objectives for each of the three stages are as follows:
L Lo fllame
(14)
L. = Ci + AvGoLyGG + AheadLhead + Awarp + AaLa + Nanchor Lanchor,
Ca Li AycoLvag + Danchor Lanchor-
(15)

---

Published as a conference paper at ICLR 2025
3.7 ACCELERATED RENDERING WITH No MLP QUERIES
One of the main advantages of Gaussian Splatting is its highly efficient rendering speed, which enables many real-time and interactive applications. To take full use of this advantage, we propose an accelerated version of our method that requires no MLP queries at inference time. Specifically, after training the model, we first cache the output of MLPs d for all head Gaussians and anchor Gaussians, then cache the view-dependent fine texture by querying the fine texture MLP MLP, conditioned on the same canonical training frame which was previously used to initialize the anchor Gaussians. The queried fine texture colors are added to the coarse color to make non-neural RGB texture. To deal with potential noise created by the fine texture MLP at the corners of the texture, we use an off-the-shelf background segmentation network (Chen et al. 2017 to compute coarse mask and clean all the pixels outside of the mask; we show the necessity of this step in the supplementary. To replace the neural warping field MLP that warps image plane coordinates to texture space, rely on the correspondence between anchor Gaussians and texture space coordinates to estimate a homography at inference time. Specifically, of the canonical training frame, and then remove any anchor Gausians that go beyond the view frustum. To deal with any potential discrepancy between the neural warping field and the anchor correspondences, we update those correspondences based on the prediction of the neural warping field on the current frame:
first project all anchor Gaussians to the image plane
fonchor (1) X + ARj.
(16)
After that, we randomly select 100 training frames and use RANSAC (Fischler & Bolles 1981) to estimate a homography between the image plane coordinates of anchor Gaussians and their cor- responding texture space coordinates, and remove anchor Gaussians that are considered outliers by RANSAC. This effectively removes any anchor deformation that cannot be described by the rigid transformation. Finally, at inference time, we perform LBS regular head Gaussians and anchor Gaussians, Based on the image plane coordinates of the anchor Gaussians x) and their correspondences fanchor, compute homography with the least square error via singular value decomposition. The estimated transformation is applied to all pixels on the image plane to find the corresponding non-neural texture, which is then blended with the head Gaussians to form the final rendering. This accelerated inference approach effectively increases the rendering speed from around 70 FPS 130 FPS.
EVALUATION
Datasets We evaluate different methods on mobile phone sequence from PointAvatar (Zheng et al. (003). 2 internet sequences from Head2Head dataset (Koujan et al.) 2020). and 4 sequences captured with mobile phones. All sequences are preprocessed with DECA (engg al. 202I and a slightly modified landmark fitting process from IMAvatar (Zheng et al. 2022). Additionally, we use DWPose Yang al. 2023) to predict 2D landmarks for nose, neck and shoulders, which are then smoothed with One Euro Filter (Casiez et al. 2012).
Baselines We compare our method with four neural head avatar methods based on various repre- sentations, including (1) INSTA (Ziclonka et al. 2022). which employs latent hash grid (Müller ct al. 2022) combined with NeRF (Mildenhall et al. 2020). (2) PointAvatar (Zheng et al. 2023), which is based on isotropic point clouds, (3) Splatting Avatar (Shao et al., 2024) and (4) GaussianA- vatars Hu et al. (2024a), which utilize Gaussian Splatting attached to local space of 3DMM meshes, and (5) GSa, baseline we implemented by changing the point cloud representation in PointAvatar to Gaussian Splatting, which is similarly deformed via neural LBS.
Seff-Reenactment We show the quantitative and qualitative results of the self-reenactment task in Tab [] and Fig Our full version demonstrates superior reconstruction performance compared to existing baselines, especially for subjects with intricate cloth textures. Our No MLP version does not consistently achieve better PSNR when compared to existing baselines, as it is unable to render pose-dependent appearance changes and intricate cloth deformation. However, we note that it consistently achieves better LPIPS, demonstrating that our No MLP version can still generate realistic and faithful renderings. This discrepancy among different metrics arises because of the high sensitivity of PSNR small misalignments in the cloth texture (Park et al. 2021). As result, PSNR
8

---

Published as a conference paper at ICLR 2025
full
head
FPS
#GS
FPS
#GS
PSNR
SSIM
LPIPSL
PSNR1
SSIM
LPIPS]
003
004
INSTA
20.59
.781
.236
29.80
.936
.059
Flash Avatar
134
13453
137
13453
SplattingAvatar Avatta
21.00
.774
.274
28.29
.925
.069
GS*
141
163830
159
125521
PointAvatar
25.21
.851
.102
30.64
.943
.042
Ours
70
58701
71
39549
FlashAvatar
21.88
.808
.127
30.33
.946
.039
Ours (No MLP)
129
58701
134
39549
GaussianAvatars
20.97
.801
.193
29.285
.941
.046
005
007
GS*
25.94
.854
.095
31.81
.947
.036
Avattr
125
13453
126 13453
Ours
26.80
.875
.070
33.06
.959
.028
GS
96
317968
131 191431
Ours (No MLP)
25.47
.859
.074
32.54
.956
.029
Ours
72
50708
69
52910
Ours (No MLP)
132
50708
127 52910
Table 1: Quatitative evaluation of self-reenactment
task We color the best and second-best methods. Our Table 2: Performance measure. We full method achieves much better performance compared report rendering FPS and the number to existing baselines. While Ours (No MLP) achieves of Gausssians for each method. The slightly lower PSNR, which is known be over-sensitive rendering speed of our no MLP ver- to small misalignments and prefers blurry results (Park sion surpasses pure Gaussian implemen- al., 202I), we show it achieves better LPIPS than exist- tation for subjects wearing extremely high-frequency cloth.
ing methods.
tends to prefer blurry reconstruction over sharp but slightly misaligned results. Notably, although we did not include specific treatments for the head region, better modeling the body also leads to better face reconstruction. The qualitative evaluation in Fig 4 demonstrates that both versions of our method can learn sharper and more robust body texture compared to existing methods.
Cross-Reenactmenn Reenactment For the cross-identity reenactment task, we render the reconstruction of the original identity with FLAME expressions and poses from the source subject. With the full version of our method, we apply an additional Euclidean transformation after warping the image plane co- ordinates with the MLP. This ensure the body texture is always aligned with the head Gaussians under novel poses; see 6 The Euclidean transformation is simply determined by fitting the MLP warped image plane coordinates the anchor Gaussians and their target coordinates the texture space. To deal with potential artifacts caused by coordinates warped to unseen corner parts in the texture, we apply the same appearance distillation process and remove the fine texture MLP. The No MLP version is applied the same way as in the self-reenactment task.
In addition to the improvement over the body texture, we observe that avatars reconstructed with our approach often give more accurate and faithful expression control, shown in Fig 5 We deduce that this is because the 3DMM-driven Gaussians only need to model the head region, leading to a more accurate reconstruction of the head model and more reliable LBS weights and expression and pose blendshapes predicted by the LBS network.
Ablation We show the effectiveness of the anchor constraint Lanchor, test-time Euclidean trans- formation and warp loss Lworp in Fig 6 Even for subjects with only slight movement in the upper body, anchor constraint is still needed to learn sharp and accurate cloth texture. Besides, without anchor Gaussians and test time Euclidean transformation, the body texture is unable to align with the head Gaussians under novel poses. The warp loss Lwarp is needed to prevent the neural warping field from mapping the background pixel to an arbitrary white pixel in the texture space. As anchor Gaussians only exist within the body region, the additional Euclidean transformation computed from anchor correspondences would significantly distort the background pixels, causing severe artifacts as shown in Fig 6 (b). Additional ablation results can be found in the supplementary.
Rendering Efficiency We report the number of Gaussians and the rendering speed for pure Gaus- sian implementation GS* Ours, and Ours (No MLP) in Tab 2 The rendering speeds are tested on an RTX4080 Ti. For subjects wearing complicated clothes, the number of Gaussians required to model the high-frequency cloth texture significantly increases for pure Gaussian implementation, hence slowing down the rendering speed, whereas our method only models the head region with Gaussians and hence requires much fewer number of Gaussians. The rendering speed of our

---

Published as a conference paper at ICLR 2025
008
INSTA
Sp Auataa
FlashAustir
MLP)
Figure 4: Qualitative comparison relfrreenttmment task. We show that both of our full version and No MLP version can recover accurate and robust body texture, under extreme poses and high-frequency cloth textures, More results in the Supplementary 7
(b)
Source
PointAvatar
GS
Ours
Ours Noo MLP
No Euclidean
Warp
Ours
Figure 5: Qualitative evaluation of cross-identity reenactment. Our method leads to both better cloth texture and more accurate expression, as LBS net- work only focuses on the head region in our ap- proach. More results in the Supplementary g
Figure Qualitative ablation for self- reenactment (a) and cross-reenactment (b).
MLP version even surpasses pure Gaussian implementation for subject 005, who wears cloth with very fregurrry texture.
CONCLUSION
We present Gaussian Head & Shoulders, method that reconstructs high-quality and animatable upper body avatars including head, chest and shoulders. By utilizing high-frequency neural texture represent the clothed body, reduce the number of Gaussians needed to represent subject. By constraining the texture warping with sparse set of anchor Gaussians, the body texture is accurately mapped the correct position even under unseen poses. By caching the neural texture and replacing the neural warping field with projective transformation estimated using anchor correspondences, we significantly improve rendering speed and reach over 130 FPS at novel poses, surpassing the rendering speed of pure Gaussian implementation for subjects with complicated cloth textures.
able to model sharp and robust cloth details and significantly
Limitation. Although method can learn faithful texture for the shoulder and chest, it cannot handle arm and hand motions, which would require specific prior and representation such as SM- PLX (Loper et al. 2015).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by UKRI Future Leaders Fellowship [grant number G104084).
REFERENCES
Marcel C. Buehler, Abhimitra Meka, Gengyan Li, Thabo Beeler, and Otmar Hilliges. Varitex: Variational neural face textures. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Géry Casiez, Nicolas Roussel, and Daniel Vogel. 1 E filter: simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '2, pp. 2527-2530, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 978145085 450310411, doi: 10. 10.1145/2207676.22066333 URL https:
//doi. 10. 1145/2207676 2208639
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. 06 2017.
Yufan Chen, Lizhen Wang, Qijing Li, Hongjiang Xiao, Shengping Zhang, Hongxun Yao, and Yebin Liu. Monogaussianavatar Monocular gaussian point-based head avatar. arXiv, 2023.
Helisa Dhamo, Yinyu Nie, Arthur Moreau, Jifei Song, Richard Shaw. Yiren Zhou, and Eduardo Pérez-Pellitero. Headgas: Real-time animatable head avatars via 3d gaussian splatting, 2023.
Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3D face model from in-the-wild images. volume 40, 2021. URL https::/ss/// //cc:////iii wwt.....o.....000 3450626. 3459933559
Martin A. Fischler and Robert C. Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381-395,
jun 1981. ISSN 0001-0782 doi: (11144///8666.35886698 URL
1145/358 668.6669.44 358692
Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8649-8658, June 2021.
Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong. Yudong Guo, and Juyong Zhang. Recon- structing personalized semantic facial nerf models from monocular video. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia), 41))), 2022. doi: 10. 145/3550454.3555000
Thomas Gerig, Andreas Forster, Clemens Blumer, Bernhard Egger, Marcel Lüthi, Sandro Schönborn, and Thomas Vetter. Morphable face models an open framework. CoRR, abs/1709.08398, 2017. URL
Philip- William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgb videos. arXiv preprint 1r.1v:2112.0/554, 2021.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric reg- ularization for learning shapes. In Proceedings of Machine Learning and Systems 2020, pp.
3569-3579. 2020.
Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. In IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), 2024a.
Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. In /EEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR). 2024h.
11

---

Published as a conference paper at ICLR 2025
Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting from monocular human videos. arXiv preprint arXiv:, 2023.
Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from single video. In Proceedings the European conference on computer vision (ECCV), 2022.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 694-711, Cham, 2016. Springer International Publishing. ISBN 978-3-
319-46475-6.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL htttt::/reeps sam. inriii htunsaa.t///ugggga///ctataaa saunggraphhh--uuusiiiae gaussian splatting/
Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. Realistic one-shot mesh-based head avatars. In European Conference of Computer vision (ECCV), 2022.
Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollöfer, and Christian Theobalt. Deep video por- traits. ACM Transactions on Graphics (TOG), (7(4): 2018.
Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner. Nersem- ble: Multi-view radiance field reconstruction of human heads. ACM Trans. Graph., 42(4), doi: 10.1145/3592455. URL
jul 2023. ISSN 0730-0301.
org/10org10... 1145/
3592455
Muhammed Kocabas, Rick Chang, James Gabricl, Oncel Tuzel, and Anurag Ranjan. Hugs: Human gaussian splats, 2023. URLLttpp:///cxxv.orr abspg//ab/233111 17910
M. Koujan, M. Doukas. A. Roussos, and S. Zafeiriou. Head2head: Video-based neural head syn- thesis. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) (FG), pp. 319-326, Los Alamitos, CA, USA, may 2020. IEEE Computer Society. doi: 10.1100/FFG48880.222222 200000008. 00048. URL https://doi. wew.coppteeeesereettyooooo/////.
11C9/FG4 7880 2020 00048
Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Danilidis. articulated template models, 2023.
Gart: Gaussian
Tianye Li, Timo Bolkart, Michael. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans. ACM Transactions Graphics, (Proc. SIGGRAPH Asia), 36(6):194:111944 2017. URL httpt:ttt///o /doi wwgg/1.1110 45/3130800 31000.113088 3130813
Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning pose- dependent gaussian maps for high-fidelity human avatar modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Jing Lin, Ailing Zeng, Haoqian Wang, Lei Zhang, and Yu Li. One-stage 3d whole-body mesh recovery with component aware transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21159-21166. 2023.
Xinqi Liu, Chenming Wu, Jialun Liu, Xing Liu, Chen Zhao Haocheng Feng, Errui Ding, and Jing- dong Wang. Gva: Reconstructing vivid 3d gaussian avatars from monocular videos. Arxiv, 2024.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), (466)248:11224:11 October 2015.
L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learn- ing 3d reconstruction in function space. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1455-4465, Los Alamitos, CA. USA, jun 2019. IEEE Computer Society, doi: 10. 10O.///...009...0000/ 900599. URL htt.httts/// //ttps:://d..... leecappterrscccettyy.. org/ 110./119///00 CVPR. 201.T.....9.0000
12

---

Published as a conference paper at ICLR 2025
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with a multiresolution hash encoding. ACM Trans. Graph, 10:::::12:1111::::00 July 2022. doi: 10.1 10.1145/522223335531222
URL https: http:////00...00000111114///////
3530127
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Gold- man, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representa- tion for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dec 2021.
Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv: 2017.
Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. Relightable gaussian codec avatars. In CVPR, 2024.
Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, and Zeyu Wang. SplattingAvataar Realistic Real-Time Human Avatars with dessh-Emeedded Gaussian Splatting. In Computer Vision and Pattern Recognition (CVPR), 2024.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http:
httt:://arxiv.oom arxiv. www.aas../....ooooo../.....a...e 1409. 1409..1/1141.155
Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regres- sion of Multiple 3D People. In ICCV, 2021.
David Svitov, Pietro Morerio, Lourdes Agapito, and Alessio Del Bue. Haha: Highly articulated gaussian human avatars with textured mesh prior, 2024.
Jie Wang, Jiu-Cheng Xie, Xianyan Li, Feng Xu, Chi-Man Pun, and Hao Gao. Gaussianhead: High- fidelity head avatars with learnable gaussian derivation, 2024.
Lizhen Wang, Xiaochen Zhao, Jingxiang Sun, Yuxiang Zhang, Hongwen Zhang, Tao Yu, and Yebin Liu. Styleavatar: Real-time photo-realistic portrait avatar from a single video. In ACM SIG- GRAPH 2023 Conference Proceedings, 2023.
Ziyan Wang, Timur Bagautdinov, Stephen Lombardi, Tomas Simon, Jason Saragih, Jessica Hod- gins, and Michael Zollh ofer. Learning compositional radiance fields of dynamic human heads. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5704-5713, 2021.
Jun Xiang. Xuan Gao, Yudong Guo, and Juyong Zhang. Flashavatar: High-fidelity head avatar with efficient gaussian embedding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Yuelang Xu, Lizhen Wang, Xiaochen Zhao, Hongwen Zhang, and Yebin Liu. Avatarmav: Fast 3d head avatar reconstruction using motion-aware neural voxels. In ACM SIGGRAPH 2023 Confer- ence Proceedings, 2023.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4211-4220, 2023.
Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang. Weichuang Li, Jiawei Huang. Ziyue Jiang, Jinzheng He. Rongji Huang, Jinglin Liu, et al. Real3d Reattd-portrait: One-shot realistic 3d talking portrait synthesis. arXiv preprint S0Vv:2401008500 2024.
Zhongyuan Zhao, Zhenyu Bao, Qing Li, Guoping Qiu, and Kanglin Liu. Psavatar: A point-based morphable shape model for real-time head avatar animation with 3d gaussian splatting, 2024.
13

---

Published as a conference paper at ICLR 2025
Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges. IM Avatar: Implicit morphable head avatars from videos. In Computer Vision and Pattern Recognition (CVPR), 2022.
Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, and Otmar Hilliges. Pointavatar: Deformable point-based head avatars from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Wojeiech Zielonka, Timo Bolkart, and Justus Thies. Instant volumetric head avatars. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4574-4584 2022. httthtt:://aa
semant icscholar.... wttg/CorppsD::2337661111
M. Zwicker, H. Pfister, J. van Baar, and M. Gross. Ewa volume splatting. In Proceedings Visualiza- tion, 2001. VIS '01., pp. 29-538, 2001. doi: 10. 10.109//ISUAL..201999442
14


Reviews:
[{"review_id": "nUUmPZ7pHk", "reviewer": "Reviewer_SS61", "summary": "This paper proposes a method to reconstruct upper body avatars from videos based on 3D Gaussian Splatting. The method models head and shoulders separately, i.e., heads by 3D Gaussians anchored on 3DMM template, and shoulders by Anchor Gaussians and neural textures. The paper proposes a Neural Texture strategy to model shoulders with less 3D Gaussians. The authors evaluate their methods on different experimental datasets and human subjects.", "questions": "Considering the 3DMM parameters and landmarks of the shoulder are acquired separately, is there a method to unify the two tracking results? It seems that the head and shoulder are misaligned in some frames.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a clear structure, and it‘s easy to follow.\nThe idea of modeling the head and shoulder separately is interesting. The neural texture strategy can model the shoulders efficiently with less 3D Gaussians.\nThe pipeline is verified on different human subjects.", "weaknesses": "The method models head and shoulders separately, i.e., different Gaussian representations and different tracking approaches. This causes a limitation, i.e., the head and shoulders are misaligned for some relatively big body movements. \n \nIt seems that the Anchor Gaussian only supports relatively simple should motions. It’s not verified that the proposed method can model pose-dependent wrinkles for the shoulders in the paper. The body movements in the supplementary demo are too simple, and I cannot observe that the clothing wrinkles vary with poses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "58eY1Zlehk", "reviewer": "Reviewer_CZUM", "summary": "This paper proposes a Gaussian splatting based deformable avatar method with a specific focus on modelling shoulders with a high fidelity. The head is represented by a set of 3D Gaussians while the shoulders are represented by a neural texture map. The gaussians on the head are deformed from the canonical space using Linear Blend Skinning while the texture-map for the shoulders are warped using a pose dependent MLP. Once in the deformed space, both the neural textures and the gaussians are rendered to yield the final image. The properties of the gaussians and the warpings are learnt through a set of RGB and deformation based losses. The neural texture warping is specifically constrained through an anchor loss defined using a set of Gaussians learnt on the shoulders. Both quantitative and qualitative show improvements over prior work.", "questions": "1) As mentioned above, I’m curious to see novel out-of-plane renders of the shoulders themselves to see how geometrically accurate they are. \n\n2) There is no ablation of exactly how robust the proposed method is to shoulder movement in the capture video. Understanding this would help evaluate the method more throughly (I do grant that prior work does not even address this problem, but I believe this is important to make the paper more complete.)\n\n3) Any reason why Splatting-avatar results are so much worse compared to the other methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1) The paper is well written and easy to follow\n2) The use of a neural texture map to model the shoulders is well motivated and intuitive\n3) The additional loss terms introduced are well ablated in both the main paper and the supplementary\n4) The proposed method both qualitatively and quantitatively outperforms prior work", "weaknesses": "1) There are no significant out-of-plane view synthesis results that are shown in the results to demonstrate whether or not the learn texture map is able to generate novel renders of the shoulders in a geometrically consistent way or it just renders them as a plane \n\n2) This paper is, I believe, a little out of scope of ICLR and is more suited at a vision or graphics conference. However, I will defer to other reviewers and the AC for that before I consider it as a strong weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1sYCStTUi1", "reviewer": "Reviewer_Dd1H", "summary": "This paper introduces a method to create a high-fidelity head avatar.\nThis method maps intricate texture to the image plane through a sparse set of anchor Gaussians driven by 3DMM, not only reconstructs the head, but also the upper body parts.\nIn addition, this paper also proposes a method to use Gaussian spraying to speed up rendering.", "questions": "- Since the modeling of the body parts starts from a canonical frame/space, will the selection of this canonical frame significantly affect the reconstruction quality?\n- I still have doubts about how this method drives the reconstruction of the upper body of the character for unseen self-driving frames and cross-driving frames. From the visual results, it seems that the upper body is not aligned with the driving frame.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper uses neural textures to model body parts simultaneously, instead of just the head.\n- The paper proposes a method to accelerate reasoning, so that the rendering speed can reach 130 frames per second.", "weaknesses": "- This paper lacks comparisons with some methods in terms of head reconstruction quality, including [1, 2].\n- The head part of this method is similar to the theory of existing methods (PSAvatar).\n- The tracking results given in Figure 11 are not convincing. In many body tracking projects, some scenes containing half of the body can still produce reasonable results. For example, One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer. The paper should describe in more detail the methods used for tracking and why they were failed, otherwise I don't see the point why not using the upper body SMPL model.\n\n- Reference\n\n        [1] Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians\n        [2] GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G7TVmY3H4n", "reviewer": "Reviewer_bZ83", "summary": "The paper addresses the challenge of creating realistic upper body avatars from monocular videos. Existing methods focus on head avatars and struggle with detailed body textures, resulting in blurry reconstructions. The proposed solution uses neural textures guided by sparse Gaussians to improve fidelity and speed, achieving up to 130 FPS, and excels in both self and cross-reenactment tasks.\nContributions:\n1. It uses anchor Gaussians to effectively model detailed upper body textures.\n2. It eliminates MLP queries for faster speeds, reaching about 130 FPS.", "questions": "1. FlashAvatar is motivated by inference acceleration. Has this paper calculated its FPS?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This method effectively models detailed upper body textures.\n2. This paper combines 3DGS and neural fields to construct head and body textures, showcasing technical novelty.\n3. Experimental results demonstrate that this method has significant advantages in rendering accuracy and inference speed while using a smaller number of 3D Gaussians.", "weaknesses": "1. The article lacks some clarity in its writing. For example, it does not explain what texture space is in the introduction.\n2. The article could include additional baselines, such as Gaussian Avatars and Gaussian Head Avatar.\n3. Regarding the rendering formula in Section 3.5, the coefficient for the third term, body texture, is the product of the opacities of the first two terms. Wouldn't this cause most of the pixels' colors in the third term to be overridden?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PKPxVRv33C", "reviewer": "Reviewer_Q3T9", "summary": "This work present the way of rendering shoulders in addition to the human faces. In general, the author propose the following methods:\n(1) to use anchor gaussians to represent the shoulder region only.\n(2) use neural texture to refine the shoulder represent and make it robust to body motions including translation, rotation, and depth-based motions;\n(3) Leverage 3D-to-2D correspondence mapping as the regularization of the shoulder learning.\n(4) an option without using MLP for fast rendering\n\nThe author conducts many experiments to validate their method.", "questions": "1. Can the authors provide the novel view synthesis for rendering the shoulder and also present the volumetric representation of the shoulder areas to let readers know how robust the model is for shoulder rendering?\n\n2. What exactly each module brings or benefits for the shoulder rendering? I think the following are possible ablations: \n(a) completely remove anchor gaussian and the 3D-2D correspondence, only relying on neural texture like in StyleAvatar\n(b) exclude Neural texture, use anchor gaussian only\n\n3. For the anchor gaussian, how the deformation achieved? If it is relying on LBS, due to that LBS is based on flame mesh that only contains the head and neck, there does not exist a real LBS for the shoulders. How should we understand the relationship between the deformation of shoulder gaussian point and neural texture? Is it a refinement of potential noisy deformed gaussian points?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The overall writing is easy to following and well-written.\n\nIt seems like compared with conventional method, which ignore the shoulder but only focus on face, this model presents better performance and more robust to shoulder renderings (like compare with PointAvatar and GS*)", "weaknesses": "Concerns of Comparisons:\n\n1. Some works already tries to explore the joint face and shoulder rendering including:\n   a. Real3D-Portrait, which uses FOMM for the shoulder motion reenactment and finally merge the face motion and shoulder motion together. \n  b. StyleAvatar, a 2D human face rendering technique which includes the face, shoulder and background together\n\nThe author did not include these two into the comparison. Is it possible to compare with this pipepile with FOMM/TPS-based methods used in Read3D-Portrait / Torso NeRFs used previously in GeneFace to see whether this method is more effective? In addition, using neural texture as an whole image enhancement method is also covered in StyleAvatar, but slightly different: this work tries to enhance the shoulder only while StyleAvatar's neural texture is for the whole image. It will be good if the authors can present very strong argument regarding this given my evaluation might be partially wrong.\n\nData:\nThe videos in the supplementary material do not present very large shoulder motions. I am not quite sure whether the model is robust in rendering the shoulder. However, I do see for slight shoulder motions, the model's performance is good.\n\nExperiment:\nI did not see the ablations for the main method part. The proposed methods include introducing the anchor gaussians for shoulder only, the 3d-2d correspondence, the neural texture compared with the baseline GS and PointAvatar. I will defer this part in the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)