FaceGPT: Self-supervised Learning to Chat about 3D Human Faces

## Abstract

Abstract We introduce FaceGPT, a self-supervised learning framework for Large Vision-Language Models (VLMs) to reason about 3D human faces from images and text. Typical 3D face reconstruction methods are specialized algorithms that lack semantic reasoning capabilities. FaceGPT overcomes this limitation by embedding the parameters of a 3D morphable face model (3DMM) into the token space of a VLM, enabling the generation of 3D faces from both textual and visual inputs. FaceGPT is trained in a self-supervised manner as a model-based autoencoder from in-the-wild images. In particular, the hidden states of LLM is projected into 3DMM parameters and subsequently rendered as 2D face image to guide the self-supervised learning process via image-based reconstruction. Without relying on expensive 3D annotations of human faces, FaceGPT obtains a detailed understanding about 3D human faces, while preserving the capacity to understand general user instructions. Our experiments demonstrate that FaceGPT not only achieves high-quality 3D face reconstructions but also retains the ability for general-purpose visual instruction following. Furthermore, FaceGPT learns fully self-supervised to generate 3D faces based on complex textual inputs, which opens a new direction in human face analysis.

## 1 Introduction

In this work, we address the problem of reasoning about 3D human faces from images and text.
Traditional monocular 3D face reconstruction methods typically estimate the parameters of a 3D morphable model [, , , , ] given 2D face images as input.
However, these methods lack the capability to reason about faces based on textual input.
Unlike these systems, humans can vividly imagine and even draw faces based solely on either face images or textual descriptions.
Motivated by recent advances in large vision-Language models (VLMs) [, ],
we aim to explore a path forward towards enabling vision-language systems to obtain an in-depth reasoning-based understanding of 3D faces.

To investigate this question, we present FaceGPT, a vision-language model with an intricate ability to reason about 3D human faces from visual and textual input.
We represent faces as 3D morphable model (3DMM) parameters that include parameters for the 3D face shape, expression, albedo, and scene illumination.
Following related work on image segmentation [] and human pose estimation [], we extend the original token space of the VLM by incorporating a new <FACE> token that is decoded into 3DMM parameters using an MLP (Fig. 1).
Thus, when combined with a differentiable computer graphics renderer [], the VLM model becomes capable to synthesize face images.
This enables us to formulate FaceGPT within a model-based autoencoder framework [], and hence to train our model in a fully self-supervised manner from in-the-wild images.
To the best of our knowledge, this is the first work combining vision-language model with an inverse graphics pipeline.
During training, we freeze the visual encoder of the VLM while training the MLP and the LLM using LORA [].
The model is trained with three types of data: (1) Standard multi-modal instruction tuning data to retain the general capability and quality of the VLM. (2) In-the-wild face images for training the <face> token and 3DMM projection layers via self-supervised inverse rendering. (3) Text-to-3DMM data that enables the generation of 3DMM parameters from text input only. We construct this dataset from a set of face images in the wild by running an off-the-shelf self-supervised monocular face reconstruction method [] and by generating textual descriptions of the faces via the original VLM.

We evaluate FaceGPT on a variety of tasks, including traditional 3D face reconstruction, text-to-3DMM face generation and general-purpose visual instruction following.
We demonstrate that FaceGPT becomes a general-purpose model that achieves competitive results at 3D face reconstruction when compared to specialized methods, while retaining competitive visual instruction following capabilities, as well as being capable of reasoning-based text-to-3D face generation.
Beyond face analysis, we believe that the design principles underlying FaceGPT are general and it also suggests a pathway towards a self-supervised integration of the "world knowledge" that VLMs derive from extensive textual data and the structured 3D representations of the visual world via computer graphics.

In summary, our work makes the following concrete contributions:

- •
To the best of our knowledge, this is the first work that enables vision-language models to learn a detailed 3D face understanding in a fully self-supervised manner.
- •
We show that VLMs can learn text-based face reconstruction, which predicts 3D human faces given user instructions, in a fully self-supervised manner.
- •
Our experiments on traditional 3D face reconstruction, visual instruction following and text-based face reconstruction, demonstrate the general face understanding capabilities of FaceGPT.

Figure: Figure 1: We introduce FaceGPT, a large vision-language model that learns to produce 3D human faces (in terms of 3DMM parameters) in a fully-self-supervised manner. When prompted with face images and task-specific questions, FaceGPT can output a special <FACE> token of which the corresponding feature embedding (red) can be decoded into 3DMM parameters, that encode the face shape $\alpha$, expression $\delta$, the texture $\gamma$, the light $\phi$ and camera $c$ parameters. When decoded with a 3DMM and differentiable renderer, this enables a fully self-supervised learning via inverse rendering. FaceGPT can produce 3D human faces from text-only input (first row), as well as multi-modal input (second row) without requiring any 3D annotations, while retaining general chatting abilities (last row).
Refer to caption: extracted/5658670/figures/fig1.png

## 2 Related works

### 2.1 Monocular model-based face reconstruction

Realistically reconstructing digital human faces has been a longstanding challenge in computer vision and graphics due to their vast potential applications. Traditional methods primarily use parametric 3D Morphable Models (3DMM) [, , ] with PCA for dimensionality reduction to simplify high-dimensional 3D face scans, serving as a 3D prior for representing unique facial characteristics and providing precise control. Recently, deep learning-based methods that map 2D images to 3D face models have gained popularity. Early methods struggled with the need for extensive 3D facial scan data paired with 2D images, which was labor-intensive and costly. This limitation was addressed with the introduction of model-based face autoencoders (MoFA) [] that enabled self-supervised 3D face reconstruction.
MoFA uses a differentiable rendering layer to minimize differences between input and rendered images, enabling end-to-end learning without ground-truth 3D faces, leading to a number of effective extensions in the self-supervised learning strategies  [, , , , ].
RingNet [] and DECA [] use landmark-based training, predicting landmarks for input images and treating them as pseudo ground truth, measuring the distance between 2D face landmarks and their projections on the 3DMM surface.
The FOCUS [] framework jointly trains a face autoencoder and an outlier segmentation network, which makes the method robust to outliers such as occlusion and make-up. These advancements significantly improved monocular model-based face reconstruction, making it more efficient and effective.
However, these methods are highly specialized and lack a deep understanding of the semantics of human faces or the ability to relate faces to language, limiting their overall scope and effectiveness.

### 2.2 Text-to-3D Face Generation and Manipulation

Text-to-3D face generation and manipulation methods aim to integrate textual information for creating and editing 3D faces.
Methods like Dreamface [] and Describe3D [] generate text-conditioned texture maps to render 3D morphable models (3DMM).
TG-3DFace [] advances this by using tri-plane neural representations and extending the 3D-aware GAN, EG3D [], for end-to-end text-conditioned generation. It incorporates a face parser and attribute classifier for fine-grained semantics and uses a camera pose conditional discriminator to guide learning. For text-guided 3D face manipulation, methods like Latent3D [] and ClipFace [] optimize intermediate layers with a CLIP-based loss to generate UV-texture maps or predict texture and expression latent codes.
These methods, however, rely on 3D scan data and to train new mappers for each text instruction.
Unlike these task-specific approaches, FaceGPT is a general-purpose model that reasons about 3D human faces from images, text, or both by leveraging general visual knowledge. Our model can interact with users through conversations, discussing facial features and providing relevant responses, while also being capable of following general user instructions.

### 2.3 Multimodal Large Language Models

Large Language Models (LLMs) are rapidly transforming various fields [, , , ]. While proprietary models like OpenAI’s ChatGPT [] and GPT-4 [] dominate the landscape, open-source alternatives such as Vicuna [], LLaMA [], and Alpaca [] support research efforts. However, LLMs mainly focus on generating text as output given text-only input. The integration of additional modalities into LLMs represents an active area of research.

Multi-Modal Large Language Models (MM-LLMs) are emerging, extending LLMs’ capabilities beyond text to encompass a broader spectrum of modalities, including images, videos, and audio. In the realm of image-text understanding, recent endeavors like LLaVA [] and MiniGPT-4 [] incorporate vision encoders to interpret images and align their features with language embeddings using projection layers. Moreover, cutting-edge models such as PandaGPT [], ImageBind [], and NeXT-GPT [] exhibit impressive versatility in handling diverse modalities, aligning embeddings from text, images, audio, and video with language as both input and output. To enhance LLM to natively output more modalities, approaches like LISA [] connects LLaVA with a decoder to generate text and segmentation masks, while ChatPose [] specializes in human pose information. However, these methods typically rely on supervised learning.

In our study, we explore the integration of human faces as a novel modality for LLMs. We assess (1) LLMs’ ability to generate 3D faces from text or image inputs and (2) their capability to understand human facial features and integrate this understanding into their broader functionality without supervision.

## 3 Method

Figure: Figure 2: Architecture of FaceGPT. Our model consists of a multi-modal LLM, which includes a vision encoder, a vision projection layer, and the LLM itself, along with a 3DMM projection layer, denoted as $\sigma$, and the parametric Basel face model []. During the training phase, we focus on training the $\sigma$ projection layer and fine-tuning the LLM, while keeping other components frozen. The network components are trained using a differentiable renderer to compute a self-supervised loss.
Refer to caption: extracted/5658670/figures/fig2.png

In this work, our aim is to augment existing Large Vision Language Models (VLMs) with the ability of reasoning about 3D human faces without requiring manual human efforts.
Inspired by established model-based face reconstruction methods, we treat the 3D human face as a separate modality within the LLM framework and represent it with 3D Morphable Model (3DMM) []parameters that represent the face shape, expression, albedo and illumination.
In particular, we introduce a <FACE> token into the token space of the LLM, which is mapped into the 3DMM parameter space and subsequently rendered into a 2D image, hence enabling self-supervised 3D facial reconstruction.

### 3.1 Model Architecture

Fig. 2 illustrates
the FaceGPT architecture.
The VLM has an extended vocabulary which includes a new token <FACE> that specifically represents the human face. Given an input image $x_{img}$ and/or text prompt $x_{txt}$, the VLM $f$ predicts text responses:

$$ ${y}_{txt}=f(x_{img},x_{txt}),$ (1) $$

where ${y}_{txt}=[t_{1},\dots,t_{N}]$ is the output sequence of tokens with corresponding hidden states $[h_{1},\dots,h_{N}]$.
If one of the output tokens $t_{n}\in y_{txt}$ is our newly defined <FACE> token, we can extract the hidden state as $h_{\texttt{<FACE>}}=h_{n}\in\mathbb{R}^{4096}$ and project it using an MLP $\sigma$ into the latent 3DMM parameters $\theta=\sigma(h_{\texttt{<FACE>}})=[\alpha,\delta,\gamma,\phi,c]\in\mathbb{R}^
{257}$,
i.e. the 3D face shape $\alpha\in\mathbb{R}^{80}$, facial expression parameters $\delta\in\mathbb{R}^{64}$ and texture $\gamma\in\mathbb{R}^{80}$ of a 3DMM, as well as the spherical harmonics illumination $\phi\in\mathbb{R}^{27}$ and camera parameters $c\in\mathbb{R}^{6}$ of the scene.
The 3D vertices and triangles, as well as the color of the face mesh are then determined using the standard 3DMM model $M(\theta)$ as described in [].
Using an orthographic camera model the reconstructed 3D face mesh can be rendered into 2D space using a differentiable renderer $\Pi$, hence producing the final reconstructed face image $\hat{y}_{rec}$. The process can be summarized as:

$$ $\displaystyle\theta$ $\displaystyle=\sigma(h_{\texttt{<FACE>}})$ (2a) $\displaystyle\hat{y}_{rec}$ $\displaystyle=\Pi(M(\theta))$ (2b) $$

Note that the FaceGPT architecture can be seen as a new type of language-based autoencoder, with a VLM as encoder and a computer graphics decoder that is based on the 3DMM.

### 3.2 Self-supervised Training

Our objective is to develop a VLM that is able to learn an enhanced comprehension of 3D human faces in a self-supervised manner.
Ultimately, the model should not only understand user instructions reliably, but also to accurately reconstruct 3D faces from visual or text input.
To accomplish this, we have devised a self-supervised approach that incorporates 3DMM understanding into existing VLMs.
This method allows us to leverage face data without the need for costly 3DMM annotations.
We also construct a text-to-3DMM dataset in an unsupervised manner that supports this training paradigm, enabling our model to learn these capabilities effectively and efficiently.

Training Objective. Inspired by the success of self-supervised learning in LLMs [], we augment the existing VLM training objective with a self-supervised face reconstruction loss $L_{face}$ that avoids the expensive 3D annotation of human faces in in-the-wild images via image-based reconstruction to regress 3DMM parameters:

$$ $L=\lambda_{txt}L_{txt}+L_{face}$ (3) $$

where $\lambda_{txt}$ and $\lambda_{face}$ are the weights for balancing the losses.

Text prediction loss.
We utilize the autoregressive objective $L_{txt}$ to guide the model to generate correct text output and hence retain its general capability to follow user instructions:

$$ $L_{txt}=CE(\hat{y}_{txt},y_{txt}),$ (4) $$

where $y_{txt}$ is the ground truth text response and $CE$ is the cross entropy loss.

Self-supervised face reconstruction loss.
To incorporate the 3DMM as a new modality into an existing VLM, we add a new token <FACE> in the vocabulary of the LLM and fine-tune the language modelling head.
For fine-tuning the VLM without relying on manually annotated data, we introduce a 2D self-supervised loss $L_{face}$, following established protocols of specialized face reconstruction models []:

$$ $L_{face}=\lambda_{pixel}L_{pixel}+\lambda_{per}L_{perc}+\lambda_{LM}L_{LM}+ \lambda_{reg}L_{reg},$ (5) $$

where $L_{pixel}=||A\odot(x_{img}-\hat{y}_{rec})||^{2}_{2}$ refers to the pixel-wise reconstruction loss between reconstructed images $\hat{y}_{rec}$ and the input images $x_{img}$.
To avoid distortions when training from in-the-wild data, a 2D skin mask $A$ is estimated from the input images using a simple pre-trained Gaussian mixture model for the skin color [].

The perceptual loss $L_{perc}=sim(f_{perc}(x_{img}),f_{perc}(\hat{y}_{rec}))$ estimates the cosine similarity between the reconstructed image $\hat{y}_{rec}$ and the input image $x_{img}$ at the perceptual level using a pre-trained feature extractor $f_{perc}$.

The landmark loss $L_{LM}=||LM_{img}-LM_{rec}||^{2}_{2}$ measures the L2 distance between the projected 2D landmarks of the estimated 3DMM $LM_{rec}$ and predicted 2D facial landmarks $LM_{img}$ using an off-the-shelf facial detector [].

$L_{reg}=||\theta||^{2}_{2}$ regularizes the estimated 3DMM parameters towards the mean value of the multi-variate Gaussian distribution of the 3DMM.

The weight parameters $\lambda_{face}=[\lambda_{pixel},\lambda_{per},\lambda_{LM},\lambda_{reg}]$ balance the respective losses, and we set them as established in prior work [].

Preserving the ability for natural conversations about faces. We noticed that VLM’s loose the ability to conduct general conversations about faces when trained self-supervised face reconstruction loss, and tend to always output 3DMM parameters when queried with a face image.
To resolve this problem, we generate a face conversation dataset with accurate textual face descriptions, by querying the VLM with face images and conversations that include multiple questions about facial attributes.
During training, we mix task-specific instructions that explicitly ask for 3DMM parameters with general conversational data to regularize the training and preserve the ability for general non-3DMM related conversations about faces.
In contrast to prior works [, ] which generally employ a simple static question-answer template that restrains the model from providing diverse responses, our proposed strategy can effectively improve the instruction following performance for general face conversations.

Self-supervised Text-to-3DMM loss.
We aim to enable the VLM to perform semantic face understanding tasks, i.e. to predict faithful 3DMM parameters when only having textual input $f(\cdot,x_{txt})$.
While related works for 3D human pose estimation [, ] rely on human-written fine-grained language descriptions of 3D human meshes, we aim to achieve text-to-3DMM generation without any human interaction.
To achieve this, we introduce a self-supervised text-based face reconstruction loss.
In particular, we first query the VLM with the face images in the training data and a question template that requests detailed face descriptions of a number of facial attributes such as head shape, skin tone, or facial expressions.
This results in a dataset with pairs of face images and detailed textual descriptions about facial attributes.
During training, we instruct the VLM to estimate 3DMM parameters only from the available detailed face descriptions $f(\cdot,x_{txt})$.
As we have the corresponding 2D image for each face description, we can re-use the self-supervised face reconstruction loss Eq. 5 to train the model at the task of text-to-3DMM generation, without the need for any human-written annotation.

## 4 Experiments

### 4.1 Training Data

Text-to-Face Data. The text-to-face dataset comprises pairs of text descriptions and face images, facilitating the development of mappings by VLM between textual descriptions and the 3DMM parameters of faces. During training, only text is used as input and the corresponding face images are only used to produce supervision signals. Given the absence of publicly available datasets linking text descriptions to 3D face meshes and existing VLMs like LLaVA present powerful Visual Question Answering(VQA) ability, we generate textual descriptions for face images. We employ templates structured as follows to guide the learning process in VLMs: "USER: {description}, can you give the 3DMM parameters of this person. ASSISTANT: Sure, it is <FACE>.", where {description} is the text description for faces. We utilize high-quality face images from CelebA-HQ [] and generate text descriptions using LLaVA-v1.5-13B [] to construct the dataset.

Image-to-Face Data. Image-to-face reconstruction data is composed of only human face images. The face images will be formatted with task-specific templates like "USER: <IMAGE> Can you give the 3DMM parameters of this person. ASSISTANT: Sure, it is <FACE>.". To enhance the diversity and relevance of the conversations centered around human face images, we also generate face-centric conversations for each face image. This approach enriches the contextual understanding of the VLM concerning the newly introduced token <FACE>. We adopt the CelebA-HQ trainset as the image to face reconstruction dataset.

Multimodal Instruction-Following Data. This data is general-purpose VQA data, and it is used to preserve the VLM’s ability of understanding a user’s instructions. Following LLaVA v1.5, We use LLaVA-v1.5-mix665k as multimodal instruction following data.

### 4.2 Experimental settings

Network Architecture. We build our model on Large Vision Language Model LLaVA-1.5-7B [] with CLIP-ViT-L-336px as vision encoder and Vicuna v1.5 as the LLM backbone. LoRA [] is applied to efficiently fine-tune the VLM. An MLP head with GeLU activations [] is appended to the last layer of the VLM to predict the 3D human face parameters.

3D Face Model. We use the Basel Face Model (BFM) 2017 [] as the 3D face model. The face is parameterized as the semantic code vector $\theta=[\alpha,\delta,\gamma,\phi,c]\in\mathbb{R}^{257}$ in [], which includes 3D shape parameters $\alpha\in\mathbb{R}^{80}$, facial expression parameters $\delta\in\mathbb{R}^{64}$, texture of 3DMM $\gamma\in\mathbb{R}^{80}$, illumination $\phi\in\mathbb{R}^{27}$ and camera parameters $c\in\mathbb{R}^{6}$.

LLaVa-Key baseline. As there are no VLM-based methods that can perform 3D face reconstruction, we introduce the LLaVa-Key. The model is finetuned to predict facial landmarks in the format of pure text given either images or textual description of human faces as input. For each input, gradient-based optimization is applied to these predicted landmarks to fit the 3DMM using Eq. 5 to obtain a 3D face reconstruction. It is important to note that LLaVa-Key employs test-time fine-tuning, making it an inherently different baseline compared to our approach and other task-specific methods, which rely solely on feed-forward prediction.
Nevertheless, we still use LLaVA-Key as a baseline to illustrate the effect of a direct 3DMM embedding in the token space of FaceGPT, which generally yields superior results, even without any test-time fine-tuning.

Implementation Details.
The training uses 8 NVIDIA 48G A40 GPUs. We utilize deepspeed [] engine and ZeRO optimizer [] for efficient training. We use AdamW [] optimizer with the learning rate and weight decay set to $2e-5$ and 0, respectively. We also follow the standard setting of using a WarmupDecayLR as the learning rate scheduler, where the warm-up iterations are set to 100. The weights of the text generation loss $\lambda_{txt}$ and the face reconstruction loss $\lambda_{face}$ are set to 1.0 and 0.1, respectively. And following [], those of the pixel loss $\lambda_{pixel}$, the perceptual loss $\lambda_{per}$, the landmark loss $\lambda_{LM}$, and the regularization loss $\lambda_{reg}$ are set to 0.5, 0.25, 5e-4 and 0.1, respectively. The landmarks of human faces are obtained with [] and we use pre-trained ArcFace [] to extract perceptual features. The batch size is set to 8 per GPU and the gradient accumulation step is set to 4.

Evaluation Metrics.
The instruction following ability is measured with GPT-assisted evaluation, which queries GPT4 to obtain the grading of generated responses. The 3D human face parameters estimation is measured with the L2 photometric error in RGB space and the L2 landmark error.

Figure: (a)
Refer to caption: extracted/5658670/images/face_rec_preds/11460.jpg

### 4.3 Image-based 3D Face Reconstruction

**Table 1: Performance at monocular 3D face reconstruction.**
| Method | photo | keypoint |
| --- | --- | --- |
| DECA [] | 0.216 | 5.2px |
| Deep3D [] | 0.073 | 3.2px |
| FOCUS [] | 0.077 | 2.2px |
| LLaVa-Key | 0.110 | 14.0px |
| FaceGPT | 0.106 | 3.0px |

We evaluate FaceGPT at image-based 3D face reconstruction and show a comparison with SOTA specialized monocular face reconstruction methods and a VLM-based generalist baseline LLaVa-Key in Table 1.

Moreover, Fig. 3 shows a qualitative comparison of FaceGPT and all baselines at the classic task of 3D face reconstruction from a single image.
As reported in prior works for VLM-based segmentation [] and human pose estimation [], our FaceGPT model does, as expected, not reach the state-of-the-art performance of specialized models.
However, we note that FaceGPT has a frozen vision encoder to preserve the generalist behavior, whereas all baseline models have a fine-tuned task-specific backbone.
Moreover, our model does outperform DECA, which is a highly competitive and widely applied baseline model [].
When compared to the VLM baseline model, FaceGPT achieves large improvements, highlighting the benefit of embedding the 3DMM parameters directly in the token space of the VLM.

### 4.4 Text-based 3D Face Reconstruction

Table 2 and Fig. 4 show the results of FaceGPT at text-based 3D face reconstruction. As there are no public unsupervised methods available that perform this task, we compare to a VLM-based baseline LLaVA-Key, where LLaVA is fine-tuned to predict the 2D coordinates of face landmarks given text description as input and an optimization-based method is utilized to fit a 3DMM on predicted landmarks.

**Table 2: Performance evaluation of text-based 3D face reconstruction (as L2 3DMM error).**
| Method | Exp. | Shape |
| --- | --- | --- |
| LLaVA-Key | 9.39 | 12.10 |
| FaceGPT | 2.89 | 4.04 |

We can clearly observe the significant advantage that FaceGPT achieves over the baseline method.
Moreover, the qualitative results demonstrate that the model can achieve faithful text-based 3D face reconstructions, despite being trained in a self-supervised manner.
These results demonstrate that embedding 3D knowledge about our world into a VLM is in principle possible without detailed human annotations, hence demonstrating the large potential of combining VLM’s with vision-as-inverse graphics for self-supervised learning.

Figure: Figure 4: Qualitative results of text-to -3DMM synthesis comparing the performance of the baseline VLM LLaVA-key and FaceGPT. Our method can reason and generate faithful 3D faces using only textual description. We also present the realistic face image which matches the description as a reference. A 3DMM reconstruction of reference images estimated by FOCUS method [] is also provided for reference. Our approach captures many facial details from the texts and produces much better visual results compared to the LLaVA-Key baseline method.
Refer to caption: x1.png

### 4.5 GPT-Assisted Evaluation

Table 5 shows that FaceGPT preserves the ability of instruction following by following LLaVA’s evaluation [] protocol, using GPT4-Assisted evaluation on LLaVA-Bench (COCO). FaceGPT compares favorably to LLaVA-v1.5-7B and reaches similar or better performance in the benchmark compared to VLMs with much more parameters. This performance advantage can be attributed to the specialized face training dataset and the usage of our face-centric conversation. These elements enhance FaceGPT’s proficiency in interpreting face-related language-based instructions, thereby improving its overall effectiveness in relevant tasks.

### 4.6 Ablation study

Influence of LLM on face understanding. Classical methods mainly rely on a vision encoder for regressing the 3DMM face parameters. In contrast, LLaVA utilizes a combination of frozen vision encoder and an LLM to perceive information from images. To study the effect of the LLM head, we train a baseline model with the vision encoder of LLaVA and an MLP to predict the human face parameters directly. The results are presented in Table 5 and show that the additional LLM helps in predicting better face parameters based on the visual representations from the frozen encoder.

Influence of face-centric conversation generation. To prove the necessity of our face-centric conversation generation strategy, we train a model only using a simple single-turn conversation template for face images, which is a common strategy used in the VLM-based image understanding works like LISA and ChatPose. The comparison results are demonstrated in Table 5. We observe that face-centric conversations help a lot in improving model’s ability in performing detailed description and complex reasoning. The face-centric conversations only have a small effect on the face reconstruction task, which is expected, as the generated conversations do not contain information about the 3DMM parameters of faces.

Comparison between supervised face loss and unsupervised face loss. Recent VLM-based works on segmentation and pose estimation generally rely on supervised learning with ground truth data or pseudo ground truth data. To study the influence of supervision on the model’s capability, we compare the model trained with supervised 3DMM losses and self-supervised 2D face losses. For the supervised 3DMM loss, we extract the 3DMM parameters on the same CelebA-HQ trainset with the state-of-the-art face reconstruction method FOCUS []. L1 loss is used for optimization on 3DMM ground truth. We observe in Table 6, that the model utilizing a supervised 3DMM loss outperforms its unsupervised counterpart, indicating that the performance ceiling is not reached yet and improvements on the self-supervised training could potentially lead to further performance gains.

**Table 3: GPT4-Assisted Evaluation on instruction-following capability. “Conv", “Details", and “Complex" correspond to three types of questions(conversation, detailed description, complex reasoning) produced by LLaVA’s data generation pipeline. GPT4 will be prompted to evaluate the answers from different models along with the ground truth answer produced by text-only GPT4. It would then give a score for each answer with an explanation. Previous method like ChatPose [] uses gpt-4-0314 for evaluation which is not accessible anymore. Thus, we provide the evaluation results with a newer gpt-4-0613.**
|  | Evaluation | Conv | Detail | Complex | All |
| --- | --- | --- | --- | --- | --- |
| LLaVA-V1.5-13B | gpt-4-0314 | 84.4 | 81.0 | 93.9 | 86.5 |
| ChatPose [] | 78.8 | 76.2 | 96.7 | 84.0 |  |
| LLaVA-V1.5-13B [] | gpt-4-0613 | 80.4 | 81.4 | 90.9 | 84.2 |
| LLaVA-V1.5-7B [] | 79.9 | 77.6 | 92.4 | 83.4 |  |
| FaceGPT | 79.6 | 81.5 | 92.6 | 84.6 |  |

**Table 6: Comparison when learning with supervised and unsupervised losses.**
| Method | photo. | keyp. |
| --- | --- | --- |
| FaceGPT(sup) | 0.092 | 2.1px |
| FaceGPT(unsup) | 0.106 | 3.0px |

## 5 Limitations and Future Work

While FaceGPT demonstrates remarkable results at reasoning about 3D faces from visual and text input with unsupervised learning, many open challenges remain.
Our work represents merely an initial step into this promising area, highlighting the vast potential for future exploration and advancement in this direction.
The model does not yet match the state-of-the-art performance of task-specific 3D face reconstruction methods, and one way to address this problem is to largely scale up the training to include various face databases.
The extension of FaceGPT to include arbitrary numbers of faces in an image is another interesting potential research direction.
Finally, the model is specific to faces and relies on the availability of a 3D morphable model for faces. A generalization to general objects would require the self-supervised learning to also include the generative object model parameters.

## 6 Conclusion

FaceGPT is the first self-supervised learning framework for Large Vision-Language Models to reason about 3D human faces.
We show that VLMs can learn to predict detailed 3D human faces from not only images, but also from textual inputs, in a fully self-supervised manner via inverse rendering.
As a generalist model, FaceGPT achieves strong results across various tasks, including traditional 3D face reconstruction, visual instruction following and text-based face reconstruction.
We believe our work also has general implications beyond face analytics, as it points towards a way forward to enable large multi-modal language-models to reason about our 3D world without supervision.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see weaknesses above. Moreover:

- Please provide a more comprehensive comparison with existing state-of-the-art methods for both 3D face reconstruction and text-driven face generation.
- What unique capabilities or insights does the language component provide that purely vision-based approaches lack? Please provide concrete examples of how the VLM enhances face understanding beyond existing methods.
- Do the authors believe that the generated faces of Fig. 3 accurately capture the input text descriptions? 
- What's the dataset used for the evaluation of Table 3? Are you comparing with SOTA? As said the results of Fig. 4 don't look very compelling as there are many details missing from the reconstructed faces and the identity is not well preserved.
soundness: 2
strengths: The combination of VLMs with face-related tasks has not been explored in literature and in its current instantiation in this paper presents some amount of novelty.  Moreover, training the VLM with a face reconstruction training objective in a self-supervised manner bears some degree of novelty.
summary: This paper describes a method where a VLM is trained with LORA to be adapted for the task of 3D face reconstruction. The VLM is supposed to provide textual information describing the face and in the end of the text a "face" token is used to predict 3DMM face parameters.
weaknesses: Unfortunately, I cannot grasp the motivation behind the proposed work as in the end of the end day it boils down how to fine-tune a VLM for 3D face reconstruction. But there are already several state-of-the-art methods of high accuracy for this task. Similarly there are already several methods for text-driven face generation. It's not clear if the proposed method is any better than methods tailored to these tasks. Importantly, these are vision tasks so it is unclear why a VLM is needed and what extra capabilities are brought into play by using for solving these tasks. The paper fails to demonstrate some newly introduced capability regarding the understanding of human faces that we have seen before. The speculative face generation task is poorly described and the evaluations do not make a lot of sense. This can be illustrated by looking at the results of Fig. 3. Clearly the method  has not really been trained successfully to produce high quality realistic faces corresponding to the textual descriptions used as input. Finally, even for face reconstruction the proposed method produces poor results as the visual results of Fig. 4 show.
Overall, the paper fails to show why VLM is needed for traditional 3D tasks, does not introduce any new capability and also fails to show decent results for the tasks it's evaluated for

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Choice of 3DMM Model:
Why does the framework utilize an older 3DMM model like BFM instead of more recent models that can capture finer facial details?

2. Reasoning Capabilities of VLMs:
Is there empirical evidence to support that VLMs possess the reasoning capabilities to accurately interpret human faces? If not, why prefer this framework over specialized existing frameworks designed for such tasks?

3. Reliability of VLM Outputs:
The framework presupposes that the VLM will consistently output the <FACE> token when analyzing a face. Are there instances where the VLM fails to produce a <FACE> token even when expected?

4. Verification of VLM-Generated Descriptions:
Is there a method to verify the accuracy of the descriptions generated by the VLM? [Ref. Lines 274-276]

5. Training Methodology:
The approach of using descriptions generated from VLM to re-train the VLM for estimating 3DMM parameters appears circular, akin to using knowledge distillation within the same model. Is there a more effective method to accomplish this?

6. Contribution of VLM to the Framework:
To what extent does the VLM contribute to the overall framework's effectiveness? Could similar results be achieved using simpler language models or the CLIP text encoder alone? [Ref. Lines 299-300]

7. Necessity of Detailed Descriptions:
In scenarios such as "Predict the face of a person who is excited about a surprise party", it seems that a simple description of the expression (e.g., "excited") might suffice. If a human would be asked to draw/imagine a face with this description, there are pretty good chances they will simply draw/imagine a face with "excited" expression on it. The additional narrative appears redundant. Do language models require this excess information to generate accurate facial expressions? Why do we really need the accompanying redundant information simply to generate a face with "excited" expression. I made the same observation in the Fig.3 examples where the faces only convey the main expression like "surprise", "lost", or "angry".

8. Modeling complex expressions:
Could the authors demonstrate complex expressions or combinations of expressions that existing models fail to capture to show the effectiveness of this framework?
soundness: 2
strengths: 1. The paper is well written and easy to follow.
2. The paper proposed a framework that can leverage large VLMs to generate 3D faces from natural description of emotions.
3. The framework doesn't require any coupled text and 3D face data.
4. The framework achieved good 3D face reconstruction results.
summary: The paper proposed a novel framework to use large VLMs to reason about 3D human faces from text/images by embedding the 3DMM face parameters into the token space. The framework is train in a self-supervised manner using image-based reconstruction and differentiable rendering. The authors claim that the proposed framework is able to leverage the existing work knowledge in VLMs to achieve semantic reasoning capabilities for 3D face generation.
weaknesses: 1. Insufficient Justification for Using VLMs:
The paper does not provide adequate justification for employing Visual Language Models (VLMs) in the 3D face synthesis task. The outcomes presented could potentially be replicated by many existing methods if trained under a conditional framework incorporating a CLIP text encoder along with detailed textual descriptions.

2. Subpar Quality of Generated Faces:
The quality of the generated faces significantly lags behind the current state-of-the-art face reconstruction methods. This is primarily attributed to the use of the outdated 3DMM model—BFM—which yields a linear texture and a very coarse mesh, limiting the detail and realism of the synthesized faces.

3. Lack of Standard Benchmarking:
It would be beneficial to evaluate the performance of this framework against standard 3D face reconstruction benchmarks, such as the Now and Realy3D datasets. Additionally, an analysis of the quality of the reconstructed mesh would provide a clearer picture of the framework's capabilities.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. What is the definition of self-supervised learning according to the authors, and how does it differ from conventional interpretations?
2. How does the paper's approach to training and data construction align with or deviate from traditional self-supervised learning methods?
3. Can the utilization of off-the-shelf models for generating 3DMM data and textual descriptions from 2D face images be considered a form of supervisory signal?
soundness: 2
strengths: This paper presents a valuable topic: constructing a unified model for generating 3D faces from both images and texts. Specifically, speculative face generation holds significant value in fields such as criminal tracking. The experiments also demonstrate the effectiveness of the constructed model in speculative face generation, explicit text-based 3D face generation, and image-based 3D face reconstruction.
summary: This paper fine-tunes a VLM (LLaVA-1.5-7B), unifying image-based 3D face reconstruction and language-based 3D face generation. Although the paper claims to be a self-supervised learning framework, the actual content indicates the use of supervision signals provided by off-the-shelf face reconstruction methods and VLM. It is effectively a supervised learning approach! Its loss function comprises two parts: the loss function for generating 3DMM output and the loss function for instruction-tuning.
weaknesses: The core idea of self-supervised learning is to set up proxy tasks that allow the model to train and capture the intrinsic structure and features of the data in the process. Although the paper claims to use a self-supervised learning framework, there seems to be some deviation from the conventional definition of self-supervised learning.
Based on the details of training and data construction in the paper, the method employed appears to be a straightforward supervised learning approach, similar to the instruction-based fine-tuning executed in papers like LLaVA. From the content on lines 193 and 236, it seems that the authors believe an algorithm can be considered self-supervised as long as it does not use manually annotated data. This perspective might reflect a different interpretation of the concept of self-supervised learning.
Although the paper does not introduce manually annotated data, it utilizes off-the-shelf face reconstruction methods and VLMs to construct 3DMM data and textual descriptions from 2D face images. This effectively means that off-the-shelf models are being used to provide supervisory signals for training.