HeadSetOff: Enabling Photorealistic Video Conferencing on Economical VR Headsets

Authors: yili  jin
xize  duan
fangxin  wang
xue  liu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 05:58:50

Abstract: Virtual Reality (VR) headsets have become increasingly popular for remote collaboration, but video conferencing poses challenges when the user's face is covered by the headset. Existing solutions have limitations in terms of accessibility. In this paper, we propose HeadsetOff, a novel system that achieves photorealistic video conferencing on economical VR headsets by leveraging voice-driven face reconstruction. HeadsetOff consists of three main components: a multimodal attention-based predictor, a generator, and an adaptive controller. The predictor effectively predicts user future behavior based on different modalities. The generator employs voice input, head motion, and eye blink to animate the human face. The adaptive controller dynamically selects the appropriate generator model based on the trade-off between video quality and delay, aiming to maximize Quality of Experience while minimizing latency. Experimental results demonstrate the effectiveness of HeadsetOff in achieving high-quality, low-latency video conferencing on economical VR headsets.

Main paper: 
 related work: 
 In this section, we present related works on traditional and VR video conferencing, along with talking head synthesis.Video conferencing has undergone a remarkable transformation since its inception in the 1960s. The early Picturephone [32] paved the way, but high costs hindered widespread adoption until the 1990s when affordable ISDN lines emerged [23] . IP-based solutions in the late 1990s revolutionized the industry, making video conferencing accessible to businesses and consumers alike [10] .Researchers focused on enhancing quality through networking [39, 47] and encoding techniques [25, 40] . The COVID-19 pandemic catalyzed an unprecedented surge in video conferencing usage across various domains [43] . However, the limitations of 2D screens have spurred interest in more immersive solutions like VR video conferencing to bridge the gap between remote and in-person interactions.In recent years, the evolution of 360°videos [21, 22] and volumetric videos [17, 18, 20, 26] has significantly heightened interest in VR video conferencing. Both researchers and companies are actively exploring diverse methods to enhance user experiences and tackle the challenges associated with VR headsets, which obscure the user's face. Two main technical solution routes have emerged: cartoon-style avatars and photorealistic reconstruction. Cartoon-style avatar solutions aim to create a virtual representation of the user using a cartoon-like avatar. VRChat [45] is a prominent example of this approach, allowing users to create and customize their avatars using a variety of tools and assets. The avatars are controlled by the user's movements and gestures, captured through VR controllers and tracking systems. Microsoft Mesh [29] is another platform that utilizes cartoon-style avatars for VR collaboration and communication. It leverages Microsoft's Mixed Reality technology to enable users to interact with virtual objects and environments alongside their avatars. These solutions provide a more immersive and engaging experience compared to traditional video conferencing, but may lack the personal connection and non-verbal cues that face-to-face interactions offer.Photorealistic reconstruction solutions, on the other hand, focus on reconstructing the user's real face and expressions under the VR headset. Apple Persona [4] is a notable example of this approach, using machine learning techniques to generate a realistic 3D model of the user's face based on information including LiDAR. The system captures the user's facial expressions and eye movements using cameras and sensors mounted on the VR headset, and maps them onto the reconstructed 3D model in real-time. Other researchers have explored similar techniques for photorealistic face reconstruction in VR. Thies et al. [42] proposed a face tracking and reconstruction method that uses an RGB-D camera to capture the user's face and a convolutional neural network to regress the facial parameters. Olszewski et al. [34] developed a system that combines a dynamic texture model with a deep appearance model to generate high-fidelity 3D face renderings in VR.The field of talking head video synthesis and manipulation has seen remarkable progress in recent years. Thies et al. [41] introduced a real-time technique for capturing and reenacting facial expressions from RGB videos, enabling convincing synthesis and manipulation of human face video content.Following this, researchers focused on enhancing various aspects of talking head synthesis. Zakharov et al. [48] few-shot learning approach using generative adversarial networks (GANs) [11] to create personalized talking head models from just a handful of frames. Deng et al. [38] proposed a GAN-based method to animate static portrait photos into dynamic videos. Wang et al. [46] introduced a high-quality neural talking head synthesis model with local free-view control that does not require 3D models.In more recent developments, researchers have harnessed the power of neural radiance fields (NeRF) [30] to achieve highly photorealistic results. Bai et al. [5] utilized a 3D-aware NeRF prior to reconstruct facial avatars from monocular video, demonstrating superior performance in novel view synthesis and face reenactment. Li et al. [24] proposed HiDe-NeRF, a subject-agnostic deformable NeRF that enables high-fidelity free-view talking head synthesis from a single shot. 
system overview: 
 Figure 1 shows the overview of HeadsetOff, and Figure 2 presents a demo. Our system consists of three main parts: predictor, generator, and controller.For the predictor, a multimodal attention-based network is utilized to predict the user's future behavior based on head motion (HM), eye blink (EB), voice (VO), and gaze direction (GD) modalities. The predictor aligns the dimensions of different modalities and encodes them using positional and temporal information. The encoded modalities are then processed by Modal Fusion Cores, which capture crossmodal and self-attention to generate fused multimodal vectors for the final prediction.For the generator, an initialization process is performed to extract facial attribute features, canonical keypoint landmarks, and a 3D Morphable Model (3DMM) from an input frame. During runtime, the generator employs voice input, head motion, and eye blink to animate the human face. The voice input is encoded and mapped to facial expression parameters, which are then integrated with the canonical keypoints. The face attribute feature is warped based on the updated keypoints, and the resulting representation is fed into a neural network to generate the output video chunk. The generator provides a range of models with varying quality levels to balance Quality of Experience (QoE) and latency.For the controller, an adaptive algorithm is employed to select the appropriate generator model based on the trade-off between video quality and delay. The controller operates on a set of available video quality levels and maintains a buffer to store generated video chunks. It aims to maximize QoE while minimizing latency by dynamically adjusting the model selection based on the current buffer level and the balance between quality and delay, controlled by a Lagrange multiplier. The controller updates the buffer level, selects the optimal video quality level, and generates the video chunk using the chosen model for each video chunk. 
predictor: 
 This section presents the predictor's overall structure. The multimodal sequences are initially processed by the Dimension Alignment and Modal Encoding modules. These components are responsible for incorporating positional information and ensuring dimensional compatibility. The processed sequences are then passed to the Modal Fusion Core, which captures multimodal attention. The following subsections provide an in-depth explanation of each module's architecture.Due to the dimensional mismatch between modalities, it is necessary to align them to a common dimension. As the user's behavior tends to exhibit continuity in movement, the temporal information within each modality plays a crucial role in prediction. To emphasize the temporal relationship of neighboring time step information in each modality, we employ a 1-D convolutional network. This process ensures that the different modalities are projected to the same dimension.The modality sequence possesses temporal characteristics. To enhance the temporal relationship between each time step 𝑡, we apply positional encoding (𝑃𝐸) to the modality sequence. Additionally, we introduce timestamp encoding (𝑇 𝐸) to further augment the modality sequence and better capture periodic information. The encoded modal sequence can be represented as: 𝐸 𝐻 𝑀,𝐸𝐵,𝑉 𝑂,𝐺𝐷 = 𝑀 𝐻 𝑀,𝐸𝐵,𝑉 𝑂,𝐺𝐷 + 𝑃𝐸 (𝑡) + 𝑇 𝐸 (𝑡).(EQUATIONThe encoded sequence is then forwarded to the modal fusion core.We employ the crossmodal attention mechanism to model the intrinsic relationships between different inputs. This subsection provides a detailed description of this process.To enhance understanding, let's discuss two distinct modalities: 𝛼 and 𝛽. The sequence vectors for these modalities are denoted as 𝑀 𝛼 and 𝑀 𝛽 , respectively.We aim to compute the crossmodal attention Φ 𝛽→𝛼 from modality 𝛽 to modality 𝛼. To facilitate this, we introduce three specific types of weight matrices: 𝑊 𝑄 𝛽→𝛼 , 𝑊 𝐾 𝛽→𝛼 , and 𝑊 𝑉 𝛽→𝛼 . These matrices project the input modality sequences into distinct representational subspaces.The Query vector 𝑄 𝛽→𝛼 is generated by applying the matrix 𝑊 𝑄 𝛽→𝛼 to the sequence 𝑀 𝛼 . This vector encapsulates the feature information of the 𝛼 modality.The Key vector 𝐾 𝛽→𝛼 is created by applying the matrix 𝑊 𝐾 𝛽→𝛼 to the sequence 𝑀 𝛽 . It plays a crucial role in weighting the attention mechanism across the different modalities.The Value vector 𝑉 𝛽→𝛼 is produced by applying the matrix 𝑊 𝑉 𝛽→𝛼 to 𝑀 𝛽 . This vector reflects the feature values of the 𝛽 modality, essential for the attention process.We use the Query vector 𝑄 𝛽→𝛼 , Key vector 𝐾 𝛽→𝛼 , and Value vector 𝑉 𝛽→𝛼 to perform the "Scaled Dot-Product. " The dot product of the Query vector and the Key vector is divided by √︁ 𝑑 𝑘 . It then passes through a softmax function and transforms into an activated sequence. The product of the activated sequence and the Value vector yields the crossmodal attention score Φ 𝛽→𝛼 . The detailed calculation formula is as follows:EQUATIONThe value of each entry in Φ 𝛽→𝛼 represents the attention score of an entry at a specific position in the 𝛼 modality sequence to a specific entry in the 𝛽 modality. For example, the 𝑖-th time step of Φ 𝛽→𝛼 is the product result of the Value vector 𝑉 𝛽→𝛼 and the 𝑖-th row in the weight matrix 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (.).Based on the crossmodal attention mechanism, we create four distinct Modal Fusion Cores for HM, EB, VO, and GD modalities. Each Modal Fusion Core includes crossmodal attention blocks from the three other modalities to the host modality and self-attention blocks based on the standard self-attention mechanism [44] . For instance, the EB modality's Modal Fusion Core calculates three crossmodal attention vectors Ψ 𝐻 𝑀→𝐸𝐵 , Ψ 𝑉 𝑂→𝐸𝐵 , Ψ 𝐺𝐷→𝐸𝐵 , and a self-attention vector Ω 𝐸𝐵→𝐸𝐵 . It then fuses these four vectors into a single fused multimodal vector 𝐹 𝐸𝐵 using a learned weight matrix 𝑊 𝑐𝑜𝑛𝑐𝑎𝑡𝑒𝑛𝑎𝑡𝑒 . The crossmodal attention block is composed of 𝐷 multi-head crossmodal attention layers, representing the attention perspective from multiple dimensions. Each input sequence to the crossmodal attention block passes through a normalization layer. The output vector from the multi-head crossmodal attention layer is processed feed-forwardly by a feed-forward layer. The self-attention block The fused multimodal vectors 𝐹 𝐻 𝑀 , 𝐹 𝐸𝐵 , 𝐹 𝑉 𝑂 , and 𝐹 𝐺𝐷 obtained from the Modal Fusion Cores are concatenated and passed through a fully connected layer to generate the final prediction. 
generator: 
 The generator, as illustrated in Fig. 3 , commences by receiving an initialization image as input to extract its features. During runtime, it employs voice, head motion, and eye blink to animate the human face. Additionally, we provide a range of models with varying quality levels, enabling the controller to optimize the balance between Quality of Experience (QoE) and latency.To initialize the generator, we first require a photo of the user. This initialization process is performed only once, prior to the video conferencing session. We begin by extracting three distinct feature categories from the input image:• Facial Attribute Feature: The primary feature we strive to extract encompasses the presence of a human face, including characteristics such as skin tone and eye color. This feature facilitates the identification and preservation of the individual's facial attributes in the synthesized view. • Canonical Keypoint Landmarks: These landmarks serve the purpose of representing the unique geometric properties of an individual's face in a neutral pose and expression. By extracting these landmarks, we can model and transform the subject's facial structure in the generated view. • 3D Morphable Model (3DMM) [6] : The 3DMM is a statistical model that represents the 3D shape and texture of human faces. By fitting the 3DMM to the input frame, we can estimate the 3D shape, pose, and expression of the face. These three features will be concurrently extracted by three dedicated networks: Facial Attribute Extractor, Canonical Keypoint Detector, and 3DMM Encoder. For the 3DMM, we leveraged the pretrained model from [8] . The design of the other two networks is described as follows: Facial Attribute Extractor: To enhance the representation of the face and its attributes, we employ an attribute feature extraction network. This network plays a pivotal role in transforming the input frame into an attribute feature representation. The primary objective of this network is to extract intricate facial features from the input frame. It accomplishes this by utilizing a sequence of downsampling blocks that strategically leverage convolution layers to convert the initial 2D features into a comprehensive representation. Subsequently, a series of residual blocks is employed to meticulously compute the final features, which encapsulate the essence of the frame's three-dimensional structure. Canonical Keypoint Detector: Drawing inspiration from prior works [9, 12, 46] , we introduce a canonical keypoint landmark detector to capture the distinctive geometric characteristics of an individual's face in a neutral pose and expression. The detector adopts a U-Net-like [36] encoder-decoder architecture. The encoder employs several convolutional layers to encode the input frame into a latent representation. Notably, rather than projecting the encoded features into a 2D space, we expand them into 3D using a 1 × 1 convolutional bottleneck layer. This enables us to extract 3D canonical keypoint landmarks instead of solely 2D landmarks. The decoder mirrors the encoder with a series of upsampling 3D convolutional layers. At each upsampling step, the resolution of the features is increased while incorporating higher-level semantic information from the corresponding encoder layer via skip connections. This facilitates the recovery of the original spatial dimensions for landmark prediction. Finally, the output of the last decoder layer generates the predicted coordinates of the canonical keypoint landmarks.As inspired by [55] , constructing model capable of generating precise facial expression from voice input presents a formidable challenge due to the inherently complex nature of mapping audio to expressions, which varies considerably across individual identities. So we incorporate 3DMM parameters into the voice encoder.Once the voice input is encoded into facial expression parameters, we integrate elements of the user's head movements and eye blinks. This comprehensive mapping results in a more nuanced and lifelike representation of facial expressions that are synchronized with the voice input.After obtaining the aforementioned features, we proceed to generate the final output video chunk through the following steps: Canonical Keypoints Fusion: The adjusted face expression is integrated with the canonical keypoints by concatenating them together. This fusion process allows for the combination of the structural information captured by the keypoints, creating a comprehensive representation that encapsulates both the facial structure and the desired expression. The fused representation combines the canonical keypoints and the adjusted face expression, enabling a holistic representation of the facial characteristics. Face Attribute Feature Transformation: Subsequently, we apply warping to the face attribute feature, utilizing the updated canonical keypoints as a guide. This warping step ensures that the extracted facial attributes are appropriately transformed and aligned with the adjusted face expression, preserving the spatial relationships between the attributes and the facial structure. The warping function, guided by the fused representation, transforms the face attribute feature to align it with the desired facial expression.Output Processing: Finally, the warped face attribute feature, enhanced with the adjusted canonical keypoints, is fed into a neural network. This network utilizes the amalgamated information to produce the output. It is important to note that this step is the most computationally intensive. To address this, we train models with varying quality levels; a high-level model with more parameters can generate high-quality video but with a longer processing delay. The choice of which model to use is determined by the controller. 
controller: 
 In this section, we present our Controller to select the appropriate generator model based on the trade-off between video quality and delay. The goal is to maximize the Quality of Experience (QoE) for the viewer while minimizing the latency in video generation.The controller aims to optimize the QoE by selecting the most suitable video quality level for each chunk, considering the tradeoff between quality and delay. The objective can be expressed as: maxEQUATIONwhere 𝑖 𝑘 is the selected video quality level for chunk 𝑘, 𝑞 𝑖 𝑘 is the corresponding quality score, 𝑑 𝑖 𝑘 is the delay for generating the video at level 𝑖 𝑘 , 𝐵(𝑡 𝑘 ) is the current buffer level at time 𝑡 𝑘 , 𝜆 is the Lagrange multiplier, and 𝛾 is a utility function parameter controlling the trade-off between quality and delay.The objective function is subject to the following constraints:EQUATION) where 𝐵 max is the maximum buffer size, and Δ𝑡 is the time elapsed since the last decision.The Controller operates on a set of available video quality levels, each associated with a quality score and a generation delay. As general video conferencing system, it maintains a buffer to store generated video chunks and aims to keep the buffer level close to a maximum threshold. The model selection is adapted based on the current buffer level and the balance between quality and delay, controlled by a Lagrange multiplier. The Controller follows these steps for each video chunk 𝑘 at time 𝑡 𝑘 :(1) Calculate the time elapsed since the last decision, Δ𝑡.( EQUATION𝑖 𝑘 ← 𝑎𝑟𝑔𝑚𝑎𝑥 𝑖 ∈𝑉 𝑞 𝑖 -𝜆 • 𝑑 𝑖 𝐵 (𝑡 𝑘 ) 𝛾 13: 𝜆 ← 𝜆 -𝛽 • (𝐵(𝑡 𝑘 ) -𝐵 max ) 14: 𝑡 last ← 𝑡 𝑘 15:Generate video chunk 𝑘 using model with quality 𝑖 𝑘 16: end for (6) Generate video chunk 𝑘 using the model with the selected quality level 𝑖 𝑘 .The Controller provides an adaptive approach to select the appropriate video generation model based on the current buffer level and the balance between video quality and delay. By dynamically adjusting the model selection, the algorithm aims to optimize the QoE for the viewer while managing the latency in video generation. The Lagrange multiplier serves as a control parameter to maintain a stable buffer level close to the maximum threshold. 
experiment: 
 This section presents experiments conducted with HeadsetOff, including a user study of the entire system and detailed evaluations of each module.Quantitatively evaluating the entire HeadsetOff system is challenging. To qualitatively assess the overall Quality of Experience (QoE) provided by HeadsetOff, we conducted a user study involving 30 participants. We randomly divided the participants into 15 pairs and asked them to engage in two video conferencing sessions. In the first session, each pair used a traditional video conferencing system for approximately 3 minutes. Subsequently, they engaged in another 3-minute session using HeadsetOff.The participants found that HeadsetOff could reconstruct the human face well, providing an experience similar to traditional 2D video conferencing with a direct camera feed. The majority of the participants expressed a positive impression of HeadsetOff, noting that the photorealistic reconstruction of facial features significantly enhanced the sense of presence and immersion. They also appreciated the smooth and lifelike animations of facial expressions driven by voice input, head motion, and eye blinks. Several participants commented that the experience felt engaging and personal, closely resembling face-to-face interactions.However, a few participants mentioned that in certain instances, the reconstructed faces appeared unnatural or exaggerated. They suggested that further refinements to the voice-to-expression mapping algorithm could improve the realism of the generated animations, especially for extreme facial expressions or rapid changes in emotion.Despite these minor limitations, the overall qualitative evaluation indicated that HeadsetOff delivers a highly immersive and engaging video conferencing experience on economical VR headsets. The photorealistic facial reconstruction and responsive animations were identified as the key strengths of the system, contributing to an enhanced sense of presence and enabling more natural communication between participants.To evaluate the performance of our multimodal attention-based predictor, we conducted experiments using a dataset collected from 30 participants. The dataset consists of head motion (HM), eye blink (EB), voice (VO), and gaze direction (GD) sequences recorded during video conferencing sessions.We compared our predictor with the following baselines:• LSTM: A long short-term memory (LSTM) network [15] that processes the concatenated multimodal sequences. • Transformer: A standard Transformer model [44] that uses self-attention to capture the dependencies within the multimodal sequences. We evaluated the predictors using two metrics: Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). MAE measures the average absolute difference between the predicted and groundtruth values, while RMSE emphasizes larger errors by taking the square root of the mean squared error. Lower values of MAE and RMSE indicate better prediction performance.Table 1 presents the prediction results on the test set. Our predictor achieves the lowest MAE and RMSE values among all the compared methods, demonstrating its effectiveness in predicting future user behavior based on multimodal sequences. The LSTM and Transformer baselines perform relatively well but fail to capture the intricate relationships between different modalities. Our predictor's ability to model the interactions between modalities and capture the dependencies within each modality contributes to its superior performance.These experimental results demonstrate the effectiveness of our multimodal attention-based predictor in capturing the complex relationships between different modalities and predicting future user behavior. The predictor's ability to anticipate user actions enables HeadsetOff to generate future video chunks in advance, significantly reducing the perceived delay during video conferencing.To evaluate the performance of our generator, we collected normal talking head videos from video conferences, along with the corresponding voice, head motion, eye blink, and gaze direction data. We used this information as input and the collected videos as ground truth. We then compared the reconstruction accuracy between the generated and original videos. We used several metrics to assess the quality and realism of the generated videos:• Frechet Inception Distance (FID) [14] : FID measures the similarity between the distributions of generated and real images. We compared our generator with a state-of-the-art audio-driven talking head synthesis method, SadTalker [55] . Table 2 presents the evaluation results.Our generator achieves comparable performance to SadTalker, indicating similar levels of realism in the generated videos. It is important to note that our generator utilizes additional input modalities, such as head motion and eye blink, which contribute to more accurate facial expression reconstruction.These experimental results demonstrate the effectiveness of our generator in producing high-quality, photorealistic videos by leveraging voice, head motion, and eye blink information. The ability to generate videos at different quality levels allows our system to adapt to various network conditions and user preferences, ensuring a smooth and engaging video conferencing experience.To evaluate the performance of our controller, we conducted experiments using a network simulator that models a range of network conditions. We utilized the FCC dataset [1] , which comprises millions of mobile throughput internet traces.We compared our controller with the following baselines:• Fixed Bitrate (FBR): A simple approach that selects a fixed video quality level, regardless of network conditions. • Buffer-Based (BB) [19] : A method that selects the video quality level based on the current buffer level, aiming to maintain a stable buffer.We evaluated the controllers using two metrics: Average Video Quality (AVQ) and Rebuffering Ratio (RR). AVQ measures the average quality level (from 1 to 5) of the generated videos throughout the session, while RR represents the percentage of time spent rebuffering due to insufficient buffer. Higher AVQ and lower RR indicate better performance.Table 3 presents the evaluation results. Our controller achieves the highest AVQ and lowest RR among the compared methods, demonstrating its effectiveness in adapting to varying network conditions. The FBR approach suffers from frequent rebuffering events, as it cannot adapt to changes in network bandwidth and latency. The BB method maintains a more stable buffer but often selects lower quality levels to avoid rebuffering.To further analyze the behavior of our controller, we investigated its performance under different network conditions. Table 4 shows the AVQ and RR for three representative network scenarios: low bandwidth (0.5-1.5 Mbps), medium bandwidth (1.5-3.0 Mbps), and high bandwidth (3.0-5.0 Mbps). As expected, the controller selects higher quality levels and achieves lower rebuffering ratios when the network bandwidth is higher. In the low bandwidth scenario, the controller prioritizes maintaining a stable buffer by selecting lower quality levels, resulting in a slightly higher RR. These results demonstrate the controller's ability to adapt to different network conditions and make appropriate quality level decisions.These experimental results demonstrate the effectiveness of our controller in maximizing the QoE while minimizing latency in video generation. By dynamically adapting the video quality level based on network conditions and the trade-off between quality and delay, the controller ensures a smooth and engaging video conferencing experience across various network scenarios. 
discussion: 
 The advancements presented in this paper have significant implications for the accessibility and adoption of immersive remote collaboration solutions. By eliminating the need for expensive, highprecision hardware, HeadsetOff lowers the barrier to entry for photorealistic VR video conferencing. This could lead to wider adoption across various domains, such as business meetings, remote education, and social interactions. The enhanced sense of presence and engagement provided by HeadsetOff has the potential to greatly improve the quality of remote communication and collaboration.However, there are several limitations and opportunities for future work that should be acknowledged. While the user study indicated a generally positive reception of HeadsetOff, some participants noted instances of slightly unnatural or exaggerated facial expressions. Further refinements to the voice-to-expression mapping algorithm [2] could improve the realism of the generated animations, especially for extreme or rapidly changing emotions. Incorporating additional modalities, such as electromyogram (EMG) signals [33] , may provide a richer input representation for more accurate expression synthesis.Another limitation is the inability to capture silent, subtle movements of the face with economical headsets due to the lack of sensors. This gap underscores an opportunity for future research. Developing methods to accurately record these nuanced expressions could dramatically improve the realism and expressiveness of generated animations. Enhancing cheap sensor or devising new software algorithms to better interpret limited data could be potential approaches to address this issue.The personalization of the HeadsetOff to individual users is another area for future exploration. Currently, the generator is trained on a diverse dataset to handle a wide range of face types and expressions. However, adapting the models to specific users through fine-tuning or few-shot learning techniques could potentially enhance the realism and consistency of the generated animations. Personalized models [27, 52, 53] may better capture the unique facial characteristics and mannerisms of each user, further improving the overall video conferencing experience.The scalability and robustness of HeadsetOff in real-world scenarios also warrant further investigation. While the experiments were conducted in controlled settings, the system should be tested under diverse network conditions [49, 50, 54] and with a larger user base [37, 51] to assess its performance and identify potential challenges. Strategies for handling network fluctuations, packet loss, and other impairments should be developed to ensure a seamless user experience.Lastly, the ethical implications of photorealistic face animation should be carefully considered [13] . While HeadsetOff aims to enhance remote communication, it is crucial to establish guidelines and safeguards to prevent misuse or deception. Users should be made aware of the capabilities and limitations of the system, and their consent should be obtained for the use of their likeness in virtual environments. Additionally, given the risk of misunderstandings caused by exaggerated, missing, or incorrect facial expressions, it is clear that any commercial system must undergo a more systematic evaluation. This evaluation should include a systematic analysis of user preferences for different options, recognizing that perfect accuracy may not always be achievable. Ongoing research and public discourse on the responsible development and deployment of such technologies are necessary to address these concerns and ensure the ethical use of photorealistic VR video conferencing. 
conclusion: 
 In this paper, we presented HeadsetOff, a novel system that achieves photorealistic video conferencing on economical VR headsets by leveraging voice-driven face reconstruction. Our system comprises three main components: a multimodal attention-based predictor, a generator that employs voice input, head motion, and eye blink to animate the human face, and an adaptive controller that dynamically selects the appropriate generator model based on the trade-off between video quality and delay. Through extensive experiments, we demonstrated the effectiveness of HeadsetOff in generating high-quality, photorealistic video conferencing experiences while minimizing latency. Our contributions pave the way for more accessible and immersive remote collaboration solutions, enabling seamless interactions in virtual environments without the need for expensive, high-precision hardware. 
: 
 Apple Vision Pro [3] , Meta Quest [28] , and HTC VIVE [16] , these devices are no longer just for immersive entertainment experiences; they are becoming integral tools for remote collaboration. This leap forward presents a wealth of opportunities as well as a unique set of challenges, particularly in the realm of video conferencing, which is a crucial function in remote working. A significant issue arises when a user wears a VR headset, which obscures the upper half of the face. In practical VR applications, users typically lack an external camera to capture the lower half of the face, necessitating reliance solely on the data collected by the headset itself.To address this problem, two main technical solution routes have emerged: cartoon-style avatars and photorealistic reconstruction. Cartoon-style avatar solutions, such as VRChat [45] and Microsoft Mesh [29] , involve creating a cartoon-like avatar that users can control, similar to playing a video game. On the other hand, photorealistic reconstruction solutions, like Apple Persona [4] , attempt to reconstruct the real human face hidden under the VR headset, providing an experience akin to traditional 2D video conferencing.However, photorealistic reconstruction solutions typically demand high-precision hardware, such as those found in the Apple Vision Pro, which features a sophisticated sensor array that includes, but is not limited to, LiDAR, Time-of-Flight (ToF) cameras, and infrared cameras. The device is also equipped with a specially designed R1 chip to process the input from these sensors. As a result, these solutions are often less accessible to the broader public due to the complexity and cost of the required hardware.So there is an interesting and meaningful task at hand: can we achieve photorealistic video conferencing on an economical VR headset? To accomplish this, we must address two main challenges: 1) How can we reconstruct the lower half of the face without using high-precision sensors to capture this information? 2) How can we decrease the delay in processing, given that video conferencing is a delay-sensitive task?To tackle the first challenge, inspired by recent developments in voice-driven talking head synthesis, we propose using voice to reconstruct the lower half of the face. By leveraging the user's voice input, we can generate a realistic representation of the mouth and jaw movements, eliminating the need for high-precision sensors. For the second challenge, we propose a two-fold approach. Firstly, we suggest predicting user behavior, including voice and head motion, for the next video chunk. By generating the future video chunk in advance, we can significantly decrease the perceived delay during video conferencing. Secondly, we propose jointly considering network adaptive bitrate with reconstruction. Traditional approaches generate the highest quality video and then send lowerquality versions based on network conditions. Instead, we recommend preparing several models with different quality levels and choosing the corresponding model based on the current network environment. This approach ensures that the video quality is optimized for the available bandwidth while also reducing the impact of reconstruction delay.We propose HeadsetOff, which consists of three main parts: predictor, generator, and controller. The predictor utilizes a multimodal attention-based network to predict the user's future behavior based on head motion, eye blink, voice, and gaze direction modalities. The generator employs voice input, head motion, and eye blink to animate the human face, providing a range of models with varying quality levels. The controller uses an adaptive algorithm to select the appropriate generator model based on the trade-off between video quality and delay, aiming to maximize Quality of Experience while minimizing latency by dynamically adjusting the model selection based on the current buffer level and the balance between quality and delay.Experiments show that our predictor achieves superior performance compared to baseline methods in predicting future user behavior, with the lowest Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) values. The generator produces high-quality, photorealistic videos by leveraging voice, head motion, and eye blink information, achieving comparable results to state-of-the-art methods in terms of Frechet Inception Distance (FID), Cumulative Probability Blur Detection (CPBD), Cosine Similarity (CSIM), and Lip Sync Evaluation (LSE) metrics. Furthermore, our adaptive controller successfully maximizes the QoE while minimizing latency by dynamically selecting the appropriate generator model based on network conditions, outperforming fixed bitrate and buffer-based approaches in terms of Average Video Quality (AVQ) and Rebuffering Ratio (RR). A user study involving 30 participants confirms the effectiveness of HeadsetOff in delivering an immersive and engaging video conferencing experience on economical VR headsets, with participants praising the photorealistic facial reconstruction and responsive animations.The main contributions of this paper are as follows:• We propose a novel system, HeadsetOff, that achieves photorealistic video conferencing on economical VR headsets by leveraging voice-driven face reconstruction. • We introduce a multimodal attention-based predictor that effectively predicts user future behavior. • We present a generator that employs voice input, head motion, and eye blink to animate the human face. • We develop an adaptive controller that dynamically selects the appropriate generator model based on the trade-off between video quality and delay. 
 
 Appendix:  


Decision: The paper was generally appreciated by the 4 reviewers. The detailed system description (R2) is clear and well written (R3). The paper - and thus overall system described - is considered to hold potential (R1) addressing a practical issue of reconstructing the otherwise invisible face from parsons wearing an HMD. The face reconstruction method is multimodal (R3, R4), considering head and eye motion to animate the user's jaw and mouth movement (R4). As such it is stated that it may contribute to promoting VR technology for daily life (R3). Addressing also low-delay transmission of the reconstructed face - in a network-adaptive manner - is thought to make HeadsOff a practical framework (R4). Here, the attention-based predictor is considered a further positive, predicting user behavior to enhance realism and responsiveness (R4). The several (R1) experiments for evaluating each part (R2) of the system and the qualitative user test are considered as another strong point (all reviewers), showing "high quality low-latency video conferencing on economic VR headsets" (R4).

The authors have provided a detailed and quite convincing rebuttal, indicating to include how they address the points of criticism in the rebuttal also into the camera-ready (CR) paper. The average rating has improved from 4.25 to 4.5 based on the constructive rebuttal, appreciated by 3 of the 4 reviewers.

Criticisms included some missing information on the baseline model (R1). Further, the imbalance of describing SoA on videoconferencing (VC) vs. talking head synthesis, considered the main contribution of the paper (R1). Here, also, some clarification wrt. reconstructing the lower vs. upper parts of the face were mentioned (R3). Constructive considerations are contained in rebuttal and indicated to be included in CR paper.
More details are requested for the figures describing the system, which have been indicated to be provided in the CR version. Further comparisons to baseline models are included in rebuttal, too (R1). While individual components are considered insufficient (R1), the authors highlight the abilities of the overall system, which appears convincing - also not criticized by other reviewers. Regarding the concern that generator and predictor should be evaluated separately in an ablation study (R1), the authors indicate that they carried this out. Here, though, it is not clear whether this new information will be included in the CR paper. 
The wish fort further example images was mentioned by several reviewers (R1, R2), and respective examples are indicated in the rebuttal, to be included in the CR paper.
Points of criticism regarding the "Average Video Quality" construct (R3) are addressed in the rebuttal as well, again to be included in a final revision. Concerns wrt. real-time operation (R3) are addressed by the predictor, as outlined in rebuttal. Here, I recommend to the authors to also clarify the descriptions in a possible CR paper version, to avoid this concern. 
The very justified question by R3 regarding the possible superiority of the approach in comparison with classical VC is well addressed in the rebuttal, but here, too, I recommend the authors to clarify this also in the CR paper. IMO, the points raised by R3 in point 5 are still valid, considering a 2D vs. 3D scenario, and should be addressed in revision. 
Some limitations will remain and may better be even more clearly be presented in the paper. An important one is that of not being able to capture silent (emotional) movements of the face with economical headsets, as no sensors exist for that. Even though somewhat mentioned, in the CR version this may be discussed more clearly as "avenue for future research" as indicated in the rebuttal.
In addition, the considerations about the initial photo quality should be included in the CR paper as well. The authors mention an additional experiment in the rebuttal, which could be included in CR if feasible.
Due to the potential of misunderstandings and the like produced by exaggerated or possibly missing or wrong expressions, it is evident that any commercial system would need to undergo a much more systematic evaluation, which however cannot be expected from such scientific paper. Here, besides the mentioned future pathways, also a systematic analysis of preferences for different options should be considered, especially if perfect accuracy cannot be achieved. This pertains to the general need for evaluation methods of generative AI.

Overall, the paper appears to be a solid contribution, hence I recommend to accept is as oral presentation.

Reviews: review: paper_summary: paper_summary: The paper presents HeadSetOff, a novel system that achieves photorealistic video conferencing on economical VR headsets by leveraging voice-driven face reconstruction. Experiments also demonstrate the effectiveness of HeadSetOff in generating high-quality, photorealistic video conferencing experiences while minimizing latency.
strength_weakness: strength_weakness: This paper intends to deal with a practical issue in VR, which is caused by the fact that the user’s face is covered by the headset. I believe the paper holds potential, and the problem it addresses is relevant to MM. Several experiments were conducted to evaluate the HeadSetOff system.
limitations: limitations: Overall, some critical information is missing, and the rationality of the compared baseline model needs to be further explained and improved. My detailed comments are as follows,

1. This paper devotes a considerable length to discussing related work about video conferencing, while providing insufficient discussion about “Talking Head Synthesis” (Section 2.3), which is more relevant to this paper. 

2. Figure 2 illustrates the overview of the proposed predictor. However, it is more like a simple diagram that can not demonstrate the details of the model structure. Moreover, figure 3 is in a similar situation.

3. This paper chooses LSTM and Transformer as two baselines, which are pretty old and not convincing. Some new sequence predicting methods should be considered, such as TimesNet and Autoformer. Same as baseline selections in Section 7.3 and Section 7.4.

4. This paper proposes a HeadSetOff system consisting of three main components. However, each individual module appears slightly insufficient.

5. This paper lacks an ablation study to prove the effectiveness of the designs of the predictor and the generator. Besides, it lacks a visual presentation of the head synthesis.

6.  Page 7, Section 7.4. Do the settings of three different bandwidth values have any basis?

Minor comments
1. The fonts of annotations on some of the figures are inconsistent. For example, the “Head motion”, “Eye blink” in Figure 1 and Figure 2.

2. The annotation (“Model Selection”) on the arrows in Figure 3 is out of bounds. Please re-design this figure and make it decent.

3. The layout of the paper needs to be optimized, as there are excessive gaps between some Sections and paragraphs. 

4. Some sentences are a bit colloquial. For example, “So there is an interesting and meaningful task at hand” on Page 1, Line 77.
paper_summary: paper_summary: In this paper, the authors present a new system for photorealistic VR communication, while wearing an HMD.
Fot that they introduced a system that is split up into 3 major parts, a predictor, a generator and a controler.
Every single part is described in depth, in that paper so that it is clear what they are talking about.
The big goal is a reconstruction of a human face while the person is wearing an HMD, so that the other end of the vr video-conferencing call can still see the face and emotions.
strength_weakness: strength_weakness: This paper is very detailed and describes all the important parts in depth, so that the reader should be able to follow the progress of the paper. Each part (predictor, generator, and controller) is described and evaluated individually to show that the authors thought about every possible point, where errors could occure.

Furthermore, a qualitative test was performed and evaluated in this paper as well, to show that the HeadSetOff also works with real Users and not only theoretically. 

The authors also discuss the limitations and future steps, which shows, that they know, that there is still work that needs to be done.
limitations: limitations: I would have liked some additional pictures of the subjective test, or atleast a reconstruction of a face, so that I could at least imagine what to expect, when using HeadSetOff.
paper_summary: paper_summary: This paper proposes a videoconferencing method that can be applied to economical VR headsets, which has some advantages over existing mature but expensive products. In the paper, the overall system design is described in detail and sufficient experiments are also carried out to demonstrate the superiority of the proposed system.
strength_weakness: strength_weakness: In this paper, HeadSetoff is proposed as a feasible videoconferencing method via an economical VR headset. The method fully integrates multimodal information and is also expected to further promote the application of VR technology in daily life. The overall system composition is clear and well-organized. The experiments in this work are also relatively sufficient to demonstrate the superiority of the system.
limitations: limitations: Despite the novelty of this paper, there are still some confusions, which I hope the author can answer:

1) Through the first two paragraphs of the introduction, I feel that the difficulty of VR videoconferencing is reconstructing the upper half of the face, due to the occlusion of the VR headset. And around L80 it is mentioned that the difficulty is reconstructing the lower half of the face. Is this a contradiction?
2) According to 1), combined with Fig. 3 and related content, I think there is no distinction between the upper and lower half of the face in Generator, but it is directly driven by the character picture and multimodal information.
3) Average Video Quality (AVQ) is mentioned at L795, but I missed the specific Video Quality Assessment Methods. May I know whether PSNR, SSIM, VMAF or other video quality assessment algorithms are used here?
4) According to my experience, Sadtalker in Table 3, although it is faster in face driver, it is far from real-time. Can Generator in the article do the driving more quickly to enable video conferencing in real-time?
5) Based on the experimental setup in Sec. 7.1, I did not experience a significant superiority of video conferencing using VR headsets compared to traditional 2D video conferencing. Since the text does not explicitly mention whether the final presentation is in 2D or 3D, I divide it into 5-a) and 5-b) to discuss them separately.
5-a) If the Generator's driving method is a 2D driving method and does not provide a free point of view for 3D content, how does the advantage of videoconferencing via a VR headset manifest itself when participating in a 2D videoconference in a bare-eye situation still achieves the same, if not better, results?
5-b) If the Generator's driving method is a 3D driving method, which can have a certain range of viewpoints, then when the character's head moves (e.g., turning the head from left to right, looking up and down), it is obvious that the character image used for initialization cannot cover the full range, and therefore the distortion generated during the synthesis needs to be further discussed.

Overall, I think 5) is the most important of these. Although the authors may have described it in Sec 2.2, I still hope that the authors can provide relevant screenshots or effects to make the readers feel more advanced.
paper_summary: paper_summary: This paper proposed a deep learning-based photorealistic human face animation framework for video conferencing on VR headsets. The framework first predicts the user’s future facial behavior based on their head motion, eye blink, gaze direction and voice. Then a generator animates the mouth and jaw movements of the picture of the user’s face. Finally, a controller algorithm selects the optimal version of the generator to maximize the quality of the human face rendering while minimizing the video conference latency.
strength_weakness: strength_weakness: -the author designed a multimodal face reconstruction scheme that considers head motion, eye motion and voice to vividly animate the user’s jaw and mouth movement.

-the author also tackled the latency issue of transmitting the reconstructed face through video conferencing, making the HeadSetOff a practical framework.

-the adaptive controller dynamically selects the appropriate generator model based on network conditions, ensuring optimal video quality and reduced delay.

-the system's multimodal attention-based predictor effectively anticipates user behavior, enhancing the realism and responsiveness of the animation.

-experimental results show the system achieves high-quality, low-latency video conferencing on economical VR headsets, making advanced VR communication more accessible.
limitations: limitations: -this framework mainly covers jaw and mouth movements that come with sound (e.g., talking). There are movements that are silent, such as smiling. Without recognizing these movements, the reconstructed face is less vivid.

-a prerequisite for the system is a photo of the user’s face, and the quality of this photo (e.g., angle, luminance) can affect animation performance. The evaluation would be more comprehensive if it included the impact of these photo variations.

-the system might produce slightly unnatural or exaggerated facial expressions in some instances, particularly with extreme or rapidly changing emotions.

-personalization to individual users is limited; fine-tuning or few-shot learning techniques could enhance the realism and consistency of the animations by better capturing unique facial characteristics.