Subjective and Objective Quality-of-Experience Assessment for 3D Talking Heads

Authors: yingjie  zhou
zicheng  zhang
wei  sun
xiaohong  liu
xiongkuo  min
guangtao  zhai
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 16:58:16

Abstract: In recent years, immersive communication has emerged as a compelling alternative to traditional video communication methods. One prospective avenue for immersive communication involves augmenting the user's immersive experience through the transmission of three-dimensional (3D) talking heads (THs).  However, transmitting 3D THs poses significant challenges due to its complex and voluminous nature, often leading to pronounced distortion and a compromised user experience. Addressing this challenge, we introduce the 3D Talking Heads Quality Assessment (THQA-3D) dataset, comprising 1,000 sets of distorted and 50 original TH mesh sequences (MSs), to facilitate quality assessment in 3D TH transmission. A subjective experiment, characterized by a novel interactive approach, is conducted with recruited participants to assess the quality of MSs in THQA-3D dataset. Leveraging this dataset, we also propose a multimodal Quality-of-Experience (QoE) method incorporating a Large Quality Model (LQM). This method involves frontal projection of MSs and subsequent rendering into videos, with quality assessment facilitated by the LQM and a variable-length video memory filter (VVMF). Additionally, tone-lip coherence and silence detection techniques are employed to characterize audio-visual coherence in 3D MS streams. Experimental evaluation demonstrates the proposed method's superiority, achieving state-of-the-art performance on the THQA-3D dataset and competitiveness on other QoE datasets. Both the THQA-3D dataset and the QoE model have been publicly released at https://github.com/zyj-2000/THQA-3D.

Main paper: 
 introduction: 
 After years of persistent efforts, significant advancements have been achieved in image and video-related domains [17, 28, 35] . Video communication, offering not only audio transmission but also real-time visual presentation, has substantially enhanced user experiences during communication, emerging as a pivotal achievement in societal interaction. Nevertheless, traditional video communication is constrained by viewpoint limitations, restricting the visual information conveyed to users. Consequently, the growing demand for immersive communication methods underscores the need for advancements in communication and media technologies. Within this context, three-dimensional (3D) talking head (TH) sequences are considered as a viable and promising technical solution to further enhance the immersion and interactivity of face-toface communications. 3D THs offer users a wider array of viewing angles compared to traditional video communication, fostering a more immersive communication experience. However, the inherent voluminous nature of 3D THs necessitates artificial compression before transmission. Furthermore, the real-time dynamics of the transmission channel introduce delays, latency, and code stream transformations, exacerbating the challenges associated with 3D TH transmission. Both artificial processing and channel dynamics impact the communication experience of users significantly. Consequently, effective assessment and monitoring of 3D TH communication quality at user terminals hold paramount importance for optimizing communication system designs and enhancing user experiences.However, conducting related quality assessment tasks poses significant challenges. On one hand, the production and collection of 3D THs entail substantial time and financial resources, resulting in few relevant datasets being available and impeding further development. On the other hand, the intricate data structures and large Table 1 : The comparison of 3D quality assessment databases and proposed database.Type Models Distortions Description BASICS [1] PCQA 1,494 GPCC, VPCC, GeoCNN [31] Humans, Animals, Architectures, Landscapes WPC [21] PCQA 740 GPCC, VPCC, Gaussian noise, Downsampling Fruit, Vegetables, Tools SJTU-PCQA [49] PCQA 420 Octree, Downsampling, Noise Humans, Statues LS-PCQA [23] PCQA 1,080 Downsampling, Noise, GPCC, VPCC Animals, Humans, Vehicles, Daily objects DHH-QA [63] MQA 1,540 Noise, JPEG, Downsampling, Quantization Scanned Real Human Heads DDHQA [62] MQA 800 Noise, JPEG, Downsampling, Quantization, Motion Distortions Dynamic 3D Digital Human SJTU-H3D [58] MQA 1,120 Noise, JPEG, Downsampling, Quantization Static 3D Digital Humans 6G-DTQA [66] MQA (QoE) 400 JPEG, Downsampling, Quantization, Stall, Rebuffer Digital Twins Transmitted Under 6G Networks THQA-3D (Proposed) MQA (QoE) 1,000 Quantization, Stall, Rebuffer, Conversion, Synchronization Scanned Real Human Heads data volumes characterizing 3D data impose rigorous demands on existing communication systems, computational processing infrastructure, and algorithmic frameworks.In this study, we address the challenge of data acquisition by leveraging existing resources. Specifically, we utilize 10 TH models sourced from the MultiFace dataset [48] introduced by Meta, employing subsequent processing to generate 1,000 distorted and 50 original mesh sequences (MSs). This endeavor culminates in the establishment of the 3D Talking Heads Quality Assessment (THQA-3D) dataset. Subsequently, we conduct a series of subjective and objective experiments on THQA-3D dataset for further research and analysis. The subjective evaluation entails the participation of volunteers in a well-organized manner, where the mean opinion score (MOS) serves as the definitive metric for assessing MSs' quality. On the other hand, we introduce a projection-based Quality-of-Experience (QoE) approach for the objective evaluation, simplifying the processing and quality assessment of mesh streams. Furthermore, the proposed method augments multimodal quality sensing capabilities through the incorporation of tone-lip coherence features and a large quality model (LQM). Experimental results demonstrate the superior performance of the proposed method compared to existing objective evaluation techniques on the THQA-3D dataset and representative QoE dataset. Consequently, the contributions of this paper are threefold:• The THQA-3D dataset is established for quality assessment for the 3D THs. This dataset simulates 7 common types of distortion during 3D streaming media transmission, providing a total of 1000 distorted MSs. Besides, the 50 original MSs are also provided in the proposed dataset. • A user-interactable subjective quality evaluation method is designed for 3D streaming media containing audio. Under the condition of using a 2D monitor, the user can shift the viewpoint to observe the 3D streaming media by using the mouse. • A multimodal QoE method assisted by LQM is proposed. In addition, the variable-length video memory filter (VVMF) proposed in this paper can effectively improve the visual perception of 3D MSs. 
related work 2.1 video communication: 
 In recent decades, advancements in communication technologies have significantly facilitated both industrial production and daily life activities. Particularly noteworthy is the refinement of video communication technology, which has enabled seamless real-time interactions, ranging from individual video calls to group web conferences. However, despite these advancements, certain limitations inherent to video communication have become increasingly apparent. Primarily, the 2D nature of video media fails to capture the depth and 3D information present in face-to-face interactions, thereby limiting the fidelity of visual experiences during video communication. Additionally, the constrained viewpoint of video capture devices restricts users to passive reception of content from specific angles, precluding the provision of truly immersive communication experiences. Furthermore, the limited interactivity options offered by traditional video communication platforms further diminish user engagement and satisfaction levels. Consequently, contemporary communication expectations extend beyond real-time interaction, with users now seeking more immersive and interactive experiences.In response to these evolving demands, immersive real-time video communication has emerged as a prominent trend in the field [12, 29, 50] . Although still in its nascent stage, immersive video communication is commonly associated with the transmission of 3D data, representing a departure from conventional 2D video streams. Within this context, two prevalent 3D data structures, namely point clouds and meshes, hold particular promise for immersive video applications. While point clouds offer a simpler data representation using discrete points to convey the position and color of 3D objects [55, 57, 65, 70] , meshes provide a more detailed characterization, delineating each polygonal facet through vertices, edges, and normals [5, 64, 69] . Consequently, meshes are chosen in this study as the preferred data structure for modeling 3D THs, enabling a more accurate depiction of facial features and expressions.The exploration of 3D data has emerged as a focal point of contemporary research [2, 39, 53, 59] . High-fidelity 3D data not only enhances subjective visual experiences for users but also facilitates computational processing in critical downstream tasks such as 3D detection and segmentation. Consequently, the evaluation of 3D data quality has become imperative. Presently, quality assessment methodologies for 3D data are categorized based on the type of data, including point cloud quality assessment (PCQA) [54, 55, 57, 60, 65] and mesh quality assessment (MQA) [61, 64, 69] . Table 1 summarizes relevant datasets in this domain, yielding several notable observations. Firstly, both point clouds and meshes are susceptible to various quality degradations during transmission and processing. Notably, noise, compression, and downsampling are prevalent distortions encountered in point clouds, whereas meshes exhibit a broader spectrum of distortion types. Besides, existing quality assessment datasets predominantly focus on single static point The Small Boy Put the worm on the hook clouds or meshes, with limited consideration given to dynamic 3D sequences.On the other hand, while many streaming media datasets and QoE quality evaluation algorithms have been proposed, QoE quality evaluation for 3D streaming media has received little attention. Although Zhang et al. introduced the 6G-DTQA dataset [66] for dynamic digital twin QoE assessment under 6G network communication conditions, its scope remains constrained, featuring a small number of personas and a limited dataset volume. Moreover, the synthetic nature of the dynamic digital twin images, generated using artificial intelligence (AI), introduces discrepancies compared to real captured character dynamics. Notably, crucial distortion types such as code-stream transformation and audio-visual synchronization are overlooked in the 6G-DTQA dataset, despite their substantial impact on user experience. In contrast, the proposed THQA-3D dataset presented in this study leverages real-time dynamic THs captured through sensors, comprehensively simulating major distortion types affecting QoE. This includes quantization distortion, jamming, buffering, code-stream transformations, and audio-video synchronization. Through comparative analysis with existing datasets, the THQA-3D dataset emerges as a comprehensive resource for guiding QoE quality assessment in 3D data transmission scenarios. 
database construction 3.1 source models collection: 
 The THQA-3D dataset proposed in this study draws its original TH models from the MultiFace dataset [48] introduced by Meta. Thus, a succinct overview of the MultiFace dataset is provided as follows. The MultiFace dataset comprises 13 distinct identities, each represented by high-quality textured mesh heads. Utilizing a multi-camera capture studio named Mugsy, facial expressions and details are captured from multiple perspectives at a frame rate of 30 frames per second (FPS). Notably, MultiFace offers highfidelity streams of MSs and texture streams, each texture with a resolution of 1024×1024, for the 13 identities. These sequences depict individuals narrating 50 sentences of speech, accompanied by corresponding audio files. As depicted in Fig. 1 , ten individuals (comprising 5 males and 5 females) are selected as raw data to construct the THQA-3D dataset. It is pertinent to mention that each talking head model is characterized by 6,172 vertices and 12,294 facets. Furthermore, for each person, as delineated in Table 2 , five fixed voices are chosen from a total of 50 voices for the QoE quality assessment study. To ensure the representativeness of the selected voices, a phonological analysis is conducted on the 50 selected voices. Specifically, the cepstrum method is employed to estimate formant frequencies for each speech, the results of which are depicted in Fig 2 . The analysis reveals several key insights: Firstly, the first formant frequencies of the 50 speech sounds range from 750Hz to 1050Hz, while the second formant frequencies range from 2750Hz to 3450Hz, indicating comprehensive distribution across both formant frequencies and rich phonological features. Moreover, across different subjects, the speech of the same sentence exhibits greater similarity in formant frequencies, reflecting a certain degree of pronunciation consistency. However, variations in pronunciation consistency are observed across different sentences, indicative of individual idiosyncrasies in pronunciation habits. Overall, the 50 selected speech sounds exhibit rich phonological features, encompassing individual differences while maintaining consistency in word pronunciation, thus affirming their representativeness.As indicated in Table 3 and Fig. 3 , during the establishment of the THQA-3D dataset, we systematically consider potential distortions affecting the MSs from four distinct perspectives: namely, the continuity of the MSs, the different encoding parameters, the random change of the code stream, and the audio-visual consistency.Prior research [16, 30, 36] has elucidated buffering and lagging as prominent factors impacting the quality of video streaming. Consequently, these distortions are equally pertinent in modeling discontinuities within 3D MSs. Specifically, to simulate initial buffering effects, a buffering time parameter 𝑇 𝑟 ∈ [1, 2] is introduced, indicating the duration of buffering preceding the onset of the 3D MSs. Subsequently, the first mesh model is replicated 𝑇 𝑟 × 𝐹 𝑃𝑆 times to emulate the buffering period. Similarly, for stuck incidents, a parameter 𝑇 𝑠 ∈ [1, 2] denotes the duration of stuck episodes. Given that the shortest speech segment lasts only 3 seconds, all stuck events are introduced after the initial 2 seconds to ensure uniform addition. Thus, mesh models at the 2-second mark are replicated 𝑇 𝑠 × 𝐹 𝑃𝑆 times and inserted following the 2-second mark to simulate stuck effects consistently.Parameters & Code Rate. Similar to conventional communication systems, the encoding and decoding processes are indispensable for transmitting 3D MSs, inevitably subjecting 3D contents to varying degrees of distortion. Notably, the choice of encoding parameters holds significant sway over user experience, particularly in the case of mesh representations. To explore this phenomenon, this study selects two common distortions of coding parameters: position quantization and texture coordinate quantization. These distortions are simulated using the Draco 1 , generating four distinct levels of coding parameter distortion. Specifically, the position quantization level is set as 𝑞 𝑝 ∈ [6, 7, 8, 9] while the texture quantization level is set as 𝑞 𝑡 ∈ [4, 5, 6, 7] . Besides, in most adaptive code rate systems, the transmitting end adjusts coding parameters in real time based on the system's instantaneous state, resulting in fluctuations in code rates. To capture these dynamics, this study models two variations: code rate boost and code rate drop. Initially, a coding parameter configuration of 𝑞 𝑝 = 8 and 𝑞 𝑡 = 6 serves as the baseline code rate. Combining the two types of coding parameters, In the absence of a standardized protocol for 3D MSs transmission, it is imperative to delineate a communication model tailored to 3D MSs. Leveraging insights from conventional video transmission paradigms, we conceptualize each TH model as an individual 3D frame as shown in Fig. 4 . Each 3D frame comprises a mesh, housing geometric information exclusively, and a texture map component, encapsulating texture and color. The sequential arrangement of these 3D frames imparts temporal continuity to the otherwise static mesh. Notably, for 3D MSs incorporating audio, such as THs, inclusion of an audio file containing voice information alongside the corresponding 3D frames is necessitated. Consequently, the mesh, texture map, and audio file are treated as distinct entities, subject to separate compression and transmission processes during communication transfer. To this end, Google's Draco is employed for mesh compression, while JPEG [24, 44] serves as the compression algorithm for texture maps, and MP3 [3] is utilized for audio compression.On the receiving end, the transmitted stream of 3D meshes can be reconstructed through decoding of the 3D frames and audio files.The data listed in Table 4 illustrates the number of encoded 3D frames for each subject when speaking different sentences. Notably, we adhere to the original parameter configuration outlined in Multiface [48] , maintaining 30 FPS during the transmission of 3D mesh streams. As a consequence, the number of 3D frames in this dataset is contingent solely upon the duration of speech segments.Given the unique characteristics of mesh streaming, a multimodal 3D streaming subjective experiment is devised, adhering to established experimental methodologies in terms of experimental settings and equipment. The subjective assessment is conducted in a meticulously controlled laboratory environment, following the guidelines outlined in ITU-R BT.500-13 [4] . An iMac monitor supporting a resolution of 4096 × 2304 pixels is utilized for the presentation of MSs, while wired headphones of superior sound quality are employed for audio perception, ensuring both a serene testing atmosphere and real-time audio. Specifically, as illustrated in the Fig. 5 , a total of 1,000 groups of distorted MSs are imported into Blender2 using the Stop-motion-OBJ plugin3 in advance. Additionally, the corresponding audio files are integrated into the Blender via added speakers, resulting in 1,000 Blender project files housing complete MSs. Subsequently, 26 male and 24 female participants are recruited to take part in the subjective quality assessment experiment. Different Blender projects are randomly accessed and showcased on the monitor, affording participants the freedom to manipulate viewing angles using the mouse, thereby observing the flow of MSs from diverse perspectives. Moreover, participants are afforded the flexibility to adjust timestamps for playback of the mesh stream or halt at specific 3D frames for detailed scrutiny. Throughout the playback session, audio playback is facilitated through wired headphones. Ultimately, participants are tasked with evaluating the quality of each mesh stream in tandem with audio-visual perception. It is worthwhile to additionally note that this subjective experiment uses Absolute Category Rating (ACR) for quality assessment.To mitigate potential discomfort [8, 37, 67] arising from prolonged exposure to 3D content, the experiment is divided into 10 phases, each comprising 100 MSs. Participants are restricted to one phase per day to prevent overexertion. Consequently, a total of 50,000 = 50 × 1,000 subjective ratings are collected upon conclusion of the experiment.Based on previous work [56, 58, 62, 63, 66, 68] , the commonly used z-scores are computed through the collection of subjective ratings. This process can be represented as:EQUATIONwhere 𝑟 𝑖 𝑗 denotes the quality rating provided by the 𝑖-th subject on the 𝑗-th 3D TH stream,𝜇 𝑖 = 1 𝑁 𝑖 𝑁 𝑖 𝑗=1 𝑟 𝑖 𝑗 , 𝜎 𝑖 = √︃ 1 𝑁 𝑖 -1 𝑁 𝑖 𝑗=1 𝑟 𝑖 𝑗 -𝜇 𝑖 ,and 𝑁 𝑖 is the number of 3D TH MSs assessed by subject 𝑖. Furthermore, quality ratings from unreliable subjects are discarded following the subject rejection procedure recommended in [4] . Subsequently, the obtained z-scores undergo linear rescaling to the range [0, 5]. The MOS of 3D TH stream 𝑗 is then computed by averaging the rescaled z-scores.To provide a more visual representation of the subjective experiment outcomes, a histogram is generated for the processed z-scores, as depicted in Fig. 6(a) . Furthermore, to delve deeper into the influence of various factors such as the 3D head models, selected speech, and different distortion types on subjective human perception, Fig. 6(b-d  
proposed qoe model: 
 In this section, a novel QoE approach is proposed and details are given. The comprehensive model framework is illustrated in Fig. 7 , encompassing three primary components: video quality assessment aided by the Large Quality Model (LQM), detection of audio-video consistency, and identification of discontinuities.Firstly, a 3D TH composed of mesh can be represented according to prior research [69] as:EQUATIONwhere 𝑉 , 𝑁𝑉 , 𝐸 represent the sets of vertices, normal vectors, and edges of the mesh, respectively. Furthermore, a set of 3D TH MSs can be denoted as:EQUATIONwhere 𝑇 𝑑 denotes the duration of the MS. As observed from the equations above, 𝑀𝑆 constitutes a chronological stream of MSs composed of 𝑀 𝑖 . Given the complex data structure of the mesh, direct computational operations on a stream of 3D MSs entail significant overhead. Utilizing a projection method, as demonstrated in prior studies [63, 69] , offers a viable alternative. These research also indicates that the frontal aspect of a face encompasses rich facial details and expressive movements. Thus, by rendering a frontal projection of a 3D TH, it is possible to preserve the effective information contained within the 3D mesh. Consequently, the center position 𝐶 can be calculated for each TH:EQUATIONwhere 𝑣 𝑖 represents the position of the 𝑖th vertex comprising the mesh, and 𝑁 𝑣 denotes the total number of vertices contained within a mesh. Utilizing 𝐶, a 3D coordinate system can be established with the frontal face direction serving as the x-axis. Subsequently, a virtual camera is positioned at a suitable location along the x-axis to capture successive frontal projections within a mesh sequence, which are then consolidated into a single video file. Finally, audio is integrated to generate the complete video of the projected TH.The overall process can be described as:EQUATIONwhere 𝐹𝑝 (•) signifies the process of front projection for each 3D frame, 𝐴 denotes the corresponding audio, ⊕ indicates the concatenation of audio and video, and 𝑉 2𝐷 denotes the resultant 2D projected video with accompanying speech obtained through rendering.With the increasing popularity and efficacy of large vision models (LVMs) in computer vision, these models are perceived to possess a comprehensive understanding of images surpassing that of traditional depth models. Consequently, in this study, a Large Quality Model (LQM) is employed for quality perception of video frames. Specifically, Q-Align [47] is chosen as the LQM for image quality perception. This paper selects the extracted video frames and presents them to Q-Align with the prompt as illustrated in Fig. 7 . Q-Align then returns the probability distribution across five grades: "excellent", "good", "fair", "poor", and "bad". The overall process is articulated as follows:EQUATIONwhere 𝑆 (•) signifies the frame extraction process, 𝐼 𝑚 𝑠 represents the 𝑚th extracted frame sensed by the LQM to obtain the probability vector 𝑃 𝑚 𝑐 encompassing five categorical probabilities. Subsequently, a fixed weight 𝑤 𝑐 is applied to the probability vector to assess the quality of the extracted video frame:EQUATIONwhere 𝑄 𝐼𝑠 𝑚 denotes the predicted quality for the extracted frame, and • denotes the vector inner product. This process is iterated until all the extracted frames in the video are evaluated, resulting in a sequence of assessed frame qualities denoted as 𝑄 𝑉 𝑠 . To better align with human subjective perception during video viewing, a variablelength video memory filter (VVMF) incorporating an Hermann Ebbinghaus forgetting curve [10] is employed to process the quality of each frame in 𝑄 𝑉 𝑠 . Specifically, the simplest exponential decay is sampled discretely according to the length 𝐿 of 𝑄 𝑉 𝑠 as:EQUATIONwhere 𝑛 = 0, 1, ..., 𝐿 -1 denotes the discrete sampling time. Additionally, to constrain the filtering result, 𝑤 𝑓 (𝑛) must satisfy the following condition:EQUATIONFinally, the filtering process of the VVMF is employed to derive the video quality 𝑞 1 using operations similar to a linear filter:EQUATIONThis equation can be equivalently understood as a sampling of the linear convolution of 𝑄 𝑛 𝐼𝑠 and 𝑤 𝑓 (𝑛) at the (𝐿 -1) moment, expressed as 𝑞 1 = [𝑄 𝑛 𝐼𝑠 * 𝑤 𝑓 (𝑛)]𝛿 (𝑛 -(𝐿 -1)).The consistency between audio and video, particularly concerning mouth movements and speech, significantly influences the user's audiovisual experience. Hence, detecting synchronization between audio and video is crucial. In this study, we employ SyncNet [6] , a classical feature extractor, for audio-lip consistency detection. Specifically, SyncNet conducts mouth cropping on the rendered frontal video and performs Mel Frequency Cepstral Coefficient (MFCC) feature extraction on the audio. Subsequently, consistency metrics are computed utilizing a siamese network. Within this component, the audio-video bias and confidence scores serve as features 𝑞 2 and 𝑞 3 characterizing audio-lip consistency.During streaming, buffering or stuck events often manifest as silent intervals in the audio signal, with its amplitude remaining at zero. This temporal pattern in the audio signal is distinctive and observable. Hence, by analyzing short-time energy and short-time over-zero rate metrics of the audio signal and setting suitable thresholds, silent intervals can be identified. Compared to traditional video-level discontinuity detection methods, this approach is more straightforward and intuitive. The onset and duration of these silent intervals are recorded as 𝑞 4 and 𝑞 5 , respectively, effectively characterizing buffering or stuck during media transmission.Following the extraction of five features from the 3D TH frontal projection video, support vector regression (SVR) is employed for final quality assessment to streamline the algorithmic process and enhance efficiency. It is pertinent to note that the SVR model utilized in this method is based on the scikit-learn package in Python, employing a radial basis function. 
experiments 5.1 experiment details: 
 To validate the efficacy of the proposed Quality of Experience (QoE) model, 23 representative quality assessment methods are selected as competing algorithms. These methodologies encompass image quality assessment (IQA), video quality assessment (VQA), point cloud quality assessment (PCQA), and quality of service assessment (QoS) methods. Notably, all evaluation methods across these categories assess video quality, except for PCQA, which evaluates quality after converting meshes into point clouds. Furthermore, based on the presence or absence of reference information, the evaluation methods utilized in this experiment can be classified into full reference (FR) and no reference (NR) quality evaluation methods.All competing algorithms are evaluated on both the proposed THQA-3D dataset and the classic Waterloo-III dataset [9] . For data partitioning, a five-fold cross-validation strategy is employed. Besides, there is no overlap in the content of all divided folds. Particularly for the THQA-3D dataset, it is important to note that Q-Align is fine-tuned with low-rank decomposition (LoRA) [14] using video frames within the coded parameter-distorted videos in each training set. However, for the Waterloo-III dataset, given the absence of audio in the Waterloo-III dataset, audio-lip consistency is replaced with video bitrate, which is a common feature in traditional QoE methods. Furthermore, experiments regarding buffering and stuck characteristics are conducted directly using the provided dataset information. Additionally, the Q-Align is fine-tuned with LoRA using all video frames in each training set. During performance testing on both datasets, the average performance over five iterations is computed to represent the algorithm's performance. Finally, it is worth stating that all methods use the source code provided by the authors and keep the original parameter settings.In order to quantify the performance of each algorithm, four widely used performance metrics are utilized, namely Spearman Rank Correlation Coefficient (SRCC), Kendall's Rank Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Squared Error (RMSE). SRCC and KRCC primarily measure the monotonicity of prediction quality, while PLCC assesses the linearity and consistency of the algorithm's prediction quality. RMSE is employed to evaluate the accuracy of the prediction results. When SRCC, PLCC and KRCC approach 1, and RMSE approaches 0, it indicates excellent performance of the algorithm.The experimental results of the performance tests are presented in Table 5 . By analyzing Table 5 , the following conclusions can be drawn: 1) The proposed algorithm achieves state-of-the-art performance on the THQA-3D dataset, significantly outperforming the other algorithms involved in the experiments.2) The performance of the proposed method on the Waterloo-III dataset remains competitive. However, considering the specific experimental setup on the Waterloo-III dataset, the results simply demonstrate that the large model-assisted QoE approach remains effective for classical QoE datasets. 3) Comparison of the experimental results on the THQA-3D and Waterloo-III datasets shows that some of the methods involved in the comparison are not stable, and even PXNR shows a complete failure result. This also highlights that the proposed method has stability. 4) The representative methods selected in the experiments exhibit inadequate performance in handling the challenge of streaming 3D TH MSs. This is mainly attributed to the insensitivity of traditional IQA, VQA, PCQA, and QoS methods to distortions caused by audio modalities, despite their comprehensive assessment of stream quality from various perspectives.To further explore the individual contributions of each module, ablation experiments are conducted on the THQA-3D dataset. The results of these experiments are presented in Table 6 . By analyzing Table 6 , the following conclusions can be drawn: 1) Large model-assisted video quality sensing, audio-lip consistency detection, speech-based buffering and stuck detection, and VVMF all positively contribute to the overall algorithm performance, indicating the effectiveness of all modules. 2) From an importance standpoint, video perception is the most crucial, followed by lip-sound consistency, while buffering and stuck detection play a relatively minor role. This is due to the intuitive nature of visual information acquisition and its dominance in human sensory perception. Additionally, humans are highly sensitive to audio-visual asynchrony, while buffering and lagging phenomena indirectly affect viewer emotions. Thus, the proposed algorithm aligns with the principles of human subjective perception. 3) Analyzing the image quality level, the performance of the large model-assisted quality perception module surpasses that of existing NR IQA methods, indicating the large model's ability to perceive more in-depth image quality features and laying a solid foundation for overall video quality perception. 
conclusions: 
 This paper contributes to the field of immersive video communication by introducing the 3D Talking Heads Quality Assessment (THQA-3D) Dataset. This dataset encapsulates seven types of distortions commonly encountered in 3D streaming communication, comprising 1,000 distorted 3D mesh sequence (MS) streams alongside 50 original counterparts. Furthermore, a tailored subjective experimental methodology is proposed for the quality assessment of 3D MSs. This approach involves importing received 3D MSs and speech audio into Blender, enabling subjects to assess quality from various viewpoints and evaluate the overall audiovisual experience subjectively. Moreover, we present a large model-assisted multimodal fusion method for objective quality assessment. The method leverages Q-Align, a cutting-edge large quality model, for image perception, and incorporates a variable-length video memory filter designed to integrate forgetting curves to derive video quality scores. Additionally, features extracted from speech signals are employed for audio-lip coherence analysis, and buffering and stuck detection in 3D streaming. These features are then fed into support vector regression to obtain the final predicted quality of the 3D streaming media. Experimental results demonstrate the superior performance of the proposed method. This work is anticipated to make significant contributions to the quality monitoring and evaluation of 3D streaming media, offering valuable insights for future research endeavors in this domain. 
 
 Appendix: https://www.blender.org/https://github.com/neverhood311/Stop-motion-OBJ (a) (b) (c) (d) 


Decision: There was a general consensus that this paper is of interest to the community. The decision is accept  but some key concerns remain and a shepherd should accompany this final acceptance.

- analysis of the viewing behaviour of the subjects and its impact on the overall should be included with some discussion. 
- writing (some of it seems to be generated) - this is a MAJOR concern of the paper in its current form.
- more details on the test set up to be added.
- writing in terms of flow and figures to be improved.

Reviews: review: paper_summary: paper_summary: This paper constructs a database, conducts a subjective experiment, and proposes a no-reference objective quality metric for "3D talking heads". Although the paper tries to cover most aspects of the subjective quality assessment domain, it does so in a superficial way.  At a higher level, although the topic is somehow motivated, what are realistic use cases of "talking heads"? Why is there a need for a metric specialized on "talking heads" and on the selected (simplistic) types of distortions?
strength_weakness: strength_weakness: + timely and interesting topic

The paper is timely and there are some interesting aspects, most importantly, the simultaneous evaluation of 3D models and audio in an interactive fashion. However, there is need for more in-depth description and analysis of the experimental factors, as well as their impact in the quality of experience of the users (see below).
limitations: limitations: - novelties
- reproducibility
- references
- writing

General:
- Most novelties of the paper are incremental. Specifically, the database construction relies on the selection of contents from an already existing repository, while the proposed objective quality metric is a composition of learning-based modules that are already available in the literature.

Related work:
- In section 2.2, the authors have completely neglected to refer to previous work on quality of assessment for both meshes and point clouds - both subjective and objective quality methods. Refer to [1] (i.e., it is the latest review I am aware of) and choose some of the methods discussed, which can be further enriched with newer submissions.

Subjective experiment:
- The subjective experiment is treated superficially. There are no sufficient descriptions of the environment, procedure, and methodology used, raising questions regarding its validity and reproducibility. For example, what were the testing setup and environmental conditions? What was the protocol used (e.g., ACR, DSIS)? What were the instructions regarding interactivity, and rating of audio and visual information? How was the training performed? 
- There is no analysis beyond histograms for subjective scores. For example, how long and how differently did the users interact with each content? Were there any differences between different types of distortions? 
- Using 3D video + audio + interactivity in the same experiment, I believe there are too many influencing factors for the rated quality from users, which makes it hard to identify where the quality variations are coming from. How do the authors comment on that, or what precautions did they take?
- In section 3.4, it is written:
"... while wired headphones of superior sound quality are employed for audio perception"
What was the exact model of the headphones used?

Objective metric:
- How realistic is the usage of the frontal view only of the mesh model in the metric? How much is the performance of the LQM affected by the inclusion of additional views? 
- There is no information regarding the placement of the virtual camera on the x-axis. Was it the same for all models? Is the projected region across models the same? How much does the performance of the LQM is affected by such choices?
- It seems that each feature is dedicated to a specific type of distortion introduced in the constructed dataset. It feels as if the authors are tailoring the problem and solution to fit each other.
- It is not clear why the authors replaced proposed features (i.e., audio-lip consistency) with a different feature (i.e., video bitrate) when evaluating their metric in Waterloo-III dataset. The authors should either include the latter feature in their proposed metric or use a subset of the proposed ones.

Experiments:
- Are the selected metrics suitable for evaluating the constructed dataset? Does any of those consider audio information? 
- What is the reason for using point cloud objective quality metrics on meshes? Why not use mesh metrics? How did the authors make the conversion from meshes to point clouds? Why did they not use state-of-the-art point cloud metric solutions?

Other comments:
- In section 2.2, second paragraph, please fix the first 2 sentences.
- In Table 3, with "Code Rate" I understand you mean "Bitrate". Moreover, "synchronicity" should have the first letter as capital to follow the same naming conventions. Please clarify/fix.
- In Figure 3, there are inconsistencies of terms: "BY" -> "BU", "T_v" -> "T_u"
- In section 3.1, there is a typo in the title: "Collectcion" -> "Collection"
- In section 3.3, the title is a bit misleading. The used pipeline is at best a simulation of streaming. In fact, authors just encode and decode the content.
- In section 4.2, provide a formal definition for Q_V_s. 

Last but not least, across the entire manuscript, there is a selection of words that is quite unlikely to have been made by humans. This is not a plus for this paper. I would like to remind the authors that, according to ACM rules, if an LLM has been used to write the paper, it is mandatory to state so in the acknowledgments.

[1] https://doi.org/10.1016/B978-0-32-391755-1.00024-9
paper_summary: paper_summary: The paper presents a large-scale 3D Talking Heads Quality Assessment (THQA-3D) dataset. The dataset includes different types of distortions encountered in 3D TH transmission. Furthermore, a multimodal QoE model for quality assessment of 3D THs is proposed. The proposed model has been extensively evaluated and is shown to perform better than state-of-the-art models using a five-fold cross-validation strategy.
strength_weakness: strength_weakness: Strengths
1) A large-scale dataset of 3D THs with MOS from 50 subjects has been created.
2) A multimodal QoE model that is shown to outperform state-of-the-art models is proposed.
3) Detailed evaluation of the proposed model.
limitations: limitations: Weaknesses
1) The test setup is not explained in detail, especially the audio part.
2) No analysis of the viewing behaviour of the subjects and its impact on the overall MOS has been conducted. The subjects were allowed to pause and also to adjust timestamps for playback which may have an impact on the perceived quality.
paper_summary: paper_summary: # summary

The paper presents a dataset, subjective and objective evaluation of quality for 3D talking heads.
The model looks interesting, but there are some unclear parts in the evaluation (see below).
Furthermore, the subjective study may be a bit hard for one participants to rate 1000 3D heads, in video tests this is usually restricted to max 1 hour of testing.


## comments

### intro
* "artificial compression before transmission", why artificial, why not just compression?
* what is a "large quality model"
### database construction
* why MP3 for audio compression and not opus or AAC, why JPEG for the textures and not something more modern?

* so each participants rated 1000 3DMS?, were they paid? (and moreover any kind of demographic overview of the participants, e.g. age?)
* eq. 2, an average calculation must not nessessarily being expressed by a separate equation
* Fig 6 is not readable

### proposed qoe model
* the only question I have here, is why a SVR and not something else was used? e.g. a RF model, or a neural network?

### experiments
* "four widely used quality metrics", I would rather call them performance metrics 
* "The proposed algorithm achieves state-of-the-art performance on the THQA-3D dataset", well but the proposed method is also trained for this dataset, the other models are not?, furthermore was this now the performance values of the 5 fold evaluation?, a bit unclear here.
* Table6 --> Table 6

### conclusion
strength_weakness: strength_weakness: * new dataset
* new model
* subjective evaluation
limitations: limitations: * evaluation and some design choices are a bit unclear
paper_summary: paper_summary: This paper proposes a new QoE Model and tests it on a new Dataset of 3D Talking Heads, which is also introduced in this paper, as well as on the Watterloo-III Dataset.
The authors compare their model with other SoA metrics on their own dataset and the Waterloo-III.
The authors show every step with all the equations of how the model is constructed and what is influencing the outcome of the model.
strength_weakness: strength_weakness: The strength of the paper is, that the model is described in detail and compare it to other models/metrics on two datasets.
The single steps that are shown are all easy to follow and the whole process seems well thought out. 
The content in the Tables and Figures is well suited to guide the reader through the paper.

The first part of the paper about the dataset is well written and shows nicely how the dataset was created. 
The second part describes the Model Creation in detail so that everyone can follow the process.
The metrics that were used to compare the model with others are a good starting point for analysis of the dataset and the model.
limitations: limitations: Besides the well planned structure of the paper it is still somewhat hard to read the paper because of all the equations and the following descriptions of the equations. This is because the description of the equations follows the same scheme over and over again (xyz denoted...)
In some cases the equation could be left out, because the text is already describing what is shown in the equation (e.g. 2 or 12).
Furthermore some descriptions of the equations are not giving additional information (e.g. in 14: SVR() denoted the SVR process (no information what the process is, and the reader would need to look that up anyways))
With leaving out some text parts that describe equations that are self explanatory, some figures like Figure2 or Figure6 could be enlarged because they are barely readable in a printed version. Maybe it is enough to enlarge the fontsize of the figures.
It would be nice if the authers would compare their dataset to other talking head datasets with objective metrics. Comparing it to the Waterloo-III is not giving the best insights to compare a talking head dataset to.

Other points:
- Typo in Table2 (heathier -> healthier)