FC-4DFS: Frequency-controlled Flexible 4D Facial Expression Synthesizing

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 08:23:16

Abstract: 4D facial expression synthesizing is a critical problem in the fields of computer vision and graphics. Current methods lack flexibility and smoothness when simulating the inter-frame motion of expression sequences. In this paper, we propose a frequency-controlled 4D facial expression synthesizing method, FC-4DFS. Specifically, we introduce a frequency-controlled LSTM network to generate 4D facial expression sequences frame by frame from a given neutral landmark with a given length. Meanwhile, we propose a temporal coherence loss to enhance the perception of temporal sequence motion and improve the accuracy of relative displacements. Furthermore, we designed a Multi-level Identity-Aware Displacement Network based on a cross-attention mechanism to reconstruct the 4D facial expression sequences from landmark sequences. Finally, our FC-4DFS achieves flexible and SOTA generation results of 4D facial expression sequences with different lengths on CoMA and Florence4D datasets. The code will be available on GitHub.

Main paper: 
 introduction: 
 4D facial expression synthesizing is a critical problem in the fields of computer vision and graphics and has broad applications in areas such as 3D animations, virtual reality, and interactive gaming. The task aims to generate a series of realistic facial mesh with diverse expressions or speech-related movements, starting from a mesh with a neutral expression.Although recent advances in generative networks have propelled the development of 2D video animation solutions [12, 25, 34] , these videos lack spatial depth, realism, and interactivity, rendering them unsuitable for real-world applications like VR. Afterward, an increasing number of researchers are beginning to explore the driving and generation of 4D facial expression sequences with the introduction of 4D facial expression sequence datasets [8, 9, 29, 30] . Among GT ours LM4D-GAN them, some previous works have focused on generating 4D facial sequences from neutral meshes driven by complex video [5] or audio signals [9, 19, 24] . However, these methods demand substantial prior knowledge and require extensive data, lacking temporal continuity, making direct application in scenarios like game scene development challenging where such priors are absent. In contrast, our research focuses on utilizing expression labels to guide a temporally coherent animation process, enhancing the applicability of facial expression sequence generation in scenarios with minimal priors.Motion3DRecently, some researchers introduced GAN [17] into the labelguided 4D facial expression generation task. First, Motion3D [27] proposed the WGAN to generate the Square Root Velocity Function (SRVF) of the 3D landmarks sequences and then they devised an S2D decoder to convert them into mesh displacements, which added to the neutral mesh and yielded a 4D facial expression. This pioneering work has laid the groundwork for its application in a variety of downstream tasks [26] . However, their generated sequences lacked robustness to different identities since they only use expression labels as guides. To address this issue, some recent methods [21, 36, 37] utilize the neutral landmark as inputs to directly generate landmark sequences, thereby enhancing the generation framework's robustness to various identities. However, the sequences generated by these methods exhibit issues such as sequence motion lacking smoothness (Figure .1 line 1), and missing expression details(Figure .1 line 2). Additionally, their application in real-world scenarios is limited since they can only generate sequences of fixed lengths.On the other hand, it is essential to transform them into a sequence of 3D meshes after obtaining the sequence of landmarks. N3DMM [3] first introduced a graph convolution method called spiral convolution, to encode and decode entire meshes. However, this method cannot effectively model the neutral mesh of unknown identities. Furthermore, S2D [27] proposed decoding mesh displacements from landmark displacements to reduce the modeling error for meshes of unknown identities. They divide the expression mesh into neutral mesh and displacement components and eliminate the need to encode and decode the dominant neutral mesh body. However, due to the limited information contained in landmark displacements and the diverse identity information, their ability to generalize the expression mesh details across different identities still requires further enhancement.In this paper, we propose a frequency-controlled 4D facial expression synthesizing method, FC-4DFS, based on frequency-controlled LSTM and multi-level identity-aware network. Specifically, we introduce a frequency-controlled LSTM (FC-LSTM) to generate 4D facial expression landmark sequences frame by frame from a given neutral landmark. Among them, we first modified the basic structure of LSTM to produce different length sequences with controlled frequency. Meanwhile, we integrate the positional information to enhance the awareness of the current frame's position within the sequence. Furthermore, we propose a temporal coherence loss to enhance the perception of temporal sequence motion and improve the accuracy of relative displacements.On the other hand, we design a multi-level identity-aware network(MIADNet) based on a cross-attention mechanism to reconstruct the 4D facial expression sequences from landmark sequences. Our MIADNet can significantly improve the robustness of various identities and facilitate the generation of detailed and identityconsistent facial expressions. Finally, our FC-4DFS achieves flexible and SOTA generation results of 4D facial expression sequences with different lengths on both the CoMA and Florence4D datasets. In summary, we make the following contributions:• We introduce a frame-by-frame 4D facial expression generation framework based on the FC-LSTM network and add temporal loss to achieve flexible control of the length of the generated sequence and enhance the smoothness of sequence motion. • We design the MIADNet, which leverages the cross-attention mechanism and fully utilizes the multi-level identity information of the neutral mesh and neutral landmark, further enhancing the decoding robustness across various identities. • Our FC-4DFS enables the generation of 4D face expression sequences with different lengths while maintaining sequence integrity and facial details. We also achieve SOTA generation results on both the CoMA and Florence4D datasets. 
related work: 
 Here, we review some of the latest advancements in related domains, including 4D facial expression sequence generation and 3D face modeling.Many researchers are committed to directly modeling 4D facial expression information [6, 7, 16] . However, because mesh has dense vertex information, it is difficult to directly drive mesh through tags to form expression sequences, and cannot effectively simulate expression details [28] . Thanks to the development of neural networks, the detection of facial landmarks is reliable and accurate [11, 13, 32, 35] , while landmarks and their motion provide a feasible way to explain facial deformation by simplifying the complexity of visual data. Researchers began to study the use of landmarks to guide expression sequence generation. The work of landmark-guided 2D expression generation has been widely studied [10, 31, 33, 34] . These methods have demonstrated the potential of using landmarks to simulate expression dynamics and generate 2D videos using generative models [17, 18, 20] , so researchers began to focus on landmark-guided Generation of 4D expression sequences. Motion3D [27] takes a step further based on MotionGAN [25] and proposes WGAN to generate the square root velocity function (SRVF) of 3D landmark sequences, and then they design an S2D decoder to convert them into mesh displacements, This displacement is added to the neutral mesh and generates a 4D expression, which is applied in downstream tasks [26] . However, the sequences they generated lacked robustness to different identities when generalizing. In order to enhance the generalization ability of the generated sequence to different identities, [21, 36, 37] uses neutral landmark sequences as input to generate complete landmark expression sequences. However, these models can only generate one [36, 37] or several [21] fixed-length sequences and cannot be effectively applied in actual game modeling scenarios.In order to obtain mesh information from the generated landmark sequences, researchers use the 3D Morphable Model (3DMM) as a bridge between landmarks and 3D facial expressions. 3DMM was originally introduced in [1] and is widely considered the premier framework for 3D facial modeling. The original model and its variants [2, 4, 15, 22, 23] rely on linear formulations to capture changes in facial shape, including identity and expression, thus limiting their modeling capabilities. Ranjan et al [30] introduced an autoencoder architecture based on a newly defined spectral convolution operator, along with pooling operations to facilitate grid down/upsampling. Brisas et al. [3] improve this approach by introducing a novel graph convolution operator that enforces consistent local ordering on the vertices of the graph via a spiral operator.Although their modeling accuracy is impressive across different expressions, recent research [14] highlights their significant shortcomings in generalizing to unknown identities. This severely limits their practical applicability in tasks such as face fitting or expression transfer. Next, S2D [27] begins to decouple neutral meshes and displacements, using landmark displacements to guide the modeling of 3D meshes. This method divides the expression mesh into neutral meshes and displacements and eliminates the need to encode and decode the main neutral mesh volume, greatly improving the ability to model expressions of unknown identities. However, due to the limited information contained in landmark displacements and the lack of diverse identity information, their ability to generalize the expressive mesh details of different identities still needs to be further enhanced. 
method 3.1 overview: 
 As shown in Figure .2, our FC-4DFS framework consists of a frequencycontrolled LSTM that generates a landmark expression sequence and a MIADNet that transfers the landmark displacements into mesh vertex displacements. Specifically, the frequency-controlled LSTM uses the landmark 𝑙𝑚 𝑡 -1 of frame 𝑡 -1 and the expression label as input and generates the landmark 𝑙𝑚 𝑡 of frame t as output, where the neutral landmark 𝑙𝑚 0 is given as the initial landmark at the first frame (See Section 3.2). Then, we subtract the initial neutral landmark 𝑙𝑚 0 from the obtained landmark sequence {𝑙𝑚 𝑡 } 𝑛 𝑡 =1 to obtain the displacement sequence for the subsequent network. Next, the MIADNet extracts multi-level latent features from the neutral mesh and predicts mesh displacement sequence with the input of landmark displacement sequence (See Section 3.3). Finally, we combine the mesh displacement sequence and the neutral mesh to obtain an expressional mesh sequence. Besides, in order to enhance the perception of temporal sequence motion and improve the accuracy of relative displacements, we also design a time consistency loss function (See Section 3.4).In this section, we propose a frequency-controlled LSTM, as shown in Figure . 3, to realize timing and frame rate perception of expression actions, realizing flexible frame-by-frame generation of variable-length sequences. Specifically, we first integrate frequency features into the LSTM for controlled long short-term memory updates, after which we introduce relative positional encoding embedding for free-framerate perception. Frequency Integration in LSTM Given the input 𝑥 𝑡 of the current frame 𝑡 extracted by the feature network and the output 𝑦 𝑡 -1 of the last frame 𝑡 -1, the LSTM needs to obtain the output 𝑦 𝑡 and update the hidden state ℎ 𝑡 in the model. In general, it first calculates the forget gate 𝑓 𝑡 and the input gate 𝑖 𝑡 as follows:× × ＋ × tanh σ σ σ y t-1 h t-1 y t h t tanh h t x t freq t           + ) ( ) ( ) ( 1 t t 1 - t P PE P PE P PE Emb i t f t o tEQUATIONwhere 𝑊 and 𝑏 are the weight matrices and bias coefficients, 𝜎 is the sigmoid function.Through the hidden state and the controls of input gate and forget gate, the LSTM achieves stable and efficient timing-dependent modeling. However, LSTM implicitly sorts the input sequence by index, and the interval between two adjacent frame inputs is always 1 when entering a sequence of different lengths. This is appropriate for inputs that only consider precedence, such as the position of each word in a sentence. However, it cannot perceive the similarity difference between the two frames at different frame rates in the input with frame rate attributes, such as an image sequence in a video.To do this, we integrate frequency information 𝑓 𝑟𝑒𝑞 𝑡 into the LSTM to control the current frame forget gate and input gate as 𝑓 𝑡 = 𝜎 𝑊 𝑓 • ℎ 𝑡 -1 , 𝑥 𝑡 , 𝑓 𝑟𝑒𝑞 𝑡 + 𝑏 𝑓 𝑖 𝑡 = 𝜎 𝑊 𝑖 • ℎ 𝑡 -1 , 𝑥 𝑡 , 𝑓 𝑟𝑒𝑞 𝑡 + 𝑏 𝑖 (2)After that, we obtain the candidate output ỹ𝑡 and the output gateEQUATIONFinally, we update the last hidden state from ℎ 𝑡 -1 to ℎ 𝑡 and calculate the output 𝑦 𝑡 as follows:ℎ 𝑡 = 𝑜 𝑡 * tanh (𝑦 𝑡 ) 𝑦 𝑡 = 𝑓 𝑡 * 𝑦 𝑡 -1 + 𝑖 𝑡 * ỹ𝑡 (4)Framerate-aware Positional Encoding Take the expression label and initial neutral landmark as inputs, to generate a landmark sequence under free-framerate control, it is necessary to obtain the current position in the complete sequence and its time change relative to the previous frame. Thus, we employ relative positional encoding to code the entire sequence. In particular, we first calculate the relative position of the current frame in the full sequence as:p𝑡 = 𝑝 𝑡 𝐿 (5)where 𝑝 𝑡 is the time point of the current frame in the complete sequence, and 𝐿 is the duration of the complete sequence. Then, we encode the relative positional encoding of the current frame 𝑃𝐸 as:EQUATIONwhere 𝑗 represents the dimension within the encoding vector, and 𝑑 is the total number of dimensions in the model. Finally, we embed the positional encoding of adjacent frames into the network as:EQUATIONwhich captures the temporal changes between the current frame and the previous frame, and realizes the free-framerate control for sequence generating.The FC-LSTM previously generated realistic sequences of facial expression landmarks. However, our ultimate objective is to animate the neutral mesh 𝑀 0 to produce novel 3D facial animation sequences {𝑀 𝑡 } 𝑛 𝑡 =1 . In this section, we propose a multi-level identityaware displacement network (MIADNet) to make full use of the identity information from the generated landmark sequence and the neutral mesh, thus further enhancing the robustness to identity when reconstructing 4D facial expression sequence from landmark sequence. Specifically, the proposed MIADNet model, as shown in Figure . 4, includes a landmark decomposition embedding module, an identity extractor, and an identity-aware mesh generator. Landmark Decomposition Embedding The generated landmark sequence after FC-LSTM contains identity and expression change information, where the identity information is shared throughout the action cycle, and provides a facial datum shape reference for the generation of the expressional mesh. We first decompose the generated landmark sequence {𝑙𝑚 𝑡 } 𝑛 𝑡 =1 to the neutral landmark 𝑙𝑚 0 and a landmark displacement sequence {Δ𝑙𝑚 𝑡 } 𝑛 𝑡 =1 . For each frame 𝑡, we embed 𝑙𝑚 0 and Δ𝑙𝑚 𝑡 to vectors in the latent code space with MLPs as the input to subsequent mesh generators, which is expressed as:EQUATION)where 𝜆 is a learnable parameter.The neutral landmark 𝑙𝑚 0 provides global identity information at low resolutions, however, it doesn't have the ability to provide facial details for high-resolution expression animation meshes. In addition, it is difficult to maintain stability throughout the sequence for mesh generation using a single-frame landmark. For this purpose, we introduce neutral mesh 𝑀 0 as input, and we use multiple spiral convolutions to extract multi-resolution identity features from 𝑀 0 . The multi-resolution feature constructs the intermediate bridge between low-resolution landmarks and high-resolution meshes, effectively improving the consistency of the generation process from landmarks to meshes and the stability in different frames.Identity-aware Mesh Generator Given the latent landmark code 𝑧 𝑡 and the multi-resolution identity features 𝐹 𝑒𝑛 of the current frame 𝑡, the mesh generator upsamples 𝑧 𝑡 and combines it with the identity features 𝐹 𝑒𝑛 by skip connections to obtain the mesh output.To utilize the neutral mesh information as a reference mesh in the generation to reconstruct a continuous and consistent sequence, we use the cross-attention mechanism to model the context dependence between the generated expression mesh and the reference neutral mesh. Specifically, the input features of the i-level decoder can be expressed as:EQUATION)where 𝐹 𝑑𝑒 are the decoder features, and 𝑑 is the feature dimension. For the generation of facial expression animations, it is crucial to ensure the high quality of each frame and maintain continuity and smoothness throughout the sequence. This requires designing a loss function that takes into account the generation quality of each frame and the relationship between frames to solve these problems. Previous methods that directly generate complete sequences often directly measure the quality of the generated sequence to ensure accuracy and perceived fidelity. But this method tends to ignore motion in sequences. To this end, we propose a hybrid reconstruction and temporal coherence loss function to evaluate the generation quality of each frame and inter-frame motion respectively, while maintaining the generation quality while strengthening the model's modeling of sequence motion. The training loss of our model can be formulated as:EQUATION)where 𝐿 𝑟𝑒 represents the single-frame reconstruction loss based on L1 distance, i.e.,EQUATION)and we set the hyperparameters 𝛼 = 0.3 in the loss function during training. Then, we use landmark motion between adjacent frames to enhance control over the smoothness of the sequence, and the temporal coherence loss 𝐿 𝑡𝑒𝑚𝑝𝑜𝑟𝑎𝑙 is described as:EQUATION)4 EXPERIMENTS AND RESULTS 
experimental details: 
 Datasets We carry out experiments on the CoMA [30] dataset and Florence4D [29] dataset. CoMA dataset comprises 144 facial expression sequences with 20,466 facial meshes collected from 12 unique subjects showcasing 12 different expressions, highlighting a vast spectrum of extreme expressions and the diversity in facial movements.The CoMA dataset faces challenges due to data schema when applied to the task of generating label-driven facial expression animations. This is mainly because such tasks require a complete sequence from neutral expression to peak expression, and expression patterns(some are from neutral to peak while others are from neutral to peak and back to neutral) and the sequence lengths in the CoMA dataset are not uniform, and most of the frames of the sequence are concentrated in neutral and peak.Existing methods, such as Motion3D, utilize sampling and interpolation to produce GT sequences, which struggle to smooth transitions and ignore motion changes of neutral and peak expressions. Therefore, we conduct a thorough refinement for the CoMA dataset to obtain the Aligned CoMA dataset. We first normalized sequence lengths, then emphasized significant expression changes, and curated the sequence to ensure the generation of animations that authentically and smoothly capture the nuances of facial expressions.Florence4D dataset provides an extensive collection of 6,650 facial sequences with 399,000 face meshes from 95 distinct identities, encompassing 43 females and 52 males, and capturing a broad array of 70 unique facial expressions. Each sequence transitions from a neutral expression to an apex expression and then returns to neutral, demonstrating a rich diversity of facial movements and expressions. Both datasets are aligned in FLAME topology and have N = 5, 023 vertices. Implement Details To maintain the independence of FC-LSTM and MIADNet, we train them separately on these datasets. For the landmark sequence generation of FC-LSTM, we first allocated the first three-quarters of the aligned datasets from CoMA and Flo-rence4D for training, reserving the final quarter for testing. Meanwhile, we utilize the Adam optimizer and set the learning rate at 1e-5. Finally, we train 1,200 epochs for CoMA and 600 epochs for Florence4D. To rigorously evaluate the generalization capability of the MIADNet, we first implement a 4-fold cross-validation protocol for both the CoMA and Florence4D datasets, same as S2D [27] . Meanwhile, we utilize the Adam optimizer for training with a learning rate of 1e-3 and a decay rate of 0.99. Then, we train 300 epochs for the CoMA dataset and 150 epochs for the Florence4D dataset with a batch size of 128.These experiments were meticulously designed to thoroughly examine the proposed framework's effectiveness and adaptability in capturing the complexities of facial expressions. Both tasks are trained on a device with 8 NVIDIA 4090 GPUs and Intel(R) Xeon(R) Platinum 8368 CPU @ 2.40GHz.Generating sequences of 4D facial expressions In this section, we compare our FC-4DFS with the state-of-the-art methods including Motion3D [27] and LM-4DGAN [21] . Motion3D uses SRVF to encode landmark displacement and their open-source code is prone to crashing during the training stage. Therefore, we use their public model on the CoMA dataset for a 30-frame sequence comparison. Furthermore, we retrained the LM-4DGAN model on two datasets for comparison. Then, we generated the landmark sequence and reconstructed the mesh sequence from them, and used the mean square error (MSE) to compare the vertex reconstruction errors in landmark and mesh, 𝐸 𝑙𝑚 (mm) and 𝐸 𝑚𝑒𝑠ℎ (mm) respectively. Finally, to characterize the perceptual accuracy of the reconstructed expression sequences relative to expression labels, we trained expression-labeled classifiers on various datasets and used classification accuracy(CA) to test the generated result by various methods. The quantitative comparison results are shown in Tab. 1. For 30 frame comparisons, the landmark per-vertex reconstruction error (𝐸 𝑙𝑚 ) using our FC-LSTM is 26% higher than Motion3D and 17% higher than LM-4DGAN. At the same time, when using the same S2D decoder for end-to-end reconstruction, the mesh reconstruction error (𝐸 𝑚𝑒𝑠ℎ ) per vertex is 18% higher than Motion3D and 7.6% higher than LM-4DGAN. In addition, when using our MIADNet to reconstruct the 4D facial expression from the generated landmark, it is improved by 21.8% compared to Motion3D and 12.5% compared to LM-4DGAN. The classification accuracy of the data generated by our method is presented in the third column of the table, demonstrating that the sequences produced, with each frame's mesh passing through a classification network, achieve results closer to the specified expressions. LM-4DGAN enhances the modeling of different identity movements due to the direct generation of landmark sequences, and its results are better than Motion3D. On the basis of utilizing neutral landmarks, our FC-LSTM generates sequences frame by frame and further considers the smoothness of inter-frame motion. Furthermore, our MIADNet introduces a neutral mesh with the cross-attention mechanism guided, which also improves the decoding result. Therefore, our method achieves the SOTA performance in all indicators and is good at generating label-driven 4D facial expression sequences with enhanced fidelity.In addition, we qualitatively compared the generation results of our method with Motion3D and LM-4DGAN on multiple subjects at 30 frames. As illustrated in Figure5, the Motion3D sequence lacks continuity between frames, and detailed information about the mouth is missing in the sequence. LM-4DGAN uses a step-bystep generation method to enhance the continuity of the sequence and directly models the complete sequence from the landmark sequence, while it is more difficult to model motion patterns. In contrast, methods using FC-LSTM to generate the next frame based on the previous frame and expression labels, can achieve smooth and effective motion pattern modeling. Moreover, our final results with MIADNet achieve closer to GT than using the S2D decoder.We mainly compared our MIADNet with several 3DMM-based methods, including the basic PCA-3DMM [2] , the Neural 3DMM (N3DMM) [3] that specifically designed to learn the nonlinear latent space of mesh changes and reconstruct the input 3D mesh and the basic S2D [27] decoder without neutral identity information for the input displacement. Landmark Decoder MIADNet is another important part of our FC-4DFS. In order to verify the robustness of the reconstruction network to unseen identities, Refers to the experiment of S2D [27] , we performed 4-fold cross-validation on the identities on the CoMA and Florence4D datasets and used per-vertex Euclidean error (mm) to measure the reconstruction performance. We compared our MI-ADNet with the basic S2D decoder and 3DMM-based methods, including the basic PCA-3DMM [2] , the Neural 3DMM(N3DMM) [3] .The quantitative results of these comparisons are detailed in Table . 2. First, the spiral convolution-based N3DMM, which is proposed to learn the nonlinear latent space of facial changes for the 3DMM mesh with FLAME topology, performs poorly on both datasets. This is because N3DMM encodes and decodes the entire mesh, while it demonstrates some generalization capabilities across different expressions, its overall performance in handling new identities is still poor [27] . Furthermore, S2D decomposes the mesh into a neutral mesh that represents the identity and displacement, then uses the spiral convolution method to decode the displacement from the landmark into the displacement of the mesh. This method avoids encoding and decoding of neutral mesh to reduce errors but misses the use of identity information while its robustness to different identities needs to be further enhanced. Finally, our MIADNet adds the multi-level identity information for the S2D decoder, strengthening the reconstruction network's modeling capabilities for unknown identities. Therefore, our MIADNet achieve a 5% improvement on the CoMA dataset and 10.3% improvement on the Florence4D dataset compared to S2D. This shows that multilevel identity information can enhance the robustness of different identities.On the other hand, we visualize the error heatmaps of the meshes reconstructed by these methods to compare their performance. Figure.6 provides qualitative examples of error heatmaps for identityindependent splitting by comparing them with PCA, N3DMM, and S2D. First, PCA-3DMM and N3DMM show high errors even for neutral meshes, demonstrating their inability to recover the identity of unseen meshes. Meanwhile, the high error of N3DMM is mainly concentrated in the lower half of the mesh, while the high error area of PCA is scattered throughout the mesh. This is because N3DMM takes advantage of the geometric topological relationship of 3DMM and uses spiral convolution to encode neutral meshes in the latent space, which can better reconstruct the topological structure of the mesh. Then, the S2D method and our method achieve lower reconstruction errors, which indicates that learning per-point displacements is better than directly learning point coordinates. Finally, our method shows significant improvement over the S2D method in handling highly varying expressions (see the neck and .2.The Impact of Our Frequency-Controlled LSTM To validate the role of LSTM in our proposed framework and the effectiveness of our novel frequency-controlled LSTM, we first use a basic MLP as the baseline for sequence regression and compare it with standard LSTM and our FC-LSTM. Due to the issues with the CoMA dataset mentioned in Section 4.1, to better compare the generated results, we trained on the aligned CoMA dataset(containing multiple sequence lengths) and the Florance4D dataset, and tested on 3 different lengths, as detailed in Table . 3. Furthermore, we use the first 3/4 identities as training data and the last 1/4 identities as test data and use per-vertex reconstruction error for evaluation.It is observed that the framework utilizing LSTM for regression consistently achieves lower per-vertex reconstruction errors across all test lengths compared to MLP. Moreover, our FC-LSTM improves performance by 14% over the baseline in the aligned CoMA dataset and by 13.8% over the baseline in the Florence4D dataset. Compared with the basic LSTM, the improvement is 4.3% in the aligned CoMA dataset and 4.1% in the Florence4D dataset.On the other hand, Figure .7 highlights the effectiveness of our FC-LSTM, demonstrating that our model can generate complete sequences of any given length and maintain fine detail across different lengths. Furthermore, the LSTM method without frequency control cannot effectively generate a complete expression sequence such as the mouth cannot be fully opened in the first row. Impact of temporal loss Sequence continuity is of paramount importance for the task of 4D facial expression generation. The displacement between adjacent frames in a sequence represents the motion of landmarks throughout the sequence, and to smooth this motion, we incorporated a temporal loss into our training process. As shown in Table .3, the LSTM with added temporal loss exhibited a performance increase of 4.5% on the Align CoMA dataset. Furthermore, it has an improvement of 2.3% after adding the temporal loss for our FC-LSTM. On the Florence4D dataset, the implementation of temporal loss resulted in an enhancement of 4.6% for the LSTM framework and 2.5% for our FC-LSTM.To further discuss the impact of hyperparameter 𝛼, we compared the per-vertex reconstruction error at different frame rates when training with different 𝛼 values. The experimental results are shown in the Table. 4 shows that the lowest reconstruction error is obtained when 𝛼 is 0.3.Impact of MIAD In the MIADNet part, based on S2D, we introduced Landmark Decomposition Embedding and Identity-aware Mesh Generator to extract multi-level identity information carried in neutral landmarks and neutral meshes respectively. To illustrate the role of multi-level identity information, we performed ablation experiments, and the results are shown in Table .2. It can be seen that adding Landmark Decomposition Embedding(S2D+lm-n) or Identity-aware Mesh Generator(S2D+mesh-n) alone does not significantly improve the reconstruction error. However, after introducing these two modules at the same time, we achieved a 5% improvement on the CoMA dataset and a 10.3% improvement on the Florence4D dataset compared to S2D.  
conclusion: 
 In this paper, we propose a label-guided 4D facial expression synthesis method, FC-4DFS, by using frequency-controlled LSTM networks (called FC-LSTM) to form a flexible generation framework. This method not only overcomes the limitation that traditional methods can only generate fixed sequences but also further enhances the smoothness of generated sequence motion. The introduction of MIADNet further enhances our framework's ability to accurately reconstruct facial expression sequences with complex identity information from facial landmarks. Experiments conducted on the CoMA and Florence4D datasets demonstrate that our model achieves state-of-the-art performance.It is worth noting that our method still needs to generate a landmark sequence first and then expand it to a mesh sequence according to the displacement. In future work, we will further study the end-to-end generation of 4D facial expression sequences. 
 
 Appendix:  


Decision: The reviewers all felt positively about this paper. The paper is well written and detailed analysis of previous related work.

Reviews: review: paper_summary: paper_summary: The task aims to generate a series of realistic facial mesh with diverse expressions or speech-related movements, starting from a mesh with a neutral expression. In this domain, this paper describes a frequency-controlled 4D facial expression synthesizing method based on frequency-controlled LSTM and multi-level identity-aware network. This is combined with a multi-level identity-aware network based on a cross-attention mechanism to reconstruct identity-consistent 4D facial expression sequences from landmark sequences.
strength_weakness: strength_weakness: Well written, detailed analysis of previous related work. A novel aspect of the proposed investigation is the use of expression labels to guide a temporally coherent animation process, enhancing the applicability of facial expression sequence generation in scenarios with minimal priors.
limitations: limitations: The processing pipeline is composed of two cascaded modules: 1) generation of the landmark expression sequence and 2) transfer landmark sequence displacements to mesh vertex displacements through the MIADNet. It is not clear how this architecture is able to process a continuous input video stream: is it necessary to decompose the stream into blocks and process the first one with the first module before passing it to the second module? This may introduce some delay and compromize smooth animated transition across different blocks.
It is really hard to find a significant difference in the data reconstructed by differente techniques as reported in Fig.5
paper_summary: paper_summary: The authors propose a frequency-controlled LSTM network (FC-LSTM) to generate 4D facial expression sequences frame by frame and enhance their temporal coherence. A Multi-level Identity-Aware Displacement Network (MIADNet) is also designed to reconstruct the 4D facial expressions from landmark sequences. The method achieves state-of-the-art (SOTA) results on CoMA and Florence4D datasets, demonstrating its effectiveness in generating realistic and flexible 4D facial expression sequences.
strength_weakness: strength_weakness: Integrating FC-LSTM for frame-by-frame generation and MIADNet for identity-aware reconstruction is a creative solution that addresses the limitations of previous methods. Introducing frequency-controlled mechanisms and cross-attention in MIADNet shows a deep understanding of the technical challenges in 4D facial expression synthesis. Using CoMA and Florence4D datasets allows for a thorough evaluation of the method's capabilities.
limitations: limitations: The framework may be complex to implement, which could limit its adoption by other researchers and practitioners. While the method shows strong results on the tested datasets, it needs to be clarified how well it generalizes to other datasets or variations in facial expressions. The paper could benefit from a discussion on the computational efficiency and scalability of the proposed method. More detailed comparative analysis with other state-of-the-art methods would be beneficial, especially regarding computational resources and scalability.
paper_summary: paper_summary: The paper discusses a method called FC-4DFS, which is a frequency-controlled 4D facial expression synthesizing method. The method uses a frequency-controlled LSTM network to generate 4D facial expression sequences frame by frame from a given neutral landmark with a given length. It also introduces a temporal coherence loss to enhance the perception of temporal sequence motion and improve the accuracy of relative displacements. Additionally, the paper describes a Multi-level Identity-Aware Displacement Network based on a cross-attention mechanism to reconstruct the 4D facial expression sequences from landmark sequences. The proposed method outperforms SOTA methods on CoMA and Florence4D datasets.
strength_weakness: strength_weakness: 1. The authors introduce a framework called FC-4DFS, which comprises a frequency-controlled LSTM (FC-LSTM) network to generate 4D facial expression sequences at frame-by-frame level and a Multi-level Identity-Aware Displacement Network (MIAD) based on a cross-attention mechanism to reconstruct the 4D facial expression sequences from landmark sequences.

2. The authors also perform extensive ablation studies and experiments on their proposed method using benchmark datasets and compare them with state-of-the-art methods.

3. In addition, the paper provides detailed explanations and descriptions of the proposed method and obtained results. Qualitative comparisons are also provided to substantiate FC-4DFS's performance.
limitations: limitations: 1. The authors did not experiment with the D3DFACS dataset, which is used in the current state-of-the-art work S2D[27].

2. The authors should use the up-arrow ($\uparrow$) and down-arrow ($\downarrow$) to indicate the directions of all metrics in the table results. For example, for vertex reconstruction errors in landmark ($\downarrow$), lower is better.