ListenFormer: Responsive Listening Head Generation with Non-autoregressive Transformers

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 05:41:44

Abstract: As one of the crucial elements in human-robot interaction, responsive listening head generation has attracted considerable attention from researchers. It aims to generate a listening head video based on speaker's audio and video as well as a reference listener image. However, existing methods exhibit two limitations: 1) the generation capability of their models is limited, resulting in generated videos that are far from real ones, and 2) they mostly employ autoregressive generative models, unable to mitigate the risk of error accumulation. To tackle these issues, we propose Listenformer that leverages the powerful temporal modeling capability of transformers for generation. It can perform non-autoregressive prediction with the proposed two-stage training method, simultaneously achieving temporal continuity and overall consistency in the outputs. To fully utilize the information from the speaker inputs, we designed an audio-motion attention fusion module, which improves the correlation of audio and motion features for accurate response. Additionally, a novel decoding method called sliding window with a large shift is proposed for Listenformer, demonstrating both excellent computational efficiency and effectiveness. Extensive experiments show that Listenformer outperforms the existing state-of-the-art methods on ViCo and L2L datasets. And a perceptual user study demonstrates the comprehensive performance of our method in generating diversity, identity preserving, speaker-listener synchronization, and attitude matching.

Main paper: 
 introduction: 
 Communication is indispensable in the process of social interaction, whether in a school setting or in a professional workplace [2, 37, 45] . In face-to-face communication [24] , participants take turns playing the roles of speaker and listener to exchange information. The speaker directly transmits information to the listener through verbal expression, while the listener actively considers the information provided by the speaker, decoding it, and offering real-time feedback Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ACM MM, 2024, Melbourne, Australia © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN primarily through non-verbal behaviors such as nodding, smiling, headshaking, etc. The speaker-centric synthesis, specifically talking head generation (THG), has received widespread attention. It plays a significant role in many human-robot interaction (HRI) applications, such as film production, games, and education. Researchers use still images and audio clips to generate vivid speaking videos, advancing towards improving lip-synchronization quality [8, 9, 17] , adding emotions [15, 23, 49] , and achieving free pose control [28, 31, 56] . However, as another crucial component of HRI, research on responsive listening head generation (LHG), is still in its early stages. The synthesis of smooth listening head videos is also crucial for successful communication [34, 40] . Through real-time feedback, the listener demonstrates their level of engagement in communication, making the conversation easier to understand for both parties. In addition to modeling everyday scenarios, it holds great potential for enriching virtual character modeling, synthesizing fake audiences, and various other applications involving responsive listeners.Similar to THG, LHG also involves the synthesis of human heads and faces. Therefore, there are many aspects that can be borrowed and applied. For instance, 3D Morphable Models (3DMM) [3, 12, 27] are often used in facial parameters modeling in the THG tasks. Similarly, this approach can be applied to LHG [35, 57] in order to maintain the stability of reconstructed faces. Meanwhile, there are differences between the two. Firstly, THG focuses solely on the speaker, while LHG spans both the speaker and the listener, requiring more consideration of how listening behaviors are influenced by the speaker signals. Additionally, LHG receives signals from both the speaker's audio and video modalities, requiring consideration of the audio-visual fusion issue.At the earliest, static images, repeated frames, or pre-scripted animations were commonly used to synthesize listeners. However, they often appeared too rigid and were unable to respond realistically to the speaker [57] . Recently, the LHG task was redefined and introduced by Zhou et al. [57] , who also curated the audio-visual ViCo dataset comprising video pairs of speakers and listeners. Almost simultaneously, Ng et al. [35] released a novel in-the-wild dataset of dyadic conversations and proposed L2L for understanding human interactional communication. Subsequently, substantial efforts [7, 21, 41, 54] have been devoted to investigating listening head generation techniques. However, these methods still encounter three primary limitations. Firstly, the quality and naturalness of the generated listener videos are currently not good enough. There is still a significant gap compared to real videos, largely due to limitations in model performance. Therefore, it is crucial to have a suitable and effective generative model for this task. Secondly, autoregressive models have inherent limitations. As seen in Fig. 1 (a), most existing methods [7, 21, 35, 41, 54, 57] employ autoregressive models, making it difficult to avoid issues such as slow synthesis speed and error accumulation. Moreover, there is insufficient attention to the fusion of audio and video signals for speakers. In LHG, the input speaker signals consist of two modalities: audio and visual motion. Most existing methods [7, 21, 54, 57] simply concatenate the signals from these two modalities, overlooking the importance of cross-modal fusion. A robust fusion method is necessary to better extract representative features from the speaker inputs.To address the aforementioned problems and meet the multifaceted requirements, we propose a non-autoregressive transformerbased model ListenFormer, which captures the speaker's audio and video signals as well as a reference image to generate highly realistic listener videos. It is important to note that, a two-stage training method is applied to ListenFormer. During the pre-training stage, we employ the teacher-forcing method. This means that real and continuous listener head coefficients were input into the decoder. In the fine-tuning stage, we modify the decoder input to consist of repeated reference image coefficients. As shown in Fig. 1(b) , in this way, we achieve the non-autoregressive prediction of ListenFormer. To capture the representative features, we propose a novel audiomotion attention fusion module (AMAF) to embed speaker's audio and motion features. The proposed module utilizes cross-modal attention to discover key information aligned along the temporal sequence. In addition, we experiment with several decoding methods to address the issue of temporal infinite extrapolation for ListenFormer. We conduct extensive experiments on ViCo and L2L datasets and achieve state-of-the-art performance on both datasets. Our code and benchmark will be released.Overall, our contributions are summarized as follows:• We propose a transformer-based model ListenFormer that can predict diverse and high-quality listening head videos in a non-autoregressive manner conditioned on the listener's reference image and speaker's audio and motion features. • The audio-motion attention fusion module (AMAF) is designed to integrate cross-modal features in order to provide representative speaker-related information to the decoder. • We present an efficient sliding-window decoding method, which addresses the transformer's inability to extrapolate infinitely.• Experimental results show a significant improvement achieved by our proposed method compared to other state-of-theart methods on the ViCo and L2L dataset in terms of visual naturalness, generation diversity, identity-preserving, speaker-listener synchronization, and attitude matching. 
related work 2.1 responsive listening head generation: 
 In early works, several rule-based methods [5, 6] were employed to produce listener heads. However, those videos fall far short in terms of naturalness and realism. Subsequently, some data-driven approaches [14, 36] based on facial keypoints were used to generate 2D listener motions, but they lost many details of facial expressions.In recent years, many 3D-based methods have been developed due to their excellent facial reconstruction capabilities. Zhou et al. [57] established a high-quality speaker-listener dataset, named ViCo. The proposed baseline utilizes long-short term memory (LSTM) as the sequential model to handle the input of speaker audio and visual signals, generating facial 3DMM coefficients for the listener. At almost the same time, Ng et al. [35] proposed a novel motionencoding VQ-VAE [47] to learn a discrete latent representation of realistic listener motion. Later, Huang et al. [21] adopted an enhanced renderer and video restoration module, improving the quality of the generated listening videos. Recently, some methods [7, 55] have attempted to incorporate semantic information into the inputs of the task with the pre-trained language model [26] . However, the methods mentioned above mostly employ autoregressive models, which cannot avoid the issue of error accumulation during the generation process. In contrast, the non-autoregressive prediction approach of Listenformer can largely overcome this limitation.Transformer [48] was initially proposed for sequence-to-sequence (seq2seq) translation in the field of natural language processing (NLP). Unlike recurrent neural networks (RNNs) that recursively process sequence tags, transformers can parallelly attend to all tokens in the input sequence, effectively modeling contextual information. The transformers have proven to be a powerful alternative to RNNs in various sequential tasks and have achieved marvelous success in audio-visual learning tasks such as speech recognition [32, 42] , emotion recognition [16, 20, 46, 58] and event detection [18, 29, 33] . Some of the most recent works on speech-driven THG [13, 22, 53] have explored the power of transformers in modeling facial features and produced impressive results.Despite its many advantages, the transformer as a generative model also has notable issues. For instance, the traditional Transformer, being an autoregressive model, suffers from the problem of error accumulation during inference. Additionally, the challenge of temporal infinite extrapolation has been a persistent concern for many researchers working with transformers [1, 43, 44, 52] . After comprehensive consideration, our work relies on a novel non-autoregressive transformer for the 3D reconstruction of the listener's face due to its excellent temporal modeling capability. Moreover, we explore various decoding approaches to address the challenge of infinite extrapolation.  
method 3.1 problem formulation: 
 We formulate the LHG task as a seq2seq learning problem. Given an input videoV 𝑠 𝑡 = {𝑣 𝑠 1 , • • • , 𝑣 𝑠 𝑡 }of a speaker head in timestamps ranging from {1, ..., 𝑡 }, containing a corresponding audio signal S 𝑡 . The goal here is to produce a model G (Fig. 2 ) that can synthesize the whole listener's head video sequence V𝑙EQUATIONwhere 𝑣 𝑙 1 denotes the reference head image of the listener. Following [57] , we apply the 3D-based method and divide the G into G 𝑚 and G 𝑟 . As shown in Fig. 2 , G 𝑚 consists of a speaker encoder and a listener decoder, which is used to predict the 3D reconstruction coefficients of listeners. And G 𝑟 is used for 3D face rendering, as depicted in the 'Renderer' part in Fig 2 . In the proposed G 𝑚 , the transformer encoder transforms audio feature H 𝑡 = {ℎ 1 , . . . , ℎ 𝑡 } into deep representation A 𝑡 = {𝑎 1 , . . . , 𝑎 𝑡 }. Meanwhile, we extract the 3D reconstruction coefficients {𝛼, 𝛽, 𝛿, 𝑝, 𝛾 } which denote the identity, expression, texture, pose, and lighting, respectively. They are split into two components: I = (𝛼, 𝛿, 𝛾) to represent relatively fixed, identity-dependent coefficients, and 𝑚 = (𝛽, 𝑝) to represent relatively dynamic, identity-independent coefficients. These identity-independent coefficients extracted from speaker videos can be denoted asM 𝑠 𝑡 = {𝑚 𝑠 1 , • • • , 𝑚 𝑠 𝑡 }.Then, the audio-motion fusion module fuses M 𝑠 𝑡 and A 𝑡 to get the fusion representation F 𝑡 . The transformer decoder receives F 𝑡 and the identity-independent coefficient 𝑚 𝑙 1 of the reference listener image to non-autoregressively predict the listener coefficients M𝑙𝑡 +1 = { m𝑙 2 , • • • , m𝑙 𝑡 +1 }.We formulate the procedure as:EQUATIONFinally, we use the pre-trained rendering model [39] to generate the realistic listening video. Formally,EQUATIONwhere I 𝑙 is the identity-dependent coefficient of the given listener.For the remainder of this section, we describe each component of the ListenFormer architecture in detail.We adopt the vanilla Transformer encoder [48] . It is composed of a sinusoidal positional encoding and a stack of sub-layers, converting the audio feature vectors H 𝑡 into contextualized representations A 𝑡 . Each encoder layer consists of multi-head self-attention and fully connected feed-forward networks. Note that layer normalization and residual connection are omitted for simplicity in Fig. 2 . The audio representations outputted by the encoder are sent to the audio-motion fusion module.The decoder is also composed of a sinusoidal positional encoding and a stack of sub-layers. Different from the encoder, each decoder layer consists of self-attention, cross-attention, and feed-forward networks. The output identity-independent 3D facial coefficients of listeners are sent to the renderer for video reconstruction.In the pre-training stage, we apply the teacher-forcing scheme, which is shown in the middle of Fig. 2 (light blue background). At each time step, the decoder receives the real target coefficientsM 𝑙 𝑡 = {𝑚 𝑙 1 , • • • , 𝑚 𝑙 𝑡 } (ground truth)along with the fusion representation, instead of using predictions generated by the model itself. This speeds up the training process and minimizes cumulative errors during the training phase. We also apply masks in the self-attention in the first training stage to prevent current output from being affected by subsequent positions according to [48] .Although the teacher-forcing scheme helps the model learn temporal continuity of the output, the model must rely on its own generated previous coefficients and generate predictions autoregressively during the inference phase, leading to inconsistency between training and inference. Therefore, we modify the input of the decoder in the fine-tuning stage. Specifically, as shown in the right of Fig. 2 (light green background), the 3D coefficient 𝑚 𝑙 1 of the reference listener image is replicated along the time axis and inputted into each time step to replace the ground truth M 𝑙 𝑡 . As shown in Fig. 1 (b), the model can perform non-autoregressive inference consistent with the training phase, thereby avoiding the issue of cumulative errors. Additionally, such approach does not require masks in self-attention and tends to provide a globally consistent motion.On the one hand, the first teacher-forcing pre-training stage forces the model to optimize in the right direction in the early stages of training. During the experimental phase, we find that skipping the pre-training stage and directly proceeding to the nonautoregressive training in the second stage does not yield more satisfactory results. For more details, refer to Section 4.5.2. This indicates the importance of the pre-training stage for the final performance of ListenFormer. On the other hand, the prediction approach in the second fine-tuning stage is the non-autoregressive method we ultimately aim for in inference. Since people usually do not make large head movements during the listening process, using repeated reference frames as the input for the decoder helps maintain the stability of the predictions. As a result, the combination of the two training stages allow ListenFormer to simultaneously learn temporal continuity and overall consistency, achieving a good balance between facial motion diversity and stability.Once the complete 3D facial coefficient sequence is produced, the model is trained by minimizing the regression loss between the decoder outputs and ground truths, which is calculated as:EQUATIONwhere β𝑙 𝑡 and p𝑙 𝑡 represent the generated expression and pose coefficients of listeners, respectively. The last two terms of Eq. 4 are applied to guarantee the inter-frame continuity, where 𝜇 (•) measures the inter-frame changes. 𝑤 1 and 𝑤 2 are the adjustable parameters for different losses.When we listen to others, the speaker's audio, facial expressions, and head motions can convey messages to us. Often, there is a strong correlation between them. Therefore, it is crucial to effectively utilize multi-modal features (audio and motion) in the LHG task. The previous works [7, 21, 54, 57] only concatenated audio and motion features, which is a coarse fusion fashion. Here, we design a novel AMAF module for finer interaction, as shown in Fig. 3 .In our view, audio conveys richer information in communication compared to motion, for example, semantic information is lacking in motion. In cross-modal fusion, we prioritize audio as the primary information stream, with motion serving as supplementary modality. Experimental results in Section 4.5.1 demonstrate that this approach performs better than regarding motion as the primary modality. Before the cross-modal fusion, to model the temporal relations of audio representation A 𝑡 , we feed it into a multi-head self-attention module. Then, enhanced representation A ′ 𝑡 interacts with motion representation M 𝑠 𝑡 in a cross-attention way, where the queries are from M 𝑠 𝑡 , keys and values are from A ′ 𝑡 , respectively:Interact(A ′ 𝑡 , M 𝑠 𝑡 ) = Softmax( M 𝑠 𝑡 𝑊 𝑄 • (A ′ 𝑡 𝑊 𝐾 ) T √ 𝑑 )A ′ 𝑡 𝑊 𝑉 (5)where 𝑊 𝑄 , 𝑊 𝐾 , 𝑊 𝑉 are learnable parameters and 𝑑 is a scaling factor.We expect that the motion feature plays a role as queries in the cross-attention mechanism, strengthening the audio representation closely associated with head motion and facial expressions to obtain a more comprehensive fused representation F 𝑡 . At last, a feed-forward layer is applied to output the fusion representation. Residual connection and layer normalization are employed after attention and feed-forward layers to ensure the training stability of the fusion process.Because there is a significant disparity in the lengths of videos in the training datasets (ranging from 1 to 71 seconds), the video clips are divided into fixed-length segments for training. This results in limited generalization ability of the transformer during the decoding phase for longer sequences. Methods to enhance the length extrapolation ability of transformers have garnered widespread attention. Existing approaches mainly fall into relative position encoding [38, 43, 44] , context window extension [1, 10, 50] , and so on. There is, however, a scarcity of methods specifically addressing seq2seq tasks. To address this issue for non-autoregressive ListenFormer, we explore three different decoding methods, corresponding to the three subfigures in Fig. 4 .Fig. 4 (a) represents the all-in decoding method, where the entire clip with length 𝑇 is inputted at once, and all predictions are generated in a single decoding step. This method not only results in high computational complexity 𝑂 (𝑇 2 ) but also yields poor performance due to limitations in extrapolation ability. Fig. 4 (b) represents the step-by-step decoding method. The input segment has a fixed window length 𝐿 and only one frame is slid in at each step. Meanwhile, the output of the last frame of each segment is concatenated to the final predictions. Although its computational complexity is reduced compared to the all-in approach, its 𝑂 (𝑇 𝐿 2 ) complexity can considerably slow down the decoding process when dealing with long videos. Furthermore, while this stepping approach performs well in autoregressive large language models (LLMs) [52] , it is not suitable for ListenFormer which computes all inputs' results in parallel. While each input is step-by-step, for non-autoregressive inference, even small changes in input can result in non-coherent outputs between each step. Therefore, the step-by-step method may lead to significant temporal jitter in the final output.Fig. 4 (c) represents the proposed sliding window with a large shift decoding method. Similar to the step-by-step method, the input window length 𝐿 remains fixed, but the sliding shift 𝑆 is expanded to approach the size of the window length 𝐿. A slight overlap helps to smooth the output at the junctions of segments. The outputs of non-overlapping frames are concatenated to the final prediction at each step. This method not only further reduces the complexity to 𝑂 (𝑇 𝐿), but also alleviates the jitter issue associated with the stepping method. To further maintain the predictions coherence, in the (b) and (c) methods, the reference image is replaced by the output of the beginning frame taken from the previous segment after the first step. More performance comparisons can be found in Section 4.5.3. 4.1.2 Evaluation Metrics. On the ViCo dataset, both the featurelevel and video-level metrics are applied for comprehensive comparison. For the former one, the L1 distance is employed to represent the disparity between the predicted angles, expressions, translation coefficients, and the ground truth. Angle and translation coefficients are two components that constitute the pose coefficient 𝑝. For the latter one, we adopt Fréchet Inception Distance (FID) [19] , Structural Similarity (SSIM) [51] , Peak Signal-to-Noise Ratio (PSNR), and Cumulative Probability of Blur Detection (CPBD) [4] . Additionally, to evaluate identity preservation, we utilize cosine similarity (CSIM) between identity features extracted from ArcFace [11] on generated and source videos.On the L2L dataset, due to the unavailability of the original videos, only feature-level metrics (L1 distance and Fréchet distance (FD)) for the expression and angle coefficients are applied. 
comparison methods.: 
 Five state-of-the-art responsive listener head generation methods are selected as comparing methods. ViCo [57] utilizes an LSTM-based sequential decoder to predict the pose and expression features of the listener subject. PCHG [21] Table 1 : The L1 Distance (×100) of different listening head generation methods on ViCo dataset. Each cell in the table represents the feature distance of angle/expression/translation coefficients respectively. Lower is better. The bold and underlined notations represent the Top-2 results. The * indicates that we directly follow the official report results of MFR-Net, while the results of other comparison methods are reproduced on our own system. modifies the post-processing approach during the rendering process based on ViCo. DSPN [54] is a dual-stream prediction network, which consists of LSTMs and temporal convolutional networks (TCN) [25] . MFR-Net [30] employs the probabilistic denoising diffusion model to predict multi-faceted response. L2L [35] learns a realistic manifold of listener motion through a novel sequenceencoding.4.1.4 Implementation Details. On the ViCo dataset, the input video frames are cropped to 256 × 256 size at 30 FPS and the audio signals are extracted into 45-dimensional acoustic features, including melfrequency cepstral coefficients (MFCC), energy, zero-crossing rate (ZCR), and loudness. The window length of the speaker clip is set to be 90 frames with a shift of 80 frames. The 3DMM coefficients are extracted with the guides of PIRender [39] . The identity-dependent features are in R 187 , and the identity-independent features are in R 70 . On the L2L dataset, the audio signals are extracted into 128-dim mel features. Following [35] , the parameters representing identityindependent features include 50 expression coefficients along with a 3D jaw rotation, as well as 3D head rotation in Euler angles.As for model details, we utilize 3 transformer encoder layers and 3 transformer decoder layers along with 4 attention heads. Due to the lack of original videos, the rendering part is not required when conducting experiments on the L2L dataset. Following [57] and [30] , results are presented for three different attitudes, along with their average values. Listenformer outperforms other existing methods on most metrics across three attitudes, with particularly outstanding performance on the D 𝑜𝑜𝑑 set. Specifically, it shows improvements of 3.75, 2.20, and 0.39 on average results of angle, expression, and translation coefficients, respectively. This could be attributed to the enhanced robustness of our non-autoregressive training and inference method, as well as the effectiveness of the proposed cross-modal fusion method in capturing representative information within audio and motion features. Meanwhile, various video-level metrics are displayed in Tab. 2. Listenformer achieves the best performance in SSIM, PSNR, and CSIM, with improvements of 0.03, 1.07, and 0.05, respectively. However, it slightly lags behind the state-of-the-art model MFR-Net in CPBD and FID. It could be due to the fact that MFR-Net has made improvements to the rendering model, giving it a certain advantage in facial reconstruction. Improving the rendering model of Listenformer is also one of our future research directions. 4.2.2 L2L dataset. Tab. 3 presents the feature-level results of our proposed method and other existing methods on the L2L dataset. One can see that the proposed method outperforms all state-of-theart approaches, which supports that the proposed non-autoregressive ListenFormer also exhibits significant advantages in modeling head motions and facial expressions on large datasets.To qualitatively evaluate different methods, we provide the responsive listening head frames generated by the proposed method and other methods in Fig. 5 . We can see that Listenformer provides a reasonable response, which may not align entirely with ground truth but remains generally consistent. Both ViCo and PCHG struggle to maintain accurate identity information, specifically in (a), where ViCo and PCHG model eye movements unnaturally, and in (b), where generated listeners consistently maintain a weird smile.Although DSPN doesn't exhibit the aforementioned glaring shortcomings, it lacks sensitivity in capturing the positive attitude in (a) and neutral attitude in (b). Conversely, our approach ensures the preservation of accurate identity information without visible artifacts. Furthermore, the generated videos present more natural facial expressions and more precise attitude conveyance. Please watch the supplementary video for the dynamic comparison.We invite 15 people to evaluate the generated listening head videos of our method with the other three methods. Each generated video along with its corresponding speaker's video is concatenated into the same video for presentation. 30 videos from the ViCo dataset are involved in this user study with human measures in overall naturalness (ON), motion diversity (MD), identity preserving (IP), speaker-listener synchronization (Sync), and attitude matching (AT). Except for attitude matching, which offers a choice between positive, negative, and neutral, the remaining four options are selected as the best among the four methods. The results from all participants are averaged and listed in Tab. 4. ListenFormer achieves the best performance among all subjective measures, especially in terms of motion diversity, identity preservation, and synchronization. This verifies the capability of our method in generating diverse and natural listening head videos.4.5.1 Effect of the audio-motion fusion module. In this section, we conduct experiments to compare three different fusion methods. "Concat" refers to directly concatenating audio and motion features. "Audio-dominated" denotes the proposed AMAF method in Section 3.4, while "motion-dominated" involves swapping the positions of audio and motion features in the AMAF module. Tab. 5 presents the performance of three methods on two datasets. It can be observed that the "audio-dominated" method outperforms the other two methods in all metrics. This not only indicates the effectiveness of the proposed audio-motion attention fusion method but also suggests that the speaker's audio may be more crucial than motion for LHG, as it can convey more information, such as semantics. to static facial expressions and induces slight but rapid back-andforth head movements. This is due to that the sinusoidal positional encoding fails to capture the modeling of position information for extended lengths. For the step-by-step method, although the generated facial expressions are no longer static, there are more pronounced back-and-forth head movements. As mentioned in Section 3.5, this may be attributed to the fact that for non-autoregressive ListenFormer, step-by-step inputs do not necessarily yield coherent results. The step-by-step approach introduces significant temporal jitters in the predictions, resulting in a visibly less smooth appearance. In comparison, our proposed method offers several advantages. On the one hand, the utilization of a sliding window helps to overcome the limitations associated with sinusoidal positional encoding for length extrapolation in decoding phase. On the other hand, the utilization of a large shift ensures that the generated frames do not exhibit jitters within the window. As a result, our method achieves superior visual quality compared to the other two methods. Additionally, it also leads to significant savings in computational resources according to Section 3.5. 
conclusion: 
 We introduce a novel transformer-based model for the responsive listening head generation task. Our proposed Listenformer achieves non-autoregressive inference through teacher-forcing pre-training and input-changed fine-tuning stage, ensuring consistency between training and inference prediction modes. Additionally, to provide more accurate responses to the speaker inputs, an audio-motion attention fusion method is proposed, which better captures the audio-motion correlation information in the speaker's signals. To further enhance performance, we propose a sliding window with a large shift approach to address infinite-length inference scenarios, which performs well in terms of both effectiveness and computational efficiency. Qualitative and quantitative experiments have validated the superiority of our method over other state-of-the-art methods in generating high-quality listening head responses. Limitations: The renderer and transformer are treated as independent components in our proposed method. In the future, we plan to explore joint optimization of these two components. Furthermore, we consider abandoning the rendering model and applying our method to 2D-based generation approaches.  
 
 Appendix:  


Decision: this paper investigates the responsive listening head generation and introduces a new approach, ListenFormer, to take full advantage of the temporal modeling capability of transformer. experiments on multiple datasets showcase competitive performance and ablation study verifies the effectiveness of the proposed method. 

the paper initially receives ratings of BR, BA, BA, BA. while there were concerns on clarity and novelty, other reviewers also praised this paper for its novel approach, the clear introduction of the proposed method, and thorough experiments. in rebuttal, the authors provided further clarification on some of the issues, and the overall ratings improved to BR, BA, WA, BA.

overall, the AC recommends Accept. since the author clearly introduced their new approach and findings, and verified them via thorough experiments, the AC believes it is worth sharing with the community. nonetheless, the authors are strongly encouraged to further improve their final manuscript based on the feedback from the reviewers.

Reviews: review: paper_summary: paper_summary: The authors propose Listenformer to predict the future listener heads based on the speaker inputs, a reference listener image and the past outputs. The experiments are conducted on the ViCo and L2L datasets. The main concern is the problem motivation of listening head generation. Compared with talking head generation, it is better for the authors to explain the motation of the problem of listening head generation. The second concern is the technical contribution of this paper, the audio-motion attention fusion method is not very novel and proposed non-auoregressive decoder is also not very novel.
strength_weakness: strength_weakness: The authors try the new problem of listener head generation.
limitations: limitations: 1. The motivation of listener head generation is not very clear.
2. The technical contributions of proposed non-autoregressive decoder with audio-motion attention fusion is not very novel.
3. The dataset size of ViCo and L2L is not large enough, compared with talking head generation problem.
paper_summary: paper_summary: The paper presents ListenFormer, a novel transformer-based model designed for generating responsive listening head videos. The primary contribution is the development of a non-autoregressive approach combined with an innovative audio-motion attention fusion module. This model effectively integrates cross-modal features and employs a sliding-window decoding method to address extrapolation limitations of transformers. Experiments on the ViCo and L2L datasets demonstrate the model's good performance in visual naturalness, generation diversity, and speaker-listener synchronization.
strength_weakness: strength_weakness: - Innovative Approach: The paper introduces a non-autoregressive transformer-based model and a novel audio-motion attention fusion module, showcasing significant advancements in responsive listening head video generation.
- Extensive Experimental Validation: The experiments are thorough, including quantitative and qualitative evaluations, user studies, and ablation studies, all of which validate the effectiveness of the proposed method.
- Clarity and Presentation: The paper is well-written and structured, making complex concepts accessible. Figures and tables are used effectively to enhance understanding.
limitations: limitations: - Reproducibility
paper_summary: paper_summary: The paper generates a listening head video based on speaker's audio and video as well as reference listener's image. It addresses limitations in existing autoregressive methods used in human-robot interactions by proposing a two-stage training approach that prevents error accumulation and improves generation speed. It integrates an audio-motion attention fusion module to better correlate audio and motion features, enhancing the model’s ability to generate realistic listener head videos.
strength_weakness: strength_weakness: - The model is evaluated extensively on the ViCo and L2L datasets, demonstrating improvements over state-of-the-art methods in several metrics. The evaluation includes a user study to validate model effectiveness in realistic scenarios.

- The paper introduces a non-autoregressive framework of ListenFormer over traditional autoregressive models
limitations: limitations: -  The two-stage training process, while effective, introduces additional complexity in training the model, requiring fine tuning. 
- The paper does not state if the model can run in real time. 
- The supp. video does not show generated videos with negative attitude.
paper_summary: paper_summary: This paper proposes an audio-motion attention fusion module, which improves the correlation of audio and motion features for accurate response, for listening head generation.
strength_weakness: strength_weakness: 1. The paper clearly describes each and every module of the proposed system.

2. The experiments have thorough ablation highlighting the usefulness of each component.

3. The ideas described in the paper appear novel.
limitations: limitations: 1. Lack of reproducibility. No source code was provided.

2. It would be useful to explain the evaluation metrics? Specifically what each of them capture.