PerFRDiff: Personalised Weight Editing for Multiple Appropriate Facial Reaction Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 16:05:26

Abstract: Human facial reactions play crucial roles in dyadic human-human interactions, where individuals (i.e., listeners) with varying cognitive process styles may display different but appropriate facial reactions in response to an identical behaviour expressed by their conversational partners. While several existing facial reaction generation approaches are capable of generating multiple appropriate facial reactions (AFRs) in response to each given human behaviour, they fail to take human's personalised cognitive process in AFRs' generation. In this paper, we propose the first online personalised multiple appropriate facial reaction generation (MAFRG) approach which learns a unique personalised cognitive style from the target human listener's previous facial behaviours and represents it as a set of network weight shifts. These personalised weight shifts are then applied to edit the weights of a pre-trained generic MAFRG model, allowing the obtained personalised model to naturally mimic the target human listener's cognitive process in its reasoning for multiple AFRs generations. Experimental results show that our approach not only largely outperformed all existing approaches in generating more appropriate and diverse generic AFRs, but also serves as the first reliable personalised MAFRG solution. Our code is provided in the Supplementary Material.

Main paper: 
 introduction: 
 Human facial reaction plays a crucial role in social interactions, which conveys individuals' (called listeners in this paper) intentions and emotional states in response to behaviours expressed by their conversational partners (called speakers in this paper) [31] . As a result, the capability to promptly generate realistic and appropriate (i.e., contextually suitable) human-style facial reactions in realtime is essential in the development of humanoid virtual agents [46] , as it facilitates natural, engaging and empathetic interactions Figure 1 : Comparison between our PerFRDiff with existing approaches: (a) deterministic approaches which aims to reproduce the corresponding 'GT' real appropriate facial reaction (AFR) specifically expressed by a human listener for responding to the input speaker behaviour [8, 17, 52] ; (b) distribution-based approaches which aims to generate multiple generic real AFRs from the input speaker behaviour [20, 51] ; (c) personalised facial reaction generation approach which generates personalised AFRs by feeding personalised factors to the pre-trained network, enforcing the network to mimic the AFR expressed by the target listener [21] ; (d) our approach generates learns a set of unique weights to obtain a personalised network with personalised weights, which naturally simulates personalised cognitive process of the target listener for generating personalised AFRs.between human users and digital entities, thereby enhancing the user experience and interaction quality.Early facial reaction generation approaches [9, 11, 27, 49] frequently attempted to reproduce the paired real facial reaction (i.e., called the 'GT' real facial reaction) triggered by each target speaker behaviour, which treats the facial reaction generation task as a 'one-to-one mapping' problem. However, in real-world dyadic interaction scenarios, the same speaker behaviour can trigger different real facial reactions expressed by different human listeners [22] , and thus facial reaction generation is an actually 'one-to-many mapping' task [41] . For example, when facing a funny scenario, human listeners with extroverted personality might react with a broad smile, while listeners with introverted personality might offer only a slight grin. As a result, instead of reproducing the corresponding single GT real facial reaction for each speaker behaviour, Multiple Appropriate Facial Reaction Generation (MAFRG) approaches target at generating multiple diverse facial reactions that are appropriate for responding to each speaker behaviour have been widely developed in the past year [8, 17, 20, 39, 51, 52] . These approaches either learn a distribution [8, 19, 51, 52] or a codebook [17] from each input speaker behaviour, based on which a set of different but appropriate facial reactions (AFRs) can be sampled.While human listeners' facial reactions in response to their conversational partners are largely depending on their personalised cognitive processes [22, 28, 34, 38, 45] , i.e., personalised cognitive processes are shaped by their personalised cognitive styles referring to 'consistent individual differences in ways of organising and processing information and experience' [23, 43] , none of existing MAFRG approaches specifically considered this crucial factor in AFRs generation. In other words, personalised AFR generation remains under-explored (Problem 1). In this sense, we draw our attention to model personalisation which allows a network to fit on a specific target. Specifically, existing model personalisation approaches are mainly achieved by fine-tuning models on personalised data [2, 13, 13, 15, 26, 32, 34, 38, 50? ] or additionally taking encoded personalised factors as an additional input [5, 12, 33, 35, 48] , to produce personalised portrait image generation [26] , talking head [21] or lip sync video [15] . Although the latter approaches avoid the complex and time-consuming personalised fine-tuning for every target individual, as illustrated in Fig. 1 , they treat the encoded personalised factors as an external input to the model for personalised feature refinement, and thus still fail to naturally simulate human cognitive processes which are shaped by internal factors (e.g., cognitive style [43] and personality [38] ) in human facial reaction generation (Problem 2).In this paper, we propose the first online personalised MAFRG approach (called PerFRDiff) which can generate multiple diverse and appropriate facial reactions in response to each input human audiovisual speaker behaviour, where the personalised cognitive style of the listener is specifically considered. To address the 'one-to-many mapping' problem occurred in MAFRG task, our approach starts with training a diffusion-based Generic Appropriate Facial Reaction Generator (GAFRG), as its denoising process naturally allows multiple different outputs (e.g., facial reactions) to be sampled based on the same condition (e.g., the input speaker behaviour) via randomly picked and different input Gaussian noises [6, 18, 24, 30, 30, 36] , i.e., achieving 'one-to-many mapping' network via 'one-to-one training'. The obtained GAFRG aims to simulate generic AFRs in response to each input speaker behaviour, simulating generic/basic cognitive processes commonly shared by different listeners [25] . Then, a Personalised Cognitive Style Modelling (PCSM) module is proposed to model the target listener's personalised cognitive style from the listener's previous facial behaviours (i.e., a short historical face video), which is represented in the form of a set of personalised weight shifts (addressing Problem 1). These learned personalised weight shifts are then applied to edit pre-trained weights of the GAFRG, resulting in a personalised instance defined by personalised weights. Manipulating the weights according to the target listener alters the fundamental way in which the GAFRG perceives the input speaker behaviour as well as how facial reactions in the latent space are formed and transformed throughout the network. In other words, this strategy allows the obtained personalised instance to naturally simulate the corresponding listener's personalised cognitive processes involved in facial reaction generation in the form of its personalised weights (addressing Problem 2). The main contributions of this paper are summarised as follows:  
related work: 
 Early deterministic facial reaction generation approaches [9, 11, 27, 49] were developed to reproduce the GT real facial reaction from each input speaker behaviour. For example, Huang et al. [10] presented a conditional Generative Adversarial Network (GAN) to generate facial reaction sketches conditioned on the facial expressions expressed by the conversational partner, which are expected to match the face sketches extracted from the paired GT real facial reaction. Woo et al. [49] utilised Recurrent Neural Networks (RNNs) to generate a multi-dimensional facial reaction signal (e.g., facial expressions, head motion, posture, etc.) in a real-time dyadic interaction. Song et al. [34, 38] simulated each listener's personalised cognitive processes using a personalised convolutional neural network (CNN) to reproduce the GT real facial reaction.. Recently, several approaches have been proposed to generate multiple diverse but appropriate facial reactions in response to each speaker behaviour. Specifically, Xu et al. [51] and Luo et al. [20] learn a distribution (i.e., Gaussian Mixture Model (GMM) / Gaussian distribution) from each input speaker behaviour, where multiple different AFRs in response to the given speaker behaviour can be sampled from it. This idea has been followed by multiple approaches [8, 52] presented in the REACT2023 challenge [39] . For example, the BEAMER [8] extended the Transformer-based Variational Autoencoder (VAE) architecture provided in [39] to predict a latent AFR distribution for each speaker behaviour. A contrastive loss between the speaker behaviour representation and the predicted facial reaction representation is computed to facilitate the model training. The winner [17] of the challenge represents a finite set of real facial reactions provided in the training set through a codebook, from which top-k AFRs are selected for responding to the input speaker behaviour. 
task definition: 
 In dyadic interaction scenarios, the goal of the Personalised Multiple Appropriate Facial Reaction Generation (PMAFRG) task is to learn a machine learning (ML) model that can generate multiple (𝑀) different but appropriate personalised facial reactionsR 𝑙 = { R𝑙 (1), • • • , R𝑙 (𝑀))} in response to the given speaker behaviour 𝐵 𝑠 . This can be formulated as:EQUATION)where 𝐹 𝑙 ℎ denotes an arbitrary facial behaviour previously expressed by the target listener 𝑙 (i.e., historical personalised facial behaviour). Here, the generated R𝑙 ∈ R 𝑙 should be similar to at least one real AFR expressed by the target listener 𝑙 (i.e., a personalised real AFR 𝐹 𝑙 (𝑛) ∈ F 𝑙 ) provided in the training set as:EQUATIONwhere F 𝑙 represents a set of real facial reactions expressed by the target listener 𝑙 in response to human behaviours that are similar to the given speaker behaviour 𝐵 𝑠 . In particular, the online PMAFRG model is expected to predict each personalised AFR R𝑙 (𝑚) in a progressive way that it predicts the current AFR segment R𝑙 [𝑡 -𝑤+1:𝑡 ] (𝑚) (i.e., represented by a small video segment consisting of 𝑤 frames at the time interval [𝑡 -𝑤 + 1 : 𝑡]) belonging to R𝑙 (𝑚) based on the previous speaker behaviour 𝐵 𝑠 expressed at the time [1 : 𝑡 -𝑤] and a historical facial behaviour 𝐹 𝑙 ℎ of the target listener as:EQUATION)where 𝑤 simulates the time delay due to the execution of human cognitive processes [4, 38] . The progressively predicted facial reaction segments { R𝑙[1:𝑤 ] (𝑚), R𝑙 [𝑤+1:2𝑤 ] (𝑚), • • • R𝑙 [𝑡 -𝑤+1:𝑡 ] (𝑚)} along the time [1 : 𝑡] forms a complete facial reaction R𝑙 [1:𝑡 ] (𝑚) in re- sponse to the speaker behaviour 𝐵 𝑠 [1:𝑡 ] .This task is different from the general MAFRG task defined in [41] and the general online MAFRG task implemented in [20, 41] , which only requires each generated AFR to be similar to at least one real AFR that can be expressed by different individuals in response to 𝐵 𝑠 . Please refer to [41] for more details. 
methodology: 
 Overview: Given an audio-visual speaker behaviour 𝐵 𝑠 To achieve this, we pre-process the input speaker facial behaviour 𝐹 𝑠 [1:𝑡 -𝑤 ] into two sets of frame-level facial features: 3D Morphable Model (3DMM) coefficients [47] 𝐸 m [1:𝑡 -𝑤 ] representing the facial appearance, as well as facial emotional representations 𝐸 e (i.e., action units (AUs), facial expression probabilities, valence and arousal intensities) [19, 42] . The raw speaker audio signal 𝐴 𝑠 The Personalised Cognitive Style Modelling (PCSM) module models the target listener's personalised cognitive processes to enable the simulation of the target listener's personalised cognitive processes for generating more appropriate and realistic facial reactions. This is achieved by learning a set of personalised weight shifts to represent the target listener's personalised cognitive style 𝑝 𝑙 that shapes the listener's personalised cognitive processes in facial reaction generation [4, 43] . Specifically, our PCSM module starts with a Personalised Style Space Learning (PSSL) block which encodes each target listener's historical facial behaviour 𝐹 𝑙 ℎ into a latent space Z in which the personalised cognitive styles 𝑝 𝑙 can be modelled as:EQUATIONThen, a Personalised Weight Shift Generation (PWSG) block is introduced to represent 𝑝 𝑙 in the form of a set of weight shifts Δ𝜃 to produce a personalised instance of the GARFG module.The PCSM module is built on the hypothesis that the personalised cognitive style 𝑝 𝑙 of the listener 𝑙 is consistent across different facial behaviours F 𝑙 = {𝐹 𝑙 (1), 𝐹 𝑙 (2), • • • , 𝐹 𝑙 (𝑁 )} expressed by this listener, but different from the personalised cognitive style of other listeners [23] ., i.e., cognitive styles 𝑝 1 , • • • , 𝑝 𝐿 reflected by facial reactions 𝐹 1 (𝑛), • • • , 𝐹 𝐿 (𝑛) expressed by different listeners for responding to the same speaker behaviour 𝐵 𝑠 (𝑛) should be different. This can be defined as:EQUATIONwhere 𝑝 𝑙 (𝐹 𝑙 (𝑛)) denotes the cognitive style reflected by the facial reaction 𝐹 𝑙 (𝑛). To obtain this personalised cognitive style from a target listener's historical facial behaviour video 𝐹 𝑙 ℎ , our PSSL is learned to map all facial behaviours to a latent space Z, where the latent representations extracted from facial behaviours expressed by the same listener are pulled together while latent representations extracted from facial behaviours belonging to different listeners are pushed apart, i.e., latent representations in Z should meet Eqa. 6 . More details for training our PSSL block are explained in Sec. 4.3. To avoid this process being influenced by identities of listeners, which are also consistent in different face videos of the same listener but different across various listeners, our PSSL first extracts a set of identity-free facial behaviour attributes 𝜓 𝑙 ℎ (e.g. 3DMM and Action Units (AUs)) from each frame of the given historical face video 𝐹 𝑙 ℎ (as shown in Fig. 3 ), aiming to model the personalised cognitive style underlying the facial behaviours regardless of the listener's identity. Then, the 𝜓 𝑙 ℎ is encoded as an embedding 𝐸 𝑝 through a Transformer encoder Enc p [44] to capture long-range dependencies within the given sequential behaviour, as the target personalised cognitive style 𝑝 𝑙 should be consistent across the entire video 𝐹 𝑙 ℎ [23] . Finally, the obtained 𝐸 𝑝 is projected into the target latent space Z through a projection head (i.e., a linear layer) as the required 𝑝 𝑙 .Personalised Weight Shifts Generation (PWSG): Based on the personalised cognitive style 𝑝 𝑙 , the PWSG block simulates the corresponding listener's cognitive processes involved in the facial reaction generation by learning a set of unique and personalised weight shifts Δ𝜃 𝑙 from 𝑝 𝑙 via a multi-branch network as:EQUATIONThese personalised weight shifts are then applied to transform the pre-trained generic appropriate facial reaction generator (GAFRG) as a personalised instance (i.e., a personalised appropriate facial reaction generator) GAFRG 𝜃 𝑙 representing the personalised facial reaction cognitive process of the listener 𝑙. Here, each branch of the PWSG network generates a set of weight shifts Δ𝑊 𝑙 𝑘 to personalise the 𝑘 th layer GAFRG 𝑘 in the GAFRG.The PAFRG module aims to naturally simulate the personalised cognitive process of the target listener in MAFRG task without requiring time-consuming personalised fine-tuning. Specifically, it first pre-trains a generic Appropriate Facial Reaction Generator (GAFRG) that is capable of generating multiple generic AFRs in response to each input speaker behaviour, simulating generic cognitive processes commonly shared by different listeners. Then, the obtained 𝑝 𝑙 representing the target listener 𝑙's personalised cognitive style is applied to edit the GAFRG's weights, producing a personalised instance GAFRG 𝜃 𝑙 , which particularly simulates the personalised cognitive processes of the listener 𝑙 in this listener's facial reaction generation.Generic Appropriate Facial Reaction Generator: Separately training/fine-tuning a personalised facial reaction generator for each listener is challenging due to the need for not only a large amount of interaction data expressed by every target listener but also substantial computational costs. Since there are commonly shared facial reactions that can be expressed by different human listeners in response to the same speaker behaviour (i.e., some facial reaction styles are commonly shared by various human listeners), we first learn a GAFRG aiming to generate generic AFRs that can be expressed by different listeners, simulating the commonly shared generic cognitive processes involved in facial reaction generation. To address the 'one-to-many mapping' problem occurring in MAFRG models' training, the employed GAFRG is designed as a transformer-based diffusion model [44] which reformulates the 'one-to-many mapping' problem as a 'one-to-one mapping' task by denoising each noisy real AFR (ground-truth provided in the training set) degraded by a random Gaussian noise to itself (a real AFR) expressed by a real listener in response to the speaker behaviour 𝐵 𝑠 [1:𝑡 -𝑤 ] during training. This process comprises two main steps: (i) a forward diffusion process where small portions of Gaussian noise are progressively added to a real AFR segment 𝐹 ( [𝑡 -𝑤+1:𝑡 ],0) in 𝐷 steps, resulting in a noisy AFR segment 𝐹 ( [𝑡 -𝑤+1:𝑡 ],𝐷 ) at the final step 𝐷; and (ii) a reverse (denoising) process where the GAFRG denoises 𝐹 ( [𝑡 -𝑤+1:𝑡 ],𝐷 ) step-by-step to recover the original AFR segment 𝐹 ( [𝑡 -𝑤+1:𝑡 ],0) expressed at the time [𝑡 -𝑤 + 1 : 𝑡]. At each reverse step, given the speaker behaviour 𝐵 𝑠 [1:𝑡 -𝑤 ] and a noisy version of AFR segment 𝐹 ( [𝑡 -𝑤+1:𝑡 ],𝑑 ) obtained at the 𝑑-th diffusion step, the GAFRG learns to predict the added noise 𝜖 𝜃 (𝐹 ( [𝑡 -𝑤+1:𝑡 ],𝑑 ) , 𝐵 𝑠 [1:𝑡 -𝑤 ] , 𝑑) at the 𝑑-th diffusion step and removes it from 𝐹 ( [𝑡 -𝑤+1:𝑡 ],𝑑 ) , reversely obtaining 𝐹 𝑡 -𝑤+1:𝑡,𝑑 -1 . The reverse process is guided by the input speaker behaviour 𝐵 𝑠 [1:𝑡 -𝑤 ] expressed at the time [1 : 𝑡 -𝑤] as the condition (i.e., as the key and value), according to which the latent representation of the predicted original AFRs (i.e., as the query) is modified via cross-attention operations. This process encourages the GAFRG to understand the relationships between the speaker behaviour 𝐵 𝑠 Simulating personalised cognitive process for AFRs generation: To consider personalised cognitive style of the target listener 𝑙 in generating multiple AFRsR 𝑙 = { R[1:𝑡 ] (1), • • • , R[1:𝑡 ] (𝑚)} in response to the speaker behaviour 𝐵 𝑠 [1:𝑡 ] ,we apply the personalised weight shifts Δ𝜃 learned from the target listener 𝑙's personalised cognitive style 𝑝 𝑙 , to edit the weights of the obtained GAFRG 𝜃 . This results in a personalised instance of GAFRG, denoted as GAFRG 𝜃 𝑙 , with its personalised weights fundamentally determining the way that GAFRG 𝜃 𝑙 interprets the input speaker behaviours and forms a set of AFRs throughout the network. Let 𝜃 be the weights of the pre-trained GAFRG, this process can be formulated as:EQUATIONwhere 𝜃 𝑙 denotes the weights of a personalised instance of GAFRG module representing the listener 𝑙's cognitive process, and can be obtained by repeating the consecutive prediction process.Δ𝜃Our PerFRDiff is trained via a two-stage strategy, where the first stage includes GAFRG and PSSL's training, while the second stage focuses on training the PWSG block.Pre-training GAFRG: The GAFRG is jointly trained with the SBE module by performing a reverse diffusion process where the GAFRG denoises a noisy facial reaction 𝐹 ( [𝑡 -𝑤+1:𝑡 ],𝑑 ) to recover the original clean real AFR 𝐹 ( [𝑡 -𝑤+1:𝑡 ],0) triggered by the speaker behaviour 𝐵 𝑠 [1:𝑡 -𝑤 ] expressed at the time [1 : 𝑡 -𝑤], which is supervised by optimising an MSE loss as:EQUATION)where F([𝑡 -𝑤+1:𝑡 ],0) (𝐹 ( [𝑡 -𝑤+1:𝑡 ],𝑑 ) , 𝑐, 𝑑) denotes the predicted original clean AFR based on the 𝐹 ( [𝑡 -𝑤+1:𝑡 ],𝑑 ) in the reverse process, and 𝑐 denotes the conditions including the speaker behaviour 𝐵 𝑠 ; F = {F 1 ∪ F 2 ∪ • • • ∪ F 𝑙 }be a union set of 𝑋 real AFRs expressed by 𝐿 different listeners in response to 𝐵 𝑠 ; 𝑖 ∈ 𝐼 ≡ {1 . . . 𝑋 } be the index of a real AFR in F; 𝐺 (𝑖) ≡ 𝐼 \{𝑖} be the set of indices excluding 𝑖; and 𝑄 (𝑖) denotes the set of indices of AFRs belonging to the same listener as the 𝑖-th AFR and 𝑖 ∉ 𝑄 (𝑖). The training process of the PSSL block is achieved by minimising a contrastive loss [14] as:EQUATIONwhere 𝑝 (𝑖) denotes the personalised cognitive style derived from the 𝑖-th AFR in F, |𝑄 (𝑖)| denotes the cardinality of 𝑄 (𝑖), and 𝜏 is a temperature parameter. This loss maximises the similarity between each pair of the personalised cognitive styles (𝑝 (𝑖), 𝑝 (𝑞)) modelled from real AFRs belonging to the same listener.Training PWSG: In this stage, we integrate the PSSL and PWSG blocks into the GAFRG, then freeze the well-trained PSSL block and GAFRG. Subsequently, we employ the loss defined in Equation 10to optimise the PWSG block. Let 𝑊 𝑘 ∈ 𝜃 be the weight matrix of the 𝑘-th layer GAFRG 𝑘 to be edited in the GAFRG and Δ𝑊 𝑙 𝑘 be the personalised weight shifts generated by the PWSG block. The forward propagation of the updated layer GAFRG 𝜃 𝑙 𝑘 is:EQUATIONwhere 𝑧 𝑙 𝑘 denotes the output of the target layer GAFRG 𝜃 𝑙 𝑘 ; 𝑏 𝑘 denotes the biases at the target layer GAFRG 𝜃 𝑙 𝑘 , and 𝜎 is an activation function. Correspondingly, the gradient of the loss L 1 with respect to the personalised weight shift Δ𝑊 𝑙 𝑘 generated by the PWSG block is computed using the rule chain as:EQUATIONwhere (𝑧 𝑙 𝑘 -1 ) 𝑇 denotes the transpose of 𝑧 𝑙 𝑘 -1 , and 𝜎 ′ denote the derivative of the activation function 𝜎. The gradient information𝜕 L 1 𝜕 (Δ𝑊 𝑙 𝑘 )can be further back-propagated through the PWSG block for the purpose of training. It is worth noting that the weights of the GAFRG are only updated based on the weight shifts generated by the PWSG block in the forward propagation at this stage, rather than being optimised in the backpropagation. 
experiments 5.1 experimental settings: 
 Dataset: Our PerFRDiff is evaluated on a publicly accessible audiovisual dyadic interaction dataset provided by REACT2024 challenge [40] 1 . It contains 2962 pairs of human speaker-listener dyadic interaction clips (including 1593 training pairs, 562 validation pairs and 806 test pairs) recorded under various contexts, with each clip lasting for 30 seconds. These clips are originally recorded by two datasets: NoXI [3] and RECOLA [29] . Implementation Details: We first crop the face region from every frame of the given clips and resize it into 224 × 224. Then, our PerFRDiff is trained with the Adam [16] optimizer with the initial learning rate of 0.0001, coupled with a cosine-annealing schedule, for optimizing GAFRG and PSSL. Additionally, we employ the SGD optimizer with 𝑙𝑟 = 0.001 to train our PWSG. Our diffusionbased GAFRG benefits from the classifier-free guidance [7] which enhances the balance between the quality and diversity of the generated facial reactions. The sampling process is implemented using a DDIM sampler [37] . All experiments are conducted on Nvidia A100 GPUs using PyTorch. More details about implementation and datasets are available in the Supplementary Material.Metrics: Following previous challenges [39, 40] , we evaluate four aspects of the generated AFRs, including: appropriateness (measured by FRCorr and FRdist), diversity (i.e., FRDiv, FRDvs and FRVar, where FRDiv measures the diversity of multiple generated AFRs in response to the same speaker behaviour), realism (FRRea) and synchrony (FRSyn). Please refer to [41] for more details about these metrics. This section conducts a series of ablation studies to evaluate various aspects our PerFRDiff. Additional analysis of three types of inputs, statistical difference analysis, model complexity analysis, and other analysis are provided in the Supplementary Material. Contributions of different modalities: Table 2 evaluates the importance of audio and visual behaviours expressed by the in generating AFRs. It can be observed that using both audio and visual modalities facilitates more appropriate and diverse AFRs generation, as indicated by the highest FRCorr and lowest FRdist. While both Effectiveness of personalised cognitive style modelling: Table 3 shows that the best appropriateness results of the generated facial reactions on both tasks are achieved by the personalised instance GAFRG 𝜃 𝑙 achieved by our approach, which naturally simulates the personalised cognitive processes (through weight editing) of the target listener involved in facial reaction generation. This validates our assumption that additionally considering individual differences (i.e., cognitive styles and cognitive processes) is crucial in generating more appropriate facial reactions (high correlations and fewer distances with corresponding real AFRs). More importantly, a common strategy which treats the personalised factors as an external input, i.e., directly feeding the obtained personalised style representation to the MAFRG (Solution 3 in Fig. 1 ), shows much less effectiveness than our strategy, which further validate our novel cognitive process simulation strategy is suitable for MAFRG and PMAFRG tasks.Weight editing of different layers: Table 4 reports the influences caused by editing the weight matrices of different layers within the GAFRG 𝜃 𝑙 on the MAFRG and PMAFRG tasks. Specifically, the GAFRG layers to be edited include the self-attention (self-attn) block, cross-attention (cross-attn) block, feed-forward block (within the transformer decoder) and the mapping layer (Fig. 2). Our observations are presented as follows: (i) editing the crossattention block is crucial as it directly interprets the input speaker behaviour, and thus decides the initial understanding of the condition; and (ii) editing multiple components generally enables more appropriate personalised facial reaction generation, compared to editing a single component. We assume this as human facial reaction generation requires multiple steps where there could be more than one step that is person-dependent (not commonly shared by different listeners), and thus personalising/editing multiple components could better simulate such processes of the target listener.Influences of the historical facial videos' lengths/contents: Table 5 first demonstrates that the performances achieved for three different length settings of the historical facial video are similar (with FRCorr around 0.36, FRdist around 98.81, and FRDiv around 13.6), suggesting that our PCSM module can effectively model the personalised cognitive style of the target human listener based on a relatively short facial behaviour video (e.g., 10s). As demonstrated in Table 6 and TSNE figures in Fig. 4 , these modelled personalised cognitive styles are also relatively invariant and robust to behaviours expressed by the target listener, regardless of the contents of the expressed facial behaviours, i.e., the personal cognitive styles learned by our approach well meet the rules described in Eqa. 6. 
conclusion: 
 This paper proposes the first online personalised MAFRG approach that can naturally simulate human listener's personalised cognitive processes in the form of a diffusion-style network with personalised weights. Results demonstrate that our approach achieved substantial improvements in both the appropriateness and diversity of the generated facial reactions compared to previous state-of-the-art approaches. Our strategy can also effectively and robustly model personalised cognitive style from an arbitrary historical facial video expressed by the target listener, based on which more appropriate facial reactions can be generated. The key limitation is that the relatively high complexity of our model when a large number of weights need to be edited, which will be addressed in future work.  
 
 Appendix: https://sites.google.com/cam.ac.uk/react2024/home 


Decision: 2xWA, 1xBA and 1xA ratings
The reviewers appreciate the novelty of the proposed appropriate reaction generation technique. The rebuttal seems to address the queries.

Reviews: review: paper_summary: paper_summary: The paper proposes a method for listeners' facial reaction generation in dyadic interactions. The method learns online the personalised cognitive style of the target listener. It incorporates that information into a pre-trained generic multiple-appropriate facial reaction generation model to mimic the target listener's behaviour. The paper reports significant improvements in this task using the REACT2024 challenge dataset.
strength_weakness: strength_weakness: - The paper proposes an online personalisation approach which leads to significant improvements
- The paper provides a thorough technical description of the proposed methodology
- The paper provides extensive evaluation and discussion of the obtained results
limitations: limitations: - I found the notation used in the paper difficult to follow and over-complicated. For the sake of readability, the paper could benefit from simplified and intuitive notations which will enable smoother reading and comprehension experience.

- Please check the formatting as in several places the text is not well formatted.

- In the provided demo video, it seems that the generated behaviours also involve considerable camera rotations and transloations. This is not observed in the other methods presented. It would be important to discuss why the output is so much influenced by the 'camera rotation' (which I guess happens because of the image cropping) and how this can be mitigated.
paper_summary: paper_summary: The paper presents a novel facial reaction generation approach that takes into account the human’s (specifically the listener's ) personalised cognitive process in appropriate facial reaction (AFR) generation. This approach learns a unique personalised cognitive style from the target human listener’s previous facial behaviours which is used in AFR generation.
strength_weakness: strength_weakness: - The problem addressed here is novel, interesting and relevant to the scope of the conference. It is interesting and critical to incorporate the listener's personal cognitive process while the facial reactions are generated. 
- The work is well presented and problem statement is well motivated.
- The Experiments are sufficient and the results discussion is interestingly presented
limitations: limitations: -There are too many notations. Defining the notations in a table might be useful for the readers to follow the explanations.
-What does 'GT' stand for in 'GT real facial reaction'
-In addition to quantitative assessment, incorporating user evaluation to assess the diversity and appropriateness of facial reaction generation would be beneficial.
- The facial reaction segments in line 570 is incomplete
- The manuscript should be proofread to correct grammatical errors and typos. Some are given below
-Typo line no 140 - [2, 13, 13, 15, 26, 32, 34, 38, 50? ]
- Typo in reaction.. 218
paper_summary: paper_summary: In this paper, the first online personalised multiple appropriate facial reaction generation (MAFRG) approach is proposed which learns a unique personalised cognitive style from the target human listener’s previous facial behaviours and represents it as a set of network weight shifts. These personalised weight shifts are then applied to edit the weights of a pre-trained generic MAFRG model, allowing the obtained personalised model to naturally mimic the target human listener’s cognitive process in its reasoning for multiple AFRs generations. The authors achieve state-of-the-art results, showing the effectiveness of their method.
strength_weakness: strength_weakness: - The proposed approach that uses personalised weight shifts to edit the weights of a pre-trained generic MAFRG model is novel.
- The authors achieved state of the art results on several aspects of the generated AFRs.
- The paper is well-written in an understandable manner.
- The visual representation is good and explains the proposed methodology quite well.
limitations: limitations: - What is the use case of such a model that generates personalised facial reactions? 
- How much is the computational time? Is the method feasible in real-time?
- There are minor typing errors. The content in line 560 is cut.
paper_summary: paper_summary: This paper proposes a personalized multiple facial reaction generation strategy via network weight shift.
strength_weakness: strength_weakness: The personalized weight shift module in proposed PerFRDiff framework is novel. Given any speakers' context and speaking information, the listener reacts according to their personality and cognitive processing of the situation. Thus, the weight editing accordingly is realistic approach.
limitations: limitations: 1. From the paper, it is not clear what does it mean by cognitive style. Given a very short interaction video, what are the verbal and nonverbal cues that can be used for representing 'cognitive style'. 
2. Also, how to get the facial behavioural history over a short interaction video is not clear.
3. If the personalized cognitive style is derived from 10/30 sec video, how does it relate with multiple reaction generation. Each target listener has one cognitive style.
4. Also, given any context there could be multiple possible reactions (seems like open ended problem). Then, how it is relatable with ground truth for training and evaluation. Please clarify this aspect. 
5. Minor issue: the opening quote is in opposite direction in few places, please edit this typo. Also, there is a question mark in line 140.