# Multi-Level Feature Dynamic Fusion Neural Radiance Fields for Audio-Driven Talking Head Generation
## 3 Method
### Efficient Audio-Visual Fusion Module
#### Audio data:  
We use a discrete-continuous approach to convert video data to image data. Meanwhile, we use existing ASR models [42; 49; 50; 51] to predict the classification logitsof each 20 ms audio segment and consider them to be its audio features, as shown in Figure 2. We do this to align audio features with discrete-continuous image features. To maintain more stable coherence of features across different video frames in the temporal dimension, we introduce an audio-filtering module based on self-attention. In addition, we adopt the sliding window method to linearly weight the features in the duration interval \([t-S/2,t+S/2]\) as the audio features of the video frame \(t\), where \(S\) represents the size of the sliding window.  
During the model training process, we will gradually optimize and learn the dependency between the attention vectors \(V\) of different planes and the audio features, particularly in the part where the facial motion features change with the audio features \(f_{a}\). This optimization aims to present better potential connections between different regional features and audio features. Since the static portion is unaffected by any factor, we set the attention vector \(V\) of different planes to zero to minimize the impact of unnecessary information on the lip area during rendering.  
\[F_{f_{a},X} =V_{a_{s},X}\odot a_{s} \tag{14}\] \[F_{f_{a},Y} =V_{a_{s},Y}\odot a_{s}\] (15) \[F_{f_{a},Z} =V_{a_{s},Z}\odot a_{s}\] (16) \[F_{a,lip} =F_{f_{a},X}\oplus F_{f_{a},Y}\oplus F_{f_{a},Z} \tag{17}\]  
where \(\odot\) denotes the Hadamard product, \(\oplus\) denotes the concatenation of characteristics, and \(F_{a,lip}\) denotes the fusion of different characteristics of the orthogonal plane of the 3D coordinate point x and the audio characteristics.  
**Eye blink:** Normal blinking is a physiological reaction that people use in daily communication. Similarly, natural blinking movements are also an indispensable part of dynamic head modeling in the talking head generation task. In data preprocessing, we adopt a continuous discrete method and use the existing OpenFace tool [52] to extract blink data from the video. Similarly, to simplify the calculation process, we use a dual-layer MLP to combine the feature dimensions of 3D coordinate points with the dimensions of blink data. During model training, we incorporate a scalar to quantify the natural variations in the motions of the eyes. During the computation procedure, we treat the scalar as a  
Figure 2: **Audio feature extraction:** Using the sliding window approach, we split the provided audio stream into several parts. Following the ASR classification of each audio clip, we apply 1D convolution and self-attention modules to temporally smooth the audio clips in order to produce the final audio feature \(a_{s}\).  
one-dimensional vector, denoted as \(f_{e}\). Unlike audio features, the activation function of the sigmoid layer generates the eye region attention vector.  
\[V_{f_{e}}=MLP_{f_{g}}\left(f_{g}\right) \tag{18}\]  
\[F_{e,out}=f_{e}\cdot Sigmoid(V_{f_{e}}) \tag{19}\]  
In this process, the spatial coordinate point feature \(V_{f_{e}}\) primarily influences the value of the blink features \(F_{e,out}\). To achieve a natural blinking action, the eye feature approaches \(f_{e}\) when \(V_{f_{e}}\) significantly affects the eye motions; otherwise, the blink feature will approach the zero vector to avoid interference from irrelevant information.  
Among them, Figure 3 shows the impact of these two audio-visual fusion practices.