# Multi-Level Feature Dynamic Fusion Neural Radiance Fields for Audio-Driven Talking Head Generation
## 4 Experiments
### Experimental Setting  
**Setting:** Our method only needs a 3-5 min video as training data for the model. It takes the discrete image sequence features of the video and the synchronized audio features as input to learn the mapping between dynamic faces and low-dimensional audio features. Given a continuous sequence of audio features during the inference process, we can query  
Figure 3: Visualization of the **audio-visual fusion module**. We display the attention map visualized through the norm of the attention vector in the generated head space image. the corresponding mapping parameters from the constructed model and render the talking head video frames synchronized with the audio stream. **Dataset:** We sourced all of our experiments' datasets from publicly released videos [6, 7, 34] to ensure the experiment's fairness and repeatability. We collected four high-quality speech-video segments with an average duration of about 6500 frames and a frame rate of 25 FPS. We cropped and resized each original video to dimensions of \(512\times 512\), and the videos provided in AD-NeRF [6] were processed to \(450\times 450\). Specifically, in addition to the [PERSON] video dataset shared by AD-NeRF, the remaining three videos were released by DFRF [7] and SSP-NeRF [34]. SSP-NeRF provides YouTube videos, which must be downloaded and cropped according to the URL. Previous studies [6, 7, 11, 12, 29, 34] have extensively used these portrait videos as training data. Moreover, following previous studies, we trained and evaluated the model by dividing each video into 11 equal parts. We used 10 of these parts for training and the remaining 1 part for evaluating the model's performance. **Head pose estimate:** In the audio-driven talking head generation task, we use the Euler angle approach [54] to estimate head posture, as the rotation amplitude of the speaking head is rather modest and does not exceed \(\pm 90\) degrees. During data preparation, we employ established tools from existing methods to calculate head posture. **Metrics:** Following the previous methods [11, 12, 36, 55], we evaluate the rendered video results using multi-dimensional quantitative indicators. To begin with, the peak signal-to-noise ratio (PSNR\(\uparrow\)) is used to measure overall image quality, the learned perceptual image patch similarity (LPIPS\(\downarrow\)) is used to evaluate details, and structure similarity (SSIM\(\uparrow\)) is used to weigh the degree of distortion between images. On this basis, we use landmark distance (LMD\(\downarrow\)) and the SyncNet confidence score (Sync\(\uparrow\)) to measure the synchronization of lip movements, and we use action unit error (AUE\(\downarrow\)) to calculate the error between facial movements. In high PSNR images, the texture details may be inconsistent with human visual perception. For better comparison, we use three no-reference evaluation methods: cumulative probabilistic blur detection (CPBD\(\uparrow\)), natural image quality evaluator (NIQE\(\downarrow\)), and image space quality evaluator (BRISQUE\(\downarrow\)). For these indicators, \"\(\downarrow\)\" means lower, better, and \"\(\uparrow\)\" means higher, better. More details are in Appendix A. **Implementation details:** In the PyTorch framework (v.1.12.1),we implement our MLDF-NeRF, which divides the modeling of the talking head generation into two stages. Talking head generation is trained for 100,000 iterations during the initial phase. In the subsequent phase, another 25,000 iterations are performed to further enhance the appearance of the eyes, mouth, and other facial features. During the training and inference process, each training iteration uses a 2D hash encoder to randomly sample \(256^{2}\) rays from an image. For model optimization, we use AdamW to achieve model convergence. The hash encoder has an initial learning rate of 0.01, while the other modules have an initial learning rate of 0.001. In this study, we conducted all experiments using an RTX A5000 GPU (NVIDIA, Santa Clara, CA, USA).