# Multi-Level Feature Dynamic Fusion Neural Radiance Fields for Audio-Driven Talking Head Generation
## 4 Experiments
### Quantitative Evaluation  
To verify the performance of the model, we evaluate it based on three criteria: head reconstruction, lip synchronization, and overall portrait quality. (1) In talking head generation, head reconstruction is a complex and important part of the generation of video keyframes. Therefore, we initially partition each video dataset into a training set and a test set to assess the quality of head reconstruction of the model. Simultaneously, we evaluate our proposed approach by comparing it to typical one-shot and special-person methods. Among them, the one-shot techniques based on one image are Wav2 Lip [20] and DiNet [56], and the special-person methods are based on NeRF, AD-NeRF [6], RAD-NeRF [11], GeneFace [45], and ER-NeRF [12]. In Table 1, for cross-modal head reconstruction, we evaluate the performance of our proposed model by comparing it with the results of different baseline models. We use a fresh video as input to ensure a fair experimental comparison, as Wav2 Lip and DiNet synthesize based on the complete original frame of the input. Therefore, the PSNR and LPIPS metrics are not applicable for evaluating these two baselines. Although one-shot methods achieve excellent audio lip sync using lip sync discriminators and can generate talking head videos from speech without training, they cannot synthesize videos with specific identities and natural movements. AD-NeRF cannot capture complex talking head details, resulting in high LPIPS and LMD scores in head reconstruction due to the limitation of vanilla NeRF. GeneFace is inefficient at modeling specific individuals due to its heavy reliance on preexisting knowledge during the pre-training phase. RAD-NeRF and ER-NeRF are new methods based on Instant-NGP, which improves the model's inference speed. However, these two methods perform poorly on indicators such as PSNR and LMD, which may be due to their lack of an effective cross-modal fusion mechanism. Compared to RAD-NeRF, ER-NeRF has a more complex model architecture, which limits the model's reconstruction effect in detail. Our proposed method obtains higher scores than other baselines in most quantitative indicators and the best sync score among NeRF-based methods. (2) To evaluate the accuracy of the lip shape of the audio-driven talking head generation task, we extract two unlearned audio clips from public videos provided by DFRF [7] and SSP-NeRF [34], respectively, and name them Audio A and Audio B. Since the 2D-based one-shot method has specific training for the lip shape part, this type of method has a higher score in lip shape accuracy. To better verify the accuracy of the lip shape of our model, we use [PERSON]'s speech video as training data and use NeRF-based methods as baselines, including AD-NeRF, RAD-NeRF, GeneFace, and ER-NeRF. However, to ensure \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Methods**} & **PSNR\(\uparrow\)** & **LPIPS\(\downarrow\)** & **SSIM\(\uparrow\)** & **LMD\(\downarrow\)** & **AUE\(\downarrow\)** & **Sync\(\uparrow\)** & **CFBD\(\uparrow\)** & **NIQE\(\downarrow\)** & **BRISQUE\(\downarrow\)** \\ \hline \multirow{2}{*}{GAN-based} & Wav2 Lip [20] & - & - & - & 5.241 & 4.361 & **9.814** & 0.169 & 15.172 & 42.263 \\  & DiNet [56] & - & - & 0.856 & 4.172 & 3.287 & 7.365 & 0.207 & 15.384 & 44.465 \\ \hline \multirow{2}{*}{\
\begin{tabular}{c} AD-NeRF [6] \\ RAD-NeRF [11] \\ NeRF-based GeneFace [45] \\ ER-NeRF [12] \\ **MLP-NeRF** \\ \end{tabular} } & 30.64 & 0.1145 & 0.889 & 3.218 & 3.874 & 5.351 & 0.153 & 16.709 & 52.467 \\  & 35.37 & 0.0934 & 0.945 & 2.703 & 3.596 & 6.426 & 0.179 & 15.443 & 43.893 \\ \multirow{2}{*}{NeRF-based GeneFace [45]} & 30.15 & 0.0917 & 0.875 & 3.356 & 3.965 & 6.531 & 0.177 & 15.335 & 45.507 \\  & 35.43 & 0.0238 & 0.950 & 2.623 & 2.832 & 7.034 & 0.201 & 14.931 & 39.268 \\ \multirow{2}{*}{
\begin{tabular}{c} ER-NeRF \\ **MLP-NeRF** \\ \end{tabular} } & **35.74** & **0.0187** & **0.963** & **2.484** & **2.752** & 7.139 & **0.211** & **14.920** & **38.316** \\ \hline \hline \end{tabular}
\end{table}