# Multi-Level Feature Dynamic Fusion Neural Radiance Fields for Audio-Driven Talking Head Generation
## 3 Method
### Multi-Level Tri-Plane Hash Representation  
Instant-NGP [8] proposes a novel neural representation that spreads feature grids in hash tables. Meanwhile, RAD-NeRF [11] has successfully used this approach to generate talking heads. However, the universal 3D representation treats all positional data in a 3D space equally, resulting in a quadratic rise of hash conflicts with a given amount of sample points. Consequently, the MLP decoder has a significant challenge in dealing with conflicting gradients. Simultaneously, when using a single-level hash grid for data sampling, treating data at any position in 3D space equally does not distinguish the importance of different regions. The speech-driven talking head generation task faces significant challenges due to the need to fuse the hash representation output from the MLP decoder with low-dimensional speech features before feeding it into the renderer to generate video.  
Inspired by EG3D's [43] use of 2D tensors to represent static 3D portraits, we can compress dynamic talking head generation from high-dimensional space into multiple low-dimensional subspaces to minimize the occurrence of hash conflicts. In other words, we specifically project the 3D spatial features onto three orthogonal 2D hash grids. The importance of different facial regions varies in talking head generation; for instance, people are more sensitive to areas such as the eyes and lips. However, the hash encoder part adopts the same method for all areas in a static resolution manner as in previous methods [6; 11; 12]. Therefore, inspired by PU-VoxelNet [44], we replace each individual level 2D hash grid with a collection of multi-level 2D hash grids to more accurately depict the significance of features in various regions.  
For each 3D sampling point x = \((x,y,z)\in\mathbb{R}^{XYZ}\), we initially project its spatial coordinates onto the three orthogonal planes, which include the X-Y, Y-Z, and X-Z planes. Then, we individually feed them into three sets of multi-level hash encoders.  
\[\mathcal{H}^{MN}:(m,n)\to f_{mn}^{MN} \tag{4}\]  
The output \(f_{mn}^{MN}\in\mathbb{R}^{LF}\) represents the plane set features of the coordinates \((m,n)\) encoded by the single-level hash encoder \(\mathcal{H}^{MN}\) of the plane \(\mathbb{R}^{MN}\), where \(L\) denotes the number of layers and F denotes the feature dimension per entry. We form the ultimate geometric feature \(f_{g}\in R^{3\times LF}\) by concatenating a succession of multi-level dynamic fusion features from three orthogonal planes as follows:  
\[\text{x}(x,y,z)\rightarrow(P(xy),P(yz),P(xz)) \tag{5}\]\[P_{x}^{\prime} =\sum_{i=1}^{N=3}w_{i}(x)E_{xy}^{i}\big{(}P(xy)\big{)} \tag{6}\] \[P_{y}^{\prime} =\sum_{i=1}^{N=3}w_{i}(x)E_{yz}^{i}\big{(}P(yz)\big{)}\] (7) \[P_{z}^{\prime} =\sum_{i=1}^{N=3}w_{i}(x)E_{xx}^{i}\big{(}P(xz)\big{)}\] (8) \[f_{g} =P_{x}^{\prime}\oplus P_{y}^{\prime}\oplus P_{z}^{\prime} \tag{9}\]  
Among them, N denotes the number of multi-level encoders; \(w_{i}(x)\) denotes the linear weight of the i-th encoder; \(P(xy)\),\(P(yz)\) and \(P(xz)\) denote the projections of 3D points onto the X-Y, Y-Z, and X-Z coordinate planes, respectively; \(E_{xy}^{i},E_{yz}^{i}\), and \(E_{xz}^{i}\) denote the hash encoders of the i-th level, each projected onto different coordinate planes, respectively.  
When modeling dynamic heads, we use the perspective direction, \(d\), audio features, \(f_{a}\), and eye features, \(f_{e}\). Furthermore, we incorporate a 3D spatial coordinate feature set \(\mathbb{C}\) consisting of \(P_{x}^{\prime},P_{y}^{\prime},P_{z}^{\prime}\), and \(f_{g}\). Meanwhile, we describe the multi-level three-plane hash representation \(\sigma\) for the implicit function in the following manner:  
\[\mathcal{F}^{\mathcal{H}}:(x,d,f_{a},f_{c};\mathbb{C})\rightarrow(c,\sigma) \tag{10}\]