# Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations
## 4 Subspace Gradient Descent  
The aim of this section is to discuss the high performances of SGNNs over GRBFNNs in terms of computational efficiency and accuracy through the lens of gradient descent. As illustrated in Section 3, SGNN has exponentially fewer trainable variables than the associated GRBFNN for high-dimensional input. In other words, GRBFNN may be over-parameterized. The recent work [29; 30; 31] has shown that optimizing a loss function constructed by an over-parameterized neural network can lead to Hessian matrices that possess few dominant eigenvalues with many near-zero ones before and after training. This means the gradient descent can happen in a small subspace. Inspired by their work, we consider the infinitesimal variation of the loss function \(J\) for GRBFNN a  
\[dJ=\bigg{[}\frac{\partial J}{\partial\bar{\mathbf{\theta}}}\bigg{]}^{T}d\bar{\mathbf{ \theta}}+\frac{1}{2}d\bar{\mathbf{\theta}}^{T}\bar{H}d\bar{\mathbf{\theta}}+h.o.t.(\|d \bar{\mathbf{\theta}}\|^{3}), \tag{28}\]  
where \(\bar{\mathbf{\theta}}\) represents a vector of all trainable weights, and  
\[\bar{\mathbf{\mathrm{H}}}=\frac{\partial^{2}J}{\partial\bar{\mathbf{\theta}}^{T}\bar{ \mathbf{\theta}}}, \tag{29}\]  
is the associated Hessian matrix. The centers and widths of Gaussian functions are assumed to be constant for simplicity. Since the Hessian matrix \(\bar{H}\) is symmetric, we can represent it in the form  
\[\bar{\mathbf{\mathrm{H}}}=\mathbf{\mathrm{P}}^{T}\bigg{(}\mathbf{\lambda}_{d}\quad\mathbf{ 0}\quad\mathbf{\lambda}_{s}\bigg{)}\mathbf{\mathrm{P}}, \tag{30}\]  
where \(\mathbf{\lambda}_{d}=\text{diag}(\lambda_{1},\lambda_{2},\ldots,\lambda_{k}, \ldots,\lambda_{dN})\) are the \(k\) dominant eigenvalues padded by \((dN-k)\) non-dominant ones (assuming \(k<dN\)), and \(\mathbf{\lambda}_{s}=\text{diag}(\lambda_{dN+1},\lambda_{dN+2},\ldots,\lambda_{N ^{d}})\) are the rest non-dominant eigenvalues. Let \(\mathbf{\theta}\) be the weights of SGNN. The variation of the mapping from \(\mathbf{\theta}\) to \(\bar{\mathbf{\theta}}\) in Equation (12) reads  
\[d\bar{\mathbf{\theta}}=\frac{\partial\mathbf{\mathrm{g}}}{\partial\mathbf{\theta}}d\mathbf{ \theta}, \tag{31}\]  
where \(\frac{\partial\mathbf{\mathrm{g}}}{\partial\bar{\mathbf{\theta}}}\): \(\mathbb{R}^{dN}\mapsto\mathbb{R}^{N^{d}\times dN}\). It should be noted that \(\frac{\partial\mathbf{\mathrm{g}}}{\partial\bar{\mathbf{\theta}}}\) is a super sparse matrix. Substitution of Equation (31) into Equation (28)  
\[dJ=\bigg{[}\frac{\partial J}{\partial\bar{\mathbf{\theta}}}\bigg{]}^{T}\frac{ \partial\mathbf{\mathrm{g}}}{\partial\mathbf{\theta}}d\mathbf{\theta}+\frac{1}{2}d\theta^ {T}\mathbf{\mathrm{H}}d\mathbf{\theta}+h.o.t.(\|d\bar{\mathbf{\theta}}\|^{3}), \tag{32}\]  
with  
\[\mathbf{\mathrm{H}} =\bigg{[}\frac{\partial\mathbf{\mathrm{g}}}{\partial\mathbf{\theta}} \bigg{]}^{T}\bar{\mathbf{\mathrm{H}}}\frac{\partial\mathbf{\mathrm{g}}}{\partial\mathbf{ \theta}} \tag{33}\] \[=\bigg{[}\frac{\partial\mathbf{\mathrm{g}}}{\partial\mathbf{\theta}} \bigg{]}^{T}\mathbf{\mathrm{P}}^{T}\bigg{(} \begin{matrix}\lambda_{d}&\mathbf{0}\\ \mathbf{0}&\lambda_{s}\end{matrix} \bigg{)}\mathbf{\mathrm{P}}\bigg{[}\frac{\partial \mathbf{\mathrm{g}}}{\partial\mathbf{\theta}}\bigg{]}.\]  
Let  
\[\bigg{(} \begin{matrix}\mathbf{\mathrm{Q}}_{d}\\ \mathbf{\mathrm{Q}}_{s}\end{matrix} \bigg{)}=\mathbf{\mathrm{P}}\bigg{[}\frac{\partial \mathbf{\mathrm{g}}}{\partial\mathbf{\theta}}\bigg{]}, \tag{34}\]where \(\mathbf{Q}_{d}\in\mathbb{R}^{dN\times dN}\) and \(\mathbf{Q}_{s}\in\mathbb{R}^{(N^{d}-dN)\times dN}\). Substituting Equation (34) into Equation (33) yields  
\[\mathbf{H}=\mathbf{Q}_{d}^{T}\lambda_{d}\mathbf{Q}_{d}+\mathbf{Q}_{s}^{T} \lambda_{s}\mathbf{Q}_{s}\approx\mathbf{Q}_{d}^{T}\lambda_{d}\mathbf{Q}_{d}. \tag{35}\]  
Therefore, the dominant eigenvalues of the Hessian of GRBFNN are also included in the corresponding SGNN.