# Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations
## 3 SGNN vs. GRBFNN
### Backward Propagation  
Accurately estimating the computational complexity of backward propagation is challenging because techniques such as autodifferentiation [28] and computational graphs [26] have optimized the original mathematical operations for improving the computational performance. Autodifferentiation evaluates the derivative of numerical functions using dual numbers with the chain rule broken into a sequence of operations such as addition, multiplication, and composition. During forward propagation, intermediate values in computational graphs are recorded for backward propagation.  
We analyze the operations of backward propagation with respect to a single neuron of the \(l\)-th layer. The partial derivatives of \(\tilde{f}(\mathbf{x})\) with respect to \(\mathcal{W}_{j}^{(l)}\), \(\mu_{j}^{(l)}\), and \(\sigma_{j}^{(l)}\) of the \(l\)-th (\(1\leq l\leq d\)) layer in SGNN are  
\[\frac{\partial\tilde{f}}{\partial\mathcal{W}_{j}^{(l)}} =\left[\frac{\partial\tilde{f}}{\partial\mathcal{N}_{j}^{(l+1)} }\right]^{T}\frac{\partial\mathcal{N}_{j}^{(l+1)}}{\partial\mathcal{W}_{j}^{( l)}}, \tag{21}\] \[\frac{\partial\tilde{f}}{\partial\mu_{j}^{(l)}} =\left[\frac{\partial\tilde{f}}{\partial\mathcal{N}_{j}^{(l+1)} }\right]^{T}\frac{\partial\mathcal{N}_{j}^{(l+1)}}{\partial\mu_{j}^{(l)}},\] (22) \[\frac{\partial\tilde{f}}{\partial\sigma_{j}^{(l)}} =\left[\frac{\partial\tilde{f}}{\partial\mathcal{N}_{j}^{(l+1)} }\right]^{T}\frac{\partial\mathcal{N}_{j}^{(l+1)}}{\partial\sigma_{j}^{(l)}}, \tag{23}\]  
with  
\[\left[\frac{\partial\tilde{f}}{\partial\mathcal{N}_{j}^{(l+1)}}\right]=\left[ \frac{\partial\tilde{f}}{\partial\mathcal{N}^{(l+2)}}\right]^{T}\left[\frac{ \partial\mathcal{N}^{(l+2)}}{\partial\mathcal{N}_{j}^{(l+1)}}\right], \tag{24}\]  
where  
\[\mathcal{N}^{(l+2)}=(\mathcal{N}_{1}^{(l+2)},\mathcal{N}_{2}^{(l+2)},\ldots, \mathcal{N}_{N}^{(l+2)})^{T}. \tag{25}\]  
The backward prorogation with respect to the \(j\)-th neuron of the \(l\)-th (\(1\leq l\leq n-1\)) layer can be divided into three steps:  
1. Compute the gradient of \(\tilde{f}\) with respect to the output of the first neuron in the \((l+1)\)-th layer, \(\mathcal{N}_{j}^{(l+1)}\), as shown in Equation (24), where \(\left[\frac{\partial\tilde{f}}{\partial\mathcal{N}^{(l+2)}}\right]^{T}\) can be accessed from the back propagation of the \((l+2)\)-th layer. This leads to \(2N\) FLOP due to the dot product of two vectors.
2. Calculate the partial derivatives of \(\mathcal{N}_{j}^{(l+1)}\) with respect to weights, center, and width. Since the calculation of derivatives is computationally cheap, the analysis below will neglect the operations used to evaluate derivatives. This shall not affect the conclusion.
3. Propagate the gradients backward. This produces \(N+2\) operations.  
Therefore, the number of FLOPs of the \(l\)-th layer is approximately \(m(3N^{2}+2N)\), where \(m\) is the volume of the input dataset. The backward propagation of the last layer leads to \(N\) operations. In total, the number of FLOPs by backward propagation is  
\[O(FLOP_{bp})=O(mdN^{2}). \tag{26}\]On the other hand, the backward propagation FLOP number of GRBFNN is  
\[O(FLOP_{\widetilde{bp}})=O(mN^{d}). \tag{27}\]