# Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations
## 7 Comparison with Deep NNs
Although SGNN has appreciably larger training epochs, this also leads to more accurate predictions. The loss values of SGNN after training are uniformly smaller than those of ReLU-NN and Sigmoid-NN, except for \(f_{10}\). In fact, for \(f_{2}\), \(f_{4}\), \(f_{6}\), and \(f_{7}\), the accuracy of SGNN is even two orders of magnitude better than the other two models. Despite the efficient training speed of Sigmoid-NN, the network is more difficult to train with random weight initialization for \(f_{1}\) and \(f_{5}\). In fact, the approximation of \(f_{5}\) by Sigmoid-NN is nowhere close to the group truth after training. When functions become more complex, SGNN outperforms ReLU-NN and Sigmoid-NN in minimizing loss through stochastic gradient descent. This could be attributed to the locality of Gaussian functions that increase the active neurons, reducing the flat subspace whose gradients diminish. Sigmoid-NN aborts with significantly fewer epoch numbers. This could be led by the small derivatives of Sigmoid functions when input stays within the saturation region, which makes it more difficult to train the network. Next, we further compare the trainability of SGNN with ReLU-DNN. We train the two networks with different configurations to approximate the function \(f_{5}\), which has a more complex geometry and is more difficult to approximate. The configuration of the NNs and the training performance are listed in Table 7.  
Because the layer of SGNN is fixed by the number of function variables, its only tunable network hyper-parameter is the number of neurons per layer. Doubling the neurons/layer of SGNN from 20 to 40 decreases the loss by two orders of magnitude. Although the training time per epoch increases by 30%, the number of epochs reduces by 60%. Consequently, the total training time is cut by almost 50%, from 38.2 to 19.4 s.  
However, the accuracy of ReLU-DNN slightly increases with the increase in width and depth of the model. Close to 50% loss reduction is achieved by adding seven more layers and 50 neurons per layer. However, the error is still three orders of magnitude higher than the error by a four-layer SGNN with one-tenth of trainable variables, with half the training time per epoch. According to the universal approximation theorem, although one can keep expanding the network structure to improve the accuracy, it is against the observation in the last row. This is because the convergence of gradient descent can be a practical obstacle when the network becomes over-parametrized. In this situation, the network may impose a very high requirement on the initial weights to yield optimal solutions. To visualize the differences in the expressiveness between SGNN and ReLU-NN, the predictions of one run in Table 7 are selected and plotted through a cross-sectional cut in the \(x_{1}-x_{2}\) plane with the other two variables \(x_{3}\) and \(x_{4}\) fixed at zero, as shown in Figure 7. The network configurations are listed in Table 8. The predictions of SGNN in Figure 7b match well the ground truth in Figure 7a. Despite the minor differences in colors near the origin, their maximum magnitude is less than 0.1. The ReLU-NN with the same structure has a much worse approximation. Although the network gradually captures the main geometric features of \(f_{5}\) by significantly augmenting its structure to 10 layers and 70 neurons per layer, the difference of magnitude can still be as large as 0.5, as shown in Figure 7f. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Subfigure** & **Network** & **Layers** & **Neurons/Layer** \\ \hline (b) & SGNN & 4 & 40 \\ (c) & ReLU-NN & 4 & 40 \\ (d) & ReLU-NN & 7 & 40 \\ (e) & ReLU-NN & 10 & 40 \\ (f) & ReLU-NN & 10 & 70 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Network configurations of subplots of Figure 7.