# Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations
## 7 Comparison with Deep NNs  
In this section, we compare the performance of SGNN with deep ReLU and Sigmoid NNs, which are two popular choices of activation functions. Through the approximation of four-dimensional candidate functions, SGNN shows much better trainability and approximability over deep ReLU and Sigmoid NNs. Table 6 presents the training time per epoch, total epoch for training, and loss after training of three deep NNs by averaging the results of 30 runs. All NNs possess four hidden layers with 20 neurons per layer. The training-set size is fixed to 16,384, with a mini-batch size of 256. As opposed to SGNN and Sigmoid-NN, which have stable training times per epoch across all candidate functions, the time of ReLU-NN fluctuates. This might be led by the difference in calculating the derivatives of a ReLU unit with an input less or greater than zero. SGNN has a longer training time per epoch because of the computation of a Gaussian function and derivative of \(\mu\) and \(\sigma\). One may argue that this comparison is unfair because SGNN has extra trainable variables. However, SGNN has fewer trainable weights (see Table 7) because no weights connect the input and first layer, and the output layer is not trainable. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**SGNN**} & \multicolumn{4}{c}{**ReLU-NN**} & \multicolumn{4}{c}{**Sigmoid-NN**} \\  & **Seconds/Epoch** & **Epoch** & **Loss** & **Seconds/Epoch** & **Epoch** & **Loss** & **Seconds/Epoch** & **Epoch** & **Loss** \\ \hline \(f_{1}\) & 0.099 & 218 & \(1.41\times 10^{-3}\) & 0.054 & 150 & \(4.86\times 10^{-3}\) & 0.063 & 39 & \(4.78\times 10^{-1}\) \\ \(f_{2}\) & 0.098 & 262 & \(2.50\times 10^{-3}\) & 0.054 & 119 & \(1.07\times 10^{-1}\) & 0.054 & 166 & \(2.90\times 10^{-1}\) \\ \(f_{3}\) & 0.106 & 193 & \(4.13\times 10^{-5}\) & 0.312 & 167 & \(5.22\times 10^{-4}\) & 0.054 & 169 & \(1.30\times 10^{-3}\) \\ \(f_{4}\) & 0.100 & 196 & \(9.12\times 10^{-5}\) & 0.234 & 161 & \(7.32\times 10^{-2}\) & 0.056 & 101 & \(2.38\times 10^{-1}\) \\ \(f_{5}\) & 0.097 & 392 & \(9.65\times 10^{-2}\) & 0.173 & 94 & \(4.97\times 10^{-1}\) & 0.066 & 29 & \(6.38\times 10^{-1}\) \\ \(f_{6}\) & 0.101 & 147 & \(2.11\times 10^{-5}\) & 0.293 & 115 & \(1.18\times 10^{-3}\) & 0.053 & 187 & \(2.94\times 10^{-3}\) \\ \(f_{7}\) & 0.099 & 246 & \(1.89\times 10^{-3}\) & 0.241 & 139 & \(2.07\times 10^{-3}\) & 0.054 & 145 & \(1.58\times 10^{-5}\) \\ \(f_{8}\) & 0.100 & 173 & \(7.76\times 10^{-5}\) & 0.344 & 109 & \(9.95\times 10^{-3}\) & 0.054 & 243 & \(3.17\times 10^{-3}\) \\ \(f_{9}\) & 0.099 & 245 & \(9.93\times 10^{-3}\) & 0.439 & 126 & \(7.66\times 10^{-3}\) & 0.054 & 374 & \(7.79\times 10^{-3}\) \\ \(f_{10}\) & 0.101 & 158 & \(9.84\times 10^{-6}\) & 0.135 & 143 & \(6.86\times 10^{-6}\) & 0.054 & 173 & \(8.20\times 10^{-8}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance comparison of SGNN and deep neural networks with ReLU and Sigmoid activation functions. Data are generated by averaging the results of 30 runs. All NNs have four hidden layers, with 20 neurons per layer. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Layers** & **Neuron/Layer** & **Parameters** & **Seconds/Epoch** & **Epoch** & **Loss** & **Min Loss** \\ \hline SGNN & 4 & 20 & 1360 & 0.097 & 392 & \(9.65\times 10^{-2}\) & \(3.91\times 10^{-2}\) \\  & 4 & 40 & 5120 & 0.130 & 149 & \(7.08\times 10^{-4}\) & \(5.32\times 10^{-4}\) \\ \hline \multirow{6}{*}{ReLU-NN} & 4 & 20 & 1381 & 0.056 & 96 & 0.497 & 0.477 \\  & 4 & 40 & 5161 & 0.067 & 99 & 0.458 & 0.409 \\  & 7 & 40 & 10,081 & 0.082 & 135 & 0.336 & 0.273 \\  & 10 & 40 & 15,001 & 0.176 & 112 & 0.324 & 0.258 \\  & 10 & 50 & 23,251 & 0.156 & 100 & 0.309 & 0.253 \\  & 10 & 60 & 33,301 & 0.250 & 96 & 0.288 & 0.232 \\  & 10 & 70 & 45,151 & 0.120 & 97 & 0.278 & 0.215 \\  & 10 & 80 & 58,801 & 0.261 & 89 & 0.291 & 0.205 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Comparison of SGNN and ReLU-based NN in approximation of \(f_{5}\). Results are generated by averaging the data of 30 runs.