# Fitness Landscape Analysis of Product Unit Neural Networks
## 7 Empirical Analysis of Loss Surface Characteristics
### Regularized Architectures
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Problem** & **Algorithm** & **Architecture** & \(\overline{\mathcal{E}}_{\mathbf{r}}\) & **Best** & \(\overline{\mathcal{E}}_{\mathbf{c}}\) & **Best** \\  & & & \(\overline{\mathcal{E}}_{\mathbf{r}}\) & \(\mathbf{\mathcal{E}}_{\mathbf{r}}\) & \(\overline{\mathcal{E}}_{\mathbf{c}}\) & \(\overline{\mathcal{E}}_{\mathbf{c}}\) & \(\mathbf{\mathcal{E}}_{\mathbf{c}}\) \\ \hline XOR & \(PSO\) & oPUNN & 0.003 & (0.002) & 0.0 & 0.003 & (0.002) & 0.0 \\  & & oSPUNN & 0.005 & (0.005) & 0.001 & 0.005 & (0.005) & 0.001 \\ \cline{3-8}  & & oPUNN & 0.001 & (0.001) & 0.0 & 0.001 & (0.001) & 0.0 \\  & & oSPUNN & 0.004 & (0.003) & 0.001 & 0.004 & (0.003) & 0.001 \\  & & rPUNN & 0.003 & (0.002) & 0.001 & 0.003 & (0.002) & 0.001 \\ \hline  & \(DE\) & oPUNN & 0.002 & (0.001) & 0.001 & 0.002 & (0.001) & 0.001 \\  & & oSPUNN & 0.002 & (0.001) & 0.001 & 0.002 & (0.001) & 0.001 \\  & & rPUNN & 0.004 & (0.004) & 0.001 & 0.004 & (0.004) & 0.001 \\ \hline Iris & \(PSO\) & oPUNN & 0.137 & (0.014) & 0.121 & 0.131 & (0.012) & 0.119 \\  & & oSPUNN & 0.152 & (0.018) & 0.125 & 0.156 & (0.015) & 0.132 \\ \cline{3-8}  & & oPUNN & 0.191 & (0.036) & 0.143 & 0.188 & (0.038) & 0.143 \\  & & oSPUNN & 0.602 & (0.21) & 0.357 & 0.669 & (0.23) & 0.354 \\  & & rPUNN & 0.47 & (0.14) & 0.286 & 0.498 & (0.141) & 0.325 \\ \hline  & \(DE\) & oPUNN & 0.127 & (0.01) & 0.119 & 0.127 & (0.011) & 0.118 \\  & & oSPUNN & 0.161 & (0.03) & 0.118 & 0.161 & (0.031) & 0.121 \\ \cline{3-8}  & & oPUNN & 0.226 & (0.028) & 0.185 & 0.222 & (0.018) & 0.197 \\  & & oSPUNN & 0.674 & (0.32) & 0.387 & 1.059 & (0.994) & 0.414 \\  & & rPUNN & 0.477 & (0.201) & 0.236 & 0.577 & (0.334) & 0.256 \\ \hline Wine & \(PSO\) & oPUNN & 0.209 & (0.004) & 0.202 & 0.242 & (0.033) & 0.2 \\  & & oSPUNN & 0.859 & (0.175) & 0.661 & 1.231 & (0.626) & 0.682 \\ \cline{3-8}  & & oPUNN & 0.271 & (0.044) & 0.229 & 37,960.156 & (75,919.546) & 0.237 \\  & & oSPUNN & 2133.681 & (1469.965) & 745.362 & 8.76\(\times 10^{12}\) & (1.75\(\times 10^{13}\)) & 165,730.721 \\  & & rPUNN & 1200.283 & (1463.764) & 201.401 & 8.35\(\times 10^{16}\) & (1.068\(\times 10^{17}\)) & 11,838.192 \\ \hline  & \(DE\) & oPUNN & 0.201 & (0.006) & 0.191 & 0.219 & (0.019) & 0.199 \\  & & oSPUNN & 1.266 & (0.287) & 0.845 & 1.398 & (0.27) & 0.938 \\ \cline{3-8}  & & oPUNN & 0.32 & (0.051) & 0.224 & 11738.17 & (2.4372.638) & 0.257 \\  & & oSPUNN & 3255.567 & (2009.275) & 668.72 & \(9.77\times 10^{18}\) & (1.95\(\times 10^{17}\)) & 485.885 \\  & & rPUNN & 3387.882 & (3452.529) & 28.73 & \(4.53\times 10^{10}\) & (9.04\(\times 10^{13}\)) & 12,196.997 \\ \hline Diabetes & \(PSO\) & oPUNN & 0.197 & (0.018) & 0.166 & 0.205 & (0.016) & 0.185 \\  & & oSPUNN & 0.226 & (0.013) & 0.204 & 3.36 & (6.239) & 0.216 \\ \cline{3-8}  & & oPUNN & 0.22 & (0.004) & 0.215 & 0.23 & (0.005) & 0.224 \\  & & oSPUNN & 0.404 & (0.136) & 0.256 & 0.423 & (0.161) & 0.232 \\  & & rPUNN & 0.326 & (0.081) & 0.257 & 0.435 & (0.273) & 0.259 \\ \hline  & \(DE\) & oPUNN & 0.192 & (0.008) & 0.18 & 0.202 & (0.017) & 0.176 \\  & & oSPUNN & 0.228 & (0.009) & 0.211 & 0.244 & (0.009) & 0.23 \\ \cline{3-8}  & & oPUNN & 0.224 & (0.002) & 0.221 & 0.225 & (0.005) & 0.219 \\  & & oSPUNN & 0.512 & (0.233) & 0.257 & 0.498 & (0.229) & 0.239 \\ \cline{3-8}  & & rPUNN & 0.365 & (0.201) & 0.231 & 265.822 & (530.72) & 0.228 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Training results for the classification problems. landscape volumes--are also correlated with worse training performance. Therefore, the performance of both PSO and DE deteriorates for loss surfaces with higher dimensionality. This agrees with findings in the literature [14] and is referred to as the \"curse of dimensionality\". The deterioration in training performance for CPLs can be explained by the observed loss surface characteristics. CPLs were found to be less searchable: possessing more complex global structures (multi-funnels) and having increased ruggedness.