# Fitness Landscape Analysis of Product Unit Neural Networks
## 7 Empirical Analysis of Loss Surface Characteristics
### Optimal Architectures  
For SUNN and PUNN loss surfaces, the nature of the PUNN loss surface is best captured by the \(G_{avg}\) and \(G_{dev}\) metrics, which are substantially larger for PUNNs for every scenario except for the XOR problem. Even for loss surfaces with smaller bounds, the PUNN \(G_{avg}\) is significantly larger than that of SUNN for the majority of the problems. Larger bounds resulted in loss surfaces with even larger gradients, especially for the diabetes and \(f_{4}\) problems. The large values for \(G_{dev}\) mean that the gradients of certain walks deviate substantially from \(G_{avg}\). This is an indication of sudden cliffs or valleys present in the loss surfaces. The \(G_{avg}\) and \(G_{dev}\) metrics portray the treacherous nature of the PUNN landscape, i.e., that of extreme gradients and deep ravines and valleys. The ruggedness of loss surfaces is estimated using entropy using the \(FEM_{0.01}\) and \(FEM_{0.1}\) metrics. The amount of entropy can be interpreted as the amount of \"information\" or variability in the loss surface [22]. There exists a prominent trend between the gradient and ruggedness measures. Loss surfaces with smaller gradients are related to very rugged surfaces, where extremely large gradients are related to smoother surfaces. This relationship is observed for all of the problems, where Iris, Wine, Diabetes, and \(f_{4}\) have large gradients and smaller \(FEM\) values. Conversely, XOR, \(f_{1}\), and \(f_{2}\) have smaller gradients and larger \(FEM\) values. Except for XOR, the PUNN loss surfaces are smoother than the SUNN loss surfaces, which is validated with the smaller values obtained for \(FEM_{0.01}\) and \(FEM_{0.1}\). SUNN landscapes tend to have more variability or \"information\", whereas PUNN landscapes tend to be smoother, with more consistent increases or decreases of loss values. Since surfaces with larger bounds have larger gradients, larger bounds tend to produce smoother loss landscapes. The macro-ruggedness values of \(FEM_{0.1}\) exceed the corresponding micro-ruggedness values of \(FEM_{0.01}\) for all scenarios, indicating that larger step sizes experience more variation in both NN loss surfaces. \(FDC_{s}\) estimates how searchable a loss surface is by quantifying how well the surface guides the search towards areas of better quality. The PUNN \(FDC_{s}\) values for the regression problems are all moderately positive, indicating that PUNN landscapes are not deceptive but possess informative landscapes, making them more searchable. PUNN loss surfaces for all regression problems except \(f_{1}\) are more searchable than those of SUNNs. This does not hold for the classification problems, where PUNN loss surfaces tend to be less searchable. Further, the searchability of both PUNN and SUNN loss surfaces decreases for classification problems. This is a result of the fact that the classification problems are higher dimensional, and thus, the volume of the landscape grows exponentially with the dimension of the landscape. Therefore, the distances between solutions of good quality become very large, producing smaller \(FDC_{s}\) values. This also explains the fact that landscapes with larger bounds are less searchable for all problems. \(DM\) indicates the presence of funnels. Negative values indicate single funnels, while positive values indicate multi-funnels. Negative values were obtained for all loss surfaces, indicating single-funnel landscapes that create basin-like structures for both PUNNs and SUNNs. It is important to note that the \(DM\) measure does not estimate modality. Therefore, it is possible and likely to still have multiple local minima residing in the global basin structure. PUNN landscapes tend to produce more negative \(DM\) values, which is indicative of a simpler global topology for PUNN surfaces. Landscapes with larger bounds produce more negative \(DM\) values, correlating with landscapes of simpler global topology.