# Fitness Landscape Analysis of Product Unit Neural Networks
## 9 Conclusions  
The main purpose of this work was to perform a fitness landscape analysis (FLA) on the loss surfaces produced by product unit neural networks (PUNNs). The loss surface characteristics of PUNNs were analyzed and compared to those of SUNNs to determine in what way PUNN and SUNN loss surfaces differ.  
PUNN loss surfaces have extremely large gradients on average, with large amounts of deviation over the landscape suggesting many deep ravines and valleys. Larger bounds and regularized PUNN architectures lead to even larger gradients. Stochastic gradient descent (SGD) failed to train PUNNs due to the treacherous gradients of the PUNN loss surfaces. The gradients of PUNN loss surfaces are significantly larger than those of SUNN loss surfaces, which explains why gradient descent works for SUNNs and not PUNNs. Therefore, optimization algorithms that make use of gradient information should be avoided when training PUNNs. Instead, meta-heuristics such as particle swarm optimization (PSO) and differential evolution (DE) should be used. PSO and DE successfully trained PUNNs of all architectures for all problems.  
PUNN loss surfaces are less rugged, more searchable in lower dimensions, and less neutral than SUNN loss surfaces. The smoother PUNN loss surfaces were strongly correlated with the larger gradient measures and were found to possess simpler overall global structures than SUNN loss surfaces, where the latter had more multi-funnel landscapes. Oversized architectures created higher-dimensional landscapes, decreasing searchability, increasing ruggedness, and having an overall more complex multi-funnel global structure. The \(FEM\), \(DM\), and \(FDC_{s}\) metrics correlated well with the poor training performance DE and PSO achieved for oversized PUNN architectures. \(FEM\), \(DM\), and \(FDC_{s}\) captured the effects of the \"curse of dimensionality\" on PUNN loss surfaces. Regularized PUNN loss surfaces were more searchable than complex PUNN loss surfaces, leading to better training and generalization performance. The \(G_{avg}\) metric described the effect regularization had on PUNN loss surfaces and was found to correlate well with the better training performance of DE and PSO for regularized PUNNs. Regularized PUNN loss surfaces had more deep ravines and valleys that trapped PSO and DE. Finally, PSO is suggested for loss surfaces that have large \(G_{dev}\) values.  
Conceptualization, A.E.; methodology, A.E. and R.G.; software, R.G.; validation, A.E. and R.G.; formal analysis, R.G.; investigation, R.G. and A.E.; resources, R.G.; data curation, R.G.; writing--original draft preparation, A.E. and R.G.; writing--review and editing, A.E.; visualization, R.G.; supervision, A.E.; project administration, A.E. All authors have read and agreed to the published version of the manuscript.  
This research received no external funding.  
Not applicable.  
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.  
The authors declare no conflicts of interest.  
The following abbreviations are used in this manuscript:  
\begin{tabular}{l l} CPL & complex product unit neural network landscape \\ DE & differential evolution \\ DM & dispersion metric \\ EA & evolutionary algorithm \\ FDC & fitness-distance correlation \\ FEM & first entropic measure \\ FLA & fitness landscape analysis \\ gbest & global best \\ GD & gradient descent \\ MRW & Manhattan random walk \\ NN & neural network \\ OPL & optimal product unit neural network landscape \\ PRW & progressive random walk \\ PSO & particle swarm optimization \\ PU & product unit \\ PUNN & product unit neural network \\ RPL & regularized product unit neural network landscape \\ SGD & stochastic gradient descent \\ SU & summation unit \\ SUNN & summation unit neural network \\ \end{tabular}