# A Survey of Advances in Landscape Analysis for Optimisation
## 2 Beyond Fitness Landscapes
### Error Landscapes  
Part of neural network (NN) training involves optimisation, where the task is to search for the weight values that minimise the error of the network model on the training data. Analogous to a fitness landscape, the error or loss landscape can be analysed to better understand the nature of a particular NN weight optimisation problem instance. There are, however, some aspects of NN weight optimisation that make it different from most black-box optimisation problems and this affects the way that landscape analysis can be done.  
Some of the distinguishing characteristics of NN weight optimisation are: ultra high-dimensional search spaces (even small NN models can have thousands of weights), expensive objective evaluation (evaluating the error value of a solution weight vector involves a full run through the training data set), unbounded search spaces (weights can theoretically take on any real value), the same solution can evaluate to different error values depending on the subset of data instances used for training and the availability of analytical gradient information of the objective function. In addition, there is the added complication that the NN training landscape may differ from the NN testing landscape. [PERSON] et al. [13] theoretically and empirically analysed the loss surfaces of multilayered neural networks and found that the training and testing error became increasingly de-correlated with the size of the network. The implication of this is that finding the global optimum in the training loss landscape is of limited use, because it will most likely not correspond with the position of the global optimum in the testing loss landscape. This presents a different picture than with most other optimisation tasks, where the focus is usually on finding the global optimum within the landscape.  
Dimension reduction techniques have been proposed for visualising portions of error surfaces [14; 15]. For example, three-dimensional visualisations of error landscapes ([[https://losslandscape.com/](https://losslandscape.com/)]([https://losslandscape.com/](https://losslandscape.com/))) are based on samples of weight vectors that fall on a two dimensional plane (a slice through the multidimensional weight space). The plane is positioned to pass through a point in the search space around which the landscape is visualised (such as the weight vector at the end of a training run). These visualisations provide a limited view that may not match the experience of a training algorithm in the high dimensional weight space. In addition, a form of numerical characterisation of error landscapes would be more useful for further analysis.  
[PERSON] et al. [16; 17; 18; 19] applied and adapted standard fitness landscape analysis techniques to error landscapes. Studies include: the influence of search space boundaries on the landscape analysis [16], the influence of regularisation on error surfaces [17], the influence of architecture settings on modality of the landscape [18], and the effect of different loss functions on the basins of attraction [19].