# Genetic Algorithm Based Deep Learning Neural Network Structure and Hyperparameter Optimization
## 2 Related Works  
### Convolutional Neural Networks  
A Convolutional Neural Network (CNN) [8] is a type of multilayer perceptron that originates from the behaviour of the visual nervous system of animals. Due to its nature, networks with layers of CNN outperform other type of layers, such as Bag of Visual Words, for tasks involving visual features.  
CNN models consist of a convolution layer, activation layer, pooling layer, and fully connected layer. Figure 1 visualises the structure of a basic CNN model. The convolution layer involves convolution filtering of input images to extract feature maps, and the activation layer introduces non-linearity to these feature maps. Non-linearity is important in neural networks because it adds complexity to the network, which allows the network to learn complex, non-linear classifiers for the input data. The pooling layer abstracts these feature maps and changes the dimension of them if required. Finally, the fully connected layer uses information obtained from the convolution and pooling layers to perform classification.  
To find the best combination of parameters that minimises the loss function (or the cost function) of the model, an optimisation algorithm is used. This algorithm updates parameters of the CNN during the backpropagation process.  
Outputs of the fully connected layer are usually represented as a linear combination of input and weights, as shown in (1).  
\[\hat{y}_{j}=f\bigg{(}\sum_{i=0}^{n}w_{i,j}x_{i},b_{j}\bigg{)} \tag{1}\]  
where \(\hat{y}_{j}\) is the output, \(x_{i}\) is the input, \(b_{j}\) is bias, and \(w_{i,j}\) is the weight of the fully connected layer. \(f\) is an activation function.  
Typical cost function for optimising fully connected networks is Mean Squared Error (MSE).  
\[J(w_{i,j})=E[e^{2}(t)]=E[(y_{j}-\hat{y}_{j})^{2}] \tag{2}\]  
where \(y_{j}\) is the ground truth value and \(J()\) represents a loss or cost function.  
Optimisation of the weight is often done using Gradient Descent (GD) [9] technique (3,4).  
\[\mathbf{w_{t+1,j}}=\mathbf{w_{t,j}}-\eta\frac{\partial J\mathbf{w_{t,j}}}{ \partial\mathbf{w_{t,j}}} \tag{3}\]  
\[\frac{\partial J\mathbf{w_{t,j}}}{\partial\mathbf{w_{t,j}}}=-2(y_{t,i}-\hat{ y}_{t,i})\mathbf{x_{t,i}} \tag{4}\]  
where \(\eta\) is a learning rate.  
As shown in (4), the optimisation algorithm is heavily affected by the input of the layer. Furthermore, the output of the layer has been processed with an activation function. Therefore, proper selection of both activation function and optimisation algorithm is essential for the performance of the network.  
The cost function, as a result of the relationship between input data and parameters of the network, defines an error surface. Figure 2 represents a simple error surface defined by the cost function. This surface affects the accuracy and convergence speed of the network. Therefore, input data does effect the performance of the neural networks significantly, and good architectural design is required to ensure this performance.  
Figure 1: Basic structure of a typical Convolutional Neural Network (CNN).  
Figure 2: Example error surface defined by the cost function, input dataset and parameters of the network.  
Recently, several CNN structures have been investigated to achieve better performance, specially for the classification tasks. These structures can be categorized into three types; Chain-shaped; Multiple-path; and Highway Network. A chain-shaped network is a simple, linear structure that is used from the beginning of CNNs. Still, many state-of-the-art networks, such as VGG [10], perform very well with this structure. Multiple-Path Networks increase the number of layers that are connected in parallel, and concatenate or add the results of these layers as the input for the next layer. ResNet [11] and GoogleNet [12] are popular examples of this block. Finally, highway network [13] uses the output of a layer as input for all proceeding layers. DenseNet [14] is a very popular network that successfully adapts this network structure.