# Genetic Algorithm Based Deep Learning Neural Network Structure and Hyperparameter Optimization
## 3 Model Optimisation  
### Structure of the Algorithm  
As the performance of the CNN model is greatly affected by both network architecture and hyperparameters, the search space suggested by Genetic-CNN is insufficient. Therefore, in this paper, in order to find the effective network architecture for a given input, we select both network configurations and hyperparameters as the search space, and use GA to search for the ideal structure. Figure 3 illustrates the procedure of our system. In order for any Genetic-type algorithm to optimise a given task, it first requires mapping of the solution into the search space of the algorithm. The key to the mapping process is to find a suitable representation scheme, so that solutions can be successfully described as chromosomes-sequences of genes. Genes can be represented by binary digits, alphabets, or floating point numbers. Figure 3: Block diagram explaining the process of our Genetic Algorithm (GA). For our algorithm, we use binary representation to encode connections between convolutional layers. An example of encoding such a connection is shown in Figure 4. We define a set of convolutional layers--layers that are in between input and pooling layers; two pooling layers; or pooling and fully connected layers--as a stage. For each layer within a stage, we call it a node. The structure shown in Figure 4 has one stage, with three nodes within the stage. We then convert the connection between each node into binary numbers. Only forward connections are allowed, and preceding connections of each node are encoded. Finally, stages are separated with bars, and nodes are separated with hyphens. The number of bits required to encode a network with \(S\) stages, where each stage has \(K_{S}\) nodes, and is calculated with (5). \[\textit{number of bits}=\sum_{S}\frac{1}{2}K_{S}(K_{S}-1) \tag{5}\]  
We also encode activation functions and optimisation algorithms into bits, and separate them with bars. The number of bits required for \(n\) hyperparameters can be calculated using (6). \[\textit{number of bits}=\lfloor\log_{2}(n)\rfloor+1 \tag{6}\]  
Definition of the encoding procedure is very important for our process, as it is used from the start of the algorithm--initialization. This step generates a population of CNN models in encoded, binary form. This step generally uses information from a previous generation to generate models, but for the first generation, it initialises the process by randomly creating a number of candidates. Each of the population is called as a chromosome, \(s\). During this process, each bit of a chromosome is sampled independently using a Bernoulli distribution. With provided population of networks, the algorithm decodes each of them and calculates the classification accuracy. These accuracies are used as fitness scores for the algorithm. The fitness score is also considered as the objective function of the algorithm, as it decides the survival of each chromosome. In our algorithm, we set an optimal fitness score, and repeat the entire procedure until a CNN model achieves this score. If none of the population achieved the optimal fitness score, our algorithm then selects number of chromosomes, using a rank-based system. In this system, chromosomes with higher classification accuracies will survive, which ensures fast convergence of the algorithm. After selection stage, survived chromosomes create the next generation of individuals through a uniform crossover process, \(c\). This process selects a random pair of parent  
Figure 4: Encoding stages of architectures into binary strings. (**a**)-set of convolutional layer; (**b**)-encoded version of the layers in (**a**). In (**b**), numbers before and after the hyphen represent connections to node \(A2\) and \(A3\), respectively. chromosomes, and generates offspring using a random variable. If this variable is larger than a predefined threshold, parent 1's gene is taken. Else, parent 2 is chosen. Finally, mutation is applied to the surviviving chromosomes.