Title: NEW LEARNING APPROACH BY GENETIC ALGORITHM IN A CONVOLUTIONAL NEURAL NETWORK FOR PATTERN RECOGNITION

Abstract: Almost all of the presented articles in the CNN 1 are based on the error backpropagation algorithm and calculation of derivations of error, our innovative proposal refers to engaging TICA 2 filters and NSGA-II 3 genetic algorithms to train the LeNet-5 CNN network. Consequently, genetic algorithm updates the weights of LeNet-5 CNN network similar to chromosome update. In our approach the weights of LeNet-5 are obtained in two stages. The first is pretraining and the second is fine-tuning. As a result, our approach impacts in learning task.

1. Background:

The CNN network presented by Fukushima (Fukushima, 1975; Fukushima, 1980; Fukushima, 1986; Fukushima, 1989; Imagawa, 1993) in 1975 to solve handwritten digit recognition problems, called it Neocognitron. Originally the CNN inspired by the works of Hubel and Wiesel (Wiese, 1962) on the neurons of the cat's visual cortex. The LeCun in (Y. LeCun, 1990) has implemented the first CNN, trained by online Backpropagation, with an extreme accurate recognition process (high accuracy percentage).

Briefly, in the paper, using the TICA filters and NSGA-II algorithms caused to be capable of training a type of CNNs called LeNet-5 by NSGA-II algorithm in two stages: pre-training and fine-tuning on the tiny pack of handwritten digits 4 (a distinct pack encompasses 50 samples from MINST dataset).

1 Convolutional Neural Network 2 Topographic independent component analysis 3 Non dominated Sorting Genetic Algorithm II 4 The MNIST database of handwritten digits has a training set of 60,000 examples, and a test set of 10,000 examples. It contents a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. The CNN extracts the features in hierarchy and creates a spectrum of input pattern upon the output layer; the obtain spectrums help to categorize the input facts. The learning method of the CNN concentrates upon extracting features of the pattern automatically.

1.1. The LeNet-5 Model:

The LeNet-5 consists of seven layers exclude the input layer (Duffner, 2007) . The size of input dimension set to 32×32 pixels. The first five layers, C1, S2, C3, S4, and C5 determine convolution and sub-sampling layers.

The small window that called receptive filed refers to a dimension of 5×5 window for convolution layers and a sub-sampling ratio factor of 2. In figure 2 shows two preliminary layers associated with the input domain called retina. In general, at the end of each convolution layer the subsample layer is attached. (Duffner, 2007) .

1.2. TICA Filters:

Topography independent component analysis is a 2D map component that has been organized with adjacent component to extract similar features (figure 4 ). The component extracts visual features from tiny patches of natural images. In fact, the model has minimized the correlations of energies with a pertinent object function applied. The model can be a CNN (K. Kavukcuoglu, 2009) or can be an optimization algorithm (Koray, 2008) . Figure 4 shows the TICA filters with the size of 16×16 that after 5000 iterations obtains them.

Figure 4 : The TICA Filters with dimensions 16×16

1.3. NSGA-II Algorithm:

One way for obtaining the answers of equivalent to minimize or maximize exploits the evolutionary algorithms; thus, the evolutionary algorithms to be applied instead of derivative calculations. In the implemented algorithms, the answers instead of gene in chromosome and the chromosome called individual and finally the individual formed a population.

The population based on natural selection competes with others for searching. The algorithm performs on population some operation etc. crossover, mutation, selection; the population size, mutation rate, crossover rate and selection strategy conducts the outcome.

The times of running the algorithm called iteration. In the iteration evaluated the population by a function so-called fitness function. Several policies exist to stopping the algorithm: finding the appropriate answer or answers, specified iteration, converge the evaluating of individuals.

In fact, the population to be generated randomly in initializing section and evolved in iterations by randomly operators. Some individuals to be chose for mutation of genes and some of them to be chose to mixture with others. At the end of iteration selects to new population in next iteration with a specific policy from some of individuals. Researcher has implemented numerous evolutionary algorithms as the NSGA-II algorithm. Srinivas and Deb presented NSGA method in 1993 (Deb, 1995) . The NSGA algorithm implements for MOP target purposes with a high performance. The regenerated version, NSGA-II presented in 2002 (Amrit Pratap, 2002) which use the number of domination for each individual by other individual. Constantly, for pertinent performance, the mentioned techniques help to acquire Pareto-Optimal 1 points with reduced complex computing by exerting non-dominated

1 When in multi-objective optimization follows the set of optimum solutions which have no superiority with themselves, the set of answers called Pareto1-Optimal which is the name of economist nominated Vilferedo Pareto.

individuals and calculating of crowding point's distance individuals. Figure 5 illustrates the NSGA-II algorithm.

Totally, the sorting functional NSGA-II algorithm has been improved by storing the frequency of each dominated individual with the others.

The NSGA-II has three crucial issues: quick sorting for non-dominated individuals; computation of crowded distance in entire Pareto-Front; method of individual selection.

For quick sorting, the entire individuals must be compared with themselves and pulled the nondominated out of them with assigning the rank 1. The task is repeated until the entire population allocates a rank from 1 to N to sort the individuals. The Pareto-Front refers to compute additional measure inside of population ranking called crowded distance. Therefore, the distance of chromosomes is computed and stored with respective adjacent in Pareto-Front by using Euclidean distance for entire members. Totally, the population rank and the crowded distance prepare the fitness value for entire members whilst the member with the low rank could be assigned to the low fitness value.

The method of individual selection determines the computation of ranking of non-dominated and crowded distance of individuals. Thus, two values acquired for ranking and computing crowded distance for comparing two members of population. The algorithm selects the member with lower priority. Competition between members of the population with identical rank caused to win the one which has the higher crowded distance. The later generation is induced by merging the winner parents and offspring in competition and after that the population`s size sets the primary length.

2. The proposal:

The proposal focused on genetic or heuristic algorithms to determine the weights of LeNet-5; without using the Backpropagation and the derivation. For better following our proposal, the steps listed below:

1) The F6 layer has been eliminated in LeNet-5 model; consequently, the length of chromosome in GA reduces to 51046 variables. 2) The train and test validation function have merged and 50 samples in 10 classes (0, 1, 2, 3, 4, 5, 6, 7, 8, and 9) selected from MINST dataset randomly. The average of errors for the entire 50 samples leads the LeNet-5 to learning. 3) All the biases in LeNet-5 set to zero. The number of individual population or particles vector size sets to 100. 4) We have two stages with specified respective parameters (table 1) and techniques in the NSGA-II algorithm usage (figure 7 ). The first stage calls Pre-training and the second calls Fine-tuning; the mentioned parameters have been extracted from our experiments and a tuned LeNet-5 with online Backpropagation. 5) The RMSE 1 and MCR 2 employed to minimize the MOP 3 in the NSGA-II or the other experimental algorithms which compute them with Eq 1 and Eq 2. 6) In the NSGA-II algorithm`s fitness function applied ax 2 and by 2 (x denote RMSE and y denotes MCR) 7) The output of LeNet-5`s model for digit 1 and desired digit 1 as depicted the table of digits labels for using in supervised learning (figure 8 ). 

√ ∑ ( ):

Where denotes value of observation in model, denotes value of output desire in i location.

( ):

Where M denotes sum of number of templates become correct, N denotes number of all the templates.

1 Root means square error 2 Misclassification rate 3 Multi-objective optimization 

Outputs Of Neurons:

Digits 0 1 2 3 4 5 6 7 8 9 1 0 0 0 0 0 0 0 0 0 0 2 0 1 0 0 0 0 0 0 0 0 3 0 0 1 0 0 0 0 0 0 0 4 0 0 0 1 0 0 0 0 0 0 5 0 0 0 0 1 0 0 0 0 0 6 0 0 0 0 0 1 0 0 0 0 7 0 0 0 0 0 0 1 0 0 0 8 0 0 0 0 0 0 0 1 0 0 9 0 0 0 0 0 0 0 0 1 0 10 0 0 0 0 0 0 0 0 0 1 Figure 8 : Up: Digit 1 in output of LeNet-5 model and digit 1 in label. Down: Table of digit labels.

2.1. Pre-training stage:

The parts of chromosome that assigns to the layer C1 and C3 in the LeNet-5 should not to be modified at the first stage; at the beginning of the NSGA-II algorithm, C1 and C3 initialized by 160 TICA filters with dimension of 16×16 in section 1.2 randomly. The filters should be resized to dimension of 5×5 to fit the LeNet-5`s weights matrix in respective layers (figure 9 ). Figure 10 illustrates the pre-training. 

2.2. Fine-training stage:

The 27 pertinent individuals have been selected from the preceding stage for initializing preliminary population. The entire layers` weights have been authorized to be modify in the LeNet-5; caused the TICA filters to be improved to extract pertinent features in the layer C1 and C3 of the LeNet-5 scheme by the NSGA-II algorithm. In figure 11 and 12 present graphs for RMSE and MCR errors.

The code of the fine-tuning algorithm declares in Algorithm 1 with remove the lines of replaced the TICA filter in the chromosome (lines 14, 18, 27) and with initialize the parameters at beginning from table 1.

2.3. The results:

Our practices (methods in table 3 and 4 and respective figures) in the article are listed at table 2. (Note: each of the practices has been rendered five times and the average of best results (minimum in RMSE and MCR) has been chosen and rounded in the specified iterations at table 3 and 4 and respective figures). It should be noted that the RMSE and MCR not have related with together; in whole results some time they are opposed (Figure 13 ).

The '*' sign marks our proposal`s results in table 2. The two stages` results (pre-training and finetuning) have been typed in column 1 and 2 of table 3 and 4 for RMSE and MCR; learning achieved with tracing the results. In figure 12 , circumstances of the results for the practices 3 up to 7 imply plenty of errors in the MCR error. Figure 13 depicted the outcomes in the proposal in second stage from original values.

Note: Cost function or fitness function equivalent in MOP in the entire experimental algorithms has applied the ax 2 +by 2 equivalent (x denotes RMSE and y denotes MCR; a=b equal to one) a=b equal to one). In SOP 1 separately applied power equivalent (x 2 ).

Finally, in the study, the table 5 shows the comparative results with the trained LeNet-5 and a derivation trained model has been practiced on the 50 samples comparing of the ability of generalization. Hence, for test phase, our assessment was carried out on 10,000 samples which 4) Appling the GA algorithm, as NSGA-II trains the paramount neurons in neural networks as the LeNet-5. Subsequently, the plethora of computation is not required in derivation and Backpropagation algorithm; therefore, the approach is advantageous to parallel processing. 5) The generalization test is well as the obtain models in the article trained with a few samples. 

3. Conclusion:

In the study presented the procedure by applying the TICA filters and the NSGA-II genetic algorithms with RMSE and MCR objective functions for training of the LeNet-5 convolutional neural network in two stages. The first stage called pre-training and the second stage called finetuning based on optimization solution. In fact, the TICA filters and NSGA-II algorithm with simple and useful methods in computations helped us to learn the input patterns.

Furthermore, the proposal is capable of rendering in parallel processing on GPU and cloud computing (future proposal).

:



Figures and Tables:

Figure1illustrates the principal architecture of LeNet-5 model which had been trained with LeCun`s Online BP 1 Algorithm.

Figure2: The convolution map and subsample of LeNet-5(Duffner, 2007).

Figure 5: Flow chart of NSGA-II algorithm

Figure 7: Stages of the proposal.

Figure 9: The five TICA filters form left to right of the TICA filters in section 1.2 that resize into Dimension of 5×5

Figure 13: The second stage in our proposal from original values without select the best them: the nominate points in fine-tuning stage shows in (a) in the last iteration for multi-objective optimization in NSGA-II algorithm (the f1 axis determines the RSME and f2 axis determines the MCR); the RSME in 1000 iterations in fine-tuning stage shows in (b); the (c) shows the MCR (in percentage) in 1000 iterations in fine-tuning stage.

The parameters in the NSGA-II at Pre-training and Fine-tuning stages

Table content: <table><tr><td>Variable</td><td>Pre-training stage</td><td/><td>Fine-tuning Stage</td></tr><tr><td>Iteration(s)</td><td>1000</td><td/><td>1000</td></tr><tr><td>Number of population</td><td>100</td><td/><td>27</td></tr><tr><td>Chromosome Lengths</td><td>51046</td><td/><td>51046</td></tr><tr><td>Variable Minimum</td><td colspan="2">-0.9000000000000000</td><td>-4.0000000000000004</td></tr><tr><td>Variable Maximum</td><td>0.9000000000000000</td><td/><td>6.0000000000000001</td></tr><tr><td>Population Crossover</td><td>0.2</td><td/><td>0.1</td></tr><tr><td>Population Mutation</td><td>0.3</td><td/><td>0.4</td></tr><tr><td>Ratio Mutation Genes</td><td>0.5</td><td/><td>0.00025</td></tr><tr><td>NSGA-II NSGA-II</td><td>NSGA-II</td><td/><td>NSGA-II</td></tr><tr><td>1000</td><td>1000</td><td/><td>1000</td></tr><tr><td>Iterations</td><td>Iterations</td><td/><td>Iterations</td></tr><tr><td>First</td><td>Second</td><td>Third</td></tr><tr><td>Stage</td><td>Stage</td><td>Stage</td></tr><tr><td>(Pre-training)</td><td>(Fine-tuning)</td><td/></tr></table>



Table content: <table><tr><td>Extra Stage</td></tr></table>

Experimental List

Table content: <table><tr><td>No Experiments</td><td>DataSet</td><td>Number of sample</td></tr><tr><td>1 *NSGA-II initialized with TICA Filters in</td><td>MNIST</td><td>50</td></tr><tr><td>two stages (second stage in our proposal)</td><td/><td/></tr><tr><td>2 *NSGA-II initialized with TICA Filters in</td><td>MNIST</td><td>50</td></tr><tr><td>two stages (first stage in our proposal)</td><td/><td/></tr><tr><td>3 NSGA-II initialized with TICA Filters</td><td>MNIST</td><td>50</td></tr><tr><td>4 Standard GA</td><td>MNIST</td><td>50</td></tr><tr><td>5 Standard PSO</td><td>MNIST</td><td>50</td></tr><tr><td>6 Standard PSO initialized with TICA Filters</td><td>MNIST</td><td>50</td></tr><tr><td>7 NSGA-II</td><td>MNIST</td><td>50</td></tr></table>

The selected RMSE (in percentage) values in experimental for 50 samples

Table content: <table><tr><td>Iteration(s)</td><td/><td/><td colspan="3">Experiments (Refer to table 2)</td><td/><td/></tr><tr><td/><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td></tr><tr><td>100</td><td>1.85</td><td>4.45</td><td>5.8</td><td>6.8</td><td>3.8</td><td>6.8</td><td>7.8</td></tr><tr><td>300</td><td>1.4</td><td>3.4</td><td>5.1</td><td>5.9</td><td>3.8</td><td>5.8</td><td>6.5</td></tr><tr><td>500</td><td>1.37</td><td>2.87</td><td>4.9</td><td>3.72</td><td>3.8</td><td>3.77</td><td>5.8</td></tr><tr><td>700</td><td>1.46</td><td>2.46</td><td>4.8</td><td>3.72</td><td>3.8</td><td>3.77</td><td>5.8</td></tr><tr><td>950</td><td>1.39</td><td>2.19</td><td>3.59</td><td>3.72</td><td>3.8</td><td>3.77</td><td>4.4</td></tr><tr><td>1000</td><td>1.34</td><td>1.94</td><td>3.51</td><td>3.72</td><td>3.8</td><td>3.77</td><td>4.22</td></tr><tr><td colspan="7">Figure 11: Compare the selected RMSE (in percentage) values from table 3</td><td/></tr><tr><td/><td/><td/><td colspan="2">in experiments of iterations</td><td/><td/><td/></tr></table>

Peer review (iclr2017):
The authors seems to have proposed a genetic algorithm for learning the features of a convolutional network (LeNet-5 to be precise). The algorithm is validated on some version of the MNIST dataset. Unfortunately the paper is extremely hard to understand and it is not at all clear what the exact training algorithm is. Neither do the authors ever motivate why do such a training as opposed to the standard back-prop. What are its advantages/dis-advantages? Furthermore the experimental section is equally unclear. The authors seem to have merged the training and validation set of the MNIST dataset and use only a subset of it. It is not clear why is that the case and what subset they use. In addition, to the best of my understanding, the results reported are RMSE as opposed to classification error. Why is that the case? In short, the paper is extremely hard to follow and it is not at all clear what the training algorithm is and how is it better than standard way of training. The experimental section is equally confusing and unconvincing. Other comments: -- The figures still say LeCun-5 -- The legends of the plots are not in english. Hence I*m not sure what is going on there. -- The paper is riddled with typos and hard to understand phrasing.