Title: New Learning Approach By Genetic Algorithm In A Convolutional Neural Network For Pattern Recognition
Authors: Mohammad Ali Mehrolhassani
Majid Mohammadi
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning
Optimization
Computer vision

Abstract: Almost all of the presented articles in the CNN  are based on the error backpropagation algorithm and calculation of derivations of error, our innovative proposal refers to engaging TICA  filters and NSGA-II  genetic algorithms to train the LeNet-5 CNN network. Consequently, genetic algorithm updates the weights of LeNet-5 CNN network similar to chromosome update. In our approach the weights of LeNet-5 are obtained in two stages. The first is pre-training and the second is fine-tuning. As a result, our approach impacts in learning task.

TL;DR: Implement new approach without exerting backpropagation in learning of CNN is useful for parallel processing Like GPU.

Review discussion:
I do not understand the algorithm and the motivation behind the algorithm proposed? Why is back propagation a bad idea? Is it the computational speed? or convergence issues? If so then why not compare your algorithm with the standard backprop algorithm in the experiments section?
More generally the paper is extremely poorly written. There are many instance where I do not understand what the authors are trying to say? LeCun-5 is LeNet-5 I believe?
Best wishes for you.
1)	Our goal is useful for parallel device like GPU. Because in every iteration, the individual was produced independently.
Consequently, the computation could be decreased with simple code.
Additionally, in our proposal obtains several trained models at the end of method.
2)	In the end of essay, I refer to compare our model with gradient descent model with training of 50 samples and compare it with 10000 samples for generalizing test. Our model had 35 percent error on MCR measure and the descent model had 55 percent error.
3)	Please help me to clarify the paragraph you mean. And also about the “LeCun-5 is LeNet-5 I believe”. Yes that is correct; I saw that someone refer to The LeCun`s model in his essay (Please refer to reference section) and the diagram of this model clear.
Reversion uploaded at 3 Dec.
This paper is currently fairly long and verbose (at 15 pages, almost twice the recommended length for conference papers, as far as I am aware).  It would be great to edit this down and make it more concise.
Further proof-reading and editing for English would also be a major improvement.  There are many odd phrases and misplaced parts of speech, which unfortunately makes the paper very hard for me to follow.  (For example, "The purpose has ferreted out a novelty approach to train the weights ...", first sentence of Sec 2; ).
For this audience, I suspect much of the background on LeCun-5 and TICA are not so much needed, and those might be a couple places to start making cuts.  The additional background on NSGA-II is nice, as I haven't seen this algorithm before myself, but I think the overall idea to this approach could be more clearly stated early on.  This section also uses terminology that I am not entirely familiar with, and explaining these would help, too.
For example, the initial explaining sentence in sec 1.3 says, NSGA-II "stores all of the information of dominated particles
and the number of domination for each particle".
But what is a "particle" in this application, and what does it mean to be "dominated"?
On a different note, I haven't yet found any discussion comparing this approach to gradient descent or stochastic gradient descent.  I feel a discussion of key differences and advantages would be helpful, as would a direct experimental comparison, since SGD is a simple and very common approach for training a LeCun-5 net.
Thanks a lot for your perfect review. I had to increase the context for clarity for all range of English levels. If the essay was accepted, I would remove additional comments by noticing the referees` comment.
1)	“The purpose has ferreted out a novelty approach to train the weights”. Means: Our Goal in the assay is finding a new way to train weights. Ferreted out is verb that means search or discover. This is ancient verb. The lords while hunting use the dogs for finding their hunts.
2)	NSGA-II "stores all of the information of dominated particles and the number of domination for each particle". Means: particle is individual. The NSGA-II use the individual how much prevail by other individual in population sorting (ranking).
In fact, in NSGA selection sort the population with the new generated individual and then resize the array of population to initialize population size.
3)	In the end of essay, I refer to compare our model with gradient descent model with training of 50 samples and compare it with 10000 samples for generalizing test. Our model had 35 percent error on MCR measure and the descent model had 55 percent error.
Reversion uploaded at 3 Dec.
I try to reconstruct the presentation to improve coherent the proposal.Please check again.
Please help me to except the paper. These paper prepared in 2013 and my researching about that return to 2011.
- typo in the title "RECOGNATION".
- Missing words and odd phrasing make reading the paper difficult, please consider having a native speaker review it. And this is after other reviewers have already mentioned this issue and the authors have updated the paper.
- paper is still too long.
Best wishes
I try to delete the additional comments in paper.
Please considering the update.
Please checking the article again.
The paper is still extremely poorly written and presented despite multiple reviewers asking to address that issue. The frequent spelling mistakes and incoherent sentences and unclear presentation make reading and understanding the paper very difficult and time consuming. Consider getting help from someone with good english and presentation skills.
I try to reconstruct the presentation to improve coherent the proposal.Please check again.
Please help me to except the paper. These paper prepared in 2013 and my researching about that return to 2011.
Unfortunately, this paper is very difficult to understand.  The current version of this paper seems improved compared to the initial version, but still far from a finished level.  I'd encourage the authors to keep editing over the language and presentation.
I also think it would be good to also try answering some of the following questions very clearly in the paper:
- What is the advantage, if any, of the proposed algorithm over SGD?  What is the motivation and goal of the work beyond MNIST benchmarking?
- Why are few training examples used?  Is this a scenario in which the system might have an advantage?
- Concretely describe the genetic algorithms terminology used in the algorithm descriptions, and what each term means in the context of the convolutional network.
- Try to make sure that the method, as described, can be understood by a reader without much prior background on genetic algorithms.
- A single experiment on MNIST is too small to adequately describe the algorithm performance.  Consider using a second or third dataset and/or experimental application.
Much work is still needed on the paper's writing before it can be understood well enough.  I hope that some of this might be useful in helping to improve. I would encourage the authors to try to find outside readers, preferably fluent in English, to work with on a frequent basis before resubmitting to another venue.
I have supplied the chronological of my paper for better understanding.
I am on improvement the writing and presentation.
My purpose is to published the paper by ICLR2017 for obtaining a scholarship or grant to continue on this research and Study in Doctoral in AI.
Please check again
The authors seems to have proposed a genetic algorithm for learning the features of a convolutional network (LeNet-5 to be precise). The algorithm is validated on some version of the MNIST dataset.
Unfortunately the paper is extremely hard to understand and it is not at all clear what the exact training algorithm is. Neither do the authors ever motivate why do such a training as opposed to the standard back-prop. What are its advantages/dis-advantages?
Furthermore the experimental section is equally unclear. The authors seem to have merged the training and validation set of the MNIST dataset and use only a subset of it. It is not clear why is that the case and what subset they use.
In addition, to the best of my understanding, the results reported are RMSE as opposed to classification error. Why is that the case?
In short, the paper is extremely hard to follow and it is not at all clear what the training algorithm is and how is it better than standard way of training. The experimental section is equally confusing and unconvincing.
Other comments:
-- The figures still say LeCun-5
-- The legends of the plots are not in english.
Hence I'm not sure what is going on there.
-- The paper is riddled with typos and hard to understand phrasing.
Exactly, Our proposed is the approach that used NSGA-II algorithm for updating the weights of a LeNet-5 model without having f6 layer in the LeNet-5 (training) in two stages with using TICA filters in Initializing population in NSGA-II. In the LeNet-5 implemented many method for learning like SGD, But they can not used in parallel processing. My motivation was to find the method that have been not back propagation; the feed forward rendering is 5 times faster than the back propagation. Our approche in parallel can be run fast ratio with the SGD.
I am studied in AI. At the opening of my research (2011), I went to find a new way to learn a CNN for learning a few pattern like human; The brain learn a few pattern with different type. For this, come under the chronological of my research:
note: the best results selected from the iterations in practices and show average them with 5 experiments.
1) started Searching by reading the all of article in CNN and deep learning.
2) Started study about Filters and vision specially on TICA filters and filters of Mammal like rats and Macaques.
3) Practiced on the heuristics and evolutionary algorithms likes GA, PSO, Coco and so on.
4) Focused on PSO algorithm (particle swarm organization) for fast converge with 100 to 1000 iterations.
6) The main PSO tested up 1000 times with set different respective parameters but the learning failed (Minimizing the RMSE); This showed in the article that the PSO practices no help to minimizing the error(RMSE=x Single-objective optimization). several functions tested in the pso`s cost function like X power 2 and log2x end others famous.
7) The main GA tested like the PSO practices. And the result showed in article failed.
8) I have tested the PSO and GA chromosome that initialized with famous vision filters like TICE filters. And the result showed in article failed.
9) I am noticed that the good individual remove from population. I should maked a decision: Change the PSO code or GA for my propose or finding the other version. I decided to find an algorithm that have been in native to do not remove the good individual.
10) I have worked on the NSGA-II one of type of GA algorithm (Multi-objective optimization;RSME and MCR). It work good but have problem to minimize the errors despite of used the TICA filters and RMSE Single-objective optimization.
11) Finally,with my scholar in this fields and get the minimum and maximum value of the SGD LeNet-5 wieghts , I have used the TICA filters for initializing their chromosomes in population.
12) For obtaining the individuals that have minimum error I used two stage for learning with different parameters (Discraibed in proposal section).
At The end, I try to rewrite the article for excepting. The article can be reference by the new researcher.
"Neither do the authors ever motivate why do such a training as opposed to the standard back-prop"
I am not opposed the back-prop, Only I want to found the new method to use the parallel operation; Because the platform for cloud computing prepared today for scholar man and also some electronic devices like FPGA developed. And also the GPU extended for deep learning.
"What are its advantages/dis-advantages":
one of advantages is not need back-prop operation only needs the feed forward operation in LeNet-5; On other hand, not needed the calculation of derivation.
That is useful in parallel processing. Perhaps, our approach could be useful in artificial brain in robots; I have still worked on this idea for learinig a few pattern in different dataset (like merging the MINST and NORB).
"the results reported are RMSE as opposed to classification error. Why is that the case":
In some figures we have selected the best RMSE in the different iterations.
In Some plot we have showed all RMSE that generated in iterations. Since the system of LeNet-5 is NP-complete the RMSE have different values. In final iteration the population sorted by ranking with two objects that minimized(RMSE and MCR);
therefore, the best individuals use for weights of LeNet-5.
"The authors seem to have merged the training and validation set of the MNIST dataset and use only a subset of it":
I had to select the few pattern from the famous dataset. My laptop has two core of cpu.
therefore I had to use small pack for calculating.
Thus, the dataset have 50 samples in 10 classes. I learn the whole sample based on k-folded. In our practice k=1;
thus, in the proposal not needed that we have training and testing validation separately.
"The experimental section is equally confusing and unconvincing. ":
The experiments are real and many times I have spent to this paper and researching. Please help me to clear the article.
I am on having new practices on this idea. If they are success, I will add the new results in this paper.
Please check again

Meta-review:
role: PC
data: comment: This paper is unfortunately quite unclear and unreadable and nowhere near ready for any conference.
 I would advise the authors to 1) restructure their paper to present first some type of context and identify a problem that they are trying to solve, 2) explain what novel method they propose to solve the identified problem and why this method is promising and how it relates to existing methods, 3) explain what their experiments are trying to do and what the results of the experiments are, 4) enlist someone fluent in English to help with writing and re-reading.
 A way to do this is to find a set of well-cited papers in the same domain with similar ideas and see how they are structured, then try to follow similar outlines.

Decision: Reject