Generative NeuroEvolution for Deep Learning
Authors: Phillip Verbancsics, Josh Harguess
ICLR 2014

Abstract: An important goal for the machine learning (ML) community is to create approaches that can learn solutions with human-level capability. One domain where humans have held a significant advantage is visual processing. A significant approach to addressing this gap has been machine learning approaches that are inspired from the natural systems, such as artificial neural networks (ANNs), evolutionary computation (EC), and generative and developmental systems (GDS). Research into deep learning has demonstrated that such architectures can achieve performance competitive with humans on some visual tasks; however, these systems have been primarily trained through supervised and unsupervised learning algorithms. Alternatively, research is showing that evolution may have a significant role in the development of visual systems. Thus this paper investigates the role neuro-evolution (NE) can take in deep learning. In particular, the Hypercube-based NeuroEvolution of Augmenting Topologies is a NE approach that can effectively learn large neural structures by training an indirect encoding that compresses the ANN weight pattern as a function of geometry. The results show that HyperNEAT struggles with performing image classification by itself, but can be effective in training a feature extractor that other ML approaches can learn from. Thus NeuroEvolution combined with other ML methods provides an intriguing area of research that can replicate the processes in nature.


Peer review:
[Review]:
The paper proposes to automatically learn the structure of a deep neural network using Neuro Evolution (NE) techniques. In particular the authors apply this technique called HyperNEAT (Hypercube based Neuro Evolution of Augmenting Topologies) to learn the structure of a deep convolutional network. This is achieved by learning an indirect encoding which encodes the weight pattern of the neural network. Once learnt, the final network is fine tuned using backpropagation. The technique is applied to the task of identifying hand written numbers in the MNIST dataset. 

In deep learning community, in a number of papers it has been shown that the choice of architecture plays an important role in the performance of the final trained model on any task. However, the question of 'how to choose the right architecture' has not got much attention. In that sense this paper tries to address this important question by automatically learning the network architecture using Neuro Evolution techniques. Unfortunately however, I think the paper falls way short of making any significant contribution in that direction. 

Novelty: I think there are not a lot novel ideas presented in the paper. Its a straight-forward application of HyperNEAT to CNNs. There is no new technique/model being proposed here. 

Quality of results: First, I feel the MNIST dataset is beaten to death and claiming it to be a real world task is probably not true any more. Second, the results reported on MNIST are quite underwhelming. The best accuracy achieved is 92.1% whereas the state-of-the-art results are in the range of 99+%. I would expect that for a technique to be considered promising it should have performance at least in the ball-park of state-of-the-art, if not best. 

Previous work: Since I'm not an expert in the area it is hard for me to point any (if there are so) missing references to the previous work in the field. That said a couple of claims by the authors are not true. For instance, references 30 and 31 do not provide the state-of-the-art. 

Clarity of presentation: I think the quality of the write-up is quite average. For instance, the authors could have explained the HyperNEAT architecture (section 2.2) much better, especially for people who are not experts in the area. I don't see the point of putting the HyperNEAT-LEO algorithm in section 2.2, since it is not used anywhere in the paper. Also the paper is littered with typos. 

Summary: Though the paper tries to address an important problem in deep learning research, I believe its falls short of delivering. The proposed approach is a straight-forward application of HyperNEAT, and the results on the standard dataset are quite poor. This to me suggests that some work needs to be done before one can consider it as a conference publication.