Regularized Evolution for Image Classifier Architecture Search

The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically. Although evolutionary algorithms have been repeatedly applied to neural network topologies, the image classifiers thus discovered have remained inferior to human-crafted ones. Here, we evolve an image classifier— AmoebaNet-A—that surpasses hand-designs for the first time. To do this, we modify the tournament selection evolutionary algorithm by introducing an age property to favor the younger genotypes. Matching size, AmoebaNet-A has comparable accuracy to current state-of-the-art ImageNet models discovered with more complex architecture-search methods. Scaled to larger size, AmoebaNet-A sets a new state-of-theart 83.9% top-1 / 96.6% top-5 ImageNet accuracy. In a controlled comparison against a well known reinforcement learning algorithm, we give evidence that evolution can obtain results faster with the same hardware, especially at the earlier stages of the search. This is relevant when fewer compute resources are available. Evolution is, thus, a simple method to effectively discover high-quality architectures. Introduction Until recently, most state-of-the-art image classifier architectures have been manually designed by human experts [20,23,24,27,44]. To speed up the process, researchers have looked into automated methods [2, 28, 31, 34, 35, 42, 46, 52]. These methods are now collectively known as architecturesearch algorithms. A traditional approach is neuro-evolution of topologies [1, 32, 41]. Improved hardware now allows scaling up evolution to produce high-quality image classifiers [29, 35, 46]. Yet, the architectures produced by evolutionary algorithms / genetic programming have not reached the accuracy of those directly designed by human experts. Here we evolve image classifiers that surpass hand-designs. To do this, we make two additions to the standard evolutionary process. First, we propose a change to the wellestablished tournament selection evolutionary algorithm [19] that we refer to as aging evolution or regularized evolution. Whereas in tournament selection, the best genotypes (architectures) are kept, we propose to associate each genotype with an age, and bias the tournament selection to choose the younger genotypes. We will show that this change turns out to make a difference. The connection to regularization will be clarified in the Discussion section. Second, we implement the simplest set of mutations that would allow evolving in the NASNet search space [53]. This search space associates convolutional neural network architectures with small directed graphs in which vertices represent hidden states and labeled edges represent common network operations (such as convolutions or pooling layers). Our mutation rules only alter architectures by randomly reconnecting the origin of edges to different vertices and by randomly relabeling the edges, covering the full search space. Searching in the NASNet space allows a controlled comparison between evolution and the original method for which it was designed, reinforcement learning (RL). Thus, this paper presents the first comparative case study of architecturesearch algorithms for the image classification task. Within this case study, we will demonstrate that evolution can attain similar results with a simpler method, as will be shown in the Discussion section. In particular, we will highlight that in all our experiments evolution searched faster than RL and random search, especially at the earlier stages, which is important when experiments cannot be run for long times due to compute resource limitations. Despite its simplicity, our approach works well in our benchmark against RL. It also evolved a high-quality model, which we name AmoebaNet-A. This model is competitive with the best image classifiers obtained by any other algorithm today at similar sizes (82.8% top-1 / 96.1% top-5 ImageNet accuracy). When scaled up, it sets a new state-ofthe-art accuracy (83.9% top-1 / 96.6% top-5 ImageNet accuracy). Related Work Review papers provide informative surveys of earlier [18, 48] and more recent [15] literature on image classifier architecture search, including successful RL studies [2, 6, 28, 51–53] and evolutionary studies like those mentioned in the Introduction. Other methods have also been applied: cascade-correlation [16], boosting [10], hill-climbing [14], MCTS [33], SMBO [28, 30], and random search [4], and grid search [49]. Some methods even forewent the idea of independent architectures [37]. There is much architecturesearch work beyond image classification too, but that is outside our scope. Even though some methods stand out due to their effiar X iv :1 80 2. 01 54 8v 6 [ cs .N E ] 2 6 O ct 2 01 8 ciency [34, 42], many approaches use large amounts of resources. Several recent papers reduced the compute cost through progressive-complexity search stages [28], hypernets [5], accuracy prediction [3, 13, 25], warm-starting and ensembling [17], parallelization, reward shaping and early stopping [51] or Net2Net transformations [6]. Most of these methods could in principle be applied to evolution too, but this is beyond the scope of this paper. A popular approach to evolution has been through generational algorithms, e.g. NEAT [41]. All models in the population must finish training before the next generation is computed. Generational evolution becomes inefficient in a distributed environment where a different machine is used to train each model: machines that train faster models finish earlier and must wait idle until all machines are ready. Real-time algorithms address this issue, e.g. rtNEAT [40] and tournament selection [19]. Unlike the generational algorithms, however, these discard models according to their performance or do not discard them at all, resulting in models that remain alive in the population for a long time—even for the whole experiment. We will present evidence that the finite lifetimes of aging evolution can give better results than direct tournament selection, while retaining its efficiency. An existing paper [22] uses a concept of age but in a very different way than we do. In that paper, age is assigned to genes to divide a constant-size population into groups called age-layers. Each layer contains individuals with genes of similar ages. Only after the genes have survived a certain age-gap, they can make it to the next layer. The goal is to restrict competition (the newly introduced genes cannot be immediately out-competed by highly-selected older ones). Their algorithm requires the introduction of two additional meta-parameters (size of the age-gap and number of agelayers). In contrast, in our algorithm, an age is assigned to the individuals (not the genes) and is only used to track which is the oldest individual in the population. This permits removing such oldest individual at each cycle (keeping a constant population size). Our approach, therefore, is in line with our goal of keeping the method as simple as possible. In particular, our method remains similar to nature (where the young are less likely to die than the very old) and it requires no additional meta-parameters.