## 2 Background  
### Milestones in the Not-So-Long History of Deep Learning  
The history of neural networks and (then) deep learning is peculiar. It includes excitements, stagnation periods, and breakthroughs. Figure 1 presents the selected milestones in that history along the timeline. We categorized most of them according to the five key topics listed in the previous section, hence the following labels: \"backpropagation\", \"initialization\", \"learning\", \"regularization\", and \"computations\". Obviously, these threads were interwoven along the way, but certain periods of specific interests can be observed. An additional category, \"structure\", is introduced to label various structure-related ideas--early models of a neuron, novel layer types or activations, contemporary deep architectures, etc. As is commonly known, it all began in the 1940s and 1950s from the first mathematical models of biological neurons proposed in the works of [3] and then [4]. Despite some initial interest, a bizarre, 20-year-long stagnation period followed, roughly from 1960 to 1980. This was, to some extent, caused by a misunderstanding involving the so-called XOR problem and the inability of the simple perceptron to cope with it, which made some people draw false general conclusions. An interesting \"lonely\" event in that period (apart from minor exceptions) is a master's thesis by [5] on _automatic differentiation_. The work contained a suitable inductive algorithm but did not pertain to neural networks explicitly. And it was not until 1986 that [PERSON], [PERSON], and [PERSON] [7] (re)discovered the algorithm for the realm of neural nets and coined the name _backpropagation_ for it. Two other works from 1980s deserve attention: firstly, the one on _Neocognitron_ by [6], who presented the early ideas of convolutional and pooling layers, and secondly, the one on zip code recognition by [PERSON] [8]. The latter turned into the first real-world application of a neural network, trained end to end with backpropagation and without pre-training. LeCun's net consisted of over 4 k units with 2 k weights (some shared), and it was organized in four hidden layers,including convolution-like and average pooling. Thus, it may be regarded as the first _deep_ learning application, even though the term did not exist at the time. The second stagnation period, lasting for about 10 years, was mainly due to difficulties experienced in training the networks, even medium-sized networks. Convergence problems and large computational powers that had to be invested in were the reasons for this frustration. Researchers started to notice the problem of vanishing gradients. As a result, in the 1990s and early 2000s, the focus of the machine learning community moved to other approaches (e.g., SVMs, boosting, HMMs, etc.) with hand-engineered features or moment-based descriptors. A notable exception was [PERSON]'s research group SRI  
Figure 1: Milestones in the history of neural networks and deep learning [3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35]. International, funded by the US government, which kept working on speech processing using neural networks. Their successful application of a speaker recognition system [11], developed in 1998-2000, proved that automatically extracted raw features may outperform hand-crafted ones, in particular the Mel-Cepstral features popular in speech processing. Also around that time, the term \"deep learning\", introduced by [10], started to appear in common use. In 2006, some revival of interest in DL occurred due to the work on deep belief nets [13]. It indicated new ways of overcoming vanishing gradients by _pre-training_ one layer at a time and then fine-training with backpropagation. Some ideas on pre-training are still valid today (transfer learning and stacked autoencoders). The major breakthrough came around 2009, and it is frequently referred to as the \"DL big bang\".