## 2 Background
### Milestones in the Not-So-Long History of Deep Learning
It arose due to a combination of three reasons: GPU technology due to NVIDIA [15], large data sets [14; 23], and convenient software that appeared shortly after. As [PERSON], [PERSON], and [PERSON] (the 2019 Turing award winners) describe it now: \"(...) the emergence of GPUs and the availability of large datasets were key enablers of deep learning and they were greatly enhanced by the development of open source, flexible software platforms with _automatic differentiation_ such as Theano, Torch, Caffe, TensorFlow, and PyTorch (...)\" [36]. Looking at the timeline (Figure 1), one can see that after 2009, key events start to become dense. In 2010, [17] published their groundbreaking work on vanishing or exploding signals/gradients in deep structures. They derived a suitable recurrence equation to analyze variance in signals and provided a method for random _weight initialization_ that keeps variance approximately constant--a state-of-the-art method up to this day. The reader should be strongly reminded that the carefree initialization of weights (making the signals vanish or explode) was one of the main reasons for DL stagnation in the 1990s. Also in 2010, [18] demonstrated the usefulness of activation functions called _rectified linear units_ (ReLUs). As a form of regularization, ReLUs contribute to higher accuracy and convergence. Other important structure- and regularization-related contributions followed in 2012: _max pooling_[20] and _dropout_ due to [PERSON], [PERSON], et al. In parallel to those, the research on faster convergence produced new algorithms with adaptive learning rates: AdaGrad [19] and RMSProp [20] in 2011 and 2012, respectively. A combination of novel ideas from that time allowed [PERSON] [23] to construct a structure with 8 layers, named _AlexNet_, that drew the attention of the whole computer vision community. AlexNet won the ImageNet 2012 competition by a large margin [14; 36], being able to accurately recognize objects from one thousand classes in natural images. The key elements of that victory were as follows: ReLUs, convolutions with max pooling, dropout regularization, and the use of multiple GPUs. Great credit must also be given to the efforts of [PERSON]'s group for collecting a training set of over 1 million labeled images [14]. It is interesting to note that many elements applied in AlexNet--inherent components of DL still nowadays, such as: \"weight sharing\" [37], convolutions and pooling [6], and ReLUs-like activation [38]--had already appeared before 1990 and thus significantly predated the contemporary era. Within barely three years since AlexNet, new impressive deeper structures came to life: VGGNet [26], GoogLeNet [27], and ResNet [30]. Around the same time, studies on efficient computations brought about novelties. Initial works on fast convolutions based on FFT [24] or Winograd transformations [28], and their GPU/CUDA implementations opened an important research topic, which is still lively today [39]. Undoubtedly, convolutional layers play a crucial role in contemporary DL (hence the commonness of the names CNNs or DCNNs, the latter standing for deep convolutional neural networks). Simultaneously, they are the most time-consuming layers, with complexity scaling roughly as a product of the number of input pixels, channels, kernels (or feature maps), and the size of a filter and batch. As a side remark, it is worth pointing out that the name \"convolution\", strongly rooted in DL's nomenclature, is not strictly correct. In forward computations, the operation commonly called a \"2D convolution\"is, in fact, a 2D _correlation_ in the mathematical sense. It is during backward computations that one deals with _convolution_ due to weights being flipped up-down and left-right. Regarding studies on optimizers, in 2014, [PERSON] and [PERSON] published their great proposition of _Adam_[25]--a learning algorithm that applies first- and second-order gradient moments estimated via exponential moving averages.