## 7 Regularization
### Dropout Regularization  
The dropout technique was proposed by [PERSON], [PERSON], et al. and presented in two articles [22; 66]. If it is known how dropout layers operate (Section 4.5), it may not be evident at first glance that they, in fact, perform _model averaging_--the approach well known from ensemble methods [2; 67]. The averaging within dropout is performed in a \"camouflaged\" way since the direct way would be prohibitively time-consuming, facing three problems: 1. training multiple large networks (preferably with different structures) requires a lot of computations, 2. large networks require large amounts of data (preferably not the same per network), and 3. evaluating multiple responses at the test stage to combine predictions is unacceptable in many applications [66]. The dropout technique addresses all three problems in a probabilistic manner.  
Consider a network with \(N\) units (neurons), where each unit can be dropped (along with its connections) with the probability \(r^{*}\) at the training stage. Since random neurons are dropped, there is almost certainly a different network structure, simpler than a complete one, trained for each data presentation. However, all of these networks _share_ same weights for units that remain present. This is a contrast with respect to traditional ensemble methods, where models are trained from scratch with independent parameters. From that perspective, a network with \(N\) units can be seen as a family of \(2^{N}\) \"thinner\" networks, and random dropout makes it possible to train a huge number of them in a reasonable time with an extensive sharing of weights [22].  
Originally [66], it was postulated to leave output signals of preserved units unchanged at the training stage and to multiply all signals by \(1-r^{*}\) at test time when the dropout is switched off. Note that this constitutes a simple _approximate averaging_ mechanism. Signals or, equivalently, weights of the final network are scaled-down versions of the trained ones. Multiplication by \(1-r^{*}\) at test time ensures that the _expected_ output (under the training-stage distribution) is the same as the actual output at the test stage. Hence, owing to scaling, the hypothetical networks become combined into a single \"mean\" network.  
In Section 4.5, we described dropout layers operating differently, namely with signals scaled by \(1/(1-r^{*})\) at the training stage and unchanged at testing--a default version in many frameworks (Keras, TensorFlow, and Pytorch). One should realize that the two approaches are equivalent.  
Based on multiple experiments, [PERSON] et al. found a superior dropout in terms of generalization than other regularization methods [66]. How can dropout be motivated? According to its authors, dropout reduces _complex co-adaptations_. This means that individual units are driven to be more robust in doing something useful without relying on many other units. They are forced to work in the presence of a random subset of other units and, therefore, with little collaboration. Without dropout, a frequent tendency is that units become co-adapted, i.e., learn a certain complex collaboration, often driven by noise in training data, that relies on the presence of many of them or even all of them. A nice, non-technical explanation of this phenomenon comes from considerations of _successful conspiracies_--quoting [66]: \"(...) Ten conspiracies each involving five people is probably a better way to create havoc than one big conspiracy that requires fifty people to all play their parts correctly (...)\".