Title: AN EXPERIMENTAL STUDY OF LAYER-LEVEL TRAIN-ING SPEED AND ITS IMPACT ON GENERALIZATION Anonymous

Abstract: How optimization influences the generalization ability of a DNN is still an active area of research. This work aims to unveil and study a factor of influence: the speed at which each layer trains. In our preliminary work, we develop a visualization technique and an optimization algorithm to monitor and control the layer rotation rate, a tentative measure of layer-level training speed, and show that it has a remarkably consistent and substantial impact on generalization. Our experiments further suggest that weight decay's and adaptive gradients methods' impact on both generalization performance and speed of convergence are solely due to layer rotation rate changes compared to vanilla SGD, offering a novel interpretation of these widely used techniques, and providing supplementary evidence that layer-level training speed indeed impacts generalization. Besides these fundamental findings, we also expect that on a practical level, the tools we introduce will reduce the meta-parameter tuning required to get the best generalization out of a deep network.

1. INTRODUCTION:

Generalization and gradient propagation are two popular themes in the deep learning literature. Concerning generalization, it has been observed that a network's ability to generalize depends on a subtle interaction between the optimization procedure and the training data (Zhang et al., 2017a; Arpit et al., 2017) . Concerning gradient propagation, several works have shown that the norm of gradients can gradually increase or decrease as a function of layer depth (i.e. vanishing and exploding gradients (Bengio et al., 1994; Hochreiter, 1998; Glorot & Bengio, 2010) ), so that some layers are trained faster than others. This work explores an interaction between generalization and the intricate nature of gradient propagation in deep networks, and focuses on the following research question: how does the speed at which each layer trains influence generalization?

Our endeavour is motivated by the following intuition: if the training data influences a neural network's generalization ability when using gradient-based optimization (Zhang et al., 2017a; Arpit et al., 2017) , the input and feedback signals that a layer receives (during the network's forward and backward passes) could also influence the generalization ability induced by the layer's training. These signals result from a transformation involving the other layers of the network such that, for example, the input signals of the last layer could be more conducive to good generalization if the first layers have been significantly updated already, instead of being randomly initialized (cfr. the works on transfer learning (Donahue et al., 2014; Oquab et al., 2014) ). More generally, the speed at which each layer trains during the network's training, since it directly influences how the input and feedback signals of the other layers evolve over training, could have an impact on generalization. Figure 1 supports our intuition with a toy example where training a single layer of an 11 layer MLP network, although always reaching 100% train accuracy, results in different test accuracies depending on the layer's localisation in the network architecture.

Our study starts from an educated guess about how to measure layer-level training speed appropriately: we measure it through layer rotation rates, i.e. the rates at which the weight vectors of layers rotate (another approach would be to measure the norm of the weight updates at each training step, as is done in Bengio et al. (1994) ; Hochreiter (1998) ; Glorot & Bengio (2010) ; Pascanu et al. (2013) ; Arjovsky et al. (2016) ). The study is then composed of the three following steps:

1. Developing tools to monitor and control layer rotation rates; 2. Using our controlling tool to systematically explore layer rotation rate configurations, varying the layers which are prioritized (first layers, last layers, or no prioritization) and the global rotation rate value (high or low rate, for all layers);1 3. Using our monitoring tool to study the layer rotation rates that emerge from standard training settings. 1 The outcomes of our study, supported by an extensive amount of experiments, are the following:

(i) layer rotation rates have a consistent and substantial impact on generalization;

(ii) weight decay is a key ingredient for enabling the emergence of beneficial layer rotation rates during SGD training;

(iii) adaptive gradient methods' impact on generalization and training speed does not result from parameter-level, but rather from layer-level adaptation of the learning rate;

While the influence of layer-level training speed on generalization has remained unstudied, our observations thus suggest that its impact is ubiquitous in current deep learning applications. Our preliminary work offers useful guidance for meta-parameter tuning and novel insights around two widely used techniques: weight decay and adaptive gradient methods. Moreover, while layer rotation rate as a measure of layer-level training speed originated from an educated guess, the impressive consistency of its impact on generalization supports the pertinence of this choice. Our work thus also contributes to the open problem of correctly measuring layer-level training speed.

To encourage further validation of our claims, the tools and source code used to create all the figures of this paper are provided at -github link hidden to preserve anonymity-(code uses the Keras (Chollet et al., 2015) and TensorFlow (Agarwal et al., 2016) libraries). We also encourage interested readers to browse the supplementary material of this paper, as additional results are presented and discussed. 

2. RELATED WORK:

Recent works have demonstrated that generalization in deep neural networks was largely due to the optimization procedure and its puzzling interaction with the training data (Zhang et al., 2017a; Arpit et al., 2017) . Our paper discloses an aspect of the optimization procedure that influences generalization in deep learning: the rate at which each layer's weight vector is rotated. This novel factor complements batch size and global learning rate, two parameters that have been extensively studied in the light of generalization (Keskar et al., 2017; Jastrzebski et al., 2017; Smith & Le, 2017; Smith & Topin, 2017; Hoffer et al., 2017; Masters & Luschi, 2018) .

The works studying the vanishing and exploding gradients problems (Bengio et al., 1994; Hochreiter, 1998; Glorot & Bengio, 2010) heavily inspired this paper. These works introduce two ideas which are central to our investigation: the notion of layer-level training speed and the fact that SGD does not necessarily train all layers at the same speed during training. Our work explores the same phenomena, but studies them in the light of generalization instead of trainability and speed of convergence.

Our paper also proposes Layca, an algorithm to control the rate at which each layer's weight is rotated during training. It is related to the works that sought solutions to the gradient propagation problems at optimization level (Pascanu et al., 2013; Hazan et al., 2015; Singh et al., 2015; Arjovsky et al., 2016; Pennington et al., 2017) . These works, however, do not use weight rotation as a measure of layer-level training speed, and also focus on speed of convergence instead of generalization.

Recently, a series of papers proposed optimization algorithms similar to Layca and observed an impact on generalization (Yu et al., 2017; Zhang et al., 2017b; Ginsburg et al., 2018) . Section A.2 in our Supplementary Material provides evidence that these methods may be equivalent to Layca in practice, despite avoiding some of Layca's operations. Our paper thus supplements these works by providing an extensive study of the phenomena underlying their observations.

Several works have recently argued that weight decay's regularization effect emerged from its ability to increase the effective learning rate (van Laarhoven, 2017; Hoffer et al., 2018; Anonymous, 2019) .

A concise description of when and to what extent weight decay increases the effective learning rate is however lacking, such that using weight decay is still necessary to benefit from its regularization effect in practice.2 Our work also analyses weight decay, but from the perspective of layer rotation rates instead of effective learning rates. We show that this new perspective enables a more succinct description of weight decay's regularizing effect, that we are able to reproduce without any additional meta-parameter tuning when using Layca, our tool for controlling layer rotation rates.

3. TOOLS FOR MONITORING AND CONTROLLING LAYER ROTATION RATES:

This paper's goal is to study the relation between layer-level training speed and generalization. However, the notion of layer-level training speed is unclear, and its control through SGD is potentially difficult because of the intricate nature of gradient propagation (cfr. vanishing and exploding gradients). Therefore, our work starts by making an educated guess about how to measure layer-level training speed and by developing tools to monitor and control this metric during training. This section presents the metric and tools, while the impact on generalization is studied in Sections 4 and 5.

3.1. HOW CAN WE MEASURE LAYER-LEVEL TRAINING SPEED?:

Training speed can be understood as the speed with which a model converges to its optimal solution -not to be confounded with learning rate, which is only one of the parameters that affect training speed in current deep learning applications. Previous work on vanishing and exploding gradients focused on the norm and variance of gradients as a measure of layer-level training speed (Bengio et al., 1994; Hochreiter, 1998; Glorot & Bengio, 2010) . Provided the empirical work on activation and weight binarization during (Courbariaux & David, 2015; Rastegari et al., 2016; Hubara et al., 2016) or after training (Agrawal et al., 2014;  Algorithm 1 Layca, an algorithm that enables control over the amount of weight rotation per step for each layer through its learning rate parameter (cfr. Section 3.2).

Require: o, an optimizer (SGD is the default choice) Require: T , the number of training steps L is the number of layers in the network for l=0 to L-1 do Require: ρ l (t), a layer's learning rate schedule Require: w l 0 , the initial multiplicative weights of layer l end for t ← 0 while t < T do

s 0 t , ..., s L-1 t = getStep(o, w 0 t , ..., w L-1 t

) (get the updates of the selected optimizer)

for l=0 to L-1 do s l t ← s l t - (s l t •w l t )w l t w l t •w l t (project step on space orthogonal to w l t ) s l t ← s l t w l t 2 s l t 2

(rotation-based normalization)

w l t+1 ← w l t + ρ l (t)s l t (perform update) w l t+1 ← w l t+1 w l 0 2 w l t+1 2

(project weights back on sphere) end for t ← t + 1 end while Carbonnelle & De Vleeschouwer, 2018) , we argue that the norm of a weight vector does not matter, but only its orientation. Therefore, we suggest to measure training speed through the rotation rate of a layer's weight vector (also denoted by layer rotation rate in this paper). More precisely, let w t l be the flattened weight tensor of the l th layer at optimization step t, then the rotation rate of layer l between steps t 1 and t 2 is defined as the angle between w t1 l and w t2 l divided by the number of performed steps t 2 -t 1 . 3 In order to visualize how fast layers rotate during training, we propose to inspect how the cosine distance between each layer's current weight vector and its initialization evolves across training steps. We denote this visualization tool by layer-wise angle deviation curves hereafter.

3.2. LAYCA: AN ALGORITHM TO CONTROL LAYER ROTATION RATES:

Given our definition of layer-level training speed, we now develop an algorithm to control it. Ideally, the layer rotation rates should be directly controllable with layer-wise learning rate parameters, ignoring the peculiarities of gradient propagation. We propose Layca (SGD-guided LAYer-level Controlled Amount of weight rotation), an algorithm where the layer-wise learning rates directly determine the amount of rotation performed by each layer's weight vector during an optimization step, in a direction specified by an optimizer (SGD being the default choice). Inspired by techniques for optimization on manifolds (Absil et al., 2010) , and on spheres in particular, Layca applies layerwise orthogonal projection and normalization operations on SGD's updates, as detailed in Algorithm 1. While Layca enables control over the rotation performed during one unique training step, the presence of noise or inconsistent directions can influence the overall rotation over multiple training steps in an uncontrolled way. Fortunately, such behaviour can be detected by inspecting the layerwise angle deviation curves and did not hinder our experiments. 1 .

4.1. LAYER-WISE LEARNING RATE CONFIGURATIONS:

Layca enables us to specify layer rotation rate configurations by setting the layer-wise learning rates. To explore the large space of possible layer rotation rate configurations, our study restricts itself to two directions of variation. First, we vary the initial global learning rate ρ(0), which affects the training speed of all the layers. During training, the global learning rate ρ(t) drops following a fixed decay scheme (hence the dependence on t), as is common in the literature (cfr. Supplementary Material B.3). Notice that the impact of the global learning rate on generalization has already been studied when using SGD (Jastrzebski et al., 2017; Smith & Le, 2017; Smith & Topin, 2017; Hoffer et al., 2017; Masters & Luschi, 2018) , but not with an algorithm like Layca where learning rate directly determines rotation rate. The second direction of variation is prioritization. We explore the impact of prioritization amongst layers by applying static, layer-wise learning rate multipliers that exponentially increase/decrease with layer depth (which is typically encountered with exploding/vanishing gradients). The multipliers are parametrized by the layer index l (in forward pass ordering) and a parameter α ∈ [-1, 1] such that the learning rate of layer l becomes:

ρ l (t) = (1 -α) 5 (L-1-l) L-1 ρ(t) if α > 0 (1 + α) 5 l L-1 ρ(t) if α ≤ 0 (1)

Values of α close to -1 correspond to prioritizing first layers, 0 corresponds to no prioritization, and values close to 1 to prioritization of last layers. Visualization of the layer-wise multipliers for different α values is provided in Supplementary Material.

To study the impact of global learning rate, we evaluate 10 logarithmically spaced values of ρ(0) (3 -7 , 3 -6 , ..., 3 2 ) in the α = 0 setting. To study the impact of prioritization, we compare 13 different values of α, and tune the initial global learning rate ρ(0) for each value separately through an iterative grid search procedure (described in Supplementary Material).

4.2. HOW LAYER ROTATION RATES INFLUENCE GENERALIZATION:

Figure 2a shows, for each of the three tasks, the test accuracies obtained for the different α and ρ(0) values. From these results, we extract two rules of thumb. First, the rotation rates should be uniform across layers, as prioritizing the first or last layers reduces the test accuracy by up to 20%. Second, we observe that the layer rotation rates should be selected as high as training allows it (when too high, training diverges), enabling gains of up to 30% in test accuracy. The observations generalize across the three tasks, and our preliminary exploration indicates that layer rotation rate configurations have a consistent and substantial impact on generalization. Let us also notice that in extreme prioritization schemes (| α |≥ 0.6), the observations are in line with Figure 1 's results, as prioritizing the first layers generalizes better than prioritizing the last layers.

Figure 2b presents the layer-wise angle deviation curves (cfr. Section 3.1) generated by the different configurations. This visualization enables us to check that, up to small deviations, Layca enables good control over the rotation rates in the tasks we consider. For example, the α = 0 setting used in the fifth column indeed leads all layers to rotate quasi synchronously. Moreover, it is useful to visualize how the layer-wise angle deviation curves look like for the different layer rotation rate configurations, as the same visualization tool will be used in Section 5 to analyse standard training settings where layer rotation rates are not controlled during training. In particular, it is useful to remember that for the three tasks, the best generalization performance is obtained when nearly all layers synchronously reach a cosine distance of 1 from their initialization. 

4.3. HOW LAYER ROTATION RATES INFLUENCE NETWORK CONVERGENCE:

It is commonly assumed that vanishing and exploding gradients slow down or even prevent training of neural networks. One might thus be tempted to believe that the bad performances on the test set obtained in Figure 2 for low and/or non-uniform layer rotation rates are caused by an equally bad performance on the training set. However, not only do these layer rotation rate configurations result in close to perfect training performance (cfr. Supplementary Material), but they also often lead the network to converge faster than the high and uniform layer rotation rate configuration. Figure 3 depicts the loss curves obtained for different values of α and ρ(0). It appears that the higher or the more uniform the layer rotation rates are, the higher the plateaus in which loss curves get stuck into. The fact that plateaus are the most prominent when all layers are rotated at high rate suggests that they are caused by some kind of interference between the layers during training. Moreover, it also suggests that, following our rules of thumb, high plateaus are additional indicators of good generalization performance. 0) values on the tiny-CNN task (for the two other tasks, see Supp. Mat.), using Layca for training. The more uniform or the higher the layer rotation rates, the higher the plateaus in which the loss gets stuck into. The sudden drop at epoch 70 corresponds to a reduction of the global learning rate by a factor 5.

5. A STUDY OF LAYER ROTATION RATES EMERGING FROM STANDARD TRAINING SETTINGS:

Section 4 uses Layca to study the impact of layer rotation rates on generalization and speed of convergence in a controlled setting. This section investigates the layer rotation rates that naturally emerge when using SGD and adaptive gradient methods for training. First of all, these experiments will provide supplementary evaluation of the rules of thumb proposed in Section 4. Second, analysing SGD and adaptive gradient methods in the light of layer rotation rates' impact on generalization will provide useful insights to explain previous observations around these methods that currently escape our understanding.

The experiments of this section are performed on the three tasks used in Section 4 and on two supplementary, extensively tuned networks from state of the art (cfr. Table 1 ). Meta-parameters for these two tasks are taken from their original implementation when using SGD and from Wilson et al. (2017) when using adaptive gradient methods for training. For the other three tasks, the learning rate is determined by grid search over 10 logarithmically spaced values (3 -7 , 3 -6 , ..., 3 2 ) independently for each (task,optimizer) pair.

5.1. ANALYSIS OF SGD AND WEIGHT DECAY:

Figure 4 (1 st line) depicts the layer-wise angle deviation curves generated by SGD and the corresponding test accuracies for each of the five tasks. We observe that the curves are far from the ideal scenario disclosed in Section 4, where the majority of the layers synchronously reached a cosine distance of 1 from their initialization. Moreover, in accordance with our rules of thumb, SGD induces a considerably lower test performance than Layca (in the high and uniform rotation rate configuration). Extensive tuning of the learning rate did not help SGD to solve its two systematic problems: 1) layers don't train at the same speed and 2) the layers' weights stop rotating before reaching a cosine distance of 1.

At this point, it is tempting to believe that Layca will improve the performance of all the deep networks trained with SGD. However, we observed that, with the unexpected help of weight decay (i.e. L 2 -regularization), SGD gains the ability to induce high and uniform layer rotation rates and the accompanying good test performance. Figure 4 (2 nd line) displays, for the 5 tasks, the layer-wise angle deviation curves generated by SGD when combined with weight decay. We observe that all layers are rotated synchronously, reaching a cosine distance of 1 from their initialization, and that the resulting test performances are on par with the ones obtained with Layca. This experiment not only provides important supplementary evidence for our rules of thumb, but also suggests a radically novel explanation of weight decay's regularization ability in deep nets: weight decay enables the emergence of high and uniform layer rotation rates during SGD training.5 On a practical level, since the same regularization effect can be elegantly achieved with tools that control layer rotation rates, without an extra parameter to tune, our results could potentially lead weight decay to disappear from the standard deep learning toolkit.

Figure 4 : Layer-wise angle deviation curves and the corresponding test accuracies generated by SGD without (1 st line) or with (2 nd line) weight decay. Colour code, axes and ∆η computation are the same as in Figure 2b . 6 Despite extensive learning rate tuning, SGD without weight decay induces test performances that are significantly below Layca. These results are coherent with our rules of thumb, as SGD is not able to induce high and uniform layer rotation rates (cfr. 5 th column of Figure 2b ). Surprisingly, weight decay solves SGD's problems, leading to high and uniform layer rotation rates and test accuracies that are on par with Layca.

5.2. ANALYSIS OF ADAPTIVE GRADIENT METHODS:

The recent years have seen the rise of adaptive gradient methods in the context of machine learning (e.g. RMSProp (Tieleman & Hinton, 2012) , Adagrad (Duchi et al., 2011 ), Adam (Kingma & Ba, 2015) ). Initially introduced for improving training speed, Wilson et al. (2017) observed that these methods also had a considerable impact on generalization. Figure 5 provides the layer-wise angle deviation curves and test accuracies obtained when using adaptive gradient methods for training of the 5 tasks described in Table 1 . Again, our rules of thumb can be applied: the overall worse generalization ability compared to Layca corresponds to low and/or non-uniform layer rotation rate configurations.

We also observe that the layer rotation rates of adaptive gradient methods are considerably different from the ones induced by SGD (cfr. Figure 4 ). For example, adaptive gradient methods tend to prioritize the last layers while SGD usually prioritizes the first layers in the tasks we study. Previous results of this paper indicate that these differences could be the reason behind adaptive gradient methods' influence on both training speed and generalization. Figure 6 confirms this hypothesis by showing that the training and test curves of adaptive gradient methods become indistinguishable from their non-adaptive equivalents when Layca is used to enforce a fixed layer rotation rate configuration. Additional evidence is provided in Supplementary Material (Section A.4), where we observe experimentally that the learning rates of adaptive gradient methods vary mostly across layers and negligibly inside layers.

Adaptive gradient methods were initially introduced with a focus on parameter-level adaptation of the learning rate in order to improve optimization in the presence of sparse gradients or optimization in on-line and non-stationary settings (Kingma & Ba, 2015) . Our work offers and empirically validates a radically different understanding of adaptive gradient methods' role in deep learning, that solely relies on their impact on the rate at which each layer's weights change during training. This new perspective reveals that adaptive gradient methods' parameter-level adaptation of the learning rate is not necessary (layer-level adaptation suffices), and that their impact on DNN training will probably be better explained by further study of layer-level training speed rather than the sparsity of gradients or the on-line aspect of the optimization procedure.

Figure 5 : Layer-wise angle deviation curves and the corresponding test accuracies generated by adaptive gradient methods (RMSProp, Adam, Adagrad, RMSProp+L 2 and Adam+L 2 respectively for each task/column) after extensive learning rate tuning. Colour code, axes and ∆η computation are the same as in Figure 2b . Our rules of thumb still apply: the overall worse generalization ability compared to Layca corresponds to low and/or non-uniform layer rotation rate configurations. We also notice that the curves are significantly different from the ones of SGD (Figure 4 ). For example, on the C10-CNN1 task, the last layers train faster than the first ones when RMSProp is used for training while the opposite happens when SGD is used. Could this explain adaptive gradient methods' impact on training speed and generalization? Training and test curves of adaptive gradient methods and their non-adaptive equivalents7 when Layca is applied on the updates of each method to fix the layer rotation rate configuration. The curves become indistinguishable, suggesting that adaptive gradient methods' impact on training speed and generalization is only due to their influence on layer rotation rates, and thus to their layer-level adaptation of the learning rate.

6. EMPHASIZING THE REMARKABLE CONSISTENCY OF LAYER ROTATION RATES' IMPACT ON GENERALIZATION:

In order to emphasize the remarkable consistency by which layer rotation rates influence generalization, this section provides evidence that the norm of updates, another sensible metric of layer-level training speed (Bengio et al., 1994; Hochreiter, 1998; Glorot & Bengio, 2010; Pascanu et al., 2013; Arjovsky et al., 2016) , does not demonstrate this consistency at all. This result supports the idea that layer rotation rates are related to a more fundamental aspect of learning, and probably constitute a relevant metric to study layer-level training speed.

The main difference between layer rotation rates and the norm of updates as layer-level training speed metrics, is that the latter doesn't take the norm of the current layer's weights into account. Our experiment targets this difference specifically, by testing if the relation between the studied metric and generalization remains consistent when the initial weights of all layers are rescaled by a constant factor f . The independence of layer-level metrics to rescaling of the weights is especially important as, with typical weight initialization schemes, weights of different layers have different scales (e.g. Glorot & Bengio (2010) ).

Using Layca to control layer rotation rates, and block-normalized SGD (Yu et al., 2017) to control the norm of weight updates, we study for several factors f the relation between test accuracy and the initial global learning rate in Figure 7 (experiment is conducted on a lighter version of the C10-CNN1 task). The difference is striking. In contrast with block-normalized SGD, Layca remains very consistent across scaling factors. For example, the optimal learning rate is 3 -3 in all cases. On the contrary, when block-normalized SGD is used, the curve that relates learning rates to train or test accuracy shifts every time the scaling factor is increased, and so does the optimal learning rate.

3 -9 3 -8 3 -7 3 -6 3 -4 3 -4 3 -3 3 -2 3 -1 3 0 ρ(0) 

7. CONCLUSION:

Inspired by the works on generalization and gradient propagation in deep networks, this paper's ambition is to disclose and study a novel way, unique to deep learning, by which optimization influences generalization: through the speed at which each layer trains. While the premises of our work are based on intuitions that escape any theoretical framework, the value of our tools and claims is supported by the consistent experimental results and the useful insights they provide. Indeed, the rules of thumb we extract about the relation between layer rotation rates and generalization could be successfully applied to all the considered tasks and training settings, explaining substantial differences in test accuracies (sometimes of the order of 20%). Moreover, we show considerable evidence that this novel way by which optimization impacts generalization could be the reason behind weight decay's and adaptive gradient methods' generalization properties, which have remained a mystery despite their ubiquity in current deep learning applications. Both methods, which are source of heavy meta-parameter tuning, could even become obsolete as practitioners start using our tools, exemplifying the practical benefit of our work. Algorithm 2 Main operations of Layca (cfr. Algorithm 1). We've noticed that in practice, operations 1 and 4 are not strictly necessary for controlling layer rotation rates.

s 0 t , ..., s (2: rotation-based normalization)

w l t+1 ← w l t + ρ l (t)s l t (3: perform update) w l t+1 ← w l t+1 w l 0 2 w l t+1 2

(4: project weights back on sphere) end for weights. Explicitly orthogonalizing the step and the weights through operation 1 is thus potentially redundant.

Operation 4 keeps the norm of weights fixed during the whole training process. First, this operation emphasizes our claim that the norm of weights doesn't really matter. Indeed, disabling changes to the norm of weights doesn't prevent the network from reaching 100% training accuracy. Second, this operation prevents the weights from increasing too much (the first three operations lead the norm of weights to increase at every training step), which causes numerical problems. However, this operation is not fundamental for controlling the layer rotation rates.

We experimented with a sub-version of Layca that does not perform Layca's operations 1 and 4. Interestingly, the resulting algorithm is equivalent to NG adap and LARS introduced by Yu et al. ( 2017) and Ginsburg et al. (2018) respectively. Both works reported improved test performance when using this algorithm. Figure 9 shows the layer-wise angle deviation curves and associated test accuracies when applying LARS (or equivalently, NG adap ) on tasks C10-CNN1, C100-resnet and tiny-CNN. 8 The layer rotation rate configuration parameters are α = 0 and ρ(0) = 3 -3 . We observe that this configuration also induces high and uniform rotation rates, and that the test accuracies are on par with Layca. This observation indicates that operations 1 and 4 of Layca can be removed in at least some practical applications.

A.3 IMPACT OF LAYER ROTATION RATES ON CONVERGENCE FOR C10-CNN1 AND C100-RESNET TASKS.

Figure 3 shows on the tiny-CNN task that the α and ρ(0) parameters, which determine the layer rotation rate configuration when using Layca for training, enable precise control over the height of

B.2 GRID SEARCH PROCEDURE FOR LEARNING RATE SELECTION IN SECTION 4:

In Section 4, the global initial learning rate parameter ρ(0) is optimized through grid search for each α value. While in the other experiments, learning rate selection is performed through grid search over 10 logarithmically spaced values (3 -7 , 3 -6 , ..., 3 2 ), such method is to demanding computationally for this experiment (it must be repeated 39 times = 13 α values * 3 tasks).

The grid search procedure starts by trying 3 values: 3 -4 , 3 -3 and 3 -2 . Then, iteratively until convergence, if the current best value is the lowest or highest of the tried values, the next value (lower or higher by a factor 3 respectively) is tried. After this first stage, the optimal ρ(0) values of two successive α values are compared (successive after sorting the values in increasing order). If the optima of two successive α values are different by a factor 3, the intermediate ρ( 0) is also tried (current value multiplied or divided by 3 0.5 ). The increased precision of the second stage was used to get smoother curves in Figure 2a .

B.3 LEARNING RATE DECAY SCHEMES:

Our work uses standard learning rate decay schemes, as follows:

• C10-CNN1: 100 epochs and a reduction of the learning rate by a factor 5 at epochs 80, 90 and 97

• C100-resnet: 100 epochs and a reduction of the learning rate by a factor 10 at epochs 70, 90 and 97 SGD AMom was designed for Section 5.2, as a non-adaptive equivalent of Adam. In particular, SGD AMom uses the same momentum scheme as Adam:

v t = m • v t-1 + (1 -m) • g t w t = w t-1 -ρ • v t

where g t is the gradient at step t, ρ the learning rate, m the momentum parameter.

B.6 FURTHER INFORMATION.:

Other details about our experiments can be found in our code at -github link hidden to preserve anonymity-.

:

Our preliminary study focuses on convolutional neural networks used for image classification.

van Laarhoven (2017) proposes to keep the norm of the weights fixed to 1 in order to eliminate the effect of weight decay, but not to reproduce it.Hoffer et al. (2018);Anonymous (2019) are able to reproduce the regularizing effect of weight decay by tuning the effective learning rate, but their tuning strategy consists in copying the effective learning rate that emerges when training the same network with weight decay.

EXPLORATION OF LAYER ROTATION RATE CONFIGURATIONS WITH LAYCASection 3 provides tools to monitor and control layer rotation rates, a tentative definition of layerlevel training speed. The purpose of this section is to use these tools to conduct a systematic experimental study of the relation between layer rotation rates and generalization. The experiments are 3 It is worth noting that our measure focuses on weights that multiply the inputs of a layer (e.g. kernels of fully connected and convolutional layers). Studying and controlling the training of additive weights (biases) is left as future work.

References: ResNet(He et al., 2016), torch blog(Zagoruyko, 2015), Wide ResNet(Zagoruyko & Komodakis, 2016),CIFAR-10 (Krizhevsky & Hinton, 2009), Tiny ImageNet(Deng et al., 2009; CS231N, 2016).

Notice that this observation is also consistent with the systematic occurrence of high plateaus (cfr. Section 4.3) in the loss curves of state of the art networks(He et al., 2016;Zagoruyko & Komodakis, 2016) (which usually use SGD with weight decay).

On the two supplementary tasks, the reference accuracy (η) is also obtained by Layca with α = 0 and ρ(0) = 3 -3 and equals 92.33% and 80.69% on C10-CNN2 and C100-WRN respectively.

SGD AMom corresponds to SGD with a momentum scheme similar to Adam (see Supp. Mat.).

While the norm of each layer's weight vector was not fixed by LARS, we still had to limit the amount of norm increase per training step to prevent numerical errors. We limited it to 0.0001 times the initial norm of each layer's weight vector.

Figures and Tables:

Figure 2: Analysis of the generalization ability induced by different layer rotation rate configurations (using Layca for training) on the three first tasks of Table1. The configurations are parametrized by α, that controls which layers are prioritized (first layers for α < 0, last layers for α > 0, or no prioritization for α = 0), and ρ(0), the initial global learning rate value shared by all layers. (a) Test accuracy in function of α and ρ(0). Two rules of thumb emerge: layer rotation rates should be uniform across layers (i.e. α = 0) and be as high as training allows it (i.e. high ρ(0) values). (b) Layer-wise angle deviation curves (cfr. Section 3.1) generated by different configurations, and their accompanying test accuracy (η). ∆η is computed with respect to the high and uniform layer rotation rate configuration (last column), which corresponds to α = 0 and ρ(0) = 3 -3 for the three tasks. Train accuracies are provided in Supplementary Material (≈ 100% in all configurations).

Figure 3: Loss curves obtained for different α and ρ(0) values on the tiny-CNN task (for the two other tasks, see Supp. Mat.), using Layca for training. The more uniform or the higher the layer rotation rates, the higher the plateaus in which the loss gets stuck into. The sudden drop at epoch 70 corresponds to a reduction of the global learning rate by a factor 5.

Figure6: Training and test curves of adaptive gradient methods and their non-adaptive equivalents 7 when Layca is applied on the updates of each method to fix the layer rotation rate configuration. The curves become indistinguishable, suggesting that adaptive gradient methods' impact on training speed and generalization is only due to their influence on layer rotation rates, and thus to their layer-level adaptation of the learning rate.

Figure7: Consistency comparison of layer rotation rates and the norm of updates (another sensible metric of layer-level training speed). Using Layca (left) or block-normalized SGD (center) to control both metrics, we test if the relation between global initial learning rates and train/test accuracy remains consistent when the initial weights of all layers are rescaled by a constant factor f . The difference is striking: block-normalized SGD does not demonstrate Layca's consistency at all. SGD is also added for comparison (right). Dotted lines correspond to train accuracy, full lines to test accuracy.

Figure8: left: Studying the impact of initialization on the MNIST experiment of Figure1. We observe that orthogonal initialization, known for its faithful propagation of signals, increases the test accuracy of every layer and slightly reduces the generalization gap between the first and last layers. right: Supplementary result suggesting that the capacity of layers to improve their own feedback signal could be a key asset enabling the first layers' superior generalization performance in Figure1. Indeed, this figure shows how the Silhouette score of the first layer's feedback with respect to the classes/targets increases even when only the first layer is trained.

tiny-CNN: 80 epochs and a reduction of the learning rate by a factor 5 at epoch 70 B.4 TRAINING ERRORS ASSOCIATED TO THE LAYER-WISE ANGLE DEVIATION CURVES.In Figures2b, 4and 5, the test accuracies corresponding to each visualization of the layer-wise angle deviation curves are provided. While it is briefly mentioned that training accuracy is close to perfect in most cases (cfr. Section 4.3), Tables 2, 3 and 4 provide the exact values for completeness.

The notion of layer-level training speed is ill-posed, since a layer does not have a loss of its own: all layers optimize the same global loss function. Given a training step, how can we know by how much each layer's update contributed to the improvement of the global loss? Or, in other words, how can we measure at what rate relevant features are learned by each layer individually?

Table content: <table/>

Summary of the tasks used for our experiments 4

Table content: <table><tr><td>Name</td><td>Architecture</td><td>Dataset</td></tr><tr><td colspan="2">C10-CNN1 25 layers deep CNN</td><td>CIFAR-10</td></tr><tr><td colspan="2">C100-resnet ResNet-32</td><td>CIFAR-100</td></tr><tr><td>tiny-CNN</td><td>11 layers deep CNN</td><td>Tiny ImageNet</td></tr><tr><td colspan="2">C10-CNN2 deep CNN from torch blog</td><td>CIFAR-10 + data augm.</td></tr><tr><td colspan="3">C100-WRN Wide ResNet 28-10 with 0.3 dropout CIFAR-100 + data augm.</td></tr><tr><td colspan="3">conducted on three different tasks which vary in network architecture and dataset complexity, and</td></tr><tr><td colspan="2">are further described in Table</td><td/></tr></table>

). Using Layca (left) or block-normalized SGD (center) to control both metrics, we test if the relation between global initial learning rates and train/test accuracy remains consistent when the initial weights of all layers are rescaled by a constant factor f . The difference is striking: block-normalized SGD does not demonstrate Layca's consistency at all. SGD is also added for comparison (right). Dotted lines correspond to train accuracy, full lines to test accuracy.

Table content: <table/>

Train accuracies associated to Figure2b

Table content: <table><tr><td/><td colspan="4">α = 0.6 α = -0.6 ρ(0) = 3 -5 ρ(0) = 3 -4</td><td>Best</td></tr><tr><td>C10-CNN1</td><td>100%</td><td>99.55%</td><td>100%</td><td>100%</td><td>99.99%</td></tr><tr><td colspan="2">C100-resnet 97.38%</td><td>97.9%</td><td>99.87%</td><td>99.99%</td><td>99.75%</td></tr><tr><td>tiny-CNN</td><td>99.98%</td><td>98.64%</td><td>99.97%</td><td>99.97%</td><td>98.91%</td></tr><tr><td/><td colspan="4">Table 3: Train accuracies associated to Figure 4</td><td/></tr><tr><td colspan="6">C10-CNN1 C100-resnet tiny-CNN C10-CNN2 C100-WRN</td></tr><tr><td>SGD</td><td>100%</td><td>100%</td><td>100%</td><td>91%</td><td>100%</td></tr><tr><td>SGD + L 2</td><td>100%</td><td>100%</td><td>89.8%</td><td>99.5%</td><td>100%</td></tr></table>

Train accuracies associated to Figure5MOMENTUM SCHEME USED BY SGD AMOM AND ADAM.

Table content: <table><tr><td/><td colspan="5">C10-CNN1 C100-resnet tiny-CNN C10-CNN2 C100-WRN</td></tr><tr><td>Adaptive methods</td><td>100%</td><td>99.98%</td><td>99.97%</td><td>98.72%</td><td>99.92%</td></tr></table>

Peer review (iclr2019):
 Paper summary: The authors propose layer rotation speed as a measure of layer-wise training speed and introduce the Layca optimizer as a means of enforcing uniform layer rotation speed throughout the network. They show empirically that layer rotation speed is linked to the generalization performance of deep neural networks and that weight decay induces uniform layer rotation speeds. Detailed comments: Overall, I felt that the paper introduced some interesting ideas but I was not left convinced that layer rotation speed is the correct measure of layer training speed. I hope that the authors can clarify this based on my questions and comments below. 1) In the introduction you refer to input and feedback signals to a layer, I assume this refers to the forward and backward pass. As I understand it, this intuition and the findings of Figure 1 do not immediately relate to the notion of layer rotation speed during training. Could you clarify what you mean by "the input and feedback signals that a layer receives could also influence the generalization ability induced by the layer*s training", to me this statement seems obvious as input+feedback signals contains training entirely. 2) In Figure 1 you show that when training one chosen layer and keeping the others fixed, if the chosen layer is deeper into the network the test accuracy is worse. I wonder to what extent this might be remedied by initialization. For example, one might expect that when sampling random square matrices there are some very small eigenvalues which "kill" information in the forward pass. If we train a layer deep in the network it may have access to less information from the data than one earlier on. Whereas training an earlier layer could allow this layer to shift mass into the parts of the eigenspace which are well represented (so-to-speak) in the future layers. Have you thought about this at all? One simple way to evaluate this would be to initialize the weights to be random orthogonal matrices, ensuring that the eigenvalues are equal. With that said, I thought that this was an interesting experiment with a fairly surprising outcome! 3) In related work you discuss the vanishing and exploding gradients problems in terms of layer-level training speed. I think that another relevant research direction may be dynamical isometry [1] which solves this problem by restricting all singular values of the Jacobian matrix to be close to 1. These ideas may also be relevant when discussing Layca and layer-rotation. 4) I found section 3.1. a little unconvincing. It is not obvious to me that layer rotation speed is necessarily a good measure of training speed. In fact, there are many updates which have large cosine distance (as you define it) but do not change the network function (for example, permuting the weight matrices in fully-connected networks). Why is the rotation defined through a vectorization of the weight matrix as opposed to e.g. the polar decomposition? Is this a computational issue? Similarly, in section 3.2 you liken Layca to optimization on a manifold but I am not convinced that this makes sense for matrices which inherently have some structure (e.g, perhaps the Stiefel manifold would be more meaningful). 6) Figure 2 shows that uniform rotation leads to improved test accuracy. But could it be the case that controlling the effective learning rate is sufficient (and layer rotation is one way to achieve this)? For example, we might take the sign of the update and use this to ensure that each weight matrix has the same effective learning rate (something like [2]). Do you expect this would have a similar effect? If not, what is unique about layer rotation that provides good test accuracy? 7) You claim that SGD and adaptive methods with weight decay works without taking extra care to control the layer-rotation rate, as weight decay provides a similar effect. Firstly, you use weight decay and L2 regularization interchangeable, could you be explicit about exactly which you mean (see e.g. [3]). Assuming you mean weight decay (and not L2 regularization), then this could also be due to the effective learning rate ([4,5,6]) which may have some interaction with layer rotation rate (i.e. Figure 4). In summary, I would have liked to see an explanation for why weight decay leads to uniform rotation speeds. 8) If I understand correctly, Figure 5 shows 5 tasks and reportedly 5 optimization schemes - each on a different task? It seems more reasonable to compare these on the same task. Overall I felt that the paper had some interesting contributions and a fairly comprehensive empirical study. However, I do not feel that the paper gives adequate attention to the notion of effective learning rate induced by weight decay and I was not totally convinced that the way layer rotation speed is defined is the correct way. Minor comments: - A lot of white space and a large caption for Figure 1. - Section 4 opens with "monitor and control", but I think the latter is really presented in section 4 and not section 3. - I think a diagram of the projection step of the Layca algorithm would be informative (for 2D weight vector). - Why does `5` appear in equation 1? Is this an arbitrary choice? - Some of the lighter colors in e.g. Figure 2(b) made some lines hard to read when printed. I do not believe that this affected the image significantly. Clarity: The paper is well written and is easy to understand. Some of the figures in the experiments are a little cluttered and the lighter colors can be hard to see (e.g. Fig 2(b)), but this is minor. Significance: The paper presents an interesting view point but I am not convinced that it offers as strong an explanation for these phenomena as other approaches. I believe with some more clarification the results could become more significant. My review score hinges mostly on the interaction between layer rotation speed and the effective layer-wise learning rate. Originality: To my knowledge, the ideas are presented in the paper are original. In particular, this is a novel way to characterize layer-wise training speed. References: [1] Pennington et al. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice" https://arxiv.org/abs/1711.04735 [2] Bernstein et al. "signSGD: Compressed Optimisation for Non-Convex Problems" https://arxiv.org/abs/1802.04434 [3] Loshchilov et al. "Fixing Weight Decay Regularization in Adam", https://arxiv.org/pdf/1711.05101.pdf [4] Laarhoven, "L2 Regularization versus Batch and Weight Normalization" https://arxiv.org/abs/1706.05350 [5] Hoffer et al. "Norm matters: efficient and accurate normalization schemes in deep networks" https://arxiv.org/abs/1803.01814 [6] Anonymous, "Three Mechanisms of Weight Decay Regularization" https://openreview.net/forum?id=B1lz-3Rct7 (Another ICLR 2019 submission)