Year: 2021

Abstract:
The optimisation of neural networks can be sped up by orthogonalising the gradients before the optimisation step, ensuring the diversification of the learned representations. We orthogonalise the gradients of the layer’s components/filters with respect to each other to separate out the intermediate representations. Our method of orthogonalisation allows the weights to be used more flexibly, in contrast to restricting the weights to an orthogonalised sub-space. We tested this method on ImageNet and CIFAR-10 resulting in a large decrease in learning time, and also obtain a speed-up on the semi-supervised learning BarlowTwins. We obtain similar accuracy to SGD without fine-tuning and better accuracy for naı̈vely chosen hyper-parameters.

text: 1 INTRODUCTION
Neural network layers are made up of several identical, but differently parametrised, components, e.g. filters in a convolutional layer, or heads in a multi-headed attention layer. Layers consist of several components so that they can provide a diverse set of intermediary representations to the next layer, however, there is no constraint or bias, other than the implicit bias from the cost function, to learning different parametrisations. We introduce this diversification bias in the form of orthogonalised gradients and find a resultant speed-up in learning and sometimes improved performance, see fig. 1.
Our novel contributions include this new optimisation method, thorough testing on CIFAR10 and ImageNet, additional testing on a semi-supervised learning method, and experiments to support our hypothesis.
In section 2 we detail the method and results to give an understanding of how this method works and its capabilities. Then, in section 3, we provide experimental justifications and supporting experiments for this method along with finer details of the implementation and limitations.

heading: 2 OVERVIEW OF NEW METHOD AND RESULTS

heading: 2.1 RELATED WORKS
text: Gradient orthogonalisation has been explored in the domain of multi-task learning (Yu et al., 2020) to keep the different tasks separate and relevant. However in this work we focus on orthogonalisation for improving single task performance.
Weight orthogonalisation has been extensively explored with both empirical (Bansal et al., 2018; Jia et al., 2017) and theoretical (Jia et al., 2019) justifications. However, modifying the weights during
training is unstable, and, in addition, it limits the weights to a tiny subspace. Deep learning is know to work well despite the immense size of the weight space, and as such we do not view this as an advantage. Xie et al. (2017) obtain improved performance over Stochastic Gradient Descent (SGD) via weight orthogonalisation and allows them to train very deep networks, we aim to achieve the same thing while being more flexible with model and optimisation method choice. We do this by orthogonalising the gradients before they are used by an optimisation method rather than modifying the weights themselves.

heading: 2.2 ORTHOGONALISING GRADIENTS
text: Given a neural network, f , with L layers made from components, c,
f = ◦Li=1 (fi) , (1) fl(x) = [cl1(x), cl2(x), . . . , clNl(x)], (2)
where ◦ is the composition operator, Nl is the number of components in layer l, cl : RSl−1×Nl−1 → RSl is a parametrised function and cli denotes cl parametrised with θli ∈ RPl giving fl : RSl−1×Nl−1 → RSl×Nl parametrised by θl ∈ RPl×Nl . Let
Gl = [∇cl1,∇cl2, . . . ,∇clNl ], (3) be the Pl ×Nl matrix of the components’ gradients. Then the nearest orthonormal matrix, i.e. the orthonormal matrix, Ol, that minimises the Frobenius norm of its difference from Gl
min Ol ‖Ol −Gl‖ subject to ∀i, j :
〈 Oli, Olj 〉 = δij ,
where δij is the Kronecker delta function, is the product of the left and right singular vector matricies from the Singular Value Decomposition (SVD) of Gl (Trefethen & Bau III, 1997),
Gl = UlΣlV T l , (4)
Ol = UlV T l . (5)
Thus, we can adjust a first-order gradient descent method, such as Stochastic Gradient Descent with Momentum (SGDM) (Polyak, 1964), to make steps where the components are pushed in orthogonal directions,
v (t+1) l = γv (t) l + ηO (t) l , and (6)
θ (t+1) l = θ (t) l − v (t+1) l , (7)
where vl is the velocity matrix, t ∈ Z0+ is the time, γ is the momentum decay term, and η is the step size. We call this method Orthogonal Stochastic Gradient Descent with Momentum (OrthogonalSGDM). This modification can clearly be applied to any first-order optimisation algorithm by replacing the gradients with O(t)l before the calculation of the next iterate.
Code for creating orthogonal optimisers in PyTorch is provided at https://anonymous. 4open.science/r/Orthogonal-Optimisers. And code for the experiments in this work is provided at https://anonymous.4open.science/r/ Orthogonalised-Gradients

heading: 2.3 RESULTS

heading: 2.3.1 CIFAR-10
text: We trained a suite of models on the CIFAR-10 (Krizhevsky et al., 2009) data set with a mini-batch size of 1024, learning rate of 10−2, momentum of 0.9, and a weight decay of 5 × 10−4 for 100 epochs. We then repeated this using Orthogonal-SGDM instead of SGDM and plot the results in figs. 2 and 3 and table 1.
Orthogonal-SGDM is more efficient and achieves better test accuracy than SGDM for every model we trained on CIFAR-10 without hyper-parameter tuning. The validation curves follow the training curves, figs. 4 and 5, and have the same patterns, this means that Orthogonal-SGDM exhibits the same generalisation performance as SGDM. More importantly though, we can see that the model
1As described in appendix B.1 2Model same as in He et al. (2015) 3From https://pytorch.org/vision/0.9/models.html
learns much faster at the beginning of training, as shown by fig. 2, this means that we do not need as many epochs to get to a well-performing network. This is especially good in light of the large data sets that new models are being trained on, where they are trained for only a few epochs, or even less (Brown et al., 2020).
For SGDM the performance of the residual networks designed for ImageNet (Deng et al., 2009) (18, 34, 50) get worse as the models get bigger. The original ResNet authors, He et al. (2015), note that unnecessarily large networks may over-fit on a small data set such as CIFAR-10. However, when trained with Orthogonal-SGDM, these models do not suffer from this over-parametrisation problem and even slightly improve in performance as the models get bigger, in clear contrast to SGDM. This agnosticism to over-parametrisation helps alleviate the need for the practitioner to tune a model’s architecture to the task at hand to achieve a reasonable performance.

heading: 2.3.2 MATCHING RESNET’S PERFORMANCE
text: Having shown that Orthogonal-SGDM speeds up learning with non-optimised hyper-parameters, we now aim to show that it can achieve state-of-the-art results. To do this we use the same hyperparameters as the original ResNet paper (He et al., 2015), which have been painstakingly tuned to benefit SGDM, to train using Orthogonal-SGDM.
This also tests the efficacy of Orthogonal-SGDM as a drop-in replacement for SGDM. OrthogonalSGDM gets close to the original results, table 2, even though the hyper-parameters are perfected for SGDM. It is the authors’ belief that with enough hyper-parameter tuning SGD or SGDM will be the best optimisation method; however, this experiment shows that Orthogonal-SGDM is robust to hyper-parameter choice and can easily replace SGDM in existing projects. Unfortunately, the authors do not have the compute-power to extensively hyper-parameter tune a residual network for
Orthogonal-SGDM, however, it is exceedingly likely that better results would be achieved by doing so.

heading: 2.3.3 IMAGENET
text: Orthogonal-SGDM also works on a large data set such as ImageNet (Deng et al., 2009) — fig. 6. Using a resnet34, mini-batch size of 1024, learning rate of 10−2, momentum of 0.9, and a weight decay of 5×10−4, for 100 epochs. SGDM achieves a test accuracy of 61.9% and a test loss of 1.565 while Orthogonal-SGDM achieves 67.5% and 1.383 respectively. While these results are a way off the capabilities of the model they still demonstrate a significant speed-up and improvement from using Orthogonal-SGDM, especially at the start of learning, and further reinforces how a dearth of hyper-parameter tuning impedes performance.

heading: 2.3.4 BARLOW TWINS
text: Barlow Twins (Zbontar et al., 2021) is a semi-supervised method that uses “the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample” to avoid collapsing to trivial solutions. While the authors do provide code, we could not replicate their
results by running it. To train within our compute limitations we used a mini-batch size of 1024 instead of 2048 however this should not affect the results since “Barlow Twins does not require large batches” (Zbontar et al., 2021). Additionally, Barlow Twins uses the Layer-wise Adaptive Rate Scaling (LARS) algorithm (You et al., 2017), which is designed to adjust the learning rate based on the ratio between the magnitudes of the gradients and weights, there should be no significant slowdown, or speed-up, in learning due to the mini-batch size. We do not orthogonalise the gradients for the dense layers (see section 3.5).
Comparing our own runs, we establish that orthogonalising the gradients before the LARS algorithm does speed up learning as shown in fig. 7, in agreement with previous experiments. This is evidence that orthogonalising gradients is also beneficial for semi-supervised learning and, moreover, that optimisation algorithms other than SGDM can be improved in this way.

heading: 2.3.5 IN COMPARISON TO ADAM
text: We compare our method to the Adam optimiser (Kingma & Ba, 2014). Adam has found its place as a reliable optimiser that works over a wide variety of hyper-parameter sets, yielding consistent performance with little fine-tuning needed. We see our optimisation method occupying the same space as Adam.
Our method outperforms Adam on all but one hyper-parameter set — table 3. In addition since we can apply our method to any previous optimisation method, we also test Orthogonal-Adam and find that it outperforms Adam too including at high learning rates where Adam suffers from blow-ups. See figs. 13 and 14 for the training plots.

heading: 3 DISCUSSION OF PROBLEM AND METHOD

heading: 3.1 NORMALISATION
text: When we perform SVD on the reshaped gradient tensor, we obtain an orthonormal matrix, since this changes the magnitude of the gradient we look at the effect of this normalisation. Normalised SGDM (N-SGDM) (Nesterov, 2003) provides an improvement in non-convex optimisation since it is difficult to get stuck in a local minimum as the step size is not dependent on the gradient magnitude. However, it hinders convergence to a global minimum since there is no way of shortening the step size; deep learning is highly non-convex and is unlikely to be optimised to a global minimum. Therefore, it stands to reason that normalising the gradient would speed up the optimisation of deep networks.
We compare N-SGDM to normalising the gradients per component — i.e. normalising the columns of Gl, eq. (3), instead of orthognormalising it — Component Normalised SGDM (CN-SGDM), as well as to SGDM and Orthogonal-SGDM. N-SGDM improves over SGDM, and CN-SGDM improves over N-SGDM except from the oft case where it diverges. Finally, Orthogonal-SGDM obtains the best solutions while remaining stable on all the models.

heading: 3.2 DIVERSIFIED INTERMEDIARY REPRESENTATIONS
text: Along with different parametrisations we also desire different intermediary representations, a model will perform better if its layers output N different representations as opposed to N similar ones.
Given xl are the resulting representations from the intermediary layers, xl = ( ◦li=1fi ) (x0)
where x0 is the input and xl is the intermediary representation after layer l. Then xli is the representation provided by cli.
We now look at the statistics of the absolute cosine of all distinct pairs of different latent features, Rl = {∣∣〈xli, xlj〉2∣∣ | i < j} .
The representations have smaller cosines when using Orthogonal-SGDM versus SGDM — fig. 8. In addition, Orthogonal-SGDM shows a steady decline in cosine similarity throughout training. This indicates that more information is being passed to the next layer as the network is trained.

heading: 3.3 DEAD PARAMETERS
text: Dead parameters occur when the activation function has a part with zero gradient, e.g. a Rectified Linear Unit (ReLU). If the result of the activation remains in this part, then the gradients of the
preceding parameters will be zero and prevented from learning. This limits the model’s capacity based on a parameterisation, however temporarily dead parameters can be beneficial and act as a regulariser, similar to dropout. To detect temporarily dead parameters, we simply look for parameters with zero gradient. Comparing the amount of dead parameters produced by SGDM versus Orthogonal-SGDM, figs. 9a and 9b respectively, shows that Orthogonal-SGDM ends with around and order of magnitude more temporarily dead parameters. This implies a much higher regularisation which helps to explain Orthogonal-SGDM’s insensitivity to over-parametrisation.

heading: 3.4 IMPLEMENTATION DETAILS
text: While the QR decomposition is the most common orthogonalisation method, it is, in practice, less stable as the gradients are rank deficient (Demmel, 1997, Section 3.5), i.e. they have at least one small singular value. Orthogonal-SGDM has a longer wall time than SGDM because of the added expense of the SVD which has non-linear time complexity in the matrix size. In practice, we have found that the calculation of the SVD is either more than made up for by the speed up in iterates or a prohibitively expensive cost, with dense layers being the largest and so most problematic.
While there exist methods for computing an approximate SVD which are faster, we have used PyTorch’s default implementation since we are more concerned with Orthogonal-SGDM’s performance and efficiency in iterates and not in wall time. Even so the overhead is small, training a resnet20 as in section 2.3.1 takes 720.3 seconds with 96.4 of them taken up by the SVD calculation — an increase of 15.5% over normal SGDM. While this is a significant amount of time we can see that our method can take only 2% of the number of epochs to reach the same accuracy — fig. 2.
It is doubtful that convergence of SVD is needed, so a custom matrix orthogonalisation algorithm, that has the required stability but remains fast and approximate, will reduce the computation overhead significantly and may allow previously infeasible networks to be optimised using OrthogonalSGDM. However, we note that even with a more suitable implementation, this method would still bias towards many smaller layers for a deeper, thinner network.
3.5 FULLY CONNECTED LAYERS
Fully connected or dense layers also fit our component model from eq. (2) where the components are based on the inner product of the input and the parametrisation,
cli(x) = σ(〈flatten(x), θli〉),
where σ is an activation function, Sl = 1 giving fl : RSl−1×Nl−1 → RNl and θl ∈ RSl−1·Nl−1×Nl as desired. Intuitively, each column of the weight matrix acts as a linear map resulting in one item in the output vector. Thus,
the gradients of fully connected layers can also be orthogonalised.
As noted in section 3.4 the extra wall time is dominated by the largest parameter, this is often the dense layer; table 5 shows that for CIFAR-10 there is no impact on the error rate from not orthogonalising the final dense layer, and the training curves are the same shape — fig. 10. While both the error rates and losses decrease when not orthogonalising the dense layer we hesitate to say that orthogonalising dense layers is detrimental since these networks only have a dense final classification layer which is qualitatively different from intermediary dense layers.

heading: 3.6 LIMITATIONS DUE TO MINI-BATCH SIZE
text: Orthogonal-SGDM does not perform as well as SGDM when the mini-batch size is extremely small, fig. 11, due to the increased levels of noise for the SVD. This is the most likely reason that the resnet20 from section 2.3.2 fails to match the original performance.
A mini-batch size of 16 is where OrthogonalSGDM starts to outperform SGDM on a resnet18 for CIFAR-10. Few models need such small mini-batch sizes, but if they do then SGDM would be a more suitable optimisation algorithm. In addition to the learning collapse, the time taken by SVD is only dependent on the parameter size and not the mini-batch size, so increasing the number of mini-batches per epoch also increases the wall time to train. The
reason for the collapse in training with small mini-batch sizes will be subject to further research.

heading: 4 CONCLUSION
text: In this work we have laid out a new optimisation method, tested it on different models and data sets, showing close to state-of-the-art results out of the box and robustness to hyper-parameter choice and over-parametrised models. Orthogonal-SGDM also has practical application in problems such as object detection and semantic segmentation since they make use of a pre-trained image classification backbone.
SGDM with a vast amount of hyper-parameter tuning still reigns supreme, but Orthogonal-SGDM is an excellent method for quick verification of models or for prototyping — when we want decent results fast, but do not need the absolute best performing model. However, as more data set sizes are growing more models are being trained on fewer to less than one epoch of data (Brown et al., 2020) leading to an extremely limited ability to tune the hyper-parameters.
Lastly, we mentioned briefly in section 1 how attention heads fit our model but, since they are beyond the scope of this work, we will explore the potential gain in using Orthogonal-SGDM with them in future work, and expect a similarly exceptional gain will be obtained.

heading: A COSINE THRESHOLD
text: We use the cosine metric 〈x, y〉2 =
x · y ‖x‖2‖y‖2
(8)
in this work since it allows the comparison of the directions of high-dimensional vectors, and so obtain insight about the surface we are optimising on. However, we note that the “significance” of the cosine between two random vectors depends on their size.
Assuming both vectors’ components are random variables
x = [x1, x2, . . . , xN ] ,
where xi ∼ N (0, σ2), then the components of their dot product
〈x, y〉 = [x1 · y1, x2 · y2, . . . xN · yN ] ,
have variance σ4. Now, from the central limit theorem, the dot product has variance σ 4
N where N is the size of the vectors. Finally, dividing by the magnitude of the vectors gives a variance of σ̄ = σ 2 N for the cosine metric.
To gain some understanding of the significance of a cosine distance, we define a four-sigma threshold on the distribution of cosines, so, assuming σ = 1, µ = 0, we get a threshold value of
µ± 4σ̄ = 0± 4σ√ N
= ± 4√ N
(9)
This is important because a distance of 0.1 might seem small, but for 10, 000-dimensional vectors, it easily clears our significance threshold.

heading: B MODEL SUMMARIES
text: B.1 BASICCNN
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− Layer ( t y p e ) Outpu t Shape Param #
================================================================ Conv2d−1 [−1 , 32 , 16 , 16] 896
BatchNorm2d−2 [−1 , 32 , 16 , 16] 64 Conv2d−3 [−1 , 32 , 8 , 8 ] 9 ,248 BatchNorm2d−4 [−1 , 32 , 8 , 8 ] 64 Conv2d−5 [−1 , 32 , 4 , 4 ] 9 ,248 BatchNorm2d−6 [−1 , 32 , 4 , 4 ] 64 L inea r−7 [−1 , 10] 5 ,130
BasicCNN−8 [−1 , 10] 0 ================================================================ T o t a l params : 24 ,714 T r a i n a b l e params : 24 ,714 Non− t r a i n a b l e params : 0 −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− I n p u t s i z e (MB) : 0 . 0 1 Forward / backward p a s s s i z e (MB) : 0 . 1 6 Params s i z e (MB) : 0 . 0 9 E s t i m a t e d T o t a l S i z e (MB) : 0 . 2 7 −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
C FULL RESULTS PLOT

heading: 10−3 0.9 1.0977± 0.0036 0.5042± 0.0055 0.6045± 0.0312 0.5829± 0.0102

heading: 10−2 0.9 0.6217± 0.0194 0.4841± 0.0087 0.8898± 0.0447 0.5903± 0.0201

heading: 10−1 0.9 0.6089± 0.0535 0.5370± 0.0230 9.0193± 5.9607 0.9803± 0.1607

Review summaries:
This paper proposes to orthogonalize the gradients of each neuron (within a layer) in order to improve training dynamics. They build on the assumption that neurons in a layer tend to learn similar features early in the training, and thus pushing them in orthogonal directions would yield to better results. The method requires calculating SVD on gradients at every iteration, thus unlikely to be practical (no FLOPs/run-time discussion). Training curves on Cifar10 and Imagenet shows faster convergence and better generalization.