Orthogonalising gradients to speedup neural network optimisation
Authors: Mark Tuddenham
Adam Prugel-Bennett
Jonathon Hare
Venue: ICLR 2022 Submitted (2022)
Keywords: machine learning
deep learning
orthogonalisation
optmisation
optimization

Abstract: The optimisation of neural networks can be sped up by orthogonalising the gradients before the optimisation step, ensuring the diversification of the learned representations. We hypothesize that components in the same layer learn the same representations at the beginning of learning. To prevent this we orthogonalise the gradients of the components with respect to each other.
Our method of orthogonalisation allows the weights to be used more flexibly, in contrast to restricting the weights to an orthogonalised sub-space. We tested this method on ImageNet and CIFAR-10 resulting in a large decrease in learning time, and also obtain a speed-up on the semi-supervised learning BarlowTwins. We obtain similar accuracy to SGD without fine-tuning and better accuracy for naïvely chosen hyper-parameters.


Under review as a conference paper at ICLR 2022
ORTHOGONALISING GRADIENTS TO SPEEDUP NEURAL NETWORK OPTIMISATION
Anonymous authors Paper under double-blind review
ABSTRACT
The optimisation of neural networks can be sped up by orthogonalising the gradi- before the optimisation step, ensuring the diversification of the learned repre- sentations. We orthogonalise the gradients of the layer's components/fiters with respect to each other separate out the intermediate representations. Our method of orthogonalisation allows the weights to be used more flexibly, in contrast to restricting the weights to an orthogonalised sub-space. We tested this method on ImageNet and CIFAR- 10 resulting in large decrease in learning time, and also obtain speed-up on the supervsdd learning BarlowTwins. We obtain sim- ilar accuracy to SGD without fine-tuning and better accuracy for naively chosen hyper-parameters.
INTRODUCTION
Neural network layers are made up of several identical, but differently parametrised, compo- nents, e.g. filters in convolutional layer, or heads in a multi-headed attention layer. Lay- consist of several components that they can provide a diverse set of intermediary rep- resentations the next layer, however, there no constraint or bias, other than the implicit bias from the function, learning differ- ent parametrisations. We introduce this diversi- fication bias in the form of orthogonalised gra- dients and find resultant speed-up in learn- ing and sometimes improved performance, see fig. 1.
0.84
6.7
Walaacinn Accuracy
033
eesree114 SGD
near 6118 with orth sec
Figure 1: An example of the speed-up obtained by orthogonalising the gradients on CIFAR-IO.
Our novel contributions include this new opti-
misation method, thorough testing on CIFAR- 10 and ImageNet, additional testing on a semi-supervised learning method, and experiments to sup- port our hypothesis.
In section 2 we detail the method and results to give an understanding of how this method works and its capabilities. Then, in section 3, provide experimental justifications and supporting exper- iments for this method along with finer details of the implementation and limitations.
OVERVIEW OF NEW METHOD AND RESULTS
2.1 RELATED WORKS
Gradient orthogonalisation has been explored in the domain of multi-task learning (Yu et al., 2020) to keep the different tasks separate and relevant. However this work we focus on orthogonalisation for improving single task performance.
Weight orthogonalisation has been extensively explored with both empirical (Bansal et al., 2018; 1018;AM Jia al., 2017) and theoretical (Jia al., 2019) justifications. However, modifying the weights during

---

Under review as a conference paper at ICLR 2022
training is unstable, and, in addition, it limits the weights to tiny subspace. Deep learning is know to work well despite the immense size of the weight space, and as such we do not view this as an advantage. Xic et al. (2017) obtain improved performance over Stochastic Gradient Descent (SGD) via weight orthogonatisation and allows them to train very deep networks, we aim to achieve the same thing while being more flexible with model and optimisation method choice. We do this by orthogonalising the gradients before they are used by an optimisation method rather than modifying the weights themselves.
2.2 ORTIIOGONALISING GRADIENTS
Given a neural network, f, with layers made from components, c,
f of (fi), f((x) = (c)( CN (x))
(I)
(2)
where is the composition operator, N2 is the number of components in layer 1, S R is parametrised function and O₁ denotes a parametrised with fts E I giving fi R S S.X parametrised by RR PxM.
Let
G: = Ven, Van..... VaNl.
be the Pt x Nt matrix of the components' gradients.
Then the nearest orthonormal matrix, i.e. the orthonormal matrix, O₁, that minimises the Frobenius norm of its difference from G₁
min ||O Grll subject Vi, (Or,, Oi,) =ōiz O:
where dig is the Kronecker delta function, is the product of the left and right singular vector matricies from the Singular Value Decomposition (SVD) of G₁ (Trefethen & Bau III, 1997).
G: UEN,"
(4)
O U.V.
(5)
Thus, we can adjust first-order gradient descent method, such as Stochastic Gradient Descent with Momentum (SGDM) (Polyak, 1964), to make steps where the components are pushed in orthogonal directions,
(++1)
(2) 01 and
v 700
0 (++) = 8/0) - (I+1)
where in is the velocity matrix, E Z is the time, y the momentum decay term, and 17 is the step size. We call this method Orthogonal Stochastic Gradient Descent with Momentum (Orthogonal- SGDM). This modification can clearly be applied to any first-order optimisation algorithm by re- placing the gradients with 0 before the calculation of the next iterate.
Code for creating orthogonal optimisers in PyTorch is provided at https: http:///nnnnyouuuuu 4cpen.scient hatpnc//e///rhhooaaaaaooea
And code for the experi- ments in this work is provided at https:: /atp://////nnmmsuumamm Aeppryenous..peenn.cceeeeee/// science/ci)n sewrgoonnaledd-rrdddaaaa
2.3 RESULTS
2.3.1
CIFAR-10
We trained a suite of models on the CIFAR-10 (Krizhevsky et al., 2009) data set with a mini-batch size of 1024, learning rate of 10 momentum of 0.9, and weight decay of 5 10-4 for 100 epochs. We then repeated this using Orthogonal-SGGe instead of SGDM and plot the results in figs. and 3 and table 1.
2

---

Under review as a conference paper at ICLR 2022
Table 1: Test loss and accuracy across suite of models on CIFAR-10 comparing normal SGDM with Orthogonal-SGDM, standard error across five runs.
Test Loss
Test Accuracy (%)
SGDM
Orthogonal-SGDM
SGDM
Orthogona--GGDM
BasicCNN'
0.7603 = 0.0061
0.6808 + 0.0038
73.60 = 0.19
76.67 0.10
resnet20
0.6728 + 0.0301
0,6766 + 0.0155
87.12 = 0.12
resnet442
0.7000 + 0.0166
0.7600 0.0299
79.81 + 0.37
88.12 + 0.20
resnet18
0.9656 0.0104
0.8427 0.0121
77.01 + 0.21
84.68 0.12
resnet34
1.0468 0.0134
0.7087 = 0.0165
75.86 = 0.26
85.42 0.33
resnet503
1.2304 + 0.0462
0.6797 + 0.0235
67.99 + 0.73
86.51 + 0.12
densenet121
1.0027 + 0.0132
0.8669 0.0132
75.26 + 0.30
84.34 0.15
densenet161
1.1399 + 0.0096
1.1688 0.1960
75.81 + 0.20
85.51 + 0.19
resnext50322x4d
1.2470 + 0.0254
0.6669 + 0.0223
68.73 + 0.30
86.37 + 0.24
wide_resnet50_22
1.4141 + 0.0337
0.7018 + 0.0091
69.42 + 0.33
87.30 + 0.12
04
eCArTD a K00
sCeeC0 11 0-h 500
denemcizi mn1
WOcroors30 wth 200
WEL.OOssP wth 200
03
20
spece
Figure 2: Validation accuracy from one run of SGDM vs Orthogonal-SGDM for a selection of models. Full plot in appendix C. Best viewed in colour.
30
KDRc50 580
TORC20 81 500
T0ne22 0-2
demsncci2i 04H
Prcc/oonc50,3 eth 200
WSstyooss3)_ wth
1.4
Figure 3: Validation losses from one run SGDM vs Orthogonal-SGDr. for selection of models. Full plot in appendix C. Best viewed in colour.
Orthogonal-SGDM is more efficient and achieves better test accuracy than SGDM for every model we trained on CIFAR-IO without hyper-parameter tuning. The validation curves follow the training curves, figs. 4 and 5, and have the same patterns, this means that Orthogonal-SGDMM exhibits the same generalisation performance as SGDM. More importantly though, we can see that model
As described appendix B.I
Model same as in He et al. (2015)
From https://pytoreet /wtgvviiioo///oo /0. 9/mode 1s. htmm

---

Under review as a conference paper at ICLR 2022
NineSS wE1 6-5 2G24
500e
demanentit with 550M
demineett2 with 500M
50te
2
7pch
Figure 4: Train accuracy from one run of SGDM vs Sethogonal-SGDM. for selection of models. Best viewed in colour.
FESRE500 SOOM
rESECTS WITh D/IN 500V
500M
40
reroonettZt with 540M
derioree127 with 261M
recre50 with Cra SG2M
1.0
26
106
Socch
Figure 5: Train losses from one run of SGDM vs Orthogonal-SGDD SGDM for selection of models. Best viewed in colour.
learns much faster at the beginning of training, as shown by fig. 2, this means that we do not need as many epochs get to a well-performing network. This especially good in light of the large data sets that new models are being trained on, where they are trained for only few epochs, or even less (Brown et al., 2020).
For SGDM the performance of the residual networks designed for ImageNet (Deng et al., 2009) (18, 34, 50) get worse as the models get bigger. The original ResNet authors. He et al. (2015). note that unnecessarily large networks may over-fit a small data set such as CIFAR-10. However, when trained with Orthogonal-SGDDa and even slightly improve in performance as the models get bigger, in clear contrast to SGDM. This agnosticism to over-parametrisation helps alleviate the need for the practitioner to tune model's architecture to the task at hand to achieve reasonable performance.
these models do not suffer from this over-parametrisation problem
2.3.2 MATCHING RESNET'S PERFORMANCE
Having shown that Orthogonal-SGDD speeds up learning with non-optimised pappe--rraeetess we now aim to show that it can achieve state-of-the-an results. To do this we use the same hyper- parameters as the original ResNet paper (He al., 2015), which have been painstakingly tuned to benefit SGDM, to train using Orthogonal-SGDM.
This also tests the efficacy Orthogonal-SGDM as a drop-in replacement for SGDM. Orthogonal- SGDM gets close to the original results, table 2, even though the hyper-parameters are perfected for SGDM. It is the authors' belief that with enough hyper-paramen be the best optimisation method; however, this experiment shows that Orth is robust to hyper-paramett choice and can easily replace SGDM in existing projects. Unfortunately. the authors do not have the power to extensively hyper-paramedt tune residual network for
tuning SGD or SGDM will
4

---

Under review as a conference paper at ICLR 2022
Table Test loss and accuracy of resnet20, in He et al. (2015), on CIFAR-10; hyper-par parameeer tuned to normal SGDM vs Orthogonal-SGDDM. standard error across five runs. Mini-bate size of 128, see section 3.6 for why this hyper-parameter value impedes Orthogonal-SGDM learning-rate rate of 0.1, momentum of 0.9, weight-decay of 10 and a learning rate schedule of x0.1 at epochs 100, 150 for 200 epochs.
Test Loss
Test Accuracy (%)
SGDM (He et al., 2015)
91.25
SGDM Orthggona--SGDM
0.4053 + 0.0054 0.4231 + 0.0043
91.17 + 0.28
90.18 18 0.30
Orthogonal-SGDM. however, is exceedingly likely that better results would be achieved by doing
SO.
2.3.3 IMAGENET
Figure Validation accuracy SGDM vs Orthogonal-SGDM on ImageNet
Orthogonal-SGDM also works large data set such ImageNet (Deng et al., 2009) fig. 6. Using a resnet34, mini-batch size of 1024, learning rate of 10-2, momentum of 0.9, and a weight decay of 5 x 10-1 for 100 epochs. SGDM achieves test accuracy of 61.9% and test loss of 1. 1.565 while Orthogonal-SGDM achieves 67.5% and 1.383 respectively. While these results are a way off the capabilities of the model they still demonstrate significant speed-up and improvement from using Orthogonal-GGDM especially at the start of learning, and further reinforces how a dearth of hyper-parameter tuning impedes performance.
2.3.4 BARLOW TWINS
Con LARS
300
203
Figure 7: Barlow Twins loss during the unsupervised phase using LARS and Orthogonal LARS on ImageNet
Barlow Twins (Zbontar al., 2021) is semi-supervised method that uses "the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of sample" to avoid collapsing trivial solutions. While the authors do provide code, could not replicate their
5

---

Under review as a conference paper at ICLR 2022
results by running it. To train within compute limitations used mini-batch size of 1024 instead of 2048 however this should not affect the results since "Barlow Twins does not require large batches" (Zbontar et al., 2021). Additionally, Barlow Twins uses the Layer-wise Adaptive Rate Scaling (LARS) algorithm (You et al., 2017), which is designed to adjust the learning rate based on the ratio between the magnitudes of the gradients and weights, there should be significant slow- down, or speed-up, in learning due to the mini-batch size. We do not orthogonalise the gradients for the dense layers (see section 3.5).
Comparing our own runs, we establish that orthogonalising the gradients before the LARS algorithm does speed up learning as shown fig. agreement with previous experiments. This is evidence that orthogonalising gradients is also beneficial for semi-supervised learning and, moreover, that optimisation algorithms other than SGDM can improved in this way.
2.3.5
IN COMPARISON TO ADAM
We compare our method to the Adam optimiser (Kingma Ba, 2014). Adam has found its place as reliable optimiser that works over wide variety of hyper-parameter sets, yielding consistent performance with little fine-tuning needed. We see our optimisation method occupying the same space as Adam.
Test accuracy across a suite of hyper- prer--pareetrr sets on CIFAR-10 on resnet20, standard error across five runs. For Adam B3 0.99.
Table 3:
SGDM
Adam
LR
Momentum
Original
Orthogonal
Original
Orthogonal
10-1
0.95
83,59 2.09
85.58 :: 0.98
38.95 1: 8.95
76.84 +: 1.53
10-2
0.95
82.66 1.0
87.72 = 0.44
74.23 :: 2.38
86.48 0.17
10-3
0.95
66.59 0.44
85.88
83.08 0.76
85.12 0.6
10-1
0.9
82.52
85.06
28. 28.264770 7.16
73.62.
10-2
0.9
79.96 0.48
87.44 0.25
73.46+ 1.19
85.26 0.38
10
0.9
60.69 0.18
84.67 0.21
83. 6+ 0.66
85.25 + 0.31
10-1
0.8
84.16 0.43
86.01
0.74
27.50 =: 6,76
71.88 4:25
10
0.8
77.42 + 0.9
87.18
0.12
72.60
# 1:76
86.75 + 0.26
10-3
0.8
53.21
82.95
= 0.40
80.89
+ 1.93
85.52
10-1
0.5
80.08 0.36
87.37
d: 0.18
18.39
4: 4.84
72. d: 2.83
10-2
0.5
68.64 1.05
86.05
:: 0.10
71.62
: 1.95
84.22 0.69
10 0.5
43.51 1.02
78.68 0.77
81.67 1.05
84.21 0.43
Our method outperforms Adam on all but one hyper-parameter set set-taile table 3. addition since we can apply our method to any previous optimisation method, we also test Orthogonal- -Adam and find that it outperforms Adam too including at high learning rates where Adam suffers from blow-ups. See figs. 13 and 14 for the training plots.
3 DISCUSSION OF PROBLEM AND METHOD
3.1 NORMALISATION
When we perform SVD on the reshaped gradient tensor, we obtain an orthonormal matrix, since this changes the magnitude of the gradient we look at the effect of this normalisation. Normalised SGDM (N-SGDM) (Nesterov, 2003) provides an improvement in non-convex optimisation since is difficult to get stuck in a local minimum as the step size is not dependent However, hinders convergence to a global minimum since there is no way of shortening the step size; deep learning is highly non-convex and is unlikely be optimised Therefore, stands to reason that normalising the gradient would speed up the optimisation of deep networks.
the gradient magnitude.
global minimum.

---

Under review as a conference paper at ICLR 2022
We compare N-SGDM to normalising the gradients per component i.e. normalising the columns of Gt. eq. (3), instead of orthognormalising it - Component Normalised SGDM (CN-SGDM), well to SGDM and Orthogonal-SGDio SGDM. N-SGDM improves over SGDM, and CN-SGDM improves over N-SGDM except from the oft case where it diverges. Finally, Orthogonal-SGiM obtains the best solutions while remaining stable all the models.
Table 4: Test accuracy for several models trained with SGDM, Normalised SGDM, Component Normalised SGDM, and Orthogonal-SGDM; trained as in section 2.3.1.
SGDM
N-SGDM
CN-SGDM Orthogonal
BasicCNN
73.68 027
73.72 0AS
74.53 0.32
76.75 0:25
resnet18
76.83
78,94
0,00
84.94
resnet50
09.
79.35 0:11
0.00
86.59 = 0.10
resnet44
79.73 1:27
83,60
84.44
87.49
densenet121
79.06 0.04
0.00 0.00
84,86 0/07
3.2 DIVERSIFIED INTERMEDIARY REPRESENTATIONS
Along with different parametrisations we also desire different intermediary representations, a model will perform better if its layers output N different representations as opposed to N similar ones.
Given xı are the resulting representations from the intermediary layers,
JJ = (1)= (ro)
where To is the input and zι is the intermediary representation after layer 1. Then 24, is the repre- sentation provided by Ct.-
We now look at the statistics of the absolute cosine of all distinct pairs of different latent features,
R₂ = (Trs #15\2| <)}
The representations have smaller cosines when using Orthogonal -SMM -SGM versus SGDM - fig. 8. In
addition, Orthogonal-SGDD shows steady decline in cosine similarity throughout training. This indicates that more information is being passed to the next layer as the network is trained.
0.6
MeAA WT Otth
askcainc
0.3
0.
0.1
0.0
2006
0000
1000
800D
10000
12000
24000
Butch + /30c
Figure 8: Mean of the absolute cosine of all distinct pairs of different intermediary representations,
E [R] E (1,2,3), for all layers of a BasicCNN trained on CIFAR- 10 as in section 2.3.1.
3.3 DEAD PARAMETERS
Dead parameters occur when the activation function has part with zero gradient, egg. a Rectified Linear Unit (ReL.U.. If the result of the activation remains in this part, then the gradients of the

---

Under review as a conference paper at ICLR 2022
preceding parameters will be zero and prevented from learning. This limits the model's capacity based on a parameterisation, however temporarily dead parameters can beneficial and act as a regulariser, similar to dropout. To detect temporarily dead parameters, simply look for param- eters with zero gradient. Comparing the amount dead parameters produced by SGDM versus Orthogonal-SGiDM. figs. 9a and 9b respectively, shows that Orthogonal-SGDDM ends with around and order of magnitude more temporarily dead parameters. This implies a much higher regularisa- tion which helps to explain Sathoognal-SGDMM. insensitivity to pece-aaramttiistionn.
17000
FIEEIS OURT JHO0
4008
2006
3500
205r
3000
(a) With SGDM.
(b) With Orthogonal-SGDM.
Figure 9: Number of temporarily dead parameters layer2[1].conv2 of a resnet50 trained as in
section 2.
3.4 IMPLEMENTATION DETAILS
While the QR decomposition is the most common orthogonalisation method, it is, in practice, less stable as the gradients are rank deficient (Demmel, 1997, Section 3.5), i.e. they have at least one small singular value. Orthogonal-SGDM has longer wall time than SGDM because of the added expense of the SVD which has non-linear time complexity in the matrix size. In practice, we have found that the calculation of the SVD is either a prohibitively expensive cost, with dense layers being the largest and so most problematic.
than made up for by the speed up in iterates
While there exist methods for computing approximate SVD which PyTorch's default implementation since we are more concerned with Orthogonal-SGDM's perfor- mance and efficiency in iterates and not in wall time. Even so the overhead small, training resnet20 as in section 2.3.1 takes 720.3 seconds with 96.4 of them taken up by the SVD calculation an increase of 15.5% over normal SGDM. While this is a significant amount of time we can see that our method can take only 2% of the number of epochs to reach the same accuracy - fig. 2.
faster, we have used
doubtful that convergence of SVD is needed, so custom matrix orthogonalisation algorithm, that has the required stability but remains fast and approximate, will reduce the computation over- head significantly and may allow previously infeasible networks to be optimised using Orthogonal- SGDM. However, we note that even with a more suitable implementation, this method would still bias towards many smaller layers for a deeper, thinner network.
3.5 FULLY CONNECTED LAYERS
Fully connected or dense layers also fit our component model from eq. (2) where the com- ponents are based on the inner product of the input and the parametrisation,
Ci = a ((flatten(z)), 0t)))
where is activation function, St giving fi S x + RR and 9; E S I-N; 1xNN as desired. Intuitively, each col- umn of the weight matrix acts as linear map resulting in one item in the output vector. Thus,
Figure 10: Orthogonalising just the convolutional filters vs both the convolutional layers and fi- nal dense layer on CIFAR-10; trained as in sec- tion 2.3.1.
8

---

Under review as a conference paper at ICLR 2022
the gradients of fully connected layers can also be orthogonalised.
Table 5: Test accuracy and loss for Orthogonal- SGDM on CIFAR-10 when orthogonalising all layers vs orthogonalising just the convolutional layers. Trained as in section 2.3.1, standard error across five runs.
SGDM
Orthogonal-Sae Maroggonl--GDMMM SGGM
Conv Orthogonal-SGite
Loss
Acc (%)
Loss
Ace (%)
Loss
Acc (%)
BasicCNN
0.7603 0.061
73.60 0.199
0.6808 0.0388
76.67 0:100
0.6732 i 0.001
76.80
resnet34
1.0468 i 00134
75.86 0266
0.7087 i 0.0165
85.42 0:333
0.6268 +: 0.0005
85.68 0.1
resnet20
0.6728 + 0.3011
79.14 0.6
0.6766 + aois
87.12 2012
0,4824 0,025
87.70 0.40
As noted in section 3.4 the extra wall time is dominated by the largest parameter, this often the dense layer; table 5 shows that for CIFAR-10 there is no impact on the error rate from not orthogonalising the final dense layer, and the training curves are the same shape - fig. 10. While both the error rates and losses decrease when not orthogonalising the dense layer hesitate to say that orthogonalising dense layers is detrimental since these networks only have a dense final classification layer which is qualitatively different from intermediary dense layers.
3.6 LIMITATIONS DUE TO MINI BANI-BATCH SIZE
Orthogonal-SGIDM does not perform as well as SGDM when the mini-hatch size is extremely small, fig. 11, due to the increased levels of noise for the SVD. This is the most likely rea- son that the resnet20 from section 2.3.2 fails to match the original performance.
0.7
valdsbin Accurscy (h0
A mini-batch size 16 is where Orthogonal- SGDM starts outperform SGDM resnet18 for CIFAR-10. Few models need such small mini-batch sizes, but if they do then SGDM would be more suitable optimisation algorithm. In addition the learning collapse, the time taken by SVD is only dependent on the parameter size and not the mini-batch size, so increasing the number of mini-batches per epoch also increases the wall time to train. The reason for the collapse in training with small mini- batch sizes will be subject to further research.
resnette with 560
nenett: with Orth 500
10
20
Opoch
Figure 11: CIFAR-10 with mini-batch size=4 trained as in section 2.3.
CONCLUSION
In this work we have laid out a new optimisation method, tested it on different models and data sets, showing close state-of-the-ar results out of the box and robustness to hyper-parameter choice and over-paran per-parrmtriieed models. Orthogonal-SGDM also has practical application in problems such as object detection and semantic segmentation since they make use of a pre-trained image classification backbone.
SGDM with a vast amount of hyper-parameter tuning still reigns supreme, but Orthogonal-SGDD) an excellent method for quick verification of models for prototyping when want decent results fast, but do not need the absolute best performing model. However, as more data set sizes are growing more models are being trained fewer to less than epoch of data (Brown et al., 2020) leading to an extremely limited ability to tune the hyper-paramedt parameters.
Lastly, mentioned briefly in section how attention heads fit our model but, since they beyond the scope of this work, we will explore the potential gain in using Orthogonal with them in future work, and expect similarly exceptional gain will be obtained.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regular- izations in training deep cnns? arXiv preprint arXiv:1810.09100 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
James W Demmel. Applied numerical linear algebra. SIAM, 1997.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. corr abs/1511..055 (2015), 2015.
Kui Jia, Dacheng Tao, Shenghua Gao, and Xiangmin Xu. Improving training of deep neural net- works via singular value bounding. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4344 4352, 2017.
Kui Jia, Shuai Li, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks. IEEE transactions on pattern analysts and machine intelligence, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint orXiv:1412.69800 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87, Springer Science & Business Media, 2003.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr compu- lational mathematics and mathematical physics, 4(5):1-17, 1964.
Lloyd N Trefethen and David Bau III. Numerical linear algebra, volume 50. Siam, 1997.
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6176- 6185, 2017.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks, 2017.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction, 2021.
10

---

Under review as a conference paper at ICLR 2022
A
COSINE THRESHOLD
We use the cosine metric
20
(x, u)2
(8)
|
in this work since it allows the comparison of the directions nigh-dimensional vectors, and so obtain insight about the surface optimising However, note that the "significance" of the cosine between two random vectors depends on their size.
Assuming both vectors' components random variables
[21, 22, F2, [NN
where ~ NO, o²), then the components of their dot product
(z, y) = [21 y/. 22 ........ IN yN],
have variance of, Now, from the central limit theorem, the dot product has variance &x where N is the size of the vectors. Finally, dividing by the magnitude of the vectors gives a variance of d² the cosine metric.
TV
To gain some understanding of the significance of cosine distance, we define four-sigma threshold on the distribution of cosines, so, assuming a = 1, / 0, we get threshold value of
+ 47 = 0 4Ø
VN
This is important because distance of 0.1 might seem small, but for 10, 000-dimensional vectors, it easily clears our significance threshold.
B MODEL SUMMARIES
B. BASICCNN
Layer (type)
Output Shape
Param #
Conv2d-1
[-1, 32, 16, 16]
896
BatchNorm2d -2
[-1, 32, 16,
16]
64
Conv2d-1
[ [- 1, 32. 8, 8]
0,248
Natch 4
[-1, 1, 32, 8, 8]
64
Conv2d
[- 32, 4, 4]
9. 9,248
BatchNorm2d
[-1, 1, 32, 4, 4]
64
Linear
[-1, 10]
5. 53130 0
BasicCNN-S
[-1. 10]
Total params: 24,714
Trainable params: 24.714
Non-trainable params: 0
Input size (MB): 0.01
Forward / backward pass size (MB): 0.16
Params size (MB): 0.09
Estimated Total Size (MB): 0.27
11

---

Under review as a conference paper at ICLR 2022
C
FULL RESULTS PLOT
BasicCNN
BasicCNN with Orth SGD
0.8
resnet18 with Orth SGD
resnet34 with Orth SGD
Validatonn Accuracy (h)
resnet50 with Orth 5GD
0.0
resnet20 with Orth 50D
0.5
resnet44 with Orth 500
densenet121 with Orth 50D
densenet161 with Orth 500
0.3
resnext50, 22200d_2xdddd_ with Orth SGD
vide_resnet50_22 wide resnet50 with Orth 5GD
0.1
20
60
80 100
(a) Validation Accuracy
3.0
BasicCNN with SGD
BasicCNN
resnet18 with SGD
2.5
resnet34 with SGD
SGD
resnet50 with SGD
2.0
SGD
Validation Loss
resnet20 with SGD
15*
resnete4 50D
resnet44 with Orth 500
densenet121 with densenet121 with Orth SGD
densenet16] with densenet161 with Orth SGD resnext50 32x4d
resnext50, 32x4d with Orth SGD
wide resnet50
wide_resnet50_2 with Orth
100
Epoch
(b) Validation Loss
Figure 12: SGDM vs Orthogonal SGDM
12

---

Under review as a conference paper at ICLR 2022
Table Test loss across suite of hyper paper-paraeeter sets on CIFAR-10 on resnet20, standard error across five runs. For Adam B2 0.99.
SGDM
Adam
LR
Momentum
Original
Orthogonal
Original
Orthogonal
10-1
0.95
0.5487 0.0133
0.4916
00420
2.4148
0.7938
10 0.95
0.5762 + 0,0311
0.4772
+ 00277
0.8395 + 0.79
0.5415 0.0533
10-3 3
0.95
0.9323 + 0.0149
0.4988
+ 00114
0.6013 # 00805
0.5998 0.0311
10-1
0.9
0.6089 = 0,0535
0.5370
0.0300
9.0193 5.9007
0.9803 0,1677
10 10-2 2
0.9
0.6217# + 0.0194
0.4841
0.0087
0.8898 0.0447
0.5903 0.0201
10
0.9
1.0977 0,0036
0.5042
0.0055
0.6045
0.5829 : 0.0102
10-1 0.8
0.5395 0.0091
0.54
1: 0.0499
4.5143 166628
1.0984 0.2710
10-2
0.8
0.6669 0.0290
0.4940
0.0163
0.8741
0,5114 0.0004
10-3
0.8
1.2805
0.0106
0.5238
0.0034
0.6950
0.0656
0.567
0,0088
10-1
0.5
0.6796
0.0158
0.5003
00148
1.68338291
15681,5352
0.9481 1 0.1072
10-2
0.5
0.8927
+ 0.0304
0.4950
0.0121
0.9594
0.0846
0.6610
0,0385
10-3
0.5
1,5309
+ 0.0189
0.6284
+ 0.0186
0.6482
+ 0.011
0.6228
0.0213
Table 7: Test loss for several models trained with SGDM, Normalised SGDM, Component Nor- malised SGDM, and Orthogonal-SGDM.
SGDM
N-SGDM
CN-SGDM
Orthogonal-SGDM
BasicCNN
0.7559+ 0.065
0.7637 0.008
0.7443 0.0994
0.6824 0.0081
resnet18
0.9252 1 0.0098
0,9726 0.0214
nan nan
0.7938 0.0083
resnet50
1.0950 0.0181
0.9454 0.0126
nan
0.6785 ::
resnet44
0.70934 0.0678
0.7641
0.0441
0.7902+
0.0477
0.7694 0.0426
densenet121
0.9357+ 0.0771
1.0096
nan
0.8142
13

---

Under review as a conference paper at ICLR 2022
resnetzo with SGDM resnet20 with Orth SGDM resnet20 with Adam resnet20 Orth Adam
0.8
0.7
Validatioon Accuracy (9)
0.3
0.2
20
60
80
100
Epach
(a) Validation Accuracy
3.0
resnet20 with SGDM
resnet20 with Orth SGDM
resnet20
resnet20 with Orth Adam
Validation
10-
20
60
100
Epoch
(b) Validation Loss
Figure 13: A compassion of Adam and SGDM, learning rate 1 x 10-3, 3, = 0.9, 32 0.99.
14

---

Under review as a conference paper at ICLR 2022
0.9
resnet20 with SGDM resnet20 Orth SGDM resnet20 with Adam resnet20 with Orth Adam
Validation Accuracy (8g
0.6
0.3
20
60
100
(a) Validation Accuracy
resnet20 with Orth SGDM
resnet20 with Adam
2.5
Validaation
0.5
0.0
20
100
Epoch
(b) Validation Loss
Figure 14: A compassion of Adam and SGDM, learning rate 1 x 10-?, 2, 3, 0.8, 82 0.99.
15


Reviews:
[{"review_id": "j5UE1qP2_LV", "reviewer": "Reviewer_S4zJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Summary*\n\nI find the research presented in this work interesting and I appreciate the simplicity in the method provided. Idea of orthogonalizing parameters (a) at initialization [1] (b) during training [2] and (c) after training [3] is widely studied, however I am not aware of any work proposing what is being proposed in this paper. Even though the idea is possibly novel, the assumptions, related work and experiments lack the depth needed for such experimental work to be useful for future research. \n\n*Weaknesses*\n- (1) ResNet34 baseline should get something around 73% on ImageNet. Similarly I found CIFAR accuracies to be very low. ResNet-20 with batch normalization should get 92-93% on Cifar10. I recommend authors to use strong baselines, otherwise it is very difficult to convince the reader whether this method is useful in practice. \n- (2) Authors should share training curves and discuss whether why orthogonalization helps training dynamics or generalization (or both). \n- (3) Would using ortogonal regularization and orthogonal initilization fix the problem? As explained briefly in the summary, orthogonality is widely studied in neural networks. Authors need to discuss and compare their methods against previous methods.\n- (4) Cost of motivated method needs to be discussed. Do we need to calculate SVD at every iteration. Can we start from previous bases? How does the cost and the gains compare with orthogonal initialization and regularization? \n- (5) Orthogonal basis is normalized, which affects the magnitude of the gradients; therefore learning rate should be searched for SGD and proposed method separately using a validation set. If computational resources are scarce I recommend authors to focus on one of the resnets in cifar (instead of trying 5 different depths). \n- (6) \"...for Orthogonal-SGDM we see that the components’ parameters are more similar than they were with SGDM and even include the initial spike. The reason behind this counter-intuitive result will be studied in the future\" This is the key motivation for provided method, which seems to be not-true. I recommend authors to revisit/update their assumptions and hypothesis in the next iteration.\n\n\n*Minor*\n- can be sped up\n- We obtain similar accuracy to SGD without fine-tuning (not sure what this refers to).\n- component->neuron.\n- The notation in 2.2 can be simplifed drastically. I recommend authors to assume fully connected layer and discuss convolutional layers in appendix. \n- dearth\n- table 2 -> Table 2\n- Figure 7 and 8 needs to be summarized like it is done in figure-6. For example seaborn.lineplot would be a good fit.\n\n\n\n1) https://arxiv.org/pdf/2001.05992.pdf\n2) https://arxiv.org/pdf/1905.05929.pdf\n3) https://arxiv.org/abs/1404.0736\n\n## After Rebuttal\nI read authors response and most of the limitations (1,3,5) are still a concern, thus I keep my score and encourage authors to address them in the future revisions. ", "summary_of_the_paper": "This paper proposes to orthogonalize the gradients of each neuron (within a layer) in order to improve training dynamics. They build on the assumption that neurons in a layer tend to learn similar features early in the training, and thus pushing them in orthogonal directions would yield to better results. The method requires calculating SVD on gradients at every iteration, thus unlikely to be practical (no FLOPs/run-time discussion). Training curves on Cifar10 and Imagenet shows faster convergence and better generalization. ", "summary_of_the_review": "Even though the idea of orthogonalization of the gradients is interesting, this work lacks the comparison to prior art and proper baselines. I recommend authors to work with proper baselines and compare their method against other orthogonalization techniques like orthogonal initialization and regularization. Furthermore the cost of doing SVD at every training iteration needs to be justified and discussed in detail. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N9hj6GI6zVl", "reviewer": "Reviewer_snPa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper touches upon orthogonalization of gradients, which is actually something that has already been explored in the literature - see for example https://arxiv.org/abs/2001.06782. Orthogonalization of gradients in general is an important technique and I am glad to see authors exploring down this direction.\n\nHowever, the authors' claims do not hold up to serious vetting.\n\n(1) The CIFAR baselines are incredibly weak. Could authors explain why?\n\n(2) When SOTA CIFAR numbers are used, the proposed method falls short of SOTA numbers. Authors state \"Unfortunately, the authors do not have the compute-power to extensively hyper-parameter tune a residual network\" but then proceed to claim that it is \"exceedingly likely\" that such experiments would work out. Unfortunately, as reviewers we cannot just accept the authors' optimism that their method would hold up in those experiments without actual hard evidence. \n\n(3) The general strategy of separating out the last dimension of gradients seems arbitrary. Why are those components important? Why not slice along another dimension? In other orthogonalizing gradient approaches, the full gradient tensor is considered, which assumes much less. Authors will need to more carefully justify this crucial design decision for their approach to pass muster. \n\n(4) No related work section and a very sparse reference list. As mentioned, gradient orthogonalization is not necessarily novel, and should authors have done a more thorough literature search they may have become aware of this and been able to refine their method. ", "summary_of_the_paper": "Authors propose orthogonalizing gradient components across the last dimension of the gradient tensor. Authors show experiments for CIFAR10 and ImageNet and claim training speedup along with performance boosts when gradients are orthogonalized. ", "summary_of_the_review": "Major issues with clarity, algorithm design, and general structure of the writeup make it difficult to endorse this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JTqTTPSU3Z6", "reviewer": "Reviewer_Pgpr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The conducted experiments do not support the author's main claim of improved training speed. This should be measured on the training loss. Therefore all presented Figures can only support claims dealing with generalization error - is there a reason for not showing the training loss curves? I find this irritating. \n\nFurthermore, the motivation of the paper that the layers representations/layers weights start aligning at the beginning of training seems not true and leads to the authors questioning their own hypthoses - see Discussion 3. I find this irritating as well.\n\nI am recommending a reject.\n\nNevertheless, the proposed idea seems interesting and I strongly urge the authors to study neural network optimization papers that look into the convergence speed of deep networks. Here, usually, experiments are not (mainly) done on very large networks / datasets and so the authors might have an easier time investigating the proposed method under limited compute (since they mention this limitation). \nSee for example the experiments conducted in https://arxiv.org/pdf/1503.05671.pdf.\n\nThe authors should tune the method they compare against as well as their own method since hyperparameters are crucial for training deep nets - since this is not done thoroughly in the paper, interpretations of the conducted experiments are difficult. \n", "summary_of_the_paper": "The paper claims that neural network training leads especially at the beginning of training to similar representations across the network.  To tackle this, they propose to orthogonalize gradients across the layer-wise feature dimensions of deep neural networks. ", "summary_of_the_review": "Experiments seem not to align with questions asked by the authors. Technical quality is not sufficient. Reject. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HO2zfEeLBBh", "reviewer": "Reviewer_CSRa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n=========\n* Orthogonalising gradients, rather than weights, seems to be a novel idea.\n\nWeaknesses\n==========\n* There is no explanation for how the authors are computing the SVD of convolutional layers. Is the doubly block circulant matrix that corresponds to the gradient of the convolutional layer explicitly constructed and then decomposed? Or are the gradients heuristically \"reshaped\" into a matrix and a quasi-SVD performed? If the former, how can this be done efficiently? If the latter, is orthogonality still actually enforced?\n* The performance comparisons are lacking. Picking one set of hyperparameters is not a sufficient procedure for comparing two algorithms. It could be the case that the particular set of hyperparameters chosen are well-suited to Orthogonal-SGDM, but maybe there exists another set of hyperparameters that allows SGDM to achieve superior performance. Given the unusually low performance of the different ResNet models considered in the experiments, this seems like a very real possibility.\n* The motivation for introducing this method is contradicted by the experimental results, so it is unclear why/when one might see benefits from using this approach. In particular, the authors state that each randomly initialised unit in a layer will converge towards the same weights near the beginning of training. I do not really follow why this would be the case, and the experiments in Section 3 seem to directly contradict this.\n* Discussion and empirical comparison to show how this is different to other orthogonalisation and projection-based optimisation/regularisation approaches is needed.\n\nPost-rebuttal\n==========\nThank you for clarifying the point about how this is implemented for convolutional layers. Thinking about this a bit more, I suspect both options I proposed are actually equivalent, which is an interesting property of your method you may wish to prove/highlight!\n\nHowever, many of my concerns remain:\n* The performance of the baselines is still very low compared to number typically seen in the literature for similar experimental setups, which indicates that something is either missing from the presentation (to explain why they are so different), or that there is a technical problem with the experiments.\n* It's good that the authors have removed the faulty motivation, but that only partially addresses my criticism: it's not obvious why orthogonalising the gradients is something we should do.\n* Perhaps I should have been more clear: I would like to see discussion about *why* orthogonalising gradients will have a different effect to orthogonalising weights.", "summary_of_the_paper": "The paper proposes inserting a gradient orthogonalisation step before each update for first-order optimisation methods. The orthogonalisation is accomplished via SVD, and the reported results show that in some settings this can lead to significant differences in performance and convergence rate compared to first-order optimisers that do not have the orthogonalisation.", "summary_of_the_review": "The work proposes a novel idea, but the presentation, evaluation, and analysis is not yet sufficient to warrant publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes orthogonalising loss gradients with respect to neural network parameters to speed up optimization and improve performance.

The reviewers are unanimous in recommending rejection of the paper. They highlight the following issues:
* weak baselines, which make it difficult to judge the contribution of this paper empirically
* lack of discussion of relevant literature and existing techniques
* arbitrary choices in the design of the algorithm, not backed up by theory or convincing arguments

The reviewers acknowledge the author response, but remain largely unconvinced of the merit of the proposed approach. I see no special reasons to disregard the reviewer assessments, and I therefore recommend not accepting this paper.
Decision: Reject