Dynamical Isometry for Residual Networks
Authors: Advait Harshal Gadhikar
Rebekka Burkholz
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
parameter initialization
dynamical isometry
ResNets

Abstract: The training success, training speed and generalization ability of neural networks rely crucially on the choice of random parameter initialization. It has been shown for multiple architectures that initial dynamical isometry is particularly advantageous. Known initialization schemes for residual blocks, however, miss this property and suffer from degrading separability of different inputs for increasing depth and instability without Batch Normalization or lack feature diversity. We propose a random initialization scheme, Risotto, that achieves perfect dynamical isometry for residual networks with ReLU activation functions even for finite depth and width. It balances the contributions of the residual and skip branches unlike other schemes, which initially bias towards the skip connections. In experiments, we demonstrate that in most cases our approach outperforms initialization schemes proposed to make Batch Normalization obsolete, including Fixup and SkipInit, and facilitates stable training. Also in combination with Batch Normalization, we find that Risotto often achieves the overall best result. 

Under review as a conference paper at ICLR 2023
DYNAMICAL ISOMETRY FOR RESIDUAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
The training success, training speed and generalization ability of neural networks rely crucially on the choice of random parameter initialization. It has been shown for multiple architectures that initial dynamical isometry is particularly advanta- geous. Known initialization schemes for residual blocks, however, miss this prop- erty and suffer from degrading separabilityy of different inputs for increasing depth and instability without Batch Normalization or lack feature diversity. We propose a random initialization scheme, RISOTTO, that achieves perfect dynamical isome- try for residual networks with ReLU activation functions even for finite depth and width. It balances the contributions of the residual and skip branches unlike other schemes, which imitially bias towards the skip connections. In experiments, we demonstrate that in most cases our approach outperforms initialization schemes proposed to make Batch Normalization obsolete, including Fixup and Skiplnit, and facilitates stable training. Also in combination with Batch Normalization, we find that RISOTTO often achieves the overall best result.
INTRODUCTION
Random initialization of weights in a neural network play a crucial role in determining the final performance of the network. This effect becomes even more pronounced for very deep models that seem to be able to solve many complex tasks more effectively. An important building block of many models are residual blocks He et al. (2016), in which skip connections between non-consecutive layers are added to ease signal propagation (Balduzzi et al.) (017) and allow for faster training. ResNets, which consist of multiple residual blocks, have since become a popular center piece of many deep learning applications (Bello et al. 2021).
Batch Normalization (BN) (Ioffe & Szegedy 2015) is a key ingredient train ResNets on large datasets. It allows training with larger learning rates, often improves generalization, and makes the training success robust to different choices of parameter initializations. It has furthermore been shown to smoothen the loss landscape (Santurkar et al. 2018) and to improve signal propagation (De & Smith 2020). However, BN has also several drawbacks: breaks the independence of samples in minibatch and adds considerable computational costs. Sufficiently large batch sizes to compute robust statistics can be infcasible the input data requires a lot of memory. Moreover, BN also prevents adversarial training (Wang al. 2022). For that reason, it is still an active area of research to find alternatives to BN Zhang et al. 2018): Brock et al.) (202T5). Scaled Weight Standardization and gradient clipping has recently outperformed BN (Brock et al.) 2021b). However, a random parameter initialization scheme that can achieve all the benefits of BN is still an open problem. An initialization scheme allows deep learning systems the flexibility drop in to existing setups without modifying pipelines. For that reason, it is still necessary to develop initialization schemes that enable learning very deep neural network models without normalization or standardization methods.
combinations of
direction of research pioneered by [axe al. (2013): Pennington et al. (001)) has analyzed the signal propagation through randomly parameterized neural networks in the infinite width limit using random matrix theory. They have argued that parameter initialization approaches that have the dynamical isometry (DI) property avoid exploding or vanishing gradients, as the singular values of the input-outpus Jacobian are close to unity. DI is key to stable and fast training (Du et al. 2019 Hu et al.] (2020). While [Pennington al. (2017) showed that it is not possible to achieve DI in networks with ReLU activations with independent weights or orthogonal weight matrices, Burkholz & Dubatovka (2019) Balduzzi al. (017)) derived way attain perfect DI even in finite ReL.U

---

Under review as a conference paper at ICLR 2023
networks by parameter sharing. This approach also be combined (B (Blumenfeld al. 2020
Balduzzi et al.] 2017)) with orthogonal initialization schemes for convolutional layers (Xiao et al. 2018). The main idea is to design random initial network that represents a linear isometric map.
We transfer a similar idea to ResNets but have to overcome the additional challenge of integrating residual connections and, in particular, potentially non-trainable identity mappings. In contrast to other ResNet initialization schemes that achieve (approxi- mate) dynamical isometry by initially scaling down the residual connections, we balance the weight of skip and residual connections and thus promote higher initial feature diversity, as highlighted by Fig. [ We thus propose RISOTTO (Residual dynam- ical isometry by initial orthogonality),, an initializa- tion scheme that induces exact dynamical isometry (DI) for ResNets (He et al., 2016) with convolutional or fully-connected layers and ReLU activation func- tions. RISOTTO achieves this for networks of finite width and finite depth and not only in expectation but exactly. We provide theoretical and empir- ical evidence that highlight the advantages of our approach. Remarkably and in contrast other initialization schemes that aim to improve signal propagation in ResNets, RISOTTO can achieve per- formance gains even in combination with BN. We hypothesize that this improvement is due the balanced skip and residual connections. Furthermore, we demonstrate that RISOTTO can success- fully train ResNets without BN and achieve the same or better performance than Zhang et (2018); Brock et al. (2)2Tb).
Risotto
Dynamical Isometry
Balanced Skip and Residual Connections
Feature Diversity
Figure 1: Features of our initialization scheme Risotto. This figure has been de- signed using images from Flaticon.com.
1.1
CONTRIBUTIONS
To explain the drawbacks of most initialization schemes for residual blocks, we derive sig- nal propagation results for finite networks without requiring mean field approximations and highlight input separability issues for large depths. Accordingly, activations corresponding to different inputs become more similar for increasing depth which makes it harder for the neural network to distinguish different classes.
We propose solution, RISOTTO, which is an initialization scheme for residual blocks that provably achieves dynamical isometry (exactly for finite networks and not only approxi- mately). A residual block is initialized that it acts orthogonal, norm and distance preserving transform.
In experiments on multiple standard benchmark datasets, we demonstrate that our approach achieves competitive results in comparison with alternatives:
We show that RISOTTO facilitates training ResNets without BN or any other nor- malization method and often outperforms existing BN free methods including Fixup, Skiplnit and NF ResNets.
It outperforms standard initialization schemes for ResNets with BN on Tiny Imagenet, CIFAR 100, and ImageNet.
1.2 RELATED WORK
Preserving Signal Propagation Random initialization schemes have been designed for multitude of neural network architectures and activation functions. Early work has focused on the layerwise preservation of average squared signal norms (Glorot Bengio, 2010 He et al. 2015 Hanin 2018) and their variance (Hanin & Rolnick] 2018). The mean field theory of infinitely wide networks has also integrated signal covariances into the analysis and further generated practical insights into good choices that avoid exploding or vanishing gradients and enable feature learning (Yang & Hul 2021 if the parameters are drawn independently (Poole et al. 2016 Raghu et al. Schoenhll et al, 2017 Yang & Schoenholz, 2017; Xiao et al.) 2018). Indirectly, these works demand that the average eigenvalue of the signal input-output Jacobian is steered towards 1. Yet. in this set-up, ReLU activation functions fail support parameter choices that lead to good trainability of very
2

---

Under review as a conference paper at ICLR 2023
deep networks, as outputs corresponding to different inputs become more similar for increasing depth (Poole et al.) 2016 Burkholz & Dubatovka] (011)). Yang & Schoenholz (2017) could show that ResNets mitigate this effect and enable training deeper networks, but also cannot distinguish different inputs eventually.
However, there are exceptions. Balanced networks can improve (Li et al. 2021 interlayer correla- tions and reduce the variance of the output. A more effective option is to remove the contribution of the residual part entirely as proposed in successful ResNet initialization schemes like Fixup (Zhang et al. 2018), SkipInit (De & Smith 2020). or ISONet (Qi et al., 2020). This, however, limits signif- icantly the iiiiill feature diversity that is usually crucial for the training success (Blumenfeld al.) 2020) and scales down the residual connections. A way to address this issue for other architectures with ReL.Us like fully-connected (Burkholz Dubatovka] 2019) and convolutional (Balduzzi al.) 2017) layers is looks-lineal weight matrix structure (Shang et al.) 2016). This idea has not been transferred to residual blocks yet but has the advantage that it can be combined with orthogonal sub- matrices. These matrices induce perfect dynamical isometry (Saxe et al. 2013 Mishkin & Matas) 2013 Poole et al. 2016 Pennington et al. 2017), meaning that the eigenvalues of the initial input- output Jacobian are identical to 1 -1 1 and not just close to unity average. This property has been shown to enable the training of very deep neural networks (Xiao et al., 2018)) and can improve their generalization ability (Hayase & Karakida 2021] and training speed Pennington et al. (2017) 2018). ResNets equipped with ReL.Us can currently only achieve this property approximately and without a practical initialization scheme (Tarnowski et al., 2019) or with reduced feature diversity Blumenfeld et al. 2020) and potential training instabilities (Zhang et al. 2018 De & Smith 2020).
ResNet Initialization Approaches Fixup (Zhang et al.] ISONet (Qi et al. (020). and ReZero (Bachlechner et al. 2021
2018).
SkipInit (De & Smith 2020
have been designed to enable training without requiring BN. yet, can usually not achieve equal performance. Training data in- formed approaches have also been successful (Zuu et al. 2021 Dauphin & Schoenholz 2019) but they require computing the gradient of the input minibatches. Yet, most methods only work well in combination with BN (loffe& & Szegedy 2015)) as it seems to improve ill conditioned initializations Gloort & Bengio 2010 He et al.] 2016) according to [Bjrrck et al. (2018), allows training with larger learning rates (Santurkar et al. 2018), and might initially bias the residual block towards the identity enabling signal to flow through De & Smith 2020). The additional computational and mem- ory costs of BN. however, have motivated research alternatives including different normalization methods (Wu & He| 2018 Salimans & Kingmal 2016 Ulyanov et al.] 2016). Only recently has it been possible to outperform BN in generalization performance using scaled weight standardization and gradient clipping (Brock et al. 2021b|a). a). but this requires careful hyperparameter tuning. In experiments, we compare our initialization proposal RISOTTO with all three approaches: normal- ization free methods, BN, and normalization alternatives (e.g NF ResNet).
2 RESNET INITIALIZATION
2.1 BACKGROUND AND NOTATION
The object of our study is a general residual network that is defined by
z" : W x, x' =0(2-)), z':= =af) Bitd -0 Wou
(I)
for 1 I T.. P(.) denotes an optional pooling operation like maxpool or average pool, J(.) residual connections, and ht.)) the skip connections, which usually represent identity mapping or a projection. For simplicity, we assume in our derivations and arguments that these functions are parameterized as (=)) W/+ (W{ +x/++// +b1)+bb and h (x*) W! skip *x' - (* denotes con- volution), but our arguments also transfer to residual blocks in which more than one layer is skipped. Optionally, batch normalization (BN) layers are placed before or after the nonlinear activation func- tion x(-). We focus on ReLUs dbxx) max(0, (Krizhevsky et al., 2012), which are among the most commonly used activation functions in practice. All biases b5 E M++, bj R.N.., and skip E Mi are assumed to be trainable and set initially to zero We ignore them in the following, since we are primarily interested in the neuron states and signal propagation at initialization. The parameters and 3 balance the contribution of the skip and the residual branch, respectively. Note that a is a trainable parameter, while 3 is just mentioned for convenience to simplify the compar- ison with standard He initialization approaches (He et al. 2015). Both parameters could also be
3

---

Under review as a conference paper at ICLR 2023
integrated into the weight parameters W/ R 1 1xNN xRI,1 x11,, W Nm
and W skip E X but they make the discussion of different mitiaaiizaionn schemes more convenient and simplify the comparison with standard He initialization approaches (He et al. 2015).
Residual Blocks Following the definition by He al. (2013), distinguish two types of residual blocks, Type B and Type C (see Figure Zah, which differ in the choice of W residual block is defined as z' a
'kipp' The Type C
h h'(x') so that shortcuts h(.) are projections with 1 x kernel with trainable parameters. The type B residual block has identity skip connections 2¹ a (x') + x'. Thus, W skip represents the identity and is not trainable.
(x f
2.2 SIGNAL PROPAGATION FOR NORMAL RESNET INITIALIZATION
Most initialization methods for ResNets draw weight entries independently at random, including Fixup and SkipInit. To simplify the theoretical analysis of the induced random networks and to highlight the shortcomings of the independence assumption, we assume:
Definition 2.1 (Normally Distributed ResNet Parameters). All biases are initialized as zero and all weight matrix entries are independently normally distributed with
w/y,2 ~ N (0. (()) w'y ~ ~N(N.. (.). and wkyskip N (0. o.
Most studies further focus on special cases of the following set of parameter choices.
Definition 2.2 (Normal ResNet Initialization). The choice 01,1
Nug Ry Ri.2 01,2
Ni+1 as used in Definition 2.1 and on, Br 2 0 that fulfill of + 37 = 1.
N1 82,2"
Siskkip
sometimes also oy # is still common if it accounts for the depth L. of the network. In case or and B1 are the same for each layer we drop the subscript 1. For instance, Fixup (Zhang et al. 2018) and SkipInit (De & Smith, 2020) satisfy the above condition with suppresses the residual branch effectively. However, in combination with He initialization (He et al.) 2015) it becomes more similar to a = 3 = v0.5. at 2021 study the case of free oq but focus their analysis on identity mappings W = and W
Another common choice is Wskip I instead of random entries. If B
0 and 3 1. De & Smith (2020) argue that BN also
skip
As other theoretical work, we focus our following investigations on fully-connected layers to sim- plify the exposition. Similar insights would transfer to convolutional layers but would require extra effort (Yang & Schoenholz] 2017). The motivation for the general choice in Definition 2.2 is that it ensures that the average squared 12-norm of the neuron states is identical every layer. This has been shown by Li et al. (2021) for the special choice W = I and W = I, 3 and by (Yang & Schoenholz 2017) the mean field limit with missing ReLU so that x' 21-1 (Hanin & Rolnick 2018) has also observed for W 'kip = and 8 = 1 that the squared signal norm increases present the most general case next and prove it in the appendix.
E1 ag For completeness,
Theorem 2.3 (Norm preservation). Let a neural network consist of fully- connected residual blocks as defined by Equ. (1) that start with a fully-connected layer at the beginning W which contains N₁ output channels. Assume that all biases are initialized as and that all weight matrix entries are independently normally distributed with w'y.2 - (0, (Z.). w/y, N o.1). and why,
N (0, oi..)))) Then the expected squared norm of the output after one fully-connected layer and L residual blocks applied to input x is given by
N₂
N4+1 -
L-1
Nm, af
E
x =
+ of ||0||2
2
2
2
1=1
Note that this result does not rely on any (mean field) approximations and applies also to other parameter distributions that have mean and are symmetric around zero. Inserting the parameters of Definition[2.1 for fully-connected networks with k = 1 leads to the following insight that explains why this is the preferred initialization choice.
Insight 2.4 (Norm preserving initialization). Acccording to Theorem 2.3 the normal ResNet initial- ization (Defiittonn 2.2) preserves the average squared signal norm for arbitrary depth L..
4

---

Under review as a conference paper at ICLR 2023
Separability of inputs
Varying the scaling
1.0
96
Ualaoon
Correlation
(lomm/DDees)
Wadyy
MALL
0.5
Frer
95
Test
W!
W!
0.0
impo
block1
block2.
block3
block4
block5
lincar
94
160
180
200
Type
Type
Epochs
(b)
(c)
Figure 2: (a) The two types of considered residual blocks. In Type C the skip connection is pro- jection with 1 kernel while in Type B the input is directly added the residual block via the skip connection. Both these blocks have been described by He et al. (2016). (b) The correlation between two inputs for different initializations as they pass through a residual network consisting of convolution filter followed by 5 residual blocks (Type C), an average pool, and linear layer on CI- FARTO. Only RISOTTO maintains constant correlations after each residual block while it increases for the other initializations with depth. (c) Balancing skip and residual connections: Performance of RISOTTO for different values alpha (a)) for ResNet 18 (C) on CIFARIO. a = is equivalent to SkipInit and achieves the lowest accuracy. The better performing 1 is implemented in Risotto.
Even though this initialization setting is able to avoid exploding vanishing signals, it still induces considerable issues, as the analysis of the joint signal corresponding to different inputs reveals. According to the next theorem, the signal covariance fulfills layerwise recurrence relationship that leads to the observation that signals become more similar with increasing depth.
Theorem 2.5 (Layerwise signal covariance). Let fully-coneetedd residual block be given as de- fined by Eq. ( with random parameters according to Definition 2.2 Let x!+1 denote the neuron states of Layer I 1 for input x and a the same neurons but for input i. Then their covariance given all parameters of the previous layers is given Er ((x ad' _d++1)
1 N
Nms +,8 2 a2
2
(x x²)) + a² 2f Nm ||>"|| |+|||
(2)
4 2
+ Ew!
(aPott
(1 Poo |a>11))) (a2o7z (lo + Poo 11=113))
where the expectation E, is taken with respect to the initial parameters W! Wi, and w! ship and the constant e fulfills 0.24 < e S 0.25.
Note that this statement holds for finite networks. To clarify what that means for the separability of inputs, we have to compute the expectation with respect to the parameters of W,- To gain an intuition, we employ approximation that holds for a wide intermediary network.
Insight 2.6 (Covariance of signal for different inputs increases with depth). Let fully-connected given. It follows from Theorem (2.3 that the outputs corresponding to different impuls become more difficult to distinguish for increasing depth L. For simplicity, let us assume that ||0||| = | Then, in the mean field limit Ne,, - oc, the covariance of the signals is lower bounded by
ResNet with random parameters as in Definition
1--1
E ((x", ,,=)) 2 7 (x, x) + +2
Σ
r = 11 (x, x) + 1- 7-71
72
(1 (1-n8) (1-
(3)
0=
for 1+ and cla² + 2)
using E ||| ~ 1.
Since 71 < 1, the contribution of the original input correlations (a, ã) vanishes for increasing depth L. Mcanwhile, by adding constant contribution in every layer, irrespective of the input correlations, ((w ( increases with and converges to the maximum value 1 (or a slightly smaller value in case of smaller width N,,)). Thus, deep models essentially map every input to almost the same
5

---

Under review as a conference paper at ICLR 2023
output vector, which makes it impossible for the initial network to distinguish different inputs and provide information for meaningful gradients. Fig. 2bdemssstsss this trend and compares it with our initialization proposal RISOTTO, which does not suffer from this problem.
While the general trend holds for residual as well as standard fully-connected feed forward networks still note mitigation for strong residual branch (3 1). The contri- bution by the input correlations decreases more slowly and the constant contribution is reduced for larger B. Thus, residual networks make the training of deeper models feasible, as they were de- signed to do (He et al. 2016). This observation is in line with the findings of Yang & Schoenholz (2117), which were obtained by mean field approximations for a different case without ReLU af- ter the residual block (so that x' = 1) It also explains how ResNet initialization approaches like Fixup (Zhang et al. 2018) and Skiplnit (De & Smith 2020) can be successful in training deep ResNets. They set a ( and B 1. If W skip = II, this approach even leads to dynamical isometry but trades it for very limited feature diversity (Blumenfeld et al., 2020) and initially broken residual branch. Figure highhliss potential ad advantages that can be achieved by A 0 if the initialization can still maintain dynamical isometry as our proposal RISOTTO.
(3 = 0), interestingly,
2.3 RISOTTO: ORTHOGONAL INITIALIZATION OF RESNETS FOR DYNAMICAL ISOMETRY
Our main objective is avoid the highlighted drawbacks of the ResNet initialization schemes that we have discussed in the last section. We aim to not only maintain input correlations on average but exactly and ensure that the input-output Jacobian of our randomly initialized ResNet is an isometry. All its eigenvalues equal thus or 1. In comparison with Fixup and Skiplnit, we also seek to increase the feature diversity and allow for arbitrary scaling of the residual versus the skip branch.
Looks-linear matrix structure The first step in designing an orthogonal initialization for residual block to allow signal to propagate through a ReLU activation without loosing half of the informa- tion. This can be achieved with the help of a looks-linear initialization (Shang et al. 2016 Burkholz & Dubatovka| 2019 Balduzzi et al. 2017), which leverages the identity æ = x(x) Φ( (x)). Ac- cordingly, the first layer maps the transformed input to a positive and a negative part. A fully- connected layer is defined by x = = = & ([U0: with respect to a submatrix UO. Note that the difference of both components defines a linear transformation of the input a1 __ U Thus, all information about U is contained in x²" The next layers con- tinue to separate the positive and negative part of a signal. Assuming this structure as input, the next layers w/+1 of o'''z') proceed with the block structure W = [u' U':U' -U". As a consequence, the activations of every layer can be separated into a positive and negative part as x² = = x so that ao || The submatrices U' can be specified as in case of linear neural network. Thus, if they are orthogonal, they induce a neural network with the dynamical isom- etry property (Burkholz & Dubatovkal (219)). With the help of the Delta Orthogonal initialization Xiao et al., 2018), the same idea can also be transferred to convolutional layers. Given a matrix H -- X. convolutional tensor is defined as W R.N+1 X.xk x as Wiyk; hag if k = [1-1/// and K/- = k2 22] and WisK( 0 otherwise. We make frequent use of the combina- tion of the idea behind the Delta Orthogonal initialization and the looks- linear structure.
Definition 2.7 (Looks- -ineer structure). A tensor W E N:+1 x XN2 X x is said to have looks-linear
structure with respect submatrix U E R [2] [M] if
=
hij ik' = | and K/2 = /2/ H=
U
-U
WISKANG
=
otherwise,
-
(4)
It has first layer looks-linear structure if [U: UU.
We impose this structure separately on the residual and skip branch but choose the corresponding submatrices wisely. To introduce RISOTTO, we only have to specify the corresponding submatrices for Wi. W! and W acts linear orthogonal map.
skip- The main idea of Risotto to choose them so that the initial residual block
The Type C residual block assumes that the skip connection is a projection such that 1((x) = EJEN, W Gjskip * a/, where W E X is a trainable convolutional tensor with kernel size 1 x 1. Thus, we can adapt the skip connections to compensate for the added activations of the residual branch visualized in Fig.

---

Under review as a conference paper at ICLR 2023
otjor a07oo
15' atjer atroo
ar
ReLU
oj 0
0 ot
ReLi
Figure 3: A Type C residual block is initialized with Risotto as visually explained.
Definition 2.8 (RISOTTO for Type C residual blocks), For residual block of the form ar++1 ++ and W
+h' where f'(a") = W (Wo (xx)) W!
ship *x', the weights W W
skip are initialized with looks-linear structure according to Def. 2.7 with the submatrices U
U2 and U'aep respectively, The matrices U1, U2 and M' be drawn independently and uniformly from all mairices with orthogonal rows or columns (depending on their dimension), while the skip submatrix is Ukiip M' aU;
The Type residual block poses the additional challenge that we cannot adjust the skip connections initially because they are defined by the identity and not trainable. Thus, we have to adapt the residual connections instead to compensate for the added input signal. To be able to distinguish the positive and the negative part of the input signal after the two convolutional layers, we have to pass it through the first ReLU without transformation and thus define W as identity mapping.
Definition 2.9 (RISOTTO for Type B residual blocks). For a residual block of the form xl41 éla f' ) + x²) where I (x') ) W dWW, x²), RISOTTO initializes the weight W| as W has looks- linear structure (according to Def. 2.7 with respect to a submatrix U!} = M' - (1/a)ll, where M' 1RNN:-12xx1//2 a random matrix with orthogonal columns or rows, respectively.
w 11ig519 = if = j. j. = k2 // kg = |k//2] and w Wiighhg = O otherwise.
Note that particularly the type B initialization consists of considcrable amount of zero entries. To induce higher symmetry breaking and promote feature diversity Blumenfeld et al. (2020), we also study a noisy variant, N-RISOTTO, in which we add a small amount of noise eΣ our RISOTTO parameters, where 10-4 and Σ follows the He Normal distribution (He et al. 2015).
As we prove in the appendix, residual blocks initialized with RISOTTO preserve the norm of the input and cosine similarity of signals corresponding to different inputs not only on average but exactly, as exemplified by Fig. (b). This addresses the drawbacks of initialization schemes that are based on independent weight entries, as discussed in the last section.
Theorem 2.10 (RISOTTO preserves signal norm and similarity). A residual block that is ini- tialized with RISOTTO maps input activations x' to output activations a!+11 so that the norm ||x = I02 stays equal. The scalar product between activations corresponding to two inputs and ă are preserved in the sense that (0 - i+] 11+1
34 _1+1) (0+
The full proof is presented Appendix A.3 It is straight forward, as the residual block is defined as orthogonal linear transform, which maintains distances of the sum of the separated positive and neg- ative part of signal. Like the residual block, the ouput- Jacobian also is formed of orthogonal submatrices. It follows that RISOTTO induces perfect dynamical isometry for finite width and depth.
Theorem 2.11 (RISOTTO achieves exact dynamical isometry for residual blocks). A residual block whose weights are initialized with RISOTTO achieves exact dynamical isometry so that the singular values X E aJ) of the input-oupput Jacobian E R NNN411 xa1xxk11xxx0 X fulfill E (-1,
The detailed proof is given in Appendix A.2 Since the weights are initialized so that the residual block acts the required spectral properties. Drawing on the well established theory of dynamical isometry (Chen et al.) 2018)) Saxe et al. 22033; Mishkin & Matas 2015 Poole et al. 2016 Pennington et al.) 2017). we therefore expect RISOTTO to enable fast and stable training of very deep ResNets, as we demonstrate next in experiments.
orthogonal transform. also the input-output Jacobian is an isometry, which has
7

---

Under review as a conference paper at ICLR 2023
3 EXPERIMENTS
In all our experiments, we use two kinds of ResNets consisting of residual blocks of Type B or Type C, as defined in Section 2.. and visualized in Fig. Za ResNet (B) contain Type B residual blocks the input and output dimension of the block is equal and Type B block otherwise, but ResNet (C) has Type C residual blocks throughout. All implementation details and our hyperparameter tuning approach are described in Appendix [A.4] We use a learnable scalar a that is initialized as to balance skip and residual connections (see Fig. 2c (c)). Results for all cases are reported on the benchmark datasets CIFARIO, CIFAR 100 (Krizhevsky al. 2014), and Tiny ImageNet (Le & Yang, (015). Additional key experiments on ImageNet are presented in Appendix A.9 that are in line with our findings on Tiny ImageNet.
Our main objective is to highlight three advantageous properties of our proposed initialization RISOTTO: (a) It enables stable and fast training of deep ResNets without any normalization methods and outperforms state-of-the
schemes designed for this purpose (see Table Table aand Fig. (center). (b) Unnormalized can compete with NF ResNets, an alternative normalization approach to BN, (see Fig. 4 (right) and Fig. 9.. (c) It can outperform alternative initialization methods in combination with Batch Normalization (BN) (see Table 2 and Fig. (left)) We hypothesize that this is enabled by the balance between skip and residual connections that is unique RISOTTO.
RISOTTO without BN We start our empirical investigation by evaluating the performance of ResNets without any normalization layers. We compare our initialization scheme RISOTTO the state-of-the-art baselines Fixup (Zhang et al., 2018) and SkipInit (De & Smith (020). Both these methods have been proposed as substitutes for BN and are designed achieve the benefits of BN by scaling down weights with depth (Fixup) and biasing signal flow towards the skip connection (Fixup and SkipInit). Fixup has so far achieved the best performance for training ResNets without any form of normalization. We observe that as shown in Table | RISOTTO is able to outperform both Fixup and Skiplnit. Moreover, we also observed our experiments that Fixup and SkipInit are both susceptible to bad random seeds and can lead to many failed training runs. The unstable gradients at the beginning due to zero initialization the last residual layer might be responsible for this phenomenon. RISOTTO produces stable results. With ResNet (C), it also achieves the overall highest accuracy for all three datasets. We have also evaluated N- N-Rissoto for ResNet50 (B) on Im- ageNet comparing with Fixup shown in Fig. 10 in Appendix A.9 These results verify that well balanced orthogonal initialization for residual blocks enables better training for different datasets and models of varying sizes.
Dataset
ResNet
RISOTTO (ours)
Fixup
SkipInit
18 (C)
93.71 + 0.11
92.05 + 0.11
10.0 + 0.03 92.57 = 0.21
CIFARIO
18 (B)
92.1 0.52
93.36 0.15
50 (C)
60.63 + 0.28
58.25 1.64
+ 0 42.40 + 0.67
CIFARI00
50 (B)
56.17 + 0.39
59.26 0.69
50 (C)
49.51 + 0.06
48.07 0.46
26.42 9.7
Tiny ImageNet
50 (B)
47.02 + 0.37
47.89 + 0.45
31.57 + 8.6
Table RISOTTO as a substitute for Batch Normalization The mean test accuracy over 3 runs and 0.95 standard confidence intervals are reported to compare RISOTTO, Fixup, and Skiplnit without using BN. RISOTTO able to achieve the overall best results for each of the benchmark datasets.
RISOTTO learns faster for deep ResNets. Fig. (center) for ResNet B consisting of 300 layers and Fig. 8b for 10 on CIFAR 100 demonstrate RISOTTO's 's ability to train deep networks without BN. Wh While RISOTTO achieves higher final performance for 300 layers and similar one Fixup for 101 layers, RISOTTO trains faster in comparison. Also in combination with BN, RISOTTO can train faster, Fig. 10 shows for ResNet50 (B) on ImageNet.
Comparison with Normalization Free ResNets NF ResNets (Brock et al. 202 that use weight standardization have been shown to outperform BN when used in combination with adaptive gradi- ent clipping. We find that RISOTTO is able to outperform NF ResNets on Tiny ImageNet with He initialization He et al. (016) as shown in Fig. 4 (right). Further experiments reported in Fig.
8

---

Under review as a conference paper at ICLR 2023
RISOTTO in combination with BN Despite its drawbacks, BN remains popular method and is implemented per default, as it often leads to the best overall generalization performance for ResNets. Normalization free initialization schemes have been unable compete with BN, even though Fixup has come close. Whether the performance of batch normalized networks can still be improved is therefore still relevant question. Table [2] compares RISOTTO with the two variants of He ini- tialization for normally distributed and uniformly distributed weights (He et al., 2016). We find that RISOTTO arrives marginally lower performance on CIFARIO but outperforms the standard methods on both CIFARIOO and Tiny ImageNet. On ImageNet, a single N-Risotto training run for ResNet50 (B) with BN obtains 65.67 test accuracy and He Uniform initialization 58.41 (see Figure
10 in Appendix A.9. These results and Table [ showcase the versatility of RISOTTO, as it enables training without BN and can even improve training with BN (see also Fig. (left)).
Dataset
ResNet
RISOTTO (ours)
He Normal
He Uniform
18 (C)
95.29 = 0.14
95.38 + 0.15
95.32 + 0.07 94.82 + 0.03
CIFARIO
18 (B)
94.93 - 0.07
94.99 + 0.12
50 (C)
73.11 0.70
76.17 + 0.25
76.21 + 0.28
CIFARI00
50 (B)
78.45 + 0.08
77.40 0.25
76.7 0.8
50 (C)
59.47 + 0.02
50.12 0.98
53.91 0.59
Tiny ImageNet
50 (B)
58.73 + 0.29
52.21 +
55.05 + 0.6
Table 2: RISOTTO with Batch Normalization The mean test accuracy over 3 runs and 0.95 stan- dard confidence intervals are reported to compare different initialization in combination with BN. RISOTTO outperforms the baseline methods with BN on CIFAR100 and Tiny ImageNet.
Can we reduce the number of BN layers? Considering the importance of BN on performance and how other methods struggle to compete with it, we note that reducing the number of BN layers in a network can still render similar benefits at reduced computational and memory costs. Often a single BN layer is sufficient and with Risotto, we are flexible in choosing its position. Fig. 4 (right) reports results for BN after the last layer, which usually achieves the best performance. Additional figures in the appendix (Fig. explore optional BN layer placements in more detail.
Tiny Img: BN after last layer
Tiny Imngenes
Test Acc
-40
40
20
Rismto (ours)
He Unifoons
Tox
20
Riadto Flg
Test
20
NF Resbir (ile Nemoo)
(ous)
Normal
NF ReaNen Eila Cardfon))
100
0
30
Upucca
100
150
0
100
Epochs
Epochs
Figure 4: (left) A single BN layer is placed after the last residual block of a ResNet50 (C) on Tiny Imagenet. (center) A 300 layer ResNet (B) with RISOTTO versus Fixup. (right) RISOTTO versus NF ResNets (C) on Tiny Imagenet.
4 CONCLUSIONS
We have introduced a new initialization method for residual networks with ReLU activations, RisOTTO, It enables residual networks of any depth and width to achieve exact dynamical isome- try at initialization. Furthermore, it can balance the signal contribution from the residual and skip branches instead of suppressing the residual branch initially. This does not only lead to higher fea- ture diversity but also promotes stable and faster training. In practice that is highly effective, as we demonstrate for multiple standard benchmark datasets. We show that RISOTTO competes with and often outperforms Batch Normalization free methods and even improves the performance of ResNets that are trained with Batch Normalization. While we have focused our exposition on ResNets with ReLU activation functions, in future, RISOTTO could also be transferred to transformer architectures similarly to the approach by Bacchechher et al. (202T). In addition, the integration into the ISONet (Qi et al. 2020) framework could potentially lead to further improvements.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Thomas Bachlechner. Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence large depth. In Uncertainty Artificial Intelligence, pp. 1361. PMLR, 2021.
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma. and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Interna- tional Conference on Machine Learning, pp. 342-350. PMLR, 2017.
Irwan Bello, Liam B Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Y Yi Lin, Jon Shlens, and Barret Richard Zoph. Revisiting resnets: Improved training methodologies and scal- ing principles. 2021.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. Advances in neural information processing systems, 31, 2018.
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. Beyond signal propagation: is feature diver- sity necessary in deep neural network initialization? International Conference on Machine Learning, pp. 960-969, PMLR, 2020.
Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation close the performance gap in unnormalized resnets. arXiv preprint arXiv:2 101.0869,, 2021a.
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pp- 1059-1071. PMLR, 2021b.
Rebekka Burkholz and Alina Dubatovka. Initialization of relus for dynamical isometry. Advances in Neural Information Processing Systems, 32, 2019.
Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks. In International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research.
PMLR, 2018.
Yann N Dauphin and Samuel Schoenholz. Metainit: Initializing learning by learning to initialize. Advances in Neural Information Processing Systems, 32, 2019.
Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 111119964-19777 2020.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterizee neural networks. In International Conference on Learning Representations, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Tch and Mike Titerington (eds.), International Conference Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research. PMLR, 2010.
Boris Hanin. Which neural net architectures give rise to exploding and vanishing gradients? In Advances in Neural Information Processing Systems, volume 31, pp. 582-591. 2018
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. In Advances Neural Information Processing Systems, 2018.
Tomohiro Hayase and Ryo Karakida. The spectrum of fisher information of deep networks achieving dynamical isometry. In International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research. PMLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1034-1034, 2015.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in opti- mizing deep linear networks. In International Conference on Learning Representations, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 148-456. PMLR, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems, volume 25, 2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. toronto. edu/kriz/cifar hmml, 55(5), 2014.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Mufan Li, Mihai Nica, and Dan Roy. The future is log-gaussiam: Resnets and their infinite-depth- and-width limit at initialization. Advances in Neural Information Processing Systems, 34:7852- 7864, 2021.
Dmytro Mishkin and Jiri Matas. All you need is good init. arXiv preprint urXiv:1511. 164422 2015.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learn- ing through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. The emergence of spectral univer- sality in deep networks. In International Conference on Artificial Intelligence and Statisties, pp- 1932, 2018.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma, and Jitendra Malik. Deep isometric learning for visual recognition. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 7824-7835. PMLR, 13-18 Jul 2020. URL https: httt::::/proceedingg..o htt.s//sss.......ss/1111....
qi20a. html
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. the expressive power of deep neural networks. In International Conference on Machine Learning, pp. 2847-2854, 2017.
Tim Salimans and Durk P Kingma. Weight normalization: simple reparameterizzation to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in neural information processing systems, 31, 2018.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint 1326v:1312.6120, 2013.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein, Deep information propagation. In International Conference Learning Representaatinss, 2017. 2017.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In International Conference Machine Learning, pp. 2217-2225, 2016.

---

Under review as a conference paper at ICLR 2023
Wojeiech Tarnowski, Piotr Warchot, Stanistaw Jastrzębski, Jacek Tabor, and Maciej Nowak. Dy- namical isometry is achieved in residual networks in a universal way for any activation function. In International Conference Artificial Intelligence and Statisticcs, pp. 2221-2223 PMLR, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprini 160,:06607.0002.. 2016.
Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, and Zhangyang Wang. Removing batch normalization boosts adversarial training. In International Conference on Machine Learn- ing, pp. 23433-23445. PMLR, 2022.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp- 3-19, 2018
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cnns: How train 10,000-layer vanilla con- volutional neural networks. In International Conference on Machine Learning, pp. 5393-5402. PMLR, 2018.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. Advances in neural information processing systems, 30, 2017.
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in fefhntt-wwithh neural net- works. In International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research. PMLR, 2021.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2018.
Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, and Tom Goldstein. Gradinit: Learning to initialize neural networks for stable and efficient training. Advances in Neural Infor- mation Processing Systems, 34:16410-16422. 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A. SIGNAL PROPAGATION
Recall our definition of a residual block in Equation T
x = WO, z'= o'= x' == auf + (x'),
x out = Wout P(x") (5)
for 1 S L with (a'' W3d( w' n²(x') = W skip where the biases have been set to zero. In the following theoretical derivations we focus on fully-connected networks for simplicity so that W E RR. Ne-1+ XMz, W IK TRNN,,x and W skip
TR. N The general principle could also be transferred to convolutional layers similarly to the mean field analysis by Xiao et al. (0018)
common choice for the initialization of the parameters defined as follows.
Definition A.1 (Normal ResNet Initialization for fonnee-co residual blocks). Let a neural network consist of fully-connected residual blocks as defined by Equ. (5) All biases are initialized as O and all weight matrix entries are independently normally distributed with w/y,2 ~ N (0.00000) (0.0(2). wh 1/y ~N (0.)).). and was -N (0, (.ui))))):: Then the Normal ResNet Initialization is defined
by the choice 01,1
Ney 01,2 =
2 Ni-1 Oroiip = v
2, and on, B 2 that fulfil Ni
af + 17 1.
Our objective is to analyze the distribution of the signals x++1 and T++1, which correspond to the random neuron state of initial neural network that is evaluated in input x T0, respectively. More precisely, we derive the average squared signal norm and the covariance of two signals that are evaluated in different inputs.
We start with the squared signal norm and, for convenience, restate Theorem 2.10 before the proof.
Theorem A.2 (Theorem 2.10 in main manuscript). Let a neural network consist of residual blocks defined by Equ. ① Equ. ( that start with fully-connected layer at the beginning W which contains N1 output channels. Assume that all biases are initialized as 0 and that all weight matrix entries are independently normally distributed with w/y.2 -N (0, o222), w/ya N (0, of. of,)
) and Wijssip ~N N (f Then the expected squared norm of the output after one cully-connected layer and L residual blocks applied to input is given by
10=))²)
|
=
Ni
00
L-1
Nm: N++1 (f
2 + B8 of ||0||
2
2
1=1
Proof. First, we study how the signal is transformed by single layer before we deduce successively the final output signal. To do so, assume that the signal of the previous layer is given. This means we condition the expectation on the parameters of the previous layers and thus x' and ø'.
For notational convenience, we define Ex(z) E (= xc a²) and skip the index in the following
derivations. We write: a x 1+| E x². z z'. f(x f'(x) w (2((W)))) h(z)
(x) Wi sk on, B = Br. Given all parameters from the previous layers, we deduce
N
E
||0]||
x = Σ (x:)²) = Ni+1E (zz)²) = N E, (21)²
(6)
2
The first equality follows from the fact that all random parameters independent and the signal components are identically distributed. The third equality holds because the distribution of each signal component is symmetric around zero that the ReI.U projects half of the signal away but
13

---

Under review as a conference paper at ICLR 2023
the contribution to the average of the squared signal is just cut in half. We continue with
Nan
N
E, (21)²
E,
12
+
B
Wikkip 1 &
i=
Kamn
Nm
N2
N1
a
Er
(w2,is) Er
+ 1
Bi
(skip, 1k)
(8)
i=1
k=
Ni
a a'o7 No, Ed
Σ
Wi, w وزا
+ B of ||2:11
(9)
2
Ni
1
a 20i.22mm
-Be
Bi
11, 11,15
8 o a |
(10)
B8 B No41 N
|
(11)
as all weight entries are independent and the expectation linear operation. To obtain Equa- tion (TO), we just repeated the same argument as for Equation take care of the ReLU. Afterwards, we used again the independence of the weights Wi.1j.
From repeated evaluation of Equations
(6) and 11
we obtain
E (1a0_11²
N1
L-1 1
N1+1
(2o + 18 of ||::||
(12)
2
2
1=1
for x = x.
To make that the signal norm neither explodes vanishes for very deep networks, would need to choose the weight variances so that 2 II Nu11 2 (a²oi Nm + /² = 1.
In the common normal ResNet initialization, this actually achieved, since 0t, 22/Nmi 01,2 si,skip = 12/Ni=1. ou = 22/N1, and a² + g2 even preserve the average norm in every layer.
How does this choice affect whether signals for different inputs are distinguishable? To answer this question, we analyze the covariance of the neuron state for two different inputs. We begin again with analyzing the transformation of single layer and condition all parameters of the previous layers. To obtain lower bound on the covariance, the following Lemma will be helpful. has been derived by Burkholz & Dubatovkal (2019) for Theorem 5.
Lemma A.3. Assume that two random variables 21 and are jointly normally distributed as N (0, V) with covariance matrix V. Then, the covariance of the ReLU transformed variables i ((21) and x2 = o(22) is
VI-P²
E (x102) = 2011222
g(p)p +
(13)
2T
1 (1) - -o) 012 + C//11122
2 021022
(14)
where p U111///U12 and g(p) is defined as g(p)
= Jo ob
exp (- du for VI-Ø""
"
72x
|e| # 1 and (11 0, g 1)= = 0.5. The constant fulfills 0.24 < c 0.25.
In the following. we assume that all weight parameters are normally distributed so that we can the above lemma. However, other parameter distributions in large networks would also lead to similar results, as the central limit theorem implies that the relevant quantities are approximately normally distributed.
14

---

Under review as a conference paper at ICLR 2023
We study the covariance of the signals x++1 and Ef+1 = z. which correspond the random neuron state of an initial neural network that evaluated input x or æ", respectively.
Theorem A.4 (Theorem 2.5 in main manuscript). Let colley-onnected residual block be given as defined by Equ. or Equ. (5). Assume that all biases are initialized as and that all weight matrix entries are independently normally distributed with w/y - (0, R.2). waya N (0, of,) ). and aay N (0, ). Let x denote the neuron states of Layer + 1 for input a and a the same neurons but for input œ. Then their covariance given all parameters of the previous layers is given as Er ((x't x(+1))
1 N
Nm +2800l (at ,0) + N4 Nm
asoo
||~|||
|
(15)
4 2
(V (10oft21000
(o + Brot [1ad1()) (afotz 0 W (W + Sf 1
Ewi
where the expectation E, is laken with respect the initial parameters W W and W!
Wass dap
Proof. Let us assume again that all parameters of the previous layers are given in addition to the parameters of the first residual layer W, and use our notation from the proof of Theorem 2.10
Based on similar arguments that we used for the derivation of average squared signal norm, we observe that z = 2" and = z are jointly normally distributed. In particular, the components 2, are identically distributed for the same input. The same component 2; and ž for different in- puts has covariance matrix V with entries U11 a²oî,2 [[)) + -B2 || a a²oî,2 [[]]] + p ||" and V12 a² a2ot (W,,)) + 8 of T.akpp (x, .).
U22
No+1
E, ((x, = Ew, Ey (12,, x)) Wi)
Σ 1=1
Ew, Eg (#(2))((2)) W1)
(16)
2 N14 (Ju12+ + eve11v22 = N441EE, (to1e + c//11
(17)
where we applied Lemma A.3 to obtain the inequality and used the fact that the entries of the variance matrix identically distributed for different (i) with respect to W1.
We can compute the first term Ew, U12 by using Lemma A.3 again, as Wi and Wiā are jointly normally distributed given and The associated covariance matrix S for one component corre- sponding different inputs has entries $11 oř,1 ||]| $22 of - | and $12 of (x. Lemma A.3 gives us therefore
Ew, (V12) = Ew, (d
+ 3 (x, (x.
(18)
2 220022N Nm,
(z, (207,
[
cof. ||||| ||ق||| + +.3 +.800 (z.
(19)
15

---

Under review as a conference paper at ICLR 2023
Determining the second part of Equation 17 is more involved:
Ew, (VU11U22)
(V
= Ew.
(1ooor2llo(W ([(((/)))))) + Bf |<E)(²)
(aPoř, Il [W(()))) + o |1
(20)
= 3 o || ||:|| Bw,
1b
W
E
+1
? 10
W
Tt
+
+1
(21)
||
2
(
(7100)
2 8 fZ issiip
skip
|al|| | Ew,y
7
1b
W
a
+1
(Wi,
Wi
+
(22)
x
M ?
Nm
2 8² f30
S.kii
skip
||-||| | Ew,
i1,1 +1
Σ
w
+1
(23)
j=1
j=M+1
2 A
Nm
= - 0f. iskii
||-||| |
EM By
20(, 2
+1
2
Nm j=1
Non
Nma 2
x
+ 1
Nmi
j=M+1
Non
Nm | |له|| (n²o)
Rd 88 8800 sk ||
|لت||
1 2
+
+-8ooo Pojssip
(24)
2
2
In Equation 22 we have used the fact that Eww (VU11U22) is monotonically increasing in the covariance 812- Thus, the minimum is attained for perfectly negative associations between and E and thus ã = x To simplify the derivation, we further study the case æ (1, 0, 0, O....... follows that either ((w1.f1) or ((-W1,)) is positive while the other one is zero. To ease the notation, by reindexing. we can assume that the first M components fulfill @(w1,11) > O, while the remaining Nm, .M components fulfill the opposite. Note that because w1.j1 is distributed symmet- rically around zero, M ~ Bin(N., 0.5) is binomially distributed random variable with success probability 0.5. Thus, EM = Nm,/2. To make the dependence on No, of the different variables more obvious, we have replaced the random variables W1,j1 that are normally distributed with stan- dard deviation 01 by standard normally distributed random variables y, with standard deviation 1. This makes the use of the law of large numbers in the last equation more apparent. Note that this approximation is only accurate for large Nm, >> 1, which is usually fulfilled in practice.
Finally, combining Equations
19 and 20). receive 17
1 a² N2 N (x, a² Ni Nm,
E, ((x, x\) 2 16"
+
B
(2.
4
+ Ew, (v
(120i22 1
W 11 ( + Fo
lee) 1l + 800 skip
(25)
N141
a² 0 Nm
82
0 +
(x, i
2
2
+ c
No+1
o-2 f11 Nm:
0 +
22020]200,1
Nm
+ 2/3 2,3207. 0?.stn) ))
(26)
2
2
2
16

---

Under review as a conference paper at ICLR 2023
To understand the problem that these derivations imply, next choose the weight parameters so that the squared norm signal is preserved from one layer to the next and, for simplicity, study the case in which | |||| 1. Then we have
1 + 82 (x, a) c(a² +2 2) = 1+ 8 (z.
1 2
Es ()x, 2
(27)
Thus, the similarity of signals corresponding to different inputs increases always by at least constant amount on average. Repeating the above bound layerwise, at Layer I we receive for 1+
71
1 S 1 and 7/2 = c(a² + 2):
L-1
E B(xt, 2 7 (x. x)
Σ k=
72 1 (1 -4-
(28)
According to bound, output signals of very deep networks become more similar with increasing depth until they are almost indistinguishabl.. This phenomenon poses great challenge for the trainability of deep resdidual neural networks with standard initialization schemes. Note that the case without skip-connections is also covered by the choice a = 1 and 3 = O. Interestingly, nonzero skip-connections (B 0) fight the increasing signal similarity by giving more weight to the original signal similarity (increased 71) while decreasing the constant contribution of 72- This enables training of deeper models but cannot solve the general problem that increasingly deep models become worse in distinguishing different inputs initially. Even the best case scenario of leads eventually to forgetting of the original input association, since " 0.5 < 1. With 72 & 0.5, the overall signal similarity E ((zt, x))) converges to I for L - irrespective of the input similarity. Thus, every input signal is essentially mapped to the same vector for very deep networks, which explains the following insight.
===0 0 and
Insight A.5 (Insight 2.6 in main paper). Let a fully- ResNet be given whose parameters are drawn according Definition A.7 It follows from Theorem 2.5 that the outputs corresponding to different inputs become more difficult to distinguish for increasing depth L. In the mean field limit N Nm, + co, the covariance of the signals is lower bounded by
E (( ab1)==
,) 2 (2, x6 + 72
Σ
(20,
23
72 (1 (1-
(29)
71
k=
for 71 and 72 cla²² + 2) and E
1+0²
|||| 1.
However, our orthogonal initialization scheme Risotto does not suffer from increasing similarity of outputs corresponding to different inputs.
A.2 DYNAMICAL ISOMETRY INDUCED BY RISOTTO
Theorem A.6 (Theorem 2.11 in the main paper). A residual block (af type or type C) whose weights are initialized with RISOTTO achieves exact dynamical isometry so that the singular values (J) of the input-output Jacobian J E R M441x144 4h441x Xixk; fulfill X E 1-..1
Proof. Consider single element of the output activation of type C residual block at layer 1. At initialization, an output activation component at layer is
++1 a Σ
Wiiker /2kk//2k2/22 /2,2
Σ mEN:
((w ( /ank/,/2k222111 mmm) +
Σ nENi
w/ ski
(30)
NEN.,
since RISOTTO initializes only the central elements of every 2-D filter to nonzero values and all the other values are zero which reduces the convolution to a simple summation. For the subsequent calculations ignore the filter dimension indices in the weights. Now the positive part of the output
17

---

Under review as a conference paper at ICLR 2023
1+1 #
where i N2, is given by
1+1 ik
a Σ Σ
w
W
p(w
+
Σ
w Sinskii nk
(31)
neNm:
JEN,
REN,
Σ
Σ
Φ(- u
Σ
w/ skip@
(32)
nENma /2
JEN:/2
nEN
Σ
Σ
2 1
m/n
Σ
u u
(33)
neNm; NENm,/2JENN /2/EN//2
nENi/2
n/N
E
(34)
1/N
Taking the derivative of the output ii
input element a nkk
0x I+1
m
(35)
dr nk
We can now obtain the input output Jacobian for [o, and j [0, NN/2 as
=
mly - otherwise
=
(36)
And since M' is orthogonal, the singular values of are one across the dimensions i.j. Due to the looks linear form of the input the weights the complete Jacobian also takes the looks linear form as
m4
if 1/41 M E N4.2/./ [0, Nr/2]
m
if h k 2 E [0, Nr/2]
mi
if k/411 kG,iE [0, [Ni/2,
(37)
m
K/41 = kG (2 e [Ni/2,
otherwise
The same argument follows for type B residual blocks. Hence, since M is orthogonal matrix, the singular values of the input output Jacobian of the residual blocks initialized with RISOTTO are exactly {1, 1 for all depths and width and not just in the infinite width limit.
A.3 SIGNAL PROPAGATION WITH RISOTTO
Now we closely analyze how RISOTTO transforms the input signal for residual blocks when initial- ized using Definitions 2.9 and 2.8 RISOTTO creates effectively an orthogonal mapping that induces DI. Note that our analysis of convolutional tensors is simplified to evaluating matrix operations since the Delta Orthogonal initialization simplifies convolution to an effective matrix multiplication in the channel dimension of the input. Specifically, we can track the changes in the submatrices used in the initialization of residual block with RISOTTO and observe the output activation. We start with a Type B residual block evaluated at input x² = [ se_ at Layer I and set = 1. The residual and skip branches at Layer are then
M' - (1/a)I -M MM'
a a!
]
f(x))- W/ 6(W) x0) = = W/ *
M' (1/a)T
M' M' (1/a)a4
- + +Ma! (1/a)a!
-M
18

---

Under review as a conference paper at ICLR 2023
Adding the skip branch to the residual branch gives
M' - M''' (1/a)
M a! -M'a'
(x') + +x =a
-M + M - (1/ox)/
+
= a
x + M
olaf(x))) + = aΦl
([-M + M'á!
MM'a' M'á!
1)=a
2.
1+1
1+1
The submatrices of the output of the residual block U/44, and UgaL_ preserve the norm of the input
signal as long as we set a 1, since U32 is an orthogonal matrix fulfilling ||U]] | 1. If a takes
a value other than or -1, the orthogonal matrix V has to be scaled such that ||u|]| 2 = 1/a. For all are experiments with type B residual blocks, however, we observe best results with a 1. Next we repeat the same computation for type C residual blocks. Note that the type C residual block can achieve an exactly orthogonal transform of the input with RISOTTO for any value a. The signal propagates through a residual block of type C initialized with RISOTTO and input xf [+1_; at_ and weights
U U
U
Ug
-U2
M' - -UUj
-M' aUg M' -'3
w =
:w
W skip
U1
- U2
U/
M² oUgU,
Layer as
af + h n'(x")
a
+ w
skip
U2o (U a U - U/ (-U Uf+4)
U2 (U) #4 - U/ U20 (-U +
④ ((M' - aUS )ay + +-MM'
e ((([MM' aU) - (M' -M' aU
M' -M'a'
M 24 + +M
x 1+ ((af(((a)) n'(x²))
We conclude that for type residual blocks the output is of linesr form and has the same norm as the input because of the orthogonal submatrices and the
structure.
We now use the above formulations to prove that RISOTTO preserves the squared signal norm and similarities between inputs for residual blocks, the key property that allows stable training.
Theorem A.7 (RISOTTO preserves signal norm and similarity). A residual block that ini- tialized with RISOTTO maps input activations x² to oulput activations a! so that the norm ||2 = |02 stays equal. The scalar product between activations correspording to two inputs x and â are preserved in the sense that (a 111 111 31+1 -/111 (#. x." x x!
Proof. We first prove that the squared signal norms are preserved for both types of residual blocks followed by the same for similarity between inputs. Consider a type C residual block. The preac- tivation of the previous layer of looks linear form z' = -1, The preactivation of the current layer is then given as the signal passes through the residual block as
z' ( + n (0(x'-1)
(38)
U (Ufat - U/ - U (-u - UgS +
(39)
④ ((M' a + (-U + U Uf
② (((-M' aU, (
(M' (M'
M'al - Mla'
(40)
-M'a'. + M'
M' MM M' MM + MM (6(-2211)
(41)
19

---

Under review as a conference paper at ICLR 2023
Then the squared norm is given by
|| || [ M ( 2/1 + 1-11-MMorz--)) +M'
M' 1/
(42)
2 [|M'o(E--)]1 1|] 1 ===11=111)1
(43)
(110(E=3)12 + |0{ 11)
(44)
-| |
(45)
||
(46)
We have used the fact the M' an orthogonal matrix such that ||M'sl| = ||>||| and the identity for ReLU activations by which x = x(2) x(-2). Since x1+1 = o(2"), we have
(( x(-21)
x
(47)
Taking the squared norm of the output then gives
| 1| = || + ||0(-=2))) 21)1 = ||
(48)
The second equality results from squaring the ReLU identity as the term
= 0.
Combining Eqns. and 48
we obtain
112 = |='1|2
(49)
In fact, the norm preservation is a special case of the preservation of the scalar product, which we prove next.
Consider two independent inputs æ and ã, their corresponding input activations at Layer I are x' and ,². We show that as result of RISOTTO the correlation between the preactivations is preserved which in turn means that the similarity between activations of the looks linear form is preserved as 11
(#4+1 - 441 34+1
[o(E')- ol o(=21)) (
- 0(-22)
(50)
[M'' -Ml0))-MM(6- M'0(-2' 0 ( Mo((=z'-1)
T (M' - M M' -'0(- 0 (-M' +
(51)
[( 6(2 (2 1)) 6 6
(6 6(-M's (M M'o(-2(-1)))
(O(M' (( 11 (1)) 6MM'
(0(M'o(-3) -2111)) - (6 -M MM
(52)
66 [0(31-1) - o1 2-) (0(3-1)) Ø( ((-))-1)
|M' ( -0(-2111))] " iM' (0(31-1) -(--1-11)
(53)
(54)
= (04 x1_ 51_ K
(55)
The above derivations follow from the looks -linear structure of the weights and the input as well as the orthogonality of matrix M The same proof strategy for both norm preservation and similarity can be followed for type B residual blocks using the signal propagation A.3] This concludes the proof.
A.4 EXPERIMENTAL SETUP AND DETAILS
In all our experiments we use Stochastic Gradient Descent (SGD) with momentum 0.9 and weight 0.0005. We use 4 NVIDIA A100 GPUs train all our models. All experiments repeated for
20

---

Under review as a conference paper at ICLR 2023
3 rums and we report the mean and 0.95 confidence intervals. In experiments with ResNet101, used a learning rate of 0.005 for all initialization schemes including ours.
We performed the following hyperparameter tuning for each initialization scheme model dataset that we compare and report. For every experiment we use the best performing learning rate between 0.1, 0.01, 0.05, 0.001 with either cosine annealing while keeping the batch size fixed to 256. To do so, we randomly split the training set into a training (90%) and validation (10%) and used the LR that performed best on the validation set to evaluate the trained model on the test set. We report the accuracies obtained on the test set.
step scheduling (whichever performs better),
Placing single BN layer In order to identify the best position to place single BN layer in ResNet, we experiments with different positions. (+) First layer: The BN was placed right after the first convolution layer before the residual blocks. (ii) BN in the middle: The BN layer was placed after half of the residual blocks in the network. (iit) BN after last res block: In this case BN was placed before the pooling operation right after the last residual block.
Correlation comparison in Figure 2b In order to compare the correlation between inputs for different initialization schemes consisting of the same number of channels (32) and kernel size of (3, 3) followed by average pooling and linear layer. The figure shows the correlation between random samples of CI- FAR10 averaged over 50 runs.
vanilla Residual Network with five residual blocks, each
Tiny Imagenet Note that the validation set provided by the creators of Tiny Imagenet (Le & Yang 2015) as test set to measure the generalization performance of our trained models.
without BN
with BN
Type
Param
RISOTTO (ours)
Fixup
SkipInit
RISOTTO (ours)
He Uniform
He Normal
LR
0.01
0.01
0.1
0.1
BS
256
256
256
256
256
256
C
Schedule
cosine
cosine
cosine
cosine
cosine
cosine
Epochs
150
150
150
150
150
150
LR
0.05
0.05
0.1
0.1
0.1
BS
256
256
256
256
256
256
B
Schedule
cosine
cosine
cosine
cosine
cosine
cosine
Epochs
150
150
150
150
150
150
Table Implementation details for ResNet18 on CIFARIO
without BN
with BN
Type
Param
RISOTTO (ours)
Fixup
SkipInit
RISOTTO (ours)
He Uniform
He Normal
LR
0.01
0.01
0.001*
0.1
0.1
0.1
BS
256
256
256
256
256
256
Schedule
cosine
cosine
cosine
cosine
cosine
cosine
Epochs
150
150
150
150
150
150
LR
0.01
0.01
0.01
0.1
0.1
0.1
BS
256
256
256
256
256
256
Schedule
cosine
cosine
cosine
cosine
cosine
cosine
Epochs
150
150
150
150
150
150
Table 4: Implementation details for ResNet50 CIFAR100. denotes that SkipInit failed train even at a very low learning rate for multiple runs as reported in Table in the main paper.
21

---

Under review as a conference paper at ICLR 2023
without BN
with BN
Type
Param
RISOTTO (ours)
Fixup
SkipInit
RISOTTO (ours)
He Uniform
He Normal
LR
0.01
0.01
0.001
0.01
0.01
0.01
BS
256
256
256
256
256
256
C
Schedule
cosine
cosine
cosine
step (30, 0.1)
step (30, 0.1)
step (30, 0.1)
Epochs
150
150
150
150
150
150
LR
0.01
0.01
0.01
0.01
0.01
0.01
BS
256
256
256
256
256
256
Schedule
cosine
cosine
cosine
cosine
cosine
cosine
Epochs
150
150
150
150
150
150
Table 5: Implementation details for ResNet50 on Tiny Imagenet. Arguments for step denote that learning rate was reduced by a factor of 0.1 every 30 epochs.
A.5 ADDITIONAL EXPERIMENTS FOR BN LAYER PLACEMENT ON CIFAR 100
Considering the importance of on performance and how other methods struggle to compete with it, we empirically explore if reducing the number of BN layers in network would still render similar benefits at reduced computational and memory costs. We observe that in such a case, the position of the single BN layer in the network plays crucial role. Figure 5 shows that RISOTTO enables training in all cases while other initializations fail if the single BN layer is not placed after the last convolutional layer normalize all features. First need to determine the best position place the BN layer the network. We report experiments in Figure (5) that placing the BN after the last convolutional layer, before the last linear layer achieves best results. In Figure we show results for Tiny Imagenet with single BN after the last residual layer where too RISOTTO outperforms the standard. BN after the last residual layer controls the norms the logits and potentially stablizes the gradients leading to better performance. Conversely, BN right after the first layer does not enable larger learning rates or better generalization. Figure 7 we show that even after optimal placement of the single BN layer, RISOTTO leads to the overall best results on Tiny ImageNet at reduced computational costs.
BN after first layer
BN in the middle
BN after last res block
100
90
Text Acc.
Text Acc.
Test
80
80
Rivctto (ours)
Uniiform
HIe Nommal
70
70
0
50
100
150
50
100
150
0
50
100
150
Epochs
Epochs
Epochs
Figure 5: Comparing different positions placing a single BN layer on a ResNet 18 (C) for CI- FARIO. In each of the cases, RiSOTTO allows stable training and converges competetive accura- cies, while standard methods fail in some cases.
A.6 TRAINING VERY DEEP NETWORKS WITH RISOTTO
We train Residual Network with 300 layers (and no BN) initialized with Risotto and show that with Risotto, such deep network is able train faster compared to Fixup (see Figure
A.7 COMPARISON WITH NORMALIZATION FREE RESNETS
Comparison with Normalization Free ResNets NF ResNets (Brock et al. 2021a) that scaled weight standardization and have been shown to compete with BN. We find that RisoTTo is able to match outperform NF ResNets with He initialization He et al. (2016) shown in Fig. While
22

---

Under review as a conference paper at ICLR 2023
BN after first layer
BN in the middlee
BN after last res block
40
Test Acc.
Risotto (ours)
40
He Lniifons Normal
20
20-
0-
100
50
100
150
50
100
150
Epochs
Epochs
Epochs
Figure 6: Comparing different positions of placing single BN layer on CIFARI00 for a ResNer50 (C), In each case, RISOTTO allows stable training and converges to competitive test accuracies, while standard methods fail in some cases. Standard methods are more unstable in this case compared to CIFAR1O 10 and only He Normal is competetive when BN is placed in the last layer.
Tiny Img: BN after last layer
RosNie (R) with layors CIFARIIO
Test Acc.
40-
40
Test Ace
Risecto (ours)
20
Uniform He Normal
20
Risorto
0
Fixus
100
50
100
150
Epochs
ppochs
(a)
Figure 7: (a) Using only one layer of BN after the last residual block in a ResNet 50 (C) on Tiny ImageNet. RISOTTO still achieves accuracy competitive to BN in every layer and performs better than Normal He initialization, while He Uniform initialization fails completely. (b) A Residual Network (type B) with 300 layers, which is a deeper version of ResNet101 with Bottleneck blocks arranged as [3 x 3, * 4, 3 23, 3 3]. is trained with Risotto and Fixup (both with LR = 0.01) and cosine annealing for 150 epochs on CIFAR 100. The rest of the hyperparameters are as Table We observe that Risotto enable faster training and better generalization.
ResNet101 (B) on CIFAR100
Risotto ResNet101
50-
Test Apc
40
Test
2.5
Risocto (ours)
20
Fixup
Rasorto (ouns)
Skiphnit
50
100
50
100
150
Epochs
Epochs
(a)
(b)
Figure 8: (a) Similar to Figure 8b compare Risotto without BN against Fixup ResNet101. Here we use an LR = 0.01 for both methods. (b) ResNet101 (C) on CIFARI00. Training with RISOTTO achieves faster good performance.
NF ResNets usually require careful hyperparameter tuning for gradient clipping, observe that they train well with vanilla SGD on smaller datasets.
A.8 RESULTS ON IMAGENET
We also report experiments on the large scale ImageNet dataset. Due to limited computational resources, we only able to report a single run ResNet50 (B), but results are in line with
23

---

Under review as a conference paper at ICLR 2023
CIFARIO
CIFARI00
Tiny Imagenet
100
40
Test Acc.
90
Tost Acc.
Test
Risoto
Riseto (ours)
20
Risctoo
NF ReaNia (He Nomaa)
25
NF ReaNss (Hec Nomad)
NF Roaet (he Nernoh)
NF (He Uniform)
dee Unittem)
ResNet Uniforno)
70
100
200
100
100
Epochs
Epochs
Epochs
Figure 9: Comparing NF ResNets to RISOTTO on all three datasets with a ResNet (C) (18 for CIFAR10 and 50 for the others). While RISOTTO performs similar to He Normal CIFAR100, it able to outperform both He Uniform and He Normal on Tiny ImageNet and CIFAR10.
findings on Tiny ImageNet. When we use our initialization for ImageNet in which we add small amount of noise eLi to our RISOTTO parameters, where 10 and > follows the He Normal distribution (He al. 2015) for the corresponding Kaiming Normal initialization. Adding this random noise enables symmetry breaking (Blumenfeld et al. (2200) and supports training according to our empirical observations. Results are reported Figure
ResNet50 (B) on ImageNet
ResNet50 (B) on ImageNet without BN
wwwmmmm
60
60
Test Ace
Test Acc
40
40
(ours)
20-
(ours)
Kaiming Uniform
Fixup
0
50
100
25
50
75
100
Epochs
Epochs
(a)
(b)
Figure 10: (a) We train a ResNet50 (B) with BN and compare Risotto (ours) with the standard Kaim- ing Normal initialization scheme. We train both the models with LR = 0.01 and cosine annealing. We observe that with our initialization scheme we are able to train faster on ImageNet and even achieve better accuracy than Kaiming Normal. (b) Similarly, we compare Risotto (ours) with Fixup on a ResNet50 (B) without BN. Both are trained with LR 0.01. Risotto tracks Fixup closely.
Remark: We would like to note that because results ImageNet from single run, they are inherently susceptible to variance between runs. Nonetheless, we have comprehensively shown the effectiveness of RISOTTO multiple datasets.
A.9 RESULTS ON PREACTIVATION RESNETS
In addition to the ResNet models we have considered so far in the paper, there is another set of mod- els in Residual Networks that may be used for certain problems. Preactivation ResNets (He al., 2015) apply the ReI.U nonlinearlity before the weight within a residual block. Our initialization scheme, Risotto, can easily transfered to Preactivation ResNets in order to achieve dynamical isometry. We report experiments on CIFAR10 for Preactivation ResNets in Figure II
A.10 ADDITIONAL EXPERIMENTS WITH RESNET (B) FOR RISOTTO
We provide additional experiments showing the effectiveness of Risotto for ResNet(B) architectures
in Figure 12
24

---

Under review as a conference paper at ICLR 2023
Preactivation Re (C) on CIFARIO
Preactivation ResNet18 (C) without BN on CIFARIO
75
75
Test Acc
Risotto (ours)
Text Acc
50
Kaiming Normal
50
Kaiming Uniform
25
25
(ours)
Fixup
50
100
150
50
100
150
Epochs
Epochs
Figure 11: We train Preactivation ResNet18 (C) with BN initialized with Risotto (ours) and com- pare with the standard Kaiming Normal and Kaiming Uniform initialization schemes. We train both the models with LR 0.1 and cosine annealing. Risotto is able enable dynamical isometry and achieve competetive generalization. Similarly we also report results withour BN comparing Risotto with Fixup, where Risotto is able outperform Fixup.
NF R (B) on CIFARIO
100 8esNet18 (B) with Last Layer BN on CIFARIO
75
Test Ace
Risotto (ours)
90
50
(Kaiming Normal)
Test
NF ResNet (Kaiming Uniform)
80
Risotto (ours)
Kaiming Normal
Kaiming Uniform
70
50
100
150
50
100
150
Epochs
Epochs
(a)
(b)
Figure 12: (a) Ne-ResNet18 (B) compared with ResNet 18 (B) initialized with Risotto (ours) and without any Batch Normalization. (b) Using single layer of BN after the last residual block in ResNet18 (B) on CIFAR10.
A.11 TRAINING 2000 LAYER RESIDUAL NETWORK
To show that Risotto enables training very deep networks due to initial dynamical isometry, we train a very deep residual network with Risotto. We use a Residual Network with Bottleneck residual blocks (Type C) arranged in the form [20 3, 20 4, 20 23, 20 with channels [64, 128, 256, 512] respectively. We use learning rate of 5e and add noise during initialization similar to We see that initializing with Risotto enable training even very deep network. While Risotto enables training such a deep network, we would like remark that training such deep network to con- vergence and competetive performance would require additional methods like gradient clipping to ensure gradients do not vanish or explode during training.
2000 layer Residual Network (C) on CIFAR100
Risotto (ours)
Test Ace
10
0
10
20
30
Epochs
Figure 13: Risotto enables training 2000 layer network on C1FAR100.
25

---

Under review as a conference paper at ICLR 2023
A.12 RISOTTO WITH NOISE: N-RISOTTO
We also report results of RISOTTO initialized with additional noise. The model is initialized with RISOTTO in which add small amount of noise eΣ our RISOTTO parameters, where 10-4 and follows the He Normal distribution (He et al. 2015) For the CIFARI0/100 experiments with N-RISOTTO we use learning rate of 0.05 with cosine annealing, except for CIFARIO0 with ResNet50 (B) where we are able to use higher learning rate of 0.1. For Tiny ImageNet we use learning rate of 0.01 with cosine annealing.
Dataset
ResNet
RISOTTO + noise (ours) with BN
RISOTTO noise (ours) without BN
18 (C)
94.88
91.88
CIFARIO
18 (B)
95.06
91.57
50 (C)
73.22
60.29
CIFARI00
50 (B)
77.74
64.46
50 (C)
58.82
49.66
Tiny ImageNet
50 (B)
59.26
47.09
Table 6: N-RISOTTO We initialize networks with RISOTTO and add small amount of noise to enable "breaking the symmetry' (Blumenfeld al. 2020
26


Reviews:
[{"review_id": "VCwBNho9yg", "reviewer": "Reviewer_GPrm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed RISOTTO (Residual dynamical isometry by initial orthogonality) to initialize residual networks. While RISOTTO achieves some success in several datasets. I feel that the experiments are inadequate, and the superiority of RISOTTO is not significant.", "summary_of_the_review": "The paper proposed an isometry dynamic based initialization. However, I still have some concerns as mentioned above. Therefore, I tend to give \"rejection\".", "strength_and_weaknesses": "Pros:\n\n- Writing is comparably easy to follow.\n\nCons:\n\n- The main difference between the proposed RISOTTO and other initialization (e.g., fixup) is that RISOTTO has analyzed the weight in residual connection (termed as C type in this paper). However, This C type is not the general case in residual nets. In a residual net, there can be a pooling layer for downsampling. And even in a case using C type, the weight is only used for downsampling, thereby not as general as no weight in residual connection. Moreover, Transformer is also a type of residual network, they do not have such a C type. Therefore, I feel that the motivation is not sufficient.\n\n- On experiments:\n  - The superiority of RISOTTO is not significant. For example, in Table 1 and Table 2, all the baselines (i.e., Fixup, SkipInit, He Normal, and He Uniform) can derive better results in some settings. Especially, Fixup achieves all better results on B type, the more general condition.\n  - The ResNets are trained for 150 epochs may not be enough. It is common to train 200 epochs at least.\n  - In Table 2, the results of two He initialization on Tiny ImageNet are incredible. Specifically, on B type, He Normal derives 52.21, and He Uniform derives 55.05. There is around a 3% distance. It is hard to believe the results as these two initialization are both in accordance with He initialization, and the only difference is the sampling strategy.\n  - A learning rate of 0.1 is a common setting. Why did this paper set 0.01 for training ResNet50 on Tiny Imagenet, while 0.1 for cifar10/100?\n  - Figure 2, Figure 3, and Figure 4 are all based on C-type nets. What are the results of B type as it is also an important architecture?\n  - In Figure 3, Fixup seems to derive better results.\n  - It is better to add Transformer as a backbone.\n  - It is better to add experiments on large-scale datasets, like ImageNet.\n\n\nOverall, regarding the above points, I vote for \"rejection\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RLsBJHumK5", "reviewer": "Reviewer_VqzH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new initialization method for ConvNets with residual blocks and ReLU activation, called RISOTTO. It enables training deep networks with residual structures without a normalization layer. This is achieved by maintaining a dynamical isometry at the initialization time, which is implemented as initializing the weight matrices as a block-wise looks-linear structure. The experiments on CIFAR and Tiny Imagenet can support the claim of this paper.", "summary_of_the_review": "In summary, this paper is interesting as it shows by designing a simple initialization method, deep networks with residual structures can be trained. On the other hand, the content of this paper can be improved a lot. I recommend acceptance for this paper but I expect to see an improved version of this paper in the discussion phase.", "strength_and_weaknesses": "Strength:\n1) The proposed method is simple and effective. It makes ResNet without normalization trainable just by changing the initialization of convolution. Compared to fixup, it does not initialize the output of the residual branch to be 0 (as in Fixup). This is surprising to me.\n2) It takes explicit consideration of ReLU activation and achieves exact dynamical isometry for residual blocks. This is different from several previous works and can be inspiring for future research.\n\nWeakness:\n1) The central theme of this paper is unclear to me. From the introduction, I think the authors want to describe the disadvantage of batch normalization and propose a way to remove batch normalization, which is shown in Table 1. However, the authors also use several paragraphs and Table 2 to show RISOTTO is compatible with batch normalization. It makes me confused because batch normalization can reduce the sensitivity of the initialization method. It is not clear why showing the compatibility between RISOTTO and BN can support the paper’s claim.\n2) In Table 1 and Table 2, it seems the proposed method is not better than the previous method (fixup) without batch normalization.\n3) This paper mentions “separability of inputs” several times but I didn’t understand what it means. The authors may want to consider adding the reference to that part. Is it equivalent to maintaining the inner product of input?\n4) Missing reference: This paper claims previous work only proposes dynamical isometry in a vanilla network (without residual structures). In [1], the paper proposes a way to achieve “approximate dynamic isometry” both at initialization and training time. There are differences between [1] and the proposed work such as [1] initializing the output of residual blocks to be 0 which is opposite to this work, but it’s worth comparing and discussing.\n5) The proposed method only shows the trainability of a network with 50 layers on CIFAR. In contrast, previous methods usually train 100 layers (fixup and [1]) on ImageNet or 10000 layers on CIFAR. Whether it can be scaled to deeper networks remains unknown. The proposed method is also only validated on CIFAR while many previous methods can train deep resnet on ImageNet without a normalization layer (fixup and [1]).\n[1] Qi et al. Deep Isometric Learning for Visual Recognition. ICML 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C96ElEw8FF", "reviewer": "Reviewer_MPAw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new random initialization scheme Risotto that achieves dynamical isometry for residual networks with Relu activation functions. The key difference from the previous work is that they achieve the dynamical isometry by balancing the signals from both residual and skip branches, unlike the previous works that mainly suppress the signal from the residual branch. They provide theoretical justification for the proposed method and extensive empirical experiments to denote the effectiveness of the proposed method.", "summary_of_the_review": "Overall I believe the paper is above the acceptance level, and I would like to raise my score if the authors address the above concerns.", "strength_and_weaknesses": "Strengths:\n1. The derived signal propagation results does not require mean field approximation but highlight input separability issues.\n2. The proposed method considers the contribution from both residual and skip branches unlike most previous works achieving dynamical isometry in residual networks.\n3. Promising empirical results on CIFAIR and Tiny ImageNet datasets.\n\nWeaknesses:\n1. It is not clear why feature diversity is important for the initialization. The authors differ the proposed method from the previous works like Fixup or SkipInit by arguing that the proposed method promotes feature diversity. However, the reason why feature diversity is needed is not well motivated in the paper. It would be good to study how the feature diversity affects the initialization.\n2. The proposed method has limited applicability. Because the method and its derivation highly rely on post-activation ResNet structure, it can not be directly applied to pre-activation ResNet (which could achieve dynamical isometry easily using skip connections). As it is limited to residual networks, it can not be applied to other models such as transformer. \n3. Large-scale evaluations are needed to show the effectiveness of the proposed method. The authors only evaluate the proposed method on CIFAIR and Tiny ImageNet datasets. It would be good to evaluate the method on full ImageNet setting. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7ArxGo5sER", "reviewer": "Reviewer_XaUD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to initialize ResNets with ReLU activations with exact dynamical isometry by combining ideas from delta-orthogonal initialization and looks-linear initialization. The proposed initialization scheme guarantees that every residual block is an exact orthogonal map. A small empirical evaluation shows that the scheme is able to train moderately deep ResNets with/without batch normalization.", "summary_of_the_review": "The proposed initialization scheme is quite complicated and it will only be adopted by community if there are clear empirical benefits. Since the experiments provided were not convincing, I recommend rejection.", "strength_and_weaknesses": "Strengths:\n\n1) To my knowledge, this is the first paper to apply looks linear initialization (with orthogonal weights) to a ResNet.\n2) Experiments suggest this scheme achieves good performance.\n\nWeaknesses:\n\n1) Note that Balduzzi et al. already combined orthogonal initialization with looks linear initialization in feedforward networks. Since this scheme already achieves dynamical isometry, it is not clear to me why the skip connection is necessary. However the paper does not explore this question? I would like to see experiments assessing whether or not the skip connection is beneficial in practice.\n\n2) Definition 2.2 (\"Normal ResNet initialization\") is not how practitioners normally initialize ResNets. In particular, practitioners do not normally ensure $\\alpha^2 + \\beta^2 = 1$, indeed almost all implementations do not include either $\\alpha$ or $\\beta$. This leads to some confusing statements later in the text.\n\n3) The experimental section is very weak. No large scale datasets (eg ImageNet) are included, and almost all experiments use the shallow ResNet-18 architecture, which is relatively easy to train with/without BatchNorm simply by ensuring the signal doesn't explode on the forward pass. Initialization becomes more important in deeper networks (eg 100+ layers). \n\n4) No comment is given describing how learning rates and other hyper-parameters are tuned, and I suspect that poor tuning accounts for many of the observed differences between initialization schemes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject