Block-level Stiffness Analysis of Residual Networks
Authors: Eliska Kloberdanz
Wei Le
Venue: Submitted to ICLR 2023 (2023)
Keywords: resnets
stiffness
ordinary differential equations

Abstract: Residual Networks (ResNets) can be interpreted as dynamic systems, which are systems whose state changes over time and can be described with ordinary differential equations (ODEs). Specifically, the dynamic systems interpretation views individual residual blocks as ODEs. The solution to an ODE is an approximation; and therefore contains an error term. If an ODE is stiff it is likely that this error is amplified and becomes dominating in the solution calculations, which negatively affects the accuracy of the approximated solution. Therefore, stiff ODEs are often numerically unstable. In this paper we leverage the dynamic systems interpretation to perform a novel theoretical analysis of ResNets by leveraging findings and tools from numerical analysis of ODEs. Specifically, we perform block level stiffness analysis of ResNets. We find that residual blocks towards the end of ResNet models exhibit increased stiffness and that there is a significant correlation between stiffness and model accuracy and loss. Based on these findings, we propose that ResNets behave as stiff numerically unstable ODEs.

Under review as a conference paper at ICLR 2023
BLOCK-LEVEL STIFFNESS ANALYSIS OF RESIDUAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Residual Networks (ResNets) can be interpreted as dynamic systems, which are systems whose state changes over time and be described with ordinary dif- ferential equations (ODEs) (Haber et al.) 2018 Weinan 2017). Specifically, the dynamic systems interpretation views individual residual blocks ODEs. Nu- merical techniques for solving ODEs result in an approximation; and therefore contain an error term. If ODE is stiff it is likely that this error is amplified and becomes dominating in the solution calculations, which negatively affects the ac- curacy of the approximated solution (Burden et al., 2015). Therefore, stiff ODEs are often numerically unstable. In this paper we leverage the dynamic systems ResNets by leveraging
interpretation to perform novel theoretical analysis findings and tools from numerical analysis of ODEs. Specifically, we perform block level stiffness analysis of ResNets. We find that residual blocks towards the end of ResNet models exhibit increased stiffness and that there is statisti- cally significant correlation between stiffness and model accuracy and loss. Based these findings, we propose that ResNets behave stiff numerically unstable ODEs.
INTRODUCTION
There are three theoretical interpretations of Residual Networks (ResNets): (1) unraveled ResNets, (2) unrolled iterative estimation, and (2) dynamical systems. The unravelled interpretation views ResNets as a collection of 2 paths along which the input data flows, where n the number of residual blocks (Veit et al., 2016]. The unrolled iterative estimation interpretation explains ResNets as iterative approximators, where the first estimate provided by the first layer and is progressively refined by subsequent layers (Greff et al. 2017). Finally, the dynamical systems view interprets ResNets as discretized dynamical systems, where ResNets are seen as ordinary differential equa- tions (ODEs) (Haber et al.) 2018 Chen al.) 2018 Lu et al. 2018). Specifically, the dynamical systems interpretation regards ResNets's residual blocks as a series of forward Euler discretizations of an initial value ODE, This connection between residual blocks and ODEs can be leveraged for novel theoretical analyses that further our understanding and interpretation of ResNets. In this paper we perform stiffness analysis of ResNets and their residual blocks by leveraging findings from numerical analysis of ODES.
Stiffness is an interesting property of an ODE that has important implications. If differential equation is stiff, the solution to the equation will have an unpredictable error that will negatively affect the accuracy of the approximated solution (Burden al.) 2015). Therefore, stiff' ODEs are often numerically unstable and their solutions have accuracy issues (Seinfeld et al. 1970: Shampine Gear 1979).
There is no rigorous definition of stiffness; however there are certain phenomena that indicate that a problem may be stiff. One way to assess stiffness of an ODE is to analyze the eigenvalues of the Jacobian of the ODE, Specifically, if the eigenvalues of the Jacobian differ greatly in magnitude [Butcher 2008 Bui & Bui 1979) or if large portion of the eigenvalues have negative real parts Burden et al.] 2015). is likely that the ODE is stiff. Unfortunately, there no specific thresh- olds regarding what constitutes a high variation in magnitude of eigenvalues or high proportion of eigenvalues with negative real parts.

---

Under review as a conference paper at ICLR 2023
In this paper we investigate whether ResNets exhibit some of the characteristics that can indicate stiffness. Specifically, we focus on analyzing the eigenvalues of the Jacobian of individual residual blocks with respect their inputs in ResNet18, ResNet34, and ResNet50. Using these eigenvalues we calculate (1) the stiffness index and (2) proportion of eigenvalues with negative real parts for each residual block and target label, where the stiffness index captures the degree of variation of the cigenvalues magnitude (Kim et al.) (021).
We find stiffness significantly varies with respect to different residual blocks. Specifically, we find that residual blocks towards the end of the network indicate increased stiffness, i.e.: they have a high stiffness index and also high proportion of eigenvalues with negative real parts. For example, the last block in ResNet50 has a stiffness index of -35.32 3.3 and 31.96% of the eigenvalues of its Jacobian have negative real parts.
We perform correlation analysis between stiffness and model accuracy/loss and show that they are correlated and that their correlation statistically significant. In particular, we calculate the pearson correlation, which ranges from -1 to I, where positive values indicate positive linear relationships and negative values indicate inverse relationships. We also compute the p-value of the correlation coefficients, where a p-value less equal to 0.05 is considered statistically significant. For example, given ResNet18's s last residual block, the stiffness index has a negative correlation with accuracy of -0.36 and the percentage of negative cigenvalues has positive correlation with loss of 0.34. The stiffness index of the last residual block in ResNet34 has a negative correlation with accuracy of -0.20 and the percentage of negative eigenvalues has negative correlation with accuracy of -0.38. Finally, given the last block in ResNet50, the correlation between the stiffiness index and accuracy is -0.27 and the correlation between the percent of negative eigenvalues and loss 0.25.
Based on these findings we propose that ResNets can be interpreted as not only as ODEs, but specif- ically as stiff ODEs. which are numerically unstable. This interpretation could be another possible explanation of why DNNs are susceptible to adversarial examples.
The rest of the paper is organized as follows. Section 2 provides related work, Section provides a detailed explanation of the dynamic systems interpretation of ResNets, which is the basis of this paper. Section 4 describes our stiffness analysis, which investigates whether individual residual blocks behave as stiff ODEs. Finally, Section Soationn]rrep reports the results of our analysis and Section summarizes our conclusions.
2 RELATED WORK
There is a large body of numerical analysis literature that studies stiffness; however, in this paper we are the first ones to connect the concepts of stiffness and ResNets via the dynamical systems interpretation (Haber et al. 2018, Weinan 2017) to propose that ResNets can be viewed as stiff ODEs. Prior related works primarlly focus on the challenges of solving stiff ODEs, which are prone to yielding unreliable results due to their stiffness.
To get an overview of on topic of stiff ODEs, please refer to SSinfeld et al. (1970), who provide a review of numerical integration techniques for stiff ODEs. Additionally, (Shampine & Gear 1979) describe the meaning of stiffness, why do stiff problems arise, how they can be recognized, and also compare the approprietness of different solution methods. |Kaps & Rentrop (1979) propose generalized Runge-Kutta methods of order four with stepsize control as a solution method for stiff ODEs. Because stiff ODEs often contain varying time scales, [Engquist & Tsai (2005) propose heterogeneous multiscale methods for stiff ODEs and show promising stability and convergence results. Enright (1978) develop a matrix updating technique that aims reduce the computational cost of matrix operations needed for solving stiff ODEs. Other works are domain specific such as (Young & Boris 1977), which proposes a new method for solving stiff ODEs describing the chemical kinetics of reactive flow problems.
One of the most recent works that focuses on solving stiff ODEs is Kim et al. 2022I) who leverage neural ODEs. Neural ODEs have been introduced by (Chen et al. 2018 8) as form of new continuous depth deep learning models that parametrize ODEs with a neural network and learn the underlying dynamic system. Kim al. (2021) study learning neural ODEs data generated from two classical stiff systems, ROBER (Robertson, 1967) and POLLU (Verwer 1994). which describe the dynamics of species concentrations in stiff chemical reaction systems. They propose derivative caleu-
2

---

Under review as a conference paper at ICLR 2023
lation that be used for neural network backpropagation and output normalization techniques for learning stiff ODEs with neural ODEs.
The dynamic systems interpretation views ResNets discretized forward Euler method. which is known to be numerically unstable solution to an ODE (Press et al. 1992). Li al. (2020) leverage this fact to establish link between numerical stability of ResNets and their adversarial robustness. They propose modifications to the ResNet architecture that effectively replace the numerically un- stable forward (explicit) Buler method with backward (implicit) Euler method, which is numerically stable. Their experiments show that the modificantion improves adversarial robustness.
3 DYNAMICAL SYSTEMS INTERPRETATION OF RESNETS
Dynamical system is system whose state changes with time (Arrowsmith & Place 1990). Dy- namical systems can be discrete or continuous depending on whether the time variable is discrete continuous. Given dynamical system defined by function f. time variable and state æ, iteration of discrete dynamical system can be described as Z(+1 f(z+), ), while a continuous system represented differential equation dr/dtt = f())). Differential equations for describing non-linear equations, e.g.: calculating the angular position of swinging pendulum is canonical dynamic system that be solved via differential equations (Brin & Stuck 2002).
especially useful
Prior works (Haber et al. 2018) Weinan 2017) have shown that ResNets can be interpreted as ordinary differential equations, where ordinary signifies that the differential equation contains function with respect to only one (as opposed to multiple) independent variable.
The ResNet architecture consists of convolutional input layer followed by residual blocks with identity skip connections and fully connected output layer. The identity skip connections allow ResNets skip block; and therefore, find the optimal number of layers. A residual block shown in Figure i the building block of ResNets. It is composed of two convolutional layers and can be defined as:
Fi+1 F(x,, W.) Ti.
where F' the residual module and W2 are its parameters.
Conv layer
F(X, w)
relu
Conv layer
F(X, xx* =xX
relu
Figure 1: Residual Block
Without loss of generality, Equation can be written as follows with an additional parameter h.
Fe+1 hF W;) + 2
(2)
Equation can be rearranged as:
Ti+1 F F(r,, Ws)
(3)
For sufficiently small h in Equation 3 the residual block becomes:
=(t) ) (=x(1), W(I)), a(0) mo for STST,
3

---

Under review as a conference paper at ICLR 2023
where x(0) and x(T) correspond the input and output feature maps and T the depth of the residual network.
The Euler forward method is an explicit method for solving ODEs using the following formula: 1++1 24 + h.f (x,, W;),
(5)
which has the same form as Equation 2 and where h represents the step size.
The equivalence of ResNets and ODENets can be demonstrated with a simple example shown in Figure which compares the results of classical ResNets with ODENets that we fitted to randomly generated numerical inputs and outputs.
15
solution
Resnet predictions
10
ODE Net predictions (Euler)
5
output
0
-5
-10
-15
Input
Figure 2: ResNets correspond to ODENets, which replace residual block with ODEs
4 STIFFNESS ANALYSIS OF RESNETS
4.1 STIFFNESS
Numerical solution to an ODE an approximation; and therefore contains an error term. an ODE is stiff it is likely that this error is amplified and becomes dominating in the solution calculations (Burden et al. (013)). Therefore, stiff ODEs are often numerically unstable. Typically, stiff ODE equations problems for which explicit methods do not work (Hairer & Wanner 1996 2002).
Stiff systems derive their name from the motion of spring and mass systems that have large spring constants, and are common, for example, in the study of vibrations, chemical reactions, electrical circuits (Burden et al. 2013), and control theory (Kaps & Rentrop, 1979). There is no rigorous mathematical definition of stiffness; however certain indicators can be used to assess if an ODE is stiff. One of those indicators are the eigenvalues of the Jacobian of the function in the ODE, where the Jacobian is matrix that consists of first order partial derivatives of function and is defined in Equation 6 Residual blocks can be viewed as functions; and therefore, we can compute their Jacobians. Moreover, because because the Jacobians of residual blocks are square matrices, can compute their eigenvalues.
0f((x) 0a
(x(())
Ox
Of(x) dzi
Ox(x) Dxn
J
(6)
fm)()
0 (x)
0f
dz
Ox
Eigenvalues are characteristic scalar values, which have special properties and can be computed only for square matrices via eigenvalue decomposition. Given square matrix A, if there is vector X E R + such that
AX XX

---

Under review as a conference paper at ICLR 2023
for scalar X, then À is the eigenvalue of A (Layton & Sussman 2017 Eigenvalue decompo- sition of a square matrix yields three matrices: Q. V, and Q such that A = QVQ -0 this matrix factorization, Q is square matrix that contains eigenvectors in its columns, V is diagonal matrix that contains eigenvalues on its diagonal, and Q simply a transpose of Q.
Stiff ODEs cause severe numerical integration problems and their solutions typically associated with stability and accuracy problems (Seinfeld et al. 1970). Both individual and systems of ODEs be stiff (Chapra & Canale 1986). An example of single stiff ODE is: dy dt 1000y + 3000 2000e
For an initial value y() 0, the analytical solution is:
y = 0.998e 1000s 2.002e
Figure compares an approximate and exact solution to the example stiff ODE in Equation 8 We approximate the solution using an explicit Fuler method, but even with step size as small as 0.0015, the approximate solution is very inaccurate and unstable.
1.4
1.4
1.0
1.0
2
0.6
0.5
0.2
Approalmate, shep slze 0.0015
Approsimate, step size 0.005
Feset
Feert
0.000
0.003
0.002
0.033
0.204
0.055
0.005
0.00
0.22
0.86
0.56
0.88
030
(a) Exact approximate solution time interval (b) Exact approximate solution time interval from 0.0 0.006
from 0.0 0.1
Figure 3: Solutions to stiff ODEs have accuracy and stability issues
4.2 STIFFNESS ANALYSIS
In section we have explained how ResNets can be interpreted as discretized forward Euler method applied to an initial value ODE as per the dynamic systems interpretation (Haber et al.) 2018 Weinan 2017). Prior works have shown that the forward Euler method numerically unsta- ble (Press et al. 1992) and attempted to modify ResNets to increase their stability (Li et al. 2020). Additionally, the phenomenon adversarial examples (Goodfellow et al.) (0155) that produce incor- rect predictions when passed inputs into ResNets, can be viewed instability (oee et al. 2021). Adversarial examples are inputs that have been carefully crafted with the goal of fooling neural network. They add small virtually imperceivable perturbations to initial inputs such that the inputs appear legitimate and unaltered, but still cause the network to output an incorrect prediction. Numerically stable methods data or conditions such that their output does not result in dramatic change in the solution to the problem (Burden al.) 2015). Given these issues and the fact that the explicit methods for solving ODEs are likely to fail if problem is stiff. it is possible that ResNets are stiff and numerically unstable.
manifestation of numerical
resilient to small changes in initial
To investigate this, we perform numerical analysis to test whether ResNets exhibit characteristics of stiff ODEs. Specifically, we focus on analyzing the eigenvalues of the Jacobian of each residual block with respect to the block inputs. Using these eigenvalues, for each residual block, we calculate (I) the stiffness index and (2) what proportion of eigenvalues contain negative real parts according respectively.
to equations 10 and I
4.2.1 STIFENESS INDEX
A problem is stiff if the eigenvalues the Jacobian of the function described by the ODE differ greatly in magnitude, which can be measured by the stiffness index (Kim et al. 2021). The stiffness
5

---

Under review as a conference paper at ICLR 2023
index is defined in Equation 10 where A corresponds to the eigenvalues of the Jacobian matrix and Re represent the real part of a complex number.
max (Rer))
S
(10)
min(Re(A.))
4.2.2 PERCENT OF EIGENVALUES WITH NEGATIVE REAL PARTS
Another indication of stiffness is if majority of those eigenvalues have negative real parts (Heath & Munson 1996). We calculate this metric as follows.
count Re > 0)
P
(11)
count(A.)
4.2.3 COMPUTATIONAL COMPLEXITY
Computing the Jacobian of residual block and its eigenvalues is computationally expensive. Given a residual block with input and output dimension equal n F IR. - RR", obtain the Jacobian matrix of F we need to compute the first partial derivative of each input dimension with respect to each output dimension, which yields a n X n matrix. Therefore, the larger the inputs and residual blocks, the more more computational power is required to compute the Jacobian and its eigenvalues. We have performed experiments on GPUs as much as 80GB memory: however, that was still not sufficient for larger ResNet architectures or larger inputs. For this reason focus R8 ResNet34, and ResNer50 that have been pre-trained on CIFARIO. Specifically, we compute the Jacobian of every residual block with respect each input to perform the analysis. To illustrate the computational time complexity we provide the number of elements in the Jacobian matrix and the number of eigenvalues of the Jacobian for CIFARIO ResNet18 and ResNet34 in Table [O and [ in the Appendix, which show that each block produces several million of these elements. For example, the first block of both ResNet18 and ResNet34 produces over 268 million Jacobian matrix elements and over 16 million eigenvalues.
5 RESULTS
We perform a stiffness analysis of ResNet18, ResNet34 and Resnet50 with CIFAR10 test images, and our results show that residual blocks towards the end of ResNets behave as stiff ODEs. More- over, we show that stiffness varies with respect to different blocks as opposed to different inputs, which means that stiffness is a property of the model itself. It also implies that our results ob- tained using CIFAR10 10 images as inputs generalize across different inputs and reflect on the model as opposed to particular inputs. Additionally, we also show that there statistically significant correlation between elevated stiffnees and model accuracy/loss.
5.1 RESNET18
ResNet18 contains eight residual blocks and in Table 2 we show that the stiffness index and percent- age of eigenvalues with negative real parts spikes in last three blocks. The average block stiffness index is -4.15, but the last residual block shows significant spike in stiffness of -12.99, which is three times higher than the average. The percentage of negative eigenvalues behaves similarly. The average percentage of negative eigenvalues in a block is 10.04, but this percentage doubles in the last three residual blocks. This indicates that the last three residual blocks of ResNet18 behave as stiff ODEs.
In addition to analyzing stiffness of different blocks we also calculate the average stiffness index and percentage of negative cigenvalues produced by different inputs with different target labels. In table 3 we show stiffness does not change with respect to different image labels. Based on that we conclude that it is ResNet18 itself that drives stiffness and not its inputs.
Next, we assess the significance of our finding that the last three residual blocks of ResNet18 behave stiff ODEs by testing if the two stiffness metrics correlated with ResNet18's accuracy and loss and if those correlations are statistically significant. Table [ We we show that the increased stiffness in the last three blocks is correlated with ResNet18's accuracy and loss. For example, the last

---

Under review as a conference paper at ICLR 2023
acc, stiffness
acc, pct
loss,
loss, pct
block
index
neg eigen values
stiffness index
neg eigen values
corr
p-value
corr
p-value
corr
p-value
corr
p-value
0.00
0.83
-0.01
0.67
0.01
0.76
0.00
0.87
0.03
0.06
-0.01
0.45
-0.02
0.19
0.01
0.45
-0.01
0.74
-0.06
0.00
0.02
0.17
0.06
0.00
-0.04
0.01
-0.10
0.00
0.04
0.03
0.10
0.00
0.00
0.97
-0.14
0.00
0.01
0.68
0.13
0.00
6
-0.09
0.00
-0.38
0.00
0.08
0.00
0.36
0.00
-0.03
0.08
-0.18
0.00
0.02
0.34
0.18
0.00
8
-0.23
0.00
-0.36
0.00
0.21
0.00
0.34
0.00
all blocks
-0.01
0.15
-0.07
0.00
0.01
0.19
0.07
0.00
Table 1:
ResNet18 8 correlation analysis between stiffness and accuracy/loss with statistically signif-
icant results in bold
residual block's percentage of negative eigenvalues has a -0.36 correlation with the model accuracy and its stiffness index has a 0.34 correlation with the model loss. The p-value of these correlations less than 0.05, i.e.: they are statistically significant. This means that higher proportion of negative cigenvalues implies lower accuracy and higher stiffness index implies higher loss.
stiffiess index
pet of negative
block
stiffness index
pet of negative eigen values
eigen values
label
-2.28
2.82
-4.31
10.46
2
-5.12
2.54
-4.22
9.71
3
-1.28
1.96
-4.02
10.31
4
-4.04
4.33
3
-4.00
11.57
5
-1.30
4.35
-4.02
9.80
6
-4.94
19.73
5
4410
10.14
7
-1.28
22.65
-4077
9.53
8
-12999
21.92
4.13
9.20
all blocks
-4.15
10.04
4.15
9.67
-.31
9.99
Table 2: ResNet18 stiffness analysis by block with stiffness index and percent of negative eigen val- ues with real parts over 10.00 in bold
Table 3: ResNet18 stiffness analysis by label
5.2 RESNET34
ResNet34 contains sixteen blocks and similarly to ResNet18, it is the residual blocks towards the end of the network that exhibit increased stiffness. In Table show that the last eight blocks show a spike in stiffness index and percentage of negative cigenvalues. The average block stiffness index -7.84, but it approximately doubles in most of the last eight blocks. For example, the sixteenth block has a stiffness index of Similar results can be observed for the percentage of negative eigenvalues Resnet34's s blocks. The average block contains 15.15 of negative eigenvalues, but that percentage is significantly higher in the last 7 blocks. For example the last block's 's Jacobian matrix has 22.34 % of negative eigenvalues and the twelfth and thirteenth blocks have 33.58 and 33.13 % respectively.
Table shows that stiffness does not seem to vary with respect to different input labels, which the same conclusion made in of ResNet18. Compared to Table 5 which shows block level analysis, the stiffness index and percentage of negative eigenvalues is virtually constant across different input image labels.
Similarly to ResNet18, we perform a correlation analysis to assess the impact of stiffness observed in the latter half of ResNet34's blocks on its accuracy and loss. Table 4 shows that the residual blocks with increased stiffness have statistically significant correlation with the model's accuracy and loss. For example, the last block's stiffness index has -0.20 correlation with the model's

---

Under review as a conference paper at ICLR 2023
acc, stiffness index
acc, pet
loss,
loss, pct
block
neg eigen values
stiffness index
neg cigen values
corr
p-value
corr
p-value
corr
p-value
corr
p-value
-0.07
0.02
-0.01
0.74
0.05
0.09
0.00
0.94
0.02
0.44
0.01
0.79
-0.03
0.37
-0.01
0.80
-0.02
0.42
0.02
0.56
0.02
0.53
-0.02
0.54
-0.01
0.73
0.03
0.39
0.00
0.95
-0.02
0.52
-0.01
0.82
-0.02
0.48
-0.01
0.82
0.02
0.59
0.06
0.03
0.00
0.94
-0.07
0.03
0.00
0.89
0.05
0.09
-0.12
0.00
-0.04
0.17
0.12
0.00
8
-0.07
0.02
-0.22
0.00
0.04
0.19
0.23
0.00
-0.07
0.03
-0.26
0.00
0.06
0.05
0.26
0.00
10
-0.17
0.00
-0.32
0.00
0.17
0.00
0.31
0.00
-0.22
0.00
-0.31
0.00
0.21
0.00
0.30
0.00
12
-0.11
0.00
-0.23
0.00
0.10
0,00
0.22
0.00
13
0.05
0.08
-0.21
0.00
-0.05
0.12
0.21
0.00
14
0.04
0.18
-0.05
0.08
-0.02
0,41
0.08
0.01
15
-0.23
0.00
-0.34
0.00
0.21
0.00
0.33
0.00
16
-0.20
0.00
-0.38
0.00
0.18
0.00
0.35
0.00
all blocks
-0.01
0.11
-0.05
0.00
0.01
0.13
0.05
0.00
Table 4: ResNer34: Correlation analysis between stiffness and accuracy/loss with statistically sig-
nificant results in bold
accuracy and that the percentage of negative eigenvalues in the last block have a 0.35 correlation with the model's loss.
stiffness index
pct of negative
block
eigen values
-3.44
3.10
2
-5.24
2.81
3
-5.91
2.78
stiffiess index
pet of negative
4
-1.10
2.50
label
eigen values
5
-6.00
4.78
-7.82
15.94
6
-5.69
4.10
-7.77
14.12
7
-5.29
5.15
-7.79
15.69
8
-1.73
7.83
-7.88
17.14
9 114.56
11.91
-7.51
15.31
10
-9.47
25.78
-7.98
15.44
11
-10.46
30.69
-7.79
14.58
12
-13.55
33.58
-8.00
14.36
13
-11.26
34.13
-7.96
14.49
14
-1.85
25.56
-7.84
14.42
15
13.59
25.39
16
166.22
22.34
Table 6: ResNet34 stiffness analysis by label
all blocks
-7.84
15.15
Table 5: ResNet34 stiffness analysis by block with stiffness index and percent of negative eigen val- ues with real parts over 10.00 in bold
5.3 RESNET50
Due to computation constraints explained in Section 4.2.3 for ResNet50, larger architecture, we compute the stiffness index and percentage of negative eigenvalues only for its sixteenth block, which the last residual block. The last residual block Jacobian is the smallest from all blocks shown in Tables 10 and TT: and is therefore the least computationally expensive. Moreover, the
8

---

Under review as a conference paper at ICLR 2023
results from our stiffness analysis on both ResNet18 and ResNet34 show that stiff residual blocks reside towards the end of the network architecture.
As shown in Table 8 the stiffness index of the last residual block in ResNet50 is 35.32. which is higher than any block stiffness index in ResNet18 or ResNet34. The percentage negative eigen- values is 31 11.96% which is also high. Similarly to ResNet18 and ResNet34, Table 9 demonstrates that the stiffness indicators do not significantly vary in response to different inputs. Finally, Table 7 shows that the stiffness index is about 20% correlated with ResNet50's accuracy and loss and that the correlation is statistically significant.
acc, stiffines index
acc, pct
loss,
loss, pct
block
neg eigen values
stiffness index
neg eigen values corr -0.15
corr
p-value
p-value
p-value
p-value 0.00
16
-0.27
0.00
0.20
0.00
0.25
0.00
Table 7: ResNet50: Correlation analysis between stiffness and accuracy/loss with statistically sig- nificant results in bold
stiffness index
pet of negative
label
eigen values
-31.28
31.91
-39.61
28.23
-32.26
32.79
stiffiness
pct of negative
block
-28.45
35.92
index
eigen values
-33.27
34.14
16
-35.32
31.96
5
-34.33
36.37
-39.56
27.23
Table 8: ResNet50 stiffness analysis of last block
-41.05
32.17
-37.34
27.77
-36.30
33.14
Table 9: ResNet50 stiffness analysis by label
CONCLUSION
In this paper we leverage the dynamic systems interpretation of ResNets that views them as ODEs whose solutions are equivalent to forward Euler discretization of an initial value ODE. We perform a novel stiffness analysis whose results lead to our proposal that ResNets can interpreted as stiff ODEs. Solutions involving stiff ODEs have numerical problems that cause poor accuracy and sta- bility, because the error term in the approximate solution becomes dominating. While stiffness does not have rigorous mathematical definition, there are some analyses that be performed to sess whether its is likely that an ODE is stiff. In this paper we focus on studying the eigenvalues of the Jacobian matrix that is created by computing the first partial derivatives of residual block with respect to its inputs. Specifically, using the eigenvalues we compute the stiffness index and percentage of negative eigenvalues with real parts for each residual block to perform stiffness analysis. Our results show that residual blocks towards the end of ResNets exhibit high stiffness index and high percentage of negative eigenvalues. We also show that stiffness varies with respect to different blocks as opposed to types of inputs, which indicates that stiffines is an inherent prop- erty of some residual blocks. To assess the significance of these results we perform correlation analysis, which indicates that stiffness has a statistically significant correlation with a model's 's accu- racy and loss. Based on these findings we conclude that some residual blocks ResNets (specif- ically the ones in the latter half of the architecture) correspond to stiff ODEs. Therefore, we pro- pose that ResNets could viewed not only as ODEs as per the dynamic systems interpretation, but specifically stiff ODEs, which are numerically unstable. Our stiffness analysis along with in- structions for reproducing results are available at: https: :oos://o /atss////nnnnooaaaee Cariiasss.ppeeeee 4open. science/ r/Sti i fress- Ana ysis ResNets 8D2/ README..

---

Under review as a conference paper at ICLR 2023
REFERENCES
David K. Arrowsmith and C. M. Place. An introduction to dynamical systems. 1990.
Michael Brin and Garrett Stuck. Introduction to dynamical systems. 2002.
Tien D. Bui and Timothy Bui. Numerical methods for extremely stiff systems of ordinary differential equations. Applied Mathematical Modelling, 3:355-358, 1979.
Richard L Burden, Douglas Faires, and Annette M Burden. Numerical analysis. Cengage learning, 2015.
John C. Butcher. Numerical methods for ordinary differential equations. 2008.
Steven C. Chapra and Raymond P. Canale. Numerical methods for engineers. 1986.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Kristjanson Duvenaud. Neural ordi- nary differential equations. ArXiv, abs/1806.07366, 2018.
Ronald A. Vorer Boris Hanin, and Guergana Petrova. Neural network approximation. Acta Numerica, 30:327 444, 2021.
Björn Engquist and Yen-Hsi Richard Tsai. Heterogeneous multiscale methods for stiff ordinary differential equations. Math. Compur., 74:1707-1742. 2005.
Wayne H. Enright. Improving the efficiency matrix operations the numerical solution of stiff ordinary differential equations. ACM Trans. Math. Softw., 4:127-136, 1978.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. CoRR, abs/ 1412.6522, 2015.
Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. ArXiv, abs/1612.07771, 2017.
Eldad Haber, Lars Ruthotto, and Elliot Holtham. Learning across seales - a multiscale method for convolution neural networks. ArXiv, abs/1703.02009, 2018.
Ernst Hairer and Gerhard Wanner. Solving Ordinary Differential Equations Equatiio II. Stiff and Differential- Algebraic Problems, volume 14. 01 1996. doi: 10.1007/978-3-662-9994777
Erstt Hairer and Gerhard Wanner. Solving ordinary differential equations ii: Stiff and differential- algebraic problems. 2002.
Michael T. Heath and F. Munson. Scientific computing: An introductory survey. 1996.
Peter Kaps and P. Rentrop. Generalized runge-kutta methods of order four with stepsize control for stiff ordinary differential equations. Numerische Mathematik, 33:55-68, 1979.
Suyong Kim, Weiqi Ji, Sili Deng, Yingbo Ma, and Christopher Rackauckas. Stiff neural ordinary differential equations. Chaos: An Interdisciplinary Journal of Nonlineur Science, (1(9):093122. 2021.
William J. Layton and Myron M. Sussman. Numerical linear algebra. 2017.
Mingjie Li, Lingshen He, and Zhouchen Lin. Implicit euler skip connections: Enhancing adversarial robustness via numerical stability. In /CML, 2020.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. ArXiv, abs/1710.101211 2018.
William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical recipes in (2nd ed.): the art of scientific computing. 1992.
H.H. Robertson. The solution of a set of reaction rate equations. numerical analysis: An introduc- tion. Academic Press, Cambridge, Massachusetts, 178-182, 1967.
10

---

Under review as a conference paper at ICLR 2023
John H. Seinfeld, Leonard Lapidus, and M. Hwang. Review of numerical integration techniques for stiff ordinary differential equations. Industrial & Engineering Chemistry Fundamentals, 9: 266-275, 1970.
Lawrence F. Shampine and Charles William Gear. A user's view of solving stiff ordinary differential equations. Siam Review, 21:1-17, 1979.
Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In NIPS, 2016.
Jan G. Verwer. Gauss-seidel iteration for stiff odes from chemical kinetics. SIAM J. Sci. Comput., 15:1243-1250, 1994.
E Weinan. A proposal machine learning via dynamical systems. 2017.
Theodore Young and Jay P. Boris. numerical technique for solving stiff ordinary differen- tial equations associated with the chemical kinetics of reactive-flow problems. The Journal of Physical Chemistry, 81:2424-2427, 1977.
A APPENDIX
elements in Jacobian
eigenvalues in Jacobian
block
268,435,,
16,777,216
2
268,435,45
16,777.216
elements
eigenvalues
3
268.435,45
16,777.216
block
in Jacobian
Jacobian
134,217,728 4
8,388,608
268,435,456
16,777,216
67,108,866 5
8,388,608
268,435,456
16,777,216
6
07,108,864
8,388,608
134,217,728
8,388,608 608
67, 07,108,86 864
8,388,608
67,108,864 866
8,388,608
8
33,554,432
4,194,304
33,554,432
4,194,304
16,777,216 9
4,194,304
6
16,777,216
4,194,304
16,777,216 10
4,194.304
8,388,608
2,097,152
16,777,216 11
4,194,304
4,194,304
2,097,152
12
16,777,216
4,194,304
total
801, 12,064
62,914.560
13
16,777,216
4,194,304
14
8,388.608
2,097,152
Table 10: ResNet18 8 number of Jacobian elements and eigenvalues
15
4,194,304
2,097,152
16
4,194.304
2,097,152
total
1,275,068,416
115,343,360
Table 11: ResNet34 number of Jacobian elements and eigenvalues


Reviews:
[{"review_id": "P5Zz0phUe1", "reviewer": "Reviewer_KUvA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper adopts the view of interpreting individual residual blocks in ResNet as ODEs, and introduces a block-wise stiffness concept for analyzing NN performance. In numerical analysis, stiff ODEs are often numerically unstable. And the authors conjectured that stiff ODEs can be one of the reasons that DNNs are susceptible to adversarial examples. They computed the block-wise stiffness for different ResNet models, and perform statistical tests between the stiffness and model accuracy to show mode with larger stiffness might lead to a lower accuracy.", "summary_of_the_review": "The main contribution of the paper is the introduction of stiffness in analyzing the performance of ResNet. It's an interesting yet pre-mature idea. The authors need to either provide a way of incorporating stiffness into improving the robustness of neural networks (as the authors conjectured in the paper - stiff ode is potentially one reason that NNs are vulnerable to adversarial attacks), or demonstrate potential use case and analysis to support proposed concepts.", "strength_and_weaknesses": "Strength: The idea of this paper is interesting, including defining the stiffness of ResNet, computing the block-wise stiffness for different models (ResNet 18, 34, and 50), and performing correlation tests between the network stiffness and their performance.\n\nWeaknesses: Although the idea is interesting, the current manuscript is considerately below the acceptance bar of ICLR. Like in a scientific experiment, you proposed a hyperthesis, then you need to design a well-controlled experiment to verify your conjecture. The current paper is proposing a hyperthesis. It defines stiffness and poses a conjecture that \"Based on these findings we propose that ResNets can be interpreted as not only as ODEs, but specifically as stiff ODEs, which are numerically unstable. This interpretation could be another possible explanation of why DNNs are susceptible to adversarial examples\". To make it a convincing argument, you need to provide evidence that this conjecture is reasonable and practically meaningful. For instance, the authors argue that ResNet is stiff ODEs which may lead to vulnerability to adversarial examples. Thus, is there a way to incorporate stiffness into the training loss and demonstrate improved robustness of neural networks/ResNet?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EoGKTk1Gsh", "reviewer": "Reviewer_j7fb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on the concept of stiffness of ODEs, which has also found elaborate use in the Neural ODE literature. The authors argue that ResNets exhibit stiffness on the block level, i.e. view ResNets as forward Euler discretized dynamical systems where the state changes across depth rather than time. They find that stiffness proxies computed for pretrained ResNet18/34/50 models on CIFAR-10 correlate with accuracy/loss of the model with statistical significance. That is, high stiffness correlates with low accuracy / high loss. Furthermore they find that mainly blocks at the end of the network exhibit increased stiffness.", "summary_of_the_review": "I vote for *rejection* of the paper. The work lacks some content that is needed to verify the conclusion the authors draw, i.e. that ResNets can be viewed as stiff ODEs. To me, judging by the results of the paper, this only applies to a specifc, architecturally equal ResNet models applied to a single classification dataset. Even though I see the argument of the computational burden of the analysis for large datasets and models, *I would expect the authors to verify/transfer their finding on different datasets and task modalities to vote for an accept*. Even though I respect the computational burden of the analysis, I do not see a problem with applying smaller ResNets to lower dimensional problems in order to verify the findings.", "strength_and_weaknesses": "***Pros***: \n- (Potentially) new insight into one of the most commonly used NN architectures\n- Statistical testing for verification of findings\n\n***Concerns/Weaknesses***:\n- *The overall content of the paper is rather shallow*, even though there are rather obvious questions to tackle, e.g. what effect do methods have that address well-behavedness of model Jacobians (L2/Lipschitz regularization [1]), on which the stiffness analysis is solely based on? There are different types of ResNet-blocks that improve their performance (different order of operations, BatchNorm, etc. [2]) - how do they compare in regards to the stiffness analysis employed?\n- *Reproducibility across datasets and different tasks*: In the paper the results only apply to pretrained ResNets on CIFAR10. Do the findings also hold for ResNets trained on different datasets and different task modalities (i.e. regression)? Even though I see that block stiffness seems to be independent of the blocks' input, I'd lack the confidence to assume that this is the case in general (as the Jacobian is in the end a function of the input when considering nonlinear blocks)\n- *The paper could provide more elaborate information regarding the exact experiment/analysis setup* (at least a short section in the appendix): To me it is not directly clear how the pearson correlation is computed with regards to the accuracy/loss of the model, and how the setup gives rise to reasonable statistics if one model (pretrained ResNet<xy>) is applied to the CIFAR10. The authors could more clearly explain if the correlation is computed across test examples etc.\n\n[1] Finlay, Chris, et al. \"Lipschitz regularized deep neural networks generalize and are adversarially robust.\" arXiv preprint arXiv:1808.09540 (2018).\n\n[2] He, Kaiming, et al. \"Identity mappings in deep residual networks.\" European conference on computer vision. Springer, Cham, 2016.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3cWIQKBKfP", "reviewer": "Reviewer_bzzZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work the authors take advantage of the analogy between ResNets and (neural) ODEs, and try to use analysis tools from ODEs to analyze blocks in a ResNet. In particular, they focus on the notion of \"stiffness\" - a set of properties which make ODEs difficult to integrate, and prone to numerical instability. They claim that the stiffness metrics are inversely correlated with loss, suggesting that \"stiffer\" resnets are worse than non-stiff ones.", "summary_of_the_review": "Overall, though the idea of using ODE theory to analyze resnets is interesting, the actual experiments seem lacking in both clarity and impact. Additionally, it is not clear how stiffness affects properties of the forward pass in a way which is detrimental to network performance.", "strength_and_weaknesses": "The idea of using well-established tools from ODE theory to study neural networks is an interesting one. In general leveraging well-developed insights from other fields of study to machine learning can be very fruitful.\n\nHowever, the analysis conducted in this paper is very unclear and underdescribed. For example, the correlation analysis in section 5 is not described at all. What are the points being correlated? Are the same networks being used for the different types of blocks? This whole section needs a lot more explanation. Even with that explanation, it is not clear if the results are relevant to network optimization and design.\n\nAn additional minor point is that the chosen metrics for stiffness are not motivated at all. While they are well-motivated in the ODE literature, most ML researchers and practitioners will not be familiar with these notions, and it is not clear how relevant they are. For example, here stiffness corresponds to some properties of the forward pass of the network; often, poor performance of ResNets is associated with learning/generalization, and not poorly-conditioned forward passes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject