Iterative convergent computation is not a useful inductive bias for ResNets
Authors: Samuel Lippl
Benjamin Peters
Nikolaus Kriegeskorte
Venue:  (2021)
Keywords: Residual neural networks
Recurrent neural networks
Computer vision

Abstract: Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful in many domains, so they might provide good solutions for neural networks to learn. Here we quantify the degree to which ResNets learn iterative solutions and introduce a regularization approach that encourages learning of iterative solutions. Iterative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consistent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer vision tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across layers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a ``recurrent" ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with a recurrent network computing the successive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normalization. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and convergent, respectively. However, neither recurrence regularization nor spectral normalization improve classification accuracy on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or on challenging recognition tasks with partial occlusions (Digitclutter). Iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets.

Under review as a conference paper at ICLR 2021
ITERATIVE CONVERGENT COMPUTATION IS NOT A USEFUL INDUCTIVE BIAS FOR RESNETS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has suggested that feedforward residual neural networks (ResNets) approximate iterative recurrent computations. Iterative computations are useful many domains, so they might provide good solutions for neural networks to learn. Here we quantify the degree to which ResNets learn iterative solutions and intro- duce regularization approach that encourages learning of iterative solutions. Iter- ative methods are characterized by two properties: iteration and convergence. To quantify these properties, we define three indices of iterative convergence. Consis- tent with previous work, we show that, even though ResNets can express iterative solutions, they do not learn them when trained conventionally on computer vi- sion tasks. We then introduce regularizations to encourage iterative convergent computation and test whether this provides a useful inductive bias. To make the networks more iterative, we manipulate the degree of weight sharing across lay- ers using soft gradient coupling. This new method provides a form of recurrence regularization and can interpolate smoothly between an ordinary ResNet and a "recurrent" ResNet (i.e., one that uses identical weights across layers and thus could be physically implemented with recurrent network computing the succes- sive stages iteratively across time). To make the networks more convergent we impose a Lipschitz constraint on the residual functions using spectral normaliza- tion. The three indices of iterative convergence reveal that the gradient coupling and the Lipschitz constraint succeed at making the networks iterative and conver- gent, respectively. However, neither recurrence regularization nor spectral nor- malization improve classification accuracy on standard visual recognition tasks (MNIST, CIFAR-10, CIFAR-100) or on challenging recognition tasks with partial occlusions (Digitclutter). Iterative convergent computation, in these tasks, does not provide a useful inductive bias for ResNets.
INTRODUCTION
An iterative method solves difficult estimation or optimization problem by starting from an initial guess and repeatedly applying transformation that is known to improve the estimate, leading to a sequence of estimates that converges to the solution. Iterative methods provide a powerful ap- proach to finding exact or approximate solutions where direct methods fail (c.g., for difficult inverse problems or solutions to systems of equations that are nonlinear and/or large).
Recurrent neural networks (RNNs) iteratively apply the same transformation their internal repre- sentation, suggesting that they may learn algorithms similar to the iterative methods used in math- ematics and engineering. The idea of iterative refinement of a representation has also driven recent progress in the context of feedforward networks. New architectures, based on the idea of iterative refinement, have allowed for the training of very deep feedforward models with hundreds of lay- Prominent architectures for achieving high depth are residual (ResNets; He et al.) (016a) and highway networks (Srivastava et al. 2015), which use skip connections to drive the network to learn the residual: pattern of adjustments the input, thus encouraging the model to learn successive refinements of representation of the input that is shared across layers.
These architectures combine two ideas. The first is to skip connections to alleviate the problem of vanishing or exploding gradients (Hochreiter) 1991). The second is to make these skip connec- tions fixed identity connections, such that the layers learn successive refinement of shared repre-

---

Under review as a conference paper at ICLR 2021
sentational format. The second idea relates residual and highway networks to RNNs and iterative methods. Learning a single transformation that can be iteratively applied is attractive because it en- ables trading speed for accuracy by iterating longer (Spoerer et al. 2019). In addition, preference for an iterative solution may provide a useful inductive bias for certain computational tasks.
However, it is unclear whether ResNets indeed learn solutions akin to iterative methods and, if they do, whether this is a useful inductive bias. The two defining features of iterative methods are (1) iteration and (2) convergence. Here we analyze what extent these features emerge in ResNets. In order to investigate whether these features provide a useful inductive bias, we introduce two simple modifications of classical ResNets and study their impact on a number of datasets. First, and MNIST (LeCun et al. (210)) as examples
we study CIFAR-10, 10, CIFAR-100 [Krizhevsky|2000 of classical vision tasks, assessing the networks' performance and sample efficiency. Since iterative and convergent inductive biases may be more useful for tasks that require some degree of recurrence, we also assess the networks' performance and sample efficiency on several variations of Digitcluter, a task which requires the recognition of multiple digits that occlude each other (Spoerer et al. 2017)
To study the effect of iteration, we manipulate the degree of weight sharing ResNets, smoothly interpolating between ordinary and recurrent ResNets. We find that a higher degree of weight sharing tends to make the network more iterative, but does not result in improved performance or sample efficiency. This suggests that in ordinary ResNets, recurrent connections do not provide useful inductive bias and the networks can harness the additional computational flexibility provided by non-recurrent residual blocks.
Recurrence implies iteration, but not convergence, and so is not sufficient for a network to imple- ment an iterative method as defined above. ResNets, whether they are recurrent (i.e. sharing weights across layers) or not, are therefore neither required nor encouraged to converge during training. We demonstrate empirically that ResNets in general do not exhibit convergent behavior and that recur- rent ResNets are more convergent than non-recurrent networks. To study the effect of convergence, we upper bound the residual blocks' Lipschitz constant. This modification adversely impacts per- formance, suggesting that the non-convergent behavior in ordinary ResNets is not merely due to lack of incentive, but underpins the networks' high performance. Across convergent ResNets, a higher degree of weight sharing does not negatively affect performance, This suggests that conver- gent ResNets, in contrasts to non-convergent ones, do not benefit from the increased computational flexibility of non-recurrent residual blocks.
Taken together, our results suggest that an inductive bias favoring an iterative convergent solution does not outweigh the computational flexibility of non-recurrent residual blocks for the considered tasks.
2 RELATED WORK
Prior theoretical work has focused explaining the success of ResNets (He et al. (016a) and the
more general class of highway networks (Srivastava et al. 2015) by studying the learning dynamics in ResNets (Hochreiter) 1991 Orhan & Pitkow 2018 Balduzzi et al. (017) and their interpretation as an ensemble of shallow networks (Veit et al. 2016]] Huang et al. 2018). as a discretized dynamical
system E 2017 Haber& & Ruthotto 2018 E et al. (2019), and as performing iterative refinement.
The iterative refinement hypothesis. Our work builds on Jastrzebski et al. (2018) who argue that the sequential application of the residual blocks in a ResNet iteratively refines a representational estimate. Their work builds on observations that dropping out residual blocks, shuffling their order, or evaluating the last block several times retains reasonable performance (Veit et al. 2016; Greff et al. 2017) and can be used for training (Huang et al. 2016). Another set of methods uses such perturbations to train deep neural networks, using stochastic depth (Huang et al.] 2016 Hu et al.,
2019 Press et al. 2020). Other methods learn to evaluate a limited number of layers that depends on the input (Graves 2017 Figurnov et al. 2017) or learn increasingly fine- meeeggra object categories across layers (Zamir al. (017). Instead of using perturbations encourage stability of the trained network, Ciccone et al. (2018) propose a method inspired by dynamical systems theory to guarantee such stability in their model.
Iterative refinement and inverse problems. The iterative refinement hypothesis is particularly important in the context of inverse problems, which are often solved using iterative methods. This is
2

---

Under review as a conference paper at ICLR 2021
particularly relevant for ResNets trained for perceptual tasks since perception is often conceptualized as an inverse problem Poggio et al. 1985 Pizlo 2001). Rao & Ballard 1999) modeled visual cortex using recurrent neural networks that iteratively infer the latent causes of hierarchical Bayesian
model. Though these networks have been applied to complex datasets (Wen et al. 2018). most
models either learn the inverse model (Rezende et al. 20.42 Kingma & Welling 2013) define an
analytically invertible forward model (Rezende & Mohamed| [2015] Gomez et al. (017)) A notable exception are invertible ResNets (Behrmann et al. 2019), whose inverse can be computed through fixed point iteration. Another set of works starts out with classical iterative algorithm and unfolds this algorithm to create a deep network, approaching a similar problem from the opposite direction
(Gregor & LeCun 2010 Hershey et al.) 2014] [Wisdom et al., 2016 2017). Moreover, Nguyen ctt al. (2019) refine CNNs to yield an inference algorithm on a specific generative model (recently
implemented by Huang et al. 2020).
Recurrent and residual neural networks. The idea of iterative refinement has motivated an in- creasing number of recurrent convolutional neural networks being applied to computer vision with- out an explicit implementation of iterative inference (Kubilius et al., 2019 Spoerer et al., 2019). ResNets may be seen as RNNs that have been unfolded for fixed number of iterations. Sharing weights between the different blocks allows us to train a recurrent residual neural network (Liao & Poggio 2016). In this framework, recurrent residual neural networks may be seen special case of residual neural networks where the weights are equal between all blocks. [Savarese & Maire (2019) and Oh et al. (2019) relaxed this constraint by defining the weights of the different blocks as a linear combination of a smaller set of underlying weights. The work by (Savarese & Maire (2irg) particularly interesting in this context, as their method yielded a more efficient parameterization of Wide ResNets (Zaooruyko & Komodakis 2017), which generally have more channels, but far fewer layers than the architectures we consider here.
Inductive bias of recurrent operations on visual tasks. The inverse problem perspective on per- ception suggests that ordinary recognition tasks in computer vision may already benefit from an iteratively convergent inductive bias. For other tasks, we have additional reasons to believe that re- current processing may be beneficial. For example, object recognition in the primate ventral stream can be well captured by feedforward models (Hung et al. 2005 Majaj et al. (0155) but benefits form recurrent processing under challenging conditions. This includes tasks where the presented objects partially occluded or degraded (Wyatte et al., 2014) or tasks that involve perceptual grouping ac- cording to local statistical regularities (Roelfsema et al. 1999) or object-level information (Vecera & Farah 1997). These observations have inspired tasks such Digitclutter and Digitdebris (Spoerer et al. 2017) as well as Pathfinder and cABC (Kim et al. 2020). For these datasets, adding recurrent networks have been shown to outperform corresponding feedforward control models (Spoerer al.,
2017 Linsley et al. 2020b)).
Implicit recurrent computations. Beyond potentially useful inductive bias, recurrent neural net- works can provide additional computational advantages. If the recurrent operation converges to a fixed point, this fixed point can be determined more efficiently by classical iterative algorithms such as ODE solvers (Chen et al.; 2018). Moreover, in this case, recurrent backpropagation (Pineda) 1987 Almeida 1990) can compute the parameters' gradients much more efficiently than backprop- agation through time (Werbos) 1988) because it does not require storing the intermediate activation. Its memory cost is therefore constant in depth. This method has inspired several promising models in computer vision. Deep equilibrium models (Bai et al. (019) harness a quasi-Newton method to find the fixed point of their recurrent operation. They have recently been applied to object recog- nition and image segmentation tasks (Bai et al., 2020). These works empirically demonstrated the existence of a fixed point under their parameterization. Other models enforce such a fixed point by the Lipschitz constraints we here employ as well. Ciccone et al. (2018) use an upper bound based on the Frobenius norm, whereas Linsley et al. (2020a) approximate the Lipschitz constant using a vector-Jacobian product. Our method (see below) is based on Yoshida & MMyatt| (2017) and has recently been employed (for different purposes) by MMyytto et al. (2018) and Behrmann et al. 20199.
3 Do RESNETS IMPLEMENT ITERATIVE COMPUTATIONS?
Iterative methods are characterized by two key properties: iteration (i.e., recurrent computation) and convergence. In this section, we examine whether residual neural networks have these proper-
3

---

Under review as a conference paper at ICLR 2021
X1
Feediorward
ResWer
1.0
1.1
d
1.2
1.0
1.0
0
Terative
0.8
<N
0.9
0.8 0
Racument
network
1.0
1.2
1.4
1.8
1.3
1/4
1.8
1.0
101..144
2
2
Figure I: A recurrent ResNet implementing simple error-corrective inverse model. The predic- tion based on the current estimate compared to the input æ (via positive and negative errors and n2). The error is used to update 2, whereas is simply retained throughout all blocks. b Trajectories of the estimates 21, &2 across blocks in a feedforward network, iterative steps in an the error-corrective algorithm, residual blocks in a trained ResNet, and residual blocks in trained recurrent ResNet. All four methods converge to the correct estimate (indicated by black 'x'). c Dropping out the fourth block (unbroken line) has minor impact on the ResNet. the last block is iteratively applied to the final estimate, the value diverges for both the residual and the recurrent network (broken lines), indicating that they not learn a convergent structure.
ties. We start by showing that ResNets can express iterative algorithms, but then demonstrate that ResNets do not automatically learn iterative algorithms. We show that this observation extends to large-scale neural networks trained on real data. Finally, we introduce paradigm that allows us to compare neural networks to iterative methods.
3.1 RESNETS CAN REPRESENT ITERATIVE COMPUTATIONS
A popular application for iterative methods is given by inverse problems of the form - (=). It is often not possible to directly compute the inverse, f-1, of the forward model, because of analytical or combinatorial intractability. Instead, approximations (Gershman & Goodman 2014) or iterative error-corrective algorithms (Donoho, (006) are used. Consider the linear forward model = f() = (a1 21, (x222). Though f has an analytical inverse, serves as an illustrative example. Based on an input 2, we can infer the latent variable using the iterative update
(0) = x,
2 g(++1)
=- (())
=
8, (a:2)
vz), 8; == sign(o;))))
where (T), for some T, is the estimate of and e 0 should be sufficiently small
This update can be implemented in small ResNet. Figure Th contains schematic illustration of one block of this network. The representation spanned by an encoder in the beginning of the network contains an input representation æ and representation of the current estimate z. In the hidden layer of the residual block, determine the positive and negative prediction errors and them to update= z:
Pi (0) = ReL.U xx),
20)(")) -o5
ReLU(r; -
((-)) = -0 8;Eap (2) + 8t6,m
This recurrent residual building block would implement an iteratively convergent computation in ResNet. The linear model is, of course, a trivial case, but serves as an illustration of the appeal of this approach. A wide neural network flexible function approximator and learning represent the prediction error instead of the prediction is easier in many cases (Zamir et al. 2017)
3.2 RESNETS DO NOT AUTOMATICALLY LEARN ITERATIVE COMPUTATIONS
The fact that ResNets can express iterative computations does not imply that they necessarily learn iterative solutions when trained via backpropagation.
Here we consider the simple example from above to better understand the behavior that may emerge. We highlight three behaviors that distinguish iterative from non-iterative computations and will ex- amine the behavior of large-scale neural networks in the following sections. As an example, we
In the of linear function, any 6 works. The example given in figure uses
0.3. eg 0.8.
4

---

Under review as a conference paper at ICLR 2021
Stage:
Stage: 2
Slage:
Average
Convergence
Recurrence
Divergence
5.50%
Error rate
90%
90%
90%
Error rate CIFAR-10 R-10
5.00%
50%
50%
4.50%
0%
10
15
12
16
10
15
Convergence
evaluated blocks
Index of skipped blocks
a additionll evaluations
b
5.00%
4.50%
0.0
0.5
1.0
0.0
0.25
Convergence Index
Recurrence Incex
Divergence Index
Divergence Index
Figure 2: Iterative convergence in ResNets with standard training. a The different perturbation meth- ods (early read-out for determining convergence, dropping-out blocks for determining recurrence, and additional evaluations the last block for determining divergence) are illustrated for the three stages of the ResNet. The axis depicts the residual block targeted by the perturbation and the y axis the error rate resulting from the corresponding perturbation (chance performance at 90%). For clarity, one of the six instances emphasized in the plots. b The resulting index values for each stage (small translucent dots) and their averages across instances (large dots). e The error rate for the individual network instances is plotted against Convergence and Divergence Index.
train a conventional ResNet, and a ResNet that uses the same weights each block (equivalent to recurrent network) to invert the function f=) 2 (22). We contrast their behavior with the iterative error-corrective algorithm outlined above.
Due to the lack of constraints, non-residual feedforward neural network changes its representation in every layer. As a consequence, the linear decoder at the end of the network is not aligned to the intermediate representation and early readout (as depicted by the orange dots in Fig- [p) leads to a meaningless estimate. In contrast, the skip connections encourage ResNet to use the same representational format across blocks. This to say that its intermediate representations are better aligned with the final decoder. Early readout is therefore possible and the representation across blocks will approach the final estimate. As consequence, the across- -cooss--locck dynamies of the non- residual network are meaningless, whereas the recurrent and residual network's early readouts are close to the final estimate and approach it smooth manner, just like the error-corrective algorithm (Figure D)). Since iterative methods iteratively refine their initial estimate, their behavior is more similar to the ResNets' monotonic convergence.
Aside from their smooth convergence, fundamental property of iterative methods is their recur- rence (i.e., the repeated use of the same computation). This means that dropping out an earlier block has the same effect dropping out the last one. We relax the requirement for exact repeti- tion (weight sharing) and require merely similar computations. Figure TT illustrates that the trained ResNet is indeed relatively robust to block dropout.
Yet the learned models fall short of an iterative method, which is apparent from a third mode of investigation. Iterative convergence would imply that applying the last block's transformation iter- atively should keep the readout in the vicinity of the actual estimate. This is clearly not the case in our toy example (Figure [d). Rather than representing a convergent estimate, this result is more compatible with understanding ResNets as approaching their final estimate at a constant speed and, in the case of late readout, moving past this estimate and overshooting at the same speed. Notably, both the non-recurrent and recurrent networks exhibit this behavior.
3.3 ITERATIVE CONVERGENCE INDICES
This behavior is not surprising. After all, the network is trained to work for a fixed number of steps and not constrained to stay within the vicinity of its final estimate if more steps are added. However, it reveals that recurrent ResNets do not automatically lear iterative convergent computations. To assess the extent to which they do learn iterative convergent computations we define three con- tinuous indices, which measure convergence, recurrence, and divergence (defined below).
5

---

Under review as a conference paper at ICLR 2021
We evaluate the indices for six instances of ResNet-101, trained on CIFAR-10 Krizhevssyy 2009 This ResNet consists of three stages of 16 residual blocks with 16, 32, and 64 channels, respectively The ResNet achieved 5.2% classification error the validation set. To characterize the extent to which the ResNets have learned an iterative convergent computation, we introduce three indices measuring different aspects of such computations.
Convergence Index. Viewing ResNets as performing iterative refinement suggests that each stage gradually approaches its final estimate before passing this estimate to the next stage using down- sampling layer. By passing the estimate at each of the residual blocks to the next stage, we can monitor how the stages approach their final estimate across blocks (see Figure 2 left panel). In accordance with previous results (Jastrzebski et al. 2018), we find that all stages smoothly approach their final estimate, confirming the earlier intuition of shared representational format. To mea- sure the rate of convergence, we compute the area under the curve (AUC) of the classification error, which we call the Convergence Index. We invert and normalize this value such that a Convergence Index of O corresponds chance level read-out at each residual block, whereas a Convergence Index of I corresponds to an instant convergence to the final classification error at the first residual block. Figure Dp. left panel, depicts this value for each stage, and averaged across stages.
Recurrence Index. To measure the degree of recurrence, we evaluated the effect of dropping out individual blocks on the error rate of the network (sce Figure 2/, middle panel). In non-recurrent ResNet, dropping out earlier blocks may have a stronger effect on the error rate than dropping out the last block. In contrast, in recurrent ResNet, the effect error rate is the same for dropping either earlier blocks or the last block. We therefore computed the difference in error rate observed after these two manipulations. We summarized the behavior by the AUC, which we refer to as Recurrence Index (RI). We invert and normalize this value such that the RI is 0 if dropping out any block leads to an error rate at chance level and the RI is in the case of recurrent algorithm. Even though we study non-recurrent ResNets, dropping out any block other than the very first leads to a negligible drop in performance, replicating previous results (Jastrzebski et al. 2018 Veit et al.) 2016). As a consequence, the RI, across all stages, is close to 1.0 (see Figure 2 middle panel).
Divergence Index, ResNets may either converge to their final estimate or simply approach it in a sequence of steps. An iterative algorithm should not be negatively affected by additional applica- tions of the same function. To examine this property, we apply the last block of each stage for an additional up to sixteen steps (see Figure 2 right panel) and determine the AUC (Divergence Index, DI, see Figure 2b, right panel). We find that no stage is particularly robust to such additional evalua- tions, though the first stage has the lowest DI, indicating that it is the most robust. This suggests that ResNets approach and move away from their final estimate in a sequence of steps, with their compu- tations bearing little similarity to iterative convergent algorithm. A high DI does not indicate that the ResNet has failed in some way. After all, it was not trained to be robust to such perturbations. However, indicates that the ResNet may not implement an iterative convergent computation.
4 MANIPULATING CONVERGENCE AND ITERATION IN RESIDUAL NETWORKS
We provided indices measuring convergence, recurrence, and divergence to assess the degree to which a ResNet implements an iterative method. Even though they able to, ResNets do not nec- essarily learn to implement purely iterative method. In particular, they show divergent behavior. Nevertheless, as we have shown above, their behavior does show some similarity itcrative meth- ods and their success has been attributed to these similarities (Greff et al., 2017]] Jastrzebski et al.) 2018]. This suggests that even though the parameterization and optimization does not promote the emergence of an iterative method in ResNet, a ResNet with iteratively convergent behavior may still have a better inductive bias. To test this hypothesis, we therefore here control the inductive bias, namely recurrence and convergence, of ResNets.
'The ResNet-101 uses the architecture recommended by He al. (01 Appendix A includes details on the architecture and the training paradigm and Appendix corresponding results on two further datasets, MNIST (L.CCun et al. 2010) and Digitclutter-3 (Spocrer et al. 2017 Seccionn contains several variations these definitions

---

Under review as a conference paper at ICLR 2021
4.1 SOFT GRADIENT COUPLING CAN INTERPOLATE SMOOTHLY BETWEEN ORDINARY AND RECURRENT OPTIMIZATION
We propose method blend between recurrent and non-recurrent networks without changing the architecture or the loss landscape. The method is motivated by the observation that we can train
a recurrent neural network by sharing the different blocks' gradients (Jastrzebski et al., 2018 Liao
& Poggio 2016). In ordinary ResNets, the residual block with the weights Wr is changed by following the gradient A = dw,L, whereas RNNs impose as gradient
A -
dW.L. L.
(I)
1=1
where the weights across residual blocks within stage must start from the same initialization. The former means that we do not employ any inductive bias towards recurrence, whereas the latter imposes possibly overly restrictive function space on the architecture. To address both limitations, we propose soft gradient coupling, which uses as its update rule
Å = =1 1) + X e 1].
For A < 1, this retains the entire space of computations enabling both non-recurrent as well as re- current computations. However, for À > 0, the optimization is biased to find more recurrent optima. In contrast to penalty regularizations or strict weight sharing models (Oh et al. 2019 Savarese & Maire 2019) this does not change the network loss landscape, but simply the accessibility of different local minima of the loss landscape.
For networks with coupled gradients (i. e., 0 < x A 1), we initialize their weights recurrently (i.e., all residual blocks within one stage share the same initialization). We unshare batch norm statistics as suggested by Jastrzebski al. 2018), but leave their parameters (softly) coupled.
4.2 SPECTRAL NORMALIZATION CAN GUARANTEE CONVERGENCE RESIDUAL NETWORKS
Iterative methods preserve a stable output when applied repeatedly. In contrast, the output of the ResNets diverged when the last block was applied repeatedly beyond the number of steps was trained for. In order to control the degree of convergence in a ResNet, we constrain the Lipschitz constant of the residual function f. is defined as the minimal value such that for any input a. y- (x) - f(y)l I Ta| y/l- The smaller I,, the more stable f. The Lipschitz constant is hard to determine accurately as is a global property of f. We therefore determine an upper bound L(f) 2 L based on the linear operations within f (see section E.1 Using this upper bound, we replace by its spectral normalization
= c: c: m=xx(p//()))))
(3)
for certain value μ. If Lf}) S 14, the corrective factor will not change the function. If f(f)) 14, it will set the corresponding upper bound of f at i(f) μ.. constraining the residual function's Lipschitz constant.
For a given input æ. a recurrent residual stage defined by the iterative application 2o : x; 21 R(z--1), where R defines the recurrently applied residual block. We wish to guarantee that 2 eventually converges
fixed point z and hope that empirically, 2y, the representation after the specified number of iterations, will be close to this fixed point. According to the Banach fixed point theorem, one way to guarantee such convergence is to require that the residual block's Lipschitz constant R be smaller than one.
We can achieve this by replacing the residual connections between adjacent blocks by residual con- nections between the input æ to the stage and each block, i. e. R((=) := x + f(z). At has the same Lipschitz constant as f. Setting < 1. we therefore guarantee that converges to a fixed point defined by æ. We call this network the properly convergent ResNer (PCR).
To get network more similar to ordinary ResNet, consider R(z) := 2 + f(z). Though convergence is not guarantced for the network defined by this residual block, we show its empirical convergence. We thus call this network the improperly convergent ResNet (ICR).
7

---

Under review as a conference paper at ICLR 2021
a
Convergence Index
Recurrence Index
Divergence Index
1.10
channels
AUC
0.7
1.05
0.4
8
0.5
1.00
0.2
0.0
0.5 (ME))
0.5
(10h-1963 pa6)
(M22
1903
2es3
(tonnecc) paca
09c1
(963
Coupling parameter (Initialization)
Error rate b
CIFAR-10
Digitclutte-55
e parameters
21.0%
10.0%
10.0%
18.0%
Emor
7.0%
7.5%
5.0%
15.0%
12.0%
5.0%
10
0.9
0.5
0.9
1
0.5 0.8 0.7 0.80.0 0.2 0.4 0.8
(non-reol post need yead (oo)
1oon-eed doe.1 (oo) uoca (reu)
Coupling parameter (Initialization)
Convergence Index Divergence Index
Figure 3: Iterative convergence and performance coupled ResNets. a Effect of gradient coupling and initialization (rec.: recurrent; non.-rec.: non-recurrent) indices of iterative convergence for architectures with different numbers of channels. Effect of gradient coupling and initialization on the performance CIFAR-I0 and Digitclutter-5. Relationship between performance and iterative convergence, i.e., Convergence (left) and Divergence Index (right). Models with the same number of parameters visualized by the same color and individual lines. Results in c are on CIFAR-IO.
EXPERIMENTS
To assess our hypotheses, we considered non-recurrently initialized (ordinary) ResNets as well as re- currently initialized ResNets with coupling parameters 0, 0.5, 0.9, and 1. In addition, we considered properly and improperly convergent ResNets across the same coupling parameters, setting μ 0.95 (we only trained these networks on CIFAR-10). We trained several instances of all these ResNets with 8, 16, 32, and 64 channels in the first stage on classical visual recognition tasks (CIFAR-10 10, MNIST, and CIFAR-100) well as Digitclutter, a challenging task with partially occluded objects, which has previously been observed to benefit from recurrent connections (Spoerer al. 2017).
5.1 SOFT GRADIENT COUPLING IMPROVES ITERATIVE RECURRENCE INDICES
As Figure 3a shows, soft gradient coupling indeed improves iterative convergence, increasing the Convergence Index and decreasing the Divergence Index. The Recurrence Index is centered closely around 1. Convergence and Divergence Index, on the other hand, tend to increase and decrease, respectively, both with higher coupling parameters and with higher number of channels. Notably, this trend appears to not hold up for fully recurrent ResNet, corresponding to a coupling parameter of 1. These results show that soft gradient coupling is an effective way of manipulating a ResNet's behavior. Increasing weight similarity across blocks leads to more iterative convergence in a ResNet.
5.2 SPECTRAL NORMALIZATION MAKES RESNETS CONVERGENT
Both the properly and improperly convergent ResNets have high Convergence Index well Divergence Index at almost zero (see Fig. 4h for ICRs and section E.2 for PCRs). The Recurrence Index is again centered around one. For improperly convergent ResNets with 16 32 channels in the first stage, higher coupling parameters generally have lower Convergence Index and higher Divergence Index. Nevertheless, these indices indicate that the spectrally normalized ResNets ex- hibit much more convergent behavior than the ordinary networks. This was only guaranteed for the recurrent, properly convergent ResNets and is therefore an important observation.
5.3 STRONGER ITERATIVE CONVERGENCE DOES NOT PROVIDE A USEFUL INDUCTIVE BIAS
We first assessed the effect of gradient coupling the performance of non-convergent ResNets. As Fig. 3D shows, a higher coupling parameter consistently leads to a higher emror rate, both for CIFAR-10 and Digitclutter. Additional supporting experiments CIFAR-100, MNIST, the Digit-
8

---

Under review as a conference paper at ICLR 2021
Convergence Index
Recurrence Index
Divergence Index
Error rale on CIFAR-1 10
channels
e
1.00
16.0%
AUG
0.9
0.8
0.99
0.10
0.15
12.0%
14.0%
8
16
0.98
10.0%
0.7
0.97
0.05
8.0%
0.5
0.9
(pes) (Pws) (pws) (me))
(18)) (rcc
pac) (We)
(Me)) (ec) (MC) (nc)
Coupling parameter (Initialization)
Coupling parameter (Initialization)
Figure 4: Iterative convergence and performance of improperly convergent ResNets. a Effects of gradient coupling and initialization (rec.: recurrent; non.-rec.: non-recurrent) on iterative conver- gence indices. b Error rates on CIFAR-I0 function of gradient coupling and initialization.
clutter task, and on sample efficiency can be found in section However, for intermediate coupling parameters 0.5 and 0.9, this increase in error rate is smaller for networks with higher capacity (i.e., more channels). This effect also be seen from the relationship between iterative convergence in- dices and performance. In particular, Figure 3 demonstrates that performance is higher for ResNets with a higher Convergence Index and a lower Divergence Index. This effect, however, is driven by the fact that ResNets with higher number of channels also show higher measures of iterative convergence (see Figure 3p). When controlling for the number of parameters (see lines in Figure Bb), we find no clear relationship between Convergence and Divergence Index and performance.
We then assessed the effect of convergence regularization on performance by training several con- vergent ResNets on CIFAR-10 10 (see Fig. 4p for ICRs). Convergence regularization led to higher error rate across all coupling parameters and architectures. A notable exception the fully coupled ResNet with 16 channels, which performs equally with and without convergence regularization. This suggests that convergence is not a useful inductive bias in ResNets. Taken together, these experiments suggest that iterative convergence may provide useful inductive bias for ResNets.
6 DISCUSSION
We introduced soft gradient coupling, new method of recurrence regularization, and demonstrated that this enabled to manipulate iterative convergence properties in ResNets. To measure these properties, we introduced three indices of iterative convergence, quantifying the effect of perturba- tions previously introduced in the literature (Veit et al. 2016 Jastrzebski et al. 2018).
Iterative methods are considered powerful approaches in particular for solving difficult inverse prob- lems. However, here we did not find iterative convergence to be useful inductive bias for ResNets. Moreover, we found that higher degrees of weight sharing did not improve a ResNet's parameter ficiency. One reason for this may be that soft gradient coupling the spectral normalization wrong methods for this purpose or require a different optimization strategy. Our findings also sug- gest, however, that deep feedforward computations ahould perhaps not be characterized iterative refinement on a latent representation, but simply as a sequence of operations smoothly approaching their final estimate. Our conclusions are based on experiments on four visual classification tasks. Visual tasks have been proposed to be inverse problems and therefore lend themselves to iterative inference algorithms. Recognition tasks like Digitclutter that involve partial occlusions of objects have in particular been shown to benefit from recurrent computations (Wyatte et al. 2014 Spoerer al., 2017). However, an iterative method like an coror-corrective algorithm that would require forward model of the data may be more complex and therefore harder to learn than purely dis- criminative model. Hence, for the four tested tasks, ResNets may learn direct inference rather than error-corrccive inference via forward model.
the
Although it did not improve performance here, soft gradient coupling provides a method for smoothly interpolating between feedforward and recurrent neural networks. More generally, soft gradient coupling provides a simple way to encourage sets of weights to remain similar to each other. This technique may find further in relaxing weight-sharing constraints and studying the benefit of various forms of weight sharing, including recurrence and convolution, in deep neural networks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Luis B. Almeida. Learning Rule for Asynchronous Perceptrons with Feedback in a Combinatorial Environment. In Artificial Neural Networks: Concept Learning, pp. 102-111. IEEE Press, 1990.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems, pp. 690-701, 2019.
Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. In Advances in Neural Information Processing Systems, volume 33, 2020.
David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: resnets are the answer, then what the ques- tion? In Proceedings of the 34th International Conference on Machine Learning, pp. 342-350, 2017.
Jens Behrmann, Will Grathwohl, Ricky Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible Residual Networks. In International Conference on Machine Learning, pp. 573-582, 2019. WtRbhtp:: patt:::/ppoceedinntt. mml.nngssmmr......mmmm wwwppppesss..... Sxxess/v97 http///ehemmannnneeee 9a html
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6583, 2018.
Marco Ciccone, Marco Gallieri, Jonathan Masci, Christian Osendorfer, and Faustino Gomez. NAIS- Net: Stable Deep Networks from Non-Autonomous Differential Equations. arXiv:1804.07209 [cs, stat], 2018. Utt/tttp://axxxx http:/ooogaa/b/oooeeeee 0000180244999999
Tim Cooijmans, Nicolas Ballas, César Laurent, Çaglar Gülçehre, and Aaron Courville. Recurrent batch normalization. In Proceedings of the 5th International Conference on Learning Represen- tations, 2017.
David Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4))1229- 1306, 2006. Publisher: IEEF
Weinan E. A Proposal on Machine Learning via Dynamical Systems. Communications
in Mathematics and Statisties, 5(1):1-11, 2017. URL httpps///oo..org110...0
34C304-017-0100
Weinan E, Chao Ma, and Lei Wu. Barron Spaces and the Compositional Function Spaces for Neural Network Models. arXiv: 1906.0809 [cs, math, stat], June 2019. URL httpatt/p/axxx.....
abs/1906. 08039
arXiv: 1906.08039.
Michael Figurnov, Maxwell D. Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially Adaptive Computation Time for Residual Networks. pp. 1039- 1048, 2017. URL http: hwwp:pccssppccccsss.ss www.cfe.ecccccc com content cat___vv___________ 20t___2111hhmm Fiqurnov_Spale ally wetpily_dppateeeeeeeeeeeeeeeeeee Crnpoolvee__mmpttttioo_oooo
2017 www.raaperr.__..... html
Samuel Gershman and Noah Goodman. Amortized inference in probabilistic reasoning. In Pro- ceedings of the annual meeting of the cognitive science society, volume 36, 2014. Issue: 36.
Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse, The Reversible Residual Network: Backpropagation Without Storing Activations. arriiv/707.000 [cs], 2017. URL arXiv: 770.004585.
04585
Alex Graves. Adaptive Computation Time for Recurrent Neural Networks. 1603.06603.0880 [cs], 08983 arXiv: 1603.08983.
February 2017. URL
Klaus Greff, Rupesh K. Srivastava, and Jürgen Schmidhuber. Highway and Residual Networks learn Unrolled Iterative Estimation. In Proceedings of the 5th International Conference on Learning Representations, 2017.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML 10, pp. 399-39994990 406, Madison, WI, USA, June 2010. Omnipress. ISBN 978- 1605588997777.
10

---

Under review as a conference paper at ICLR 2021
Eldad Haber and Lars Ruthotto. Stable Architectures for Deep Neural Networks. Inverse Problems, 34(1):014004, January 2018. ISSN 0266-5611, 1361-6420. doi: (0.1088/13311-6622/aaaaaaoo
httpthttp///tttt org/ at/aa/bs11055/ 03341 arXiv: 1705.03341.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Recti- fiers: Surpassing Human-Level Performance ImageNet Classification. pp. 1026-1034,
2015. URL https: www. cv foundat on org ottp///opeacccee//oooootooooooee
201 5t/5m//mm//.mmmmmmmmmmmmmmmmm Sirfe_Delviine wtwplliin__eeeitttt____t Ticice__ccv__000 2012__apppprrr____ html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645, Springer. 2016b.
John Hershey, Jonathan Le Roux, and Felix Weninger. Deep Unfolding: Model-Basee -Bsed Inspiration of Novel Deep Architectures. arXiv: 1:Xv:2440.2554 [cs, stat], November 2014. URL http://
wwi.vorg//as.... 01/./1409..25 2574 arXiv: 1409.2574.
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. PhD thesis, 1991.
Hanzhang Hu, Debadeepta Dey, Martial Hebert, and J. Andrew Bagnell. Learning Anytime Predic- tions in Neural Networks via Adaptive Loss Balancing. Proceedings of the AAAI Conference on Artificial Intelligence, (1001):3112-8821,1 July 2019. ISSN 2374-3468, doi: 10. 10.16///aaaii33000 33013812. URL https:://// //wp:/://wwwww/// aaa www.aaat.ojj.////aaoataaooooooot 4268
Number: 01.
Furong Huang, Jordan Ash, John Langford, and Robert Schapire. Learning Deep ResNet Blocks Se- quentially using Boosting Theory. In International Conference on Machine Learning, pp. 2058- 2067, 2018
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep Networks with Stochastic Depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, Lecture Notes in Computer Science, pp. 646-661, Cham, 2016. Springer International Publishing. ISBN
doi: 10. 11110079778-3-31--444993
Yujia Huang, James Gornet, Sihui Dai, Zhiding Yu, Tan Nguyen, Doris Y. Tsao, and Anima Anand- kumar. Neural Networks with Recurrent Generative Feedback. arXiv:2007.09220 00000000000000 [cs, stat], July 2020. URL /ttptt/pp/aaxii orq hbtpg//b/2200..0 200000
arxiv: 2007.09200.
Chou P Hung, Gabriel Kreiman, Tomaso Poggio, and James DiCarlo. Fast readout of object identity from macaque inferior temporal cortex. Science, 310(5749):863- 866, 2005. Publisher: American Association for the Advancement of Science.
ResNet Yerlan Idelbayev. Res Implementation htt.:p///gthoooo.coo///aaasssssr
Immpeeeeat for CIFAR1O/CIFARIO0 in PyTorch, 2018. URL
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03316 2015.
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio.
Residual Connections Encourage Iterative Inference. In International Conference on Learning Representations, 2018. URL
net www.mmriiioo SJJ -ttp://wwww..eeeeeee//////-----
Junkyung Kim, Drew Linsley, Kalpit Thakkar, and Thomas Serre. Disentangling neural mechanisms for perceptual grouping. In International Conference on Learning Representations, 2020. URL hittps://openrre wtt.eeewweeee/eeeeee htt.:/orrrmmll--ixxA44EE
Diederik Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
1rXiv:13/2.6114,22 2013.
Alex Krizhevsky, Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
11

---

Under review as a conference paper at ICLR 2021
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L Yamins, and James J DiCarlo. Brain-Like Object Recognition with
Shallow Recurrent ANNs. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' texeeuuutesen Alché-Buc, F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 12805-12816. Curran Associates, Inc., 2019. URL h wwpppprss.iiss.oo. nips. cc ccpp..ccppperr http:/::ppppppp..iiiieee
9441 brain- like object www.oeeiiiioo... with high performing shallow
recurrent-ann
pdf
Yann LeCun, Corinna Cortes, and C. J. Burges. MNIST handwritten digit database. 2010.
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint 160..03600 2016.
Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recurrent vision models. arXiv:2005. /136 [cs], May 2020a. URL http://aaxxv..ooo/////200533333
arXiv: 2005.11366
Drew Linsley, Junkyung Kim, Alekh Ashok, and Thomas Serre. Recurrent neural circuits for contour detection. In International Conference on Learning Representations, 2020b. URL
https://openrev//
net forum? forummidd-HlcBBRRVSS
Najib J Majaj. Ha Hong. Ethan A Solomon, and James J DiCarlo. Simple Iearned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition perfor- mance. Journal of Neuroscience, 35(39):13402-1344 2015. Publisher: Soc Neuroscience.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. International Conference on Learning Representations, 2018. htRL:tts:
net.eeewwweeeeeeeeeeee..ee.eeeee wttp://wwweeoettoeeeeeeee B1gg10 QRgziI
Tan Nguyen, Nhat Ho, Ankit Patel, Anima Anandkumar, Michael I. Jordan, and Richard G. Bara- niuk. A Bayesian Perspective Convolutional Neural Networks through Deconvolutional Gen- erative Model. arXiv: /. 02265 [cs, stat], December 2019. URL http:://axxiv..
abs/ 02657 XrXiv: 2811.02657.
Jeeheh Oh, Jiaxuan Wang, Shengpu Tang, Michael W. Sjoding, and Jenna Wiens. Relaxed Pa- rameter Sharing: Effectively Modeling Time-Varying Relationships Clinical Time-Series. In Machine Learning for Healthcare Conference, pp. 27-52, October 2019. URL http:
//proceedings. mlr Learning.
www.iings........ee.e.. wwl..press/viie wwt.sp/1l/s////.ss.1.. html ISSN: 1938-7228 Section: Machine
Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Emin Orhan and Xaq Pitkow. Skip Connections Eliminate Singularities. In International Confer- ence on Learning Representations, 2018. URL httpps://opereviieee net/ forum?rrmmmmm ?
HKwBEMWCZ
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmai- son, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py- Torch: An Imperative Style, PerfoPeefoemaacce Deep Learning Library. lach, H. Larochelle. A. Beygelzimer, F. d detetuuuueee Alché-Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL
Wal-
cc/paper/
9015 cat--pytrrccheee an -et--aan-iimmmee imperatiieeeeeee style high performaace ceciiceee-eeee-ee sleep-learnin Library.
pdf
Thomas Lin Pedersen. patchwork: The Composer of Plots. 2019. URL https://CRAN,
R proporooct wttp:///ppcaaagepaacccc..oo
Fernand Pineda. Generalization of back pro-paropngttion to recurrent neural networks. Physical
review letters, 59(19) 1987. Publisher: APS.
12

---

Under review as a conference paper at ICLR 2021
Zygmunt Pizlo. Perception viewed as inverse problem. Vision Research, :1
November 2001. ISSN 0042-6989. doi: 101000600002-0999000000003. URL http::/
www. www.sciencccieeetccoooooo
com/ htt.a//c......ooooooooco.ooaooa.
Tomaso Poggio. Vincent Torre, and Christof Koch. Computational vision and regularization theory.
Nature, 317(6035)3344-319
September 1985. ISSN 1476-4687. doi: 10.1038/317314a) URL
https: www.ww.tt........mm.m....wwwww.w http://wwww..aaarrc.cccooo/oaaaa Publishing Group.
Number: 6035 Publisher: Nature
Ofir Press, Noah A. Smith, and Omer Levy. Improving Transformer Models by Reordering their Sublayers. arXiv: 1911.03864 [cs], April 2020. URL http:::/aaxxiv..rg/abbbssss
03864 arXiv: 1911 190.648.
Core Team. R: A Language and Environment for Statistical Computing. Foundation for Statis- tical Computing, Vienna, Austria, 2019. htttp://www/ projerrrrjct.....o org/
Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: functional in- terpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79-87, January 1999. ISSN 1546-1726. doi: 10.1038/4580. URL httt::ttt///rtt
com/
articles/nnn199_1
Danilo Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Interna- tional Conference on Machine Learning, pp. 1530-1538, 2015.
Danilo Jimenez. Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference Deep Generative Models. In International Conference on Machine Learning, pp. 1278-1286, 2014.
Pieter R Roelfsema, H Steven Scholte, and Henk Spekreijse. Temporal constraints on the grouping of contour segments into spatially extended objects. Vision Research, 39(3):1509-152, 1999. Publisher: Elsevier.
Pedro Savarese and Michael Maire. Learning Implicitly Recurrent CNNs Through Parameter
Sharing. In International Conference on Learning Represemtations, 2019. URL https:
//openreview. aeee//oormmiid-uggyzuu
Andri Signorell. DescTools: Tools for Descriptive Statistics. 2020. URL https://crar.
t-project.oor//pee hwt.eett.orr/ppccageeeeotooooooo
Courtney J. Spoerer, Patrick McClure, and Nikolaus Kriegeskorte. Recurrent Convolutional Neural Networks: A Better Model of Biological Object Recognition. Frontiers in Psychology, 8, 2017. Publisher: Frontiers.
Courtney Spoerer, Tim C. Kietzmann, and Nikolaus Kriegeskorte. Recurrent networks can re- cycle neural resources to flexibly trade speed for accuracy in visual recognition. bioRxiv, pp. 677237, June 2019. doi: 10.1101/677237. URL httpt:://///t.... :/ww://wwwwwo http://wwwooxxvv.oo//oooeeeee wwt.orggccoon/////o/ooooo 10.1101/6 67723722
Publisher: Cold Spring Harbor Laboratory Section: New Results.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 1rXiv:1505.00387, 2015.
Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1-4414-1269-7.
Shaun P Vecera and Martha J Farah. Is visual image segmentation a bottom-up or an interactive process? Perception & Psychophysics, 5988) 1997. Publisher: Springer.
Andreas Veit, Michael Wilber, and Serge Belongie. Residual Networks Behave Like En- sembles of Relatively Shallow Networks. In D. D. Lee. M. Sugiyama, U. V. Luxburg, Guyon, and R. Gameet (eds.), Advances in Neural Information Processing Systems 29, Curran Associates, Inc., 2016. URL http:: ni cc/paper/ 2556-resiin networks org.r:/eeee..aaeeeeeeeeeeeeeeeer pdf
pp. 550-558
13

---

Under review as a conference paper at ICLR 2021
RV Mises and Hilda Pollaczek- Pollaczek-Geiringer. Geiringer. Praktische Verfahren der Gleichungsauflösung. ZAMM-Joumal of Applied Mathematics and Mecssnics/Zeitscrrfff für Angewandte Mathematik und Mechanik, 9(1):58-7) 77, 1929. Publisher: Wiley Online Library.
Haiguang Wen, Kuan Han, Junxing Shi, Yizhen Zhang, Eugenio Culurciello, and Zhongming Liu. Deep Predictive Coding Network for Object Recognition. In International Conference on Ma- chine Learning, pp. 5266-5275, 2018.
Paul Werbos. Generalization backpropagation with application a recurrent gas market model. Newral networks, (44):339335, 1988. Publisher: Elsevier.
Hadley Wickham, g8plo12: Elegant Graphics for Data Analysis. Springer- Verlag New York, 2016. ISBN 978-3-319-242772 URL
wtwpllotttiiiyee www.yiiiiseee.... org
Hadley Wickham and Lionel Henry. ridy:: Tidy Messy Data. 2019. URL https:
Hadley Wickham, Romain François. Lionel Henry, and Kirill Müller, dplyr: A Grammar of Data Manipulation. 2019. URL http:::tt///aatt R pro wttp::ttt...p.oaa.eeeeeeeeeeeeee
Scott Wisdom, Thomas Powers, James Pitton, and Les Atlas. Interpretable Recurrent Neural Net- works Using Sequential Sparse Recovery. arXiv: 1.0755 [cs, stat], November 2016. URL
httt:///axist www.oogg//.....o..o...o......../ alaaab/b////// /
07252
arXiv: 1611.07252.
Scott Wisdom, Thomas Powers, James Pitton, and Les Atlas. Deep recurrent NMF for speech separation by unfolding iterative thresholding. In 2017 IEEE Workshop on Applications of Sig- nal Processing to Audio and Acoustics (WASPAA), pp. 254-258, October 2017. doi: 1091 MAAPAAA.0017.817005 ISSN: 1947-1629.
Dean Wyatte, David Jilk, and Randall C O' Reillly. Early recurrent feedback facilitates visual object recognttin under challenging conditions. Frontiers in psychology, 5:674, 2014. Publisher: Frontiers.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: 10941, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. 1605v:1605.077 146 [cs], June 2017. URL/tttp//aaxxi/ wwtp://w.w.g/a///aaa/oo//oosss/ a/65 07146
arXiv: 1605.07 146.
Amir Zamir, Te-Lin Wu, Lin Sun, William B Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1308-1317, 2017.
A NETWORK TRAINING
A.1 SOFTWARR
We trained the networks using PyTorch (Paszke et al. 2019). PytorchLightning, numpy (Oliphant, 2006), and pandas in Python (Van Rossum Drake 2009). The remaining analysis was conducted in R (R Core Team 2019). using ggplor2 (Wickham 2016). dplyr (Wickham et al. 2019). tidyr
Wickham & Henry, 2019, i patchwork (Pedersen 2019), and DescTools (Signorell (020). The im-
plementation of ResNet- 104 was significantly supported by an existing implementation (Idelbayev)
2018).
A.2 DATASETS
We trained and evaluated the networks on four datasets: CIFAR- CIFAR-100 (Krizhevsky) 2009).
MNIST (LeCun et al. 2010), and Digitclutter (Spoerer al. 2017). All images were normalized before being provided to the network. For CIFAR-IO, we used a training/validation sample split of 45000/5000, augmented the data by random cropping and random horizontal flips during training. For MNIST. we used split of 00000/10000 and augmented the data by random cropping. Finally, for Digitclutter, we used between two and five overlapping digits (referred to as Digitclutter-3) with split of 100000/1 00000 without data augmentation.
14

---

Under review as a conference paper at ICLR 2021
Training data
Velidation data
Enror rate CIFAR -10
60%
Coupling parameter (Initaaiizaiin))
0 (non-rec.)
40%
0 (rec.)
0.5 (rec.)
(rec.)
20%
1 (rec.)
0%
200
100
Epoch
Figure 5: This figure depicts the evolution of mean error rate on training (left) and validation data (right) during training. This pertains to 16 channels in the first stage across all combinations of coupling parameter and initialization.
A.3 ARCHITECTURE DETAILS
As our base architecture we use a ResNet with the preactivation unit recommended by He et al. 2016b). This means that residual block consists of
BN1 - ReLU - Conv1 - BN₂ + ReL. + Conv2,
where BN stands for batch normalization (loffe & Szegedy 2015), ReLU for rectified linear unit, and Cony for a convolutional layer.
For CIFAR-1O, CIFAR-100 100,, and Digitclutter, this ResNet contained three stages with 16 blocks each. This is common depth for ResNet. For example He et al. (2016a) use ResNet-110. Two stages were connected by a downsampling block, which halved width and height of the representation and doubled the number of channels. The downsampling block consisted of a residual block where the first convolutional layer had stride 2 and doubled the number of channels. The shortcut connection (otherwise simply the identity function) decreased the image resolution by taking into account only every second column and row. To increase the number of channels, the shorteut connection added a number of layers initialized as zero. This downsampling block was never removed as part of the perturbations defining the indices of iterative convergence.
This means that for a standard ResNet with 16 channels in the first stage, the second stage consisted of (16x16) represeetttsst with 32 channels, and the third stage consisted of (8x8) represettttnt with 64 channels. In contrast, we only used one stage with 16 blocks for MNIST.
A.4 TRAINING DETAILS
We initialized our networks using Kaiming initialization (He et al. 2015) for the convolutional and linear weights. In the case of recurrent initialization, these were equal across residual blocks of stage. In the case of non-recurrent initialization, they were drawn independently for each block. We initialized the batch normalization's 's scale y with and its bias with
We trained the original networks for 400 epochs using gradient descent with momentum 0.9. For CIFAR- 10, we used an initial learning rate of 0.1, which was divided by after 200 and 300 epochs (see figure for the evolution of the learning rate of the ResNets with 16 channels in the first stage). For CIFAR-100 100 and Digitclutter, we trained the networks for 200 epochs and used an initial learning rate of 0. which was divided by 10 after 100 and 150 epochs. For MNIST, we used an initial learning rate of 0.025, which was divided by 2.5 after 200 and 300 epochs. For Digitclutter, we used initial learning rate of 0.05, which divided by after 200 and 300 epochs. We then identified the epoch at which the network had obtained the best validation classification error and used the corresponding model for all analyses.
15

---

Under review as a conference paper at ICLR 2021
B ALTERNATIVE INDICES OF ITERATIVE CONVERGENCE
We have defined our indices of iterative convergence as based on the accuracy with respect to the true image labels. This makes sense because computer vision models are usually evaluated on accuracy and we may not about perturbations which do not affect accuracy. Nevertheless, only studying the perturbations' effect on accuracy may lead us to miss certain phenomena. For example, an intermediate representation may have similar accuracy the unperturbed model, but make largely inconsistent predictions. This would be important to know. To address this limitation, we studied the accuracy of intermediate representations with respect to the predictions made by the unperturbed model instead of the ground truth. Fig. [6 depicts the indices of iterative convergence for the non- convergent ResNets trained on CIFAR-IO. As we see, this alternative definition does not change the conclusions we can draw from the experiment.
C PARAMETER EFFICIENCY
C.I EFFECTIVE PARAMETER COUNT
Softly coupled ResNets have the same number of raw parameters as ordinary ResNets with the same architecture. Their soft weight sharing, however, increases the similarity between weights of different blocks. This weight similarity is not reflected in the raw parameter count. We therefore introduce an effective parameter count (EPC). For this measure, three quantities are of interest: the parameters that are not coupled with any other parameters, W Wuncoupled: the mean parameters
for a given stage 8, (*) and the normalized deviations from this mean of a parameters within particular layer, ((1) == (W(())
(+)). As measure of the effective number of parameters which is sensitive to different degrees of weight sharing, we now propose
s
EPC(W) W wncooppedd/
lio
+ ΣΣ
D 9 11,
where the To-morm || corresponds to the parameter count, and S and T. to the number of stages and blocks, respectively. This allows us to interpolate between the raw parameter counts of non- recurrent and recurrent ResNet. For softly coupled ResNets, the EPC measure lies between the raw parameter count of non-recurrent and recurrent ResNets as a function of the weight similarity across weights (see Fig. -
This definition also allows us to examine the effect of gradient coupling on the average deviation of the parameters from the mean. As expected higher coupling parameter lead lower average deviation and = leads to no deviation at all (see Fig. 7c).
C.2 STRONGER GRADIENT COUPLING DOES NOT INCREASE PARAMETER EFFICIENCY
We hypothesized that ResNets may employ implicitly recurrent computations. This would mean that there is a more recurrent network which is more parameter-efficien. If this is the case, soft gradient coupling might be able to find such solution. For given number of parameters, however, recurrent and softly gradient-coupled networks do not outperform ordinary ResNets (see Figure B left). Softly coupled networks even tend do worse. This because pure parameter counts do not take into account the soft weight sharing across blocks. We therefore developed a measure of the effective parameter count which takes the similarity of weights across layers into account (see Section C.I). Figure [Zc. right, shows that for a given effective parameter count, ordinary, softly gradient-coupled, and recurrent ResNets all perform equally well. Nonetheless, this suggests that we cannot increase parameter efficiency by making the ResNets more recurrent.
D TRAINING VARIATIONS
On CIFAR- 10, we explored several variations the training algorithm, which were consistent with the findings presented above.
16

---

Under review as a conference paper at ICLR 2021
# channels
Coupling parameter (Initialization)
(non-rec.)
(roc.)
0.5 troc.)
0.9 (rec.)
1 (rec.)
Convergence Index
Recurrence Index
Divergence Index
1.10
0.7
AUC
1.05
0.6
1.00
0.4
(non-rec..(rec.)
(rec.) (rec.)
(non-rec.) (rec.)
(rec.)
Inon-roc.) (rec.)
Inec.)
(rec.)
Coupling parameter (Initialization)
Stage
Stage
Stage
10.0
3.0
1.0
0.3
1e-00
1e-02
Tucldenn
10-04
10
10
# #evaluated blocks
Stage
Stage
Stage
100
Suctecon Crntsentroyy
10
1e-02
1e-02
1e-04
#additional evaluations
Figure 6: a The indices of iterative convergence as based on accuracy with respect the predictions made by the unperturbed model rather than the ground truth. b,c The evolution of crossentropy and the Euclidean distance between intermediate representations and the final representation across the number of evaluated blocks and additional evaluations.
17

---

Under review as a conference paper at ICLR 2021
@ rra parameters
effective parameters
Enar rob on CIFAR-3 R-
7.0%
5.0%
Coupling parameter
coon-rec
101
10'
10'
(recc
0.5 (rec.]
#offectie parametess
Deviadon per parerreter
0.8 (rec.]
(reC)
0.4
0.2
10f
10'
0
0.25
0.5
0.9
(nec.)
(nec.
rrw parameters
Coupling parameter (Initaaiiztion)
Figure 7: This figure plots the effective parameter count of the networks studied in the main text against the raw parameter count.
D.I RECURRENT BATCH NORMALIZATION
Cooijmans al. (2017) proposed initialize the scale of the batch normalization 0.1 in recur- rent neural networks. When applying this method to softly gradient-coupled networks, we find that the method strongly improves performance for À 0.9, but not for any other coupling parameters (see figure (1).
D.2 TRIANGULAR GRADIENT COUPLING
We may generalize soft gradient coupling to the coupling rule
T
A
Σ
((s,;))00 T.
(5)
where x is some kernel depending on the coupling parameter A. Ordinary soft gradient coupling can be recovered with a uniform kernel
x(s t; A) = (1 A) A)+ A/T
= 1
(6)
W/T
Alternatively, we may want to couple adjacent blocks more strongly than blocks that are far away from each other. This can, for example, be achieved with a triangular kernel
A(s,t:)) = (1-2. 18- +)+ (1-11/6 x T) (s-
if 2 0.5, if A < 0.5,
(7)
where
(+) = max(-;0)
Again, 0 corresponds to an ordinary ResNet and = corresponds to fully recurrent ResNet, whereas intermediate values for À smoothly interpolate between the two.
18

---

Under review as a conference paper at ICLR 2021
# channels
Error CIFAR-1D
15.0%
16
10.0%
64
5.0%
Batzhnorm in/tialization
0.0%
gamma-1
(non-rec.) (rec.)
(rec.)
0.5
trec.
0.9
(rec.]
(non-rec.) (rec.)
trec.)
(rec.)
(recil
gamma-0.1
Coupling parameter (Initialization)
Training data
Va Eddatiinn data
Kernel type
ralte CIFAR -10
uniform
triangular
5.0%
channels
2.5%
32
0.0%
0.25
0.5
0.9
1
0
0
0.25
0.5 (mc.))
0.9 (roc.)
(rcc.)
(roc.)
(mc.)
(rec.)
Inon-rer. (noc..
(roc.)
(roc.)
Coupling parameter (Initialization)
# channels
Training data
Validation data
Error CIFAR -10
10.0%
7.5%
5.0%
2.5%
Batchnorm initialization
0.0%
gamman1
(rec.)
frec.)
Irec.)
gamma-001
Coupling parameter (Initialization))
Figure 8: Performance training variations on CIFAR-10. a The effect initializing batchnorm with y 0.1 instead of 1. b The effect of using triangular kernel for gradient coupling instead of uniform kernel. e variation of gradient coupling where the first five blocks in each stage were uncoupled.
19

---

Under review as a conference paper at ICLR 2021
Enor rate on CIFAR-30
10.0%
Coupling parameter (Initialization)
O (non-rec.)
(rec.)
0.25 (rec.)
(rec.)
0.9 (mc.)
(rec.)
5.0%
10a
10
10a
10'
raw parameters
effective parameters
Figure 9: This figure plots the error rate of all models trained on CIFAR-10 against their raw and effective parameter count.
Figure Sp shows the effect of triangular compared uniform gradient coupling. For given coupling parameter, this method appears to outperform the uniformly gradient-coupled network, but neither method outperforms an ordinary ResNet.
D.3 START COUPLING AT LATER LAYERS
Finally, we uncoupled the first five residual blocks and only constrained the remaining blocks. For the fully recurrent case, this corresponds the implementation by Jastrzebski et al. (2018). We also studied the effect of this late coupling together with batchnorm initialization of 7 0.1. In both cases, the late coupling means that the coupled networks are better than coupled networks which start coupling at block 0, but still worse than an ordinary ResNet.
D.4 PARAMETER EFFICIENCY
Figure 9 shows that across all these extended networks, the relationship between effective parameter count and performance is also largely unaffected by coupling parameter.
D.5 ITERATIVE CONVERGENCE INDICES
Figure 10 demonstrates that for the extended training variations, higher coupling parameters gen- erally tend to increase the Convergence Index and decrease the Divergence Index, well. The extended training variations also predominantly yield networks with Recurrence Index around 1.
E DETAILS ON THE SPECTRAL NORMALIZATION
E.1
UPPER BOUND ON THE LIPSCHITZ CONSTANT
To determine an upper bound on Lipschitz constant of f, we use an alternative characterization based the Jacobian (æ). The spectral norm ||Al|z of matrix A is defined its maximal singular value. The Lipschitz constant then given by
L(f) max (x) x
According to the chain rule,
20

---

Under review as a conference paper at ICLR 2021
Batchnorm initialization
gamma-1 gamma-0.1
# #channels - 8 - 16 - 32
Convergence
Recurrence
Divergence Index
1.00
ICome type: unfromm
Soutt cougting a
0.75
0.50
Start coupling at:0 Koreet type trbingufer
1.0
AUC
1.0
Start couping Ktrnel typer Uyperuuttrrn untorm
0.5
-0.5
Coupling parameter
Figure 10: This figure plots the indices of iterative convergence against the coupling parameters for the different training variations.
1y(x) = JCony. (22) -/BN,oConvi -JRLLU(Z)) JBN,
Here 21 and 22 given by the representation at the appropriate intermediate stages of the residual function, i. c. 21 is the representation after BN, and Ø2 is the representation after BN2. The Jacobians of the batch normalizations and convolutions do not depend the input as these linear operations. Since JROLU and JBN, are both diagonal matrices, they commute and therefore,
(x(x) = JCanv, -JRLLU (22): JBN,oConvi .BN, JneLu(E1)
Joanvz JROLU (22) JnNyoConv,oBN, I
We have therefore split the Jj into a product of two constant functions and two Jacobian of the rectified linear unit.
The spectral norm || ll2 is known to sub-multiplicative. This means that for matrices and B, |BAl]2 & ||B])|||A||| Moreover, JRELU, depending on whether its input is positive negative is given by diagonal matrix with ones or zeros on the diagonal. Its singular values are therefore at most 1. Putting this together, we can upper bound the Lipschitz constant as
L(f) max [Jyy())
I max [JConvzll2l| oasss ll2
z
< || 2
Ca JNN |l2 i(f).
Theoretically, we could determine the maximal singular value of the convolution using a singular value decomposition. However, the singular value decomposition of such large matrix is compu- tationally expensive. Instead, Yoshida & Miyato (2017) lay out how the maximal singular value can be approximated using power iteration (von Mises & Pollaczek-Geiin Geieiingerrrrrrr 1929). The only differ- ence to their method consists in the fact that our first convolution additionally involves multiplying the input by the diagonal matrices given by the two batch normalizations.
21

---

Under review as a conference paper at ICLR 2021
channels +
8
+ +
16
+
22
Convergence
Divergence
1,000
0.006
1,010
AUC
3.995
0.004
1.005
0.002
1,000
0.000
(ncn-ncc.rec.)
(mc.)
Inon mnnn-rec_mmc..
(non-roc.)rec.)
Coupling parameter (Initialiiation)
Figure 11: This figures plots the indices of iterative convergence for the properly convergent ResNets trained on CIFAR-10.
rait: on CIFAR-10
12.00%
channels
11.00%
10.00%
16
32
9.00%
8.00%
Inon-roc. [nec.] (rec.) (roc.)
Coupling parameter (Initialiiation)
Figure 12: This figures depicts the performance of properly convergent ResNets on CIFAR-10.
E.2 RESULTS ON PROPERLY CONVERGENT RESNETS
Fig. demonstrates that the properly convergent ResNets indeed converge. This is guaranteed for the recurrent ResNet, but empirically we see that is also the case for non-recurrent properly convergent ResNets. Fig. T2 demonstrates that these ResNets tend perform bit worse than the improperly convergent ResNets we studied in the main article.
F RESULTS ON DIGITCLUTTER, MNIST, CIFAR-100 AND SAMPLE EFFICIENCY
Fig. depicts the performance of the gradient-coupled networks on Digitclutter, CIFAR-100 100, MNIST, and CIFAR-10 trained with fewer samples. In none of these case, higher gradient coupling parameters systematically improve performance.
22

---

Under review as a conference paper at ICLR 2021
channels
64
Two dig 1100,000 samples)
Four digits (100.000 samples]
Two digits (2048 samples)
4.00%
0.240%
14.0%
3.60%
0.200%
10.0%
3.20%
0.160%
rale
8.0%
2,80%
Three digis (100 000 samples)
Fire digits (100,000 samples)
Four digits (8192 samples)
4.50%
46.0%
4.00%
21.0%
44.0% 04.0%
18.0%
42.0%
40.0% 0%
2.00%
12.0%
0.5
0.9
0
0.5
0.9
0
0.5
0.9
kec. (rec.) (nec.) (rec.)
(nonwrac drac. (rec.)
(rec.) (rec.)
[non-rec (rec.) (fec.) (fec.) (rac.)
Coupling parameter (Initialization)
CIFAR-100
CIFAR -10 (2048 samples)
MNIST
5.500%
40.0%
0.400%
Error
30.0%
30.0%
0.350%
26.0%
300%
0.5
Inon-rec(ec..) (rec.) (rec) (rec.)
Inon-reciee.. (rec.) (rec.) (rec.)
(non-tec(ee.) (rec.) (rec.) (rec.)
Coupling parameter (Initialization)
Coupling parameter (Initializatinn)
Coupling parameter (Initialization
Performance of gradient- ResNets on variations of Digitclutter with different number of overlapping digits and different size of training data. b Performance of gradient-coupled ResNets on CIFAR-100 100, CIFAR-11 10 with few training data, and MIST.
Figure 13:
23


Reviews:
[{"review_id": "vGVFGJbwqw9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "> Summary: This paper investigates the relationship between deep ResNets and (implicitly) iterative computations. The authors introduce two main hypotheses that are at the core of the investigation: 1) whether the iterative inductive bias improves ResNet performance; and 2) whether recurrent ResNets are more parameter-efficient. The paper also proposes three metrics for studying the convergence and divergence behaviors of these networks in order to investigate this matter.\n\n----------------\n\nPost-rebuttal thoughts:\n\nI would like to thank the authors for their detailed response and the revisions made to the paper. I'm updating my score to 5 as part of my concerns are satisfactorily addressed, and I wished I could have more opportunities to discuss with the authors on their response. In general, my opinion is that the authors have introduced too many \"artificial\" components to the study (e.g., soft gradient coupling, the convergence/divergence indices) that make me slightly dubious of how generalizable this characterization is. For example, as the authors indicated, spectral normalization creates a different phenomenon (at a cost of worse performance), but with no change to the structure itself (so unlike the soft gradient coupling), a different phenomenon could be challenging the conclusion of the paper.\n\nMy suggestion would be that the authors delve deeper into the observations here and better integrate the revisions with their original approach (e.g., the high-dimensional discussion; the spectral normalization discussion, etc.)\n\n----------------\n  \n- I feel that in the rebuttal phase the authors made certain important new edits to the paper (e.g., \n\nMy general opinion is that this paper investigates an interesting direction on the learning behavior of ResNets, but is still not quite ready for publication in a venue like ICLR. There is an obvious gap in related work (see my detailed comment below) on implicit deep networks; moreover, the definition of the various indices (e.g., convergence index) is also rather confusing to me. The empirical results are not strong enough evidence, in my view, to make most of the claims conclusively. I also have some doubts on the motivation for the methodology that the authors are using. \n\nPros:\n\n1. Interesting direction; as the author shows, the ResNet architecture itself is expressive enough for implementing iterative computations/algorithms. So it is worthwhile to study its behavior along this trail.\n2. The paper is overall written in a clear manner and the author explained their methodology well.\n\nCons:\n\n1. Many arguments are too hand-wavy and I don't particularly find the metrics the authors define to analyze convergence/divergence particularly convincing. (See my comment below)\n2. The experimental setup is mostly on small scales.\n3. Even with the small scale setups, the experimental results don't seem conclusive enough (at least to me) to draw the conclusion that the authors were trying to claim. The verification of hypothesis 2 is especially hasty.\n4. The motivation behind the soft gradient coupling is not clear to me.\n5. There is a clear missing gap in the related work that I think the authors should pay attention to.\n\n--------------------------------------\n\nI will expand on some of the Cons above, and provide the following detailed comments/questions:\n\n1. Again, I think it is interesting to investigate the relationship between ResNets and iterative computations. But besides the canonical, plain unrolling of the layers that the authors have looked at, *implicit models* (i.e., models that study the continuous dynamics of a layer $f$) like Neural ODEs [1] and Deep Equilibrium Models [2] (there's a ResNet version of it) are both also looking at compact recurrent networks. In particular, the deep equilibrium models especially targets the convergence (i.e., the \"fixed point\" of the layer), and seems to demonstrate state-of-the-art level performances. In contrast to what the authors provided in the last paragraph of Section 1, I would therefore argue (based on Neural ODEs and deep equilibrium nets) that recurrence does offer some notable advantages like constant memory cost and analytical gradients. The other related thread of work is simply the classical recurrent backprop (RBP) theories, which study the convergence of recurrent networks and how one can leverage such property for the backward pass of these networks. I found the current version of the paper did not discuss either aspect of this, which I believe is important literature that actually is on the opposite side (partially) of what the authors are trying to claim. \n\n2. There are actually many ways that I can think of to make recurrent residual blocks converge when you infinitely repeat it. For example, with spectral normalization [3], we can simply make the Jacobian of the block have an operator norm $<1$. Then Banach fixed point theorem will guarantee convergence. Other methods are also possible (e.g., via a provably convergent optimization perspective). These are not discussed in the paper (nor are they the main focus, I guess), but this doesn't mean that ResNets do not converge in general. The authors argue that \"some balance between feedforward and iterative computations might have been learned by the ResNets\", but there is actually a lot of noise in the analysis... for example, the networks could be overfitting, etc. The point is, as long as you regularize the model in that direction, the ResNets could still converge.\n\n3. One main problem that I found about this paper is its definition of the convergence/divergence indices. The \"convergence\" concept in this paper is constrained to look at the accuracy convergence, by which the authors look at the inverse of the AUC of the classification rate curve. But given the nature of softmax and classification task itself, I don't think a convergence in accuracy is a good \"index\" for measuring convergence of an architecture, which Section 3.1 looks at (for $\\hat{z}_i^{(t)}$). For example, softmax is constant up to a shift of constant. And for classification of, let's say 10 objects $(x_1, \\dots, x_{10})$, getting $x_1, \\dots, x_5$ correct is still different from getting $x_6, \\dots, x_{10}$ correct, even though they both have \"50% accuracy\". The paper investigates CIFAR-10, where one can achieve >94% accuracy, but in cases like ImageNet where 70% accuracy is normal, these two 50% are certainly non-convergent to me. Also, I'm assuming the entire Figure 1 is on the simple 2-dimensional linear task? Does the phenomenon in Figure 1d repeat in high dimensionality? If so, what does it look like? (My experience with this suggests that if you keep stacking the same block, the activations will eventually oscillate, if not converge, but it could differ by initialization.)\n\n4. Some arguments are also a bit handwavy to me and I'd appreciate if the authors can expand on them. For example, in Section 3.2, the paper claims \"in contrast, the skip connections encourage a ResNet to use the same representational format across blocks... [and] are therefore better aligned with the final decoder\". As another example, the paper claims ResNets learn a balance between \"feedforward and iterative computations\". These are all intuitively reasonable arguments indeed, but considering that this is an empirical study paper, I think actually verifying these would make the paper stronger.\n\n5. About the soft gradient coupling, doesn't this simply mix the gradients and inject more stochasticity to them? In general, would you expect (when $0 < \\lambda < 1$) that just like in typical SGD, this stochasticity will be averaged out by the optimization procedure of deep networks? Since the $\\tilde{\\Delta}_t$ no longer fully reflect the mini-batch gradient descent direction, have you checked how the block parameters within the same stage gradually deviate from one another as you optimize the network (e.g., how does the standard deviation of $\\mathbf{W}_l^{(s)}$ over all layer $l$'s in the same $s$ change over training iterations? Do they deviate or stay around? If these weights are eventually still different, why can one still consider them to be \"similar\" (other than the RI metric, which I find to be a debatable metric given the #3 above...)? \n\n6. For the EPC, have you computed the EPC of an ordinary ResNet and a purely recurrent ResNet? How do their EPC look like when compared to the soft gradient coupled ResNets (e.g., $\\lambda=0.5$)?\n\n7. In Section 5.3, the paper claims that \"if this is the case, we would expect soft gradient coupling to find such a solution.\" Why? And isn't a soft gradient coupled ResNet still a non-recurrent ResNet (in the sense that you can't simply unroll a single layer to get the output; you still need to store all parameters of the network, rather than only a single layer of it)?\n\n--------------------------------------------\n\nI look forward to the authors' response on my questions/comments above. I'm happy to consider adjusting my score accordingly.\n\n\n[1] https://arxiv.org/abs/1806.07366\n[2] https://arxiv.org/abs/1909.01377\n[3] https://arxiv.org/abs/1802.05957", "extra_scores": {}, "extra_text": {}}, {"review_id": "aHAvnTpjGpA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n\nThis paper investigates the extent to which the computations implemented by an optimized ResNet resemble those of a recurrent network. They develop new tools to study this question, and find evidence that ResNet performance is *hurt* when it is forced to act like a specific kind of RNN.\n\nStrengths:\n\nI am excited that the authors are re-examining the assumptions of now classic work likening RNNs to ResNets. They develop elegant tools for continuously transforming between the two, and compare performance on several image classification tasks.\n\nWeaknesses:\n\nI'm not totally convinced by the author's pitch for inverse problems. This idea of the visual system acting as a generative model is still far from worked out, and I'd prefer the authors hedge their language on it. There's also other tasks that are more closely linked to recurrent processing than the ones studied here. For example, I recommend the authors experiment with Pathfinder or cABC of [1], which are solved in fewer samples by recurrent networks vs. feedforward models. This would allow you to plot the amount of gradient coupling on one axis, and the number of samples need to solve (e.g.) Pathfinder on the other axis, which I think would be very elegant.\n\nI appreciate the authors laying out their hypotheses like they did, but I found the language to be indirect. Is there a simpler way to motivate these hypotheses?\n\nFigure 1 is difficult to understand. Are you plotting activities against each other? What do the different dots represent for the feedforward model? Is the interaction in (d) meaningful or is this just by happenstance, and the divergence from x is the meaningful quality. \n\nWhy ResNet-104?\n\nWhen dropping ResNet blocks, how do you deal with the subsampling between layers? When you drop out the first layer do you also drop out the max pooling at that layer? Is there any chance that these distinctions in computations and resolutions between blocks that you're dropping could bias your observed results?\n\nRegarding the Divergence index, the authors should review [2]. I don't think a high divergence index necessarily means that the ResNet isn't learning the function of an RNN — only that the learned function is not stable, which makes sense given ResNet hyperparams. This paper suggests that if you change the model nonlinearities to globally contractive ones like tanh or sigmoid (or use their algorithm) you'll control this problem.\n\nThe gradient coupling is forcing a fixed combination between the gradients of successive layers. But gated RNNs are standard for recurrent vision models, and these do not have such a constraint. Is it possible that the ResNet without shared weights is learning a the function of a gated RNN rather than the vanilla RNNs that you're comparing to here?\n\n\"Our findings also suggest, however, that deep feedforward computations may not be characterized as iterative refinement on a latent representation, but, at most, as non-iterative refinement on this representation.\" How do you refine non-iteratively/incrementally? More generally, I felt like the authors overloaded \"iterative\" and the manuscript would benefit from a more careful treatment of the exact computations they're /referring to. Give concrete examples.\n\nAre you comparing ResNets to RNNs or iterative algorithms (as are alluded to in the intro)? I am confused by the motivation here, which changes from paragraph to paragraph.\n\n[1] Kim et al. Disentangling neural mechanisms for perceptual grouping. ICLR 2020.\n[2] Linsley et al. Stable and expressive recurrent vision models. NeurIPS 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RZZc865Tn9v", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an empirical study that characterizes and quantifies the implicitly recurrent nature of residual networks (ResNets). A ResNet can be construed as a general formulation of a recurrent neural network (RNN) unfolded for a fixed number of time steps. In particular, the authors propose \"soft gradient coupling,\" a novel way to control the degree of weight-sharing between the different residual blocks. This gives them the ability to smoothly interpolate between a \"no weight sharing\" scenario to a \"full weight sharing scenario.\" Soft gradient coupling imparts the ability to share similar \"computations\" without necessarily sharing the same weights. They introduce metrics such as convergence, recurrence, divergence indices, and an effective parameter count to quantify \"iterative\" behavior numerically. Finally, they also test the impact of \"iterative\" computations in a ResNet trained for non-trivial visual recognition tasks.\n\nPros:\nThe problem that the authors tackle is undoubtedly interesting and useful. This is particularly true in light of a growing literature analyzing ResNets as discretized dynamical systems. The demonstration that ResNets can express iterative algorithms but do not learn such algorithms by default is intuitive and powerful. The authors use a simple toy example (a linear function) to articulate the desiderata and then work with larger datasets.\n\nThough the manipulations to test iterative computations and implicit recurrence (early read out to determine convergence; residual block drop outs to determine recurrence; and repeated application of residual blocks to determine divergence) are not entirely novel, they are applied quite aptly. The most salient observation is that of divergence and is reminiscent of stability analysis of RNNs for vision. The notion that ResNets learn to tradeoff (and balance) feedforward vs. iterative computations is an interesting proposal. \n\nThe proposed \"soft gradient coupling\" scheme allows for different residual blocks to implement similar \"computations\" without necessarily sharing weights or changing the primary optimization problem. This is an interesting suggestion. \n\nThis paper also presents fairly extensive numerical experiments.\n\nCons:\nThe strong conclusion that ResNets do not benefit from recurrence regularization is premature, given the current set of experiments presented in this manuscript. As the authors themselves point out, \"iterative computation\" is an inductive bias. However, there is little reason to believe that this inductive bias is the right one for a classification problem. Have the authors tried to consider problems other than image classification? For instance, there has been recent literature on the relevance of iterative computation (visual routines) for contour detection and segmentation problems. Moreover, \"accuracy\" is not the only way to quantify the benefit. Have the authors tried to measure sample efficiency? i.e., can a ResNet employing iterative computations learn from fewer training samples than a non-iterative ResNet?\n\nHow does the performance benchmarking of a \"fully recurrent\" ResNet compare to a comparable-sized LSTM/GRU trained on this task? Or even a weight-shared ResNet? These comparisons seem to be necessary to discern if the soft gradient coupling is introducing other artificial biases.\n\nIt is unclear why the authors believed that a high degree of soft-gradient coupling would help with the divergence issue in the first place. Implementing the same computation repeatedly only converges (and stays there) given certain other properties of the transformation function applied (for instance, the spectral radius of each Residual block's Jacobian). There is quite a bit of theoretical/empirical work in the literature in this regard.\n\nThe manuscript would benefit from some discussion on theoretical results from the RNN literature that outline necessary and sufficient conditions for the forward pass of RNNs to behave like convergent dynamical systems. Given this paper's focus on iterations, convergence, and divergence, this body of work seems relevant.\n\nMinor:\n(Fig. 3a) The recurrence index was normalized, yet there are points with a recurrence index greater than 1. Is there any explanation for this?\n\nThe recurrence index measure also does not seem to add much value. (Fig 2a,b; second to left panel)\n\nAre the values reported in Table 1., for example, point estimates? Did the authors estimate some confidence intervals on these values by running a few repeats of the experiments?\n\nClarity: (Pg. 2) \"Encouraging iterative behavior in this way therefore does not improve the inductive bias\": Is not iterative behavior *the* inductive bias?\n\n(Fig. 3a) There must be a discussion on the non-monotonicity of these curves (especially convergence/divergence).\n\nThe paper can do with a through reformatting of the reference list to make all entries consistent in citation style (for ex: including URLs, DOIs, proper and consistent journal/conference abbreviations, etc.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "JFD0ijnlZ03", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Paper Summary\n\nThis paper studies the correspondence between residual networks and iterative algorithms that repeat computations and converge to a solution. The authors suggest that residual networks can in principle implement such iterative algorithms and experimentally show that networks trained in practice do not naturally learn them. They also define three indices to quantify the degree to which a ResNet shows properties of iterative algorithms. Finally, they show that while soft gradient coupling across layers within stages can ensure that learned ResNets behave more like iterative algorithms, this does not appear to provide a useful inductive bias for image classification tasks.\n\n## Strengths\n\nStudying the nature of programs that are learned by neural networks of various architectures is an interesting and important research problem. This paper makes a contribution to it by examining the extent to which ResNets implement algorithms similar to iterative solvers.\n\nThe authors define numerical indices to formalize the criteria for \"iterative-ness\" that they are looking for, which are useful for comparisons.\n\nThe paper contains a negative result about the utility of forcing iterative behavior on ResNets using the proposed gradient coupling trick. This negative result may be useful to researchers interested in similar ideas in the future.\n\n## Weaknesses\n\nThe motivation for this paper is somewhat weak, or at least weakly justified. The authors define an iterative method/algorithm as one that uses repeated iterations and convergences to a solution. It is then hypothesized that such behavior might be a good inductive bias for neural networks, but it is not discussed why this might be expected. After all, iterative algorithms are designed to converge after a (non-fixed) number of steps, and neural nets are not. I think that either the motivation should be justified better, or it is simply a question without a strong motivation (this doesn't necessarily make it unimportant, just less important).\n\nMoreover, if we'd like the outputs of the neural network to be \"stable\" for reasons other than metrics like accuracy, there are certain methods and lines of research on this subject, such as Ciccone et al. cited by the authors. Those works already claim that computations learned by ResNets are not stable, and suggest methods to make them so. Doesn't that make the question investigated (whether ResNets learn iterative convergent behavior) in this paper somewhat redundant?\n\nFinally, the negative experimental results are interesting but I think they need to be stronger to be convince a reader that this is a result that can be expected to generalize. Due to the simplicity of the datasets (and no confidence intervals on the numbers in Table 1), evidence for the negative impact of encouraging iterative convergent behavior on performance is still preliminary. More datasets and tasks of higher complexity will certainly help here.\n\n## Review Summary\n\nSince the paper lacks strong motivations, clear significance and highly convincing results, I am currently unable to recommend an acceptance. If the authors can elaborate on their motivations and discuss them in light of related work as mentioned above, I am willing to reconsider my score. \n\n## After Author Response\n\nI think the changes to the paper have improved it. In particular, the reference to Spoerer et al. gives more weight to the motivations of the paper. I'm increasing my score slightly as a result.\nHowever, the relation to prior work remains hazy. The model of Ciccone et al., for example, has shared weights, stability over increased depth, and still performs as well as ResNets generally. Doesn't this go against the conclusions of this paper?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work provides evidence against the hypothesis that ResNets implement iterative inference, or that iterative convergent computation is a good inductive bias to have in these models. The reviewers indicate that they think this hypothesis is interesting and relevant to the ICLR community, but they do not find the current work sufficiently convincing. Both theoretically and experimentally the paper does not fully demonstrate the claim that iterative inference is not useful in ResNets, and the reviewers are unanimous in their recommendation to reject the paper until the evidence for this claim is strengthened.

Decision: Reject