The Uncanny Similarity of Recurrence and Depth

1 INTRODUCTION . It is well-known that adding depth to neural architectures can often enhance performance on hard problems ( He et al. , 2016 ; Huang et al. , 2017 ) . State-of-the-art models contain hundreds of distinct layers ( Tan & Le , 2019 ; Brock et al. , 2021 ) . However , it is not obvious that recurrent networks can experience performance boosts by conducting additional iterations , since this does not add any parameters . On the other hand , increasing the number of iterations allows the network to engage in additional processing . In this work , we refer to the number of sequential layers ( possible not distinct ) as effective depth . While it might seem that the addition of new parameters is a key factor in the behavior of deep architectures , we show that recurrent networks can exhibit improved behavior , closely mirroring that of deep feed-forward models , simply by adding recurrence iterations ( to increase the effective depth ) and no new parameters at all . In addition to the success of very deep networks , a number of works propose plausible concrete benefits of deep models . In the generalization literature , the sharp-flat hypothesis submits that networks with more parameters , often those which are very wide or deep , are biased towards flat minima of the loss landscape which are thought to coincide with better generalization ( Keskar et al. , 2016 ; Huang et al. , 2020 ; Foret et al. , 2020 ) . Feed-forward networks are also widely believed to have layer specialization . That is , the layers in a feed-forward network are thought to have distinct convolutional filters that build on top of one another to sequentially extract hierarchical features ( Olah et al. , 2017 ; Geirhos et al. , 2018 ) . For example , the filters in shallow layers of image classifiers may be finely tuned to detect edges and textures while later filters may be precisely tailored for detecting semantic structures such as noses and eyes ( Yosinski et al. , 2015 ) . In contrast , the filters in early and deep iterations of recurrent networks have the very same parameters . Despite their lack of layer-wise specialization and despite the fact that increasing the number of recurrences does not increase the parameter count , we find that recurrent networks can emulate both the generalization performance and the hierarchical structure of deep feed-forward networks , and the relationship between depth and performance is similar regardless of whether recurrence or distinct layers are used . Our contributions can be summarized as follows : • We show that image classification accuracy changes with depth in remarkably similar ways for both recurrent and feed-forward networks . • We introduce datasets of mazes , a sequential reasoning task , on which we show the same boosts in performance by adding depth to recurrent and to feed-forward models . • With optimization based feature visualizations , we show that recurrent and feed-forward networks extract the very same types of features at different depths . 1.1 RELATED WORKS . Recurrent networks are typically used to handle sequential inputs of arbitrary length , for example in text classification , stock price prediction , and protein structure prediction ( Lai et al. , 2015 ; Borovkova & Tsiamas , 2019 ; Goldblum et al. , 2020 ; Baldi & Pollastri , 2003 ) . We instead study the use of recurrent layers in place of sequential layers in convolutional networks and on non-sequential inputs , such as images . Prior work on recurrent layers , or equivalently depth-wise weight sharing , also includes studies on image segmentation and classification as well as sequence-based tasks . For example , Pinheiro & Collobert ( 2014 ) develop fast and parameter-efficient methods for segmentation using recurrent convolutional layers . Alom et al . ( 2018 ) use similar techniques for medical imaging . Also , recurrent layers have been used to improve performance on image classification benchmarks ( Liang & Hu , 2015 ) . Recent work developing transformer models for image classification shows that weight sharing can reduce the size of otherwise large models without incurring a large performance decrease ( Jaegle et al. , 2021 ) . Additionally , the architectures we study are related to weight tying . Eigen et al . ( 2013 ) explore very similar dynamics in CNNs , but they do not address MLPs or ResNets , and Boulch ( 2017 ) study weight sharing only in ResNets . More importantly , neither of those works have analysis beyond accuracy metrics , whereas we delve into what exactly is happening at various depths with feature visiualization , linear separability probes , and feature space similarity metrics . Bai et al . ( 2018 ) propose similar weight tying for sequences , and they further extend this work to include equilibrium models , where infinite depth networks are found with root-finding techniques ( Bai et al. , 2018 ; 2019 ) . Equilibrium models and neural ODEs make use of the fact that hidden states in a recurrent network can converge to a fixed point , training models with differential equation solvers or root finding algorithms ( Chen et al. , 2018 ; Bai et al. , 2019 ) . Prior work towards understanding the relationship between residual and highway networks and recurrent architectures reveals phenomena related to our study . Greff et al . ( 2016 ) present a view of highway networks wherein they iteratively refine representations . The recurrent structures we study could easily be thought of in the same way , however they are not addressed in that work . Also , Jastrzebski et al . ( 2017 ) primarily focus on deepening the understanding of iterative refinement in ResNets . While our results may strengthen the argument that ResNets are naturally doing something iterative , this is tangential to our broader claims that for several families of models depth can be achieved with distinct layers or with recurrent ones . Similarly , Liao & Poggio ( 2016 ) show that deep RNNs can be rephrased as ResNets with weight sharing and even include a study on batch normalization in recurrent layers . Our work builds on theirs to further elucidate the similarity of depth and recurrence , specifically , we carry out quantitative and qualitative comparisons between the deep features , as well as performance , of recurrent and analogous feed-forward architectures as their depth scales . We analyze a variety of additional models , including multi-layer perceptrons and convolutional architectures which do not include residual connections . Our work expands the scope of the aforementioned studies in several key ways . First , we do not focus on model compression , but we instead analyze the relationship between recurrent networks and models whose layers contain distinct parameters . Second , we use a standard neural network training process , and we study a variety of architectures . Finally , we conduct our experiments on image data , as well as a reasoning task , rather than sequences , so that our recurrent models can be viewed as standard fully-connected or residual networks but with additional weight-sharing constraints . For each family of architectures and each dataset we study , we observe various trends as depth , or equivalently the number of recurrence iterations , is increased that are consistent from recurrent to feed-forward models . These relationships are often not simple , for example classification accuracy does not vary monotonically with depth . However , the trends are consistent whether depth is added via distinct layers or with iterations . We are the first to our knowledge to make qualitative and quantitative observations that recurrence mimics depth . 2 RECURRENT ARCHITECTURES . Widely used architectures , such as ResNet and DenseNet , scale up depth in order to boost performance on benchmark image classification tasks ( He et al. , 2016 ; Huang et al. , 2017 ) . We show that this trend also holds true in networks where recurrence replaces depth . Typically , models are made deeper by adding layers with new parameters , but recurrent networks can be made deeper by recurring through modules more times without adding any parameters . To discuss this further , it is useful to define effective depth . Let a layer of a neural network be denoted by ` . Then , a p-layer feed-forward network can be defined by a function f that maps an input x to a label y where f ( x ) = ` p ◦ ` p−1 ◦ · · · ◦ ` 2 ◦ ` 1 ( x ) . In this general formulation , each member of the set { ` i } pi=1 can be a different function , including compositions of linear and nonlinear functions . The recurrent networks in this study are of the following form . f ( x ) = ` p ◦ · · · ◦ ` k+1 ◦mr ◦mr ◦ · · · ◦ ` k ◦ · · · ◦ ` 1 ( x ) ( 1 ) We let mr ( x ) = ` ( m ) q ◦ · · · ◦ ` ( m ) 1 ( x ) and call it a q-layer recurrent module , which is applied successively between the first and last layers . The effective depth of a network ( feed-forward or recurrent ) is the number of layers , not necessarily unique , composed with one another to form f . Thus , the network in Equation equation 1 has an effective depth of p+ nq , where n is the number of iterations of the recurrent module . A more concrete example is a feed-forward network with seven layers , which has the same effective depth as a recurrent model with two layers before the recurrent module , two layers after the recurrent module , and a single-layer recurrent module that runs for three iterations . See Figure 1 for a graphical representation the effective depth measurement . We train and test models of varying effective depths on ImageNet , CIFAR-10 , EMNIST , and SVHN to study the relationship between depth and recurrence ( Russakovsky et al. , 2015 ; Krizhevsky , 2009 ; Cohen et al. , 2017 ; Netzer et al. , 2011 ) . For every architecture style and every dataset , we see that depth can be emulated by recurrence . The details of the particular recurrent and feed-forward models we use in these experiments are outlined below . For specific training hyperparameters for every experiment , see Appendix A.2.1 The specific architectures we study in the context of image classification include families of multilayer perceptrons ( MLPs ) , ConvNets , and residual networks . Each recurrent model contains an initial set of layers , which project into feature space , followed by a module whose inputs and outputs have the same shape . As a result of the latter property , this module can be iteratively applied in a recurrent fashion an arbitrary number of times . The feed-forward analogues of these recurrent models instead stack several unique instances of the internal module on top of each other . In other words , our recurrent models can be considered the same as their feed-forward counterparts but with depth-wise weight sharing between each instance of the internal module . 2.1 MULTI-LAYER PERCEPTRONS . The MLPs we study are defined by the width of their internal modules which is specified per dataset in Appendix A.1 . We examine feed-forward and recurrent MLPs with effective depths from 3 to 10 . The recurrent models have non-recurrent layers at the beginning and end , and they have one layer in between that can be recurred . For example , if a recurrent MLP has effective depth five , then the model is trained and tested with three iterations of this single recurrent layer . A feed-forward MLP of depth five will have the exact same architecture , but instead with three distinct fully-connected layers between the first and last layers . All MLPs we use have ReLU activation functions between full-connected layers . 1Code for reproducing the experiments in this paper is available in the code repository at < suppressed for anonymity > .

Summary: In this work, the authors compare and contrast models wherein layer depth is replaced with equivalent number of recurrent time steps. In particular, they explore feed-forward, CNN and residual deep network wherein the intermediate layers are replaced by an equivalent recurrent block. For each architecture, the authors compare accuracy of the stacked and recurrent variants for image classification on 4 datasets- CIFAR-10, EMNIST, SVHN and ImageNet and a novel Maze task wherein the network has to assign binary labels to each image pixel (0= not a valid point in path, 1=valid point in path to solve the maze). Overall, the authors observe that across all models and tasks, the recurrent and stacked networks achieve similar performance. They also verify this with batch normalization and dilated CNNs in some tasks. Next, they investigate the number of positive activations at each recurrent step to find that filters are active/recycled across recurrent depth. They also analyze the linear classification accuracy at different depths of the stacked and recurrent CIFAR-10 feedforward model, finding comparable accuracy at each depth. Finally, they visualize filter activations at different depths for both CIFAR-10 & ImageNet, finding comparable feature maps in both architectures.