Go with the Flow: the distribution of information processing in multi-path networks

1 INTRODUCTION . The architectures of Convolutional Neural Network ( CNN ) classifiers are an important influencing factor regarding their predictive performance and computational efficiency , with many designs being proposed over the years ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Szegedy et al. , 2015 ; Tan & Le , 2019 ; Sandler et al. , 2018 ) . These architectures can be generally subdivided into two distinct categories . First , the sequential architectures like VGG16 by Simonyan & Zisserman ( 2015 ) , which essentially consist of a sequence of layers leading from the input to the output of the network . While these architectures are simplistic in structure , they have been surpassed by multi-path architectures in efficiency and predictive performance . Multipath architectures can be described as a directed acyclic graph , with the nodes representing the layers . Information provided at the input of a multipath network can be processed by different sequences of layers that are intertwined in the overall neural architecture . The Inception-family of networks is an example for such an architecture ( Szegedy et al. , 2015 ; 2016 ; 2017 ) , where each building block features a different set of layers and allows for the parallel extraction of heterogeneous features . Networks featuring skip-connections like ResNet , MobileNetV2 , MobileNetV3 and EfficientNet ( He et al. , 2016 ; Sandler et al. , 2018 ; Howard et al. , 2019 ; Tan & Le , 2019 ) are a special case of multipath architectures , since they may only feature a single sequence of layers . However , since the skip-connections effectively allow signals to skip layers , it is possible for any signal from the input to take multiple paths to the output , thus making these architectures multipath architectures . On the other hand , multi-path architectures require the model to implicitly make decisions regarding the distribution of the inference process , since some routes connecting two layers may differ in their capacity ( number of parameters ) , depth and receptive field size . In essence , during training , the model will learn to utilize the different pathways , implicitly deciding how the inference process is distributed in branching paths and whether to skip certain layers . These decisions have been discussed theoretically by authors like He et al . ( 2016 ) , who for example elaborate the possibility of identity-mappings in skip-connections , which effectively enable the network to remove layers from the qualitative inference process . In this work , we empirically investigate how parallel pathways and skip connections are utilized in multi-path CNN architectures . Our contributions can be summarized as follows : • We find that networks with skip connections reliably skip layers when a mismatch between receptive field and input resolution occurs . This finding is in agreement with the findings of Richter et al . ( 2021b ) and Richter et al . ( 2021a ) . • We show that networks with parallel pathways of strongly different depth will prefer the shorter pathway over the longer pathway due to a partial vanishing gradient . • We demonstrate that multi-path architectures with parallel pathways of roughly similar size advance the solution quality in a similar pace and with a high degree of redundancy . 2 RELATED WORK . 2.1 MULTI-PATH NETWORKS . Since this work is focussed on multi-path architectures , we will introduce the most significant works regarding this type of neural architecture as well as the contemporary reasoning for the introduction of the proposed neural architectures . After the advent of CNNs , there were several works that focused on improving their performance by introducing novel architectures ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ) . Although one of the most intuitive ways to improve the accuracy of CNNs is by increasing the number of layers ( depth ) , there is an upper limit to it . After a specific level , there are certain hurdles that occur such as network overfitting , vanishing gradient ( Hochreiter , 1998 ; Dong et al. , 2015 ) , and also it becomes computationally expensive to train them . Therefore , to train such deep networks , several works proposed the idea of multipath networks ( He et al. , 2016 ; Larsson et al. , 2017 ; Huang et al. , 2018 ; Kuen et al. , 2017 ) . The multipath networks presented in ( Mao et al. , 2016 ; Tong et al. , 2017 ; Srivastava et al. , 2015 ) introduced shortcuts in the networks structure that facilitate a customized flow of information such that the vanishing gradient problem is reduced . The work by Szegedy et al . ( 2015 ) introduced sparsely connected architectures motivated by Hebbian principle to solve the problem of overfitting and then clustering them to dense matrices ( Ümt V. Çatalyürek et al. , 2010 ) so that the network can be trained efficiently on the hardware . Further improvements to this network were introduced ( Szegedy et al. , 2016 ; 2017 ) to increase its accuracy and training efficiency . Since the information available to the shallower layers is not available to the deeper layers , He et al . ( 2016 ) introduced the Residual Networks ( ResNets ) which used skip connections to preserve the gradient . The skip connections are operated by vector addition whereas in the method proposed by Huang et al . ( 2018 ) , the skip connections are operated by concatenation of the feature maps . 2.2 RELEVANT ANALYSIS TOOLS AND SIMILARITY METRICS . Since we are interested in the information processing of CNNs , a set of analysis tools is required that allows us to analyze the processing within the neural architecture . Our main analysis tool that is used extensively throughout this work are logistic regression probes ( LRP ) by Alain et al . ( 2020 ) . LRP are logistic regressions trained on the output of a hidden layer . Since classifiers implicitly maximize the linear separability of the data , the predictive performance of the logistic regression models on the test set allows us to track the progress of the intermediate solution quality while the data is propagated from layer to layer ( see Fig . 1 ) . The usefulness of LRP has been demonstrated by the works of Richter et al . ( 2020 ) , Richter et al . ( 2021a ) and Richter et al . ( 2021b ) , which utilize LRP to identify inefficiencies in neural architectures caused by layers not contributing qualitatively to the inference process . Alain et al . ( 2020 ) find by experimenting on multilayer perceptrons with skip connections that the skipping of a sequence of layers is observable by logistic LRP by a degradation in performance . A reproduction of this experiment on MNIST using a similar architecture can be seen in Fig . 2 . In section 3 we furthermore utilize saturation proposed by Richter et al . ( 2020 ) as an additional tool for analyzing the output of hidden layers . Saturation uses PCA to approximate the subspace the data is processed , and can be interpreted as the percentage of dimensions utilized by the data in the output of space of a given layer . Thereby , highly saturated layers can be usually seen as active , in the sense that they contribute qualitatively to the prediction , while layers that are low saturated relative to the rest of the network tend to be unproductive . A sequence of unproductive layers is referred to as a tail pattern. . Since we are interested in the information processing of ( partial ) networks , tools that easily allow the comparison of learned feature representation are required . Several works ( Raghu et al. , 2017 ; Morcos et al. , 2018 ; Feng et al. , 2020 ) proposed such tools to gauge the similarity of the features learned by two models having the same architectures but trained with different initialization . The similarity is calculated by taking the hidden state activations from each model in the form of a feature matrix and finding a measure of correlation between them . The method proposed by Wang et al . ( 2018 ) uses a subspace match model to find the similarity in terms of maximum and simple matches . Raghu et al . ( 2017 ) uses Canonical Correlation Analysis ( CCA ) proposed in Golub & Zha ( 1995 ) coupled with Singular Vector Decomposition ( SVD ) to compare the representations presented by two neural networks in a process which is invariant to invertible linear transformations . The technique by Morcos et al . ( 2018 ) further improves this method by giving more weight to those intermediate CCA vectors which are more important for the representations . Another metric proposed by Feng et al . ( 2020 ) called the Transferred Discrepancy ( TD ) focuses on the practical usage of the representations by stating their similarity based on their performance in downstream tasks . Another approach to compare layer activations , that has gained popularity over the last years is Centered Kernel Alignment ( CKA , Cortes et al. , 2012 ) , which we will use for this work . It is introduced as a normalized version of the Hilbert-Schmidt Independence Criterion ( HSIC , Gretton et al. , 2005 ) , and when used with linear kernels , it is equivalent to the RV coefficient Robert & Escoufier ( 1976 ) . In this form it has been applied successfully in recent works as an efficient alternative to SVCCA and other methods ( Nguyen et al. , 2021 ; Kornblith et al. , 2019 ) , majorly because it requires less number of data points ( Kornblith et al. , 2019 ) . 3 DISTRIBUTION OF INFORMATION PROCESSING IN NETWORKS WITH SKIP-CONNECTIONS . In this section , we will investigate the first scenario we identified in which a network with a skip connection will reliably choose not to utilize the layers encapsulated by a skip connection . Skipping of layers could be described as an autogenous pruning technique , where the model decides during training to not utilize certain layers . Therefore , we hypothesize that layers that would be unproductive are likely to be skipped if the network is given the opportunity . We find that it is possible to guarantee a convolutional layer will not be able to contribute to the quality of the solution by utilizing the knowledge about the relation of receptive field and input resolution presented by Richter et al . ( 2021b ) . The authors show experimentally that convolutional layers can only enhance the quality of the solution if the receptive field of the layer ’ s input is smaller than the input image . Simply speaking , if the layer is unable to integrate novel information into a feature map position by convolving the kernel over the input , the layer will not improve the quality of the intermediate solution . In case of a simple sequential architecture neural network like VGG16 the layers pass the solved problem from layer to layer while not enhancing the intermediate solution quality , as Fig . 3 ( a ) exemplifies . The observed effect is caused by training the model on Cifar10 using the native resolution of 32 × 32 pixels . When repeating this experiment on DenseNet18 , we can see that the probe performance no longer stagnates . Instead , the final dense-block of the model is skipped entirely , which is apparent when looking at the decaying probe performance in Fig . 3 ( b ) . A similar effect can be observed when training ResNet34 in the same scenario . In this case , unproductive residual blocks are skipped . Since the input of a residual block is added to its output , the performance recovers after each building block and thus recovering the intermediate solution quality . This is reflected by a zig-zag-pattern in the LRP . Based on these results , we can see that unproductive layers are behaving differently in architectures where these layers can be bypassed . Instead of learning to functionally emulate a pass-through layer , the layers learn a representation which is functionally equivalent to an identity mapping . With functionally equivalent , we refer to the behavior of the layers being similar to a pass-through layer or an identity mapping regarding the predictive performance . We find that skipped layers do in-fact not learn an identity mapping . Instead , these layers learn a harmless non-zero representation that does not hurt the intermediate solution quality when added to the feature map containing a good solution . This makes it also impossible to remove these skipped layers in a primitive pruning step without retraining the model , since the changes in the data caused by the unproductive layers are still expected by the following layers . Nevertheless , removing the skipped layers and retraining the pruned model will lead to a more efficient network ( Richter et al. , 2021b ) .

Summary: This paper analyzes the distribution of information processing in multi-path networks (including skip-connection models such as ResNet and DenseNet). They apply logistic regression on the hidden layers, namely logistic regression probes, to track the progress of the intermediate solution quality, and they analyze in which condition (depth of the path and receptive field size) the neural network prefers to skip the paths. They also measure the CKA similarity of the hidden representations learned by the multi-path model when the paths are homogeneous. They claim that with their analysis, later layers in ResNet and DenseNet can be skipped as pruning due to the unproductive layers. In addition, they find that for multi-path model, the shorter path is dominant when the depth of the paths are very different, and when the pathways are homogeneous, the behavior of the pathways changes to a coexisting behavior.