Go with the Flow: the distribution of information processing in multi-path networks
Authors: Mats Leon Richter
Krupal Shah
Anna Wiedenroth
Saketh Bachu
Ulf Krumnack
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Learning
Multi-path Networks
Similarity of Representations
Regression Probe
Centered Kernel Alignment

Abstract: The architectures of convolution neural networks (CNN) have a great impact on the predictive performance and efficiency of the model.
Yet, the development of these architectures is still driven by trial and error, making the design of novel models a costly endeavor. 
To move towards a more guided process, the impact of design decisions on information processing must be understood better. 
This work contributes by analyzing the processing of the information in neural architectures with parallel pathways.
Using logistic regression probes and similarity indices, we characterize the role of different pathways in the network during the inference process.
In detail, we find that similar sized pathways advance the solution quality at a similar pace, with high redundancy.
On the other hand, shorter pathways dominate longer ones by majorly transporting (and improving) the main signal, while longer pathways do not advance the solution quality directly. 
Additionally, we explore the situation in which networks start to ``skip'' layers and how the skipping of layers is expressed.

Under review as a conference paper at ICLR 2022
Go WITH THE FLOW: THE DISTRIBUTION OF INFOR- MATION PROCESSING IN MULTI-PATH NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The architectures of convolution neural networks (CNN) have great impact on the predictive performance and efficiency of the model. Yet, the development of these architectures still driven by trial and error, making the design of novel models a costly endeavor. To move towards a more guided process, the impact of design decisions on information processing must be understood better. This work contributes by analyzing the processing of the information in neural architectures with parallel pathways. Using logistic regression probes and similarity indices, we characterize the role of different pathways in the network during the inference process. In detail, we find that similar sized pathways advance the solution quality at similar pace, with high redundancy. On the other hand, shorter pathways dominate longer ones by majorly transporting (and improving) the main signal, while longer pathways do not advance the solution quality directly. Additionally, we explore the situation in which networks start to "skip" layers and how the skipping of layers is expressed.
INTRODUCTION
The architectures of Convolutional Neural Network (CNN) classifiers are an important influencing factor regarding their predictive performance and computational efficiency, with many designs being proposed over the years (Simonyan & Zisserman, 2013 He et al., 2016) Szegedy et al. [015]] Tan
Le 2019 Sandler et al. 2018). These architectures can be generally subdivided into two distinct categories. First, the sequential architectures like VGG16 by (Simonyan & Zisserman (2015), which essentially consist of sequence of layers leading from the input to the output of the network. While these architectures are simplistic in structure, they have been surpassed by multi path architectures in efficiency and predictive performance.
Multipath architectures can be described as a directed acyelic graph, with the nodes representing the layers. Information provided at the input of multipath network can be processed by different sequences of layers that are intertwined in the overall neural architecture. The Inception-family of networks is an example for such architecture (Szegedy al.) 2015) 2016 2017), where each building block features a different set of layers and allows for the parallel extraction of heteroge- neous features. Networks featuring 8kip-connections like ResNet. MobileNetV2. MobileNetV3 and EfficientNet (He et al. 2016 Sandler et al. 2018 Howard et al. 2019; Tan & Le (2019) are a special case of multipath architectures, since they may only feature single sequence layers.
However, since the coineconnsssonss effectively allow signals to skip layers, it is possible for any signal from the input take multiple paths the output, thus making these architectures multi- path architectures. On the other hand, multi-path architectures require the model to implicitly make decisions regarding the distribution of the inference process, since some routes connecting two layers may differ in their capacity (number of parameters), depth and receptive field size. In essence, during training, the model will learn to utilize the different pathways, implicitly deciding how the inference process is distributed in branching paths and whether to skip certain layers.
These decisions have been discussed theoretically by authors like He et al. (016).. who for ex- ample elaborate the possibility of drentity-mapping in skip-connections, which effectively enable the network to remove layers from the qualitative inference process. In this work, we empirically investigate how parallel pathways and skip connections utilized in multi-path CNN architectures.

---

Under review as a conference paper at ICLR 2022
Our contributions can be summarized as follows:
We find that networks with skip connections reliably skip layers when a mismatch between receptive field and input resolution occurs. This finding is in agreement with the findings of Richter et al. (2021b) and Richter et al. (2021a).
We show that networks with parallel pathways of strongly different depth will prefer the shorter pathway over the longer pathway due to partial vanishing gradient.
We demonstrate that multi-path architectures with parallel pathways of roughly similar size advance the solution quality in a similar pace and with high degree of redundancy.
RELATED WORK
2.1 MULTI-PATH NETWORKS
Since this work is focussed on multi-path architectures, we will introduce the most significant works regarding this type of neural architecture as well as the contemporary reasoning for the introduction of the proposed neural architectures. After the advent of CNNs, there were several works that fo- cused on improving their performance by introducing novel architectures (Krizhevsky et al.) 2012] Simonyan & Zisserman 2015). Although one of the most intuitive ways to improve the accuracy of CNNs is by increasing the number of layers (depth), there is an upper limit to it. After a specific level, there are certain hurdles that occur such as network overfitting, vanishing gradient (Hochreiter) Dong et al.) 2015), and also it becomes computationally expensive to train them. Therefore, to train such deep networks, several works proposed the idea of multipath networks (He et al. 2016
1998
Larsson et al. 2017 Huang et al. 2018 Kuen al. 2017). The multipath networks presented in
(Mao et al.) 2016: Tong et al. | Srivastava et al. (2155) introduced shortcuts in the networks structure that facilitate customized flow of information such that the vanishing gradient problem is reduced. The work by Szegedy et al. (2015) introduced sparsely connected architectures moti- vated by Hebbian principle solve the problem of overfitting and then clustering them to dense matrices (Umt V. Çatalyürek et al. 2010) so that the network can be trained efficiently on the hard- ware. Further improvements to this network were introduced (Szegedy et a 2017) to increase its accuracy and training efficiency. Since the information available to the shallower layers is not available to the deeper layers, He et al. (2016) introduced the Residual Networks (ResNets) which used skip connections to preserve the gradient. The skip connections are operated by vector addi- tion whereas in the method proposed by Huang et al. (2018), the skip connections are operated by concatenation of the feature maps.
2.2 RELEVANT ANALYSIS TOOLS AND SIMILARITY METRICS
Since we are interested in the information processing of CNNs, a set of analysis tools required that allows us to analyze the processing within the neural architecture. Our main analysis tool that is used extensively throughout this work are logistic regression probes (LRP) by Alain et al. (2020). LRP are logistic regressions trained on the output of hidden layer. Since classifiers implicitly maximize the linear separability of the data, the predictive performance of the logistic regression models the test set allows us to track the progress of the intermediate solution quality while the data is propagated from layer to layer (see Fig. []. The usefulness of LRP has been demonstrated by the works Richtr et al. 22020) Richter et al. (2021a) and Richter et al. (2021b), which utilize LRP to identify inefficiencies in neural architectures caused by layers not contributing qualitatively to the inference process. Alain et al. (2020) find by experimenting on multilayer perceptrons with skip connections that the skipping of a sequence of layers is observable by logistic LRP by a degradation in performance.
reproduction of this experiment on MNIST using a similar architecture can be seen in Fig. 2 In section 3 we furthermore utilize saturation proposed by |Richter et al. (2020) as an additional tool for analyzing the output of hidden layers. Saturation uses PCA to approximate the subspace the data is processed, and can be interpreted as the percentage of dimensions utilized by the data in the output of space of a given layer. Thereby, highly saturated layers can be usually seen as active, in the sense that they contribute qualitatively to the prediction, while layers that are low saturated relative to the rest of the network tend to be unproductive. A sequence of unproductive layers is referred to fail pattern.
2

---

Under review as a conference paper at ICLR 2022
Probe Accuracy
model test acc.: 91.34%
1.00
Probe Acc.
0.75
0.50
0.6
0.25
Layers
Figure I: Logistic Regression Probes allow the practitioner to observe the evolution of the interme- diate solution quality from layer to layer. The probe performance each layer of VGG16 trained on Cifar10. The performance is increasing layer by layer, indicating that the problem is solved incrementally and that the inference process is evenly distributed among layers.
Softmax
DenseLayer 128
* 8
+
DenseLayer 128
x 128
DenseLayer 128
(a) The multi-layer perceptron architecture used for the reproduction of
the experiment skip connections by Alain al. (2020). The architec-
ture deviates slightly from the original by Alain et al. (200 to make the model easier to train.
1.0
Logistic Regmession Probes
Tost Accuracy
0.8
Model Test Accuracy: 96, 45%
0.6
1000
Layers
(b) The performance of the logistic regression probes in the order of the forward pass. Note the stark difference in behavior between the first part (encapsulated by the skip connection) and the second part of the network.
Figure 2: The images show the basic neural architecture with skip connection (a) and the probe performances of each layer (b). The setup is designed to provoke the network to "skip" the 128 layers by learning some identity-mappinn analog. By observing the probe performances, this behavior can be observed. After the initial layer, the performance degrades until reaching chance level. The probe performances recover as soon as the skip connection is added to the layers again. We will observe similar behavior on convolutional neural networks over the course of this work.
Since we are interested in the information processing of (partial) networks, tools that easily allow the comparison of learned feature representation required. Several works (Raghu et al. 2017

---

Under review as a conference paper at ICLR 2022
Morcos al. 2018 Feng al. 2020) proposed such tools gauge the similarity of the features learned by two models having the same architectures but trained with different initialization. The similarity is calculated by taking the hidden state activations from each model in the form of feature matrix and finding a measure of correlation between them. The method proposed by [Wang et al. (2018) uses subspace match model to find the similarity in terms of maximum and simple matches. Raghu et al. (2017) uses Canonical Correlation Analysis (CCA) proposed in Golub & Zha (1995) coupled with Singular Vector Decomposition (SVD) to compare the representations presented by two neural networks in a process which is invariant to invertible linear transformations.
The technique by [Morcos et al.] (2018) further improves this method by giving more weight to those intermediate CCA vectors which are more important for the representations. Another metric proposed by Feng et al. (2020) called the Transferred Discrepancy (TD) focuses on the practical usage of the representations by stating their similarity based on their performance in downstream tasks.
Another approach compare layer activations, that has gained popularity over the last years is Cen- tered Kernel Alignment (CKA,,Cortes et al. 2012), which we will use for this work. It is introduced as normalized version of the Hilbert-Schmine Independence Criterion (HSIC, Gretton al. 2005),
and when used with linear kernels, it is equivalent to the RV coefficient Robert & Escoufier 1976..
In this form it has been applied successfully in recent works as efficient alternative S SVCCA and other methods (Nguyen et al.] [021] Kornblith et al., 2019), majorly because it requires less number of data points (Kornblith et al. 2019).
3. DISTRIBUTION OF INFORMATION PROCESSING IN NETWORKS WITH SKIP-CONNECTIONS
In this section, we will investigate the first scenario we identified in which network with skip connection will reliably choose not to utilize the layers encapsulated by a skip connection. Skip- ping of layers could be deseribed during training to not utilize certain layers. Therefore, we hypothesize that layers that would be unproductive are likely be skipped if the network is given the opportunity. We find that it is pos- sible to guarantee a convolutional layer will not be able to contribute to the quality of the solution by utilizing the knowledge about the relation of receptive field and input resolution presented by Richter et al. (2021b). The authors show experimentally that convolutional layers can only enhance the quality of the solution if the receptive field of the layer's input smaller than the input image. Simply speaking, if the layer is unable to integrate novel information into feature map position by convolving the kernel over the input, the layer will not improve the quality of the intermediate solution.
autogenous pruning technique, where the model decides
In case of a simple sequential architecture neural network like VGGI6 the layers pass the solved problem from layer to layer while not enhancing the intermediate solution quality, as Fig. 3 (a) ex- emplifies. The observed effect is caused by training the model on Cifar 10 using the native resolution of 32 x 32 pixels. When repeating this experiment DenseNet18, 8, can that the probe per- formance no longer stagnates. Instead, the final dense-block of the model is skipped entirely, which apparent when looking at the decaying probe performance in Fig. (b). A similar effect can be observed when training ResNet34 in the same scenario. In this case, unproductive residual blocks are skipped. Since the input of residual block is added its output, the performance recovers after each building block and thus recovering the intermediate solution quality. This is reflected by a zig-zag- aazzaaaattternn in the LRP.
Based on these results, we can see that unproductive layers are behaving differently in architectures where these layers can be bypassed. Instead of learning to functionally emulate a pass-through layer, the layers learn representation which is functionally equivalent to an identity mapping. With functionally equivalent, we refer to the behavior of the layers being similar to a pass-through layer or identity mapping regarding the predictive performance. We find that skipped layers do fn-fact not learn an identity mapping. Instead, these layers learn a harmless non-zero representation that does not hurt the intermediate solution quality when added to the feature map containing a good solution. This makes it also impossible to remove these skipped layers in a primitive pruning step without retraining the model, since the changes in the data caused by the unproductive layers
4

---

Under review as a conference paper at ICLR 2022
Accuracy
Saturation
probe arourary
saturation
model test acc:: 88.93%
0.0
0.0
Layers
(a) VGG16 tail layers maintain the quality of the intermediate
lution.
Saturation
pro accuracy
saturation
model test acc.: 07.07%
Test Accuracy
0.0
0.0
Layers
(b) The tail of DenseNet18 shows decay in probe performance. indicating the last DenseBlock skipped entirely (Alain al. 2020
Tost Accuracy
probe accuracy
saturation
model test acc.: 83.44%
Saturation
1.0
1.0
Layers
ResNet34 skips most residual blocks in the tail, which is ap- parent by the zig-zag pattern in probe performances caused by the starts and end of skip-connections (Alain et al. 2020).
Figure 3: Depending the neural architecture, tail patterns may deviate in their appearance in probe performance. In sequential architectures (a) the layers maintain the quality of the intermediate so- lution. If shorteut connections exist in the architecture, layers may be skipped. Skipped layers apparent by their decaying probe performance AAlin et al. (2020). This apparent on DenseNet18 (b) and ResNet34 (b) where single DenseBlock and multiple ResidualBlocks skipped respec- tively. All models are trained on Cifar10 at native resolution.
still expected by the following layers. Nevertheless, removing the skipped layers and retraining the pruned model will lead to a more efficient network (Richter et al.) (0211b)
DISTRIBUTION OF INFORMATION PROCESSING IN MULTI-PATH ARCHITECTURES
We proceed to investigate networks with parallel sequences of layers. To make the results of this investigation easier to visualize, we use a simple CNN architecture that composed of building blocks that contain two distinct pathways. The basic template for this architecture can be seen in Fig. The architecture itself consists of four stages consisting of k building blocks each. The filter size is doubled from stage to stage, while the first layers process the data in every stage downsample the filter map with convolutions with stride size 2. The downsampling and the increases in filter sizes over multiple stages are common feature in modern neural architectures such as ResNet He et al. (2016) or EfficientNet (Tan & Le (2019)) The individual building blocks have two-pathway layout, see Fig. 4 (b). Each pathway sequence of layers using the same kernel size. The number of layers and kernel sizes in each pathway are varied throughout the experiments and will explicitly mentioned the respective experiment. The pathways are reunited using an element-wise addition. We decided to utilize element-wiss addition instead of concatenation. since avoids the otherwise necessary 1 x convolution for dimension reduction after each building block. From additional experiments, we find that the 1 x 1 convolutions also do not impact the results of the experiments substantially. We refer to this multipath architectures as MPNet x (n, : 81 81, 912 82 82), where K is the number of building blocks, ni and ni refer to the number of
5

---

Under review as a conference paper at ICLR 2022
layers for the first and second pathway and 11 and k2 refer to the kernel sizes used in the first and second pathway respectively. We refer to the sequence of layers composed by only the pathways that feature the smaller kernel size as Pt,min and to the path with the larger kernel sizes as marr. While these pathways are intertwined in the architecture, we visualize these two pathways as separate sequences, which makes the visualizations more digestible.
Softmax
Aug Pool
K
Block 256
K
Block 120
* K
Pathway
Block 64
K
Pathway
Cone
(a) Overall structure
(b) Building Block
Figure 4: The architecture used for the experiments in this section. Each stage consists of k blocks and has double the filters of the previous stage. The first layer of each stage a downsampling layer with stride size of 2. Each building block consists of two pathways, which resemble sequences of convolutional layers.
Training conducted on the datasets of ImageNette and Cifar10. We choose ImageNette over the full ImageNet dataset, since the large number of experiments is necessary to make the results pre- sented in this work quantifiable. Since LRPs are very resource and time intensive to train on large datasets, make this concession practicality. Training is conducted using the same preprocess- ing as He et al. (2016), training is conducted for 30 epochs using stochastic gradient descent with a learning rate of 0.1. The learning rate is reduced by factor of 0.1 every 10 epochs.
4.1 DOMINANT AND NON-DOMINANT PATHWAYS IN HETEROGENEOUS BUILDING BLOCKS
We first investigate scenarios where the pathways behave substantially different when measured with LRP. We find that this can be achieved by increasing the heterogeneity between the different pathways. To exemplify this, a building block is used with 3 x 3 pathway with 4 layers and 7 x 7 pathway with a single layer. These two pathways differ in three important properties. The number of layers, resulting in strong difference in capacity, receptive field and gradient flow. We train these models on ImageNette with an 250 x 250 input resolution. When using a single building block per stage, we can see in Fig. 5a that now the second building block is only utilizing the short pathway. When doubling the number of building blocks per stage, we observe in Fig. [5b] that this behavior has spread all building blocks. While this indicates that pt,min is utilized while layers of Pt.ma are skipped, we were unable to confirm this experimentally. When removing the skipped layers, we find that the performance decreases by 2.16%-poinnss on average, which indicates that's these paths are still not entirely irrelevant for the inference process. For this reason, we will not refer to these paths as skipped. Instead, we will refer to the path as dominant when they advance the intermediate solution quality directly and non-dominant when their contribution is not directly measurable by logistic regression probes.
4.1.1 PARTIAL VANISHING GRADIENT NON-DOMINANT PATHWAYS
So far, we observe two distinct causes for degradation in LRP performance. First, the degrada- tion caused by excessively large receptive field in section S second the dominam/non-dommnnn behavior observed in section 4.1 caused by heterogeneity in the pathways. The pathways of the pre- viously tested models in section 4.1 differ in depth and receptive field size. We hypothesize that the

---

Under review as a conference paper at ICLR 2022
LRP Tost Acc.
Plaae
test acc.: 88 29%
LRP Test: Acc.
Pi.
Pumto
Test acc.: 89.14%
0.8
0.8
0.6
Layers
Layers
(a) M1 : x 3,4:
(b) MPNel, 3
Figure 5: When training MPNet with pathways of different depth, deeper pathway is skipped for building block (a), which is apparent by the deteriorating probe performance. This effect occurs on all building blocks when the depth is increased. The models are trained on Cifar10.
observed behavior is caused by one of those two factors. We test this hypothesis by using a building
Pl max
Pl,mi
test acc.:
LRP Test Acc.
0.8
Meen Aboohte Ordden
shatt layers path lopors
0.4
Layers
(a) MPNeta(8: 3 3.1: 21 21)
(b) MPNeta :101:334 7)
Figure 6: Different from tail patterns, the shallower pathway is dominating the deeper pathway even their receptive field sizes are identical (a). We attribute the serrinating-bhhavie to vanishing gradient in the deeper pathway, evident in the overall lower size of the gradients (b).
block with pathway using 8 83x3 3 layers and second pathway using single 21 21 convolutional layer. The result of this unusual setup is that the single layer on the mt. -pttwwa will increase the receptive field more than the stack of 3 in the other pathway. If the receptive field is responsible for this behavior, we should be able to observe a skipping-behavinn behavior in the shorter path. As we can see in Fig. 6a this not the case. Instead, the layers still appear to be dominant for the most part. A better explanation for this behavior is provided by analyzing the accumulated gradients of a neu- ral architecture with a clear dominant/non-dominan behavior lime MPNet. 3 x 3, 4 trained on ImageNette using an 160 x 160 input resolution in Fig. 6b This is done by computing the average over the absolute mean of all gradients, which gives us an idea on the size of the gradient that is propagated back through given layer. We can clearly see that the gradients of the long, non-dominant pathway are strictly smaller than the gradient of the dominant pathway. We interpret this partial vanishing gradient problem, where the gradient flow is following the path of least resistance, implicitly preferring shorter pathways to longer ones, where the signal may be more diluted by the number of layers the gradient has to pass through.
7)
4.2 COEXISTENCE OF HOMOGENEOUS PARALLEL PATHWAYS IN MULTIPATH ARCHITECTURES
We continue to analyze scenarios in which both pathways are less heterogeneous. For this reason, we train MPNet, (2 3 x 3,2 7 x 7) on Cifar10. The equal number of layers in both pathways ensures that the gradient travels through similar amount of layers both pathways. We chose the kernel sizes to make both pathways slightly heterogeneous, similarly to the different pathways in the Inception architectures. We additionally train MPNett 3 x 3, 2 7 x 7). which features one less 3 x 3 convolution to increase the heterogeneity of the pathways. We train both models on Cifarl0 and train LRP on each layer of the architecture. From the results in Fig. 7 we observe that the probe performance of the two pathways Pl.min and Pi,maz almost indistinguishable. We interpret this as a cocxisting behavior, these pathways do not necessarily extract different features to achieve the depicted increase in LRP accuracy. We will see further evidence of this in the next section.

---

Under review as a conference paper at ICLR 2022
LRP Tost Acc.
Plawe
test acc.: 84.59%
LRP Test Acc.
Pr.
Pumto
Test acc.: 89.54%
0.8
0.6
Layers
Layers
(a) MPNet222 3,
(b) MPNeet211 3,
Figure 7: The MPNet architectures with only slightly different pathways in each building block show a coexisting behavior, which is indicated by the probe performances of the 3 x 3-pathway Pi,min and the 7 pathway Pl,max increasing at similar rate.
4.2.1 CORRELATION ANALYSIS IN MULTIPATH ARCHITECTURES
To provide additional insights and also different perspective the division of information in multipath networks, we utilize the CKA as a metric to quantify correlations. While our main analysis tool, LRP, measures the quality of the information being carried in terms of accuracy, the CKA can provide the correlation measurement between the latent representations of layers. We compute the CKA for all the layers (for both the paths) with the final output of the network. This is calculated in a similar way as the LRP accuracy (see section 2.2
The resulting graphs can be seen in the Fig. g
cka,,
max
test 82.69%
ckai, max
Pt max
test 83.73%
cka, men
Pi
cka m/n
Pt
LRP COK
Adc.
0.5
0.5
0.0
Layera
Layers
(a) MPNela(1 3,2, 7) trained on Cifar10 dataset.
(b) 3,
trained on Cifar10
dataset.
ckal,
Pt
test ace.: 74.04%
cka
Pt min
LRP OSA
0.0
Layers
(c) (8 3 3, 7 x 7) trained on Ima- geNette200 dataset.
Figure 8: Comparison the representational similarity score given by the Centered Kernel Align- ment (Cortes et al. 2012 Kornblith et al. 2019) and the probe performance of MPNet architectures with different sets of pathways in each building block. The CKA value is calculated to quantify each layer's representational similarity with the final output of the network. The min and maz subscript refer to the path with the smaller and larger kernel sizes. The CKA values are in accordance with the probe performance, indicating that the shorter paths and the longer paths are extracting more similar features in early layers of the network when compared to later layers.
We observe that initially both the shorter and the longer paths have relatively similar internal repre- sentations because the plots of CKA for these paths close to each other. In the later stage, see that the CKA values are significantly different from each other, which signifies that shorter paths and longer paths learning different representations in the later stage of the network. Another
8

---

Under review as a conference paper at ICLR 2022
trend that observe is, that the shorter path exhibits higher internal representation correlation with that of the final output of the neural network than the longer path. This is substantial as the CKA values of the shorter path are higher throughout compared to the CKA values of the longer path.
0.99
0.87
0.66
0.63
0.23
0.18
0.2
0.2
0.8
0.39
0.73
0.84
0.86
0.79
0.71
0.73
0.74
0.7
Pathway2
0.35
0.55
0.73
0.87
0.84
0.84
0.83
0.84
0.19
0.39
0.59
0.79
0.88
0.9
0.9
0.9
0.14
0.92
0.4
0.12
0.23
0.39
0.67
0.83
0.9
0.89
0.89
0.3
0.11
0.22
0.38
0.66
0.84
0.91
0.9
0.89
Pathway1
Figure 9: 2 3, 3x trained on ImageNette dataset showing the correlation among 8 layers deep parallel pathways. We observe that layers of the two pathways have high redundancy with their respective counterparts in the other pathway, indicating a parameter inefficiency.
Additionally, we are interested in learning how much work is shared when pathways are very deep and identical their structure. We test this by analyzing the last building block of MPNetz 3 x 3, 8 3 x 3) trained on ImageNette. Since both pathways are identical, we can compare the layers in a building block directly with their counterparts from the other pathway. This is different compared the previous experiment, where measured CKA similarity of all layers with the final output of the network. When looking at the CKA similarity heatmap of a building block in Fig. 9 we can observe that layers in the first pathway show high similarity with their respective counterparts in the other pathway, implying a high degree of redundancy in the extracted features. This could be considered parameter inefficiency.
5 DISCUSSION AND CONCLUSION
In this work, we analyzed the distribution of the inference process multipath architectures. We first investigated sequential models with skip connections like ResNet and DenseNet. For these chitectures, we find that layers are skipped when they are part of the tail pattern (see the works of Richter et al. (2021a) and Richter et al. (20215)), which can be predicted by applying the border- layer rule of Richter et al. (20216). We then moved on to architectures with multiple pathways in each building block. Our analysis suggests, that only the pathways with fewer layers are utilized if the parallel pathways differ strongly in depth. We were able to attribute this to localized van- ishing gradient problem experimentally. When the pathways are homogeneous, the behavior of the pathways changes to coexisting behavior, where both pathways improve the intermediate solution quality at a similar pace. We find that slight differences in the pathways result increasingly distinct extracted features in later layers. However, if the pathways are identical, also the extracted features become identical according to our CKA analysis.
It can be argued that the parameters of the layers in multipath architectures are not utilized fully in the heterogeneous and homogeneous case. In one scenario because of the redundancy extracted features, and in the other case because of layers not directly advancing the intermediate solution quality. Following the line of thought, architectures with parallel pathways may be prone to inef- ficiencies. This may also explain why most efficient state-of the-art architectures in recent years like ResNet. MobileNet and EfficientNet only feature skip connection, which do not suffer from this problem.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Guillaume Alain, Maxime Chevalier-Boisvers Frédéric Osterrath, and Remi Tailler. Deep- drummer Generating drum loops using deep learning and a human in the loop. CoRR, abs/2008. 04391, 2020. URL
abs /ts./220. /ttp//aas.//.//.//////// 2008.000....00 04391
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kemelss based on centered alignment. Journal of Machine Learning Research, 13(1):795-822 March 2012. ISSN 1532-4435.
Chao Dong. Chen Change Loy. Kaiming He, and Xiaoou Tang, Image super-resolution using deep convolutional networks, 2015.
Yunzhen Feng, Runtian Zhai, Di He, Liwei Wang, and Bin Dong, Transferred discrepancy: Quanti- fying the difference between representations, 2020.
Gene H. Golub and Hongyuan Zha. The canonical correlations of matrix pairs and their nu- merical computation. In Linear Algebra for Signal Processing, pp. 27-49. Springer New York, 1995. doi: 10.
URL https: .rrg/10.11000/..
185-1-4612 4228- 3
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic Learning Theory (ALT), pp. 63-77. Springer, 2005, ISBN 978-3-540-29242-5. doi: 00.1007/11564089.77
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90 (ttptt / doi org. http://...111////// 1111/1199..../ 2016.90
Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainry, Fuzziness and Knowledge- Based Systems, 06 (02):107-116. April 1998. doi: 121144//0211888599000000
URL URL pitttts://doi.crre oor/100...rg/....
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Search- ing for mobilenetv3. In 2019 JEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2. 2019, pp. 1314-1324. IEEE, 2019, doi: htthttt:: doi www.oogg111....... 10. 1109/ ICCV. 2019.
10.1109//CCV2201..00011
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2018.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neu- ral network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning (ICML). volume 97 of Proceedingy of Machine Learning Research, pp. 3519-3529. PMLR, June 2019. URL http::/ppocceee ngs .am.s.m.rr... www.mmww........mmmmmmmmmmmm. http://99//oooollloeem 9t.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL https http::::://occcccceeeeeeoo hww.eeiiiiiinnnnnnnneeeeeeeee httpp//pp..cc/ppprr///2///E// E399862d3b9d613
pdf
Jason Kuen, Xiangfei Kong, Gang Wang, and Yap-Peng Tan. Delugenets: Deep networks with efficient and flexible cross-layer information inflows, 2017.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural net- works without residuals, 2017.
Xiao-Jiao Mao, Chunhua Shen. and Yu-Bin Yang. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections, 2016.
10

---

Under review as a conference paper at ICLR 2022
Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation, 2018.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In Interna- tional Conference on Learning Representations, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Confer- ence Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 6076-6085, 2017. URL httpo/////aaaep/ppooeedo.oo.oee. htts:/c6677555775386663663636663
neurip. cc/ cc/paper/2017.
Mats L.
Richter, Justin Shenk, Wolf Byttner, Anders Arpteg, and Mikael Huss. Feature space sat- uration during training. CoRR, abs/2006.08679, 2020. URL https://arxiv. org/ abs/
0006.08677
Mats L.
Richter, Wolf Byttner, Ulf Krumnack, Ludwig Schallner, and Justin Shenk. (input) size matters for convolutional neural network classifiers. In Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter (eds.), Artificial Neural Networks and Machine Learning (ICANN), volume 12892 of Lecture Notes in Computer Science, pp. 133-144. Springer, Cham, 2021a. ISBN 978-3-030-86339-5 doi: 10. 10000/978-30000-66660 8-3-00-8633--11111
Mats L. Richter, Julius Schöning. and Ulf Krumnack. Should you go deeper? optimizing volutional neural network architectures without training by receptive field analysis. CoRR, abs/2106. 12307, 2021b. URL/sttss://axxi org/ absaabss1s200 12307
P. Robert and Y. Escoufier. A unifying tool for linear multivariate statistical methods: The rv- coefficient. Journal of the Royal Statistical Society. Series (Applied Statistics), 25(3):257-26,, 1976. ISSN 00359254, 14679876. doi: 10.2307/2347233. URL http:/ wwwp/wwwww.k jstor org/
stable/2347233
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks, 2018. URL 18C1. 04381 cite 10t.11801.04811.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks, 2015,
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. doi: 10. D0.1109/CVPRR221557995
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 20/6 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016.
doi: 10. 10.1109/CVP..201.3333
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, AAAI'17, pp. 4278 4284. AAAI Press, 2017.
Mingxing Tan and Quoe V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali-
International Conference fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6105-61 14. PMLR, 2019. URL wtw.http:// httpt::tpppoceeeeeeeeeee ngs www.llrrrrsssseee aaress/v97.. tan 19. html

---

Under review as a conference paper at ICLR 2022
Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. Image super-resolution using dense skip connec- tions. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 18809-8119 2017. doi: 10.1 20.1109/
Ümt V. Çatalyürek, Cevdet Aykanat, and Bora Uçar. On two-dimensional sparse matrix partitioning: Models, methods, and a recipe. SIAM Journal on Scientific Computing, 32(2):656-63,, January 2010. doi: 10.1137/080737770.
org/10..//.. 1137708077777
Liwei Wang. Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hoperoft. Towards understanding learning representations: To what extent do different neural networks learn the same representation, 2018.
12


Reviews:
[{"review_id": "vbhwQ0aIJ7i", "reviewer": "Reviewer_Hchc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Pros]\n\n* The paper proposes interesting new experimental framework to analyse the behaviour of learned representations in popular deep CNNs. The idea to concurrently use LRPs, representation saturation (% of dimensions required after PCA) and techniques such as Centered Kernel Alignment (CKA) to analyse post-hoc behaviour is valid\n\n* The conclusion that minor heterogeneity in network pathways leads to distinct laterr feature deeper in the network is insightful and can potentially lead to new network designs in the future\n\n[Cons]\n\n* The bulk of the experiments was performed on the hypothetical network shown in Figure 4 that was constructed to feasibly allow analysis of the effect of different paths. However, it seems as if the results are very specific to this network. There is no mention of how these findings generalise to multi-path networks in the literature. Further more, if heterogeneity/homogeneity of paths is important for representations and performance, I would have liked to see these measured in relation to popular network to better understand why a ResNet might perform better than another.\n\n* Section 3 and Figure 3 are very similar to Figure 2 from Richter et al. 2021b. The main novelty of this work appears to simply extend the experimental procedure of Richter to morre complex network architectures that go beyond sequential networks (VGG style)\n\n* I am missing what the overall conclusion of this paper is. The experiments aim to demonstrate the redundancy of featurres learned or the fact certain kernels do not learn anything meaningful when comparing short versus long-paths. However, there are no real recommendations to take these results onboard. Could these metrics be used in a data-driven way through neural architecture search or to prune layers iteratively during training? How might one use the observations of this paper when designing new methods? I feel that the observations do not lead to significant insights that may help advance the field.\n\n[Other]\n\n* I would have liked to see the results are different stages of training. Further information on the dynamics of gradient updates, feature similarity and representation saturation in different multi-path networks might help better understand the results and may point to new training schemes. For instance https://proceedings.neurips.cc/paper/2017/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf showed bottom-up convergence, leading to the idea of Freeze Training.\n\n", "summary_of_the_paper": "This is an experimental paper that seeks out to investigate information processing in multi-path networks i.e. networks such as ResNet, EfficientNet, Inception-style. The goal was to investigate how different pathways process information in such networks in order to better understand learned representations to inform new network architectures in the future. The authors used logistic regression probes (LRP) and representation saturation as metrics to power their analysis. Through the analysis of a network composed of many multi-path networks with differing number of layers or different receptive fields, the authors demonstrate that shorter pathways often dominate longer pathways. Further, if pathways are slightly difference, distinct features will be promoted in later layers.", "summary_of_the_review": "The paper is interesting and uses a suite of analytical tools to evaluate the representations learned in different multi-path network settings.\n\nHowever, in the current format, it is hard to see how the results of the experiments generalise to networks used in the literature, and which aren't directly consistent with the network used for experiments. Furthermore, the analysis is very similar to Richter et al. and the main novelty is its application to multi-path network.\n\nFor this reason, I am currently recommending Rejection", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wy4uTzCMWrc", "reviewer": "Reviewer_PRea", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The motivation is interesting that we can design better neural networks without training. Some of the experiments are interesting that they find latter layers of ResNet and DenseNet can be skipped for model efficiency and without performance drop. \n\n\nWeakness:\n1) It seems to me that the Section 3 is redundant since the message is the same as the previous paper in the citation. [1]\n\n2) I think the Section 4 should be ordered better. It is not necessary to discuss the heterogeneous and homogeneous cases since the paper mainly cares about the receptive field size and the depth of paths. Then you just analyze the blocks with different settings of receptive field size and the depth. And the results that shorter paths are dominant and the longer paths that suffers from gradient vanishing is not surprising.\n\n3) There're multiple typos and errors in the paper. For example, at the end of page 5, the $s_1$ and $s_2$ should be $k_1$ and $k_2$. I think the citation [2] in Figures 3 is wrong. For Section 4.1, it says that a building block is used with a 3 * 3 pathway with 4 layers and a 7 * 7 pathway with a single layer, however, in Figure 5, it seems the pathway with 7 * 7 kernels has 4 layers instead. \n\n4) For the homogenous case in the paper, can we have the results that only on pathway is used? \n\n\nReferences:\n[1] Mats L. Richter, Julius Schoning, and Ulf Krumnack. Should you go deeper? optimizing con-volutional neural network architectures without training by receptive field analysis.\n\n[2] Guillaume Alain, Maxime Chevalier-Boisvert, Frederic Osterrath, and Remi Piche-Taillefer. Deep- drummer : Generating drum loops using deep learning and a human in the loop.", "summary_of_the_paper": "This paper analyzes the distribution of information processing in multi-path networks (including skip-connection models such as ResNet and DenseNet). They apply logistic regression on the hidden layers, namely logistic regression probes, to track the progress of the intermediate solution quality, and they analyze in which condition (depth of the path and receptive field size) the neural network prefers to skip the paths. They also measure the CKA similarity of the hidden representations learned by the multi-path model when the paths are homogeneous. They claim that with their analysis, later layers in ResNet and DenseNet can be skipped as pruning due to the unproductive layers. In addition, they find that for multi-path model, the shorter path is dominant when the depth of the paths are very different, and when the pathways are homogeneous, the behavior of the pathways changes to a coexisting behavior.", "summary_of_the_review": "Although the motivation of the paper is interesting, the analysis results of the paper is weak and not that surprising. In addition, part of the experiments are redundant since it is mentioned in the previous paper. There's little new insights or takeaways from the analysis. As a result, I recommend to reject the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LI37povzGv", "reviewer": "Reviewer_ac9X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has following strengths:\n1.\tStudying multi-path networks with respect to receptive field and depth is an interesting problem.\n2.\tThe paper is well-written and motivation is very good.\n\nThe paper has following weaknesses:\n1.\tUnfortunately, the conclusions that the authors could draw from their study are quite limited and, in most cases, completely obvious (or already established by previous papers). For instance, their first contribution in introduction: input size vs. receptive field size (as cited by the authors themselves) was already published in Richter et al. 2021a [R1] and 2021b [R2]. Other discussions about layers getting completely skipped in ResNets and variants is extremely well-established at this point (see also the third point below). The second contribution in introduction on preferring shorter pathway over longer pathway due to vanishing gradients: At first, this is also obvious since many papers (starting with the ResNet paper [R3] itself) have established that consecutive convolution layers without skip connections will result in diminishing gradients. However, these gradient properties can be modified if each branch in the multipath network further had skip connections within individual branches too. In such more complex cases, the conclusions of this study will not hold. The third contribution is also completely obvious, and the reviewer is not sure what else one can expect. If both branches in the multipath network are identical, then of course they will move the solution at a similar pace and will have high redundancy.\n2.\tAnother concerning issue is that it is not immediately clear if the findings in this paper will apply to more complex real-world problems or how to practically use the knowledge from this paper. The authors have conducted all experiments on CIFAR-10 and ImageNette datasets. The reviewer understands that doing such studies require a lot of ablations and time-/resource-consuming experiments. However, questions like “would a network more effectively utilize a shallower, higher receptive field branch over a deeper, low receptive field branch?” should also depend on the complexity of the problem itself. CIFAR-10 is a toy dataset, and while ImageNette has a larger input size, the creators of that dataset still classify it as a very simple dataset (see: “Imagenette is a subset of 10 easily classified classes from Imagenet” on https://github.com/fastai/imagenette). So, the reviewer is left wondering if this analysis will hold for more complex datasets (say CIFAR-100, tiny imagenet, etc.)? In the current form, the study seems inconclusive.\n3.\tThe reason why many of the conclusions appear obvious to the reviewer (particularly anything related to deeper pathways not being used in DNNs over shorter pathways when skip connections are used) is because of the existing literature. Two very important papers in this direction have not been discussed. Veit et al. already established long ago that ResNets behave like an ensemble of shallower networks [R4]. More recently, Bhardwaj et al. theoretically and empirically discussed the gradient flow properties through network topologies with skip connections [R5]. They derive theory on DenseNet-type models but show empirical results for MobileNets/ResNets as well. These works must be discussed in detail.\n\n[R1] Mats L. Richter, Wolf Byttner, Ulf Krumnack, Ludwig Schallner, and Justin Shenk. (input) size matters for convolutional neural network classifiers. In Igor Farkaˇs, Paolo Masulli, Sebastian Otte, and Stefan Wermter (eds.), Artificial Neural Networks and Machine Learning (ICANN), volume 12892 of Lecture Notes in Computer Science, pp. 133–144. Springer, Cham, 2021a.\n\n[R2] Mats L. Richter, Julius Sch¨oning, and Ulf Krumnack. Should you go deeper? optimizing convolutional neural network architectures without training by receptive field analysis. CoRR, abs/2106.12307, 2021b. URL https://arxiv.org/abs/2106.12307.\n\n[R3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770–778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90. URL https://doi.org/10.1109/CVPR.2016.90.\n\n[R4] Veit, Andreas, Michael J. Wilber, and Serge Belongie. \"Residual networks behave like ensembles of relatively shallow networks.\" Advances in neural information processing systems 29 (2016): 550-558.\n\n[R5] Bhardwaj, Kartikeya, Guihong Li, and Radu Marculescu. \"How does topology influence gradient propagation and model performance of deep networks with DenseNet-type skip connections?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n", "summary_of_the_paper": "The paper studies information processing in (1) networks with skip connection like ResNets and DenseNets, and (2) multi-path networks where the model has multiple branches with homogeneous or heterogeneous layers. The study focuses on problems like: (a) does learning depend on receptive field and/or depth of various branches, (b) how features differ in representation at different stages of the network", "summary_of_the_review": "The paper seems like a good first attempt towards an important and difficult problem. However, most of the conclusions seemed obvious or inconclusive (e.g., since the paper does not account for “how difficult a given problem is”, it is not easy to conclude how much receptive field size we really need over depth, etc.). More comprehensive evaluation on difficult datasets (not necessarily large datasets) is needed for a better contribution. Note that, most statements made in the paper themselves are correct and well-supported. The reviewer has concerns about the claims being obvious.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes the flow of information in convolutional neural networks with parallel pathways by using logistic regression probes and similarity indices. 
Following the analysis, the authors concluded that:
- pathways of similar size have similar contributions to learning and have high redundancy
- shorter pathways directly improve solution quality to a greater extent than longer pathways
- pathways of different lengths also lead to grater variety amid features in the 'downstream' layers of the network

The novelty in this type of analysis and its thoroughness was appreciated by the reviewers. The insight about the benefits of pathways of different length is also valuable; although there is a sense in the community that long pathways without skip connections bring diminishing returns, as pointed by reviewer ac9X, there is still some benefit in quantifying it through such an analysis and establishing the correct mix of long/short pathways. [note: This paper is not there yet, but has the potential.]

On the other hand, the experiments were performed on a single network, a network selected such that this instrumentation is possible, so there is the issue of whether these conclusions generalize, as pointed out by reviewer Hchc.

There were other comments, in terms of structure, errors and typos, as pointed out by reviewer PRea. 

The authors have not responded to the comments, nor updated their manuscript.
In its current form, the paper is not ready for acceptance.
Decision: Reject