Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes to modify a standard CNN by requiring all of its layers to share the same filter set, essentially allowing it to be expressed as an iterative (or recurrent) network.  This also has the effect of forcing the same number of feature channels to be used throughout the network.  For ResNet-like architectures with bottleneck blocks, sharing occurs at the level of the block (3 conv layers in series that are repeated).  Another variant of the sharing pattern inserts unshared 1x1 convolutional layers after shared layers or blocks; this adds some flexibility while still reducing parameters compared to standard CNNs.

On CIFAR-10, CIFAR-100, and Tiny ImageNet, experiments demonstrate the ability of the sharing scheme to reduce parameters without impacting accuracy (or more drastically reduce parameters at the cost of accuracy) (Tables 1ab, 2a).

However, results are less compelling on ImageNet (Table 2b), where SL-ResNet-50 and SL-ResNet-34 are both less accurate than the baseline standard ResNets as well as ShaResNet [Boulch, 2018].  The accuracy gap between SL-ResNet and ResNet on ImageNet (Table 2b) is significant (approx 5% Top-1 and 2% Top-5 accuracy) and might make it difficult to justify use of the proposed method in this setting.  As ImageNet is the most challenging of the datasets used, this is cause for concern.

There is also a major concern with respect to novelty and related work.  Unfortunately, the paper appears to have completely missed the following highly related publication from ICLR 2019:

Learning Implicitly Recurrent CNNs Through Parameter Sharing
Pedro Savarese, Michael Maire
ICLR 2019

This prior work proposes a network structure in which a set of L layers share a set of k parameter templates.  The templates and sharing coefficients are learned as part of the standard training procedure.  This prior work demonstrates both parameter savings and accuracy improvements when training networks in this manner.  Additionally, this prior work shows that some learned networks can be converted into explicitly recurrent forms as a post-processing step.

The paper under review appears be a special case of this prior work with the number of templates k = 1 (shared between all layers).  It is possible this is an important special case, worthy of significant attention on its own.  Notably, [Savarese and Maire, 2019] considered sharing across at most all layers within the same stage of a residual network, rather than all layers in the network.  However, arguing for the importance of this special case would require focused experimental comparison and analysis, which is not present in the current version of the paper.

Novelty is clearly limited in light of this overlooked prior work.  At minimum, citation, discussion, and experimental comparison to the above ICLR 2019 paper is necessary.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents an approach to reduce the number of a neural network by sharing the convolutional weights among layers. To convert the first layer into the right number of features padding is used. The last layer I suppose is instead a normal classifier on the fully connected representation (for VGG) or on the average pooling(for ResNet). Results on different datasets and architectures show that the proposed approach can highly compress the number of needed parameters with a minimal reduction of the network test accuracy.

I lean to reject this paper because, in my opinion is very similar to ("Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex" Qianli Liao and Tomaso Poggio), which is not mentioned in related work. This paper, published in 2016 was already proposing the idea of reducing the number of parameters of ResNet by sharing the weights of each layer and therefore consider ResNet with shared weights as a recurrent net.
In this paper the setting are slightly different, authors add also a variant with additional 1x1 convolutions and show also results with additional compression. However, in my opinion, the main idea is the sharing of convolutional weights, and this is not new.


Additional Comments:
- This paper considers only the number of learnable parameters of a network. However, in many cases, for applications, it is more important to save memory (which is not the case as the activations should still be saved for backpropagation) and computation. In my understanding the final computation of the model is actually increased because it uses more channels at lower layers (which corresponds to high resolution features maps). Authors should comment about that.
- In section 4, VGGNet-like Architectures and ResNet-like architectures the authors mention a baseline E-VGGNet or E-ResNet with exactly the same architecture as the shared weights network (thus same number of channels at each layer), but without sharing. However I could not find the performance of that interesting baseline in the results.





review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose to use the *same* convolutional layer in every layer of a DNN. The network effectively is converted into repeatedly applying the same convolutional filter at multiple scales. The idea is motivated by wavelet decompositions and related work. The authors show that by repeatedly applying the same filter, the number of parameters that need to be stored for a model reduces proportionally to the depth of the network. At the same time, experimental evidence is provided that the performance of these models is not affected, when compared to the baseline (full) model. 

COMMENTS:
- the paper is well written, but overly verbose. there are several areas where the explanation can be compressed, making room to add more informative details (which are in the appendix), or increasing the size of the figures (which are too small)

- the paper seems lacking a bit in experiments. If the authors can show that the same filter applied L times achieves about the same performance, why not also experiment with different L? i.e. does VGGNet actually need L layers? what if only 2 layers are used? this will help with the overparametrization problem as well. 

- Figures 3 and 4 are hard to read. Please increase their size.

- page 5 line 2: how does padding the input with (n-3) empty channels affect performance? If you're learning the filters through backprop, will they not always be learning to fit to 0 ? or am i missing something? 

- along the above lines, why not have the input layer to be a different filter with 3 channels and then have a common filter for all upstream layers?

- in multiple places in the text, you refer to the number of "independent" parameters. I dont see why the parameters need to be independent. Unless there's some orthogonalization happening at the weights, calling them independent is incorrect. 

- paragraph above sec4: you add separate "linear" layers for the 'SL' models. Can you describe how many addditional parameters you have to learn?
review_title: Official Blind Review #2

Meta-review: This submission proposes an interesting experiment/modification of CNNs. However, it looks like this contribution overlaps significantly with prior work (that the authors initially missed) and the comparison in the (revised) manuscript seem to not clearly delineate and acknowledge the similarities and differences.

I suggest the authors improve this aspect and try submitting this work to next venue. 
Decision: Reject