Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary
---
This paper proposes to learn simultaneously all the parameters of grouped convolutions by factorizing the weights of the convolutions as a sum of lower rank tensors. This enables architecture search for the convolution parameters in a differentiable and efficient way.

Comments
---
I think was paper is well written, and was clear at least until 3.2. I believe some clarifications could be useful here, it is not written clearly that t' and u', the 2 first dimensions of the core are R times smaller than t and u. There is some explanation in the bracket (3) but 1) it should be stated clearly in the text, 2) I believe there are several typos on the 4th line of bracket (3) making it hard to understand.

I did not know about the expansion function, and while I trust the authors that it is correctly used, I would have like either more explanations on how it works or some reference.

Can you justify the softmax and the very high temperature? For N = 8, s_1 will be sampled 98.2% of the time s_2 1.8% and the other sampling probabilities are close to neglibigle. While I understand it seems to work better in practice, it looks extremely aggressive.

In 4.4 you say you perform finetuning for 150 epochs, which is huge, while on the abstract you said "GroSS represents a significant step towards
liberating network architecture search from the burden of training and finetuning". Can you comment?

As you say GroSS is an alternative to NAS (for the convolutions parameters that is), is the GroSS method proposed really faster and more accurate than a NAS baseline for finding these architectures?

I don't find the column titles in Table 3 to be always informative. "After train" means after the finetuning? I took me some time to realize the delta was the delta in accuracies, it is not very informative and it was not clear for me for some time what it meant. Either the titles should be chosen more carefully or the caption should be more precise I believe.

In figure 1, the legend should be more informative, at least incorporate a "alpha" or "temperature" title in the legend.

Conclusion
---
While the method is interesting I am wondering whether GroSS enables more efficient architecture search that tradional methods as there is still a long finetuning step, furthermore it can only be applied to grouped convolutions parameters. As the authors present it in the abstract and introduction as an alternative to NAS, I believe a comparison to a NAS would be needed.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary: The authors introduce GROSS---a reformulation of block tensor decomposition, which allows multiple grouped convolutions (with varying group sizes) to be trained simultaneously. The basic idea is to reformulate the BTD so that higher-order decompositions can be expressed as functions of lower-order decompositions. Given this nesting, it is possible to implicitly train the lower-order decompositions while training the higher-order ones. 

The authors frame this contribution as a form of "neural architecture search" (NAS), arguing that this allows researchers to simultaneously train grouped-convolution CNNs with varying group sizes. After the simultaneous training based on the GROSS approach, the researcher can then select the group size that gives the best performance/accuracy tradeoff. The selected model can be further fine-tuned on the task, and the authors found that this improved performance. 

Empirical results on the CIFAR-10 dataset show that the proposed approach performs as expected, allowing for the simultaneous training of CNNs with grouped convolutions of varying orders. The results show the the proposed approach can find "better" solutions than a simple search over fixed architectures. 

Assessment: Overall, this is a well-written and soundly derived contribution. However, it is quite niche and---while the authors frame it as a form of NAS---in my view, this contribution is more in the realm of hyperparameter search for grouped convolutions, and not NAS in general. I would recommend reframing the introduction to make this fact more explicit, as the approach does not provide a general strategy for differentiable NAS.  In addition, the empirical results are relatively shallow, with only one dataset and without detailed discussion of the variance of the results.

Reasons to accept:
- Well-written
- Sound and well-motivated algorithm
- Potential applications in cases where grouped convolutions are useful
- Empirical results demonstrate validity of the proposed approach

Reasons to reject:
- Relatively niche contribution incorrectly framed as general contribution to NAS
- Limited empirical analysis (e.g., only one dataset). 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose to express the weight of a convolutional neural network as a coupled Tucker decomposition. The Tucker formulation allows for an efficient reformulation. The weigths of the sum of Tucker decompositions allows is randomly set at each iteration during training, with each term of the sum having a different rank.
                                                                                                                                                                                                            
The method is interesting, however the novelty is low. There is already large bodies of work on parametrizing neural networks with tensor decomposition, including coupled decomposition.


How does the proposed method compared to the related method DART? And to a simple coupled decomposition?
How is the method different to training several network with the same Tucker parametrization but different ranks? What about memory and computational efficiency?
In any case, these should be compared to.

The notation should be kept consistent throughout: e.g. either use t,u,v,w or d1,d2,d3,d4. Notation should be unified in the text and captions (e.g. Table 1).
In 3.2, when specifying the size of G, should it be G_r? Same for B and C.

Why the convolution R should be grouped? Should it not be a regular convolution?

For ot', u' being the group-size, what is o? It was not introduced.

The response reconstruction is only useful if the same, uncompressed network is already trained, would not be applicable for end-to-end training.

The model is a simple 4-layer network, not fully described. An established architecture, such as ResNet should be employed.

Experiments should be carried on ImageNet, or at least not just on CIFAR10.

There is no comparison with existing work, e.g. parametrization of the network with Tucker [1, 2] or CP [3]

[1] Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications, ICLR 2016
[2] T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor, CVPR 2019
[3] Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition, ICLR 2015

review_title: Official Blind Review #3

Meta-review: The authors use a Tucker decomposition to represent the weights of a network, for efficient computation. The idea is natural, and preliminary results promising. The main concern was lack of empirical validation and comparisons. While the authors have provided partial additional results in the rebuttal, which is appreciated, a thorough set of experiments and comparisons would ideally be included in a new version of the paper, and then considered again in review.

Decision: Reject