Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper proposed a sparse-complementary convolution as an alternative to the convolution operation in deep networks. In this method, two new types of kernels are developed, namely the spatial-wise and channel-wise sparse-complementary kernels. The authors argue that the proposed kernels are able to cover the same receptive field as the regular convolution with almost half the parameters. By adding more filters or layers in the model while keeping the same FLOPs and parameters, the models with the proposed method outperform the regular convolution models. The paper is easy to follow and the idea is interesting. However, the novelty of the paper is limited and the experiments are not sufficient.

Strengths:
1. The authors proposed the sparse-complementary convolution to cover the same receptive field as the regular convolution. 

2. The authors implement the proposed sparse-complementary convolution on NVIDIA GPU and achieved competitive speed under the same computational load to regular convolution.

3. The authors demonstrated that, given the same resource budget, the wider networks with the proposed method are more efficient than the deeper networks due to the nature of GPU parallel mechanism.

Weak points:

1. The novelty of this paper is limited. The main idea is to design complementary kernels that cover the same receptive field as the regular convolution. However, the performance improvement is marginal and may come from the benefit of wide networks rather than the proposed complementary kernels. Moreover, the experiments are not sufficient to support the arguments. For example, how is the performance of a model containing SW-SC or CW-SC without deepening or widening the networks? Without such experiment, it is unclear whether the improved performance comes from the sparse-complementary kernels or the increased number of kernels.

2. The relationship between the proposed spatial-wise kernels and the channel-wise kernels is not very clear. Which kernel is better and how to choose between them in a deep network? There is no experimental proof in the paper.

3. The proposed two kernels introduce sparsity in the spatial and channel dimension, respectively. The two methods are used separately. Is it possible to combine them together?

4. The proposed method only considers the “+-shape” and “x-shape” sparse pattern. Given the same receptive field with multiple complementary kernels, is the kernel shape important for the training? There is no experimental result to verify this.

5. As mentioned in the paper, there are many methods which introduce sparsity in the convolution layer, such as “random kernels”, “low-rank approximated kernels” and “mixed-shape kernels”. However, there is no experimental comparison with these methods.

6. In the paper, the author mentioned another sparse-complementary baseline (sc-seq), which applies sparse kernels sequentially. It yields smaller receptive field than the proposed method when the model depth is very small. Indeed, when the model goes deeper, the receptive field becomes very close to that of the proposed method. In the experiments, it is strange that this method can also achieve comparable or better results. So, what is the advantage of the proposed “sc” method compared to the “sc-seq” method?


8. Figure 5 is hard to understand. This figure only shows that training shallower networks is more effective than training the deeper networks on GPU. However, it does not mean training the wider networks is more efficient than training the deeper ones.

review_title: This paper proposed a sparse-complementary convolution as an alternative to the convolution operation in deep networks. The paper is easy to follow and the idea is interesting. However, the novelty of the paper is limited and the experiments are not sufficient.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 

This paper presented interesting ideas to reduce the redundancy in convolution kernels. They are very close to existing algorithms.

(1)	The SW-SC kernel (Figure 2 (a)) is an extension of the existing shaped kernel (Figure 1 (c)).
(2)	The CW-SC kernel (Figure 2 (c)) is very similar to interleaved group convolutions. The CW-SC kernel can be regarded as a redundant version of interleaved group convolutions [1]. 

I would like to see more discussions on the relation to these methods and more strong arguments for convincing reviewers to accept this paper. 

[1] Interleaved Group Convolutions. Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. ICCV 2017. http://openaccess.thecvf.com/content_ICCV_2017/papers/Zhang_Interleaved_Group_Convolutions_ICCV_2017_paper.pdf
review_title: Very close to mixed-shaped kernels and interleaved group convolutions

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces a new design of kernels in convolutional neural networks. The idea is to have sparse but complementary kernels with predefined patterns, which altogether cover the same receptive field as dense kernels. Because of the sparsity of such kernels, deeper or wider networks can be designed at the same computational cost as networks with dense kernels.

Strengths:
- The complementary kernels come at no loss compare to standard ones
- The resulting wider networks can achieve better accuracies than the original ones

Weaknesses:
- The proposed patterns are clear for 3x3 kernels, but no solution is proposed for other dimensions
- The improvement over the baseline is not very impressive
- There is no comparison against other strategies, such as 1xk and kx1 kernels (e.g., Ioannou et al. 2016)

Detailed comments:
- The separation into + and x patterns is quite clear for 3x3 kernels. However, two such patterns would not be sufficient for 5x5 or 7x7 kernels. This idea would have more impact if it generalized to arbitrary kernel dimensions.

- The improvement over the original models are of the order of less than 1 percent. I understand that such improvements are not easy to achieve, but one could wonder if they are not due to the randomness of initialization/mini-batches. It would be more meaningful to report average accuracies and standard deviations over several runs of each experiment.

- Section 4.4 briefly discusses the comparison with using 3x1 and 1x3 kernels, mentioning that an empirical comparison is beyond the scope of this paper. To me, this comparison is a must. In fact, the discussion in this section is not very clear to me, as it mentions additional experiments that I could not find (maybe I misunderstood the authors). What I would like to see is the results of a model based on the method of Ioannou et al, 2016 with the same number of FLOPS.

- In Section 2, the authors review ideas of so-called random kernel sparsity. Note that the work of Wen et al., 2016, and that of Alvarez & Salzmann, NIPS 2016, do not really impose random sparsity, but rather aim to cancel out entire kernels, thus reducing the size of the model and not requiring implementation overhead. They also do not require pre-training and re-training, but just a single training procedure. Note also that these methods often tend not to decrease accuracy, but rather even increase it (by a similar magnitude to that in this paper), for a more compact model.

- In the context of random sparsity, it would be worth citing the work of Collins & Kohli, 2014, Memory Bounded Deep Convolutional Networks.

- I am not entirely convinced by the discussion of the grouped sparsity method in Section 3.1. In fact, the order of the channels is arbitrary, since the kernels are learnt. Therefore, it seems to me that they could achieve the same result. Maybe the authors can clarify this?

- Is there a particular reason why the central points appears in both complementary kernels (+ and x)?

- Why did the authors change the training procedure of ResNets slightly compared to the original paper, i.e., 50k training images instead of 45k training + 5k validation? Did the baseline (original model) reported here also use 50k? What would the results be with 45k?

- Fig. 5 is not entirely clear to me. What was the width of each layer? The original one or the modified one?

- It would be interesting to report the accuracy of a standard ResNet with 1.325*width as a comparison, as well as the runtime of such a model.

- In Table 4, I find it surprising that there is an actual speedup for the model with larger width. I would have expected the same runtime. How do the authors explain this? 

review_title: An interesting idea, but generalization is not clear and the experiments not entirely convincing

Meta-review: The paper studies factorizations of convolutional kernels. The proposed kernels lead to theoretical and practical efficiency improvements, but these improvements are very, very limited (for instance, Figure 5). It remains unclear how they compare to popular alternative approaches such as group convolutions (used in ResNeXt) or depth-separable convolutions (used in MobileNet). The reviewers identify a variety of smaller issues with the manuscript.
Decision: Reject