Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents a new CNN architecture: CrescendoNet. It does not have skip connections yet performs quite well.

Overall, I think the contributions of this paper are too marginal for acceptance in a top tier conference.

The architecture is competitive on SVHN and CIFAR 10 but not on CIFAR 100. The performance is not strong enough to warrant acceptance by itself.

FractalNets amd DiracNets (https://arxiv.org/pdf/1706.00388.pdf) have demonstrated that it is possible to train deep networks without skip connections and achieve high performance. While CrescendoNet seems to slightly outperform FractalNet in the experiments conducted, it is itself outperformed by DiracNet. Hence, CrescendoNet does not have the best performance among skip connection free networks.

You claim that FractalNet shows no ensemble behavior. This is clearly not true because FractalNet has ensembling directly built in, i.e. different paths in the network are explicitly averaged. If averaging paths leads to ensembling in CrescendoNet, it leads to ensembling in FractalNet. While the longest path in FractalNet is stronger than the other members of the ensemble, it is nevertheless an ensemble. Besides, as Veit showed, ResNet also shows ensemble behavior. Hence, using ensembling in deep networks is not a significant contribution.

The authors claim that "Through our analysis and experiments, we note that the implicit ensemble behavior of CrescendoNet leads to high performance". I don't think the experiments show that ensemble behavior leads to high performance. Just because a network performs averaging of different paths and individual paths perform worse than sets of paths doesn't imply that ensembling as a mechanism is in fact the cause of the performance of the entire architecture. Similary, you say "On the other hand, the ensemble model can explain the performance improvement easily." Veit et al only claimed that ensembling is a feature of ResNet, but they did not claim that this was the cause of the performance of ResNet.

Path-wise training is not original enough or indeed different enough from drop-path to count as a major contribution.

You claim that the number of layers "increase exponentially" in FractalNet. This is misleading. The number of layers increases exponentially in the number of paths, but not in the depth of the network. In fact, the number of layers is linear in the depth of the network. Since depth is the meaningful quantity here, CrescendoNet does not have an advantage over FractalNet in terms of layer number. Also, it is always possible to simply add more paths to FractalNet if desired without increasing depth. Instead of using 1 long paths, one can simply use 2, 3, 4 etc. While this is not explicitly mentioned in the FractalNet paper, it clearly would not break the design principle of FractalNet which is to train a path of multiple layers by ensembling it with a path of fewer layers. CrescendoNets do not extend beyond this design principle.

You say that "First, path-wise training procedure significantly reduces the memory requirements for convolutional layers, which constitutes the major memory cost for training CNNs. For example, the higher bound of the memory required can be reduced to about 40% for a Crescendo block with 4 paths where interval = 1." This is misleading, as you need to store the weights of all convolutional layers to compute the forward pass and the majority of the weights of all convolutional layers to compute the backward pass, no matter how many weights you intend to update. In a response to a question I posed, you mentioned that we you meant was "we use about 40% memory for the gradient computation and storage". Fair enough, but "gradient computation and storage" is not mentioned in the paper. Also, the reduction to 40% does not apply e.g. to vanilla SGD because the computed gradient can be immediately added to the weights and does not need to be stored or combined with e.g. a stored momentum term.

Finally, nowhere in the paper do you mention which nonlinearities you used or if you used any at all. In future revisions, this should be rectified.

While I can definitely imagine that your network architecture is well-designed and a good choice for image classification tasks, there is a very saturated market of papers proposing various architectures for CIFAR-10 and related datasets. To be accepted to ICLR, either outstanding performance or truly novel design principles are required.
review_title: Good network that is not sufficiently different from FractalNet and underperforms DiracNet

final_score: rating: null
review_content: Dear authors,

you say that path-wise training can reduce memory usage by 40%. How is this? If I freeze the weights in all but one path, I still need to have those weights in memory to compute the forward and backward pass of the network. Just because the weights in some layers aren't updated doesn't mean they aren't needed in memory to compute the activations and gradients of other weights.

Thanks,
review_title: Another reviewer question

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors propose  a new network architecture, CrescendoNet, which is a simple stack of building blocks without residual connections. To reduce the memory required for training, the authors also propose a path-wise training procedure based on the independent convolution paths of CrescendoNet. The experimental results on CIFAR-10, CIFAR-100 and SVHN show that CrescendoNet outperforms most of the networks without residual connections.
 
Contributions:

1 The authors proposed Crescendo block that consists of convolution paths with increasing depth. 

3 The authors conducted experiments on three benchmark datasets and show promising performance of CrescendoNet .

3 The authors proposed a path-wise training procedure to reduce memory requirement in training.

Negative points:

1 The motivation of the paper is not clear. It is well known that the residual connections are important in training deep CNNs and have shown remarkable performance on many tasks. The authors propose the CrescendoNet which is without residual connections. However, the experiments show that CrescendoNet is worse than ResNet. 

2  The contribution of this paper is not clear. In fact, the performance of CrescendoNet is worse than most of the variants of residual networks, e.g., Wide ResNet, DenseNet, and ResNet with pre-activation. Besides, it seems that the proposed path-wise training procedure also leads to significant performance degradation.

3 The novelty of this paper is insufficient. The CrescendoNet is like a variant of the FractalNet, and the only difference is that the number of convolutional layers in Crescendo blocks grows linearly.

4 The experimental settings are unfair. The authors run 700 epochs and even 1400 epochs with path-wise training on CIFAR, while the baselines only have 160~400 epochs for training.

5 The authors should provide the experimental results on large-scale data sets (e.g. ImageNet) to prove the effectiveness of the proposed method, as they only conduct experiments on small data sets, including CIFAR-10, CIFAR-100, and SVHN.


6 The model size of CrescendoNet is larger than residual networks with similar performance.


Minor issues:

1In line 2, section 3.4, the period after “(128, 256, 512)” should be removed.

review_title: The authors propose  a new network architecture without residual connections. The motivation of the paper is not clear. The contribution of the paper is not sufficient.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new convolutional network architecture, which is tested on three image classification tasks.

Pros:
The network is very clean and easy to implement, and the results are OK.

Cons:
The idea is rather incremental compared to FractalNet. The results seem to be worse than existing networks, e.g., DenseNet (Note that SVHN is no longer a good benchmark dataset for evaluating state-of-the-art CNNs). Not much insights were given.

One additional question: Skip connections have been shown to be very useful in ConvNets. Why not adopt it in CrescendoNet? What's the point of designing a network without skip connections?

review_title: review

final_score: rating: null
review_content: Dear authors,

Thank you for answering my 2 questions. I just posted my review. I am curious: Can you see the review? Because when I log out of my account, I can no longer see it. Hence, the review isn't public. I am wondering whether at least you can see it.

Thanks,
review_title: Can you see the review?

final_score: rating: null
review_content: Dear authors,

I would like to confirm you is that you do not use any ReLU or other nonlinearities at all in the convolutional part of the network. Also, how many fully-connected layers do you use and what, if any, nonlinearities do you use before, between or after them?

Also, how did you initialize the weights in each layer?

Thanks,
review_title: Reviewer question

Meta-review: The paper proposes a new convolutional network architecture, called CrescendoNet. Whilst achieving competitive performance on CIFAR-10 and SVHN, the accuracy of the proposed model on CIFAR-100 is substantially lower than that of state-of-the-art models with fewer parameters; the paper presents no experimental results on ImageNet. The proposed architecture does not provide clear new insights or successful new design principles. This makes it unlikely the current manuscript will have a lot of impact.
Decision: Reject