Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to manually design a new 12-depth CNN architecture ParNet based on parallel subnetworks instead of traditionally deeply stacked blocks. Experiments show that ParNet is the first CNN achieving over 80% accuracy on ImageNet with 12 depth only. ParNet also achieves a competitive AP of 48% on MS-COCO for object detection.
main_review: 1) My major concern is that the authors argues contribution of parallel subnetworks, while I think the traditional RepVGG-wise blocks adopted in ParNet contributes significantly more to prediction accuracy. Even if ParNet adopts 1 branch only, ParNet's accuracy can still achieve 75% more, as "ParNet-M-OneStream" shown in Table 8. Comparison of Table 7 and Table 10 shows that the RepVGG blocks is significantly more important than adopted number of branches on improving accuracy. 

2) As shown in Table 2, ParNet-L/ParNet-XL (12-depth) indeed has fewer depth compared to ResNet50 (50-depth). However, ParNet-L (54.9M)/ParNet-XL (85.0M) has significantly more parameters compared to ResNet50 (25.6M). A potential problem is that if ParNet runs on low-power computing hardware with limited cuda cores, I am not sure whether ParNet still has the advantage on speed compared to ResNet50.

3) The paper proposes a new block RepVGG-SSE by introducing SSE into traditional RepVGG, while ParNet's accuracy improvement from SSE is only 1.53%, as shown in Table 7. Therefore, I think contribution of RepVGG SSE is not sufficient enough.
summary_of_the_review: The authors argues contribution of parallel subnetworks, while the experiments show that the traditional RepVGG-wise blocks adopted in ParNet contributes significantly more to prediction accuracy. Therefor, my rating is "5: marginally below the acceptance threshold".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: - **Motivation**. The paper argues that deep networks have several limitations
  - (a) deep nets have a higher latency;
  - (b) deep nets are hard to parallelize;
  - (c) deep nets are not suitable for applications.


- **Method**.
Motivated by these observations, the paper aims to fill the performance gap between shallow networks and deep networks. The paper proposed a 12-layer shallow model framework, which contains proposed RepVGG-SSE blocks, fusion modules for multi-scale processing, and parallel streams.  


- **Experiments**.
The proposed model, ParNet, is verified on CIFAR-10, CIFAR-100, and ImageNet for classification, MS-COCO for detection. 
main_review: I have four considerations:
- (a) Table 2 only shows the details for ParNet-L and -XL. It could be better to show the full details and comparisons of ParNet-S, -M, -L, and -XL.
- (b) What are the details of speed testing in Table 2? Do the ResNet and ParNet use the same setting for speed benchmark? If ParNet uses multiple GPUs in parallel to benchmark the inference speed, e.g., 2 GPUs with 128 global batch size, does the ResNet uses 2 GPUs within the global batch size of 128 as well? It could be better to add more words in the third paragraph on Page 6 to show how to conduct the experiments. 
- (c) ParNet develops the RepVGG-SSE based on the work of RepVGG, CVPR 2021. But there is no comparison with this high-related work. It could be better to compare with RepVGG in Table 1 and Table 2.
- (d) The paper argues that one of the advantages is parallel. But according to Table 2 and RepVGG's Table 4, ParNet has more parameters and larger FLOPs than the competitors. More parameters mean that the model will cost more memory during training and inference. This fact fades the significance of ParNet. It could be better to discuss this and highlight why parallelism is more important than the other two aspects for selling the work.
summary_of_the_review: Please erase the above concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a way of designing a network fixing the depth yet involving more branches. The authors try to support the claim which is that low-depth networks can achieve a good result by providing experiments on the ImageNet and the CIFAR datasets
main_review: Pros)
+ This paper is easy to follow.
+ The concept of branch-parallelism in a model looks promising.

Cons)
- I am agreeing that the network depth is a crucial design element towards model efficiency but the computational costs in including the number of parameters, FLOPs, and memory footprints seem to be overlooked in this paper: the proposed model has a larger # of parameters and FLOPs, so the overall computational costs are bigger than the baseline models. 
- The architectural design with the proposed building block in Figure 2 is presented without providing any intuitions or design philosophy.
- The comparisons with the baselines in model-parallelism is not sufficient

Some comments and questions)
- The model-parallelism can be done with a single-branch model in training or inferring with multiple images (i.e., with a larger mini-batch size), then the speed gain from the branch-parallelism scheme may vanish. Please clarify a multi-branch architecture also has an advantage over the widened networks such as WideResNet in this case.

- The performance comparison should be done fairly. For example in Table 2, ParNet-L is compared with much smaller networks such as R34 and R50 in terms of the computational budgets, so this is not fair. The models with widening the width while fixing depth (e.g., Wide ResNet) can achieve much better accuracy barely increase the latency.

- Memory consumption would be a matter for the proposed architecture. Please specify the memory footprints when training (or inference) with the fixed batchsize.

- ResNets can also be leveraged fusing methods including skip connections and BNs. Did the authors compare the latency with these fused ResNets with the proposed ParNets in Table 2?

- Why vanilla SE-block is not suitable for the proposed model? Why Rep-VGG block has been adopted?

- Please specify why ParNets cannot outperform DenseNet-100 even using more parameters on the CIFAR datasets in Table 6. I am just wondering whether there is a different earning behavior of a shallow network on a particular dataset.
summary_of_the_review: - See the main review
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new architecture of a convolutional neural network for image classification. The authors are motivated by inference efficiency and showing that networks with about a dozen of layers can be competitive in classification accuracy with 50 layers and more. So instead of growing depth, they propose to grow width and have multiple subnetworks of the same depth within one model. The proposed architecture is evaluated on CIFAR, ImageNet and COCO classification and detection tasks, and shows that networks with 12 layers can be competitive with deeper counterparts.
main_review: Strength:
- empirical result that low-depth networks can be competitive with deeper models is significant. Confirming observations in prior works, the number of parameters and computational complexity of low-depth networks need to be much higher to match the accuracy of deeper networks.

Weaknesses:
- architecture complexity: the authors claim outperforming ResNet in efficiency, but do not mention that the architecture is far more complex. It is also more complex than RepVGG. This needs to be stressed in the introduction. This is separate from computational complexity.
- weak baseline: authors use Squeeze-and-Excitation blocks in their architecture which is known to improve ResNet performance, yet compare to ResNet without SE layers (section 4, tables 1-2)
- missing baseline: the authors introduce parallel streams in their network, which might be working as an ensemble, known to be improving classification performance on the tested datasets. An important baseline then is treating the stream subnetworks as separate models applied on different resolutions, which is simpler than the proposed architecture and would not suffer from communication overhead. A baseline with ResNet and ParNet ensembles needs to be added.
- statistical significance: test errors on CIFAR and ImageNet have high variance, and the authors show results of a single run for each experiment. A common practice to reduce variance is to train multiple networks and report mean and standard deviation for each experiment.
- SiLU: the authors replace ReLU with SiLU motivated by the limited representational power of ReLU. However, SiLU was introduced to improve training of deep networks, which is not the case in the proposed architecture. Moreover, in the ablation study (table 7) it does not seem to bring a significant improvement over ReLU, there is also no statistical significance analysis of this result. Using SiLU seems like a needless complication of the proposed architecture.

Arguable weakness:
- definition of depth: the approach takes advantage of the vague definition of depth in neural networks. In prior works, depth is defined as a number of layers in a single stream network. By introducing multiple parallel streams the authors effectively take a deep network and move layers around, so another, perhaps more fair, definition of depth could be counting the total number of layers in all substreams.

Notes to authors:
- table 3 is misleading because it does not show the number of GPUs used in the last row. It is also not clear if it is fused or not.
summary_of_the_review: The paper shows a significant result that a network with 12 layers can be competitive with deeper networks on well established image classification benchmarks. However, the architecture is more complex than ResNet or RepVGG, and the empirical evaluation is unsatisfactory due to weak and missing baselines, and the lack of statistical significance analysis. I hesitate between reject and weak reject.
review_title: null

Meta-review: This paper shows the possibility to design a relatively shallow architecture, ParNet, based on parallel subnetworks, instead of traditionally deeply stacked blocks. During discussions, the reviewers pointed out two important concerns: (1) the current design heavily hinges on the recently proposed RepVGG block, whose comparison was even missed in the original submission (later added in rebuttal); (2) comparing ParNet with RepVGG, there seems no performance advantage. Although RepVGG is 2.5 times deeper than ParNet, it is still faster due to highly optimized layers.

The authors mainly argued that their contribution is to answer the scientific question “is it possible to build high-performing non-deep neural networks?” While this is indeed an interesting question, AC feels: (1) it is perhaps unfair for this paper to claim as the first work proving the feasibility. WideResNet provided similar insight much earlier, among others; (2) the presented results, with tools being not novel, are pre-mature as they display no real appeal of using ParNet, in any aspect. Probing a new question is of course valuable, but presenting an immature and novelty-lacking answer shouldn't automatically grant publication. 

In sum, the reviewers were unanimously UN-convinced by this paper's value, nor was the AC. The authors are suggested to very seriously take into account reviewers' suggestions to make improvements, before submitting their work to the next venue.
Decision: Reject