Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors explore how to reduce the depth of neural networks while keeping the model capacity. They propose ParNet, which has a depth of only 12 and can achieve comparable performance with deep neural networks. The authors also show some potential advantages of using non-deep networks for a smaller latency on future hardwares.
strengths_and_weaknesses: Strengths:
- The whole idea of non-deep networks is novel and not well-explored in previous works, as increasing the depth is usually a common practice for most neural network architectures to scale up themselves.
- Achieving similar performance as 'deep networks' with a depth of 12 is pretty impressive.
- The authors also discuss the potential advantages of using shallow networks to fully utilize the parallelism of hardware (e.g. GPUs).
- Some previous works are reproduced under same setting for fair comparison.
- The key limitations are well discussed.
- The writing is good and all details are presented very clearly.

Weaknesses:
- The technical contributions are not significant, as most of the crucial components of ParNet were presented in other papers. For example, the overall ParNet structure looks very similar to HRNet [1] and the ParNet block looks like a variant of Inception [2].
- Although the authors intend to improve the inference speed by reducing the network depth, ParNet still introduces some element-wise operations that are not parallism-friendly and hard to optimize (e.g. global average pooling and sigmoid in SSE layer and SiLU activation).
- To reduce the network depth, ParNet sacrifces too much on the number of parameters and FLOPs (as well as inference memory usage even it's not shown in the paper).

[1] Hu, Jie, Li Shen, and Gang Sun. "Squeeze-and-excitation networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[2] Szegedy, Christian, et al. "Going deeper with convolutions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
questions: - From my perspective, ParNet is generally comparing to models with a large computing budget like ResNet. It is interesting to see whether ParNet architecture can also achieve comparable performance with modern compact models (for example EfficientNet). Compat models are more widely used in environment that requires low-latency and some of the design philosophy might help to reduce the high computation cost of ParNet.
limitations: The main limitation of this work (inference speed on current hardwares) is well-discussed in the paper. In my option, figuring out a way to avoid the use of global average pooling and sigmoid while keeping the same depth/performance would be a huge technical contribution, as they are indeed slow on GPUs and hard to parallize or optimize.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Prevailing wisdom suggests neural networks must be of sufficient depth in order to achieve competitive or useful accuracy.  The authors challenge this belief and design a family of networks called ParNets, which use parallel structures that fork quickly from the input and converge again close to the output.  ParNets are built of ParNet Blocks, made up of Rep-VGG blocks with Skip-SE layers to increase the receptive field and avoid increasing depth.  The authors present results on a variety of tasks and against a variety of baseline networks.  Generally, ParNets are able to achieve competitive accuracy and latency.  Ablation studies show the impact of several strategies for boosting accuracy and how scaling various dimensions of ParNets (input resolution, number of streams, number of parameters) affects their accuracy.

strengths_and_weaknesses: == Strengths ==

The submission is intriguing - it boldly challenges conventional wisdom and shows that networks need not be very deep at all to achieve useful, much less competitive, accuracy.  I found the network design methodology to be interesting and grounded in reason.  I imagine much of the NeurIPS attendees and readership will find the topic and approach compelling, at least from a theoretical standpoint.

The text is not hard to follow; I rarely found myself needing to re-read a sentence due to having trouble parsing it or placing it into context with the preceding text.

The breadth of experiments is appreciated: not only are there accuracy results, but we also see latency for some experiments, comparisons with different baselines, ablations studies of all types, and different tasks are represented.

Finally, the ultimate goal of showing that a shallow network can compete with deep networks is successful: the accuracy of the 12-layer networks, regardless of the number of streams, stream widths, or input size, is sufficient to make the field take notice.

== Weaknesses ==

For the successes I describe above, I think the exposition and successive analysis are in need of attention before I can recommend the submission for acceptance.

I found the motivation to be lacking nuance and the focus on network depth to be myopic.  
- In the abstract, it is asserted that "latency is fundamentally dependent on the depth of the network."  This is also true of the number of parameters width of each layer - wider layers require more FLOPs, increasing latency on a given piece of hardware.  Similarly, latency is fundamentally dependent on the input resolution, choice of activation function (is it a simple clamp() function, like ReLU, or does it involve a transcendental function?), and choice of layer types (3x3 convolution vs. 1x1 convolution vs. fully-connected vs. depthwise vs. …).
- This focus on depth gets stranger with line 20's "the lowest achievable latency is d/f" - layer-seconds per cycle is an odd unit of latency.  I think I understand the intent, but I can't help but feel it could be stated less sensationally.
- The focus on depth is similarly out of place in Section 3.5's Line 177-178: "All these factors make non-deep parallel structures advantageous…", but nothing in this section was related to network depth, just parallel operations.  
- Line 264 suggests that "it is surprising that a mere depth-12 network could achieve…. This further indicates that non-deep networks can work as well as deeper counterparts."  Strictly speaking, any surprise at the results does not indicate anything about non-deep networks' relative performance to their deeper counterparts.  What's really shown is that ParNet's structure allows for shallower networks than the other network structures studied.

An obvious experiment was left un-performed, in light of the claim that non-deep networks can work as well as deeper counterparts: what happens if the streams of ParNet are made deeper?  Does performance not improve, supporting this assertion?  Conversely, what if depth is decreased - is there a minimum necessary depth for ParNet?  Any results here would not diminish the contributions, but they would help put them into perspective.

I noted several unfair or misleading comparisons in the results:
- Figure 1 mentions that "for fairness," the authors "exclude results with longer training, higher resolution, or multi-crop testing."  However, the authors also seem to report their own networks' accuracy results with higher resolution in Table 4's results, as described in Line 237: "We use this higher image resolution for ParNext-XL and ParNet-XL-CSP."
- Table 2 presents latency results for several baselines networks and several ParNets, with the stated conclusion that, "In spite of communication overhead, ParNet is faster than similar-performing ResNets."  This is not a fair comparison, as the ParNets are using 3x as many compute resources, detailed in Line 210-211: "… for the multi-GPU version, we use 3 GPUs."  This detail is not reflected in the table, where most readers would notice.
- Table 6 is organized such that models with similar sizes are grouped together, and then ParNets are compared only within those groups - this shows that ParNets are within the top three most accurate networks in each group.  However, this disguises other potential comparisons, such as DenseNet (Bottleneck+Compression) with a depth of 250, with the final group.  If it were included, it would show that a network with less than half as many parameters has higher accuracy than the largest ParNet model.
	
I was waiting for a comparison with ensembling, which is an obvious cousin to the ParNet structure: send an input to separate networks, collect the results, and reduce to a single output.  However, the only comparison is with ensembling ParNet-M models.  A better choice might be ResNet-50; as presented in Table 2, it has higher accuracy, fewer parameters, and fewer FLOPs than ParNet-M.  Ensembling two RN50 models would have roughly the same number of parameters as a single ParNet-L.  Three RN50s would make use of the three GPUs afforded to ParNet-L in Table 2, so latency for either ensemble would be roughly identical.  If the latency is roughly the same for improved accuracy (pending the experiment!), then why would ParNets have an advantage?

The conclusion from Figure 3 on Lines 299-301 is that "Based on these charts, we see no saturation in performance while scaling ParNets.  This indicates that by increasing compute, one could achieve even higher performance with ParNet while maintaining low depth."  This is in contrast to Table 10, though, which shows explicitly that scaling from a third to a fourth branch reduces Top-1 accuracy.  It seems that Figure 3 simply does not scale far enough to see the saturation shown in Table 10.

questions: What happens if the streams of ParNet are made deeper?
Conversely, what if depth is decreased - is there a minimum necessary depth for ParNet?

Am I missing a fundamental relationship between depth and latency that doesn't exist between other axes, like layer width, and latency?  (Formulating an equation for minimum latency seems like it should be based on FLOP count and bandwidth requirements rather than something more abstract like "depth.")

limitations: Yes, the limitations have been discussed adequately.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new CNN architecture designed for the purpose of having a relatively low depth (12 layers) while still providing results on par with (deeper) classic ResNets on various vision benchmarks, demonstrating that there may be other mechanisms than depth at play to obtain state of the art results with CNNs. The strong expressive power of the architecture is obtained thanks to parallel substructures in the network, each operating at different scale, then fusing together their information content before the final layers. In-depth ablations study are done to demonstrate the relevance of many components of the design.
This work is motivated by the goal of decreasing latency, which can be impacted by depth. In order to improve this metrics, it is shown that, once trained, the network can be expressed equivalently as a classic single stream CNN, increasing its speed at inference.
strengths_and_weaknesses: $\textbf{Strengths}$

The paper is clearly motivated and very well written. The experiments are numerous, thoroughly comparing the newly introduced CNN architecture to deeper ones on several classic vision datasets. An in-depth ablation study quantifies the impact of most parts of the new design, reporting impressive results for a 12 layers deep network. Comparison between this new design and other standard architectures in terms of latency is done, showing promising results for ParNet.

$\textbf{Weaknesses}$

While the effect of increasing the number of parameters and the number of streams have been studied on this new architectures, the effect of the depth is suprisingly not shown while being the highlighted metric of this paper. How ParNets with a number of layers of similar order of magnitude (e.g., 10,11,13 or 14) perform ? If depth/latency was not an issue, would scaling ParNet to greater depths further increase its performances on standard benchmarks ?
questions: * Why/How did you chose the depth of 12 ?
* What is the effect of the depth in your new architecture ?
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The presented paper questions the importance of the depth in neural architectures to provide state-of-the-art-results. There are two main motivations for this paper, the first being that very deep networks are inherently ill-suited for real-time systems, since input data needs to be processed sequentially through each layer. The second motivation is that computations within shallow networks using parallel substructures can be parallelized, hence (theoretically) reducing the effective computation time. However, depth being a central feature allowing neural architecture to perform complex classification tasks, the main challenge addressed by this paper is to derive a class of shallow architectures whose performances remain competitive against state-of-the-art deep architecture. The resulting architecture, called ParNet, is rather flexible in the sense that one can scale its representational power by increasing the number of parallel substructure. The authors specifically discuss the importance of certain module with respect to the final performance reported in the experiment section.
strengths_and_weaknesses: The paper motivates its research effort to derive efficient shallow architectures with several arguments, the first one being that shallow networks are highly desirable in real-time applications. The author also motivates their effort from two "wall-clock" computation arguments, the first one being that deep architectures are hardly biologically plausible given the speed of human reaction with respect to certain stimuli. The authors also try to make reasonable assumptions on the improvement that is likely to be provided by the next generation of hardware, pointing out that neural architectures enabling module parallelism (different from data-parallelism), should benefit from such improvements. Three experiments are conducted to assess the relevance of the architecture:
1) experiments show that ParNet, an architecture of depth only 12, provides performance on par with ResNet-like and VGG-like architectures on CIFAR10, CIFAR100, ImageNet and MS-COCO. The authors also carefully discuss speed versus accuracy in the case of ImageNet.
2) the impact of specific block design and engineering tricks found in the literature with respect to the final performance are discussed to show how they can be employed to reduce depth while maintaining high performance. The claims are backed by an ablation study.
3) rules to increase the statistical capacity are presented, and the increased statistical capacity results in increased performance for ParNet.

Some weaknesses are exposed by the authors themselves. In table 2, RepVGG was found to be faster than ParNet, however authors point out that RepVGG benefits from a highly optimized design which is coherent given that this architecture has been tested and improved on many times over. Another key weakness in the presented paper (not the contribution itself) is the lack of real-world scenario where parallelism is actively employed to reduce computation time, hence showcasing the advantage of ParNet in such setting. Table 3 attempts to show that parallelizing the streams on different GPU does reduce the effective computation time, but it is likely that neither the software nor the hardware are optimized for ParNet paradigm, since they were initially developped to tackle data-parallelism.
questions: Can the author provide a setting example where they can lower-bound the time it would take for a ResNet-like or VGG-like architecture to process an input, both theoretically and empirically, and show to what extent parallelism gives a crucial advantage ?
Throughout the paper, the authors mention situations where multi-stream shallow networks should provide much faster computations.
For example, table 3 compare ParNet with and without parallelism but do not compare the speed with other architectures.
Furthermore, the authors promote the need for robust shallow architecture to reduce latency in real-time systems such as autonomous vehicles. The closest experiment addressing this issue is the MS-COCO challenge as it is a detection rather than a classification challenge, but parallelism is not employed here, thus resulting in less than 25% improvement in latency while the depth is reduced by 75% compared to the baseline. Can the authors explain why they did not include more experiments on model parallelism while strongly motivating their paper for this particular purpose ?
limitations: One limitation of the paper, actually discussed by the authors, is that non-deep networks are not a replacement for their deep counterparts. They motivate such claim by saying that non-deep networks still require a high number of parameter and flops. From a more theoretical side, however, the reviewer strongly wonders if there are classes of functions actually requiring a certain depth to be effectively approximated by a neural network. This last point could help identify the use cases where non-deep networks will never be an acceptable replacement for their deep counterparts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a non-deep network for low latency. The key designs and techniques include multi-stream topology, RepVGG-style structural re-parameterization and SSE block. Reasonable results on ImageNet, CIFAR and COCO are reported. The results show that shallow models can perform competitively with deep ones.
strengths_and_weaknesses: Strengths:
S1. This paper shows that a 12-layer model can achieve accuracy of over 80% on ImageNet and an AP of 48% on MS-COCO. The results show that CNNs do not need to be very deep and research on shallow models is still useful, which is valuable to the literature. This is the primary reason I recommend accepting this paper.
S2. The key techniques include RepVGG-style structural re-parameterization, multi-stream design and SSE block. The motivations are well explained and the effects are verified.
S3. The results with custom hardware settings are particularly interesting. (For the multi-GPU version, each stream is launched on a separate GPU) The results and discussions of the communication overhead are useful.

Weaknesses
W1. The reported results are all small- and middle-sized models. It is not sure if heavy-weight models show different depth v.s. performance trade-offs. Of course, I understand the authors may have limited computing resources and did not lower my rating for lack of such large-scale experiments.
W2. Though the discussions of the future hardware are interesting, this paper shows no clue on how to realize it. Maybe the authors can show a potential direction.
W3. The writing can be improved. Please see the suggestions below.
questions: Suggestions:
1. A discussion with RepVGG may highlight the contributions of this paper as well as show the differences. For example, "this paper focus on the depth while RepVGG was more about the simplicity, .........., this paper is related to RepVGG because a RepVGG-style overall architecture is suitable for shallow model design".

2. L103: "VGG-style block" seems confusing. A reader may question why a single 3x3 conv is called a block. I would suggest the authors use "VGG-style architecture" or "RepVGG-style design".

3. L133: missing reference. L143: missing reference

4. Caption of Table 2: I would suggest the authors use 3$\times$3 instead of 3X3.

5. L311: "consists of consists of"

limitations: Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes. I appreciate the discussions.
review_title: null

Meta-review: metareview: This work considers the task of training state-of-the-art CNNs with limited depth. The benefits considered in this work are related to the potential parallelization which is induced by depth reduction. This paper generated a fair bit of discussion with the reviewers about the motivations and the basic thesis. The authors do a good job of representing their viewpoint, and adding a version of this discussion to the final manuscript will undoubtedly be needed. The empirical results look quite promising, but the authors are also encouraged to further discuss adding an additional motivation to reducing depth (e.g., a theoretical reason, as proposed by reviewer zBDf, which is currently only one paragraph long in the related section) and/or performing a deeper study of hyper-parameters affecting accuracy/latency. With proper framing of the question studied here, the scope of evaluation and the assumptions on the hardware, this will be an interesting contribution to NeurIPS
confidence: Less certain
Decision: Accept