Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a neural architecture search (NAS) algorithm which automatically finds a efficient network architecture, FasterSeg, for real time semantic segmentation. In designing a NAS algorithm the author takes cue from recent architectural advances introduced for faster segmentation as well as improved accuracy. For instances a) it explores and integrates multi-resolution branches from BiSeNet during NAS b) simultaneously optimizes the loss for accuracy and latency (as done in CAS algorithm) and c) knowledge distillation for semantic segmentation. However, the usage of these blocks in FasterSeg has been well refined to integrate with NAS search. To be precise, their improved version of latency loss avoids architectural collapse during latency-constrained search and it claims to be the first work to co-search for teacher and student network using NAS. Empirical experiments on benchmark dataset suggests that FasterSeg  is more than 30 percent faster with similar accuracy as state-of-the-art real-time segmentation algorithms.

This paper weakly leans towards rejection. Some of the contributing factors 
1) Overall presentation of algorithm leaves one more confused. Perhaps, the paper is targeted at small set of audience who primarily works on NAS. More about specific comment in 'Clarification'.
2) There is not a single concrete contribution. For example, NAS search in semantic segmentation using cells and downsampling rates was done in Auto-Deeplab. Further, resource-constrained search for segmentation was introduced in Zhang et al while distillation for segmentation task was proposed by Liu et al. 
3) No doubt that it achieves improved efficiency. But at the cost of accuracy. On Camvid and BDD, the competitive algorithm is 1.7 % better in absolute terms. On cityscapes it performs on par. However, the large improvement in accuracy can be attributed to distillation process (Table 3: absolute 2%), without which the overall performance of NAS is suboptimal.

Clarification:
1. It is not clear what is the form of initial network which is pre-trained for 20 epochs ? My guess is, initial network consists of b=3 branches with L=16 sequential layers and for each cell in a layer, the network pre-trains 5 operators as well as for different expansion ratios. Is it correct ?
2. Can you explain 697 unique paths as well as 10^55 unique combinations ?
3. It is noted that by default b=2 is used. However, in FasterSeg network shown in figure 6, I note three branches s={8,16,32}. Am I missing something ? Also, how more branches will introduce more latency ? Branches operate in parallel with max sensitivity s=0.01 and max L=16.
4. Next, as pointed in 3.4 the discrete architecture is obtained by computing \argmax_l over \beta. In that case, there should only be single connection which branches out from s->2s. In figure 6, I note two branches from 8->16 (4th and 6th cell).
5. If the teacher and student network shares the same weight, then what is the need for distillation ? Only difference I currently note is in the expansion ratio. May be you want to say same pretrained network ?
6. Can you explain with example how \gamma's are updated using backpropagation and lookup-table ? 
7. Are you employing STE for Gumbel-Max trick ? 
8. The individual terms in eq (3) optimizes for \alpha, \beta and \gamma respectively ?
9. In eq (2), each cell output O is linear combination of different operator ? 
10. Once the discrete architecture is obtained, is it retrained on cityscapes from scratch or fine-tuned ?

Request for ablation:
1. What is the variation in NAS output with changes in \lambda ? Precisely, can one tradeoff accuracy for improved latency just by tuning \lambda ?
2. What happens if teacher network is also optimised over \gamma ? The difference between teacher and student will then only be in loss function.
3. Currently, discrete architecture is greedily extracted. This need not be the best. Instead one can utilize sequential beam search (vitterbi algorithm). With this it is possible to visualise the accuracy and latency distribution of, say top 100 architecture obtained by NAS.

Minor comments:
1. Seachable -> Searchable

Updates:
I read through the reviews of other reviewers as well as the rebuttal posted by authors. Overall, I am satisfied with the authors response and hence Improving my scores to Weak Accept.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Summary:
- key problem: neural architecture search (NAS) to improve both accuracy and runtime efficiency of deep nets for semantic segmentation;
- contributions: 1) a novel NAS search space leveraging multi-resolution branches, efficient operators ("zoomed convolutions"), and parametrized expansion ratios, 2) a decomposition and normalization of the latency objective to avoid a bias towards very fast but weak architectures, 3) a natural extension of the optimization problem to simultaneously search for teacher and student architectures in one pass, 4) a novel state-of-the-art efficient architecture (FasterSeg) found by the aforementioned NAS algorithm, 5) a detailed experimental evaluation on 3 datasets and an ablative analysis quantifying the benefits of the aforementioned contributions.

Recommendation: Accept.

Key reason 1: solid experimental results backing the claims.
- When compared to related efficient architectures, the proposed method results in competitive accuracy at significantly higher frame rates.
- This is validated on Cityscapes, CamVid, and BDD with the architecture found on Cityscapes. 
- The resulting architecture (FasterSeg) is actually interpretable and makes sense, extending the handcrafted architectures used as inspiration.
- The ablative analysis shows that the numerous individual contributions are significant, esp. the multi-branch formulation and student co-searching.

Key reason 2: well-motivated method with a collection of multiple novel contributions that are interesting and practical.
- The multi-resolution branches formulation is simple and extends typical NAS focusing on single paths through the supernet.
- Teacher/student co-searching via learning two sets of architectures in one supernet seems novel, simple, and effective. Always picking the largest expansion ratios for the teacher and applying a distillation loss in addition to the latency loss for the student is sensible and seems to beat the standard pruning approach at no significant extra cost during NAS.
- The zoomed convolution operator seems like a novel efficient alternative to (expensive) dilated convolutions. Although it is very simple (bilinear downsampling -> 3x3 conv -> bilinear upsampling), it is not commonly used as an operator (as far as I know), and yet is found to be a key part of the final architecture (Table 7 appendix I) due to its low latency. The closest related operator / block I could think of might be blocks found in stacked hourglass networks (Newell et al).
- The optimization of the expansion ratios using the Gumbel-Softmax trick is interesting, although this is also explored in the very recent paper by Shaw et al. 2019 (possibly the closest related work that should be discussed in a bit more depth in Section 2);
- Decomposing and normalizing the latency objective to avoid "architecture collapse" (convergence to anemic architectures stemming from certain architectural factors dominating latency) is principled and effective.
- Caveat regarding novelty: I could not find the ideas proposed here in the literature, but its hard to be sure due to 1) the recent explosion of NAS papers, 2) the simplicity of certain ideas (e.g., "zoomed convolutions").


Additional Feedback:
- how is the student trained after NAS? Is the teacher first retrained from scratch? Is the student retrained from scratch on the teacher (after NAS or retraining)? in general, more details on what happens after co-searching would be helpful;
- "human designed CNN architectures achieve superior accuracy performance nowadays": this is a surprising statement considering the cited NAS papers report performance improvements (e.g., Zoph and Le 2016);
- missing reference also using multi-scale NAS for efficient and accurate semantic segmentation: "Searching for Efficient Multi-Scale Architectures for Dense Image Prediction", Chen et al, NeurIPS 2018;
- missing reference on NAS for efficient semantic segmentation that also uses distillation: "Fast neural architecture search of compact semantic segmentation models via auxiliary cells", Nekrasov et al, CVPR 2019;
- missing reference on joint NAS and quantization: "Joint Neural Architecture Search and Quantization", Chen et al, arxiv 2018;
- "we choose a sequential search space (rather than a directed acyclic graph of nodes (Liu et al., 2018b)), i.e., convolutional layers are sequentially stacked in our network": "stacked" is confusing here;
- "we allow each cell to be individually searchable across the whole search space": what do you mean? Anything beyond each cell containing different operators after learning?
- if \alpha = \beta in eq. 6 of appendix C, then w and hence Target(m) does not depend on latency, isn't this a typo?
- "Gumbel-Max" is typically called "Gumbel-Softmax" (cf. "Categorical Reparameterization with Gumbel-Softmax", Jang et al, ICLR'17);
- typos: "find them contribute", "the closet competitor", "is popular dense predictions".
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper presents an automatically designed semantic segmentation network utilising neural architecture search. The proposed method is discovered from a search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To calibrate the balance between the goals of high accuracy and low latency, the authors propose a decoupled and fine-grained latency regularization, that effectively overcomes the observed phenomenons that the searched networks are prone to “collapsing” to low-latency yet poor-accuracy models. Moreover, the authors extend the proposed method to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model’s accuracy. Experimental results on Cityscapes, CamVid, and BDD verified the efficacy of the proposed method.

The writing is clear and the presentation is good. I like the motivation of this paper. The problem solved in this paper aligns with reality.

My concerns regarding this paper are as below.
1) The datasets used for evaluation are quite old except BDD, which make the results not so convincing. More experiments on more recent challenging semantic segmentation benchmarks are needed to verify the superiorities claimed in this paper. e.g., MHP v2.0 [Zhao et al., ACM MM 2018], etc.
2) The methods compared in Tab.6 are quite old, please add comparisons with more recent SOTAs.
3) Please add qualitative analysis to gain insight into the proposed method and to show why works better than other SOTAs.
4) The results in Tab.4,5,6 of the proposed method are not the best for mIoU on testing protocol, is this a trade-off between acc and speed?

Based on the above overall comments, I decide to give the rate of Weak Accept for this paper.
review_title: Official Blind Review #1

Meta-review: This paper presents neural architecture search for semantic segmentation, with search space that integrates multi-resolution branches. The method also uses a regularization to overcome the issue of learned networks collapsing to low-latency but poor accuracy models. Another interesting contribution is a collaborative search procedure to simultaneously search for student and teacher networks in a single run. All reviewers agree that the proposed method is well-motivated and shows promising empirical results. Author response satisfactorily addressed most of the points raised by the reviewers. I recommend acceptance. 
Decision: Accept (Poster)