FasterSeg: Searching for Faster Real-time Semantic Segmentation
Authors: Wuyang Chen
Xinyu Gong
Xianming Liu
Qian Zhang
Yuan Li
Zhangyang Wang
Venue:  (2020)
Keywords: neural architecture search
real-time
segmentation

Abstract: We present FasterSeg, an automatically designed semantic segmentation network with not only state-of-the-art performance but also faster speed than current methods. Utilizing neural architecture search (NAS), FasterSeg is discovered from a novel and broader search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To better calibrate the balance between the goals of high accuracy and low latency, we propose a decoupled and fine-grained latency regularization, that effectively overcomes our observed phenomenons that the searched networks are prone to "collapsing" to low-latency yet poor-accuracy models. Moreover, we seamlessly extend FasterSeg to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model’s accuracy. Experiments on popular segmentation benchmarks demonstrate the competency of FasterSeg. For example, FasterSeg can run over 30% faster than the closest manually designed competitor on Cityscapes, while maintaining comparable accuracy.

Published as a conference paper at ICLR 2020
FASTERSEG: SEARCHING FOR FASTER REAL-TIME SEMANTIC SEGMENTATION
Wuyang Chen',, Xinyu Gong", Xianming Liu?, Qian Zhang', Yuan Li,, Zhangyang Wang' 'Department of Computer Science and Engineering, Texas A&M University Hoorizon Robotics Inc.
{wuyang. chen, xy-gong, atlaswagg http:aswwwwaaaamam.e.ommm @tamu. edu
{xianming ng 1iu,, qian01 zhang, yuan leeoonniiieehheeeeeeeeeeea weworiooorooooooo https://githun com/ http://www....////VV....../ TAAM--IIA/aatte:://
ABSTRACT
We present FasterSeg, an automatically designed semantic segmentation network with not only state-of-the-an performance but also faster speed than current meth- ods. Utilizing neural architecture search (NAS), FasterSeg is discovered from novel and broader search space integrating multi-resolution branches, that has been recently found to vital in manually designed segmentation models. To better calibrate the balance between the goals of high accuracy and low latency, we propose a decoupled and fine-grained latency regularization, that effectively overcomes observed phenomenons that the searched networks prone "collapsingg" to low-latency yet poor-accuracy models. Moreover, we seamlessly extend FasterSeg to a new collaborative search (co-searching) framework, simul- tancously searching for teacher and a student network in the same single run. The teacher-student distillation further boosts the student model's accuracy. Ex- periments on popular segmentation benchmarks demonstrate the competency of FasterSeg. For example, FasterSeg can run over 30% faster than the closest manu- ally designed competitor on Cityscapes, while maintaining comparable accuracy.
INTRODUCTION
Semantic segmentation predicts pixel-level annotations of different semantic categories for im- age. Despite its performance breakthrough thanks to the prosperity of convolutional neural networks (CNNs) (Long al. 2015), as dense structured prediction task, segmentation models commonly suffer from heavy memory costs and latency, often due to stacking convolutions and aggregating multiple-scale features, as well as the increasing input image resolutions. However, recent years witness the fast-growing demand for real-time usage of semantic segmentation, e.g., autonomous driving. Such has motivated the enthusiasm on designing low-latency, more efficient segmentation networks, without sacrificing accuracy notably (Zhao et al., 2018 Yu et al. 2018a).
The recent success neural architecture search (NAS) algorithms has shed light on the new horizon in designing better semantic segmentation models, especially under latency of other resource con- straints. Auto-Deepl.at (Liu et al. 2019a) first introduced network-level search space to optimize resolutions (in addition to cell structure) for segmentation tasks. Zhang al. (2019) and Li et al. (2019) adopted pre-defined network-level patterns of spatial resolution, and searched for operators and decoders with latency constraint. Despite handful of preliminary successes, observe that the successful human domain expertise in designing segmentation models appears to be not fully integrated into NAS frameworks yet. For example, human-designed architectures for real-time seg- mentation (Zhao et al.) 20188; Yu et al., 2018a) commonly exploit multi-resolution branches with proper depth, width, operators, and downsample rates, and find them contributing vitally the suc- cess: such flexibility has not been unleashed by existing NAS segmentation efforts. Furthermore, the trade-off between two (somewhat conflicting) goals, i.e.. high accuracy and low latency, also makes the search process unstable and prone to "bad local minima" architecture options.
Work done during the first two authors' research internships with Horizon Roboties Applicd AT Lah.

---

Published as a conference paper at ICLR 2020
As the well-said quote goes: "those who do not learn history are doomed to repeat it". Inheriting and inspired by the successful practice in hand-crafted efficient segmentation, we propose a novel NAS framework dubbed FasterSeg, aiming to achieve extremely fast inference speed and competi- tive accuracy. We designed a special search space capable of supporting optimization over multiple branches of different resolutions, instead of single backbone. These searched branches are adap- tively aggregated for the final prediction. To further balance between accuracy versus latency and avoiding collapsing towards either metric (e.g. good lateney yet poor accuracy), we design a decou- pled and fine-gr ane-grained latency regularization, that facilitates a more flexible and effective calibration between latency and aceuracy. Moreover, our NAS framework can be easily extended to a collabo- rative search (co-searching), i.e., jointly searching for complex teacher network and a fight-weight student network in a single run, whereas the two models are coupled by feature distillation order to boost the student's s accuracy. We summarize our main contributions as follows:
A novel NAS search space tailored for real-time segmentation, where multi-resolution branches can be flexibility searched and aggregated.
A novel decoupled and fine-grained latency regularization, that successfully alleviates the "architecture collapse" problem in the latency search.
A novel extension to teacher-student co-searching for the first time, where we distill the teacher the student for further accuracy boost of the latter.
Extensive experiments demonstrating that FasterSeg achieves extremely fast speed (over 30% faster than the closest manually designed competitor on CityScapes) and maintains competitive accuracy.
2 RELATED WORK
Human-designn CNN architectures achieve good accuracy performance nowadays (He al. 2016 Chen et al. 2018b Wang et al. 2019 Sun et al. 2019). However, designing architectures to balance between accuracy and other resource constraints (latency, memory, FLOPs, etc.) requires more human efforts. To free human experts from this challenging trade-off, neural architecture search
(NAS) has been recently introduced and drawn a booming interest (Zoph & Le 2016 Brock al.
2017 Pham et al. 2018: : Liu et al. 2018a Chen et al.] [2015 8a Bender et al. 2018 Chen et al. (2018c)
Gong et al.] 2019). These works optimize both accuracy and resource utilization, via a combined
loss function Wu et al. 2019), or a hybrid reward signal for policy learning (Tan et al. 2019 Cheng
al. 2018 or constrained optimization formulation (Dai al. 2019).
Most existing resource-aware NAS efforts focus on classification tasks, while semantic segmentation has higher requirements for preserving details and rich contexts, therefore posing more dilemmas for efficient network design. Fortunately, previous handcrafted architectures for real-tine siml-time segmentation have identified several consistent and successful design patterns. ENet (Paszke et al.. 2016) adopted early downsampling, and ICNet (Zhao et al.) 2018) further incorporated feature maps from multi- resolution branches under label guidance. BiSeNet (Yu et al. (018a) fused a context path with fast downsampling and a spatial path with smaller filter strides. More works target segmentation efficiency in terms of computation cost (He et al.) 2019] Marin et al. 2019) and memory usage Chen al. 2019). Their multi-resolution branching and aggregation designs ensure sufficiently large receptive fields (contexts) while preserving high-resolution fine details, providing important clues on how further optimize the architecture.
There have been recent studies that start pointing NAS algorithms to segmentation tasks. Auto- Deepl.ab (Liu et al. 2019a) pioneered in this direction by searching the cells and the network- level downsample rates, to flexibly control the spatial resolution changes throughout the network. Zhang et al. (2019) and Li et al. (2019) introduced resource constraints into NAS segmentation. A multi-scale decoder was also automatically searched (Zhang et al., 2019). However, compared with manually designed architectures. those search models still follow a singlebbacboor
design and did not fully utilize the prior wisdom (e.g., multi-resolution branches) in designing their search spaces.
Lastly, we briefly review knowledge distillation (Hinton et al.) 2015), that aims to transfer learned knowledge from a sophisticated teacher network to a light-weight student, to improve the (more ef- ficient) student's accuracy. For segmentation, [Liu et al. (2019h) Nekrasoo et al, (2019) proposed to leverage knowledge distillation to improve the accuracy of the compact model and speed-up con- vergence. There was no prior work in linking distillation with NAS yet, and we will introduce the extension of FasterSeg by integrating reacher-student model collaborative search for the first time.
2

---

Published as a conference paper at ICLR 2020
3 FASTERSEG: FASTER REAL-TIME SEGMENTATION
Our FasterSeg is discovered from an efficient and multi-resolution search space inspired by pre- vious manual design successes. A ane-grained latency regularization is proposed to overcome the challenge of "architecture collapse" (Cheng et al. 2018). We then extend our FasterSeg to a teacher- student co-searching framework, further resulting a lighter yet more accurate student network.
3.1
EFFICIENT SEARCH SPACE WITH MULTI- RESOLUTIN BRANCHING
The core motivation behind our search space is search multi-resolution branches with overall low latency, which has shown effective in previous manual design works (Zhao et al. 2018 Yu et al. 2a. Our NAS framework automatically selects and aggregates branches of different resolutions, based on efficient cells with searchable superkernels.
Downsample Rate (x)
Cells
Head
L-2
multi-resolution
16
branches
Tonsor,
(H,CH)
Tanaora (aaHHol!
3x33
Calles
3x3 x2
zoomed 3x3 comv.
zoomed 3x3 com. x2
searchable superkernal expansion ratios
The multi-resolution branching search space for FasterSeg, where aim to optimize multiple branches with different output resolutions. These outputs progressively aggregated together in the head module. Each cell individually searchable and may have two inputs two outputs, both different downsampling rates (s). Inside each cell, we enable searching for expansion ratios within a single superkemnll.
Figure 1:
3.1.1 SEARCHABLE MULTI- RESOLUTION BRANCHES
Inspired by (Liu et al. 2019a), we enable searching for spatial resolutions within the L-layer cells (Figure
Downnsmppo Rate 69) Downsemple Rate 64)
T.. where each cell takes inputs from two connected predecessors and outputs two feature maps of differ- ent resolutions. Hand-crafted networks for real time segmentation found multi-branches of different reso- lutions to be effective (Zhao et al. 2018 Yu et al.] 2018ap. However, architectures explored by current NAS algorithms are restricted to a single backbone.
Our goal is select b (b 1) branches of different resolutions in this L-layer framework. Specifically, we could choose b different final output resolutions for the last layer of cells, and decode each branch via back- trace (section 3.4). This enables our NAS framework to explore b individual branches with different reso- lutions, which are progressively "learned to aggre- gated" by the head module (Figure
Figure 2: Our multi-resolution search space covers existing manual designs for real-time segmentation (unused cells omitted). Top: IC- Net (Zhao et al. 2018). Bottom: BiSeNet YYu et al. 2018a)
We follow the convention to increase the number of channels at each time of resolution downsam- pling. To enlarge the model capacity without incurring much latency, first downsample the input image to original scale with our stem module, and then set our searchable downsample rates (8, 16, 32). Figure 2 shows that multi-resolution search space is able cover existing human-designed networks for real-time segmentation. See Appendix [ for branch selection details.
3.1.2 CHOOSING EFFICIENT OPERATORS WITH LARGE RECEPTIVE FIELDS
As we aim to boost the inference latency. the speed of executing an operator is direct metric (rather than indirect metrics like FI.OPs) for selecting operator candidates O. Meanwhile, previously
3

---

Published as a conference paper at ICLR 2020
discussed, it is also important to ensure sufficiently large receptive field for spatial contexts. We analyze typical operators, including their common surrogate latency measures (FLOPs, parameter numbers), and their real- cel--meauured latency NVIDIA 1080Ti GPU with TensorRT library, and their receptive fields, as summarized in T
Compared with standard convolution, group convolution is often used for reducing FLOPs and num- ber of parameters (Sandler et al. 2018 Ma et al.) 2018). Convolving with two groups has the same receptive field with standard convolution but is 13% faster, while halving the parameter amount (which might not be preferable as it reduces the model learning capacity). Dilated convolution has an enlarged receptive field and is popular in dense predictions (Chen al. 2018b] Dai al. (017). However, as shown in Table (and as widely acknowledged in engineering practice), dilated convo- lution (with dilation rate 2) suffers from dramatically higher latency, although that was not directly reflected in FLOPs nor parameter numbers. In view of that, we design new variant called "zoomed convolution", where the input feature map sequentially processed with bilinear downsampling, standard convolution, and bilinear upsampling. This special design enjoys 40% lower latency and 2 times larger receptive field compared to standard convolution. Our search space hence consists of the following operators:
skip connection
3x3 conv.
3x3 conv. x2
"zoomed conv.": bilinear downsampling 3x3 conv. bilinear upsampling "zoomed conv. x2": bilinear downsampling + 3x3 conv. x2 bilinear upsampling
As mentioned by Ma et al. (2188) network fragmentation can significantly hamper the de- gree of parallelism, and there- fore practical efficiency. There- fore, we choose a sequential search space (rather than a di- rected acyclic graph of nodes (Liu et al.) 2018b)), i.e., con- volutional layers are sequentially stacked in our network. In Figure each cell is diffecentiable (Liu et al., 2018b
Table 1:
Specifications of different convolutions. Latency is measured using an input of size 1 x256 x2 x 64 on 1080Ti with TensorRT library, Each operator only contains one convolutional layer. The receptive field (RF) is relatively compared with the standard convolution (first row).
Layer Type
Latency (1n))
FLOPs (G)
Params (M)
RF
conv.
0.15
1.21
0.59
group2
0.13 (31)
0.60 (-50%)
0.29 (-51%) (1%)
conv. dilation?
0.25 (+-67%)
1.10 (-9%)
0.59
zoomed conv.
0.09 (-40%)
0.30 (-75%)
0.59
2019a and will
contain only one operator, once the discrete architecture is derived (section (34). It is worth noting that we allow each cell to be individually searchable across the whole search space.
3.1.3 SEARCHABLE SUPERKERNEL FOR EXPANSION RATIOS
We further give each cell the flexibility to choose different channel expansion ratios. In our work, we search for the width of the connection between successive cells. That is however non-trivial due the exponentially possible combinations of operators and widths. To tackle this problem, we propose a differentiably searchable superkemnl,, i.e., directly searching for the expansion ratio x within a single convolutional kernel which supports set of ratios C N Inspired by (Yul et al. 2018c) and (Stamoulis et al., 2019). from slim to wide our connections incrementally take larger subsets of input/output dimensions from the superkernel. During the architecture search, for each superkernel. only one expansion ratio is sampled, activated, and back poo-ppropggtted in each step of stochastic gradient descent. This design contributes to a simplified and memorry-effcciee super network and is implemented via the renowned "Gumbel-Softmax" trick (see Appendix( details).
To follow the convention to increase the number of channels as resolution downsampling, in our search space we consider the width X X 8, where E (8, 16, 32). We allow connections between each pair of successive cells flexibly choose its own expansion ratio, instead of using a unified single expansion ratio across the whole search space.
3.1.4 CONTINUOUS RELAXATION OF SEARCH SPACE
Denote the downsample rate as and layer index as l. To facilitate the search of spatial resolutions, we connect each cell with two possible predecessors' outputs with different downsample rates:
I = 00,0 --,01+ 18,
(1)
Each cell could have at most outputs with different downsample rates into its successors:
4

---

Published as a conference paper at ICLR 2020
O
0
8-,,
Σ
0 8-8
O
X., =)
k=1
(2)
10
Σ
star stride 2).
The expansion ratio X2, is sampled via "Gumbel-Softmax" trick according to p(x= X1,1 2 Here, a, 3, and y are all normalized scalars, associated with each operator O+ O, each predeces- sor's output O and each expansion ratio X E X' respectively (Appendix D They encode the architectures to optimized and derived.
3.2 REGULARIZED LATENCY OPTIMIZATION WITH FINER GRANULARITY
Low latency is desirable yet challenging to optimize. Previ- ous works (Cheng et al. 2018 Zhang et al.) 2019) observed that during the search procedure, the supernet search pol- icy often fall into bad "local minimums" where the generated architectures are of extremely low latency but with poor curacy, especially in the early stage of exploration. In addi- tion, the searched networked tend tions instead of choosing low expansion ratios (Shaw et al.) 2019). This problem termed as "architecture collapse" in our paper. The potential reason is that, finding architectures with extremely low latency (e.g. trivially selecting the most light- operators) is significantly easier than discovering meaningful compact architectures of high accuracy.
Latency Imst
29
20-
10.
Corcoed 0.993
more skip connec-
Estimated Latency (ms)
Figure Correlation between network latency and its estimation via our la- tency lookup table (linear coefficient: 0.993). Red line indicates "y = x"
To address this "architecture collapse" problem, we for the first time propose to leverage a fine-grained, decoupled latency regularization. We first achieve the continuous relaxation of latency similar to the cell operations in section 3.1.4] via replacing the operator in Eqn. D and with the corresponding latency. We build a latency lookup table that covers all possible operators to support the estimation of the relaxed latency. Figure F demonstrates the high correlation of 0.993 between the real and estimated latencies (see details in appendix EE).
We argue that the core reason behind the "architecture collapse" problem the different sensitivities of supernet to operator O, downsample rate 8, and expansion ratio x- Operators like "3x3 conv. x2" and "zoomed conv." have huge gap in latency. Similar latency gap (though more moderate) exists between slim and wide expansion ratios. However, downsample rates like "8" and "32" do dif- fer much, since resolution downsampling also brings dou- bling of the number of both input and output channels.
Table 2: Supernet's sensitivity to latency under different granularities. Input size: (1, 1024. 2048).
Granularity
Laatency (ms)
O (operator)
10.42
(downsample rate)
0.01
X (expansion ratio)
5.54
We quantitatively compared the influence of O, 8, and towards the supernet latency, by adjusting one of the three aspects and fixing the other two. Taking O as the example, we first uniformly initialize 3 and 7, and calculate ALatency (O) as the gap between the supernet which dominantly takes the slowest operators and the one adopts the fastest. Similar calculations were performed for and X. Values of ALatency in Table 2 indicate the high sensitivity of the supemee's latency to operators and expansion ratios, while not to resolutions. Figure 44a) shows that the unregularized latency optimization will bias the supernet towards light- operators and slim expansion ratios to quickly minimize the latency, ending up with problematic architectures with low accuracy.
Based this observation, propose regularized latency optimization leveraging different granu- larities of our search space. We decouple the calculation of supernet's 's latency into three granularities of our search space (O, 8, x). and regularize each aspect with different factor:
Latency (O,ancy(O,,,. x) wi Latency(O]ss, x) w,JTatencyss|O,x x) + wggTatency(xx0. (3)
5

---

Published as a conference paper at ICLR 2020
where by default set w1 0.001, W2 0.997, wa regularization successfully addresses this "architecture collapse"
0.002"
This decoupled and fine-gu ane-graindd problem, as shown in Figure b).
0.24
0.32
925
0.22
Lateno (ms)
0.30
9.20
0.20
mlou (%)
0.18
9.10
9.05
0.12
9.00
0.30
5
10
15
20
10
15
20
Epoch
Epoch
(a) Naive latency optimization.
(b) Proposed fine grained latency regularization.
Figure Comparing mloU (%) and latency (ms) between supernets. search conducted on Cityscapes training set and mloU measured on the validation set.
3.3 TEACHER/STUDEN CO-SEARCHING FOR KNOWLEDGE DISTILLATION
Knowledge Distillation is effective approach to transfer the knowledge learned by a large and complex network (teacher T) into much smaller network (student S). In our NAS framework, we can seamlessly extend to teacher-student co- searching, ie., collaboratively searching for two networks in a single run (Figure 5). Specifically, we search a complex teacher and light-weight student simultaneously via adopting two sets of architectures in one supernet: (aT, B8) and Bss Note that the teacher does not search the ex- pansion ratios and always select the widest one.
Segmentation
Supernet dz
Teacher T
derive
shared W
KLdtte
Supernet Ps
Student S
Segmenlation Latency
Figure Our co-searching framework, which optimizes two architectures dur- ing search (left orange) and distills from complex teacher to light student dur- ing training from scratch (right green).
This extension does not bring any overhead in memory us- age or size of supernet since the teacher and student share the same supernet weights W during the search process. Two sets of architectures iteratively optimized during search (please see details in Appendix F and apply the latency constraint only on the student, not on the teacher. Therefore, our searched teacher sophisticated network based on the same search space and supernet weights W used by the student. During training from scratch, we apply a distillation lass from teacher T to student
Edistillation E,eR (L ).)
(4)
KL denotes the KL divergence. qi and of are predicted logit for pixel from S and T. respectively. Equal weights (1.0) are assigned the segmentation loss and this distillation loss.
3.4 DERIVING DISCRETE ARCHITECTURES
Once the search is completed, we derive our discrete architecture from 3, and ?;
a, We select the optimum operators and expansion ratios by taking the argmaxx of a and y. We shrink the operator "skip connection" to obtain shallower architecture with less cells.
8: Different from (Liu et al. 2019a), for each cells,1 consider 3 and 33 as probabilities of two outputs from cellg.t-1 and cell. into cell,,t- Therefore, by taking the 1 = argmax, find the optimum position (cell,,t-) where downsample the current resolution ($ + +))
It is worth noting that, the multi-resolution branches will share both cell weights and feature maps their cells are the same operator type, spatial resolution, and expansion ratio. This design contributes to a faster network. Once cells in branches diverge, the sharing between the branches will be stopped and they become individual branches (See Figure ())
'hese values are obtained by solving equations derived from Table 2 order achieve balanced sensitiv- ities on different granularities: 10.42 x u1 0.01 Wy = 5.54 x w1, s.t. 22 + we + urs
For branch with two searchable downsampling positions, we consider the argmax over the joint proba- bilities (I7, (2) argmax (fh.s (2..12)-

---

Published as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATIONS
We use the Cityscapes (Cordts et al., 2016) testbed for both our architecture search and ablation studies, After that, we report our final accuracy and latency on Cityscapes, CamVid (Brostow et al.) 2008), and BDD (Yu et al. 2018b). In all experiments, the class mloU (mean Intersection over Union per class) and FPS (frame per second) are used as the metrics for accuracy and speed, respectively. Please Appendix G for dataset details.
In all experiments, we use Nvidia Geforee GTX 1080Ti for benchmarking the computing power. We employ the nigh-performance inference framework TensorRT v5.1.5 and report the inference speed. During this inference measurement, an image of a batch size of 1 is first loaded into the graphics memory, then the model is warmed up to reach a steady speed, and finally, the inference time is measured by running the model for six seconds. All experiments are performed under CUDA 10.0 and CUDNN V7. Our framework is implemented with PyTorch. The search, training, and latency measurement codes are available at https::///// //gttbbb com Sata//TAAAVVIA/atteeeee...
4.2 ARCHITECTURE SEARCH
We consider total of L 16 layers in the supernet and our downsample rate {8, 16, 32). In our work we use number of branches b = 2 by default, since more branches will suffer from high latency. We consider expansion ratio Xs.I X {4, 6, 8, 10, 12) for any "downsample rate". s and layer l. The multi-resolution branches have 1695 unique paths. For cells and expansion ratios, we have (1 + x 5) (11++1++13) + 53 x 3.4 x 1055 unique combinations. This results in a search space in the order of 1058, which is much larger and challenging, compared with preliminary studies.
Architecture search is conducted on Cityscapes training dataset. Figure 6 visualizes the best spatial resolution discovered (FasterSeg). Our FasterSeg achieved mutti-resolutions with proper depths. The two branches share the first three operators then diverge, and choose to aggregate outputs with downsample rates of 16 and 32. Oper- ators and expansion ratios are listed Table 7 in Appendix [ where the zoomed convolution is heavily used, suggesting the importance of low latency and large receptive field.
Downsample Rate (5)
Stem
stared by teo beanches tearch of 32
Head
18
Figure 6: FasterSeg network discovered by our NAS framework.
4.3 EVALUATION OF RESTIIRESOLUTION SEARCH SPACE AND CO-SEARCHING
We conduct ablation studies on Cityscapes evaluate the effective- ness of our NAS framework. More specifically,
Table Ablation studies different search and training strategies. mloU is measured on Cityscapes validation set. The input resolution is 1024 x 2048, O: operator; s: downsample rate; x: expansion
examine the im- pact of operators (O), downsam- ple rate (s), expansion ratios (). and also distillation on the accu- racy and latency. When we expand from single backbone (b = to FPS
ratios; b: number of branches; "="; knowledge distillation.
Settings
mloU (%)
FPS
FLOPs
#Params
0/ 8
66.9
177.8
G7.0 42.8
6.3 M
0 8,b=3
69.5
119.9
10.8 M
Teacher (T) Student (S) Co-searching
multi-branches (b = 2), drops but we gain a much improve- ment on mloU, indicating the multi- resolution design is beneficial for segmentation task. By enabling the search for expansion ratios (x), we discover a faster network with FPS 163.9 without sacrificing accuracy (70.5%), which proves that the scarchable superkernel gets the benefit from eliminating redundant channels while maintaining high accuracy. This is our student network (S) discovered in our co-searching framework (see below).
S: O, x/b 2
70.5
163.9
28.2 G
M
pruned
66.1
146.7
29.5 G
4.7M
T + S (FasterSeg)
73.1
163.9
28.2
4.4
We further evaluate the efficacy of our teacher-student co-searching framework. After the collabora- tively searching, obtain teacher architecture (T) and student architecture (S). As mentioned above, S is searched with searchable expansion ratios (X), achieving an FPS of 163.9 and an mloU of 70.5%. In contrast, when we directly compress the teacher (channel pruning via selecting the
7

---

Published as a conference paper at ICLR 2020
slimmest expansion ratio) and train with distillation from the well-trained original cumbersome teacher, it only achieved mloU = 66.1% with only FPS = 146.7, indicating that our architecture co- searching surpass the pruning based compression. Finally, when we adopt the knowledge distillation from the well-trained cumbersome teacher to our searched student, we boost the student's accuracy to 73.1 %, which is our final network FasterSeg. This demonstrates that both student discovered by co-searching and training with knowledge distillation from the teacher are vital for obtaining an accurate faster real-time segmentation model.
4.4 REAL-TIME SEMANTIC SEGMENTATION
In this section, we compare our FasterSeg with other works for real-time semantic segmentation on three popular scene segmentation datasets. Note that since target on real-time segmentation, we measure the mloU without any evaluation tricks like flipping, multi-scale, etc.
Cityscapes: We evaluate FasterSeg on Cityscapes val- idation and test sets. We use original image resolution of x024x2048 to measure both mloU and speed infer- ence. In Table 4 we see the superior FPS (163.9) of our FasterSeg, even under the maximum image resolution. This high FPS is over 1.3x faster than human-designed networks. Meanwhile, our FasterSeg still maintains competitive accuracy, which is 73.1 % on the val- idation set and 71.5% on the test set. This accuracy is achieved with only Cityscapes fine-annotated images, without using any extra data annotteed images, ImageNet, etc.).
Table mloU and inference FPS Ciytscapes validation and test
mloU (%)
Method
FPS
Resolution
val
test
ENet (Paszke al.
2016
58.3
76.9
512x 1024
ICNet Zhaa ct al
2018)
67.7
69.5
37.7
x024x2048
BiSeNet Yu al. 2018a
69.0
68.4
105.8
768x1536
CAS (Zhang el 2019)
71.6
70.5
108.0
768x
Fast-SCNN (Poudel el
2019
68.0
123.5
1024x2048
DF1-Seg-die (Li el al. 2019)
72.4
71.4
136.9
1024xx208
FasterSeg (ours)
73.1
71.5
163.9
x024x
CamVid: We directly transfer the searched architecture on Cityscapes to train on Cam Table 5 reveals that without sacrificing much accuracy, our FasterSeg achieved an FPS of 398.1. This extremely high speed is over 47% faster than the closest competitor in FPS (Yu al. 2018a), and over two times faster than the work with the best mloU (Zhang et al. 2019). This impressive result verifies both the high performance of FasterSeg and also the transfcrability of our NAS framework.
BDD: In addition, we also directly transfer the learned architecture to the BDD dataset. In Table 6 we compare our FasterSeg with the baseline provided by Yu et al. (2018b). Since no previous work has considered real-time segmentation on the BDD dataset, we get 15 times faster than the DRN-D- 22 with slightly higher mloU. Our FasterSeg still preserve the extremely fast speed and competitive accuracy on BDD.
Table mloU and inference on Cam Vid test
Table mloU and inference FPS on BDD validation set. The input resolution is 720 x 1280.
set. The input resolution 720
960.
Method
mIoU (%)
FPS
Method
mloU (%)
FPS
ENet (Paszke et al.
2016)
68.3
61.2
DRN-D D-22 YYu et al. 2017
53.2
21.0
ICNet (CNettZhap Zha al. 2018
67.1
27.8
DRN-D-38 Yu al. 2017
55.2
12.9
BiSeNet Yu et al. 2018a
65.6
269.1
FasterSeg (ours)
55.1
318.0
CAS (Zhang et al. FasterSeg (ours)
2019
71.2
169.0
71.1
398.1
CONCLUSION
We introduced novel multi-resolution NAS framework, leveraging successful design patterns in handcrafted networks for real-time segmentation. Our NAS framework can automatically discover FasterSeg, which achieved both extremely fast inference speed and competitive accuracy. Our search space is intrinsically of low-latency and is much larger and challenging due to flexible searchable expansion ratios. More importantly, we successfully addressed the "architecture collapse" problem, by proposing the novel regularized latency optimization of fine- grranaaaarrtt. We also demonstrate that by seamlessly extending to teacher-student co-searching, our NAS framework can boost the student's accuracy via effective distillation.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing. pp. 549-558, 2018.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston, Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint arXiv: 700. 2017.
Gabriel Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recog- nition using structure from motion point clouds. In European conference on computer vision, pp- 44-57. Springer, 2008
Blang-Chieh Chen, Maxwell Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In Advances in Neural Information Processing Systems, pp. 8699-8710 2018a.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018b.
Wuyang Chen, Ziyu Jiang, Zhangyang Wang, Kexin Cui, and Xiaoning Qian. Collaborative global- local networks for memory-efficient segmentation of ultra-high resolution images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8924-8933, 8933, 2019.
Yukang Chen, Gaofeng Meng, Qian Zhang, Xinbang Zhang, Liangchen Song, Shiming Xi- ang. and Chunhong Pan. Joint neural architecture search and quantization. arXiv preprint arXiv:1811.09426, 2018c.
An-Chich Cheng, Chich Hubert Lin, Da-Cheng Juan, Wei Wei, and Min Sun. Instanas: Instance- aware neural architecture search. arXiv preprint arXiv: 181 I. 10201, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, et al. Chamnet: Towards efficient network design through platform-an aware model adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattemn Recognition, pp. 11398- 11407, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3224 3234, 2019.
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications. NBS Applied Mathematics Series, 33, 1954.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan. Knowledge adaptation for efficient semantic segmentation. In Proceedings of the IEEE Conference on Com- pufer Vision and Pattern Recognition, pp. 578-587, 2019.

---

Published as a conference paper at ICLR 2020
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1XX3::503.005333 2015.
Xin Li, Yiming Zhou, Zheng Pan, and Jiashi Feng. Partial order pruning: for best speed/accuracy trade-off in neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9145-9153, 2019.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018a.
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 82-92, 2019a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0900 2018b.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2604-2613, 2019b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1430-,44,, 2015.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Chris J Maddison, Daniel Tarlow, and Tom Minka. A sampling, In Advances in Neural Information Processing Systems, pp. 3086-3099 2014.
Dmitrii Marin, Zijian He, Peter Vajda, Priyam Chatterjee, Sam Tsai, Fei Yang, and Yuri Boykov. Efficient segmentation: Learning downsampling near semantic boundaries. In Proceedings of the IEEE International Conference Computer Vision, pp. 2131-2141, 2019.
Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid. Fast neural architecture search of com- pact semantic segmentation models via auxiliary cells. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9126-9135, 2019.
Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. Enet: A deep neu- ral network architecture for real-time semantic segmentation. arXiv preprint 1eXXiv1606600 2016.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03266. 2018.
Rudra PK Poudel, Stephan Liwicki, and Roberto Cipolla. Fast-senn: fast semantic segmentation network. arXiv preprint 190p::402200 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Albert Shaw, Daniel Hunter, Forrest landola, and Sammy Sidhu. Squeezenas: Fast neural architec- ture search for faster semantic segmentation. arXiv preprint arXiv: 2019.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang. Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculeseu. Single-path nas: Designing hardware-efficient convnets in less than hours. arXiv preprint arXiv: 02877. 2019.
10

---

Published as a conference paper at ICLR 2020
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolttion representation learning for human pose estimation. arXiv preprint arXiv:1902.09220 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2820-2828, 2019.
Jingdong Wang. Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, al. Deep high-resolution representation learning for visual recognition. arXiv preprint arXiv: 1ef.::1908079999 2019.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 10734-10742, 2019.
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilat- eral segmentation network for real-time semantic segmentation. Proceedings of the European Conference on Computer Vision (ECCV), pp. 325-341 2018a.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 472-480, 2017.
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. 0dd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint arXiv.1805.04653 2018b.
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv: 18 12..8.2....85 2018c.
Yiheng Zhang, Zhaofan Qiu, Jingen Liu, Ting Yao, Dong Liu, and Tao Mei. Customizable archi- tecture search for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11641-11650, 2019.
Hengshuang Zhao, Xiaojuan Qi. Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 405-420, 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:161..0157,, 2016.

---

Published as a conference paper at ICLR 2020
A STEM AND HEAD MODULE
Stem: Our stem module aims quickly downsample the input image to 1 resolution while increas- ing the number of channels. The stem module consists of five 3 x 3 convolution layers, where the first, second, and fourth layer are of stride two and double the number of channels.
Head: As shown in Figure 1 feature map of shape (C2s x H x W) is first reduced in channels by convolution layer and bilinearly upsampled to match the shape of the other feature map (C, x 2H 2W). Then, two feature maps concatenated and fused together with 3 x 3 convolution layer. Note that we not necessarily have C2. 2C, because of the searchable expansion ratios.
B BRANCH SELECTION CRITERION
Since our searchable downsample rates s {8, 16, 32} and the number selected branches b = 2, supernet needs select branches of three possible combinations of resolutions: {8, 16), {8, 32}, and {16, 32}. For each combination, branches of two resolutions will be aggregated by our head module.
Our supernet selects the best b branches based on the criterion used in (Tan et al. 2019):
Target(m) = ACC(m) x LAT(()) 1"
(5)
T
where m a searched model aggregating b branches, with accuracy ACC(m) and latency (AT(m).
w is the weight factor defined as:
a, if LAT(m) ST
w
(6)
B otherwise.
We empirically set 3 -0.07 and the target latency T = 8.3 ms in our work.
C CGUMBEL-SOFTMAX TRICK FOR SEARCHING EXPANSION RATIOS
Formally, suppose we have our set of expansion ratios X' C N+, and we want to sample one ratio from X. For each x' we have an associated probability 7², where Σ = "Gumbel-Softmax" trick (Gumbell 1954 Maddison et al. 2014) helps us approximate differentiable sampling. We first sample "Gumbel-Noise o' log( log(u)) with Unif(0, 1]. Instead of selecting x such that j argmax,(7") we choose j = argmax, exp( lox(r)))
We set the temperature
m excp Togreet Ingrst))
parameter r = 1 in our work.
D NORMALIZED SCALARS a, 8,7
a, and y are all normalized scalars and implemented as softmax. They act probabilities associ- ating with each operator O E O, each predecessor's ' output O;-1, and each expansion ratio X E X, respectively:
Σ
a
1,, and
a 2 Vk. s.l
30, + +11 1,,s, and
ph By 2 0, 1
(7)
Σ
1, and
2 0, Vj
where is downsample rate and 7 is index of the layer in supernet.
E LATENCY ESTIMATION
We build latency lookup table that covers all possible situations and use this lookup table as building blocks estimate the relaxed latency. To verify the continuous relaxation of latency,
12

---

Published as a conference paper at ICLR 2020
randomly sample networks of different operators/downsample rates/expansion ratios out of the pernet M, and measured both the real and estimated latency. We estimate the network latency by accumulating all latencies of operators consisted in the network. In Figure B] correlation between the two measurements, with a correlation coefficient of 0.993. This accurate estimation of network latency benefits from the sequential design of our search space.
the high
F TRAINING
Given our supernet M, the overall optimization target (loss) during architecture search is:
Tseg (M) + A Lat(M)
(8)
We adopt cross-entropy with hhw--hadddeee--mmineeee as our segmentation loss Leeg. (M) is the continuously relaxed latency of supemet, and À is the balancing factor. We set À 0.01 in our work.
As the architecture 3, and y are now involved in the differentiable computation graph, they can be optimized using gradient descent. Similar in (Liu et al., 2019a). we adopt the first-order approx- imation ((Liu et al.) 2018b)), randomly split our training dataset into two disjoint sets trainA and trainB, and alternates the optimization between:
1. Update network weights W by Vw Teeg (MW, a, a, on trainA,
2. Update awcciiecttrreeeeee ax, B, by V Va,a, B Lung (M] (, (,)) LAM. on trainB.
When extend to our teacher-student co-searching where have two sets of architectures (o',, (3) and (as, As, 7s), our alternative optimization becomes:
1. Update network weights W by Vw T (M[W, OT, (T)) on trainA,
2. Update network weights W by Vw Loog (M], as, As, is) on trainA,
3. Update architecture oT; BT by STT ( aT, BT) on trainB.
4. Update architecture 0ss, 1s. 75
by Vos.8s.rs sep (M(W, as 7s)
Vagi8s LAT (s,7s) 7s) on trainB.
In search experiments, we first pretrain the supernet without updating the architecture parameter for 20 epochs, then start architecture searching for 30 epochs.
To train multiple widths within a superkemel, it is unrealistic to accumulate losses from all different width options in the super network. Therefore, approximate the multi-widths optimization by training the maximum and minimum expansion ratios. During pretraining the super network, train each operator with the smallest width, largest width, and random widths. When we are searching for architecture parameters, we train each operator with the smallest width, largest width, and the width that is sampled from the current y by Gumbel Softmax.
Optimizing Multiple Widths:
G BENCHMARK DATASETS
Cityscapes (Cordts et al. 2016) is a popular urban street scene dataset for semantic segmentation. contains high- high-quality level annotations, 2,975 images for training and 500 images for val- idation. In our experiments, we only use the fine annotated images without any coarse annotated image or extra data like ImageNet (Deng et al., (209).. For testing, it offers 1,525 images without ground-truth for fair comparison. These images all have resolution of 1024 x 2048 (HxW), where each pixel is annotated to pre-defined 19 classes.
CamVid (Brostow et al. 2008) is another street scene dataset, extracted from five video sequences taken from driving automobile. contains 701 images in total, where 367 for training, 101 for validation and 233 for testing. The images have resolution of 720 x 960 (Hx W) and 11 semantic categories.
BDD: The BDD Yu et al. 2018b) is recently released urban scene dataset. For the segmentation task, it contains 7,000 images for training and 1,000 for validation. The images have a resolution of 720 x 1280 (Hx W) and shared the same 19 semantic categories used in the Cityscapes.
13

---

Published as a conference paper at ICLR 2020
H ARCHITECTURE SEARCH IMPLEMENTATIONS
As stated in the second line of Eqn. 2 both to reduce spatial size and double the number of filters. Bilinear upsampling is used for all upsampling operations.
stride convolution used for all - 2s connections,
We conduct architecture search the Cityscapes dataset. We use 160 x 320 random image crops from half-resolution (512 x 1024) images the training set. Note that the original validation set or test set is never used for our architecture search. When learning network weights W, we use SGD optimizer with momentum 0.9 and weight decay of 5 5x10- We used the exponential learning rate decay of power 0.99. When learning the architecture parameters a, B, andy, we Adam optimizer with learning rate x 10-1, The entire architecture search optimization takes about 2 days on one 1080Ti GPU.
FASTERSEG STRUCTURE
In Table 7 we list the operators (O) and expansion ratios (x) selected by our FasterSeg. The down- sample rates s Table and Figure S match. We have the number of output channels Cout 8 x X- We observed that the zoomed convolution is heavily used, suggesting the importance of low latency and large receptive field.
Table Cells FasterSeg. Left: cells for branch with final downsample rate 16. Right: cells branch with final downsample rate of 32. s: downsample rate. x expansion ratio. cuout number of output channels.
Cell
Operator
c.out
Cell
Operator
X
c.out
1
x2
32
conv. x2
32
2
conv. x2
4
32
2
conv.
32
3
zoomed conv. x
3
zoomed cony. x2
4
4
zoomed conv.
8
64
zoomed conv,
5
zoomed conv. x2
16
4
64
5
conv. x2
32
6
zoomed
x2
64
6
conv. x2
4
32
zoomed conv.
7
zoomed cony. x2
16
4
64
8
zoomed conv. x2
64
zoomed cony. x2
16
192
9
zoomed conv. x2
32
128
9
zonmed conv. x2
128
10
zoomed x2
32
256
J VISUALIZATION
We show some visualized segmentation results in Figure 7 From the third to the forth column (*O,s|x s|x 8,
"O, 8, 2"), adding the extra branch of different scales provides more consistent segmentations (in the building and sidewalk area). From the fifth to the sixth column ("T pruned T FasterSeg). our FasterSeg surpasses the prunned teacher network, where the segmentation is smoother and more accurate.
traffic
vegetal
ferrain
sky
person
rider
car
truck
bus
taih
motarcycle
bicycle
Figure Visualization on Cityscapes validation set. Columns from left to right correspond to original images, ground truths, and results of "O,s|x s|x = 8,b = 1", "O, six = 8,b = 2", "T - pruned T", FasterSog (sec Table for annotation details), Best viewed in color.
14


Reviews:
[{"review_id": "SkevyWznYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- key problem: neural architecture search (NAS) to improve both accuracy and runtime efficiency of deep nets for semantic segmentation;\n- contributions: 1) a novel NAS search space leveraging multi-resolution branches, efficient operators (\"zoomed convolutions\"), and parametrized expansion ratios, 2) a decomposition and normalization of the latency objective to avoid a bias towards very fast but weak architectures, 3) a natural extension of the optimization problem to simultaneously search for teacher and student architectures in one pass, 4) a novel state-of-the-art efficient architecture (FasterSeg) found by the aforementioned NAS algorithm, 5) a detailed experimental evaluation on 3 datasets and an ablative analysis quantifying the benefits of the aforementioned contributions.\n\nRecommendation: Accept.\n\nKey reason 1: solid experimental results backing the claims.\n- When compared to related efficient architectures, the proposed method results in competitive accuracy at significantly higher frame rates.\n- This is validated on Cityscapes, CamVid, and BDD with the architecture found on Cityscapes. \n- The resulting architecture (FasterSeg) is actually interpretable and makes sense, extending the handcrafted architectures used as inspiration.\n- The ablative analysis shows that the numerous individual contributions are significant, esp. the multi-branch formulation and student co-searching.\n\nKey reason 2: well-motivated method with a collection of multiple novel contributions that are interesting and practical.\n- The multi-resolution branches formulation is simple and extends typical NAS focusing on single paths through the supernet.\n- Teacher/student co-searching via learning two sets of architectures in one supernet seems novel, simple, and effective. Always picking the largest expansion ratios for the teacher and applying a distillation loss in addition to the latency loss for the student is sensible and seems to beat the standard pruning approach at no significant extra cost during NAS.\n- The zoomed convolution operator seems like a novel efficient alternative to (expensive) dilated convolutions. Although it is very simple (bilinear downsampling -> 3x3 conv -> bilinear upsampling), it is not commonly used as an operator (as far as I know), and yet is found to be a key part of the final architecture (Table 7 appendix I) due to its low latency. The closest related operator / block I could think of might be blocks found in stacked hourglass networks (Newell et al).\n- The optimization of the expansion ratios using the Gumbel-Softmax trick is interesting, although this is also explored in the very recent paper by Shaw et al. 2019 (possibly the closest related work that should be discussed in a bit more depth in Section 2);\n- Decomposing and normalizing the latency objective to avoid \"architecture collapse\" (convergence to anemic architectures stemming from certain architectural factors dominating latency) is principled and effective.\n- Caveat regarding novelty: I could not find the ideas proposed here in the literature, but its hard to be sure due to 1) the recent explosion of NAS papers, 2) the simplicity of certain ideas (e.g., \"zoomed convolutions\").\n\n\nAdditional Feedback:\n- how is the student trained after NAS? Is the teacher first retrained from scratch? Is the student retrained from scratch on the teacher (after NAS or retraining)? in general, more details on what happens after co-searching would be helpful;\n- \"human designed CNN architectures achieve superior accuracy performance nowadays\": this is a surprising statement considering the cited NAS papers report performance improvements (e.g., Zoph and Le 2016);\n- missing reference also using multi-scale NAS for efficient and accurate semantic segmentation: \"Searching for Efficient Multi-Scale Architectures for Dense Image Prediction\", Chen et al, NeurIPS 2018;\n- missing reference on NAS for efficient semantic segmentation that also uses distillation: \"Fast neural architecture search of compact semantic segmentation models via auxiliary cells\", Nekrasov et al, CVPR 2019;\n- missing reference on joint NAS and quantization: \"Joint Neural Architecture Search and Quantization\", Chen et al, arxiv 2018;\n- \"we choose a sequential search space (rather than a directed acyclic graph of nodes (Liu et al., 2018b)), i.e., convolutional layers are sequentially stacked in our network\": \"stacked\" is confusing here;\n- \"we allow each cell to be individually searchable across the whole search space\": what do you mean? Anything beyond each cell containing different operators after learning?\n- if \\alpha = \\beta in eq. 6 of appendix C, then w and hence Target(m) does not depend on latency, isn't this a typo?\n- \"Gumbel-Max\" is typically called \"Gumbel-Softmax\" (cf. \"Categorical Reparameterization with Gumbel-Softmax\", Jang et al, ICLR'17);\n- typos: \"find them contribute\", \"the closet competitor\", \"is popular dense predictions\".", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkenQmO19S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural architecture search (NAS) algorithm which automatically finds a efficient network architecture, FasterSeg, for real time semantic segmentation. In designing a NAS algorithm the author takes cue from recent architectural advances introduced for faster segmentation as well as improved accuracy. For instances a) it explores and integrates multi-resolution branches from BiSeNet during NAS b) simultaneously optimizes the loss for accuracy and latency (as done in CAS algorithm) and c) knowledge distillation for semantic segmentation. However, the usage of these blocks in FasterSeg has been well refined to integrate with NAS search. To be precise, their improved version of latency loss avoids architectural collapse during latency-constrained search and it claims to be the first work to co-search for teacher and student network using NAS. Empirical experiments on benchmark dataset suggests that FasterSeg  is more than 30 percent faster with similar accuracy as state-of-the-art real-time segmentation algorithms.\n\nThis paper weakly leans towards rejection. Some of the contributing factors \n1) Overall presentation of algorithm leaves one more confused. Perhaps, the paper is targeted at small set of audience who primarily works on NAS. More about specific comment in 'Clarification'.\n2) There is not a single concrete contribution. For example, NAS search in semantic segmentation using cells and downsampling rates was done in Auto-Deeplab. Further, resource-constrained search for segmentation was introduced in Zhang et al while distillation for segmentation task was proposed by Liu et al. \n3) No doubt that it achieves improved efficiency. But at the cost of accuracy. On Camvid and BDD, the competitive algorithm is 1.7 % better in absolute terms. On cityscapes it performs on par. However, the large improvement in accuracy can be attributed to distillation process (Table 3: absolute 2%), without which the overall performance of NAS is suboptimal.\n\nClarification:\n1. It is not clear what is the form of initial network which is pre-trained for 20 epochs ? My guess is, initial network consists of b=3 branches with L=16 sequential layers and for each cell in a layer, the network pre-trains 5 operators as well as for different expansion ratios. Is it correct ?\n2. Can you explain 697 unique paths as well as 10^55 unique combinations ?\n3. It is noted that by default b=2 is used. However, in FasterSeg network shown in figure 6, I note three branches s={8,16,32}. Am I missing something ? Also, how more branches will introduce more latency ? Branches operate in parallel with max sensitivity s=0.01 and max L=16.\n4. Next, as pointed in 3.4 the discrete architecture is obtained by computing \\argmax_l over \\beta. In that case, there should only be single connection which branches out from s->2s. In figure 6, I note two branches from 8->16 (4th and 6th cell).\n5. If the teacher and student network shares the same weight, then what is the need for distillation ? Only difference I currently note is in the expansion ratio. May be you want to say same pretrained network ?\n6. Can you explain with example how \\gamma's are updated using backpropagation and lookup-table ? \n7. Are you employing STE for Gumbel-Max trick ? \n8. The individual terms in eq (3) optimizes for \\alpha, \\beta and \\gamma respectively ?\n9. In eq (2), each cell output O is linear combination of different operator ? \n10. Once the discrete architecture is obtained, is it retrained on cityscapes from scratch or fine-tuned ?\n\nRequest for ablation:\n1. What is the variation in NAS output with changes in \\lambda ? Precisely, can one tradeoff accuracy for improved latency just by tuning \\lambda ?\n2. What happens if teacher network is also optimised over \\gamma ? The difference between teacher and student will then only be in loss function.\n3. Currently, discrete architecture is greedily extracted. This need not be the best. Instead one can utilize sequential beam search (vitterbi algorithm). With this it is possible to visualise the accuracy and latency distribution of, say top 100 architecture obtained by NAS.\n\nMinor comments:\n1. Seachable -> Searchable\n\nUpdates:\nI read through the reviews of other reviewers as well as the rebuttal posted by authors. Overall, I am satisfied with the authors response and hence Improving my scores to Weak Accept.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xVcoRjFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an automatically designed semantic segmentation network utilising neural architecture search. The proposed method is discovered from a search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To calibrate the balance between the goals of high accuracy and low latency, the authors propose a decoupled and fine-grained latency regularization, that effectively overcomes the observed phenomenons that the searched networks are prone to “collapsing” to low-latency yet poor-accuracy models. Moreover, the authors extend the proposed method to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model’s accuracy. Experimental results on Cityscapes, CamVid, and BDD verified the efficacy of the proposed method.\n\nThe writing is clear and the presentation is good. I like the motivation of this paper. The problem solved in this paper aligns with reality.\n\nMy concerns regarding this paper are as below.\n1) The datasets used for evaluation are quite old except BDD, which make the results not so convincing. More experiments on more recent challenging semantic segmentation benchmarks are needed to verify the superiorities claimed in this paper. e.g., MHP v2.0 [Zhao et al., ACM MM 2018], etc.\n2) The methods compared in Tab.6 are quite old, please add comparisons with more recent SOTAs.\n3) Please add qualitative analysis to gain insight into the proposed method and to show why works better than other SOTAs.\n4) The results in Tab.4,5,6 of the proposed method are not the best for mIoU on testing protocol, is this a trade-off between acc and speed?\n\nBased on the above overall comments, I decide to give the rate of Weak Accept for this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents neural architecture search for semantic segmentation, with search space that integrates multi-resolution branches. The method also uses a regularization to overcome the issue of learned networks collapsing to low-latency but poor accuracy models. Another interesting contribution is a collaborative search procedure to simultaneously search for student and teacher networks in a single run. All reviewers agree that the proposed method is well-motivated and shows promising empirical results. Author response satisfactorily addressed most of the points raised by the reviewers. I recommend acceptance. 
Decision: Accept (Poster)