Title: FASTERSEG: SEARCHING FOR FASTER REAL-TIME SEMANTIC SEGMENTATION

Abstract: We present FasterSeg, an automatically designed semantic segmentation network with not only state-of-the-art performance but also faster speed than current methods. Utilizing neural architecture search (NAS), FasterSeg is discovered from a novel and broader search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To better calibrate the balance between the goals of high accuracy and low latency, we propose a decoupled and fine-grained latency regularization, that effectively overcomes our observed phenomenons that the searched networks are prone to "collapsing" to low-latency yet poor-accuracy models. Moreover, we seamlessly extend FasterSeg to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model's accuracy. Experiments on popular segmentation benchmarks demonstrate the competency of FasterSeg. For example, FasterSeg can run over 30% faster than the closest manually designed competitor on Cityscapes, while maintaining comparable accuracy.

1. INTRODUCTION:

Semantic segmentation predicts pixel-level annotations of different semantic categories for an image. Despite its performance breakthrough thanks to the prosperity of convolutional neural networks (CNNs) (Long et al., 2015) , as a dense structured prediction task, segmentation models commonly suffer from heavy memory costs and latency, often due to stacking convolutions and aggregating multiple-scale features, as well as the increasing input image resolutions. However, recent years witness the fast-growing demand for real-time usage of semantic segmentation, e.g., autonomous driving. Such has motivated the enthusiasm on designing low-latency, more efficient segmentation networks, without sacrificing accuracy notably (Zhao et al., 2018; Yu et al., 2018a) .

The recent success of neural architecture search (NAS) algorithms has shed light on the new horizon in designing better semantic segmentation models, especially under latency of other resource constraints. Auto-DeepLab (Liu et al., 2019a) first introduced network-level search space to optimize resolutions (in addition to cell structure) for segmentation tasks. Zhang et al. (2019) and Li et al. (2019) adopted pre-defined network-level patterns of spatial resolution, and searched for operators and decoders with latency constraint. Despite a handful of preliminary successes, we observe that the successful human domain expertise in designing segmentation models appears to be not fully integrated into NAS frameworks yet. For example, human-designed architectures for real-time segmentation (Zhao et al., 2018; Yu et al., 2018a) commonly exploit multi-resolution branches with proper depth, width, operators, and downsample rates, and find them contributing vitally to the success: such flexibility has not been unleashed by existing NAS segmentation efforts. Furthermore, the trade-off between two (somewhat conflicting) goals, i.e., high accuracy and low latency, also makes the search process unstable and prone to "bad local minima" architecture options.

As the well-said quote goes: "those who do not learn history are doomed to repeat it". Inheriting and inspired by the successful practice in hand-crafted efficient segmentation, we propose a novel NAS framework dubbed FasterSeg, aiming to achieve extremely fast inference speed and competitive accuracy. We designed a special search space capable of supporting optimization over multiple branches of different resolutions, instead of a single backbone. These searched branches are adaptively aggregated for the final prediction. To further balance between accuracy versus latency and avoiding collapsing towards either metric (e.g., good latency yet poor accuracy), we design a decoupled and fine-grained latency regularization, that facilitates a more flexible and effective calibration between latency and accuracy. Moreover, our NAS framework can be easily extended to a collaborative search (co-searching), i.e., jointly searching for a complex teacher network and a light-weight student network in a single run, whereas the two models are coupled by feature distillation in order to boost the student's accuracy. We summarize our main contributions as follows:

• A novel NAS search space tailored for real-time segmentation, where multi-resolution branches can be flexibility searched and aggregated. • A novel decoupled and fine-grained latency regularization, that successfully alleviates the "architecture collapse" problem in the latency-constrained search. • A novel extension to teacher-student co-searching for the first time, where we distill the teacher to the student for further accuracy boost of the latter.

• Extensive experiments demonstrating that FasterSeg achieves extremely fast speed (over 30% faster than the closest manually designed competitor on CityScapes) and maintains competitive accuracy.

2. RELATED WORK:

Human-designed CNN architectures achieve good accuracy performance nowadays (He et al., 2016; Chen et al., 2018b; Wang et al., 2019; Sun et al., 2019) . However, designing architectures to balance between accuracy and other resource constraints (latency, memory, FLOPs, etc.) requires more human efforts. To free human experts from this challenging trade-off, neural architecture search (NAS) has been recently introduced and drawn a booming interest (Zoph & Le, 2016; Brock et al., 2017; Pham et al., 2018; Liu et al., 2018a; Chen et al., 2018a; Bender et al., 2018; Chen et al., 2018c; Gong et al., 2019) . These works optimize both accuracy and resource utilization, via a combined loss function (Wu et al., 2019) , or a hybrid reward signal for policy learning (Tan et al., 2019; Cheng et al., 2018) , or a constrained optimization formulation (Dai et al., 2019) .

Most existing resource-aware NAS efforts focus on classification tasks, while semantic segmentation has higher requirements for preserving details and rich contexts, therefore posing more dilemmas for efficient network design. Fortunately, previous handcrafted architectures for real-time segmentation have identified several consistent and successful design patterns. ENet (Paszke et al., 2016) adopted early downsampling, and ICNet (Zhao et al., 2018) further incorporated feature maps from multiresolution branches under label guidance. BiSeNet (Yu et al., 2018a) fused a context path with fast downsampling and a spatial path with smaller filter strides. More works target on segmentation efficiency in terms of computation cost (He et al., 2019; Marin et al., 2019) and memory usage (Chen et al., 2019) . Their multi-resolution branching and aggregation designs ensure sufficiently large receptive fields (contexts) while preserving high-resolution fine details, providing important clues on how to further optimize the architecture.

There have been recent studies that start pointing NAS algorithms to segmentation tasks. Auto-DeepLab (Liu et al., 2019a) pioneered in this direction by searching the cells and the networklevel downsample rates, to flexibly control the spatial resolution changes throughout the network. Zhang et al. (2019) and Li et al. (2019) introduced resource constraints into NAS segmentation. A multi-scale decoder was also automatically searched (Zhang et al., 2019) . However, compared with manually designed architectures, those search models still follow a single-backbone design and did not fully utilize the prior wisdom (e.g., multi-resolution branches) in designing their search spaces.

Lastly, we briefly review knowledge distillation (Hinton et al., 2015) , that aims to transfer learned knowledge from a sophisticated teacher network to a light-weight student, to improve the (more efficient) student's accuracy. For segmentation, Liu et al. (2019b) and Nekrasov et al. (2019) proposed to leverage knowledge distillation to improve the accuracy of the compact model and speed-up convergence. There was no prior work in linking distillation with NAS yet, and we will introduce the extension of FasterSeg by integrating teacher-student model collaborative search for the first time.

3. FASTERSEG: FASTER REAL-TIME SEGMENTATION:

Our FasterSeg is discovered from an efficient and multi-resolution search space inspired by previous manual design successes. A fine-grained latency regularization is proposed to overcome the challenge of "architecture collapse" (Cheng et al., 2018) . We then extend our FasterSeg to a teacherstudent co-searching framework, further resulting in a lighter yet more accurate student network.

3.1. EFFICIENT SEARCH SPACE WITH MULTI-RESOLUTION BRANCHING:

The core motivation behind our search space is to search multi-resolution branches with overall low latency, which has shown effective in previous manual design works (Zhao et al., 2018; Yu et al., 2018a) . Our NAS framework automatically selects and aggregates branches of different resolutions, based on efficient cells with searchable superkernels. Downsample Rate (𝒔) Figure 1 : The multi-resolution branching search space for FasterSeg, where we aim to optimize multiple branches with different output resolutions. These outputs are progressively aggregated together in the head module. Each cell is individually searchable and may have two inputs and two outputs, both of different downsampling rates (s). Inside each cell, we enable searching for expansion ratios within a single superkernel. (Zhao et al., 2018) . Bottom: BiSeNet (Yu et al., 2018a) Inspired by (Liu et al., 2019a) , we enable searching for spatial resolutions within the L-layer cells (Figure 1 ), where each cell takes inputs from two connected predecessors and outputs two feature maps of different resolutions. Hand-crafted networks for real-time segmentation found multi-branches of different resolutions to be effective (Zhao et al., 2018; Yu et al., 2018a) . However, architectures explored by current NAS algorithms are restricted to a single backbone.

1 2 3 L-1 L •••••• •••••• •••••• 𝑪𝒆𝒍𝒍 𝒔,𝒍&𝟏 𝑪𝒆𝒍𝒍 𝒔,𝒍 skip connect 3×3 conv. 3×3 conv. ×2 zoomed 3×3 conv. zoomed 3×3 conv. ×2

3.1.1. SEARCHABLE MULTI-RESOLUTION BRANCHES:

Our goal is to select b (b > 1) branches of different resolutions in this L-layer framework. Specifically, we could choose b different final output resolutions for the last layer of cells, and decode each branch via backtrace (section 3.4). This enables our NAS framework to explore b individual branches with different resolutions, which are progressively "learned to be aggregated" by the head module (Figure 1 ).

We follow the convention to increase the number of channels at each time of resolution downsampling. To enlarge the model capacity without incurring much latency, we first downsample the input image to 1 8 original scale with our stem module, and then set our searchable downsample rates s ∈ {8, 16, 32}. Figure 2 shows that our multi-resolution search space is able to cover existing human-designed networks for real-time segmentation. See Appendix B for branch selection details.

3.1.2. CHOOSING EFFICIENT OPERATORS WITH LARGE RECEPTIVE FIELDS:

As we aim to boost the inference latency, the speed of executing an operator is a direct metric (rather than indirect metrics like FLOPs) for selecting operator candidates O. Meanwhile, as we previously discussed, it is also important to ensure sufficiently large receptive field for spatial contexts. We analyze typical operators, including their common surrogate latency measures (FLOPs, parameter numbers), and their real-measured latency on an NVIDIA 1080Ti GPU with TensorRT library, and their receptive fields, as summarized in Table 1 .

Compared with standard convolution, group convolution is often used for reducing FLOPs and number of parameters (Sandler et al., 2018; Ma et al., 2018) . Convolving with two groups has the same receptive field with a standard convolution but is 13% faster, while halving the parameter amount (which might not be preferable as it reduces the model learning capacity). Dilated convolution has an enlarged receptive field and is popular in dense predictions (Chen et al., 2018b; Dai et al., 2017) . However, as shown in Table 1 (and as widely acknowledged in engineering practice), dilated convolution (with dilation rate 2) suffers from dramatically higher latency, although that was not directly reflected in FLOPs nor parameter numbers. In view of that, we design a new variant called "zoomed convolution", where the input feature map is sequentially processed with bilinear downsampling, standard convolution, and bilinear upsampling. This special design enjoys 40% lower latency and 2 times larger receptive field compared to standard convolution. Our search space hence consists of the following operators:

• skip connection • 3×3 conv. • 3×3 conv. ×2

• "zoomed conv.": bilinear downsampling + 3×3 conv. + bilinear upsampling • "zoomed conv. ×2": bilinear downsampling + 3×3 conv. ×2 + bilinear upsampling As mentioned by Ma et al. (2018) , network fragmentation can significantly hamper the degree of parallelism, and therefore practical efficiency. Therefore, we choose a sequential search space (rather than a directed acyclic graph of nodes (Liu et al., 2018b )), i.e., convolutional layers are sequentially stacked in our network. In Figure 1 , each cell is differentiable (Liu et al., 2018b; 2019a) and will contain only one operator, once the discrete architecture is derived (section 3.4). It is worth noting that we allow each cell to be individually searchable across the whole search space.

3.1.3. SEARCHABLE SUPERKERNEL FOR EXPANSION RATIOS:

We further give each cell the flexibility to choose different channel expansion ratios. In our work, we search for the width of the connection between successive cells. That is however non-trivial due to the exponentially possible combinations of operators and widths. To tackle this problem, we propose a differentiably searchable superkernel, i.e., directly searching for the expansion ratio χ within a single convolutional kernel which supports a set of ratios X ⊆ N + . Inspired by (Yu et al., 2018c) and (Stamoulis et al., 2019) , from slim to wide our connections incrementally take larger subsets of input/output dimensions from the superkernel. During the architecture search, for each superkernel, only one expansion ratio is sampled, activated, and back-propagated in each step of stochastic gradient descent. This design contributes to a simplified and memory-efficient super network and is implemented via the renowned "Gumbel-Softmax" trick (see Appendix C for details).

To follow the convention to increase the number of channels as resolution downsampling, in our search space we consider the width = χ × s, where s ∈ {8, 16, 32}. We allow connections between each pair of successive cells flexibly choose its own expansion ratio, instead of using a unified single expansion ratio across the whole search space.

3.1.4. CONTINUOUS RELAXATION OF SEARCH SPACE:

Denote the downsample rate as s and layer index as l. To facilitate the search of spatial resolutions, we connect each cell with two possible predecessors' outputs with different downsample rates:

I s,l = β 0 s,l O s 2 s,l-1 + β 1 s,l O s s,l-1 (1)

Each cell could have at most two outputs with different downsample rates into its successors:

O s s,l = |O| k=1 α k s,l O k s s,l (I s,l , χ j s,l , stride = 1) O s 2s,l = |O| k=1 α k s,l O k s 2s,l (I s,l , χ j s,l , stride = 2).

(2)

The expansion ratio χ j s,l is sampled via "Gumbel-Softmax" trick according to p(χ = χ j s,l ) = γ j s,l . Here, α, β, and γ are all normalized scalars, associated with each operator O k ∈ O, each predecessor's output O l-1 , and each expansion ratio χ ∈ X , respectively (Appendix D). They encode the architectures to be optimized and derived. Low latency is desirable yet challenging to optimize. Previous works (Cheng et al., 2018; Zhang et al., 2019) observed that during the search procedure, the supernet or search policy often fall into bad "local minimums" where the generated architectures are of extremely low latency but with poor accuracy, especially in the early stage of exploration. In addition, the searched networked tend to use more skip connections instead of choosing low expansion ratios (Shaw et al., 2019) . This problem is termed as "architecture collapse" in our paper. The potential reason is that, finding architectures with extremely low latency (e.g. trivially selecting the most light-weight operators) is significantly easier than discovering meaningful compact architectures of high accuracy.

3.2. REGULARIZED LATENCY OPTIMIZATION WITH FINER GRANULARITY:

To address this "architecture collapse" problem, we for the first time propose to leverage a fine-grained, decoupled latency regularization. We first achieve the continuous relaxation of latency similar to the cell operations in section 3.1.4, via replacing the operator O in Eqn. 1 and 2 with the corresponding latency. We build a latency lookup table that covers all possible operators to support the estimation of the relaxed latency. Figure 3 demonstrates the high correlation of 0.993 between the real and estimated latencies (see details in appendix E). We argue that the core reason behind the "architecture collapse" problem is the different sensitivities of supernet to operator O, downsample rate s, and expansion ratio χ.

Operators like "3×3 conv. ×2" and "zoomed conv." have a huge gap in latency. Similar latency gap (though more moderate) exists between slim and wide expansion ratios. However, downsample rates like "8" and "32" do not differ much, since resolution downsampling also brings doubling of the number of both input and output channels.

We quantitatively compared the influence of O, s, and χ towards the supernet latency, by adjusting one of the three aspects and fixing the other two. Taking O as the example, we first uniformly initialize β and γ, and calculate ∆Latency(O) as the gap between the supernet which dominantly takes the slowest operators and the one adopts the fastest. Similar calculations were performed for s and χ. Values of ∆Latency in Table 2 indicate the high sensitivity of the supernet's latency to operators and expansion ratios, while not to resolutions. Figure 4 (a) shows that the unregularized latency optimization will bias the supernet towards light-weight operators and slim expansion ratios to quickly minimize the latency, ending up with problematic architectures with low accuracy.

Based on this observation, we propose a regularized latency optimization leveraging different granularities of our search space. We decouple the calculation of supernet's latency into three granularities of our search space (O, s, χ), and regularize each aspect with a different factor:

Latency(O, s, χ) = w 1 Latency(O|s, χ) + w 2 Latency(s|O, χ) + w 3 Latency(χ|O, s) (3) where we by default set w 1 = 0.001, w 2 = 0.997, w 3 = 0.0021 . This decoupled and fine-grained regularization successfully addresses this "architecture collapse" problem, as shown in Figure 4 Knowledge Distillation is an effective approach to transfer the knowledge learned by a large and complex network (teacher T ) into a much smaller network (student S). In our NAS framework, we can seamlessly extend to teacher-student cosearching, i.e., collaboratively searching for two networks in a single run (Figure 5 ). Specifically, we search a complex teacher and light-weight student simultaneously via adopting two sets of architectures in one supernet: (α T , β T ) and (α S , β S , γ S ). Note that the teacher does not search the expansion ratios and always select the widest one. This extension does not bring any overhead in memory usage or size of supernet since the teacher and student share the same supernet weights W during the search process. Two sets of architectures are iteratively optimized during search (please see details in Appendix F), and we apply the latency constraint only on the student, not on the teacher. Therefore, our searched teacher is a sophisticated network based on the same search space and supernet weights W used by the student . During training from scratch, we apply a distillation loss from teacher T to student S:

distillation = E i∈R KL(q s i ||q t i ).

(4) KL denotes the KL divergence. q s i and q t i are predicted logit for pixel i from S and T , respectively. Equal weights (1.0) are assigned to the segmentation loss and this distillation loss.

3.4. DERIVING DISCRETE ARCHITECTURES:

Once the search is completed, we derive our discrete architecture from α, β, and γ: • α, γ: We select the optimum operators and expansion ratios by taking the argmax of α and γ. We shrink the operator "skip connection" to obtain a shallower architecture with less cells. • β: Different from (Liu et al., 2019a) , for each cell s,l we consider β 0 and β 1 as probabilities of two outputs from cell s 2 ,l-1 and cell s,l-1 into cell s,l . Therefore, by taking the l * = argmax l (β 0 s,l ), we find the optimum position (cell s,l * ) where to downsample the current resolution ( s 2 → s).2 It is worth noting that, the multi-resolution branches will share both cell weights and feature maps if their cells are of the same operator type, spatial resolution, and expansion ratio. This design contributes to a faster network. Once cells in branches diverge, the sharing between the branches will be stopped and they become individual branches (See Figure 6 ).

4. EXPERIMENTS 4.1 DATASETS AND IMPLEMENTATIONS:

We use the Cityscapes (Cordts et al., 2016) as a testbed for both our architecture search and ablation studies. After that, we report our final accuracy and latency on Cityscapes, CamVid (Brostow et al., 2008) , and BDD (Yu et al., 2018b) . In all experiments, the class mIoU (mean Intersection over Union per class) and FPS (frame per second) are used as the metrics for accuracy and speed, respectively. Please see Appendix G for dataset details.

In all experiments, we use Nvidia Geforce GTX 1080Ti for benchmarking the computing power. We employ the high-performance inference framework TensorRT v5.1.5 and report the inference speed. During this inference measurement, an image of a batch size of 1 is first loaded into the graphics memory, then the model is warmed up to reach a steady speed, and finally, the inference time is measured by running the model for six seconds. All experiments are performed under CUDA 10.0 and CUDNN V7. Our framework is implemented with PyTorch. The search, training, and latency measurement codes are available at https://github.com/TAMU-VITA/FasterSeg.

4.2. ARCHITECTURE SEARCH:

We consider a total of L = 16 layers in the supernet and our downsample rate s ∈ {8, 16, 32}. In our work we use number of branches b = 2 by default, since more branches will suffer from high latency. We consider expansion ratio χ s,l ∈ X = {4, 6, 8, 10, 12} for any "downsample rate" s and layer l. The multi-resolution branches have 1695 unique paths. For cells and expansion ratios, we have (1 + 4 × 5) (15+14+13) + 5 3 ≈ 3.4 × 10 55 unique combinations. This results in a search space in the order of 10 58 , which is much larger and challenging, compared with preliminary studies. Architecture search is conducted on Cityscapes training dataset. Figure 6 visualizes the best spatial resolution discovered (FasterSeg). Our FasterSeg achieved mutli-resolutions with proper depths.

The two branches share the first three operators then diverge, and choose to aggregate outputs with downsample rates of 16 and 32. Operators and expansion ratios are listed in Table 7 in Appendix I, where the zoomed convolution is heavily used, suggesting the importance of low latency and large receptive field. We conduct ablation studies on Cityscapes to evaluate the effectiveness of our NAS framework. More specifically, we examine the impact of operators (O), downsample rate (s), expansion ratios (χ), and also distillation on the accuracy and latency. When we expand from a single backbone (b = 1) to multi-branches (b = 2), our FPS drops but we gain a much improvement on mIoU, indicating the multiresolution design is beneficial for segmentation task. By enabling the search for expansion ratios (χ), we discover a faster network with FPS 163.9 without sacrificing accuracy (70.5%), which proves that the searchable superkernel gets the benefit from eliminating redundant channels while maintaining high accuracy. This is our student network (S) discovered in our co-searching framework (see below).

4.3. EVALUATION OF MULTI-RESOLUTION SEARCH SPACE AND CO-SEARCHING:

We further evaluate the efficacy of our teacher-student co-searching framework. After the collaboratively searching, we obtain a teacher architecture (T ) and a student architecture (S). As mentioned above, S is searched with searchable expansion ratios (χ), achieving an FPS of 163.9 and an mIoU of 70.5%. In contrast, when we directly compress the teacher (channel pruning via selecting the slimmest expansion ratio) and train with distillation from the well-trained original cumbersome teacher, it only achieved mIoU = 66.1% with only FPS = 146.7, indicating that our architecture cosearching surpass the pruning based compression. Finally, when we adopt the knowledge distillation from the well-trained cumbersome teacher to our searched student, we boost the student's accuracy to 73.1%, which is our final network FasterSeg. This demonstrates that both a student discovered by co-searching and training with knowledge distillation from the teacher are vital for obtaining an accurate faster real-time segmentation model.

4.4. REAL-TIME SEMANTIC SEGMENTATION:

In this section, we compare our FasterSeg with other works for real-time semantic segmentation on three popular scene segmentation datasets. Note that since we target on real-time segmentation, we measure the mIoU without any evaluation tricks like flipping, multi-scale, etc. (Zhao et al., 2018) 67.7 69.5 37.7 1024×2048 BiSeNet (Yu et al., 2018a) 69.0 68.4 105.8 768×1536 CAS (Zhang et al., 2019) 71.6 70.5 108.0 768×1536 Fast-SCNN (Poudel et al., 2019) CamVid: We directly transfer the searched architecture on Cityscapes to train on CamVid. Table 5 reveals that without sacrificing much accuracy, our FasterSeg achieved an FPS of 398.1. This extremely high speed is over 47% faster than the closest competitor in FPS (Yu et al., 2018a) , and is over two times faster than the work with the best mIoU (Zhang et al., 2019) . This impressive result verifies both the high performance of FasterSeg and also the transferability of our NAS framework.

BDD::

In addition, we also directly transfer the learned architecture to the BDD dataset. In Table 6 we compare our FasterSeg with the baseline provided by Yu et al. (2018b) . Since no previous work has considered real-time segmentation on the BDD dataset, we get 15 times faster than the DRN-D-22 with slightly higher mIoU. Our FasterSeg still preserve the extremely fast speed and competitive accuracy on BDD. (Zhao et al., 2018) 67.1 27.8 BiSeNet (Yu et al., 2018a) 65.6 269.1 CAS (Zhang et al., 2019) 71.2 169.0 FasterSeg (ours) 71.1 398.1

Table 6 : mIoU and inference FPS on BDD validation set. The input resolution is 720 × 1280.

Method mIoU (%) FPS:

DRN-D-22 (Yu et al., 2017) 53.2 21.0 DRN-D-38 (Yu et al., 2017) 55.2 12.9 FasterSeg (ours) 55.1 318.0

5. CONCLUSION:

We introduced a novel multi-resolution NAS framework, leveraging successful design patterns in handcrafted networks for real-time segmentation. Our NAS framework can automatically discover FasterSeg, which achieved both extremely fast inference speed and competitive accuracy. Our search space is intrinsically of low-latency and is much larger and challenging due to flexible searchable expansion ratios. More importantly, we successfully addressed the "architecture collapse" problem, by proposing the novel regularized latency optimization of fine-granularity. We also demonstrate that by seamlessly extending to teacher-student co-searching, our NAS framework can boost the student's accuracy via effective distillation.

A STEM AND HEAD MODULE:

Stem: Our stem module aims to quickly downsample the input image to 1 8 resolution while increasing the number of channels. The stem module consists of five 3 × 3 convolution layers, where the first, second, and fourth layer are of stride two and double the number of channels.

Head: As shown in Figure 1 , feature map of shape (C 2s × H × W ) is first reduced in channels by a 1 × 1 convolution layer and bilinearly upsampled to match the shape of the other feature map (C s × 2H × 2W ). Then, two feature maps are concatenated and fused together with a 3 × 3 convolution layer. Note that we not necessarily have C 2s = 2C s because of the searchable expansion ratios.

B BRANCH SELECTION CRITERION:

Since our searchable downsample rates s ∈ {8, 16, 32} and the number of selected branches b = 2, our supernet needs to select branches of three possible combinations of resolutions: {8, 16}, {8, 32}, and {16, 32}. For each combination, branches of two resolutions will be aggregated by our head module.

Our supernet selects the best b branches based on the criterion used in (Tan et al., 2019) :

EQUATION

where m is a searched model aggregating b branches, with accuracy ACC(m) and latency LAT (m). w is the weight factor defined as:

EQUATION

We empirically set α = β = -0.07 and the target latency T = 8.3 ms in our work.

C "GUMBEL-SOFTMAX" TRICK FOR SEARCHING EXPANSION RATIOS:

Formally, suppose we have our set of expansion ratios X ⊆ N + , and we want to sample one ratio χ from X . For each χ i we have an associated probability γ i , where (Gumbel, 1954; Maddison et al., 2014) helps us approximate differentiable sampling. We first sample a "Gumbel-Noise"

|X | i=1 γ i = 1. "Gumbel-Softmax" trick

o i = -log(-log(u)) with u ∼ Unif[0, 1]. Instead of selecting χ j such that j = argmax i (γ i ), we choose j = argmax i ( exp( log(γ i )+o i τ ) |X | m=1 exp( log(γ m )+o m τ )

). We set the temperature parameter τ = 1 in our work.

D NORMALIZED SCALARS α, β, γ α, β, and γ are all normalized scalars and implemented as softmax. They act as probabilities associating with each operator O k ∈ O, each predecessor's output O l-1 , and each expansion ratio χ ∈ X , respectively:

EQUATION

where s is downsample rate and l is index of the layer in our supernet.

E LATENCY ESTIMATION:

We build a latency lookup table that covers all possible situations and use this lookup table as building blocks to estimate the relaxed latency. To verify the continuous relaxation of latency, we randomly sample networks of different operators/downsample rates/expansion ratios out of the supernet M, and measured both the real and estimated latency. We estimate the network latency by accumulating all latencies of operators consisted in the network. In Figure 3 , we can see the high correlation between the two measurements, with a correlation coefficient of 0.993. This accurate estimation of network latency benefits from the sequential design of our search space.

F TRAINING:

Given our supernet M, the overall optimization target (loss) during architecture search is:

EQUATION

We adopt cross-entropy with "online-hard-element-mining" as our segmentation loss L seg . Lat(M) is the continuously relaxed latency of supernet, and λ is the balancing factor. We set λ = 0.01 in our work.

As the architecture α, β, and γ are now involved in the differentiable computation graph, they can be optimized using gradient descent. Similar in (Liu et al., 2019a) , we adopt the first-order approximation ( (Liu et al., 2018b) ), randomly split our training dataset into two disjoint sets trainA and trainB, and alternates the optimization between:

1. Update network weights W by ∇ W L seg (M|W, α, β, γ) on trainA, 2. Update architecture α, β, γ by ∇ α,β,γ L seg (M|W, α, β, γ)+λ•∇ α,β,γ LAT (M|W, α, β, γ) on trainB.

When we extend to our teacher-student co-searching where we have two sets of architectures (α T , β T ) and (α S , β S , γ S ), our alternative optimization becomes: In all search experiments, we first pretrain the supernet without updating the architecture parameter for 20 epochs, then start architecture searching for 30 epochs.

Optimizing Multiple Widths: To train multiple widths within a superkernel, it is unrealistic to accumulate losses from all different width options in the super network. Therefore, we approximate the multi-widths optimization by training the maximum and minimum expansion ratios. During pretraining the super network, we train each operator with the smallest width, largest width, and 2 random widths. When we are searching for architecture parameters, we train each operator with the smallest width, largest width, and the width that is sampled from the current γ by Gumbel Softmax.

G BENCHMARK DATASETS:

Cityscapes (Cordts et al., 2016 ) is a popular urban street scene dataset for semantic segmentation. It contains high-quality pixel-level annotations, 2,975 images for training and 500 images for validation. In our experiments, we only use the fine annotated images without any coarse annotated image or extra data like ImageNet (Deng et al., 2009) . For testing, it offers 1,525 images without ground-truth for a fair comparison. These images all have a resolution of 1024 × 2048 (H×W), where each pixel is annotated to pre-defined 19 classes.

CamVid (Brostow et al., 2008) is another street scene dataset, extracted from five video sequences taken from a driving automobile. It contains 701 images in total, where 367 for training, 101 for validation and 233 for testing. The images have a resolution of 720 × 960 (H×W) and 11 semantic categories.

The BDD (Yu et al., 2018b ) is a recently released urban scene dataset. For the segmentation task, it contains 7,000 images for training and 1,000 for validation. The images have a resolution of 720 × 1280 (H×W) and shared the same 19 semantic categories as used in the Cityscapes.

:

These values are obtained by solving equations derived from Table

in order to achieve balanced sensitivities on different granularities: 10.42 × w1 = 0.01 × w2 = 5.54 × w1, s.t. w1 + w2 + w3 = 1.2 For a branch with two searchable downsampling positions, we consider the argmax over the joint probabilities (l * 1 , l * 2 ) = argmax l 1 ,l 2 (β 0 s,l 1 • β 0 2s,l 2 ).

Figures and Tables:

Figure2: Our multi-resolution search space covers existing manual designs for real-time segmentation (unused cells omitted). Top: IC-Net(Zhao et al., 2018). Bottom: BiSeNet(Yu  et al., 2018a)

Figure 3: Correlation between network latency and its estimation via our latency lookup table (linear coefficient: 0.993). Red line indicates "y = x".

Proposed fine-grained latency regularization.

Figure 4: Comparing mIoU (%) and latency (ms) between supernets. The search is conducted on the Cityscapes training set and mIoU is measured on the validation set.

Figure 6: FasterSeg network discovered by our NAS framework.

Update network weights W by ∇ W L seg (M|W, α T , β T ) on trainA, 2. Update network weights W by ∇ W L seg (M|W, α S , β S , γ S ) on trainA, 3. Update architecture α T , β T by ∇ α T ,β T ,γ T L seg (M|W, α T , β T ) on trainB. 4. Update architecture α S , β S , γ S by ∇ α S ,β S ,γ S L seg (M|W, α S , β S , γ S ) + λ • ∇ α S ,β S ,γ S LAT (M|W, α S , β S , γ S ) on trainB.

Specifications

Table content: <table><tr><td>Layer Type</td><td>Latency (ms)</td><td>FLOPs (G)</td><td>Params (M)</td><td>RF*</td></tr><tr><td>conv.</td><td>0.15</td><td>1.21</td><td>0.59</td><td>1</td></tr><tr><td>conv. group2</td><td>0.13 (-13%)</td><td>0.60 (-50%)</td><td>0.29 (-51%)</td><td>1</td></tr><tr><td>conv. dilation2</td><td>0.25 (+67%)</td><td>1.10 (-9%)</td><td>0.59</td><td>2</td></tr><tr><td>zoomed conv.</td><td>0.09 (-40%)</td><td>0.30 (-75%)</td><td>0.59</td><td>2</td></tr></table>

Supernet

Table content: <table><tr><td colspan="2">'s sensitivity to latency</td></tr><tr><td colspan="2">under different granularities. Input size: (1,</td></tr><tr><td>3, 1024, 2048).</td><td/></tr><tr><td>Granularity</td><td>∆Latency (ms)</td></tr><tr><td>O (operator)</td><td>10.42</td></tr><tr><td>s (downsample rate)</td><td>0.01</td></tr><tr><td>χ (expansion ratio)</td><td>5.54</td></tr></table>

Ablation studies of different search and training strategies. mIoU is measured on Cityscapes validation set. The input resolution is 1024 × 2048. O: operator; s: downsample rate; χ: expansion ratios; b: number of branches; " ": knowledge distillation.

Table content: <table><tr><td/><td>Settings</td><td>mIoU (%)</td><td>FPS</td><td colspan="2">FLOPs #Params</td></tr><tr><td colspan="2">O, s|χ = 8, b = 1</td><td>66.9</td><td colspan="2">177.8 27.0 G</td><td>6.3 M</td></tr><tr><td colspan="2">O, s|χ = 8, b = 2</td><td>69.5</td><td colspan="2">119.9 42.8 G</td><td>10.8 M</td></tr><tr><td/><td colspan="4">Teacher (T ) Student (S) Co-searching</td></tr><tr><td colspan="2">S: O, s, χ|b = 2</td><td>70.5</td><td colspan="2">163.9 28.2 G</td><td>4.4 M</td></tr><tr><td>T</td><td>pruned T</td><td>66.1</td><td colspan="2">146.7 29.5 G</td><td>4.7 M</td></tr><tr><td>T</td><td>S(FasterSeg)</td><td>73.1</td><td colspan="2">163.9 28.2 G</td><td>4.4 M</td></tr></table>

mIoU and inference FPS on Ciytscapes validation and test sets.

Table content: <table><tr><td>Method</td><td colspan="2">mIoU (%)</td><td>FPS</td><td>Resolution</td></tr><tr><td/><td>val</td><td>test</td><td/><td/></tr><tr><td>ENet (Paszke et al., 2016)</td><td>-</td><td>58.3</td><td>76.9</td><td>512×1024</td></tr><tr><td>ICNet</td><td/><td/><td/><td/></tr></table>

Meanwhile, our FasterSeg still maintains competitive accuracy, which is 73.1% on the validation set and 71.5% on the test set. This accuracy is achieved with only Cityscapes fine-annotated images, without using any extra data (coarse-annotated images, ImageNet, etc.).

Table content: <table><tr><td>Cityscapes: We evaluate</td><td/><td/></tr><tr><td>FasterSeg on Cityscapes val-</td><td/><td/></tr><tr><td>idation and test sets. We</td><td/><td/></tr><tr><td>use original image resolution</td><td/><td/></tr><tr><td>of 1024×2048 to measure</td><td/><td/></tr><tr><td>both mIoU and speed infer-</td><td/><td/></tr><tr><td>ence. In Table 4, we see</td><td/><td/></tr><tr><td>the superior FPS (163.9) of</td><td/><td/></tr><tr><td>our FasterSeg, even under the</td><td/><td>68.6 68.0 123.5 1024×2048</td></tr><tr><td>maximum image resolution. This high FPS is over 1.3×</td><td>DF1-Seg-d8 (Li et al., 2019) FasterSeg (ours)</td><td>72.4 71.4 136.9 1024×2048 73.1 71.5 163.9 1024×2048</td></tr><tr><td>faster than human-designed</td><td/><td/></tr><tr><td>networks.</td><td/><td/></tr></table>

mIoU and inference FPS on CamVid test set. The input resolution is 720 × 960.

Table content: <table><tr><td>Method</td><td>mIoU (%)</td><td>FPS</td></tr><tr><td>ENet (Paszke et al., 2016)</td><td>68.3</td><td>61.2</td></tr><tr><td>ICNet</td><td/><td/></tr></table>

Peer review (iclr2020):
Summary: - key problem: neural architecture search (NAS) to improve both accuracy and runtime efficiency of deep nets for semantic segmentation; - contributions: 1) a novel NAS search space leveraging multi-resolution branches, efficient operators (*zoomed convolutions*), and parametrized expansion ratios, 2) a decomposition and normalization of the latency objective to avoid a bias towards very fast but weak architectures, 3) a natural extension of the optimization problem to simultaneously search for teacher and student architectures in one pass, 4) a novel state-of-the-art efficient architecture (FasterSeg) found by the aforementioned NAS algorithm, 5) a detailed experimental evaluation on 3 datasets and an ablative analysis quantifying the benefits of the aforementioned contributions. Recommendation: Accept. Key reason 1: solid experimental results backing the claims. - When compared to related efficient architectures, the proposed method results in competitive accuracy at significantly higher frame rates. - This is validated on Cityscapes, CamVid, and BDD with the architecture found on Cityscapes. - The resulting architecture (FasterSeg) is actually interpretable and makes sense, extending the handcrafted architectures used as inspiration. - The ablative analysis shows that the numerous individual contributions are significant, esp. the multi-branch formulation and student co-searching. Key reason 2: well-motivated method with a collection of multiple novel contributions that are interesting and practical. - The multi-resolution branches formulation is simple and extends typical NAS focusing on single paths through the supernet. - Teacher/student co-searching via learning two sets of architectures in one supernet seems novel, simple, and effective. Always picking the largest expansion ratios for the teacher and applying a distillation loss in addition to the latency loss for the student is sensible and seems to beat the standard pruning approach at no significant extra cost during NAS. - The zoomed convolution operator seems like a novel efficient alternative to (expensive) dilated convolutions. Although it is very simple (bilinear downsampling -> 3x3 conv -> bilinear upsampling), it is not commonly used as an operator (as far as I know), and yet is found to be a key part of the final architecture (Table 7 appendix I) due to its low latency. The closest related operator / block I could think of might be blocks found in stacked hourglass networks (Newell et al). - The optimization of the expansion ratios using the Gumbel-Softmax trick is interesting, although this is also explored in the very recent paper by Shaw et al. 2019 (possibly the closest related work that should be discussed in a bit more depth in Section 2); - Decomposing and normalizing the latency objective to avoid *architecture collapse* (convergence to anemic architectures stemming from certain architectural factors dominating latency) is principled and effective. - Caveat regarding novelty: I could not find the ideas proposed here in the literature, but its hard to be sure due to 1) the recent explosion of NAS papers, 2) the simplicity of certain ideas (e.g., *zoomed convolutions*). Additional Feedback: - how is the student trained after NAS? Is the teacher first retrained from scratch? Is the student retrained from scratch on the teacher (after NAS or retraining)? in general, more details on what happens after co-searching would be helpful; - *human designed CNN architectures achieve superior accuracy performance nowadays*: this is a surprising statement considering the cited NAS papers report performance improvements (e.g., Zoph and Le 2016); - missing reference also using multi-scale NAS for efficient and accurate semantic segmentation: *Searching for Efficient Multi-Scale Architectures for Dense Image Prediction*, Chen et al, NeurIPS 2018; - missing reference on NAS for efficient semantic segmentation that also uses distillation: *Fast neural architecture search of compact semantic segmentation models via auxiliary cells*, Nekrasov et al, CVPR 2019; - missing reference on joint NAS and quantization: *Joint Neural Architecture Search and Quantization*, Chen et al, arxiv 2018; - *we choose a sequential search space (rather than a directed acyclic graph of nodes (Liu et al., 2018b)), i.e., convolutional layers are sequentially stacked in our network*: *stacked* is confusing here; - *we allow each cell to be individually searchable across the whole search space*: what do you mean? Anything beyond each cell containing different operators after learning? - if alpha = _x0008_eta in eq. 6 of appendix C, then w and hence Target(m) does not depend on latency, isn*t this a typo? - *Gumbel-Max* is typically called *Gumbel-Softmax* (cf. *Categorical Reparameterization with Gumbel-Softmax*, Jang et al, ICLR*17); - typos: *find them contribute*, *the closet competitor*, *is popular dense predictions*.