FasterSeg: Searching for Faster Real-time Semantic Segmentation

1 INTRODUCTION . Semantic segmentation predicts pixel-level annotations of different semantic categories for an image . Despite its performance breakthrough thanks to the prosperity of convolutional neural networks ( CNNs ) ( Long et al. , 2015 ) , as a dense structured prediction task , segmentation models commonly suffer from heavy memory costs and latency , often due to stacking convolutions and aggregating multiple-scale features , as well as the increasing input image resolutions . However , recent years witness the fast-growing demand for real-time usage of semantic segmentation , e.g. , autonomous driving . Such has motivated the enthusiasm on designing low-latency , more efficient segmentation networks , without sacrificing accuracy notably ( Zhao et al. , 2018 ; Yu et al. , 2018a ) . The recent success of neural architecture search ( NAS ) algorithms has shed light on the new horizon in designing better semantic segmentation models , especially under latency of other resource constraints . Auto-DeepLab ( Liu et al. , 2019a ) first introduced network-level search space to optimize resolutions ( in addition to cell structure ) for segmentation tasks . Zhang et al . ( 2019 ) and Li et al . ( 2019 ) adopted pre-defined network-level patterns of spatial resolution , and searched for operators and decoders with latency constraint . Despite a handful of preliminary successes , we observe that the successful human domain expertise in designing segmentation models appears to be not fully integrated into NAS frameworks yet . For example , human-designed architectures for real-time segmentation ( Zhao et al. , 2018 ; Yu et al. , 2018a ) commonly exploit multi-resolution branches with proper depth , width , operators , and downsample rates , and find them contributing vitally to the success : such flexibility has not been unleashed by existing NAS segmentation efforts . Furthermore , the trade-off between two ( somewhat conflicting ) goals , i.e. , high accuracy and low latency , also makes the search process unstable and prone to “ bad local minima ” architecture options . ∗Work done during the first two authors ’ research internships with Horizon Robotics Applied AI Lab . As the well-said quote goes : “ those who do not learn history are doomed to repeat it ” . Inheriting and inspired by the successful practice in hand-crafted efficient segmentation , we propose a novel NAS framework dubbed FasterSeg , aiming to achieve extremely fast inference speed and competitive accuracy . We designed a special search space capable of supporting optimization over multiple branches of different resolutions , instead of a single backbone . These searched branches are adaptively aggregated for the final prediction . To further balance between accuracy versus latency and avoiding collapsing towards either metric ( e.g. , good latency yet poor accuracy ) , we design a decoupled and fine-grained latency regularization , that facilitates a more flexible and effective calibration between latency and accuracy . Moreover , our NAS framework can be easily extended to a collaborative search ( co-searching ) , i.e. , jointly searching for a complex teacher network and a light-weight student network in a single run , whereas the two models are coupled by feature distillation in order to boost the student ’ s accuracy . We summarize our main contributions as follows : • A novel NAS search space tailored for real-time segmentation , where multi-resolution branches can be flexibility searched and aggregated . • A novel decoupled and fine-grained latency regularization , that successfully alleviates the “ architecture collapse ” problem in the latency-constrained search . • A novel extension to teacher-student co-searching for the first time , where we distill the teacher to the student for further accuracy boost of the latter . • Extensive experiments demonstrating that FasterSeg achieves extremely fast speed ( over 30 % faster than the closest manually designed competitor on CityScapes ) and maintains competitive accuracy . 2 RELATED WORK . Human-designed CNN architectures achieve good accuracy performance nowadays ( He et al. , 2016 ; Chen et al. , 2018b ; Wang et al. , 2019 ; Sun et al. , 2019 ) . However , designing architectures to balance between accuracy and other resource constraints ( latency , memory , FLOPs , etc . ) requires more human efforts . To free human experts from this challenging trade-off , neural architecture search ( NAS ) has been recently introduced and drawn a booming interest ( Zoph & Le , 2016 ; Brock et al. , 2017 ; Pham et al. , 2018 ; Liu et al. , 2018a ; Chen et al. , 2018a ; Bender et al. , 2018 ; Chen et al. , 2018c ; Gong et al. , 2019 ) . These works optimize both accuracy and resource utilization , via a combined loss function ( Wu et al. , 2019 ) , or a hybrid reward signal for policy learning ( Tan et al. , 2019 ; Cheng et al. , 2018 ) , or a constrained optimization formulation ( Dai et al. , 2019 ) . Most existing resource-aware NAS efforts focus on classification tasks , while semantic segmentation has higher requirements for preserving details and rich contexts , therefore posing more dilemmas for efficient network design . Fortunately , previous handcrafted architectures for real-time segmentation have identified several consistent and successful design patterns . ENet ( Paszke et al. , 2016 ) adopted early downsampling , and ICNet ( Zhao et al. , 2018 ) further incorporated feature maps from multiresolution branches under label guidance . BiSeNet ( Yu et al. , 2018a ) fused a context path with fast downsampling and a spatial path with smaller filter strides . More works target on segmentation efficiency in terms of computation cost ( He et al. , 2019 ; Marin et al. , 2019 ) and memory usage ( Chen et al. , 2019 ) . Their multi-resolution branching and aggregation designs ensure sufficiently large receptive fields ( contexts ) while preserving high-resolution fine details , providing important clues on how to further optimize the architecture . There have been recent studies that start pointing NAS algorithms to segmentation tasks . AutoDeepLab ( Liu et al. , 2019a ) pioneered in this direction by searching the cells and the networklevel downsample rates , to flexibly control the spatial resolution changes throughout the network . Zhang et al . ( 2019 ) and Li et al . ( 2019 ) introduced resource constraints into NAS segmentation . A multi-scale decoder was also automatically searched ( Zhang et al. , 2019 ) . However , compared with manually designed architectures , those search models still follow a single-backbone design and did not fully utilize the prior wisdom ( e.g. , multi-resolution branches ) in designing their search spaces . Lastly , we briefly review knowledge distillation ( Hinton et al. , 2015 ) , that aims to transfer learned knowledge from a sophisticated teacher network to a light-weight student , to improve the ( more efficient ) student ’ s accuracy . For segmentation , Liu et al . ( 2019b ) and Nekrasov et al . ( 2019 ) proposed to leverage knowledge distillation to improve the accuracy of the compact model and speed-up convergence . There was no prior work in linking distillation with NAS yet , and we will introduce the extension of FasterSeg by integrating teacher-student model collaborative search for the first time . 3 FASTERSEG : FASTER REAL-TIME SEGMENTATION . Our FasterSeg is discovered from an efficient and multi-resolution search space inspired by previous manual design successes . A fine-grained latency regularization is proposed to overcome the challenge of “ architecture collapse ” ( Cheng et al. , 2018 ) . We then extend our FasterSeg to a teacherstudent co-searching framework , further resulting in a lighter yet more accurate student network . 3.1 EFFICIENT SEARCH SPACE WITH MULTI-RESOLUTION BRANCHING . The core motivation behind our search space is to search multi-resolution branches with overall low latency , which has shown effective in previous manual design works ( Zhao et al. , 2018 ; Yu et al. , 2018a ) . Our NAS framework automatically selects and aggregates branches of different resolutions , based on efficient cells with searchable superkernels . 3.1.1 SEARCHABLE MULTI-RESOLUTION BRANCHES 1 2 4 8 16 32 D ow ns am pl e R at e ( 𝒔 ) CFF CFFsub24 sub124 1 2 4 8 16 32 D ow ns am pl e R at e ( 𝒔 ) spatial path context path FFM ARM ARM Figure 2 : Our multi-resolution search space covers existing manual designs for real-time segmentation ( unused cells omitted ) . Top : ICNet ( Zhao et al. , 2018 ) . Bottom : BiSeNet ( Yu et al. , 2018a ) Inspired by ( Liu et al. , 2019a ) , we enable searching for spatial resolutions within the L-layer cells ( Figure 1 ) , where each cell takes inputs from two connected predecessors and outputs two feature maps of different resolutions . Hand-crafted networks for real-time segmentation found multi-branches of different resolutions to be effective ( Zhao et al. , 2018 ; Yu et al. , 2018a ) . However , architectures explored by current NAS algorithms are restricted to a single backbone . Our goal is to select b ( b > 1 ) branches of different resolutions in this L-layer framework . Specifically , we could choose b different final output resolutions for the last layer of cells , and decode each branch via backtrace ( section 3.4 ) . This enables our NAS framework to explore b individual branches with different resolutions , which are progressively “ learned to be aggregated ” by the head module ( Figure 1 ) . We follow the convention to increase the number of channels at each time of resolution downsampling . To enlarge the model capacity without incurring much latency , we first downsample the input image to 18 original scale with our stem module , and then set our searchable downsample rates s ∈ { 8 , 16 , 32 } . Figure 2 shows that our multi-resolution search space is able to cover existing human-designed networks for real-time segmentation . See Appendix B for branch selection details . 3.1.2 CHOOSING EFFICIENT OPERATORS WITH LARGE RECEPTIVE FIELDS . As we aim to boost the inference latency , the speed of executing an operator is a direct metric ( rather than indirect metrics like FLOPs ) for selecting operator candidatesO . Meanwhile , as we previously discussed , it is also important to ensure sufficiently large receptive field for spatial contexts . We analyze typical operators , including their common surrogate latency measures ( FLOPs , parameter numbers ) , and their real-measured latency on an NVIDIA 1080Ti GPU with TensorRT library , and their receptive fields , as summarized in Table 1 . Compared with standard convolution , group convolution is often used for reducing FLOPs and number of parameters ( Sandler et al. , 2018 ; Ma et al. , 2018 ) . Convolving with two groups has the same receptive field with a standard convolution but is 13 % faster , while halving the parameter amount ( which might not be preferable as it reduces the model learning capacity ) . Dilated convolution has an enlarged receptive field and is popular in dense predictions ( Chen et al. , 2018b ; Dai et al. , 2017 ) . However , as shown in Table 1 ( and as widely acknowledged in engineering practice ) , dilated convolution ( with dilation rate 2 ) suffers from dramatically higher latency , although that was not directly reflected in FLOPs nor parameter numbers . In view of that , we design a new variant called “ zoomed convolution ” , where the input feature map is sequentially processed with bilinear downsampling , standard convolution , and bilinear upsampling . This special design enjoys 40 % lower latency and 2 times larger receptive field compared to standard convolution . Our search space hence consists of the following operators : • skip connection • 3×3 conv . • 3×3 conv . ×2 • “ zoomed conv. ” : bilinear downsampling + 3×3 conv . + bilinear upsampling • “ zoomed conv . ×2 ” : bilinear downsampling + 3×3 conv . ×2 + bilinear upsampling As mentioned by Ma et al . ( 2018 ) , network fragmentation can significantly hamper the degree of parallelism , and therefore practical efficiency . Therefore , we choose a sequential search space ( rather than a directed acyclic graph of nodes ( Liu et al. , 2018b ) ) , i.e. , convolutional layers are sequentially stacked in our network . In Figure 1 , each cell is differentiable ( Liu et al. , 2018b ; 2019a ) and will contain only one operator , once the discrete architecture is derived ( section 3.4 ) . It is worth noting that we allow each cell to be individually searchable across the whole search space .

Summary: This paper presents an automatically designed semantic segmentation network utilising neural architecture search. The proposed method is discovered from a search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To calibrate the balance between the goals of high accuracy and low latency, the authors propose a decoupled and fine-grained latency regularization, that effectively overcomes the observed phenomenons that the searched networks are prone to “collapsing” to low-latency yet poor-accuracy models. Moreover, the authors extend the proposed method to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model’s accuracy. Experimental results on Cityscapes, CamVid, and BDD verified the efficacy of the proposed method.