Sample and Computation Redistribution for Efficient Face Detection

1 INTRODUCTION . Face detection is a long-standing problem in computer vision with many applications , such as face alignment ( Bulat & Tzimiropoulos , 2017 ) , face reconstruction ( Feng et al. , 2018 ) , face attribute analysis ( Zhang et al. , 2018 ; Pan et al. , 2018 ) , and face recognition ( Schroff et al. , 2015 ; Deng et al. , 2019 ) . Following the pioneering work of ( Viola & Jones , 2004 ) , numerous face detection algorithms have been designed . Among them , the single-shot anchor-based approaches ( Najibi et al. , 2017 ; Zhang et al. , 2017b ; Tang et al. , 2018 ; Li et al. , 2019 ; Ming et al. , 2019 ; Deng et al. , 2020 ; Liu et al. , 2020 ; Zhu et al. , 2020 ) have recently demonstrated very promising performance . In particular , on the most challenging face detection dataset , WIDER FACE ( Yang et al. , 2016 ) , the average precision ( AP ) on its hard validation set has been boosted to 93.4 % by TinaFace ( Zhu et al. , 2020 ) . Even though TinaFace ( Zhu et al. , 2020 ) achieves impressive results on unconstrained face detection , it employs large-scale ( e.g . 1 , 650 pixels ) testing , which consumes huge amounts of computational resources . In addition , TinaFace design is based on a generic object detector ( i.e . RetinaNet ( Lin et al. , 2017b ) ) , directly taking the classification network as the backbone , tiling dense anchors on the multi-scale feature maps ( i.e . P2 to P7 of neck ) , and adopting heavy head designs . Without considering the prior of faces , the network design of TinaFace is thus redundant and sub-optimal . One approach of optimizing such networks ’ performance is computation redistribution . Since directly taking the backbone of the classification network for object detection is sub-optimal , the recent CR-NAS ( Liang et al. , 2020 ) reallocates the computation across different resolutions to obtain a more balanced Effective Receptive Field ( ERF ) , leading to higher detection performance . In BFbox ( Liu & Tang , 2020 ) , a face-appropriate search space is designed , based on the observation of scale distribution gap between general object detection and face detection . In ASFD ( Zhang et al. , 2020a ) , a differential architecture search is employed to discover optimized feature enhance modules for efficient multi-scale feature fusion and context enhancement . Even though ( Liu & Tang , 2020 ; Zhang et al. , 2020a ) have realized the limitation of directly applying general backbone , neck and head settings to face detection , CR-NAS ( Liang et al. , 2020 ) only focuses the optimization on backbone , BFbox ( Liu & Tang , 2020 ) neglects the optimization of head , and ASFD ( Zhang et al. , 2020a ) only explores the best design for neck . Another optimization approach , is the sample redistribution across different scales . Due to the extremely large scale variance of faces in real-world scenarios , different scale augmentation strategies are employed to introduce scale adaptation into the face detector . The most widely used scale augmentation approaches include random square crop ( Zhang et al. , 2017b ; Deng et al. , 2020 ; Zhu et al. , 2020 ) and data anchor sampling ( Tang et al. , 2018 ) . Nevertheless , the scale augmentation parameters in these methods are manually designed for all different network structures . Therefore , traditional multi-scale training in face detection is also tedious and sub-optimal . Since VGA resolution ( 640 × 480 ) is widely used for efficient face detection on numerous mobile phones and digital cameras , we focus on efficient face detection from low-resolution images in this paper . In Fig 1 ( a ) , we give the cumulative face scale distribution on the WIDER FACE validation dataset . Under the VGA resolution , most of the faces ( 78.93 % ) in WIDER FACE are smaller than 32×32 pixels . Under this specific scale distribution , both network structure and scale augmentation need to be optimized . In this work , we present a meticulously designed methodology of search space optimization , that addresses both the redistribution between the backbone , neck and head , and the sample redistribution between the most needed scales . As the structure of a face detector determines the distribution of computation and is the key in determining its accuracy and efficiency , we first discover principles of computation distribution under different flop regimes . Inspired by ( Radosavovic et al. , 2020 ) , we control the degrees of freedom and reduce the search space . More specifically , we randomly sample model architectures with different configurations on backbone ( stem and four stages ) , neck and head . Based on the statistics of these models , we compute the empirical bootstrap ( Efron & Tibshirani , 1994 ) and estimate the likely range in which the best models fall . To further decrease the complexity of the search space , we divide the computation ratio estimation for backbone and the whole detector into two steps . To handle extreme scale variations in face detection , we also design a search-able zoom-in and zoom-out space , specified by discrete scales and binary probabilities . In experiments , the proposed computation redistribution and sample redistribution yield significant and consistent improvement on various compute regimes , even surpassing a range of state-of-the-art face detectors by using much fewer flops as shown in Fig . 1 ( b ) . To sum up , this paper makes following contributions : • We have proposed a simplified search space , as well as a two-step search strategy for computation redistribution across different components ( backbone , neck and head ) of a face detector . The proposed computation redistribution method can easily boost detection performance through random search . • We have designed a search-able zoom-in and zoom-out space for face-specific scale augmentation , which automatically redistributes more training samples for shallow stages , enhancing the detection performance on small faces . • Extensive experiments conducted on WIDER FACE demonstrate the significantly improved accuracy and efficiency trade-off of the proposed SCRFD across a wide range of compute regimes . 2 RELATED WORK . Face Detection . To deal with extreme variations ( e.g . scale , pose , illumination and occlusion ) in face detection ( Yang et al. , 2016 ) , most of the recent single-shot face detectors focus on improving the anchor sampling/matching or feature enhancement . SSH ( Najibi et al. , 2017 ) builds detection modules on different feature maps with a rich receptive field . S3FD ( Zhang et al. , 2017b ) introduces an anchor compensation strategy by offsetting anchors for outer faces . PyramidBox ( Tang et al. , 2018 ) formulates a data-anchor-sampling strategy to increase the proportion of small faces in the training data . DSFD ( Li et al. , 2019 ) introduces small faces supervision signals on the backbone , which implicitly boosts the performance of pyramid features . Group sampling ( Ming et al. , 2019 ) emphasizes the importance of the ratio for matched and unmatched anchors . RetinaFace ( Deng et al. , 2020 ) employs deform-able context modules and additional landmark annotations to improve the performance of face detection . HAMBox ( Liu et al. , 2020 ) finds that many unmatched anchors in the training phase also have strong localization ability and proposes an online high-quality anchor mining strategy to assign high-quality anchors for outer faces . BFbox ( Liu & Tang , 2020 ) employs a single-path one-shot search method ( Guo et al. , 2019 ) to jointly optimize the backbone and neck for face detector . ASFD ( Zhang et al. , 2020a ) explores a differential architecture search to discover optimized feature enhance modules for efficient multi-scale feature fusion and context enhancement . All these methods are either designed by expert experience or partially optimized on backbone , neck and head . By contrast , we search for computation redistribution across different components ( backbone , neck and head ) of a face detector across a wide range of compute regimes . Neural Architecture Search . Given a fixed search space of possible networks , Neural Architecture Search ( NAS ) automatically finds a good model within the search space . DetNAS ( Chen et al. , 2019b ) adopts the evolution algorithm for the backbone search to boost object detection on COCO ( Lin et al. , 2014 ) . By contrast , CR-NAS ( Liang et al. , 2020 ) reallocates the computation across different stages within the backbone to improve object detection . NAS-FPN ( Ghiasi et al. , 2019 ) uses reinforcement learning to search the proper FPN for general object detection . As there is an obvious distribution gap between COCO ( Lin et al. , 2014 ) and WIDER FACE ( Yang et al. , 2016 ) , the experience in the above methods is not directly applicable for face detection but gives us an inspiration that the backbone , neck and head can be optimized to enhance the performance of face detection . Inspired by RegNet ( Radosavovic et al. , 2020 ) , we optimize the computation distribution on backbone , neck and head based on the statistics from a group of random sampled models . We successfully reduce the search space and find the stable computation distribution under a particular complex regime , which significantly improves the model ’ s performance . 3 METHODOLOGY . To efficiently and accurately detect small faces from low-resolution images ( e.g . VGA 640 × 480 ) , we propose two methodologies that , when combined , outperform the state-of-the-art . In Sec . 3.1 , we explore the computation redistribution across different stages of backbone , as well as different components ( i.e . backbone , neck and head ) of the whole detector , given a pre-defined computation budget . Then , in Sec . 3.2 , we investigate the redistribution of positive training samples across different scales of feature maps by searching optimized scale augmentations . 3.1 COMPUTATION REDISTRIBUTION . As illustrated in Fig . 2 , we apply our search method on a network consisting of ( 1 ) RetinaNet ( Lin et al. , 2017a ) , with ResNet ( He et al. , 2016 ) as the backbone , ( 2 ) Path Aggregation Feature Pyramid Network ( PAFPN ) ( Liu et al. , 2018 ) as the neck , and ( 3 ) stacked 3 × 3 convolutional layers for the head . Despite the generally simple structure , the total number of possible network configurations of the search space becomes unwieldy . Therefore , we attempt to simplify the tremendous search space and arrive at a low-dimensional design space , consisting of simple and effective networks . 3.1.1 SEARCH SPACE DESIGN . Inspired by RegNet ( Radosavovic et al. , 2020 ) , we explore the structures of face detectors , assuming fixed standard network blocks ( i.e. , basic residual or bottleneck blocks with a fixed bottleneck ratio of 4 ) . In our case , the structure of a face detector includes : • the backbone stem , three 3× 3 convolutional layers with w1 output channels ( He et al. , 2019a ) . • the backbone body , four stages ( i.e . C2 , C3 , C4 and C5 ) operating at progressively reduced resolution , with each stage consisting of a sequence of identical blocks . For each stage i , the degrees of freedom include the number of blocks di ( i.e . network depth ) and the block width wi ( i.e . number of channels ) . • the neck , a multi-scale feature aggregation module by a top-down path and a bottom-up path with n channels ( Liu et al. , 2018 ) . • the head , with hi channels of m blocks to predict face scores and regress face boxes . The search space can be initially designed as follows . As the channel number of the stem is equal to the block width of the first residual block in C2 , the degree of freedom of the stem w1 can be merged into w2 . In addition , we employ a shared head design for three-scale of feature maps and fix the channel number for all 3 × 3 convolutional layers within the heads . Therefore , we reduce the degrees of freedom to three within our neck and head design : ( 1 ) output channel number n for neck , ( 2 ) output channel number h for head , and ( 3 ) the number of 3 × 3 convolutional layers m. We perform uniform sampling of n ≤ 256 , h ≤ 256 , and m ≤ 6 ( both n and h are divisible by 8 ) . The backbone search space has 8 degrees of freedom as there are 4 stages and each stage i has 2 parameters : the number of blocks di and block width wi . Following RegNet ( Radosavovic et al. , 2020 ) , we perform uniform sampling of di ≤ 24 and wi ≤ 512 ( wi is divisible by 8 ) . As state-ofthe-art backbones have increasing widths ( Radosavovic et al. , 2020 ) , we also constrain the search space , according to the principle of wi+1 ≥ wi .

Summary: The authors proposed face detection algorithm based on the optimized network architecture and data sampling strategy.  Their novelties are two-fold: one is Computation Redistribution (CR) which optimally reallocates the computation between the backbone, neck and head of the model, and another is Sample Redistribution (SR) which automatically redistributes more training samples for shallow stages.  The ablation study showed that both of proposed methods are effective and the comparative study showed that their whole pipeline achieved the highest accuracy among state-of-the-art methods on the public WIDER FACE dataset.