Multi-Task Learning via Scale Aware Feature Pyramid Networks and Effective Joint Head

1 INTRODUCTION . In the past few years , object detection and instance segmentation results were rapidly improved by the powerful baseline system Mask R-CNN ( He et al . ( 2017 ) ) , which extends Faster R-CNN ( Girshick ( 2015 ) ) by adding a branch for predicting an object mask in parallel with the existing part for bounding box recognition . This method is conceptually natural and offers extensibility and robustness , shows a surprisingly smooth , flexible , and fast system for instance segmentation results . However , this remarkable multi-task learning method suffers from a common problem of modern detection methods . That is scale variation , since Convolutional Neural Network is sensitive to scales . And what ’ s more , performing multi-tasks needs more credible feature extraction execution and multi-scale complementary features integration . Therefore , it is urgent to tackle this problem . Feature pyramid is a common practice . FPN ( Lin et al . ( 2017a ) ) augmented a top-down path with lateral connections for object detection . It exploits the inherent multi-scale , pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost . Rethinking the extracted multi-scale features of general FPN , the top-down pathway FPN only introduces high-level semantic information to low-level feature , while ignore the role of low-level feature for localization . FPN still has room for improvement . Another aspect for improvement of Mask R-CNN in multi-task learning is about the parallel isolated branches of Mask R-CNN . The segmentation branch of Mask R-CNN is based on the output of Region Proposal Network ( RPN ) in training stage , which ignores the inherently tie in those two tasks and is inconsistent with testing processes . In common sense , instance segmentation is connected detection based on the bounding box strictly , which is more meticulous than the bounding box . However , the bounding box is easy to obtain than the masking label . It is worthy of trying to explore and enhance the interrelations between object detection and instance segmentation . 1Code will be available soon . In this paper , we make a natural extension of Mask R-CNN architecture , merging the detection branch and the instance segmentation branch into single branch . This smart framework contribution of our work named EJ-Head ( Effective Joint Head ) , including three operations : “ Interleaving ” , “ Enriched Feature ” and “ Boundary Refinement ” . EJ-Head promotes both two tasks consistently and provides an example for improving multi-task learning . In general , we improve two aspects in Mask R-CNN by proposing SA-FPN ( Scale Aware Feature Pyramid Networks ) and EJ-Head ( Effective Joint Head ) . Experimental results on the challenging COCO benchmark show that when using our proposed modules , detection and instance segmentation performances are improved by about 1.6 and 1.4 percent AP increment in box AP and mask AP , respectively . The main contributions of our work highlighted as follows . ( 1 ) We propose SA-FPN ( Scale Aware Feature Pyramid Networks ) , which effectively integrates multi-scale complementary feature and solves the problem of scale variation in an innovative way . ( 2 ) We slickly mix the mask branch and detection branch into one branch and introduce EJ-Head ( Effective Joint Head ) , which can reinforce each task and also eliminate the gap between training and testing processes . ( 3 ) We propose a newly enhanced Mask R-CNN , which provides a reference and is helpful for further research on multi-task learning . 2 RELATED WORK . 2.1 DEEP OBJECT DETECTORS . Deep learning based methods ( Girshick et al . ( 2014 ) , He et al . ( 2015b ) ) have tremendously pushed forward the remarkable progress in object detection over a short period of time . Mainstream object detection frameworks roughly fall into two categories . Two-stage methods like Faster RCNN ( Girshick ( 2015 ) ) , R-FCN ( Dai et al . ( 2016b ) ) , Mask R-CNN ( He et al . ( 2017 ) ) generate a sparse set of candidate proposals that contain all objects while filtering out the majority of negative locations in the first stage , and then classify the proposals into foreground classes or background in the second stage . Single-stage approaches , such as SSD ( Liu et al . ( 2016 ) ) , YOLO ( Redmon & Farhadi ( 2016 ) ) , RetinaNet ( Lin et al . ( 2017b ) ) directly regress to predict the bounding boxes . Detection frameworks with multiple stages like Cascade R-CNN ( Dai et al . ( 2016a ) ) are also popular and bring tremendous improvement for object detection . 2.2 SCALE VARIATION . Scale variation across object instances has been treated as one of the most knotty problem in modern development of detection . To address this challenge , several approaches have been proposed . Image pyramid is an intuitive way , SNIP ( Singh & Davis ( 2018 ) ) and SNIPER ( Singh et al . ( 2018 ) ) select a specific scale for each resolution during multi-scale training . However , these methods suffer a lot from the inevitable increase of inference time . Instead of taking multiple images as input , feature pyramid method uses multi-level features of different layers . FPN ( Lin et al . ( 2017a ) ) is the most famous representative of this strategy . It takes the fused feature map with the highest resolution to pool features and achieves superior performance . SSD ( Liu et al . ( 2016 ) ) , DSSD ( Fu et al . ( 2017 ) ) and MS-CNN ( Cai et al . ( 2016 ) ) perform object detection at multiple layers for objects of different scales . PANet ( Liu et al . ( 2018 ) ) boosts information flow in proposal-base instance segmentation framework , enhances the entire feature hierarchy with accurate localization signals in lower layers by bottom-up path augmentation , which shortens the information path between lower and top layers feature . 2.3 INSTANCE SEGMENTATION . Instance segmentation is a task to predict class label and pixel-wise instance mask in an image . There are mainly two streams of methods in instance segmentation , segmentation based methods and detection based methods . The former are less commonly used nowadays . Segmentation based methods like ( Zhang et al . ( 2016 ) ) first predict the labels of each pixel and then identify object instances therefrom . However the performances of these methods are always unsatisfactory . Detection based methods follow a similar diagram : get the region of each instance and then predict the mask , showing a strong connection to object detection . Instance-FCN ( Dai et al . ( 2016a ) ) proposes instance-sensitive FCNs to generate the position-sensitive to obtain the final masks . FCIS ( Li et al . ( 2017 ) ) takes position-sensitive maps with inside/outside scores to generate the instance segmentation results , but exhibits systematic errors on overlapping instances and creates spurious edges . MaskLab ( Chen et al . ( 2018b ) ) produces instance-aware masks by combining semantic and direction predictions . Cascade Mask R-CNN is a multi-stage object detection and instance segmentation framework derived from Cascade R-CNN ( Cai & Vasconcelos ( 2018 ) ) , which comprises multiple stages where the output of each stage is fed into the next one for higher quality refinement . But this multi-stage mechanism inevitably brings extra computation overhead of inference . 3 FRAMEWORK . In this section , we will describe our proposed new framework for object detection and instance segmentation in details . Overview . Based on Mask R-CNN , as shown in Figure 1 , it is distinctive in two aspects : ( 1 ) It aims to remedy the problem of scale variation by fusing Top-Down style FPN and Bottom-Up style FPN into a novel neck module called SA-FPN . ( 2 ) It explores the interrelations between object detection and instance segmentation , eliminating the gap between training and testing processes and enhancing both two tasks . Overall , these changes to the framework architecture effectively improve two tasks and also provides a reference for further research on multi-task learning . 3.1 SCALE AWARE FEATURE PYRAMID NETWORKS . Scale variation across object instances is one of the most challenging problems in object detection , especially for very small or huge objects . Simply using multi-scale image pyramids has some improvement in accuracy but suffers a lot from increasing inference time . Feature pyramid is popular to deal with this knotty problem , and Top-Down style FPN in ( Lin et al . ( 2017a ) ) is the most widely used feature pyramid structure . As illustrated in Figure 2 ( a ) , a general FPN is a Top-Down style structure that feeds the last output of the backbone into the top of feature pyramid , and the information flow direction is from shallow to deep in the backbone and from the top to bottom in feature pyramid . Up-sampling operation and lateral connection fusion are applied step by step to get a larger feature until the bottom of the pyramid . Thus , deep feature map with high semantic information and shallow feature map with high resolution are combined . This greatly boosts the performance on small objects detection and therefore improves the overall detection performance . However , the top-down pathway of general FPN only introduces high-level semantic information to low-level feature , but ignore the subsidiary role of low-level feature like edges and textures which also are important for accurately localizing instances . Most of the improvements caused by TopDown FPN come from more accurate detection on small objects . Shallow features with high resolution contain useful low-level features . Different layers are of different resolutions , the fusion operation between neighbor shallower or deeper feature is also important . Following the discussion above , we design two new Bottom-Up style FPNs , as illustrated in Figure 2 ( b ) BU-FPN and ( c ) BUF-FPN . Experiments in Table 3 show that ( c ) BUF-FPN performs better . The main difference between ( b ) and ( c ) is the feature integration operation , Details are shown in purple dotted boxes in ( b ) and in orange dotted boxes in ( c ) . In ( b ) each layer of feature pyramid combines with only neighboring shallower layer , while in ( c ) both neighboring shallower and deeper layers . So we use ( c ) BUF-FPN as the default Bottom-Up style FPN for further experiments . BU-FPN takes every level output of the backbone as the input of the next step and applies downsampling and fusion operation step by step to build a feature pyramid from bottom to top . Letting low-level information guide high-level semantic information , BU-FPN is especially better at accurate detection of large objects . When we combine TD-FPN ( Top-Down FPN ) and BUF-FPN ( Bottom-Up-Fusion FPN ) together , a scale sensitive neck module called SA-FPN ( Scale Aware FPN ) is proposed . Experiments in Table 3 show that concatenating performs better than element-wise addition . Then we add 1x1 convolutions to reduce dimensions . In line with expectations , SA-FPN absorbs the advantages of two style FPNs and improves both detection and instance segmentation in different object scales . 3.2 EFFECTIVE JOINT HEAD .. For multi-task learning , parallel execution of tasks is a common practice . However , instance segmentation in Mask R-CNN is such a detection based method that relies heavily on detection . Thus , the interaction between these two tasks is worth further exploring . Firstly , we simply connect the segmentation branch behind the detection head in series when training , feeding the bounding box predictions of the detection into the mask head . As shown in Figure 1 ( b ) , green dotted boxes “ Interleaving ” represents this execution . “ Interleaving ” is an operation to filter predicted bounding box which have IoU ( Intersection over Union ) with ground truth boxes of at least 0.5 and feed positive samples of mask into instance segmentation branch . In this way , the segmentation branch can take advantage of the detection process , realizing the interaction between two tasks . Moreover , the training and testing pipelines are highly consistent . And experiments in Table 4 prove this operation indeed improved performance . Merging heavy mask branch into box branch may slightly increase the burden of box branch . We enlarge the extracted RoI features from 7 x 7 to 14 x 14 , and name it “ Enriched Feature ” as shown in blue dotted boxes in Figure 1 ( b ) . The quality of RoI features is enhanced by the enlarged receptive field . Richer features further help enhance both the detection and instance segmentation . Experiments in Table 4 show the effectiveness of our operation . The whole flow of information goes through an unified path . Despite increased resolution , we can add additional convolutions on this pathway to further boost localization especially for details and edges of objects . As shown in Figure 3 the yellow dotted boxes “ Boundary Refinement ” means this execution . Boundary Refinement module is a residual structure for boundary alignment . Experiments in Table 5 show that “ Boundary Refinement ” is more superior than directly two stacked 3 x 3 simple convolutions together .

Summary: This paper works on the problem of improving object detection and instance segmentation. It is realized by two independent contributions: 1). adding a high-to-low-resolution connection in FPN and 2) adding more connecting layers between the mask and classification heads. Experiments show both contributions give a small improvement (~1AP) on both detection and instance segmentation task.