Multi-Task Learning via Scale Aware Feature Pyramid Networks and Effective Joint Head

Abstract: As a concise and classic framework for object detection and instance segmentation, Mask R-CNN achieves promising performance in both two tasks. However, considering stronger feature representation for Mask R-CNN fashion framework, there is room for improvement from two aspects. On the one hand, performing multi-task prediction needs more credible feature extraction and multi-scale features integration to handle objects with varied scales. In this paper, we address this problem by using a novel neck module called SA-FPN (Scale Aware Feature Pyramid Networks). With the enhanced feature representations, our model can accurately detect and segment the objects of multiple scales. On the other hand, in Mask R-CNN framework, isolation between parallel detection branch and instance segmentation branch exists, causing the gap between training and testing processes. To narrow this gap, we propose a unified head module named EJ-Head (Effective Joint Head) to combine two branches into one head, not only realizing the interaction between two tasks, but also enhancing the effectiveness of multi-task learning. Comprehensive experiments show that our proposed methods bring noticeable gains for object detection and instance segmentation. In particular, our model outperforms the original Mask R-CNN by 1~2 percent AP in both object detection and instance segmentation task on MS-COCO benchmark. Code will be available soon.

Reviews and author responses:

I apologize for a mistake in the penultimate paragraph of my above review. Of course, p(b|features)p(m|b,features) = p(m, b | features) and thus it *is* joint prediction.

I am sorry for this very basic mistake!

This paper works on the problem of improving object detection and instance segmentation. It is realized by two independent contributions: 1). adding a high-to-low-resolution connection in FPN and 2) adding more connecting layers between the mask and classification heads. Experiments show both contributions give a small improvement (~1AP) on both detection and instance segmentation task.

Overall, the method seems reasonable and the improvements are healthy. The main concern is the technical novelty. It is not supervising that adding more connections inside the network can improve some performance. People have tried a lot of that (e.g. M2Det). These kinds of improvements come with a cost of slowdown and are usually not that appealing in practice.

The motivation for leveraging the relationship between segmentation and detection as a joint model is interesting and relatively new. However, the proposed method of feeding back the regressed bbox for segmentation seems straightforward and far from the full potential. It also requires a larger RoI feature map, which makes the contribution less clear. A fancier method or a more thorough analysis of how the information is shared between the two tasks is demanded for an ICLR publication.

The paper proposes several modifications to the Mask R-CNN model:
1. a variation of Feature Pyramid Networks (FPN): SA-FPN which merges features top-down and bottom-up.
2. the "effective joint head" (EJ-Head) which consists of
    a) moving the segmentation head behind the detection one,
    b) Doubling resolution of RoI crops, calling this "enriched feature".
    c) adding Boundary Refinement (really just a residual block).
Experiments show slightly improved scores on MS-COCO


I vote to reject this paper.

First of all, it is badly written. Not only are grandiose formulations of "smart framework", "innovative way", "we slickly mix" not professional, but the whole write-up of the method is very confusing. After reading it multiple times, I am still not 100% sure I fully understood the SA-FPN.

Second, it is not clear to me that SA-FPN is really novel, the original FPN paper already compared top-down and bottom-up performances (Tables 1-3).

Then, two modifications which simply increase capacity and could explain all improved scores are not ablated: increased resolution of RoI crops, and added "boundary refinement", which is really just a residual block.

Furthermore, moving the segmentation *after* the bounding-box prediction is *not* joint prediction. If predicting the box is called p(b) and predicting the mask is called p(m), the Original Mask R-CNN does p(b|features)p(m|features), the proposed model in this paper does p(b|features)p(m|b,features), and actual joint prediction would be p(b,m|features).

Finally, I think the paper is much better suited for a conference like ICCV/ECCV or CVPR and will get better reviewers than me there.

Meta-review: All three reviewers gave scores of Weak Reject. Only a brief rebuttal was offered, which did not change the scores. Thus the paper connect be accepted. 

Decision: reject