Trident Pyramid Networks: The importance of processing at the feature pyramid level for better object detection

1 INTRODUCTION . Many computer vision tasks such as object detection and instance segmentation require strong features both at low and high resolution to detect both large and small objects respectively . This is in contrast to the image classification task where low resolution features are sufficient as usually only a single object is present in the center of the image . Networks developed specifically for the image classification task ( e.g . Simonyan & Zisserman ( 2014 ) ; He et al . ( 2016a ) ; Xie et al . ( 2017 ) ) , further denoted by backbones , are therefore insufficient for multi-scale vision tasks . Especially poor performance is to be expected on small objects , as shown in Lin et al . ( 2017a ) . In order to alleviate this problem , named the feature fusion problem , top-down mechanisms are added ( Lin et al. , 2017a ) to propagate semantically strong information from the low resolution to the high resolution feature maps , with improved performance on small objects as a result . Additionally , bottom-up mechanisms can also be appended ( Liu et al. , 2018 ) such that the lower resolution maps can benefit from the freshly updated higher resolution maps . These top-down and bottom-up mechanisms can now be grouped into a layer , after which multiple of these layers can be concatenated , as done in Tan et al . ( 2020 ) . We call this part of a computer vision network the core , laying in between the backbone and the task-specific head ( see Figure 1 ) . In general , we define a core module to be any module taking as input a feature pyramid and outputting an updated feature pyramid . These top-down and bottom-up operations can be regarded as communication-based processing operating on two feature maps , as opposed to content-based self-processing operating on a single feature map . Existing cores such as FPN ( Lin et al. , 2017a ) , PANet ( Liu et al. , 2018 ) and BiFPN ( Tan et al. , 2020 ) mostly focus on communication-based processing , as this nicely supplements the backbone merely consisting of self-processing . However , when having multiple communication-based operations in a row , communication tends to saturate ( everyone is up to date ) and hence becomes superfluous . We argue it is therefore more effective to alternate communication-based processing with sufficient self-processing , such that feature maps have the time to come up with new findings to be communicated . Based on this observation , we design the Trident Pyramid Network ( TPN ) core consisting of sequential top-down and bottom-up operations alternated with parallel self-processing mechanisms . The TPN core is equipped with hyperparameters controlling the amount of communication-based processing and self-processing . During the experiments , we empirically investigate what the optimal balance is between communication-based processing and self-processing ( see Subsection 4.3 ) . The TPN core is compared to various baselines on the COCO object detection benchmark ( Lin et al. , 2014 ) . Specific care is taken to ensure the baselines have similar computational characteristics , such that a fair comparison can be made . Using a ResNet-50 backbone and a simple one-stage detector head , our TPN core peaks at 41.8 AP on the COCO validation set when using the 3x training schedule ( see Subsection 4.2 ) . This is a 1.5 AP improvement over a BiFPN core of similar computational expense . When having additional compute to improve performance , practitioners typically decide to replace their backbone with a heavier one . A ResNet-50+FPN network for example gets traded for the heavier ResNet-101+FPN network . Yet , one might wonder whether it is not more beneficial to add additional computation into the core ( i.e . at the feature pyramid level ) by using a ResNet-50+TPN network , rather than into the backbone by using a ResNet-101+FPN network . When comparing both options under similar computational characteristics , we show a 1.7 AP improvement of the ResNet-50+TPN network over the ResNet-101+FPN network . This empirically shows that it is more beneficial to add additional computation into the core , highlighting the importance of performing computation at the feature pyramid level in modern-day object detection systems . We hope this new insight drives researchers to design even better cores in the future . 2 RELATED WORK . In order to obtain multi-scale features , early detectors performed predictions on feature maps directly coming from the backbone , such as MS-CNN ( Cai et al. , 2016 ) and SSD ( Liu et al. , 2016 ) . As the higher resolution maps from the backbone contain relatively weak semantic information , top-down mechanisms were added to propagate semantically strong information from lower resolution maps back to the higher resolution maps as in FPN ( Lin et al. , 2017a ) and TDM ( Shrivastava et al. , 2016 ) . Since , many variants and additions have been proposed : PANet ( Liu et al. , 2018 ) appends bottom-up connections , M2det ( Zhao et al. , 2019 ) uses a U-shape feature interaction architecture , ZigZagNet ( Lin et al. , 2019 ) adds additional pathways between different levels of the top-down and bottom-up hierarchies , NAS-FPN ( Ghiasi et al. , 2019 ) and Hit-Detector ( Guo et al. , 2020 ) use Neural Architecture Search ( NAS ) to automatically design a feature interaction topology , and BiFPN ( Tan et al. , 2020 ) modifies PANet by removing some connections , adding skip connections and using weighted feature map aggregation . All of the above variants focus on improving the communication between the different feature maps . We argue however that to be effective , extra content-based self-processing is needed in between the communication flow . Not all methods use a feature pyramid to deal with scale variation . TridentNet ( Li et al. , 2019 ) applies parallel branches of convolutional blocks with different dilations on a single feature map to obtain scale-aware features . In DetectoRS ( Qiao et al. , 2021 ) , they combine this idea with feature pyramids , by applying their switchable atrous convolutions ( SAC ) inside their recursive feature pyramids ( RFP ) . Note that to avoid any name confusion with TridentNet , we call our core by its abbreviated name TPN as opposed to Trident Pyramid Network . Our TPN core is also related to networks typically used in segmentation such as U-Net ( Ronneberger et al. , 2015 ) and stacked hourglass networks ( Newell et al. , 2016 ) , given that these networks also use a combination of top-down , self-processing and bottom-up operations . A major difference of these networks with our TPN core however , is that they do not operate on a feature pyramid in the sense that lower resolution maps are only generated and used within a single layer ( e.g . within a single hourglass ) and are not shared across layers ( e.g . across two neighboring hourglasses ) . Finally , note that some works such as Guo et al . ( 2020 ) and Bochkovskiy et al . ( 2020 ) refer to the the network part connecting the backbone with the head as the neck ( instead of the core ) . That name implies that the neck is merely a connection piece between the backbone and the head , and is of little importance . Yet , we show that the neck is in fact an important part of the network , and therefore call it the core instead . 3 METHOD . 3.1 TPN CORE ARCHITECTURE . Generally speaking , the core receives a feature pyramid as input , and outputs an updated feature pyramid . Here , a feature pyramid is defined as a collection of feature maps , with feature maps defined as a collection of feature vectors ( called features ) organized in a two-dimensional map . More specifically , feature map Pl denotes a feature map of level l which is 2l times smaller in width and height compared to the initial image resolution . A popular choice for the feature pyramid ( Lin et al. , 2017b ) is to consider feature maps { P3 , P4 , P5 , P6 , P7 } , which we will use as the default setting throughout our discussions and experiments . The core is constructed from three building blocks : top-down operations , self-processing operations and bottom-up operations ( see Figure 2 ) . In this subsection , we focus on how these operations are best combined , independently of their precise implementations . We call this configuration of operations making up a core , the core architecture . The specific implementations corresponding to the top-down , self-processing and bottom-up operations will be discussed in Subsection 3.2 and Subsection 3.3 . Using these general building blocks , we can recreate the popular FPN ( Lin et al. , 2017a ) and PANet ( Liu et al. , 2018 ) core architectures in Figure 3 . Note that the architectures slightly differ from those found in the original works ( Lin et al. , 2017a ; Liu et al. , 2018 ) , as the input layers are missing . Given that these input layers are meant to transition from backbone feature sizes to core feature sizes , we decided to move these transition layers from the core to the backbone instead , such that multiple core layers can easily be concatenated . Note moreover that Figure 3 only defines the architecture of the core , without specifying the implementation of the top-down and bottom-up operations . These implementations could hence differ from those found in the original works ( Lin et al. , 2017a ; Liu et al. , 2018 ) . From the FPN and PANet core architectures from Figure 3 , we make following two observations . First , we can see that the top-down and bottom-up operations are sequential . Secondly , we observe the lack of self-processing operations in both core architectures . In what follows , we discuss both aspects in more depth . First , we discuss the trade-off between sequential and parallel operations in greater detail . By sequential operations , we mean that Pl is updated with the new Pl±1 instead of with the old one , forcing the operations to be performed sequentially as the new feature maps must be available . Alternatively , one could instead opt for parallel operations by solely relying on the old feature maps . The choice between parallel and sequential could be regarded as a trade-off between speed and accuracy , while maintaining a similar memory consumption . Given that the top-down and bottom-up operations can be quite expensive , especially on high resolution maps , it is important to get the most out of every single operation . We hence believe that the sequential variant should be preferred here for the top-down and bottom-up operations , as found in the FPN and PANet core architectures . Secondly , we discuss the lack of self-processing operations in the FPN and PANet core architectures . When looking at the PANet architecture , we see that bottom-up operations immediately follow the top-down operations . We argue that this is sub-optimal . Take for example a look at the P4-P3 topdown operation , followed immediately by the P3-P4 bottom-up operation . The P3 map was just updated with information from P4 and now P3 must immediately communicate its content back to P4 before having the possibility to digest and work on this new information . We hence argue that the top-down and bottom-up operations should be separated with self-processing operations . This gives the feature maps the opportunity to work on themselves before communicating back to their peers . By combining the insights from previous two discussions , we arrive at the Trident Pyramid Network ( TPN ) core architecture consisting of sequential top-down and bottom-up operations alternated with self-processing operations ( see lower part of Figure 4 ) . The name is inspired by the top-down , first self-processing and bottom-up operations resembling a trident . Note that the TPN self-processing operations happen in parallel . This is not necessary , but we believe this to be the most natural choice .

Summary: Typically, object detection CNNs consist of a backbone for feature extraction, a core for feature refinement, and a head for predictions. The core is usually a simple arrangement of top-down and/or bottom-up convolutional blocks, used for effectively combining feature maps at different resolutions and this kind of processing is referred to as communication-based processing. This paper claims that the detection performance can be enhanced by the use of a more complex core that enables additional self-processing of individual feature maps (by running them through a series of convolutional layers)  before combining them with feature maps of different resolutions. The paper proposes a new core architecture in object detection CNNs and names it the Trident Pyramid Network (TPN).  In addition, the paper claims that given an additional computational budget, it is better to invest it in a heavier core rather than a heavier backbone, contrary to the mainstream approach. The proposed TPN has two tunable parameters B and L, that control the amount of self processing (B) and the amount of communication-based processing (L). Experimental studies on the COCO object detection benchmark show that the proposed approach with ResNet-50 backbone and TPN core outperforms existing methods with ResNet-50/101 backbone with FPN or PANet as their cores, under similar computational cost.