Trident Pyramid Networks: The importance of processing at the feature pyramid level for better object detection
Authors: Cédric Picron
Tinne Tuytelaars
Venue: ICLR 2022 Submitted (2022)
Keywords: feature pyramid
network architecture
object detection
deep learning

Abstract: Feature pyramids have become ubiquitous in multi-scale computer vision tasks such as object detection. Based on their importance, we divide a computer vision network into three parts: a backbone (generating a feature pyramid), a core (refining the feature pyramid) and a head (generating the final output). Most existing networks operating on feature pyramids, named cores, are shallow and mostly focus on communication-based processing in the form of top-down and bottom-up operations. We present a new core architecture called Trident Pyramid Network (TPN), that allows for a deeper design and for a better balance between communication-based processing and self-processing. We show consistent improvements when using our TPN core on the COCO object detection benchmark, outperforming the popular BiFPN baseline by 1.5 AP. Additionally, we empirically show that it is more beneficial to put additional computation into the TPN core, rather than into the backbone, by outperforming a ResNet-101+FPN baseline with our ResNet-50+TPN network by 1.7 AP, while operating under similar computation budgets. This emphasizes the importance of performing computation at the feature pyramid level in modern-day object detection systems. Code will be released.

Under review as a conference paper at ICLR 2022
TRIDENT PYRAMID NETWORKS: THE IMPORTANCE OF PROCESSING AT THE FEATURE PYRAMID LEVEL FOR BETTER OBJECT DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Feature pyramids have become ubiquitous in multi-scale computer vision tasks such as object detection. Based on their importance, we divide a computer vi- sion network into three parts: backbone (generating a feature pyramid), core (refining the feature pyramid) and head (generating the final output). Most exist- ing networks operating on feature pyramids, named cores, are shallow and mostly focus communication--asee processing in the form of top-down and bottom- up operations. We present a new core architecture called Trident Pyramid Net- work (TPN), that allows for deeper design and for better balance between communication-basee processing and self-processing. We show consistent im- provements when using our TPN core the COCO object detection benchmark, outperforming the popular BiFPN baseline by 1.5 AP. Additionally, we empiri- cally show that it is more beneficial to put additional computation into the TPN core, rather than into the backbone, by outperforming a ResNet-101+FPP 101+FPN base- line with our ResNet-50+TPP network by 1.7 AP, while operating under similar computation budgets. This emphasizes the importance of performing computation at the feature pyramid level in modern-day object detection systems. Code will be released.
INTRODUCTION
Many computer vision tasks such as object detection and instance segmentation require strong fea- tures both at low and high resolution to detect both large and small objects respectively. This is in contrast to the image classification task where low resolution features are sufficient as usually only single object is present in the center of the image. Networks developed specifically for the image classification task (e.g. (agoonnny & Zisserman (2014); He et al. 62), Xie et al. (2017). ). further denoted by backbones, are therefore insufficient for multi-scale vision tasks. Especially poor performance is to be expected on small objects, as shown in Lin et al. (2017a).
In order alleviate this problem, named the feature fusion problem, top-down mechanisms added (Lin et af.) 2017a) to propagate semantically strong information from the low resolution to the high resolution feature maps, with improved performance on small objects result. Addition- ally. bottom-up mechanisms can also be appended (Liu et al. 20188) such that the lower resolution maps can benefit from the freshly updated higher resolution maps. These top-down and bottom-up mechanisms can now be grouped into layer, after which multiple of these layers can be concate- nated, as done in Tan al. (2020). We call this part of a computer vision network the core, laying in between the backbone and the task- task-specific head (see Figure D.. In general, we define a core module to be any module taking as input a feature pyramid and outputting an updated feature pyramid.
These top-down and bottom-up operations can be regarded as communication--on based processing op- erating on two feature maps, as opposed to content- based self-processing operating on a single fea- ture map. Existing cores such as FPN (Lin et al. 2017a), PANet (Liu et al. 2018) and BiFPN (Tan) et al. 2020) mostly focus on communication based processing, as this nicely supplements the back- bone merely consisting of self- processccss.ng. However, when having multiple communication-bann based operations in a row, communication tends to saturate (everyone is up to date) and hence becomes superfluous. We argue it is therefore more effective to alternate communication-basee processing

---

Under review as a conference paper at ICLR 2022
FP
FP
image
backbone
core
head
loss/ pres/pred
Figure 1: High-level view of a computer vision network. The backbone (left) processes the image to output set of feature maps (i.e. feature pyramid). The core (middle) takes in feature pyramid (denoted by FP) and returns an updated feature pyramid. Finally, the head (right) produces the loss during training and makes predictions during inference from the final feature pyramid. In this work, we focus on improving the core.
with sufficient pelf-processing, such that feature maps have the time to come up with new findings to be communicated.
Based on this observation, we design the Trident Pyramid Network (TPN) core consisting of sequen- tial top-down and bottom-up operations alternated with parallel self-processing mechanisms. The TPN core is equipped with hyperparameters controlling the amount of communication-basee pro- cessing and self proceesiccss During the experiments, we empirically investigate what the optimal balance is between communication baaeu processing and self- self-prroessing (see Subsection 4.3
The TPN core is compared various baselines the COCO object detection benchmark (Lin et al. 2014). Specific care is taken to ensure the baselines have similar computational characteristics, such that fair comparison can be made. Using ResNet-50 backbone and simple one-stage detector head, our TPN core peaks at 41.8 AP on the COCO validation set when using the 3x training schedule (see Subsection 4.2) This is 1.5 AP improvement over BiFPN core of similar computational expense.
When having additional compute improve performance, practitioners typically decide to replace their backbone with heavier one. A ResNet-50+FPH network for example gets traded for the heavier ResNet-101+FPN network. Yet, one might wonder whether it not more beneficial add additional computation into the core (i.e. at the feature pyramid level) by using a ResNet-50+TPN network, rather than into the backbone by using both options under similar computational characteristics, ResNet-50+TPN network over the ResNet- MeaNeet-111+H +PNN network. This empirically shows that is more beneficial to add additional computation into the core, highlighting the importance of performing computation at the feature pyramid level in modern-day object detection systems. We hope this new insight drives researchers to design even better cores in the future.
DesNet-101+FPM network. When comparing
show a 1.7 AP improvement of the
2 RELATED WORK
In order obtain multi-scale features, early detectors performed predictions feature maps directly coming from the backbone, such as MS-CNN (Cai et al., 2016) and SSD (Liu al. 2016). As the higher resolution maps from the backbone contain relatively weak semantic information. top-down mechanisms were added to propagate semantically strong information from lower resolution maps back to the higher resolution maps in FPN (Lin al., 2017a) and TDM (Shrivastava et al., 2016). Since, many variants and additions have been proposed: PANet (Liu et al.) 2018) appends bottom-up connections, M2det (Zhao al., 2019) uses U-shape feature interaction architecture, ZigZagNet (Lin et al. (2019) adds additional pathways between different levels of the top-down and bottom-up hierarchies. NAS-FPN (Ghiasi al. 2019) and Hit-Detector (Guo et al. 2020) use Neural Architecture Search (NAS) to automatically design a feature interaction topology. and BiFPN (Tan al. 2020) modifies PANet by removing some connections, adding skip connections and using weighted feature map aggregation. All of the above variants focus on improving the communication between the different feature maps. We argue however that to be effective, extra content-based self-processing is needed in between the communication flow.
Not all methods use a feature pyramid to deal with scale variation. TridentNet (Li et al. 2019) applies parallel branches of convolutional blocks with different dilations single feature map to obtain scale-aware features. In DetectoRS (Qiao et al. 2021). they combine this idea with feature pyramids, by applying their switchable atrous convolutions (SAC) inside their recursive feature pyramids (RFP). Note that to avoid any name confusion with TridentNet, we call our core by its abbreviated name TPN opposed to Trident Pyramid Network.
2

---

Under review as a conference paper at ICLR 2022
Pr+1
Bottom-up
P
Self
P
Pr
Top-down
Pt
Pi-1
Figure 2: Collection of building blocks for core architecture design. Here Pt denotes feature map of level 1 which is 2' times smaller compared to the initial image resolution. (Left) General top-down operation updating feature map P. with information from lower resolution map Pi+1- (Middle) General self proc-sscsssing operation updating feature map Pt with information from itself, i.e. from feature map Pt. (Right) General bottom-up operation updating feature map Pt with information from higher resolution map P:-1-
P4
Py
Pb
P
F/
Pb
P3
P3
P3
P3
P4
P4
Pa
P.
Ps P.
Ps
Figure 3: (Leff) A layer from the FPN core architecture. (Right) A layer from the PANet core architecture.
Our TPN core is also related networks typically used in segmentation such as U-Net (Ronneberger et al. 2015) and stacked hourglass networks (Newell et al. 2016), given that these networks also use combination of top-down, self- poff-processin and bottom-up operations. A major difference of these networks with our TPN core however, is that they do not operate on a feature pyramid in the sense that lower resolution maps are only generated and used within a single layer (e.g. within a single hourglass) and are not shared across layers (e.g. across two neighboring hourglasses).
Finally, note that some works such as Guo et al. (2020) and /Bockooskiy et al. (2020) refer to the the network part connecting the backbone with the head as the neck (instead of the core). That name implies that the neck merely a connection piece between the backbone and the head, and little
importance. Yet, we show that the neck is in fact important part of the network, and therefore call it the core instead.
METHOD
3.1 TPN CORE ARCHITECTURE
Generally speaking, the receives feature pyramid input, and outputs updated feature pyramid. Here, feature pyramid is defined collection of fcature maps, with feature maps defined a collection of feature vectors (called features) organized in two-dimensional map. More specifically, feature map P. denotes feature map of level / which is 2' times smaller in width and height compared to the initial image resolution. A popular choice for the feature pyramid (Lin et al., (01 is to consider feature maps {P3, Pa, Ps. Pa. Pr), which we will use as the default setting throughout our discussions and experiments.
The core is constructed from three building blocks: top-down operations, self- celf-processing operations and bottom-up operations (see Figure D this subsection, we focus on how these operations best combined, independently of their precise implementations. We call this configuration of operations making up a core, the core architecture. The specific implementations corresponding to the top-down, seof-procsssing and bottom-up operations will be discussed in Subsection 3.2 and Subsection 3.3
Using these general building blocks,
recreate the popular FPN LLn et al. 2017a) and PANet (Liu et al. 2018) core architectures Figure Note that the architectures slightly differ from those
found in the original works (Lin et al. 2017a Liu et al. 2018), as the input layers missing. Given
3

---

Under review as a conference paper at ICLR 2022
xB
P
Pt
P7
P7
Po
Po
P3
m
Ps
P4
P4
P3
Ps
L
Figure 4: Our TPN core architecture consisting of L consecutive TPN core layers (bottom), with each seoo-procsssing operation consisting of B consecutive bottleneck layers (top).
that these input layers meant transition from backbone feature sizes core feature sizes,
decided to move these transition layers from the core to the backbone instead, such that multiple core layers can easily be concatenated. Note moreover that Figure 3 Jonly defines the architecture of
the core, without specifying the implementation of the top-down and bottom-up operations. These implementations could hence differ from those found in the original works (Lin et al. 201 Liu et al. 2018).
From the FPN and PANet core architectures from Figure we make following two observations. First, we can see that the top-down and bottom-up operations are sequential. Secondly, we observe the lack of self-processing operations both core architectures. In what follows, we discuss both aspects in more depth.
First, we discuss the trade-off between sequential and parallel operations in greater detail. By se- quential operations,
mean that Pt is updated with the new P instead of with the old one, forcing the operations to be performed sequentially as the new feature maps must be available. Al- ternatively, one could instead opt for parallel operations by solely relying the old feature maps. The choice between parallel and sequential could be regarded as trade-off between speed and ac- curacy, while maintaining similar memory consumption. Given that the top-down and bottom-up operations can be quite expensive, especially on high resolution maps, it is important to get the most of every single operation. We hence believe that the sequential variant should be preferred here for the top-down and bottom-up operations, as found in the FPN and PANet core architectures.
Secondly, discuss the lack of self-processing operations the FPN and PANet core architectures. When looking at the PANet architecture, we see that bottom-up operations immediately follow the top-down operations. We argue that this sub-optimal. Take for example a look the P1-Ps top- down operation, followed immediately by the P3-Pa bottom-up operation. The P3 map was just updated with information from P4 and now P3 must immediately communicate its content back to P1 before having the possibility digest and work this new information. We hence argue that the top-down and bottom-up operations should be separated with self colf-processing operations. This gives the feature maps the opportunity to work on themselves before communicating back to their peers.
By combining the insights from previous two discussions, arrive at the Trident Pyramid Network (TPN) core architecture consisting of sequential top-down and bottom-up operations alternated with self-processing operations (see lower part of Figure 4) The name is inspired by the top-down, first

---

Under review as a conference paper at ICLR 2022
Pi
Pt
Figure 5: Bottleneck layer used base self- processing operation. It is skip-connection operation with a residual branch consisting of three convolution operations: convolution operation of kernel size 1 reducing the original feature size to the hidden feature size, a content convolution operation of kernel size 3 applied on the hidden feature size, and finally a convolution operation of kernel size 1 expanding the hidden feature size back to the original feature size. Note that each convolution operation (i.e. pink convolution node) consists of the actual convolution preceded (He ct al. 2016b) by group normalization (Wu & He 2018) and a ReLU activation function.
self-processing and bottom-up operations resembling trident. Note that the TPN self- seff-processing operations happen in parallel. This not necessary, but we believe this to be the most natural choice.
3.2 COMMUNICATION VS. CONTENT
When looking at the TPN core in Figure 4 we have on the one hand communication--asee top-down and bottom-up operations, and on the other hand content- content-based self- celf-processing operations. One might now wonder what the optimal balance between communication and content is. In order to investigate this matter, we introduce hyperparameters enabling us to control the amount of content- based processing on the one hand, and the amount of communication-basen processing on the other hand.
First, we take a closer look at the eeffprocessing operation. In general, we consider this self- processing operation to be a sequence of layers, with each layer containing the base self-processing operation. In this paper, chose the bottleneck architecture from He et al. (2016a) (see Figure as the base self- self-processing operation. The number of bottleneck layers B per self- pelfeesrocssingg operation then determines the amount of content- based processing within TPN layer.
Secondly, with each TPN layer consisting of one top-down and one bottom-up up sequence, the number of TPN core layers L determines the amount of Communication--roceee
within the TPN core. Note that while the total amount of communication -based processing only depends on the number of TPN layers T., the total amount of self self-processing within TPN depends both the number of bottleneck layers per self-processing operation B, as well as on the total number of TPN layers I.. In Figure overview of our TPN core architecture shown, displaying the number of bottleneck layers B and the number of TPN core layers L. By varying these hyperparameters B and L, we can hence control the balance between content-basee processing and communication-bun based processing respectively. In Subsection 4.3 we empirically find out which combinations work best.
3.3 TOP-DOWN AND BOTTOM-UP OPERATIONS
Let us now take closer look the top-down and bottom-up operations. Generally speaking, these operations update a feature map based on a second feature map, either having a lower resolution (top-down case) higher resolution (bottom-up case). Our implementation of the top-down and bottom-up operations are shown in Figure 6 The operations consist of adding a modified version of P1+1 to Pt. This is similar traditional skip-connection operations, with the exception that the residual features originate from different feature map. The residual branch of the top-down operation consists of a linear projection followed by bilinear interpolation. The presence the línear projection is important here, as it makes the expectation of the residual features zero at initialization. Failing to do so can be detrimental, especially when building deeper core modules, as correlated features add up without constraints. An alternative consists in replacing the blue addition nodes with averaging nodes. This however fails to keep the skip connection computation free (due to the 0.5 factor), which is undesired (He et al. 2016b). The residual branch of the bottom-up operation is similar to the bottleneck residual branch in Figure [5] Only the middle 3 convolution has stride 2 instead of stride 1, avoiding the need for interpolation step later in the residual branch.
5

---

Under review as a conference paper at ICLR 2022
P
P
P141
P
Pt
Pi-1
Figure 6: Implementation of the top-down (left) and bottom-up (right) operations. The pink convolu- tion nodes are defined as in Figure with the subscript denoting the kernel size and the superscript denoting the stride (stride 1 when omitted). The green node is interpolation node resizing the input feature map to the required resolution by using bilinear interpolation.
EXPERIMENTS
4.1 SETUP
Dataset. We perform our experiments the 2017 COCO dataset (Lin 2014). where train on the 118k training images and evaluate on the remaining 5k validation images.
Implementation details. Throughout our experiments, pretrained ResNet-50 (or ResNet-101) backbone (He et al. 2016a), with frozen stem, stage and batchnorm layers (see Radosavovic et al. (2020) for used terminology).
an ImageNet (Deng et al. 2009)
Our feature pyramid consists of five feature maps, ranging from P3 to Pr. each having feature size 256. Our initial feature pyramid is constructed based on the backbone output feature maps C3 to Cs from stages 2, and respectively. Remember that the subscript denotes how many times the feature map was downsampled with factor 2 compared to the input image. The initial P3 P3 maps are obtained by applying simple linear projections on C3 to C5. whereas the initial Pg and Py maps obtained by applying a simple network on C consisting of convolutions with stride 2, with ReLU activation in between (similar to Lin et al. (2017b)). Throughout our TPN core modules, we use group normalization (Wu He 2018) with 8 groups. For the bottleneck layers (see Figure 5 we use a hidden feature size of 64.
As detection head, we use the one-stage detector head from RetinaNet (Lin al. 2017b), with 1 or 4 hidden layers in both classification and bounding box subnets. We follow the implementation and settings from Wu et al. (2019), except that the last layer of the subnets has kernel size 1 (instead of 3) and that we normalize the losses per feature map (instead of over the whole feature pyramid).
We train our models with the AdamW optimizer (Loshchilov Hutter) 2017) with weight decay 10-4 using an initial learning rate of 10-3 for the backbone parameters and an initial learning rate of 10-4 for the remaining model parameters. Our main experiment results Subsection 4.2 are obtained by using the 3x training schedule, consisting of 36 epochs with learning rate drops after the 27th and 33rd epoch with a factor 0.1. Our models are trained on 2 GPUs with batch size 2, while using the same data augmentation scheme in Carion et al. (020)_
4.2 MAIN TPN EXPERIMENTS
Baselines. this subsection, we perform experiments to evaluate the TPN core. As baseline, we consider the BiFPN core architecture from Tan et al. (2020). with Swish-1 activation functions placed by ReLU activation functions. Multiple of these BiFPN layers will be concatenated such that the BiFPN core shares similar computational characteristics compared to the tested TPN modules.

---

Under review as a conference paper at ICLR 2022
P7
P
Pq
Po
Pa
Pg
Pg
ID
Ps
P3
TD
Ps
P4
P4
P4
Pa
P3
7D
Ps
Pa
Ps
Figure 7: (Left) The baseline bFPN core architecture simulating a heavier backbone followed by a single FPN layer. (Right) The baseline hFPN core architecture simulating a single FPN layer followed by heavier head.
Table 1: Experiment results the 2017 COCO validation set of different TPN cores (top four rows) and its baselines (bottom five rows). The five leftmost columns specify the network (Back = Back- bone, Core, B Number of bottleneck layers per self-processing node, Number of layers, Number of hidden layers in classification and bounding box subnets), the middle six columns show its performance and the five rightmost columns show its computational characteristics. These char- acteristics were obtained on a GeForce GTX 1660 Ti GPU by applying the network on batch of two 800 x 800 images, each containing 10 ground truth objects during training. The training characteris- ties are found under the columns 'Params', 'tFPS' and 'tMem',, whereas the inference characteristics are found under the 'iFPS' and 'iMem' columns. Here the FPS metric should be interpreted as the number of times the GPU can process above input. Note that both forward and backward passes (with parameter update from the optimizer) are used to obtain the training characteristics.
Back
Core
AP50
APts
APs
AP
AP,
Params
LFPS
tMem
iFPS
iMem
R50
TPN
41.3
60.5
44.2
26.3
45.9
52.5
36.3
1.7
3.31 GB
5.3
0.50 GB
R50
TPN
41.6
60.9
44.6
26.4
45.8
53.2
36.2 M
1.7
3.21 GB
5.5
0.50 GB
R50
TPN
41.8
61.1
444
26.2
16.1
53.7
36.7
1.6
3.27
0.50 GB
R50
TPN
41.8
61.2
45.0
26.0
46.3
53.4
37.1
1.6
3.22 GB
5.3
0.50 GB
R50
BiFPN
40.3
59.8
43.3
21.5
44.1
52.2
31.7 M
1.9
3.28 GB
6.1
0.19 GB
R50
bFPN
14
1
39.6
60.3
42.4
24.2
43.5
51.3
36.1 M
1.7
3.26 GB
5.4
0.49 GB
R50
bFPN
40.0
60.2
43.0
25.6
13.9
51.1
1.7
3.26
0.49 GB
R101
FPN
60.1
44.0
52.7
3.20
0.57 GB
R101
TPN
40.9
61.0
44.2
25.0
45.3
52.6
51.7 M
1.6
3.20 GB
4.6
0.55 GB
As the FPN layer was not designed be concatenated many times, we provide two additional base- lines using single FPN (see Figure(7). Here the bFPN baseline performs additional self- selfeprocessing before the FPN simulating a heavier backbone, while the hFPN baseline performs additional self- processing after the FPN simulating a heavier head. As such, we will not only be able to evaluate whether the TPN core outperforms other cores, also whether outperforms detection networks using simple core with heavier backbones heads, while operating under similar computation budgets.
Finally, we also compare ResNet-101+FPP and ResNet-101+TPN networks with ResNet-50+TPN network of similar computation budget, to further assess whether it is more beneficial to put addi- tional computation into the backbone into the core.
The experiment results evaluating four different TPN configurations against the five base- lines, are found in T
Results.
First, notice how the B and L hyperparameters (defining the TPN configuration) are chosen in order to obtain models with similar computational characteristics. Here hyperparameter B denotes the number of bottleneck layers per self processocssiig node, while hyperparameter L denotes the number of consecutive core layers (see Figure These similar computational characteristics ensure us that fair comparison can be made between the different models.

---

Under review as a conference paper at ICLR 2022
0OCD 200mIIR
COLD oiiss pakiin Stancy
ahhenee Seancy
100
0000
14
Figure 8: Accuracy vs. efficiency comparisons between 15 different (I,, B) TPN configurations using the "parameters' (left), 'training latency' (middle) and 'inference latency' (right) efficiency metries. The accuracies correspond to the COCO validation APs, obtained after training the mod- els for 12 epochs using the Ix schedule. The TPN configurations yielding the best accuracy vs. efficiency trade-off at various computation budgets, are highlighted in magenta.
Secondly, we observe that the results between the four different TPN configurations are very similar, all four obtaining between 41.3 and 41.8 AP. At first glance, it appears that having more TPN core layers L is slightly more beneficial than having more bottleneck layers B under similar computa- tional budgets. In Subsection 4.3 we will further investigate which TPN configurations yield the best accuracy efficiency trade-off at various computation budgets.
Thirdly, when comparing our TPN cores (top four rows) with the BiFPN core (fifth row), we clearly the superiority of the TPN core. All four TPN configurations outperform the BiFPN core with 1.0 up to 1.5 AP, where improvements are found across all object sizes. Note that the BiFPN baseline has slightly fewer parameters and is slightly faster compared the used TPN cores. However, when using additional BiFPN layers to better align the BiFPN computation budget with the TPN budgets, we did not obtain any performance improvements. We therefore chose report the results of slightly lighter BiFPN core instead. In Appendix A we provide additional results where we compare the TPN core with the BiFPN core when using a large backbone.
Fourthly, when comparing the ResNet-50+TPP networks (top four rows) with the ResNet-50+bbPN, Resheet-O+hhFN,, -eesNetttOO+FPy and see that the ResNet-50+TPP networks work best. The best-performing baseline ReesNet-101+TPM from this category outperformed by all four ResNet-50+TPN configurations with 0.4 up to 0.9 AP. Note that the 1OP+ baseline has considerably more parameters and is clearly slower at inference, but still does not match the performance higher computational cost. This hence shows that the TPN such as BiFPN, but also other detection networks using heavier backbones or heads while operating under similar overall computation budgets. This highlights the importance of core modules operating on feature pyramids in general object detection networks.
baselines (bottom four rows), we again
the ResNet-50+TPN networks despite its
only outperforms other cores
4.3 COMPARISON BETWEEN DIFFERENT TPN CONFIGURATIONS
In this subsection, we investigate which TPN contigurations yield the best accuracy vs. efficiency trade-off at various computation budgets. Here, TPN configuration is determined by the hyperpa- rameter pair (L, B), respectively denoting the number of TPN layers and the number of bottleneck layers per self- self-processing node. We use the same settings as explained in Subsection 4.1 except
that we only train for 12 epochs using the Ix training schedule. In Figure g we compare 15 dif- ferent (Z, B) TPN configurations using the 'parameters', 'training latency" and 'inference latency' efficiency metrics, with the latency metrics obtained using the same methodology as in Table
We can see from the magenta curves yielding the optimal TPN configurations at various computation budgets, that having a good balance between communication-ban layers L) and self-processing (in the form of bottleneck layers per self-processing node B) is im- portant. We can for example see that the balanced (2, 2) configuration outperforms the unbalanced (3. 1) and (1, 5) configurations. The same observation can be also made at higher computation bud- gets, where the balanced (2, 4), (4, 2) and (3, 3) configurations outperform the unbalanced (6, 1) and (7, 1) configurations.
processing (in the form of TPN
8

---

Under review as a conference paper at ICLR 2022
Table 2: Comparison of best-performing TPN model with other prominent object detection networks on the 2017 COCO validation set (see Table [ for the definitions of B, L and C). All models use ResNet-50 (He et al. 2016a) backbone. The number of FLOPS are computed in Carion et al. (2020), by applying the www.ppountoooooe a tors tool from Detectron2 (Wu et al. 2019] on the first 100 images of the validation set. The number of inference FPS 'iFPS' calculated
as explained in Table | while using the implementations provided by MMDetection (Chen et al. 2019) for the baseline models.
Model
B
L
C
Epochs
AP
APse
APys
APs
APM
AP1
Params
GFLOPS
IFPS
Faster R-CNN+FPN (Wo et RetinaNet-FPN [Wa 2019
2019
40.2
43.8
24.2
43.5
52.0
41.5
180
4.1
4
37
38.7
58.0
41.5
23.3
42.3
30.3
37.7 M
208
5.1
DETR CCrion
500
44.2
20.5
45.8
41.3
86
7.3
Deformable DETK ZBB ml. 2020
50
43.8
62.6
47.7
26.4
47.1
38.0
39.8 M
173
3.5
RetinaNet TPN (ours)
41.8
44.4
26.2
46.1
53.7
36.7
121
5.3
We hence empirically show that balanced (L, B) configurations with L 2 2 and B 2 2 are to be preferred over unbalanced configurations such as (I., 1) and (1, B). By having only one self- processing operation in between each communication-bon bbased based operation, existing cores with a (I. 1) configuration such as PANet (Liu et al. 2018) and BiFPN (Tan et al. 2020) crucially lack self- processing. The TPN core solves this problem by introducing the (L, B3 hyperparameter pair, such that a better balance between communication- based processing and self-processing can be chosen.
4.4 COMPARISON WITH OTHER OBJECT DETECTION NETWORKS
In Table 2 we compare the performance of our best-performing model against other prominent object detection networks. We make following two sets of observations.
First, we compare our model with the popular two-stage object detector Faster R-CNN+FPN (Ren cct al. 2015; Lin et al. 2017a) and the popular one-stage object detector RetinaNet+FPN (Lin et al. 2017b a) (top two rows). Under similar training schedules, we observe that our model performs better across all object sizes, while being computationally cheaper (three rightmost columns). Note that the RetinaNet+FPN model closely resembles our RetinaNet+TPN model, except that they use a FPN core and a heavier head using hidden layers in the classification and bounding box subnets Wu al., 2019) instead of 1. The results again show that it is more beneficial to put additional computation into the core (RetinaNet+ TPNN rather than into the head (RetinaNet+FPN),
Secondly, we compare our model with the DETR (Carion al. 2020) and Deformable DETR (Zhu et al. 2020) models (middle two rows). These are two recent object detectors based on transformers with set prediction. On small objects, we observe similar results for our model and Deformable DETR, while DETR performs significantly worse. This can be understood from the fact that DETR only operates on a single low resolution feature map instead of a feature pyramid, once again proving the importance of feature pyramids. On large objects on the other hand, we see clear superiority of the transformer-based detectors with set prediction compared to our model using a simple RetinaNet-like head. Given that our TPN model only specifies the core module, combine with this new type of detection head that uses a transformer decoder combined with set prediction (Carion et al. 2020). We believe this should improve the performance of our TPN model for large objects in a similar way as for the DETR and Deformable DETR models. We will further investigate this in future work.
could
5 CONCLUSION
In this paper, we introduced new type of core architecture, called the Trident Pyramid Network (TPN). We show consistent improvements when using our TPN core on the COCO object detec- tion benchmark compared to various baselines. From our TPN experiments, we see that both communication basdeiaaaai-dddddddddeeeee processing and seofepsrcsssing are crucial within core modules order to obtain good results. We additionally observe that TPN performance could not be matched by heavier backbones or heads under similar overall computation budgets, showing the importance and effectiveness of the core within object detection systems.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Alexey Bochkovskiy, Chien-Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
Zhaowei Cai, Quanfu Fan, Rogerio S Feris, and Nuno Vasconcelos. A unified multi-scale deep convolutional neural network for fast object detection. In European conference on computer vision, pp. 354-370. Springer, 2016.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprini arXiv: 199900001555 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Golnaz Ghiasi, Tsung-Y Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7036-7045, 2019.
Jianyuan Guo, Kai Han, Yunhe Wang, Chao Zhang, Zhaohui Yang. Han Wu, Xinghao Chen, and Chang Xu. Hit-detector: Hierarchical trinity architecture search for object detection. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11405- 11414, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 20.
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Scale-aware trident networks for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6054 6063, 2019.
Di Lin, Dingguo Shen, Siting Shen, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or Or, and Hui Huang. Zigzagnet: Fusing top-down and bottom-up context for object segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 7490-7499, 2019,
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision. pp- 2980-2988, 2017b.
Shu Liu, Lu Qi. Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8759-8768, 2018
10

---

Under review as a conference paper at ICLR 2022
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.
arXiv preprint
arXiv:1711.05101 2017.
Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European conference on computer vision, pp. 483 499, Springer, 2016.
Siyuan Qiao, Liang-Chide Chigh- Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10213-10224, 2021.
Iliaa Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10436, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv: 1506.01497 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Abhinav Shrivastava, Rahul Sukthankar, Jitendra Malik, and Abhinav Gupta. Beyond skip connec- tions: Top-down modulation for object detection. arXiv preprint 4rXiv:1612.06887, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 144i::1409.1505 2014.
Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 10781-10790, 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 1492-1500, 2017.
Qijie Zhao, Tao Sheng, Yongtao Wang, Zhi Tang, Ying Chen, Ling Cai, and Haibin Ling. M2det: A single-shot object detector based on multi-level feature pyramid network. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 9259 026999266, 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A. COMPARISON BETWEEN TPN AND BIFPN WHEN USING LARGE BACKBONE
In this appendix subsection, we provide additional results comparing the TPN core with the BiFPN core, when using the large 100133444
(Xie et al. [017 Zhu et al. 2020) backbone. To carry out these experiments, we chose the TPN core with neck layers per self self-processing node. We follow the same settings as explained in Subsection 4. except that we use the training schedule and train on only one GPU instead of two.
TPN layers, and B bottle-
The results are found in Table We can see that the TPN core outperforms the BiFPN core by 0.7 AP, while having a similar computation budget. We can morcover see that the TPN core has better performance across all object scales. These additional experimental results show that the superiority of the TPN core compared to the BiFPN core generalizes to larger backbones.
Table 3: Experiment results on the 2017 COCO validation set comparing the TPN core (top row) with the BiFPN core (bottom row), when using the large The six leftmost columns specify the model and training settings, the six middle columns show the model performance, and the five rightmost columns contain the computational characteristics of the model. These characteristics are obtained as explained Table [] except that we use a batch of two 600 600 images (instead of two 800 800 images) avoid out-of-memory (OOM) error the used GeForce GTX 1660 Ti GPU.
S00-X-10--322-DDCNEC backbone.
Backbone
Core
Epochs
AP
APp)
APrs
APg
APM
AP,,
Parums
tFPS
iMem
iFPS
iMem
XICI XI0I-DCNV2
TPN
12
41.1
61.9
47.7
27.9
48.6
58.0
59.1 M
3.60 GB
4.7
0.39 GB
43.4
56.7
3.59
12


Reviews:
[{"review_id": "e0VUEZJ4bOm", "reviewer": "Reviewer_YsZ7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work proposes a new core architecture (called the Trident Pyramid Network) for object detection CNNs and claims that:\n1. A heavier core enhances object detection performance. \n2. It is better to invest any extra computational budget on a heavier core rather than on a heavier backbone. \n\nExperimental studies that compare the proposed method with counterparts like PANet, FPN, and Faster-RCNN+FPN on the COCO object detection benchmark corroborate the above claim 1. However, empirical evidence for claim 2 is not convincing enough. The quality of the paper may be enhanced if some concerns (see cons below) are addressed during the rebuttal period.\n\n--------------------------\nPros:\n--------------------------\n1. The paper is well-written. The flow of the paper is engaging. The method is well-motivated and the literature review is sufficient. \n\n2. Figures are informative. For example, Figure 4 helps the reader visualize the architecture of the  proposed TPN core.\n\n3. The trade-off between computational cost and detection performance of the proposed TPN can be controlled by two hyperparameters, namely B (controlling the amount of self-processing) and L (controlling the amount of communication-based processing). This enables the user to choose the necessary amount of self-processing versus communication-based processing.\n\n4. Experimental studies with various B and L values for TPN seem to outperform other methods under comparison. The proposed method’s performance is shown to be 3 AP and 1.5 AP better than PANet and BiFPN respectively (with the same backbone network and under similar computational cost). In addition, TPN outperformed RetinaNet and Faster-RCNN+FPN by 3.1 AP and 1.6 AP respectively,  convincing the reader that the proposed method is effective. \n\n5. The proposed method is shown to outperform other CNN based  architectures like Faster-RCNN+FPN and RetinaNet.\n\n6. Although TPN underperforms compared to vision transformer based architectures like, DETR and deformable DETR, the paper claims to adopt transformer based processing in the TPN core to enhance performance in future work.\n\n\n--------------------------\nCons:\n--------------------------\n1. The paper claims that using self processing in the core is beneficial, especially by allowing individual feature maps “work” on themselves before being combined with feature maps of different resolution. It would be beneficial to see more explanation on the intuition behind this. Specifically, why is it better to let the feature map work on itself at the core stage as opposed to letting it develop better at the backbone stage, using a heavier backbone?\n\n2. Why does the residual branch as seen in Figure 5 have C1-C3-C1 and not C1-Cx-C1, where x could be 5 or 7 for example? A larger convolutional kernel covers a larger area of the activation map, has a larger receptive field, and may result in better detection performance (although more expensive) when objects of different size (especially larger) are present in the dataset.\n\n3. The top-down and bottom-up blocks as shown in Figure 6 have a specific and distinct structure. What is the reason behind this? Why doesn’t the top-down block have C1-C3-C1-I as opposed to just C1-I?\n\n4. It seems that varying L also varies the total number of self-processing blocks in the TPN core. That is, if L=3 and B=2 (per L), the total number of self-processing blocks in TPN is 2*3=6. However, varying B doesn’t affect the overall number of communication based blocks. Is there a way of varying L independent of B? If not, it is worth mentioning it in the paper for the benefit of the reader.\n\n5. Although different values of B and L result in similar performance, the paper provides empirical evidence that using a larger value of L compared to B yields marginally better performance, in some cases. Even though pointing this out is helpful, the results do not demonstrate that the AP will increase as L increases (while decreasing B to maintain similar cost), e.g., B=2, L=3 (36.7M parameters) yields 41.8 AP, while decreasing B to 1 and increasing L to 5 also yields the same AP, although this setting has 0.4M more parameters. Moreover the paper states the following - “It appears that having more TPN core layers L is slightly more beneficial than having more bottleneck layers B under similar computational budgets. Especially on large objects we observe noticeable differences ranging from 52.5 to 53.7 AP_L”. This statement may imply that increasing L while keeping the cost similar (by decreasing B) yields better performance. However AP_L decreases from 53.7 to 53.4 when L is increased from 3 to 5 (and B decreases from 2 to 1), contradicting this notion. Please provide additional experimental evidence (try out other L>B values) to support the claim above in quotes. Specifically, given similar cost, would increasing L (and decreasing B) always result in better performance? Or are there desirable values of L>B for which the performance is optimal and further increasing L does not help?\n\n6. Paper claims that using more computation at the TPN core results in better performance compared to using a heavier backbone and reiterates the importance of processing at the core level in object detection networks many times. An attempt is made to empirically demonstrate this by showing that ResNet-50+TPN outperforms ResNet-101+FPN, under similar computational cost. However, this result is not sufficient to verify the claim above. It would be interesting to see how ResNet-101+TPN  performs compared to ResNet-50+TPN, under a similar computation budget. Would it be the case that ResNet-50+TPN outperforms ResNet-101+TPN? If so, the above statement claiming that more computation at the core yields better performance can be verified convincingly.\n\n7. Table 2 in page 8 has TPN in the last row, it is not clear what the B and L values are for this configuration. Upon referral to Table 1, it seems like B=2 and L=3. It would be beneficial to mention the B and L values of TPN in Table 2.\n", "summary_of_the_paper": "Typically, object detection CNNs consist of a backbone for feature extraction, a core for feature refinement, and a head for predictions. The core is usually a simple arrangement of top-down and/or bottom-up convolutional blocks, used for effectively combining feature maps at different resolutions and this kind of processing is referred to as communication-based processing. This paper claims that the detection performance can be enhanced by the use of a more complex core that enables additional self-processing of individual feature maps (by running them through a series of convolutional layers)  before combining them with feature maps of different resolutions. The paper proposes a new core architecture in object detection CNNs and names it the Trident Pyramid Network (TPN).  In addition, the paper claims that given an additional computational budget, it is better to invest it in a heavier core rather than a heavier backbone, contrary to the mainstream approach. The proposed TPN has two tunable parameters B and L, that control the amount of self processing (B) and the amount of communication-based processing (L).\nExperimental studies on the COCO object detection benchmark show that the proposed approach with ResNet-50 backbone and TPN core outperforms existing methods with ResNet-50/101 backbone with FPN or PANet as their cores, under similar computational cost.  \n", "summary_of_the_review": "The paper is well-written, easy to follow, and the literature review is adequate. The proposed method is well-motivated in the introduction.  Figures describing the proposed method help in better visualization.  Of the two main claims of the paper, one is empirically well-supported, while the other lacks convincing evidence (see main review above). Experimental studies compare the proposed TPN to existing baselines like PANet and FPN, among others, on the COCO dataset. Additionally, the paper  demonstrates empirically that TPN underperforms compared to state-of-the-art transformer based architectures like DETR and Deformable DETR, and in future work, the authors intend to incorporate transformer based processing in the proposed TPN core to further enhance detection performance. The paper may be improved if the cons (see above) are addressed. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MenuC4IrTS9", "reviewer": "Reviewer_NXJS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors point out the importance of the core network (the feature aggregation part between the backbone network and the task-specific head) and make a clear point that more computations should be allocated to the core network. The authors also provide a concrete implementation of the core network, called TPN, which performs \"self-processing\" along with \"communication-based processing\" during the aggregation of feature pyramids. TPN is compared with the SOTA on the COCO object detection benchmark, and the results support the main claim of the paper. \n\nWeakness: \n1. There are many new notions in the paper, including the \"core\" network, \"communication-based processing,\" and \"self-processing.\" Some statements are obscure, e.g. \"too much communication can lead to the core going in circles ... within the core communication-based processing should be alternated with some healthy portion of self-processing.\" Authors are advised to express their meanings using common notions that are more accessible by researchers in the field. \n2. The evaluation is not comprehensive. The backbone-core(neck)-head structure can also be used for semantic segmentation, which is one of the three important tasks in image domain (the classification task is mainly about backbone, and not quite relevant to this architecture). It is a pity that the authors do not validate their design in the segmentation task, leaving the generalization capability of TPN in the fog. \n3. It is well-received that consecutive bottleneck layers are added to most feature layers and between top-down and bottom-up aggregation (Fig.4). However, it is not clear why the top-down and bottom-up operations require \"self-processing\" too (Fig.6). This reviewer does not find any ablation study about whether (and to what extent) this complicated fusion structure outperforms the original simple TD and BU modules, given that \"self-processing\" modules have already been inserted between TD's and BU's. \n", "summary_of_the_paper": "This paper presents a new network architecture for fusing pyramid features in deep neural networks for visual object detection. The authors divide an object detection network into a backbone, a core, and a head. The proposed trident pyramid network, or TPN for short, is a core network structure. The main idea in TPN is that we shall do feature processing along with aggregation in the core part of a network. Experiments are carried out to justify this design. ", "summary_of_the_review": "This paper provides some insights to the design of a three-part network for fine-grained vision tasks. A concrete design is presented and performance gains are demonstrated in the object detection task. However, the experiments are not comprehensive, lacking the evaluation on a relevant task and necessary ablation. The writing of the paper also has room for improvement. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KjXStcGsUhL", "reviewer": "Reviewer_MSxk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written. The experiments demonstrate TPN brings solid improvement (1.5% on COCO) over existing works on RetinaNet w/ ResNet50. \n\nThe technical novelty of this paper is limited. Stacking FPN is not new as BiFPN and PANet also stacks multiple FPN to gain better performance which is also mentioned by the authors. So the major difference between TPN and existing works is that TPN replaces the plain skip connections within each FPN with residual modules. It is interesting to see that replacing skip connections with residual module is a more effective way to improve performance than stacking FPNs. But this change is incremental, and I do not consider this to be a novel design.\n\nTPN introduces two hyper-parameters to control the number of stacks and the number of residual modules. Experiments in table 1 show that how the performance change with the two new hyper-parameters under a fixed computational budget. It would be interesting to see how the performance would change with the computational budgets and different hyper-parameter choices. The authors can consider an efficiency and accuracy tradeoff curve, which would give a better understanding on the importance of the residual modules.\n\nThe authors should try their approach on a deeper network such as ResNet101. It is unclear if the proposed approach would generalize to deeper networks. The authors should also include FPS in Table 2 when they compare TPN with other prominent models. There are other factors that can impact the inference speed of a model such as memory latency and whether the network is parallelizable other than GFLOPS.", "summary_of_the_paper": "This paper proposes a new feature pyramid network, trident pyramid network (TPN). TPN stacks multiple FPNs together and a FPN processes the features produced by its previous stage. TPN also replaces the skip connections within each FPN with residual modules to further process the features. Experiments show that TPN outperforms existing works such as BiFPN and PANet on ResNet50 under a similar computational budget by varying the number of FPNs and residual modules.", "summary_of_the_review": "Although TPN shows some interesting empirical results and solid improvement over existing works on smaller networks, it is not technically interesting, and the novelty is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "26Wu5cEfryK", "reviewer": "Reviewer_EG97", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe results are reasonable, and the method is easy to follow.\n\nCons:\n\n- First, the existing FPN-based detectors (RetinaNet, etc.) have self-processing module, where a set of conv layers will be applied in $P_{l}$ and $P_{l+1}$ before summing up, and the new aggregated feature map $P_{l}$ will also be processed by a conv layer before output. In other words, the claim that the original FPN has no self-processing module is not correct, and the proposed module is simply to replace the vanilla conv layer with residual block, which is too incremental and heuristic. \n\n- Second, it is a common sense that the FPN module is specifically designed for multiscale representations which is extremely suitable for object detection, and thus it's not surprising that enhancing the representation of FPN outperforms heavier backbone with the same computation budget. This topic has been widely studied in the past few years and the similar idea of the paper (stacking FPN, etc.) has already been proposed by the existing frameworks (e.g., BiFPN and PANet), which leads to limited novelty.\n\n- Third, though it's not an academic publication, the idea of dividing the detection framework into three modules (backbone, core and head) has already proposed in the implementation of mmdetection [1] (backbone, neck and head), which is a widely used platform for object detection.\n\n- Finally, what’s the performance if the proposed TPN added in large backbone, e.g, X-101-DCNv2? To validate the generalization ability of the detector, experiments on large backbones are required.\n\n[1] Chen K, Wang J, Pang J, et al. MMDetection: Open mmlab detection toolbox and benchmark[J]. arXiv preprint arXiv:1906.07155, 2019.\n", "summary_of_the_paper": "This paper disentangles the typical detection algorithms into three modules: backbone, core and head, and it proposes a new TPN module to enhance the representation of core module. More specifically, multiple core modules are stacked, each of which is equipped with a residual block for self-processing.", "summary_of_the_review": "The limited novelty is my main concern to accept this paper, and the module design is too heuristic. I recommend \"Reject\" as the pre-rebuttal score.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission receives mixed ratings initially. Three reviewers are on the borderline and one reviewer EG97 leans negatively. The raised issues mainly reside on the technical contribution, technical correctness, and experimental validation. In the rebuttal, the authors have tried to address the raised issues and discussed them in-depth with reviewers. However, the discussion does not change the reviewer's mind. After checking all the reviews, rebuttals, and discussions. The AC stands for the reviewer side that the technical contribution is a major issue that ought to be solved. The proposed TPN comes from the summarization of the existing FPN based structure and there are not sufficient insights to make significant improvements. Besides, there are still unsolved issues regarding the technical presentation and experimental validations. The authors are suggested to improve the current manuscript based on these reviews and welcome to submit for the next venue.
Decision: Reject