Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new network architecture for fusing pyramid features in deep neural networks for visual object detection. The authors divide an object detection network into a backbone, a core, and a head. The proposed trident pyramid network, or TPN for short, is a core network structure. The main idea in TPN is that we shall do feature processing along with aggregation in the core part of a network. Experiments are carried out to justify this design. 
main_review: In this paper, the authors point out the importance of the core network (the feature aggregation part between the backbone network and the task-specific head) and make a clear point that more computations should be allocated to the core network. The authors also provide a concrete implementation of the core network, called TPN, which performs "self-processing" along with "communication-based processing" during the aggregation of feature pyramids. TPN is compared with the SOTA on the COCO object detection benchmark, and the results support the main claim of the paper. 

Weakness: 
1. There are many new notions in the paper, including the "core" network, "communication-based processing," and "self-processing." Some statements are obscure, e.g. "too much communication can lead to the core going in circles ... within the core communication-based processing should be alternated with some healthy portion of self-processing." Authors are advised to express their meanings using common notions that are more accessible by researchers in the field. 
2. The evaluation is not comprehensive. The backbone-core(neck)-head structure can also be used for semantic segmentation, which is one of the three important tasks in image domain (the classification task is mainly about backbone, and not quite relevant to this architecture). It is a pity that the authors do not validate their design in the segmentation task, leaving the generalization capability of TPN in the fog. 
3. It is well-received that consecutive bottleneck layers are added to most feature layers and between top-down and bottom-up aggregation (Fig.4). However, it is not clear why the top-down and bottom-up operations require "self-processing" too (Fig.6). This reviewer does not find any ablation study about whether (and to what extent) this complicated fusion structure outperforms the original simple TD and BU modules, given that "self-processing" modules have already been inserted between TD's and BU's. 

summary_of_the_review: This paper provides some insights to the design of a three-part network for fine-grained vision tasks. A concrete design is presented and performance gains are demonstrated in the object detection task. However, the experiments are not comprehensive, lacking the evaluation on a relevant task and necessary ablation. The writing of the paper also has room for improvement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper disentangles the typical detection algorithms into three modules: backbone, core and head, and it proposes a new TPN module to enhance the representation of core module. More specifically, multiple core modules are stacked, each of which is equipped with a residual block for self-processing.
main_review: Pros:

The results are reasonable, and the method is easy to follow.

Cons:

- First, the existing FPN-based detectors (RetinaNet, etc.) have self-processing module, where a set of conv layers will be applied in $P_{l}$ and $P_{l+1}$ before summing up, and the new aggregated feature map $P_{l}$ will also be processed by a conv layer before output. In other words, the claim that the original FPN has no self-processing module is not correct, and the proposed module is simply to replace the vanilla conv layer with residual block, which is too incremental and heuristic. 

- Second, it is a common sense that the FPN module is specifically designed for multiscale representations which is extremely suitable for object detection, and thus it's not surprising that enhancing the representation of FPN outperforms heavier backbone with the same computation budget. This topic has been widely studied in the past few years and the similar idea of the paper (stacking FPN, etc.) has already been proposed by the existing frameworks (e.g., BiFPN and PANet), which leads to limited novelty.

- Third, though it's not an academic publication, the idea of dividing the detection framework into three modules (backbone, core and head) has already proposed in the implementation of mmdetection [1] (backbone, neck and head), which is a widely used platform for object detection.

- Finally, what’s the performance if the proposed TPN added in large backbone, e.g, X-101-DCNv2? To validate the generalization ability of the detector, experiments on large backbones are required.

[1] Chen K, Wang J, Pang J, et al. MMDetection: Open mmlab detection toolbox and benchmark[J]. arXiv preprint arXiv:1906.07155, 2019.

summary_of_the_review: The limited novelty is my main concern to accept this paper, and the module design is too heuristic. I recommend "Reject" as the pre-rebuttal score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Typically, object detection CNNs consist of a backbone for feature extraction, a core for feature refinement, and a head for predictions. The core is usually a simple arrangement of top-down and/or bottom-up convolutional blocks, used for effectively combining feature maps at different resolutions and this kind of processing is referred to as communication-based processing. This paper claims that the detection performance can be enhanced by the use of a more complex core that enables additional self-processing of individual feature maps (by running them through a series of convolutional layers)  before combining them with feature maps of different resolutions. The paper proposes a new core architecture in object detection CNNs and names it the Trident Pyramid Network (TPN).  In addition, the paper claims that given an additional computational budget, it is better to invest it in a heavier core rather than a heavier backbone, contrary to the mainstream approach. The proposed TPN has two tunable parameters B and L, that control the amount of self processing (B) and the amount of communication-based processing (L).
Experimental studies on the COCO object detection benchmark show that the proposed approach with ResNet-50 backbone and TPN core outperforms existing methods with ResNet-50/101 backbone with FPN or PANet as their cores, under similar computational cost.  

main_review: This work proposes a new core architecture (called the Trident Pyramid Network) for object detection CNNs and claims that:
1. A heavier core enhances object detection performance. 
2. It is better to invest any extra computational budget on a heavier core rather than on a heavier backbone. 

Experimental studies that compare the proposed method with counterparts like PANet, FPN, and Faster-RCNN+FPN on the COCO object detection benchmark corroborate the above claim 1. However, empirical evidence for claim 2 is not convincing enough. The quality of the paper may be enhanced if some concerns (see cons below) are addressed during the rebuttal period.

--------------------------
Pros:
--------------------------
1. The paper is well-written. The flow of the paper is engaging. The method is well-motivated and the literature review is sufficient. 

2. Figures are informative. For example, Figure 4 helps the reader visualize the architecture of the  proposed TPN core.

3. The trade-off between computational cost and detection performance of the proposed TPN can be controlled by two hyperparameters, namely B (controlling the amount of self-processing) and L (controlling the amount of communication-based processing). This enables the user to choose the necessary amount of self-processing versus communication-based processing.

4. Experimental studies with various B and L values for TPN seem to outperform other methods under comparison. The proposed method’s performance is shown to be 3 AP and 1.5 AP better than PANet and BiFPN respectively (with the same backbone network and under similar computational cost). In addition, TPN outperformed RetinaNet and Faster-RCNN+FPN by 3.1 AP and 1.6 AP respectively,  convincing the reader that the proposed method is effective. 

5. The proposed method is shown to outperform other CNN based  architectures like Faster-RCNN+FPN and RetinaNet.

6. Although TPN underperforms compared to vision transformer based architectures like, DETR and deformable DETR, the paper claims to adopt transformer based processing in the TPN core to enhance performance in future work.


--------------------------
Cons:
--------------------------
1. The paper claims that using self processing in the core is beneficial, especially by allowing individual feature maps “work” on themselves before being combined with feature maps of different resolution. It would be beneficial to see more explanation on the intuition behind this. Specifically, why is it better to let the feature map work on itself at the core stage as opposed to letting it develop better at the backbone stage, using a heavier backbone?

2. Why does the residual branch as seen in Figure 5 have C1-C3-C1 and not C1-Cx-C1, where x could be 5 or 7 for example? A larger convolutional kernel covers a larger area of the activation map, has a larger receptive field, and may result in better detection performance (although more expensive) when objects of different size (especially larger) are present in the dataset.

3. The top-down and bottom-up blocks as shown in Figure 6 have a specific and distinct structure. What is the reason behind this? Why doesn’t the top-down block have C1-C3-C1-I as opposed to just C1-I?

4. It seems that varying L also varies the total number of self-processing blocks in the TPN core. That is, if L=3 and B=2 (per L), the total number of self-processing blocks in TPN is 2*3=6. However, varying B doesn’t affect the overall number of communication based blocks. Is there a way of varying L independent of B? If not, it is worth mentioning it in the paper for the benefit of the reader.

5. Although different values of B and L result in similar performance, the paper provides empirical evidence that using a larger value of L compared to B yields marginally better performance, in some cases. Even though pointing this out is helpful, the results do not demonstrate that the AP will increase as L increases (while decreasing B to maintain similar cost), e.g., B=2, L=3 (36.7M parameters) yields 41.8 AP, while decreasing B to 1 and increasing L to 5 also yields the same AP, although this setting has 0.4M more parameters. Moreover the paper states the following - “It appears that having more TPN core layers L is slightly more beneficial than having more bottleneck layers B under similar computational budgets. Especially on large objects we observe noticeable differences ranging from 52.5 to 53.7 AP_L”. This statement may imply that increasing L while keeping the cost similar (by decreasing B) yields better performance. However AP_L decreases from 53.7 to 53.4 when L is increased from 3 to 5 (and B decreases from 2 to 1), contradicting this notion. Please provide additional experimental evidence (try out other L>B values) to support the claim above in quotes. Specifically, given similar cost, would increasing L (and decreasing B) always result in better performance? Or are there desirable values of L>B for which the performance is optimal and further increasing L does not help?

6. Paper claims that using more computation at the TPN core results in better performance compared to using a heavier backbone and reiterates the importance of processing at the core level in object detection networks many times. An attempt is made to empirically demonstrate this by showing that ResNet-50+TPN outperforms ResNet-101+FPN, under similar computational cost. However, this result is not sufficient to verify the claim above. It would be interesting to see how ResNet-101+TPN  performs compared to ResNet-50+TPN, under a similar computation budget. Would it be the case that ResNet-50+TPN outperforms ResNet-101+TPN? If so, the above statement claiming that more computation at the core yields better performance can be verified convincingly.

7. Table 2 in page 8 has TPN in the last row, it is not clear what the B and L values are for this configuration. Upon referral to Table 1, it seems like B=2 and L=3. It would be beneficial to mention the B and L values of TPN in Table 2.

summary_of_the_review: The paper is well-written, easy to follow, and the literature review is adequate. The proposed method is well-motivated in the introduction.  Figures describing the proposed method help in better visualization.  Of the two main claims of the paper, one is empirically well-supported, while the other lacks convincing evidence (see main review above). Experimental studies compare the proposed TPN to existing baselines like PANet and FPN, among others, on the COCO dataset. Additionally, the paper  demonstrates empirically that TPN underperforms compared to state-of-the-art transformer based architectures like DETR and Deformable DETR, and in future work, the authors intend to incorporate transformer based processing in the proposed TPN core to further enhance detection performance. The paper may be improved if the cons (see above) are addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new feature pyramid network, trident pyramid network (TPN). TPN stacks multiple FPNs together and a FPN processes the features produced by its previous stage. TPN also replaces the skip connections within each FPN with residual modules to further process the features. Experiments show that TPN outperforms existing works such as BiFPN and PANet on ResNet50 under a similar computational budget by varying the number of FPNs and residual modules.
main_review: The paper is very well written. The experiments demonstrate TPN brings solid improvement (1.5% on COCO) over existing works on RetinaNet w/ ResNet50. 

The technical novelty of this paper is limited. Stacking FPN is not new as BiFPN and PANet also stacks multiple FPN to gain better performance which is also mentioned by the authors. So the major difference between TPN and existing works is that TPN replaces the plain skip connections within each FPN with residual modules. It is interesting to see that replacing skip connections with residual module is a more effective way to improve performance than stacking FPNs. But this change is incremental, and I do not consider this to be a novel design.

TPN introduces two hyper-parameters to control the number of stacks and the number of residual modules. Experiments in table 1 show that how the performance change with the two new hyper-parameters under a fixed computational budget. It would be interesting to see how the performance would change with the computational budgets and different hyper-parameter choices. The authors can consider an efficiency and accuracy tradeoff curve, which would give a better understanding on the importance of the residual modules.

The authors should try their approach on a deeper network such as ResNet101. It is unclear if the proposed approach would generalize to deeper networks. The authors should also include FPS in Table 2 when they compare TPN with other prominent models. There are other factors that can impact the inference speed of a model such as memory latency and whether the network is parallelizable other than GFLOPS.
summary_of_the_review: Although TPN shows some interesting empirical results and solid improvement over existing works on smaller networks, it is not technically interesting, and the novelty is limited.
review_title: null

Meta-review: The submission receives mixed ratings initially. Three reviewers are on the borderline and one reviewer EG97 leans negatively. The raised issues mainly reside on the technical contribution, technical correctness, and experimental validation. In the rebuttal, the authors have tried to address the raised issues and discussed them in-depth with reviewers. However, the discussion does not change the reviewer's mind. After checking all the reviews, rebuttals, and discussions. The AC stands for the reviewer side that the technical contribution is a major issue that ought to be solved. The proposed TPN comes from the summarization of the existing FPN based structure and there are not sufficient insights to make significant improvements. Besides, there are still unsolved issues regarding the technical presentation and experimental validations. The authors are suggested to improve the current manuscript based on these reviews and welcome to submit for the next venue.
Decision: Reject