Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### summary
This paper introduces capsule network for object detection. To solve the issue of capsule network when applied to large-scale detection problems, this paper develops deformable capsules, a new  prediction head SplitCaps, and a dynamic routing algorithm, SE-Routing. Experiments are conducted on COCO where it performs slightly worse than the baselines but arguably predicts less false positives.

### pros
I appreciate the spirit of developing new architectures for a highly-competitive task like object detection. It's totally acceptable that the performance cannot fully match the SOTA results in this kind of scenario. This paper is insightly as it studies several important issues of capsule network when being applied to large-scale visual tasks. The proposed solutions have at least make the framework run-able on standard hardware.

### cons
1. My current rating is not purely based on the performance, but it is one of the major concerns here. The proposed framework (1) performs worse than baseline -- CenterNet, and (2) performs much worse then the SOTA solutions (Tab.1) in terms of both performance and speed. Note that the network weights are initialized from CenterNet. 
2. The whole analysis in Appendix A.2 reads very confusing for several reasons:
 * CenterNet achieves higher AP (TP/(TP+FP)) and also high FP, which is a quite common thing to me. Ignoring the ratio and simply compare the absolute value of FP doesn't seems to be the correct thing to do.
 * Some visualizations are weird like Fig.4 human and dog. Why NMS cannot remove such highly-redundant predictions? It seems more like a bug to me.
 * On page 12 top row it writes "CenterNet consistently producing fairly confident false predictions (e.g. > 0.4)", how does this threshold 0.4 is chosen? It's not a standard thing and in fact if we improve it to 0.5 or higher, most of the FPs about CenterNet will disappear. Moreover, is this how FP gets counted? If yes, I think this is problematic as the total number (TP+FP) is changing across methods. A more fair way is to choose top-k (k=100 in coco) predictions and compute the metrics.
3. Lots of reference are missing. For example, Tab.1 compares multiple different methods but most of them are not cited. The related work only cites few very classical papers, but most recent works are missing.
4. The writing, especially the approach section, isn't very clear. I had a relatively hard time to understand how do each modules work, and what are the motivations behind these design choices. 
5. Moreover, there are couple of new things proposed, and each one of them have specific design choices and parameters. However, there are no ablation studies properly studying them. As a results I don't understand which module is working, and how does it work.

### questions
1. How do the FPS numbers in Tab.1 get computed? Are they simply borrowed from original paper or some published work? Some of them look strange to me. 
2. Why KL in Sec.3.2? Does the asymmetric natural of KL influence the results?

review_title: An interesting idea of using capsule network for object detection; The experimental sections are relatively weak and cannot support the claim.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes to use the capsules to perform object detection on COCO. Capsules, while showing promises, are usually too expensive for tasks beyond MNIST and Cifar. The authors propose three key improvements in DeformCaps, SplitCaps and SE-Routing to improve the efficiency and therefore allow capsules to be applied on larger tasks such as object detection. The authors claim novelties in:

-- First ever capsule-based object detection on COCO.

-- Deformable capsules that let parent to samples child capsules instead of having fixed connections to improve efficiency.

-- SplitCaps: a reparametrization trick to reduce the number of capsules needed.

-- Squeeze-and-excitation routing that finds agreement without iterative loop.

I agree with the authors' novelty claims.

Strengths:

-- Impressive results: Capsule Networks are well known to be computationally expensive. Getting it to work on large images is certainly no easy task. There has been other works operating on larger images, as noted by this paper, but they usually cannot achieve quality on par with CNNs. Having said that, I need to note that the SOTA for COCO has consistently improved, and is at ~51% AP [a], quite better than the numbers listed in Table 1.

-- Good novelty: DeformCapsule and SplitCaps both follow a straightforward pattern to dissect prohibitively large tensor predictions into smaller ones. And together with the SE-routing methods, the novelty should be sufficient for this venue.

-- Clear writing. Figure 1 and 2 helped a lot with the understanding of the key components as well.

[a]: SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization

Weaknesses:

-- Insufficient experimentation: There is only a single experimental table that contains both comparisons to the SOTA and ablations to DeformCaps and SE-Routing. It would be nice to see additional results, be it on another dataset / task (e.g. sem seg), or even additional ablations. For example, one premise of capsules was the requirement of less training data. Is it true here as well? 

Conclusion:

I'd recommend acceptance because of the strong results and good novelty. The only major weakness is the lack of additional ablations to highlight the advantages (if not on AP) over CNNs. Nonetheless, I believe this work is a good step towards more interests and advances in capsules, and will be of interests to the audience of this venue.

Post-rebuttal:
I concur with R1 in that the results look poor when taking into the account how close the proposed method is compared to CenterNet and that it was finetuned from a CenterNet. Therefore, I'll lower my rating to 6.

review_title: Impressive results, would be nice to have more ablations

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:
It is indeed that capsules have many promising attributes but they are not quite easy to be exploited in object detectors. This paper has addressed many problems existed when applying capsule architectures for the object detection task. In particular, deformable capsules, SplitCaps, and SE-Routing are respectively introduced to help tackle the object detection with capsules. I believe the techniques of this paper will attract interests from researchers in the corresponding areas. The writing is also good.

Cons:
There are several points that prevent me from giving a higher rating:
1) I feel that the descriptions or presentations of introduced techniques are not quite sufficient. In particular, I am confused about the details of deformable capsules. What are the exact operations performed by the deformable capsules? Is it to make parent capsules only aggregate information from a smaller set of their children and the sampling of such a smaller set is implemented by deformable operations? I believe it will be much better to present some detailed formulations or equations to illustrate this operation. Similarly, I also recommend adding detailed equations to describe SE-Routing.  

2) Without sufficient information about operations, I found that the descriptions of motivations are also quite weak, especially in the introductory part. It seems that the authors only mention the problems they will address and the techniques they proposed to address problems. There is not sufficient content briefly explaining why the proposed techniques can tackle the mentioned problems, or at least what are the advantages of the proposed techniques for tackling the problems. 

3) The figures also have many confusing points. For example, in Figure 1, the authors marked that solid red arrows represent 'up' but I can only find dotted red arrows in the figure. Moreover, in what place the modules inside the big blue box is implemented in the detector? After reading, I think it should represent what's inside the SplitCaps, but I do recommend the authors to add some indicative symbols to corresponding related modules. 

4) In addition to the reviewed literature, I found that there are some missing articles that also study how to borrow the capsule concepts for various computer vision tasks. For example:
[1] Vijayakumar T. Comparative study of capsule neural network in various applications[J]. Journal of Artificial Intelligence, 2019, 1(01): 19-27.
[2] Zhao Y, Birdal T, Deng H, et al. 3D point capsule networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2019: 1009-1018.
[3] Chen Z, Zhang J, Tao D. Recursive Context Routing for Object Detection[J]. International Journal of Computer Vision, 2020: 1-19.

Overall, I would like to have some feedback from the authors regarding the above issues before making my final decision. 
review_title: A good subminssion that can be improved further

Meta-review: This work proposes capsule networks with deformable capsules for tackling object detection. All reviewers agreed that object detection is an important problem that is interesting to the ICLR community. Reviewers also agree that the proposed approach is novel and interesting, and in particular they mention that proposing an efficient capsule network for detection is non-trivial. On the less positive side, during the discussion phase all reviewers had concerns about the weak experimental validation, particularly missing ablation studies to analyse the effectiveness of their contributions. At the end, all reviewers felt that, while this is a promising direction of research for object detection, the experimentation should be improved.
Decision: Reject