Deformable Capsules for Object Detection
Authors: Rodney LaLonde
Naji Khosravan
Ulas Bagci
Venue:  (2021)
Keywords: Representation Learning
Capsule Networks
Object Detection

Abstract: Capsule networks promise significant benefits over convolutional networks by storing stronger internal representations, and routing information based on the agreement between intermediate representations' projections. Despite this, their success has been mostly limited to small-scale classification datasets due to their computationally expensive nature. Recent studies have partially overcome this burden by locally-constraining the dynamic routing of features with convolutional capsules. Though memory efficient, convolutional capsules impose geometric constraints which fundamentally limit the ability of capsules to model the pose/deformation of objects. Further, they do not address the bigger memory concern of class-capsules scaling-up to bigger tasks such as detection or large-scale classification. In this study, we introduce deformable capsules (DeformCaps), a new capsule structure (SplitCaps), and a novel dynamic routing algorithm (SE-Routing) to balance computational efficiency with the need for modeling a large number of objects and classes. We demonstrate that the proposed methods allow capsules to efficiently scale-up to large-scale computer vision tasks for the first time, and create the first-ever capsule network for object detection in the literature. Our proposed architecture is a one-stage detection framework and obtains results on MS COCO which are on-par with state-of-the-art one-stage CNN-based methods, while producing fewer false positive detections.

methods, while producing fewer Under review as a conference paper at ICLR 2021
DEFORMABLE CAPSULES FOR OBJECT DETECTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Capsule networks promise significant benefits over convolutional networks by stor- ing stronger internal representations, and routing information based on the agree- ment between intermediate representations' projections. Despite this, their success has been mostly limited to small-scale classification datasets due to their computa- tionally expensive nature. Recent studies have partially overcome this burden by locally-constrainingg the dynamic routing of features with convolutional capsules. Though memory efficient, convolutional capsules impose geometric constraints which fundamentally limit the ability of capsules to model the pose/deformation of objects. Further, they do not address the bigger memory concern of class-capsules scaling-up to bigger tasks such as detection or large-scale classification. In this study, we introduce deformable capsules (DeformCaps). a new capsule structure (SplitCaps)) and novel dynamic routing algorithm (SE-Routing) balance compu- tational efficiency with the need for modeling a large number of objects and classes. We demonstrate that the proposed methods allow capsules to efficiently scale-up to large-scale computer vision tasks for the first time, and create the first-ever capsule network for object detection in the literature. Our proposed architecture is a one-stage detection framework and obtains results on MS COCO which are on-par with tate-off-the-ar one-stage false positive detections.
INTRODUCTION
Capsule networks promise many potential benefits over convolutional neural networks (CNNs). These include practical benefits, such as requiring less data for training or better handling unbalanced class distributions (Jiménez-Sāncher et al., 2018), and important theoretical benefits, such as building- in stronger internal representations of objects (Punjabi et al., 2020), and modeling the agreement between those intermediate representations which combine to form final object representations (e.g. part-whole relationships) (Kosiorek et al. 2019 Sabour et al. 2017). Although these benefits might not be seen in the performance metrics (e.g. average precision) on standard benchmark computer vision datasets, they important for real-world applications. As an example, it was found by Alcor et al.] (2019) that CNNs fail to recognize 97% of (heir pose space, while capsule networks have been shown to be far more robust pose variations of objects (Hinton et al.) 2018); further, real-world datasets are not often as extensive and cleanly distributed as ImageNet or MS COCO.
These benefits are achieved in capsule networks by storing richer vector (or matrix) representations of features, rather than the simple scalars of CNNs, and dynamically choosing how to route that information through the network. The instantiation parameters for feature are stored in these capsule vectors and contain information (e.g. pose, deformation, hue, texture) useful for constructing the object being modeled. Early studies have shown strong evidence that these vectors do in fact capture important local and global variations across objects' feature components (or parts) within class (Punjabi et al. 2020 Sabour et al., (017). Inside their networks, capsules dynamically route their information, sceking to maximize the agreement between these vector feature representations and the higher-level feature vectors they are attempting to form.
Despite their potential benefits, many have remained unconvinced about the general applicability of capsule networks to large-scale computer vision tasks. To date, no capsule- study has achieved classification performance comparable to CNN datasets such ImageNet, instead relegated to smaller datasets such as MNIST or CIFAR. Worse still, to the best of our knowledge, no capsule network has shown successful results in object detection, very important problem in computer

---

Under review as a conference paper at ICLR 2021
vision, robotics, and medical imaging. Now, the argument can be made that standard benchmark datasets such as ImageNet or MS COCO likely contain majority of objects in that 3% range of usual poses, and thus CNNs will appear to perform extremely well when measured in terms of accuracy, stripping capsule networks of one of their largest advantages. However, until capsule networks can perform on-par with CNNs these typical object poses, few will care about the benefits of stronger internal representations and better generalization to unseen poses.
Summary of Our Contributions: (1) We propose the first ever capsule-based object detection framework in the literature. Our network is one-stage (single-shot) architecture, where objects are both localized and classified using capsules, and can perform on-par with the state-of-ther of-the-art CNNs on a large-scale dataset (MS COCO). (2) We address the geometric constraint of convolutional capsules routing) by introducing deformable capsules, where parent capsules learn to adaptively sample child capsules, effectively eliminating rigid spatial restrictions while remaining memory efficient. (3) We design a new capsule-base based prediction head structure, SplitCaps, which reformulates the projections of an objects' instantiation parameters, presence, and class, eliminating the previous dimensional increase of capsules by the number of classes. This crucial addition enables the training of capsule networks on large-scale computer vision datasets for the first time in the literature. (4) To route information across SplitCaps' unique structure, we introduce a novel Squeeze- and-Excitation inspired dynamic routing algorithm, SE-Routing, which seeks to maximize agreement between child capsule projections, without the need of iterative loops
(and
2 DEFORMABLE CAPSULES: FIXING LOCALLY- DYNAMIC ROUTING
The capsule network architecture proposed by Sabour et al. (2017) acted global information, where digit capsules represented the pose and presence of digits in an image regardless of spatial location. The information from all children in the previous layer was sent to every parent in the following layer, weighted via the routing coefficients found in a cosine similarity routing algorithm. While this proved to be highly-effective strategy, it was also computationally expensive, limiting its use to only small-scale datasets. Recent works attempted to scale up capsule networks to larger problems such as biomedical image segmentation (LaLonde & Bagci] 2018) or action detection in video (Duarte et al.) 2018) by using convolutional capsules and -orallyy-ontttiiinng the routing algorithm. Although efficient solutions were presented in those studies, the representation power of capsule networks was fundamentally limited due to imposing local constraints. This is because convolutions, by design, have a fixed geometric structure (Dai et al., 2017), and such geometric constraint significantly inhibits capsules' ability to model phrt- re relationships, relying parts of objects to fall within fixed local grid. Therefore, it is unreasonable to expect a capsule to effectively represent the pose and deformations of an object when the information related to the parts of that object are locked into fixed spatial relationship.
In this study, we propose to effectively solve this aforementioned problem by introducing method that balances efficiency with the ability for a capsule to represent any pose and deformation of an object (i.e. where child capsules the same parent). In such a formulation, global information is not explicitly required, but it does require parent capsules have more flexibility over which child capsules they draw information from. Our proposed solution is deformable capsules. The idea behind the proposed algorithm is simple: if parent capsules are supposed to capture common deformations of the objects they represent within their vectors, then the choice of which children to aggregate information from must be handled in a deformable manner as well. Deformable capsules allow parents to adaptively gather projections from non-spatially-fixxx set of children, and thus effectively and efficiently model objects' poses.
be found in different spatial relationships to one another for
To achieve this overall goal, we follow the same efficient convolutional capsule paradigm, where projection vectors are formed via convolution operation with a kernel centered on the parent capsules' spatial location, but now we learn an additional set of weights for each parent capsule. These learnable weights are the same shape as each parent's 's kernel, and represent the offset values for the spatial sampling of child capsules for that parent. Based on the child capsule representation vectors in the previous layer, these weights learn which children a parent capsule should adaptively sample from for a given input image. Dynamic routing then determines how to weight the information coming from each of these children based on their agreement for each projected parent.
2

---

Under review as a conference paper at ICLR 2021
Let us give concrete example to better illustrate the parameter savings of this technique. Given H = 128 by W = 128 grid of child capsules with ç; = 32 capsule types of a, = 8 atoms each, being routed to set of === 10 parent capsule types of a; 16 atoms each, the fully- fully-connected capsules offSabour et al. (017) would require x W xx X x02 = 128x 128 x 32 xx 10 16 z 671M
xa,
parameters for this layer alone (assuming the goal classification, with detection requiring multiplicative increase by the detection grid size). Instead, using our proposed deformable capsules with k2 52 kernel, we only require 2 x k a, X Cj X 0j = 2 x 5 5 8 10 16 se 64K parameters. Convolutional capsules with Docall-conssrrinee needing the additional spatial offsets kernel, but mentioned above, they are fundamentally limited in the poses and deformations that they can represent. In our experiments, we found deformable capsules to converge faster and much higher performance than convolutional capsules.
routing require 32K parameters, not
3 OBJECTS AS CAPSULES: SplitCaps WITH SE-Routing
We propose a novel one-stage (single-shot) capsule network architecture for object detection, called DeformCaps, where objects are detected, classified, and modeled with capsules. Our overall network architecture, shown in Fig. I is built upon CenterNer by Zhou et al. (2019a) who proposed to represent objects in images scalar point values located the center of their bounding boxes. The authors then regress the remaining characteristics of the object (e.g. height, width, depth) for each center-point detected. In our work, we follow the same center-point detection paradigm, but represent objects with capsule vectors instead. Since several recent studies have found utilizing CNN backbone before forming capsule types to be beneficial to overall performance (Duarte et al. 20188) Kosiorek et al. 2019 T'sai et al., 2020), adopt the preferred backhone of CenterNet, DLA-34 (Yu et al. 2018), as this gave the best trade-off between speed and accuracy. Features extracted from the backbone are sent to capsule object detection head and bounding box regression head. In order to perform object detection using capsules, we introduce new capsule structure, called SplitCaps, composed of class-agnostic and class-presence capsules, and novel routing algorithm, called SE SE-Routing.
Head.1:
Inside SplitCaps:
Input
:Ouppot
w,
32
Baokbona
Heatmap
Head
:Outport BBox (Hight, Width, x-off, y-off)
SplitCaps
8
16
16
Conv
Up
Unity
SE Routing
Daformable Conv
Sum
Figure Deformable capsule architecture for object detection.
3.1 SPLITCAPS: CLASS- CAPSULES AND CLASS-PRESENCE CAPSULES
As discussed in Section 2 the original CapsNet by Sabour al. 2017 extremely expensive in computation and our proposed deformable capsules is a best possible solution to balance non-rigid deformations of objects while remaining memory efficient. However, there is
significant memory hurdle capsule networks must overcome when sealing up to large-scale datasets, such as MS COCO. In their current implementation, capsule networks represent each class with its parent
3

---

Under review as a conference paper at ICLR 2021
capsule vector. On small scale classification datasets (and using deformable routing), this is not an issue; it amounts to X k x k x a, x Cj X a parameters and N x c; x Cj x aj x bytes to store the intermediate representations to be routed for the parents, where Cj is usually around 10 classes to represent. Let us suppose we have 5 x 5 kernels with 32 input capsule types of 8 atoms per capsule, 10 output capsule types of 16 atoms per capsule, and batch size of 32. In total, would have 2 x 5 x 5 x 8 x 10 x 16 = 64K parameters and 32 X 32 X 10 X 16 X 4 & 655 KB.
When we scale this up to object detection, we now need store representations for every possible object location, and for MS COCO we need to represent 80 possible classes. This gives us 2 x k x k x x x C3 0 + 2 5 5 x 8 x 80 16 512K parameters and N x H W x c, X Cj X a, X 4 bytes + 32 x 128* x 128 x 32 x 80 x 16 x 4 = 86 GB for the intermediate representations, where we assume the output grid of detections is 128 x 1 128 with a single detection (i.e. bounding box) predicted per class per location. The problem is not any better for large-scale classification datasets such as ImageNet either, where we lose the grid of predictions but grow to 1000 classes, which would require 2 x 5 x 5 x 8 x 1000 x 16 = 6.4M parameters and 32 x 32 x 1000 16 x 4 R 66 GB for the intermediate representations. Clearly, with most GPU memories limited to 12-24 GB, a solution is needed to be found for capsule networks to scale up to larger-scale computer vision tasks.
To overcome this issue, we propose a new type of capsule architecture, SplitCaps, to more efficiently scale capsule networks to large-scale computer vision tasks. SplitCaps contains two parent capsule types, each with a different number of atoms per capsule, for each location of the detection grid. As before, the idea is to balance efficiency with the ability to learn powerful representations. Towards this goal, SplitCaps proposes to divide up between its two parent capsules the tasks of (i) learning the instantiation parameters necessary model the possible variations of an object and (ii) predicting which classes of objects are present in a given input. The first capsule type we refer is our class- agnostic object instantiation capsules, and the second we refer is class presence capsules.
Class Class-agnostic object instantiation capsules: The purpose of these capsules is similar to those in previous works: model the possible variations (in pose, deformation, texture, etc.) of objects within a vector of instantiation parameters, the span of which should cover all possible variations for that object at test (hence why capsules are better than CNNs at generalizing to unseen poses). While previous capsule networks did this in a class-wise manner, argue such a formulation is not required and possibly it is redundant. Many variations (e.g. rotation, skew, stroke thickness) may be class-independent, and thus to model these variations class wise would require repetition across each capsule type. Instead, we propose to model all classes within a single capsule type (i.e. class-agnostic). In this way, while class-dependent variations would each require their own dimensions of the vector, class-independent variations can each be modeled along a single dimension for all possible objects. Since it is reasonable to assume there will be at least some class-specific variations, we increase the default capsule vector dimension from 16 to 64 to accommodate for possible class-specific instantiation parameters.
During training, if an object is present at a given spatial location, the 64-dimensional capsule vector for that location is fed to a reconstruction regularization sub-network to construct the mask of that object as similar to the reconstruction regularization used by [Sabour et al. (2017) for classification. This sub-network is a relatively small and fast addition: a set of three Reltvvact 1 x 1 convolutional layers with 256 filters each, followed by final sigmoid-activated 1 x 1 convolution with N n² = 28² = 784 filters, before reshaping outputs to n X n. Since objects' scales vary dramatically, we scale normalize all objects' ground-truth masks to be 28 x 28 (by following He et al. (2017)) ). Supervised training is conducted by computing the Dice loss (Milletari et al., 2016) between the predicted reconstruction, r, and the object's 's mask, m
2 N N rmm; ΣN
C,
(1)
N ? + ΣN N m?
where C, is used to provide regularization signal to the instantiation parameters being learned.
Class presence capsules: The class presence capsules attempt to model which classes of objects are present in the input at each spatial location, if any. We accomplish this by setting the atoms per capsule to the number of classes being represented (i.e. 80 for MS COCO). Just as Hinton et al. (2018) separately modeled pose (with matrix) and activation (with scalar), this 80-dimensional vector can be viewed as a flass-dependent set of activation values. The activation values are then passed through sigmoid function and thresholded; if one more activation values are above the
4

---

Under review as a conference paper at ICLR 2021
threshold, object is determined to be at that spatial location with the strongest activated dimension determining the class label.
In order to produce smooth loss function during training, create ground truth heatmap by fitting a Gaussian distribution rather than a single point, to the center- center-point of each object's s bounding box, with variance proportional to the size of the box following /Zou et al. (2019a). More specifically, we create a heatmap E [0, 1] x x containing each down- ground truth center-poim
p (2) d for class Is K using Gaussian kernel Hykk exp
+yy- where d is (x- 1e-Pa))
20
the amount of downsampling in the network and op is an object-size-adape -adiiivve standard deviation (Law) Deng 2018). In the case of overlapping Gaussians, we take the element-wise maximum. To handle the large class imbalance between objects and background in our heatmaps, we use penalty-reduced pixel-wise logistic regression with a focal loss (Lin et al. 2017):
(1 Hayk) log(Hzzl.
H Hayk 1,
Ln
P 2yk (1 (1- Hayk) (flzyk) Log(1 Hayk). otherwise;
where a, 3 are hyper-parameters of the focal loss, P is the number of center-points in the input, used to normalize all positive focal loss instances to 1 (Zhou et al.] (2019)). We use = 2 and |3 4 in all our experiments, following Law & Deng (2018). At test, to efficiently retrieve the object's exact center, we a 3 x 3 max-pooling over the thresholded spatial map.
To predict the height and width of the bounding boxes of objects and recover the x, offsets needed to map back to the upscaled image dimensions, follow the same formulation [Zhou et al. (2019a) and pass the backbone features through a 3 x 3 convolutional layer with 256 feature maps, then 1 x 1 convolutional layer with 2 feature maps. These layers predict the local offset, Ô E R t x x2 and size prediction, S IR 4 x 4 x2, for each center-point and are supervised by
| Σ C. = P O - (2 --- and L. = 1 Σ Sp 5
(02 x1, - wi)/
(3)
respectively. Our final objective function is thus defined as L Ln + N, + AoC. + JoCo. We keep A = 0.1 and Ao = as done in Zhou et al. (2019a), and set A = 0.1 initially, then step up to A = 2.0 at the half-way in training.
3.2 SE-ROUTING: SPLITCAPS NEW ROUTING ALGORITHM
As core component of capsule networks, dynamic routing seeks to maximize the agreement between child capsule projections for parent capsules and to fully-leverage the richer representations being stored. Since SplitCaps introduces unique capsule head structure, where instantiation parameters and activations are split across different capsule types, previous dynamic routing algorithms can no longer be directly applied. To overcome this, we propose new dynamic routing algorithm that takes inspiration from illustrated in Fig. 2
lateeeee-andExxcttatte networks (Hu et al. 2018), which we call SE-Routing, and
Previously proposed dynamic routing algorithms (e.g. Saboou et al. (017)) and Hinton et al. (2018) were typically iterative, requiring hand-tuned loop of routing iterations, which proved be slow and temperamental in practice. Different studies found different numbers of iterations to be effective, and one meta-study of five different iterative dynamic routing algorithms found them all to be largely ineffective (Paik et al. 2019). To avoid this pitfall, we propose a new routing algorithm to dynamically assign weights child capsule projections based on their agreement, computed in a single forward pass using simple gating mechanism with sigmoid activation. Unlike Sabour et al. (2017) which uses a routing softmax force a one-hot mapping of information from each child to parents, our proposed SE-Routing learns allow multiple children to be emphasised for each parent.
mwti-mmttllll-ya -xxclusiv relationship between children and parents to
Creating child capsule projection descriptors (squeeze): Following the Squeeze-and-Excine paradigm, we first must compute the squeeze (i.e. a set of descriptors which summarize relevant information about each feature) to create set of child capsule projection descriptors. In (u et al.) (2018), the authors proposed to use the global average activation of each channel with the goal of modeling channel interdependencies. In this study, our goal is to maximize the agreement
5

---

Under review as a conference paper at ICLR 2021
Deformably sampled
capacde projection vactors
Onjet SpilCcpa
0
Class
Presenor
pro)
Oigh
0
Instan
Raconsttccion
Brasci
ONT
Mask
Prams.
Deformable Caps
proj
Obyi2 Spritcpa
Class
Clase Prediction
proj.
Mask
Child Capsule Descriptors
2-Layer Bottleneck
Routing Confficients Capsula Projections
SE-Routing
Coe, angle wi Mean
KL Diverg: wi Agt- Distrilt.
Class Distrib, Varience
1
Figure 2:
Proposed deformable SplitCaps formulation which, in one-shot, localizes all objects to their centers, determines their classes, and models their instantiation parameters in two parallel parent capsule types. Information is dynamically routed from children to parents by passing three chosen child capsule descriptors through two-layer Squeeze-ande -ndiiiitttiion (S&E) bottleneck.
between child projections, for both the instantiation parameters and class presence of the object being modeled. With that motivation, we compute three separate descriptors which are fed into the excitation phase of the routing: cosine angle between the mean projection vector and each child's projection, which captures object instantiation agreement; (ii) Kullback-Leibler (KL) divergence of each child's predicted class distribution and an aggregated distribution, which captures class presence agreement; and (iii) variance of each child's predicted class distribution, which captures class presence uncertainty.
The cosine angle descriptor, is calculated in similar manner to Sabour et al. (0017). A mean projection vector, û 1/N Σ ú; is first computed using the set of child capsule projections, Ô (ū,, , U22 ..., J Then we compute a set cosine angles between each individual projection and this mean, {01, 02 ..., ON where = (u á.) lā,l).
In a similar fashion, we compute a KL divergence descriptor, b, by first creating an aggregate object class distribution. To create this aggregate distribution, follow the work of Clemen & Winkler (1999), insofar as each child capsule type is treated as an expert giving its prediction about the true underlying class distribution. First, we compute simple linear opinion pool Stone 1961), p(ž)=
EN o,(2))/N, where p=)) is the aggregated probability distribution, Z (1), 22, ...... nN } the set of child class presence projection vectors, and 02 (2))) c Ek = for j = (1, .... K}.
{1.... .N) w} the softmax function used transform projection vectors into normalized probability distributions over the K classes. Then, we measure the agreement between each child's predicted distributions, (2;), and the aggregate distribution, p()), as the KL. divergence between them b, = Σ p(žx)
Lastly, take child capsules' predicted distributions, os (z:), and compute their variance to estimate the uncertainty each child has: c, = Σx (a.(2)) - Σk (2 Our three sets of descriptors efficiently computed for all capsules simultaneously (i.e. for entire batch and across spatial locations) on GPU in parallel with matrix operations. They are then concatenated, b @ c, and fed to the excitation layers of our routing mechanism (Fig. 2).
Determining routing coefficients (excitation): The excitation stage of the SE- Routiot algorithm has the task of learning mapping from the concatenated of capsule descriptors, 8, into set

---

Under review as a conference paper at ICLR 2021
of routing coefficients for the child capsule projections. Since parents capsules types longer different classes in our formulation, but rather two separated aspects of modeling objects, we compute single set of routing coefficients each spatial location for both parents. Formally, this mapping is (W22( (122@(ie) Wi (W))), where W1 E
I & x3N W2 E R aN å is the ReLU activation function, a is the sigmoid activation function, and & is the reduction ratio used to form this mapping into two tully-connected (FC) layer bottleneck. A brief note: although excitation routing has interesting parallels to self-attention (e.g. dynamically conditioned on the input), our learned mapping non-mutuae ne--muuuall--exxleee
computed as
while self-attention and CapsNet's dynamic routing both rely on applying softmax function over outputs.
Finally, with the determined routing coefficients r = (ri: T2, .... TN): we can compute the output of the SplitCaps detection head. Projection vectors from each child to each parent are computed using the proposed deformable capsules (as described in Section 2) These projections are then combined and weighted by the routing coefficients to form the final parent capsules. These final parents contain the instantiation parameters, Vobj Σx ry rydobjli and class presence, Veta = =Ei ry rydedsje of any objects being represented at the given spatial location within the detection grid.
4 DEFORMABLE CAPSULES ON MS COCO
We evaluated our deformable capsule object detection framework on the MS COCO dataset (Lin) et al.j 20014), which contains 118K training, 5K validation and 20K hold-out testing images. Average precision (AP) is reported over all IOU thresholds and at thresholds 0.5 (AP50) and 0.75 (AP75). We followed the training procedure proposed in Zhou et al. (2019a). training 512 x 512 pixel inputs, yielding 128 x 128 detection grids, using random flip. random scaling (between 0.6 to 1.3), cropping, and color jittering data augmentation, and Adam (Kingma & Ba 2014) to optimize our objective function. Due to limited compute resources, we initialized the backhone network weights from CenterNet and only train for 40 epochs with batch size of 12 and learning rate of 5c-4 with 5x drops at 5, 15, and 25 epochs. Longer training would likely yield superior results, as found by Zhou al. (019a) who obtained better results for CenterNet when increasing from 140 to 230 epochs.
In Table we provide results of our proposed deformable capsule network with and without flip and multi-scale augmentations following Zhou al. (2019a). Inference time on our hardware (Intel Xeon E5-2687 CPU, Titan V GPU, Pytorch 1.2.0, CUDA 10.0, and CUDNN 7.6.5) was consistent with those reported by Zhou et al. 2019a) While DeformCaps performs slightly worse than CenterNet in terms of AP, it does so while producing far fewer false positive detections, as shown in Table 2 in Appendix[A.2 [A.2 For ablations, we trained a version of DeformCaps which replaces the proposed deformable capsules with the standard locally-constrained convolutional capsules (non- DeformCaps), and a version which removed the routing procedures (No-Routing). These ablations show the contribution of each component of the proposed method.
Method
Backbone
FPS
AP50
AP75
APM
APL
MaskRCNN (lle al.
2717
ResNeXt 101
11
39.8
62.3
43.4
22.1
43.2
51.2
Deform-v2 (Zh
2019
ResNet-101
67.9
50.8
27.8
49.1
59.5
SNIPER Singn el ar. 2018
DPN.98
2.5
46.1
67.0
51.6
29.6
48.9
58.1
PANet
ResNeXt-101
67.2
51.7
TridentNet [i al. 2019)
ResNet DCN
0.7
48.4
69.7
53.5
31.8
51.3
60.3
Kedmin 7
ni.,, 2018)
DarkNer-33
20
57.9
25.4
41.9
RetinaNet Lin el al. 2017
ResNeXL FPN
5.4
40.8
61.1
44.1
24.1
44.2
51.2
RelineDei Zhank
ResNet-101
36.4/4188
45.5
16.6 25.6
39.9
54.4/
CenterNet
Zhou el
2019a
DLA-34
28
/9.2
57.1 / 60.3
42.8 45,1
/9.5
/
$1.4 56.0
Deform
(Ours]
DLA-34
15
55.5
23.0
42.5
56.4
Non Delornt.pps
14
20
27.27
30.1
4187455
29.67 32.6
13.37 15.6
30.07 30.8
35.077 40.6
No-Rouling
DLA-34
15
38.9
52.7
39.1/42.4
20.9
42.0
54.4
Table 1: Single-scale / multi-scale results on COCO test-dev. Top: Two-stage detectors; Middle: One-stage detectors; Bottom: Ablation studies on deformable capsules and SF--Rutting.
We will make code publicly available for the community for reproducible research.
7

---

Under review as a conference paper at ICLR 2021
5
RELATED WORKS
5.1 CAPSULE NETWORKS
The idea of capsules was first introduced by Hinton et al. ). Sabour et al. (011)) extended this and proposed dynamic routing between capsules. The EM routing algorithm was then modified by Hinton et al. (0118) Recently, capsule networks could achieve the state-of-the-art performance for wide range of applications: video object segmentation (Duarte et al.
2019). point cloud segmentation (Zhao et al. 2019). explainable medical diagnosis (LaLonde et al., 2020 text classification (Zhao et al.
2018
sentiment analysis (Wang al. 2018). and various other applications (Vijayakumar 2019
5.2
OBJECT DETECTION
Region proposal-basee approaches: R-CNN was one of the first successful deep object detectors, which a selective search algorithm was used to select a number of region proposals, CNN features were extracted from each of the region proposals and were used to both classify the object and regress its bounding box (Girshick et al.] 2014). The later addition of Fast R-CNN (Girshick) 2015) provided end-to-end training and addressed the speed and efficiency issues of R-CNN.
Anchors-based approaches: Anchors-based approaches sample fixed-shape bounding boxes (an- chors) around low-resolution image grid, then attempt to classify anchors into object classes. Faster R-CNN (Ren et al. 2013) generates region proposals in a first stage network, then attempts to classify and regress bounding boxes for the top-k highest scoring anchors in second stage network. Later studies such as [eedmon et al. (016) dramatically speed up the process by converting the proposal classifier multi-class one-stage detector. Since then, researchers have been working on improving one-stage detectors by including shape priors (Redmon & Farhadi] 2017 2018), multiple feature
resolutions (Liu al. 2016), re-weighting the loss among different samples (Lin al. 2017). or
modeling channel-wise attention (Chen et al. 2020).
Keypoint estimation- based approaches: CornerNet (Law & Deng 2018) attempts detect objects by predicting two bounding box corners as keypoints. ExtremeNet (Zhou et al.) (019b) extends CornerNet's 's approach by estimating all corners and the center of the objects' bounding box. However, these methods rely on significantly slow combinatorial grouping post-processing stage. /houu et al. (20T9a) proposed CenterNet which attempts to predict only an objects' center point, and regress all other necessary values from there without the need for grouping or post-processing.
6 DISCUSSIONS, LIMITATIONS, & FUTURE WORK
Our proposed deformable capsules (DeformCaps) with SplitCaps object-class representations and Squeeze-and- -nde-xxttttiiiiiioonnnn inspired SE- SE-Routing algorithm represents an important step for capsule networks to scale-up to large-scale computer vision problems, such as object detection large-scale classification. Our proposed one-stage object detection capsule network is able to obtain results on MS COCO which are on-par with other state-of-the--rr one-stage CNN-based networks for the first time in the literature, while also producing fewer false positives. Examining the qualitative results, provided in Appendix AA.3] lends empirical evidence that DeformCaps can better generalize to unusual poses/viewpoints of objects than CenterNet (Zhou et al. 2019a). We hope our work will inspire future research into the considerable potential of capsule networks.
Limitations: Our study contains some limitations, discussed in greater detail in Appendix A. Briefly, (1) we had difficulty integrating the bounding box regression values into our capsule object detection head; (2) the choice of descriptors used in the squeeze is somewhat handcrafted, and is open to further investigation; (3) the choice of dimensions to model the class-agnostic instantiation parameters of objects was chosen semi-arbitrarily and could likely improve from fine-search, and (4) the choice of reconstructing objects' masks versus image patches is not thoroughly explored.
The reconstruction sub-network of DeformCaps could possibly be trained to produce a fast single-shot instance segmentation framework. At test, potentially detected objects could have their instantiation vectors reconstructed into objects' masks, then these masks would simply be resized to the predicted bounding-boow boxes, similar to He et al. (2017) but without needing to have the initial reshape and ROI alignment required in their two-stage approach.
Future directions:
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Michael A Alcom, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4845-4854, 2019.
Zhe Chen, Jing Zhang, and Dacheng Tao. Recursive context routing for object detection. International Journal of Computer Vision, pp. 1-19, 2020.
Robert T Clemen and Robert Winkler. Combining probability distributions from experts in risk analysis. Risk analysis, 19(2):187-203, 1999.
Jifeng Dai, Haozhi Qi, Yuwen Xiong. Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In The IEEE International Conference on Computer Vision (ICCV), pp. 764-773, 2017.
Kevin Duarte, Yogesh Rawat, and Mubarak Shah. Videocapsulenet: A simplified network for action detection. In Advances in Neural Information Processing Systems, pp. 7610-7619, 2018.
Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. Capsulevos: Semi-supervised video object segmentation using capsule routing. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Ross Girshick. Fast r-cnn. In The IEEE International Conference on Computer Vision (ICCV), pp. 1440-1448, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580-587, 2014.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Geoffrey Hinton, Alex Krizhevsky, and Sida Wang. Transforming auto-encoders. In International conference on artificial neural networks, pp. 14-51..priinn 2011.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In Interna- tional Conference on Learning Representations (ICLR), 2018.
Jie Hu, Li Shen, and Gang Sun. Squeezz-anddexcitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Amelia Jiménez-Sánchez, Shadi Albarqouni, and Diana Mateus. Capsule networks against medical imaging data challenges. In Intravascular Imaging and Computer Assisted Stenting and Large- Scale Annotation of Biomedical Data and Expert Label Synthesis, pp. 150-160. Springer, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2014.
Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoencoders. In Advances Neural Information Processing Systems, pp. 15486-15496. 2019.
Rodney LaLonde and Ulas Bagci. Capsules for object segmentation. In Medical Imaging with Deep Learning (MIDL), 2018
Rodney LaLonde, Drew Torigian, and Ulas Bagci. Encoding visual attributes in capsules for explainable medical diagnoses. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, 2020.
Hei Law and Jia Deng. Comernet: Detecting objects as paired keypoints. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 734-750, 2018.

---

Under review as a conference paper at ICLR 2021
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Scale- trident networks for object detection. In Proceedings of the IEEE international conference on computer vision, pp. 6054-6063, 2019.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In The IEEE International Conference on Computer Vision (ICCV), pp. 2980-2988, 2017.
Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8759-8768, 2018.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 21-37. Springer, 2016.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision (3DV), pp. 565-571. IEEE, 2016.
Inyoung Paik, Taeyeong Kwak, and Injung Kim. Capsule networks need improved routing algorithm. arXiv preprint arXiv: 13327, 2019.
Arjun Punjabi, Jonas Schmid, and Aggelos K Katsaggelos. Examining the benefits of capsule neural networks. arXiv preprint arXiv:2001.10964 2020.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition (CVPR), pp. 7263-7271, 2017.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv: 02767, 2018
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Bharat Singh, Mahyar Najibi, and Larry S Davis. Sniper: Efficient multi-scale training. In Advances in neural information processing systems, pp. 9310-9320, 2018
Mervyn Stone. The opinion pool. The Annals of Mathematical Statistics, pp. 1339-1342, 1961.
Yao-Hung Hubert Tsai, Nitish Srivastava, Hanlin Goh, and Ruslan Salakhutdinov. Capsules with inverted dot-product attention routing. In International Conference on Learning Representations (ICLR), 2020.
T Vijayakumar.
Comparative study of capsule neural network in various applications. Journal of Artificial Intelligence, 1(01):19-27, 2019.
Yequan Wang, Aixin Sun, Jialong Han, Ying Liu, and Xiaoyan Zhu, Sentiment analysis by capsules. In Proceedings of the World Wide Web Conference, pp. 1165-1174, 2018.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 24122412, 2018.
10

---

Under review as a conference paper at ICLR 2021
Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Li. Single-shot refinement neural network for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1203-4212, 2018.
Wei Zhao, Jianbo Ye, Min Yang, Zeyang Lei, Suofei Zhang, and Zhou Zhao. Investigating capsule networks with dynamic routing for text classification. arXiv preprint arXiv:1804.0055 2018.
Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3d point capsule networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1009-1018, 2019.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint arXi::1904.07855. 2019a.
Xingyi Zhou, Jiacheng Zhuo, and Philipp Krahenbuhl. Bottom-up object detection by grouping extreme and center points. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 850-859, 2019b.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9308-9316, 2019.
A APPENDIX
A.1 EXTENDED EXPLANATIONS OF LIMITATIONS AND POSSIBLE RECOMMENDATIONS
In the discussion section of the main body of our paper, we mentioned four potential limitations in our study. We would like discuss these in bit more detail here. Since many components of our method are newly introduced, there is a wide range of choices which could be investigated and improved by future researchers and engineers, and we suggest a few of those here:
(1) We had difficulty in integrating the bounding box regression values into our capsule object detection head. In our implementation, the class-agnostic capsules are trained to predict scale- normalized masks of 28 x 28. Ultimately, we would like to integrate predicting the object masks and the boxes for those masks together, as these tasks surely share mutual information. However, to the best of our knowledge, no published works exist for using capsules on real-valued regression task.
(2) For our proposed SE.-Routing. as with the original Sgneeze-and-Fxeitatie descriptors computed in the squeeze is somewhat handcrafted. We propose to use the cosine angle, KL divergence, and variance, and provide justifications for each of these choices, then allow the excitation to learn which of these pieces of information is most beneficial dynamically for each given input. Nonetheless, it is completely plausible that different descriptors could yield superior results. We unfortunately do not have the compute resources to run ablation studies over each of these chosen descriptors individually.
network, the choice of
(3) The choice of 64 dimensions to model the class-agnostic instantiation parameters was decided somewhat empirically. As we argued in the main paper, it is unlikely that all variations across object poses are completely class independent; thus, to represent these extra dimensions of variation, we increase our vector lengths considerably (16 > 64). However, it is possible that the number of class- independent and class-dependent variations significantly higher or lower than the value chosen, and largely will depend on the complexity of the data being modeled. This difficulty is analogous to determining the optimal number of convolutional filters use at every given layer of a CNN. Related to this, there is the potential for the dlass-dependent dimensions of the instantiation vectors to have unwanted influence over the cosine angle descriptors when attempting to represent objects of other classes. It could be beneficial to pass class information from the class presence capsule type over to the object instantiation capsule type to dynamically attend to the relevant dimensions of its vector for given object. In similar manner, it could be beneficial when computing the probability aggregation using the linear opinion pool to weight the expert opinions in proportion to their uncertainty instead of uniformly.
(4) We chose to reconstruct object's masks with the motivation of forcing the network to learn variations in shape, pose, and deformations. Since CNNs are known to be biased to texture information
11

---

Under review as a conference paper at ICLR 2021
over shape, chose not to explicitly supervise the learning of any texture information. Nonetheless, it is plausible that reconstructing the object with texture could yield superior performance. Further, we chose to set the value of the reconstruction regularization' 's contribution to the loss to 0.1, following what was found most beneficial by CenterNet (Zhou et al.] 2019a) for weighting the size loss contribution, and from concern not over-regularize the network early in training, then stepped this value to 2.0 half-way through training to make its value roughly equal to the other loss terms. From our experience, the accuracy remained fairly consistent across values up to 2.0 for this term, while setting its weight to 0.0 resulted in degradation of performance. We found that increasing the value during training led to faster improvements performance, consistent with other works in the literature that use such a regularization term. Engineering efforts on this parameter, such as temperature function to automatically increase this weight during training, may prove beneficial the goal is to reach the maximum possible accuracy.
A.2 ANALYSIS OF FALSE POSITIVES
DeformCaps tends to be more conservative with its detections than CenterNet. This can be observed both by the slightly lower confidence scores (typically 0.1 less than CenterNet for most detections), and by the overall fewer amount of boxes placed in scenes. CenterNet tends to produce far more false positives than DeformCaps, both in the case of incorrect detections and of multiple detections for the same object which failed to be suppressed by the NMS algorithm. Though Deform producing slightly lower confidence scores might account for some of the reduction in false positives, we observe CenterNet consistently producing fairly confident false predictions while DeformCaps does not produce a detection in the same region at all (see qualitative examples in Appendix (A.3). A quantitative analysis of this is provided in Table 2 These number are generted using the official MS COCO evaluation code in it's standard operation. However, instead of only returning the average precision (AP) ratio of true positives (TP) and false positives (FP), namely TP/ FP), also return the raw FP count as well.
AFP 50:95
AFP 50
AFP75
CenterNet
144.270
218.842
154.387
DeformCaps
139.516
209.465
149.904
Table 2: Comparison of mean average false positives (AFP) between the proposed DeformCaps and CenterNet at the standard IOU thresholds for MS COCO.
In the default operation of CenterNet, there no non-maximum suppression (NMS) operation performed. Instead, sort of pscudo-NMS is performed by passing 3 x 3 Max Pooling operation over the detection grid activations to extract objects' centers. When running CenterNet with multi- scale testing, a NMS operation is then added effectively choose which scale is the best fit for given object detection. Therefore, the false positives being seen in Appendix [A.3]ar are direct results of multiple object centers being predicted incorrectly for the same object or object centers being predicted where there are no objects. We find that DeformCaps, which predicts objects as capsule vectors and not scalar, does not suffer from these former class of FPs.
This observation of less false positive detections is consistent with what we would expect from a capsule network with dynamic routing as compared to a traditional convolutional neural network (CNN). Where a CNN passes on all activations to the next layer, capsule networks utilize a dynamic routing algorithm to only pass on activations if there is agreement amongst the child capsule projec- tions. In our proposed method specifically, with SplitCaps structure and SE- SE-Routing, the agreement computed for projections of both the pose and class of the object being represented. It follows naturally that this would limit the amount of false positive detections which are produced, by reducing the amount of activations that get passed on. Further, we find from a survey of these qualitative examples that DeformCaps is better able to detect objects when being presented in unusual pose or from an usual viewpoint than its CenterNet counterpart. This gives empirical support to one of the purported benefits of capsule networks, to be able to better generalize to unseen poses and viewpoints.
12

---

Under review as a conference paper at ICLR 2021
A.3 QUALITATIVE RESULTS ON MS COCO
Included in this section are number of qualitative examples for CenterNet (Zhou et al. 2019a) and the proposed DeformCaps the MS COCO test-dev dataset (Lin al. 2014) using both flip and multi-scale augmentations. Results for CenterNet were obtained using the official code and trained models provided by the authors (Zhou al. 2019a). While we do make any sweeping claims, we wish to comment on few general patterns that seemed to emerge in these examples. In Figure 3
we show a prototypical example of the general trend we are describing.
we include more examples of this trend. In Figures O 7 we include a set of interesting examples of unusual object viewpoints or poses being better captured by DeformCaps than by CenterNet.
Figures 4 5
Figure 3: Qualitative example for CenterNet (Zouu et al. 2019a) (leftmost), the proposed Deform- Caps (center), and the ground-truth annotations (rightmost) the MS COCO test-dev dataset. While CenterNet produces slightly higher average precision (AP) values than DeformCaps, also seems produce more false positives on average than DeformCaps.
13

---

Under review as a conference paper at ICLR 2021
orange
oronge
Figure Qualitative examples showing CenterNet (Zhou al. 2019a) (leftmost) consistently producing higher number of false positive detections than the proposed DeformCaps (center), as compared to the ground-truth annotations (rightmost) on the MS COCO test-dev dataset.
14

---

Under review as a conference paper at ICLR 2021
TRED ENOUGH ALBEA
THED ENOUGH ALDEA STOP the PORK
TRED EHONGH ALRG
STOP the PORK
STOP
the PORK
d6cL0.6
Figure Some more qualitative examples showing CenterNet (Zhou al. 2019a) (leftmost) consistently producing higher number of false positive detections than the proposed DeformCaps (center), as compared the ground-truth annotations (rightmost) the MS COCO test-dev dataset.
15

---

Under review as a conference paper at ICLR 2021
NE N
ARLO
NIE
ARLONIE
VIE ARLO
N A
personut10
Figure 6: Qualitative examples showing CenterNet Zhou et al. 2019a) (leftmost) struggling to detect objects being presented in unusual poses or from unusual viewpoints, while the proposed DeformCaps (center) successfully captures these cases, compared to the ground-truth annotations (rightmost) on the MS COCO test-dev dataset.
16

---

Under review as a conference paper at ICLR 2021
Figure 7: A few more qualitative examples showing CenterNet Zhou al. 2019a) (leftmost) struggling to detect objects being presented in anusual poses or from unusual viewpoints, while the proposed DeformCaps (center) successfully captures these cases, as compared to the ground-truth annotations (rightmost) on the MS COCO test-dev dataset.
17


Reviews:
[{"review_id": "ewg_yz1Zjx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\nIt is indeed that capsules have many promising attributes but they are not quite easy to be exploited in object detectors. This paper has addressed many problems existed when applying capsule architectures for the object detection task. In particular, deformable capsules, SplitCaps, and SE-Routing are respectively introduced to help tackle the object detection with capsules. I believe the techniques of this paper will attract interests from researchers in the corresponding areas. The writing is also good.\n\nCons:\nThere are several points that prevent me from giving a higher rating:\n1) I feel that the descriptions or presentations of introduced techniques are not quite sufficient. In particular, I am confused about the details of deformable capsules. What are the exact operations performed by the deformable capsules? Is it to make parent capsules only aggregate information from a smaller set of their children and the sampling of such a smaller set is implemented by deformable operations? I believe it will be much better to present some detailed formulations or equations to illustrate this operation. Similarly, I also recommend adding detailed equations to describe SE-Routing.  \n\n2) Without sufficient information about operations, I found that the descriptions of motivations are also quite weak, especially in the introductory part. It seems that the authors only mention the problems they will address and the techniques they proposed to address problems. There is not sufficient content briefly explaining why the proposed techniques can tackle the mentioned problems, or at least what are the advantages of the proposed techniques for tackling the problems. \n\n3) The figures also have many confusing points. For example, in Figure 1, the authors marked that solid red arrows represent 'up' but I can only find dotted red arrows in the figure. Moreover, in what place the modules inside the big blue box is implemented in the detector? After reading, I think it should represent what's inside the SplitCaps, but I do recommend the authors to add some indicative symbols to corresponding related modules. \n\n4) In addition to the reviewed literature, I found that there are some missing articles that also study how to borrow the capsule concepts for various computer vision tasks. For example:\n[1] Vijayakumar T. Comparative study of capsule neural network in various applications[J]. Journal of Artificial Intelligence, 2019, 1(01): 19-27.\n[2] Zhao Y, Birdal T, Deng H, et al. 3D point capsule networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2019: 1009-1018.\n[3] Chen Z, Zhang J, Tao D. Recursive Context Routing for Object Detection[J]. International Journal of Computer Vision, 2020: 1-19.\n\nOverall, I would like to have some feedback from the authors regarding the above issues before making my final decision. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "VOekTEqKm3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use the capsules to perform object detection on COCO. Capsules, while showing promises, are usually too expensive for tasks beyond MNIST and Cifar. The authors propose three key improvements in DeformCaps, SplitCaps and SE-Routing to improve the efficiency and therefore allow capsules to be applied on larger tasks such as object detection. The authors claim novelties in:\n\n-- First ever capsule-based object detection on COCO.\n\n-- Deformable capsules that let parent to samples child capsules instead of having fixed connections to improve efficiency.\n\n-- SplitCaps: a reparametrization trick to reduce the number of capsules needed.\n\n-- Squeeze-and-excitation routing that finds agreement without iterative loop.\n\nI agree with the authors' novelty claims.\n\nStrengths:\n\n-- Impressive results: Capsule Networks are well known to be computationally expensive. Getting it to work on large images is certainly no easy task. There has been other works operating on larger images, as noted by this paper, but they usually cannot achieve quality on par with CNNs. Having said that, I need to note that the SOTA for COCO has consistently improved, and is at ~51% AP [a], quite better than the numbers listed in Table 1.\n\n-- Good novelty: DeformCapsule and SplitCaps both follow a straightforward pattern to dissect prohibitively large tensor predictions into smaller ones. And together with the SE-routing methods, the novelty should be sufficient for this venue.\n\n-- Clear writing. Figure 1 and 2 helped a lot with the understanding of the key components as well.\n\n[a]: SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization\n\nWeaknesses:\n\n-- Insufficient experimentation: There is only a single experimental table that contains both comparisons to the SOTA and ablations to DeformCaps and SE-Routing. It would be nice to see additional results, be it on another dataset / task (e.g. sem seg), or even additional ablations. For example, one premise of capsules was the requirement of less training data. Is it true here as well? \n\nConclusion:\n\nI'd recommend acceptance because of the strong results and good novelty. The only major weakness is the lack of additional ablations to highlight the advantages (if not on AP) over CNNs. Nonetheless, I believe this work is a good step towards more interests and advances in capsules, and will be of interests to the audience of this venue.\n\nPost-rebuttal:\nI concur with R1 in that the results look poor when taking into the account how close the proposed method is compared to CenterNet and that it was finetuned from a CenterNet. Therefore, I'll lower my rating to 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-30Rk_cNJ32", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### summary\nThis paper introduces capsule network for object detection. To solve the issue of capsule network when applied to large-scale detection problems, this paper develops deformable capsules, a new  prediction head SplitCaps, and a dynamic routing algorithm, SE-Routing. Experiments are conducted on COCO where it performs slightly worse than the baselines but arguably predicts less false positives.\n\n### pros\nI appreciate the spirit of developing new architectures for a highly-competitive task like object detection. It's totally acceptable that the performance cannot fully match the SOTA results in this kind of scenario. This paper is insightly as it studies several important issues of capsule network when being applied to large-scale visual tasks. The proposed solutions have at least make the framework run-able on standard hardware.\n\n### cons\n1. My current rating is not purely based on the performance, but it is one of the major concerns here. The proposed framework (1) performs worse than baseline -- CenterNet, and (2) performs much worse then the SOTA solutions (Tab.1) in terms of both performance and speed. Note that the network weights are initialized from CenterNet. \n2. The whole analysis in Appendix A.2 reads very confusing for several reasons:\n * CenterNet achieves higher AP (TP/(TP+FP)) and also high FP, which is a quite common thing to me. Ignoring the ratio and simply compare the absolute value of FP doesn't seems to be the correct thing to do.\n * Some visualizations are weird like Fig.4 human and dog. Why NMS cannot remove such highly-redundant predictions? It seems more like a bug to me.\n * On page 12 top row it writes \"CenterNet consistently producing fairly confident false predictions (e.g. > 0.4)\", how does this threshold 0.4 is chosen? It's not a standard thing and in fact if we improve it to 0.5 or higher, most of the FPs about CenterNet will disappear. Moreover, is this how FP gets counted? If yes, I think this is problematic as the total number (TP+FP) is changing across methods. A more fair way is to choose top-k (k=100 in coco) predictions and compute the metrics.\n3. Lots of reference are missing. For example, Tab.1 compares multiple different methods but most of them are not cited. The related work only cites few very classical papers, but most recent works are missing.\n4. The writing, especially the approach section, isn't very clear. I had a relatively hard time to understand how do each modules work, and what are the motivations behind these design choices. \n5. Moreover, there are couple of new things proposed, and each one of them have specific design choices and parameters. However, there are no ablation studies properly studying them. As a results I don't understand which module is working, and how does it work.\n\n### questions\n1. How do the FPS numbers in Tab.1 get computed? Are they simply borrowed from original paper or some published work? Some of them look strange to me. \n2. Why KL in Sec.3.2? Does the asymmetric natural of KL influence the results?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes capsule networks with deformable capsules for tackling object detection. All reviewers agreed that object detection is an important problem that is interesting to the ICLR community. Reviewers also agree that the proposed approach is novel and interesting, and in particular they mention that proposing an efficient capsule network for detection is non-trivial. On the less positive side, during the discussion phase all reviewers had concerns about the weak experimental validation, particularly missing ablation studies to analyse the effectiveness of their contributions. At the end, all reviewers felt that, while this is a promising direction of research for object detection, the experimentation should be improved.
Decision: Reject