Capsules with Inverted Dot-Product Attention Routing
Authors: Yao-Hung Hubert Tsai
Nitish Srivastava
Hanlin Goh
Ruslan Salakhutdinov
Venue:  (2020)
Keywords: capsule networks
routing
attention

Abstract: We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote. 
The new mechanism 1) designs routing via inverted dot-product attention; 2) imposes Layer Normalization as normalization; and 3) replaces sequential iterative routing with concurrent iterative routing.
When compared to previously proposed routing algorithms, our method improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100, and it performs at-par with a powerful CNN (ResNet-18) with 4x fewer parameters.
On a different task of recognizing digits from overlayed digit images, the proposed capsule model performs favorably against CNNs given the same number of layers and neurons per layer.  We believe that our work raises the possibility of applying capsule networks to complex real-world tasks.

Published as a conference paper at ICLR 2020
CAPSULES WITH INVERTED DOT-PRODUCT ATTENTION ROUTING
Yao-Hung Hubert Tsai Nitishh Srivastava', Hanlin Goh', Ruslan Salakhutdinov Apple Inc., "Carnegie Mellon University
ya yaohungt&css cmu, edu, (nitish.srivastave srivastava, hanlin, rea.kakutccccoooyaappp......
ABSTRACT
We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to parent based only agreement between the par- ent's state and the child's vote. The new mechanism 1) designs routing via inverted dot-product attention; 2) imposes Layer Normalization normal- ization; and 3) replaces sequential iterative routing with concurrent iterative routing. When compared to previously proposed routing algorithms, method improves performance on benchmark datasets such as CIFAR-I0 and CIFAR-I00, and it performs al-par with powerful CNN (ResNet-18) with 4x fewer parameters. On a different task of recognizing digits from over- layed digit images, the proposed capsule model performs favorably against CNNs given the same number of layers and neurons per layer. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks. Our code is publicly available at: /https://gitee com/ hatpa/ppleemmmmmm htts/ullcappsessss oereeleeeeeeeeeeeettteeeeeoooooo rout An alternative implementation is available at: https://gittub.cr com/
attentien
master/READML.C
INTRODUCTION
Capsule Networks (CapsNets) represent visual features using groups of neurons. Each group (called a "capsule") encodes a feature and represents one visual entity. Grouping all the information about one entity into one computational unit makes it easy to incorporate priors such as "a part can belong to only one whole" by routing the entire part capsule to its parent whole capsule. Routing is mutually exclusive among parents, which ensures that one part cannot belong to multiple parents. Therefore, capsule routing has the potential to produce an interpretable hierarchical parsing of a visual scene. Such structure is hard to impose in typical convolutional neural network (CNN). This hierarchical relationship modeling has spurred a lot of interest in designing capsules and their routing algorithms
(Sabour et al., 2017: Hinton et al. 2018, Wang & Liu 2018 Zhang et al. 2018
Li et al. 2018
Rajasegaran et al. 2019 Kosiorek et al. 2019).
In order to do routing, each lower-level capsule votes for the state of each higher-level capsule. The higher-level (parent) capsule aggregates the votes, updates its state, and uses the updated state to explain each lower-level capsule. The ones that are well-explained end up routing more towards that parent. This process is repeated, with the vote aggregation step taking into account the extent which part routed that parent. Therefore, the states of the hidden units and the routing probabilities are inferred in an iterative way, analogous to the M-step and E-step, respectively. of an Expectation-Mo -aairiiizaaiiiiiiionnnn (EM) algorithm. Dynamic Routing (Sabour et al.) (017) and EM- routing (Hinton al. 2018) 8) can both be
variants of this scheme that share the basic iterative structure but differ in terms of details, such as their capsule design, how the votes are aggregated, and whether non-linearity is used.
We introduce a novel routing algorithm, which we called Inverted Dot- Dot-Product Attention Routing. In method, the routing procedure resembles an inverted attention mechanism, where dot products are used to measure agreement. Specifically, the higher-level (parent) units compete for the attention of the lower-level (child) units, instead of the other way around, which is commonly used in attention

---

Published as a conference paper at ICLR 2020
Convolutional
Convolution
Backbone
LayerNorm
Routing
Routing
Classifier
Convolutional Features
Capaules
Primary
Convolutional
Capsules
Fully-Connected
Capsules
Capsules
Class
Logits
Figure Illustration Capsule network with backhone block, 3 convolutional capsule layers, fully- connected capsule layers, and a classifier. The first convolutional capsule layer is called the primary capsule layer. The last colle-connected capsule layer is called the class capsule layer.
models. Hence, the routing probability directly depends on the agreement between the parent's pose (from the previous iteration step) and the child's vote for the parent's pose (in the current iteration step). We also propose two modifications for our routing procedure - (1) using Layer Normalization (Ba al., 2016) normalization, and (2) doing inference of the latent capsule states and routing probabilities jointly across multiple capsule layers (instead of doing it layer-wise). These modifications help scale up the model to more challenging datasets.
Our model achieves comparable performance as the state-of saee--ffthheeeee ff-the-art convolutional neural networks (CNNs), but with much fewer parameters, on CIFAR-10 (95.14% 14% test accuracy) and CIFAR-100 (78.02% test accuracy). We also introduce challenging task to recognize single and multiple over- lapping objects simultaneously. To be more precise, we construct the DiverseMultiMNIS1 dataset that contains both single-digit and overlapping-dight the same number neurons per layer, the proposed CapsNet has better convergence than a baseline CNN. Overall, argue that with the proposed routing mechanism, it is no longer impractical to apply CapsNets on real-world tasks. We will release the source code to reproduce the experiments.
images. With the same number of layers and
2 CAPSULE NETWORK ARCHITECTURE
An example of our proposed architecture is shown in Figure The backbone is a standard feed- forward convolutional neural network. The features extracted from this network are fed through another convolutional layer. At each spatial location, groups of 16 channels are made to create capsules (we assume a 16-dimensional pose in a capsule). LayerNorm is then applied across the 16 channels to obtain the primary capsules. This is followed by two convolutional capsule layers, and then by two fully-connected capsule layers. In the last capsule layer, each capsule corresponds to class. These capsules then used to compute logits that feed into softmax to computed the classification probabilities. Inference in this network requires a feed-forward pass up to the primary capsules. After this, our proposed routing mechanism (discussed in the next section) takes over.
In prior work, each capsule has a pose and some way of representing an activation probability. In Dynamic Routing CapsNets (Sabour al. 2017), the pose is represented by vector and the activa- tion probability is implicitly represented by the norm of the pose. In EM Routing CapsNets (Hinton) al. 2018 the pose is represented by a matrix and the activation probability is determined by the EM algorithm. In our work, we consider a matrix-structured pose in a capsule. We denote the capsules layer as ph and the i-th capsule layer as p! The pose p Rdr. in vector form and will be reshaped to IR v V when representing it as a matrix, where d, is the number of hidden units grouped together to make capsules in layer L. The activation probability is not plicitly represented. By doing this, we are essentially asking the network to represent the absence of a capsule by some special value of its pose.
INVERTED DOT-PRODUCT ATTENTION ROUTING
The proposed routing process consists of two steps. The first step computes the agreement between lower-level capsules and higher-level capsules. The second step updates the pose of the higher-level capsules.
Step 1: Computing Agreement: To determine how capsule j in layer + (p/+1) agrees with capsule in layer L. (p(), we first transform the pose po the vote v for the pose p L+1 This
2

---

Published as a conference paper at ICLR 2020
dot produuct
Highar-leval Capsules
pos
routing coofficietss
Laveroore
Soltmee:
weight
reshipp
Lower-level Capsules
agreement
muttiplicat
Dat-Product Attention
multiplication
Routing Coefficiants Normalized Agreement
Pose Update
Figure 2: Illustration of the Inverted Dot-Product Attention Routing with the pose admitting matrix structure.
Procedure Inverted Dot-product Attention Routing algorithm returns updated poses the cap- sules in layer L given poses in layer and L + and weights between layer L and L + 1.
1: procedure INVERTED DOT-PRODUCE ATTENTION ROUTING pL+1 W
2: for all capsule in layer and capsule j in layer (L + 1): vL
ff f w.
1j P
P vote
3: for all capsule in layer and capsule j in layer (L + 1): oi
+ Pj
L+1
D agreement routing coefficient
4: for all capsule in layer L: r exp(al)) / Eg exp(att))
5: for all capsule in layer (L + 1): Pj [++1 + E, riy
pose update normalization
6: for all capsule in layer (L + 1): Pj 1+1 + IayerNorm(p, /
7: return PL+1
transformation is done using learned transformation matrix W
WL P
(1)
where the matrix WL Rdd41xdd. if the pose has vector structure and WL E R ij
iy
(requires d=+1 = dz) if the pose has a matrix structure. Next, the agreement (a is computed by the dot-product similarity between pose pj [++ and vote v.y-
of = I+IT
The pose Pj
[++1 is obtained from the previous iteration of this procedure, and will be set to 0 initially.
Step 2: Computing Poses: The agreement scores a! are passed through softmax function to determine the routing probabilities r
exp(at)) Eg exp(aty))
(3)
where r is an inverted attention representing how higher-level capsules compete for attention of lower-level capsules. Using the routing probabilities, we update the pose p, I++1 for capsule j in layer L + from all capsules in layer L:
p / = LayerNorm
Σ Entivs
We adopt Layer Normalization (Ba et al. 2016) the normalization, which empirically find it to be able to improve the convergence for routing. The routing algorithm is summarized in Proce-
dure D and Figure 2
INFERENCE AND LEARNING
To explain how inference and learning are performed, we use Figure | as an example. Note that the choice of the backbone, the number of capsules layers, the number of capsules per layer, the design of the classifier may vary for different sets of experiments. We leave the discussions of configurations in Sections and 6 and in the Appendix.

---

Published as a conference paper at ICLR 2020
Procedure 2 Inference. Inference returns class logits given input images and parameters for the model. Capsule layer denotes the primary capsules layer and layer N denotes the class capsules layer.
1: procedure INFERENCE(I: 0,
1 Pre-Capsules Layers: backbone features extraction */
2: F + backbone(I: ())
backhone feature
1 Capsules Layers: initialization
3: pl f- -Layeroorm (convolution(: 0))
primary capsules
4: for L in layers 2 to N: P 0s
non-primary capsules
1e Capsules Layers (1sf [eraation:: sequential routing */
5: for L in layers 1 to (N
do
6: p4+1 + Routing (P',tt,, W
> non non-primary capsules
1 Capsules Layers (2nd to tth Iteration): concurrent routing %/
7:
for (t - 1) iterations do for L in layers 1 to (N - 1): 1/+1 + Routing , ,p/+, w for in layers 2 to N: pL p
8:
9:
b non-primary capsules
1st Post- Cast-Capuules Layers: classification */ for all capsule in layer N: h. t classificr(p; :p,":) return y
class logits
11:
4.1 INFERENCE
For ease exposition, we decompose a CapsNet into pre-capsule, capsule and post-capsule layers.
The goal is to obtain a backbone feature F from the input image I. The backbone model can be either single convolutional layer or ResNet computational blocks (He
Pre-Capsule Layers:
et al. 2016
The primary capsules P are computed by applying a convolution layer and Layer Normalization to the backbone feature F. The non-primary capsules layers p2:N are ini- tialized to be zeros] For the first iteration, we perform one step of routing sequentially in each capsule layer. In other words, the primary capsules are used to update their parent convolutional capsules, which are then used to update the next higher-level capsule layer, and so on. After doing this first pass, the rest of the routing iterations are performed concurrently. Specifically, all capsule layers look at their preceding lower-level capsule layer and perform one step of routing simultane- ously. This procedure is example of parallel-in-time inference method. We call it "concurrent routing" as it concurrently performs routing between capsules layers per iteration, leading to bet- ter parallelism. Figure illustrates this procedure from routing iteration 2 to l. It worth noting that, our proposed variant of CapsNet is a weight-tied concurrent routing architecture with Layer Normalization, which Bai et al. (2019) empirically showed could converge fixed points.
Capsule Layers:
Previous CapsNets (Sabour et al.) 2017: Hinton et al. 2018) used sequential layer-wise iterative routing between the capsules layers. For example, the model first performs routing between layer L --1 1 and layer L for few iterations. Next, the model performs routing between layer L and L+1 1 for few iterations. When unrolled, this sequential iterative routing defines very deep computational graph with a single path going from the inputs to the outputs. This deep graph could lead to a vanishing gradients problem and limit the depth of CapsNet that can trained well, especially if any squashing non-linearities are present. With concurrent routing, the training can be made more stable, since each iteration has a more cumulative effect.
The goal is to obtain the predicted class logits ý from the last capsule layer (the class capsules) PN, In our CapsNet, we use a linear classifier for class i in class capsules: ÿ: classificr(p.") This classifier is shared across all the class capsules.
Post-Capsie Layers:
As compared to 0 initialization, we observe that a random initialization leads to similar converged perfor- mance but slower convergence speed. We also tried to learn biases for capsules' initialization, which results in similar converged performance and same convergence speed. As summary, we initialize the capsule's 's value 0 for simplicity.
4

---

Published as a conference paper at ICLR 2020
(Initaalized Zeros)
(Initialized Zeros) (Initiaiized to Zeros) (Initilized Zeros)
Convolutional Features
Primary Copsules
Convolutiona Capsules
Consclutional Capsules
Fuly- Connected Capsules
Closs Capsules
copy
routing
routing
routing
routing
Iteration
Primary Capsules
Comvolutional Capsules
Convolutional Capsulcs
Fuly- Connected Capsules
Cless
copy
Ieeaaion
Primary Capsules
Fuly- Connacted Capsules
Capsules
Capsules
Capsules
Fouting
Fouting
routing
Primary
Convolutionll Capsullio
Convolutional Capsulis
Cless Capaules
Logits
Capsullos
Figure 3: Illustration the proposed concurrent routing from iteration with example Figure The concurrent routing is a parallel-in intiiee-i-mmee routing procedure for all capsules layers.
Table 1: Classification results on CIFAR-IO/CIFAR 100 without ensembling models. We report the best performance for CapsNets when considering to 5 routing iterations. We report the performance from the best test model for baseline routing methods, routing method, and ResNet (He et al. 2016).
Test Accuracy (# of parameters)
Method
Backbone
CIFAR-10
CIFAR-100 100
Dynamic Routing (Sabour et al., 2017
simple simple
84.08% (7.99M)
56.96% (31,59 31155MM)
EM Routing (Hinton el al 2018
82.19 (0.45M)
37.73% (0.50M)
Inverted Dot- Product Attention Roulig (ours)
simple
85.17 (0.56M)
57.32% (1.46M)
Dynamic Routing (Sabour et al. 2017
ResNet
92.65% (12.45M)
71.70% (36.04M)
EM Routing (Hinton 2018
ResNet
92.15% (L.7IM)
58.08% (1.76M)
Inverted Dot- Product Attention Rouing (ours)
ResNet
95. 14% (1.8M)
78.02% (2.80M)
Baseline CNN (simple)
87.10% (18.92M)
62.30% (19.01M)
ResNet-18 (He et al. 2016
95.11% (11.17M
77.92% (11.22M)
4.2 LEARNING
We update the parameters 0, W1:N-1 by stochastic gradient descent. For multiclass classification, we use multiclass cross-entropy loss. For multilabel classification, we use binary cross-entropy loss. We also tried Margin loss and Spread loss which are introduced by prior work (Sabour et al.) 2017 Hinton et al. 2018). However, these losses do not give us better performance against cross cross-yntropy and binary cross-entropy losses.
4.3 COMPARISONS WITH EXISTING CAPSNET MODELS
Having described our model in detail, we can now place the model in the context of previous work. In the following table, we list the major differences among different variants of CapsNets.
Dynamic Routing Sabour et al. 2017
EM Routing
Inverted Dot-Prode Product Attention Routing
Hiinton et al., 20188)
(ours)
Routing
sequential iterative routing
sequential iterative routing
concurrent iterative routing
Poses
vector n/a (norm of poses)
matrix determined by EM n/a
matrix
Activations
n/a
Non-lincarity
Squash function n/a
Normalization
n/a
Layer Normalization
Loss Function
Margin
Spread loss
Cross Entropy/Binnary Cross Entropy
EXPERIMENTS ON CIFAR- 10 AND CIFAR-100
CIFAR-10 and CIFAR- 100 datasets (Krizhevsky et al. 2009) consist of small 32 x 32 real-world color images with 50, 000 for training and 10, 000 for evaluation. CIFAR-10 has 10 classes, and CIFAR- 100 has 100 classes. We choose these natural image datasets to demonstrate our method since they correspond more complex data distribution than digit images.
5

---

Published as a conference paper at ICLR 2020
Dynemic Routing
BM Routing
Inverted Dot-Product Attention Routing
80
Test Accuracy
Tea Acturacy
Test Accuracy
iterationts)
fer ationis]
6D
iterationis) 3 Iteration(s)
40-
2 ioorationss
Itor Iteration(s)
3 ibor (on)) iefiie
40
iteration(s)
20-
Iborationiss
sssttionist
attontttt
20
iterationts)
100
200 of Epochs
330
100
200
300
100
200 Epochs
300
of Epochs
Routing Method
Text Accuracy for Various Routing lterationss
Inverted Dot-Product Attention Routing
2
std
400
220
Dymamiic Subvur et al
2017
84.08%
83. 505
62.88%
82. 12.901%
82.11%
0.64%
EM [ihior er x 2018
58.08%
82.19%
78.11%
46.115%
31.41%
19.20%
900
905
Inverted Dor ProdPrdue Allesomm-A
66.83%
65.83%
66 22%
77.03%
10.00%.
23.96/%
lovested Det. Product Attenton-B B5
84.24%
83.35
50.21%
83.57%
82.07%
1.40%
lnvested ArediitAteeiionn
82.55%
63.12%
82.38%
81.97%
leverted Det-Prodict Attemion
84.24%
85.05%
84.83%
85.17%
85.095
0.34%
Minenry (SB)
Thiiing Time (oo)
Exsluation Time
Figure 4: Convergence analysis for CapsNets on CIFAR-I0 with simple backbone model. Top: convergence
plots for different routing mechanisms. Bottom left: classification results with respect to different routing iterations. Inverted Dot- Product Attention-A denotes routing approach without Layer Normalization. In- verted Dot-Product Attention-B denotes our routing approach with sequential routing. Inverted Dot-Produce Attention-C C denotes our routing approach with activations in capsules. indicates a uniform prediction. Bot- tom right: memory usage and inference time for the proposed Inverted Dot-Produce Attention Routing. For fairness, the numbers are benchmarked using the same 8-GPU machine with hatch size 128. Note that for fu- ture comparisons, we refer the readers to an alternative implementation for model: https: //tttpa:///ttii //github. com/ wwtpyyycchhhhcccc Sttphuuunnt//pppuleeeeee Capsulesss Inverted eertted-tttte Aetentionn Rout uses much less memory, has a significantly faster inference speed, and retains the same performance.
www.://aaaeeeeeeeeeeeeeeeeeeeeee SarestteeRRRDMEEE /READMEEE....... .mal
Comparisons with other CapsNets and CNNs: In Table I we report the test accuracy obtained by our model, along with other CapsNets and CNNs. Two prior CapsNets are chosen: Dynamic Routing CapsNets (Sabour et al. 2017) and EM Routing CapsNets (Hinton et al.) 2018). For each CapsNet, we apply two backbone feature models: simple convolution followed by ReLU nonlinear activation and a ResNet (He et al.] 2016) backbone. For CNNs, we consider a haseline CNN with 3 convolutional layers followed by fully-connected classifier layer. ResNet-18 is selected as a representative of SOTA CNNs. See Appendix A. for detailed configurations.
First, we compare previous routing approaches against ours. In a general trend, the proposed Cap- sNets perform better than the Dynamic Routing CapsNets, and the Dynamic Routing CapsNets perform better than EM Routing CapsNets. The performance differs more on CIFAR-100 than on CIFAR- 10. For example, with simple convolutional backbone, EM Routing CapsNet can only achieve 37.73% test accuracy while ours can achieve 57.32%. Additionally, for all CapsNets, we see improved performance when replacing a single convolutional backbone with ResNet backbone. This result is not surprising since ResNet structure has better generalizability than a single convo- lutional layer. For the number of parameters, ours and EM Routing CapsNets have much fewer as compared to Dynamic Routing CapsNets. The reason is due to different structures of capsule's 's pose. Ours and EM Routing CapsNets have matrix-structur poses, and Dynamic Routing CapsNets have vector-structure poses. With matrix structure, weights between capsules are only O(d) with d being pose's ss dimension; with vector structure, weights are ( To conclude, combining the proposed Inverted Dot-Product Attention Routing with ResNet backbone gives us both the advantages of a low number of parameters and high performance.
Second, we discuss the performance difference between CNNs and CapsNets. We see that, with a simple backbone (a single convolutional layer), it is hard for CapsNets to reach the same per- formance as CNNs. For instance. our routing approach can only achieve 57. 57.32% test accuracy on CIFAR-108 while the baseline CNN achieves 62.30%. However, with a SOTA backbone structure (ResNet backbone), the proposed routing approach can reach competitive performance 195.14% on CIFAR-10) as compared to the SOTA CNN model (ResNet- 18 with 99.11%........ on CIF.R-10).
In Figure 4 top row, we analyze the convergence for CapsNets with respect to the number of routing iterations. The optimization hyperparameters are chosen optimally for each routing mechanism. For Dynamic Routing CapsNets (Sabour et al. 2017), we observe a mild performance drop when the number of iterations increases. For EM Routing CapsNets (Hinton et al,, 2018), the best-performed number of iterations is 2. Increasing decreasing this number severely hurts the performance. For our proposed routing mechanism, we find a positive correlation between performance and number of routing iterations. The performance variance is also the smallest among
Convergence Analysis:

---

Published as a conference paper at ICLR 2020
Deselin DMV va Capabat
Drewkiin ENN
Method
Pose Stracture
Test Ace.
paramss.
Capshic (Voetar 2 rouding horations) CapsNet (hor 2 routing fiorotions
CopsNet'
vector
83.39%
42.4SM
CapsNet
80.59%
9.96M
CapsNet
voctor
85.74%
42.18M
Best Chh
Baseline CNN
79.8155
19,55M
Casslett with natii poses
Capeliets with vecter peeas
22.5
360
263
505
Steps 60,000 traiing samples)
Time
Figure 5: Table and convergence plot for baseline CNN and CapsNets with different pose structures. We consider the same optimizer, the same number of layers, and the same number of neurons per layer for the models. For fairness, memory usage/ inference time are benchmarked using the same 8-GPU machine with batch size 128. CapsNet" denotes the Dynamic routing method and CapsNet denotes proposed Inverted Dot-Product Attention Routing method. Note that for future comparisons, we re- fer the readers an altemative implementation for our model: [bttps: //gips:///tte thub , com/ yaohungt// mdl It uses much less mem- ory, has a significantly faster inference speed, and retains the same performance. The memory usage and inference time in this implementation now are only marginally higher than the baseline CNN.
[Cassulcs Invernee-dd---e At tention Rout i inggb/// haig///bbb... ma www.tte///EaA....
the three routing mechanisms. This result suggests our approach has better optimization and stable inference. However, selecting a larger iteration number may not be ideal since memory usage and inference time will also increase (shown in the bottom right in Figure 4p. Note that, we observe sharp performance jitters during training when the model has not converged (especially when the number of iterations is high). This phenomenon is due to applying LayerNorm on a cow-dimeniional vector. The jittering is reduced when we increase the pose dimension in capsules.
Ablation Study: Furthermore, we inspect our routing approach with the following ablations: 1) In- verted Dot-Product Attention-A: without Layer Normalization; 2) Inverted Dot-Product Attention- B: replacing concurrent to sequential iterative routing; and 3) Inverted Dot-Product Attention-C: adding activations in capsules The results presented in Figure bottom row. When remov- ing Layer Normalization, performance dramatically drops from our routing mechanism. Notably, the prediction becomes uniform when the iteration number increases to 5. This result implies that the normalization step is crucial to the stability of our method. When replacing concurrent with sequential iterative routing, the positive correlation between performance and iteration number no longer exists. This fact happens in the Dynamic Routing CapsNet as well, which also uses sequential iterative routing. When adding activations to our capsule design, we obtain a performance deterio- ration. Typically, squashing activations such as sigmoids make it harder for gradients to flow, which might explain this. Discovering the best strategy to incorporate activations capsule networks an interesting direction for future work.
6 EXPERIMENTS ON DIVERSEMULTIMNIST
The goal in this section is to compare CapsNets and CNNs when they have the same number of layers and the same number of neurons per layer. Specifically, we would like to examine the dif- ference of the representation power between the routing mechanism (in CapsNets) and the pooling operation (in CNNs). A challenging setting is considered in which objects may be overlapping with each other, and there may be diverse number of objects in the image. To this end, we construct the DiverseMultiMNIST dataset which is extended from MNIST (LeCun et al. 1998). and it contains both mingle-digit and two overlapping digit images. The task will be multilabel classification, where the prediction is said to be correct if and only if the recognized digits match all the digits in the image. We plot the convergence curve when the model is trained 21 M images from DiverseMul- tiMNIST. Please see Appendix B.2 for more details on the dataset and Appendix B.1 for detailed model configurations. The results are reported in Figure
First, we compare our routing method against the Dynamic routing one. We observe an improved performance from the CapsNet* the CapsNet (83.39% to 85. 74% with vector-structuree poses). The result suggests a better viewpoint generalization for our routing mechanism.
Second, we compare baseline CNN against our CapsNet. From the table, we see that CapsNet has better test accuracy compared to CNN. For example, the CapsNet with vector-structured poses reaches 85. 75%74% test accuracy, and the baseline CNN reaches 79.81% test accuracy. In our CNN implementation, we use average pooling from the last convolutional layer to its next fully-connected
We consider the same kind of capsules activations as described in EM Routing CapsNets Hinton et al.
2018
7

---

Published as a conference paper at ICLR 2020
layer. We can that having a routing mechanism works better than pooling. However, one may argue that the pooling operations requires no extra parameter but routing mechanism does, and hence it may not be fair to compare their performance. To address this issue, in the baseline CNN, we replace the pooling operation with a fully-connected operation. To be more precise, instead of using average pooling, we learn the entire transformation matrix from the last convolutional layer to its next fully-connected layer. This procedure can be regarded as considering pooling with learnable parameters. After doing this, the number of parameters in CNN increases to 42.49.,, and the corresponding test accuracy is 84.84%, which is still lower than 85. 74% from the CapsNet. We conclude that, when recognizing overlapping and diverse number of objects, the routing mechanism has better representation power against the pooling operation.
Last, we compare CapsNet with different pose structures. The CapsNet with vector-structured poses works better than the CapsNet with -ttuii--tr poses (80.59% vs 85.74%). However, the former requires more parameters, more memory usage, and more inference time. If we increase the number of parameters in the matrix-pose CapsNet to 42M, its test accuracy rises to 91.17%. Nevertheless, the model now requires more memory usage and inference time as compared to using sector-structured poses. We conclude that more performance can be extracted from vector-structured poses but at the cost of high memory usage and inference time.
7 RELATED WORK
The idea of grouping a set of neurons into a capsule was first proposed in Transforming Auto- Encoders (Hinton et al., 2011). The capsule represented the multi-scale recognized fragments of the input images. Given the transformation matrix, Transforming Auto-Encoders learned to dis-
cover capsules' instantiation parameters from an affine- tranfforrdd image pair. [Sabour et al. (2017)
extended this idea to learn part-whole relationships in images systematically. Hinton et al. (2018) cast the routing mechanism as fitting a mixture of Gaussians. The model demonstrated an im- pressive ability for recognizing objects from novel viewpoints. Recently, Stacked Capsule Auto- Encoders (Kosiorek al., (019) proposed to segment and compose the image fragments without any supervision. The work achieved SOTA results on unsupervised classification. However, despite showing promising applications by leveraging inherent structures in images, the current literature on capsule networks has only been applied on datasets of limited complexity. Our proposed new routing mechanism instead attempts to apply capsule networks to more complex data.
Our model also relates to Transformers (Vaswani et al. 2017) and Set Transformers (Lee et al., 2019), where dot-product attention is also used. In the language of capsules, a Set Transformer can be seen as model in which higher-level unit can choose to pay attention to K lower-level units (using K attention heads). Our model inverts the attention direction (lower-level units "attend" to parents), enforces exclusivity among routing to parents and does not impose any limits on how many lower-level units can routed to any parent. Therefore, it combines the ease and parallelism of -oo-prrduct routing derived from a Transformer, with the interpretability of building a hierarchical parsing of a scene derived from capsule networks.
There are other works presenting different routing mechanisms for capsules. Wang & Liu (2018) formulated the Dynamic routing (Sabour et al., 2017) as an optimization problem consisting of a clustering loss and KL regularization term. Zhhng al. (2018) generalized the routing method within the framework of weighted kernel density estimation. Li et al. (2018) approximated the routing process with two branches and minimized the distributions between capsules layers by an optimal transport divergence constraint. [Phaye et al.] (2188) replaced standard convolutional struc- tures before capsules layers by densely connected convolutions. It is worth noting that this work was the first to combine SOTA CNN backbones with capsules layers. RRajasegaran et al. (2019) pro- posed DeepCaps by stacking 10+ capsules layers. achieved 92.74% test accuracy on CIFAR-10, which was the previous best for capsule networks. Instead of looking for agreement between cap- sules layers, nceers,Choiie et al. (2019) proposed to learn deterministic attention scores only from lower-level capsules. Nevertheless, without agreement, their best-performed model achieved only 88.61% test accuracy on CIFAR-10. 10. In contrast to these prior work, we present a combination of inverted dot- product attention routing, layer normalization, and concurrent routing. To the best of our knowledge, we are the first to show that capsule networks can achieve comparable performance against SOTA CNNs. In particular, achieve 95.1 14% test accuracy for CIFAR-10 and 78.02% for CIFAR-100.
8

---

Published as a conference paper at ICLR 2020
8 CONCLUSION AND FUTURE WORK
In this work, we propose novel Inverted Dot-Product Attention Routing algorithm for Capsule networks. Our method directly determines the routing probability by the agreements between parent and child capsules. Routing algorithms from prior work require child capsules to be explained by parent capsules. By removing this constraint, we are able to achieve competitive performance against SOTA CNN architectures
CIFAR-10 and CIFAR-100 with the use of low number of parameters. We believe that it is no longer impractical to apply capsule networks to datasets with complex data distribution. Two future directions can be extended from this paper:
In the experiments, we show how capsules layers can be combined with SOTA CNN back- bones. The optimal combinations between SOTA CNN structures and capsules layers may be the key to scale up to much larger dataset such as ImageNet.
The proposed concurrent routing is parallel-in-time and weight-tied inference process.
The strong connection with Deep Equilibrium Models (Bai et al. 2019) can potentially lead infinite-iteration routing.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint (Xx7:/607.064 06450, 2016.
Shaojie Bai. J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Neural Information Processing Systems (NeurIPS), 2019.
Jaewoong Choi, Hyun Seo, Suee Im, and Myungju Kang, Attention routing between capsules. arXiv 2019.
preprint
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton, Alex Krizhevsky, and Sida Wang. Transforming auto-encoders. Interna- fional Conference on Artificial Neural Networks, pp. 51- Springer, 2011.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1412. 2014.
Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey Hinton. Stacked capsule autoen- coders. arXiv preprint arXiv:1906.066 0608.8888..,88 2019.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2288-2324, 1998.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer. framework for aitention-based permutation-invariie neural networks. In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3744-3753, 2019.
Hongyang Li, Xiaoyang Guo, Bo DaiWanli Ouyang, and Xiaogang Wang. Neural network encap- sulation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 252-267, 2018.
Sai Samarth R Phaye, Apoorva Sikka, Abhinav Dhall, and Deepti Bathula. Dense and diverse capsule networks: Making the capsules learn better. arXiv preprint arXiv: 1805. ,00000000,,000000000000 2018.

---

Published as a conference paper at ICLR 2020
Jathushan Rajasegaran, Vinoj Jayasundara, Sandaru Jayasekara, Hirunima Jayasekara, Suranga Seneviratne, and Ranga Rodrigo. Deepcaps: Going deeper with capsule networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 10725-10733, 2019. 2019.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp- 3856-3866, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-5998999 6008, 2017.
Dilin Wang and Qiang Liu. An optimization view on dynamic routing between capsules. 2018.
Suofei Zhang, Quan Zhou, and Xiaofu Wu. Fast dynamic routing based on weighted kernel density estimation. In International Symposium Artificial Intelligence and Robotics, pp. 301-309. Springer, 2018.
A MODEL CONFIGURATIONS FOR CIFAR-
A.1 MODEL SPECIFICATIONS
The configuration choices of Dynamic Routing CapsNets and EM Routing CapsNets are followed by prior work (Sabour et al. 2017 Hinton et al., 2018). We empirically find their configurations perform the best for their routing mechanisms (instead of applying our network configurations to their routing mechanisms). The optimizers are chosen to reach the best performance for all models. We list the model specifications in Table 2
and
We only show the specifications for CapsNets with a simple convolutional backbone. When con- sidering ResNet backbone, two modifications performed. First, replace the simple feature backbone with ResNet feature backbone. Then, the imput dimension the weights after the back- bone is set 128. A ResNet backbone contains 3 x 3 convolutional layer (output 64-dim.), three 64-dim. residual building block (He et al.) 2016) with stride 1, and four 128-dim. residual building block with stride 2. The ResNet backbone returns a 16 16 128 tensor.
For the optimizers. we use stochastic gradient descent with learning rate 0.1 for our proposed method, baseline CNN, and ResNet-18 (He et al.] (016). We Adam (Kingma & Ba 2014) with learning rate 0.001 for Dynamic Routing CapsNets and Adam with learning rate 0.01 for EM Routing CapsNets. We decrease the learning rate by 10 times when the model trained 150 epochs and 250 epochs, and there are 350 epochs in total.
A.2 DATA AUGMENTATIONS
We consider the same data augmentation for all networks. During training. we first pad four zero- value pixels to each image and randomly crop the image to the size 32 x 32. Then, we horizontally flip the image with probability 0.5. During evaluation, the model is trained on 8-GPU machine with batch size 128.
do not perform data augmentation. All
B MODEL CONFIGURATIONS FOR DIVERSE_ MULTIMNIST
B.1 MODEL SPECIFICATIONS
To fairly compare CNNs and CapsNets, we fix the number of layers and the number of neurons per layer in the models. These models consider the design: 36x36 image + 18x18x1002 1024 neurons + 8x8x1024 neurons + 6x6x 1024 neurons - 640 neurons - 10 class logits. The configurations are presented in Table 10 and 12, We also fix the optimizers across all the models. We use stochastic gradient descent with learning rate 0.1 and decay the learning rate by 10 times when the models trained on 150 steps and 250 steps. One step corresponds to 60, 000 training samples, and train the models with total of 350 steps.
10

---

Published as a conference paper at ICLR 2020
B.2 DATASET CONSTRUCTION
Diverse MultiMNIST contains both single-digit and overlapping-digit images. We generate im- ages on the fly and plot the test accuracy for training models over 2 (21M 350(steps) x 60, (00(images) generated images. We also generate the test images, and for each evaluation step, there are 10, 000 test images. Note that we make sure the training and the test images are from the disjoint set. In the following, we shall present how we generate the images. We set the probability of generating a single-digit image as 6 and the probability of generating an overlapping-dee image as 5 6'
The single-digit image in DiverseMulliMNIST training/ test set is generated by shifting digits in MNIST (LeCun et al. 1998) training/ test set. Each digit shifted up to pixels in each direction and results 36 36 image.
Following Sabour et al. (2017). we generate overlapping-dige images DiverSeMultMNIIT train- ing/ test set by overlaying two digits from the same training/ test set of MNIST. Two digits selected from different classes. Before overlaying the digits, we shift the digits in the same way which we shift for the digit single-digit image. After overlapping, the generated image has size 36 x 36.
We consider data augmentation for both training and evaluation. All the model is trained 8-GPU machine with batch size 128.
Table 2: Baseline CNN for CIFAR-10.
Operation
Output Size
input.dim=3. dim=114, 3x3 conv, stride=2, padding=1 ReLU
16x16x1022
input_dim=1024, 1utpur_dim=1024, 3x3 conv, stride=2, padding=1
pardde==.padddin-11
8x8x1024
ReLU Batch Norm
2x2 average pooling, padding=0
4x4x1024
input dim=1024, output dim=1024, 3x3 conv, stride=2, padding=1
2x2x1024
ReL.U Batch Norm
2x2 average pooling, padding-0
Ix1x1024
Flatten
1024
input.dim=1024, output dim==10, linear
Table 3: Baseline CNN for CIFAR-100.
Operation
Output Size
input dim=3, output dim=1024, 3x3 conv, stride=2, padding=1 ReLU
16x 16x16x102
input dim=1024, output dim=1024, 3x3 conv, stride=2, padddngg
8x8x1024
ReI.U Batch Norm
2x2 average pooling, padding=0
4x4x1024
input_dim=1024, output_dim=100 3x3 conv, stride=2, padding=1
2x2x1024
ReLU Batch Norm
2x2 average pooling, padding=0
1x1x1024
Flatten
1024
input d0m== outputudiine dim==10, linear
100

---

Published as a conference paper at ICLR 2020
Table 4: Dynamic Routing with Simple Backbone for CIFAR-I0.
Operation
Output Size
dim=3,, output dim=256, 9x9 conv, stride=1, padding-0
24x24x256
ReLU
dim=226,, output dim-=26, 9x9 conv, stride=2, padding=0
8x8x256
Capsules reshape
8x8x32x8
Squash
Capsules flatten
2048x8
Linear Dynamic Routing to 10 16-dim. capsules
10x16
Squash
Table 5: Dynamic Routing with Simple Backbone for CIFAR-100.
Operation
Output Size
input.dim=3, output. 9x9 conv, stride=1, padding-0
24x24x256
ReLU
input.dim=256, output. 9x9 conv, stride=2, padding=0
8x8x256
Capsules reshape
8x8x32x8
Squash
Capsules flatten
2048x8
Linear Dynamic Routing to 100 16-dim. capsules
100x16
Squash
Table 6: EM Routing with Simple Backbone CIFAR-IO.
Operation
Output Size
input.dim=3, dutput. 4x4 conv, stride=2, padding=1 Batch Norm + ReLU
16x16x256
input_ 511 Ixl conv, stride=1, padding=0 &
16x16x512
& 16x16x32
input.dim=256, dutput 1x1 conv, stride=1, 1adddng=, Sigmoid
16x 16x16x32xxx4
Capsules reshape (only for poses)
&
16x 16x16x32
7x7x32x4x4
Conv EM Routing to 32 4x4-dim. capsules, 3x3 conv, stride=2
&
7x7x32
5x5x32x4x4
Conv EM Routing to 32 4x4-dim. capsules, 3x3 conv, stride=1
5x5x32
800x4x4
Capsules flatten
&
800
10x4x4
Linear EM Routing 10 4x4-dim. capsules
&
10
12

---

Published as a conference paper at ICLR 2020
Table EM Routing with Simple Backbone CIFAR-100.
Operation
Output Size
dim=3,, output dim=256, 4x4 conv, stride=2, padding=1 padding]_
16x
Batch Norm + ReLU
input dim=256, output dim=512, Ixl conv, stride=1, padding=0
16x16x512
&
& 16x16x32
input_dim=256, dim 1x1 conv, stride=1, 1
Sigmoid
16x16x32x4x4
Capsules reshape (only for poses)
&
16x16x32
7x7x32x4x4
Conv EM Routing to 32 4x4-dim. capsules, 3x3 conv, stride=2
&
7x7x32
5x5x32x4x4
Conv EM Routing to 32 4x4-dim. capsules, 3x3 conv, stride=1
&
5x5x32
800x4x4
Capsules flatten
& 800
100x4x4
Linear EM Routing to 100 4x4-dim. capsules
& 100
Table Proposed Inverted Dot-Product Attention Routing with Simple Backbone for CIFAR-10.
Operation
Output Size
input.dim=3, sutput 3x3 conv, stride=2, padding=1
16x16x255
ReLU
input.dim=256, dutput. Ixl conv, stride=1, padding-0
16x16x512
Layer Norm
Capsules reshape
16x16x32x4x4
Conv Inverted Dot-Product Attention Routing to 32 4x4-dim. capsules, 3x3 conv, stride=2 Layer Norm
7x7x32x4x4
Cony Inverted Dot-Product Attention Routing 32 4x4-dim, capsules, 3x3 cony, stride=1 Layer Norm
5x5x32x4x4
Capsules flatten
800x4x4
Linear Inverted Dot-Produch Attention Routing to 10 4x4-dim, capsules Layer Norm
10x4x4
Reshape
10x16
input_dim=16, 16, output linear
10x1
Reshape
10
13

---

Published as a conference paper at ICLR 2020
Table 9: Proposed Inverted Dot-Product Attention Routing with Simple Backbone for CIFAR-100.
Operation
Output Size
input dim=3,, output d2m, 3x3 conv, stride=2, padding=1
16x16x128
ReLU
input.dim=128, output dimi:112,m1115,, 1xl conv, stride=1, padding-0
16x 1122
Layer Norm
Capsules reshape
16x16x32x6x6
Conv Inverted Dot-Product Attention Routing to 32 6x6-dim. capsules, 3x3 conv, stride=2 Layer Norm
7x7x32x6x6
Conv Inverted Dot-Product Attention Routing 32 6x6-dim. capsules, 3x3 cony, stride=1
5x5x32x6x6
Layer Norm
Capsules flatten
800x6x6
Linear Inverted Dot-Product Attention Routing to 20 6x6-dim. capsules Layer Norm
20x6x6
Linear Inverted Dot-Produs Attention Routing to 100 6x6-dim. capsules Layer Norm
100x6x6
Reshape
100x36
input_ output.dim==, linear
100x1
Reshape
100
Table 10: CapsNet with matrix-structured poses DiverseMultiMNIST.
Operation
Output Size
input.dim=3, ditputddim=1024, 3x3 conv, stride=2, padding=1 ReLU
18x18x1024
input.dim=1024, output.dim=104 3x3 conv, stride=2, padding=0
8x8x1024
Layer Norm
Capsules reshape
8x8x16x8x0
Cony Inverted Dot-Product Attention Routing to 16 8x8-dim. capsules, 3x3 conv, stride=1 Layer Norm
6x6x16x8x8
Capsules flatten
576x8x8
Linear Inverted Dot-Product Attention Routing to 10 8x8-dim. capsules Layer Norm
10x8x8
Reshape
10x64
dim- output linear Sigmoid
10x1
Reshape
10
14

---

Published as a conference paper at ICLR 2020
Table 11: CapsNet with vector-structured poses for DiverseMulliMNIST.
Operation
Output Size
input.dim=3, 1up 3x3 conv, stride=2, padding=l ReLU
18x18x1024
dim=1104, output output_dim=1024 dim=1024, 3x3 conv, stride=2, padding=0
8x8x1024
Layer Norm
Capsules reshape
8x8x16x64
Conv Inverted Dot-Product Attention Routing 16 64-dim. capsules, 3x3 conv, stride=1 Layer Norm
6x6x16xx6
Capsules flatten
576x64
Linear Inverted Dot-Product Attention Routing to 10 8x8-dim. capsules Layer Norm
10x64
dip-t.dim=64, output dimm=,, linear Sigmoid
10x
Reshape
10
Table 12: Baseline CNN for DiverscMultiMNIST.
Operation
Output Size
input.dim=3, dutput.dim=1024. 3x3 conv, stride=2, padding=1 ReLU
18x18x1024
input_dim=1024, cutput_dim=100 3x3 conv, stride=2, padding-0
8x8x1024
ReLU Batch Norm
output d02m=004, 3x3 conv, stride=1, padding=0
padding0
6x6x1024
ReLU Batch Norm
input dim=1024, output dim=640, linear
6x6x640
6x6 average pooling, padding=0
1x 1x1x640
Flatten
640
input dim=640, output dim=10, linear Sigmoid
10
15


Reviews:
[{"review_id": "rygSM4-q9r", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper presents a new simpler routing mechanism for capsule networks and achieves good performance on real world data sets making use of this new capsule structure along with a restnet backbone. Strong performance on the cifar10 and cifar100 datasets are presented and the network outperforms earlier versions of capsule networks. This new structure also performs well on an augmented MNIST dataset of overlapping digits (similar to the one used by Sabour et al 2017). \n\nOverall the paper is well written and presents solid results. The paper also presents a thorough comparison of two earlier versions of capsules which is a worthwhile contribution in its own right.\n\nThe paper could be improved by clearing up a few ambiguities:\n\n- is the learning rate schedule the same for all three models? in figure 4 it looks like the learning rate is decayed at two distinct points for your model, but only one distinct point for both the EM and Dynamic routing models.  \n\n-\"Notably, the prediction becomes random guess when the iteration number increases to 5.\" this sentence is a little confusing. Do you mean when the iteration number the performance is equivalent to not random assignments?  \n\n- This new algorithm requires that the capsules in L+1 have initialized poses with which to compare agreement between the poses in L. This is initial value seems like it may greatly effect the performance of the model. In the paper it is set to 0 and not expanded upon. It would be interesting to see if randomizing this value, or learning a bias for it would effect performance.   \n\n-unlike the two previous versions of capsules, the inverted dot product capsules show in figure 4 sudden huge decreases in test accuracy while training. These moments seem to be overcome quite quickly and the model ends up outperforming the other two. But it would be worth mentioning this behavior and perhaps attempting to explain it.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SklNZ1259S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors improve upon dynamic routing between capsules by removing the squash function (norm normalization) and apply a layerNorm normalization instead. Furthermore, they experiment with concurrent routing rather than sequential routing (route all caps layers once, then all layers concurrently again and again). This is an interesting development since provides better gradient in conjunction with layerNorm. They report results on Cifar10 and Cifar100 and achieve similar to CNN (resnet) performance.\n\nFirst, I want to point out that inverted attention is exactly what happens in dynamic routing (sabour et al 2017), proc. 1 line 4,5, and 7. In dynamic routing the dot product with the next layer capsule is calculated and then normalized over all next layer capsules. The only difference that I notice between alg. 1 here and proc. 1 there is replacement of squash with layer norm. There is no \"reconstructing the layer bellow\" in Dynamic routing as authors suggest in intro. \n\nSecond, the Capsules are promised to have better viewpoint generalizability than CNNs while having comparable performance. Replacing the 1 convolution layer with a ResNet backbone and replacing the activation with a classifier on top seems reducing the proposed CapsNet to the level of CNNs in terms of Viewpoint Generalization. Why should someone use this network rather than the ResNet itself? Fewer number of parameters by itself is not interesting, the reason it is reported usually is that it indicates lower memory consumption or fewer flops. Is that the case when comparing the baseline ResNet with the proposed CapsNet? Otherwise, a set of experiments showcasing the viewpoint generalizability of proposed CapsuleNetworks might only justify the switch between resnets to the proposed capsnets.\n\nThirdly, Fig. 4 top images seems to indicate all 3 routing procedures are following the same Learning Rate schedule. In the text it is said that optimization hyperparameters are tuned individually. Did authors tune learning rate schedule individually?\n\nForth, the proper baseline for the current study is the dynamic routing CapsNet. Why the multiMNIST experiment lacks comparison with dynamic routing capsnet?\n\nFor the reasons above, the manuscript in its current format is not ready for publication.\n\n------------------------------------------------------rebuttal\nThank you for your response. I acknowledged the novel contributions of this work. My comment was that some claims in the paper are not right. i.e. \"inverted dot-product attention\" is not new and \"reconstructing the layer bellow\" does not happen in Sabour et al . Parallel execution + layer norm definitely is novel and significant.\n\nRegarding the LR-schedule, I am not sure how fair it is to use same hyper-params tuned for the proposed method on the baselines. \n\nRegarding the viewpoint, the diverseMultiMNIST is two over lapping MNIST digits shifted 6 pixels. There is no rotation or scale in this dataset. An example experiment verifying the viewpoint generalizability of the proposed model is training on MNIST testing on AFFNIST. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1x0kqZJ5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a simple and effective routing algorithm for capsule networks. The paper is well written. A nice analysis of the proposed routing algorithm is provided. Experiments of varying the routing iterations demonstrate the stableability of proposed routing algorithm compared to others.\n\nHere are some issues:\n1. Would the authors release the code for reproducing the results in the paper? It will be helpful for future research in this area.\n\n2. In Fig.5, it would be better to give some brief explanations about why CasNet (Matrix) occupies much more memory while possessing less parameters.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a routing algorithm for capsule networks, and demonstrates empirical evaluation on CIFAR-10 and CIFAR-100. The results outperform existing capsule networks and are at-par with CNNs. Reviewers appreciated the novelty, introducing a new simpler routing mechanism, and achieving good performance on real world datasets. In particular, removing the squash function and experimenting with concurrent routing was highlighted as significant progress. There were some concerns (e.g. claiming novelty for inverted dot-product attention) and clarification questions (e.g. same learning rate schedule for all models). The authors provided a response and revised the submission , which addresses most of these concerns. At the end, majority of reviewers recommended accept. Alongside with them, I acknowledge the novelty of using layer norm and parallel execution, and recommend accept.

Decision: Accept (Poster)