Trans-Caps: Transformer Capsule Networks with Self-attention Routing
Authors: Aryan Mobiny
Pietro Antonio Cicalese
Hien Van Nguyen
Venue:  (2021)
Keywords: capsule network
self-attention

Abstract: Capsule Networks (CapsNets) have shown to be a promising alternative to Convolutional Neural Networks (CNNs) in many computer vision tasks, due to their ability to encode object viewpoint variations. The high computational complexity and numerical instability of iterative routing mechanisms stem from the challenging nature of the part-object encoding process. This hinders CapsNets from being utilized effectively in large-scale image tasks. In this paper, we propose a novel non-iterative routing strategy named self-attention routing (SAR) that computes the agreement between the capsules in one forward pass. SAR accomplishes this by utilizing a learnable inducing mixture of Gaussians (IMoG) to reduce the cost of computing pairwise attention values from quadratic to linear time complexity. Our observations show that our Transformer Capsule Network (Trans-Caps) is better suited for complex image tasks including CIFAR-10/100, Tiny-ImageNet, and ImageNet when compared to other prominent CapsNet architectures. We also show that Trans-Caps yields a dramatic improvement over its competitors when presented with novel viewpoints on the SmallNORB dataset, outperforming EM-Caps by 5.77% and 3.25% on the novel azimuth and elevation experiments, respectively. Our observations suggest that our routing mechanism is able to capture complex part-whole relationships which allow Trans-Caps to construct reliable geometrical representations of the objects.

Under review as a conference paper at ICLR 2020
TRANS-CAPS: TRANSFORMER CAPSULE NETWORKS WITH CELE-AITNNTON ROUTING
Anonymous authors
Paper under double-blind review
ABSTRACT
Capsule Networks (CapsNets) have shown to be a promising alternative to Con- volutional Neural Networks (CNNs) in many computer vision tasks, due to their ability to encode object viewpoint variations. The high computational complexity and numerical instability of iterative routing mechanisms stem from the challeng- ing nature of the part-object encoding process. This hinders CapsNets from being utilized effectively in large-scale image tasks. In this paper, we propose a novel non-iterative routing strategy named self-attention routing (SAR) that computes the agreement between the capsules in one forward pass. SAR accomplishes this by utilizing learnable inducing mixture of Gaussians (IMoG) to reduce the cost of computing pairwise attention values from quadratic to linear time complexity. Our observations show that our Transformer Capsule Network (Trans-Caps) is better suited for complex image tasks including CIFAR-100/00, Tiny-ImageNet and ImageNet when compared to other prominent CapsNet architectures. We also show that Trans- Caps yields a dramatic improvement over its competitors when presented with novel viewpoints on the SmalINORB dataset, outperform- ing EM-Caps by 5.77% and 3.25% on the novel azimuth and elevation experi- ments, respectively. Our observations suggest that our routing mechanism is able to capture complex part-whole relationships which allow Trans-Caps to construct reliable geometrical representations of the objects.
INTRODUCTION
Convolutional Neural Networks (CNNs) have achieved oftte-of-thh-att performance in many dif- ferent computer vision tasks (Krizhevsky al., 2012 He et al., 2016). This is achieved by local connectivity and parameter sharing across spatial locations so that useful local features learned in one receptive field can then be detected across the input feature space. While such a mechanism is sufficient to learn relationships between nearby pixels and to detect the existence of objects of interest, CNNs often fail to detect objects presented in radically new viewpoints due to the complex effects of the viewpoint changes the pixel intensity values. This limitation forces us to train each CNN with a large number of data points which is computationally expensive.
Capsule Networks (CapsNets) were introduced explicitly learn viewpoint invariant representa- tion of the geometry of an object. In CapsNets, each group of neurons (called a "capsule") encodes and represents the visual features of higher-level object in instantiation parameter vector ma- trix (which we refer to as the pose vector or matrix throughout this paper). The lower-level capsules (which refer as part capsules) estimate the pose of the object parts and hierarchically com- bine them to predict the pose of the whole object in the next layer. The object-part relationship is siewpoint-invarian meaning that changes in the viewpoint change the pose of parts and objects in a coordinated way. Therefore, regardless of the viewpoint, we can infer the pose of the whole object from its parts using set of trainable viewpoint-invarian transformation matrices. Capsule routing mechanisms can therefore learn the underlying spatial relationships between parts and objects. This improves the generalization capabilities of the network due to the underlying linear relationship be- tween the viewpoint changes and the pose matrices. In order to route information between capsules, the part capsules vote for the pose of the higher-level capsules (which we refer A routing-by bouting-by-agreemee agrremeetnt mechanism is employed to aggregate votes (which has been traditionally accomplished using a recurrent clustering procedure) effectively computing the contribution each part the object pose.
to as object capsules).

---

Under review as a conference paper at ICLR 2020
While various proposed iterative routing mechanisms (such Dynamic (Sabour et al. 2017 7) and EM (Hinton et al. 2018) routing) have been shown to be effective in the detection of viewpoint variations, their iterative nature increases computational cost. Prior research has additionally shown that these routing mechanisms may fail to properly construct parse tree between each set of part and object capsules, partly due the inability of the network to learn routing weights through back-propagation (Peer et al., 2018). This ultimately limits the performance of CapsNets in real- world image classification tasks. Additionally, the correct number of routing iterations serves as an additional data-dependent hyper-paramett parameter that needs to be carefully selected; failing to optimize the number of routing operations can result in increased bias or variance in the model (Hinton al.) 2018). This issue is amplified when training networks with multiple capsule layers.
In this paper, we introduce a novel routing algorithm called self-attention routing (SAR), which is inspired by the structural resemblance between CapsNets and Transformer networks (Vaswani et al. 2017). This mechanism eliminates the need for recursive computations by replacing unsu- pervised routing procedures with a self-attention module, making the use of CapsNets effective in complex and large-scale image classification tasks. Our algorithm also reduces the risk of under and over-fitting associated with selecting small and large number of routing iterations, respec- tively. We compare our proposed routing algorithm to two of the most prominent iterative methods, namely dynamic and EM routing, and the recently published non-iterative self-routing mechanism (Hahn et al. 2019). We evaluate performance on several image classification datasets including SVHN, CIFAR- I0, CIFAR-100, Tiny-ImageNet, ImageNet, and SmalINORB. Our results show that our model outperforms other baseline CapsNets and achieves better classification performance and convergence speed while requiring significantly fewer trainable parameters, fewer computations (in FLOPs), and less memory. Moreover, our experimental result on the SmalINORB dataset with novel viewpoints shows that the proposed model significantly more robust to changes in the viewpoint and is able to retain its performance under severe viewpoint shifts. All source code will be made publicly available.
RELATED WORK
2.1 CAPSULE NETWORKS
; here they pose computer vision tasks as inverse graphics problems deal with variations in an object's instantiation parameters. This architecture learns to reconstruct an dffine-tranfformed version of the input image, therefore learning represent each input as a combination of its parts and their respective characteristics. (Sabour et al. (2017) introduced capsules with Dynamic Routing (DR- Caps), which allows the network learn part-whole relationships through an iterative unsupervised clustering procedure. In DR-Caps, capsules output a pose vector whose length (norm or magnitude) implicitly represents the capsule activation. The vector norm should be able to scale depending on the pose values; representing existence with the vector norm can therefore potentially weaken the representation power of any given capsule layer. HHnnon et al. (2018) proposed capsules with EM routing (EM-Caps), where capsule activations and pose matrices from part capsules through a mixture of Gaussians. While powerful, capsule network's routing procedures have several fundamental limitations:: 1) Iterative routing operations are the bottleneck of CapsNets due to their computational complexity, which limits their widespread applicability in complex, large-scale datasets (Zhang al. 2018: al. 2018). 2) The number of routing iterations are siyper-parameter that need to be carefully tuned to prevent under and over fitting (Hinton et al., 2018). 3) Lin et al. (2018) showed that even after seven iterative routing operations, the entropy of the coupling coefficient was still large, indicating that part capsules pass information to all object capsules. 4) EM-Caps have difficulty converging and have been shown to be numerically unstable, which limits their applicability in complex tasks (Ahmed & Torresani 2019 Gritzman, 2019).
CapsNets were originally introduced in Transforming Autoencoders by Hinton et al. (201T);
segregated to fit the votes
Several studies have proposed non-iterative methods to replace the traditional iterative routing mech- anisms in CapsNets. STAR-CAPS (Ahmed & Torresani 2019) combines an attention gate with straight-through estimator to make a binary decision to either connect or disconnect the route be- tween each part and object capsule. Tsai et al. (2020) proposed an inverted dot- dot-product attention routing mechanism (IDPA-Caps which generates the routing coefficients between capsules; they unroll the iterative routing procedure and perform the iterations concurrently which helps improve
2

---

Under review as a conference paper at ICLR 2020
parallelization. While powerful, the number of concurrent iterations is a hyper-parametes that needs to be tuned, and the unrolling process creates a very large network that is memory intensive. Inspired by Mixture-of-FFr Experts,eetttiiiee self-routing attaches stationary routing weights to specific locations which limits its ability to gener- alize novel viewpoints. This also increases the required number of trainable parameters, making it impractical for high-dimensional images.
Mf Hahn al. (2019) introduced self-routing mechanism. While non-iterative,
2.2 SELF-ATTENTION
Attention operations bias a network to more informative components of the input in order to improve the discriminative capabilities of the model. This operation has been used to tackle the problem of long-range interactions in sequence modeling and has seen great success across the fields of Nat- ural Language Processing (NLP) (Bahdanau et al., 2014), genomics (Zaheer et al.] (020). speech recognition (Chorowski et al., 2015), and computer vision (Hu et al. [0118] Wang et al. 2017). Various attention mechanisms have been used to improve CNNs by allowing the network cap- ture interactions between elements of the encoded feature space, which is difficult for a stand-alone convolutional operation (Woo et al. 2018 Hu et al.] 20188). Transformer based architectures were introduced by VVasaani et al. (2017) and utilize self-attention as the primary mechanism for repre- sentation learning. Self-attention employs the standard dot product operation to generate attention coefficients that effectively capture the long-range interactions between inputs and outputs. These architectures have outperformed recurrent neural networks in wide range of tasks and have become the SOTA for representation learning (Devlin et al. 2018 Radford et al. 2019 Huang et al.) 2018). This concept was then expanded to computer vision applications by treating each output pixel as an element in the self-attention operation, thus allowing CNN to learn global dependencies between receptive fields (Bello al., 2019). While powerful, this mechanism generates global dependen- cies for all pixels, making it memory intensive and computationally cumbersome. This issue was later addressed by restricting the scope of each self-attention operation to local patches as opposed to applying self-attention to the global feature space (Hu et al. 2019 Ramachandran et al. 2019). The Set Transformer is an encoder-decoder architecture that util utilizes a sell-attention mechanism to cluster group of independent inputs by modeling the interactions among the elements of the set (Lee et al. 2019). Given that the order of the part capsules does not contribute to the understanding of each object capsule, we took inspiration from the Set Transformer to replace the recurrent routing mechanism with a self-attention- aggregation of the "votes" in a permutation invariant manner.
3
TRANSFORMER CAPSULE NETWORK
3.1 MODEL ARCHITECTURE OVERVIEW
The Transformer Capsule Network (Trans-Caps) is capsule-based neural network architecture where each capsule represents an encoded pose matrix. Trans-Caps starts with a convolutional backbone, followed by sequence of capsule layers. Note that the choice of the convolutional back- bone, the number of capsule layers and the number of capsules per layer varies for different sets of experiments. Given an input image X I2x' DD the role of the convolutional backbone is to extract set of features F E H XW' from the input. The backbone can be either single con- volutional layer, a cascade of a few convolutional layers, or a cascade of residual blocks (He et al.] 2016). Previous studies have shown improved performance for more complex datasets when using a residual backbone (T'ai et al. 2020 Hahn et al., 2019). We provide a detailed discussion of the various backbone configurations in the Experiments section. The first capsule layer (PrimaryCaps) is a convolutional layer, followed by BN applied to the output backbone features F. The outputs are then reshaped to form the primary capsule pose elements. All layers following this layer are convolutional capsule layers (ConvCaps) with SAR performed between the layers to construct the pose of the object capsules. With our non-iterative SAR mechanism, the pose computation at all stages of the network can be performed sequentially in one forward pass, yielding numerical sta- bility and efficiency. The final capsule layer (ClassCaps) has as many capsules as the number of classes and predicts the pose of the objects, Pf where E [}) J and L represent the number of object capsules and the total number of layers, respectively. This layer is followed by a linear Mlassified which is shared across all class capsules and computes the final class logits as
3

---

Under review as a conference paper at ICLR 2020
PRP 200 SAL PONEE
weltlottoo Fgg A4
#FF
P
Figure 1: Overview of a Trans-Caps layer with SAR.
0c Classifier (P.)) where P is the output of the final ConvCaps layer and E {1,..., C) in
a C' class problem.
3.2 INPUT POSE TRANSFORMATION
Figure describes one objt-t capsule layer with SAR. Let Ω4 denote the sets of capsules in layer E L}. Each part capsule E Z outputs pose matrix P Rpxp in which each of the p x p entries of the pose matrix represent some instantiation parameter, such as location, size, and orientation. In our implementation, the capsule activation probability is separately defined, but we instead enable the network to encode the presence or absence of the given part in the pose matrix itself. Each capsule
S2 transforms its pose matrix P and casts a vote Vij E I2x as the prediction made by the hh part capsule for the pose of the jth object capsule (j E 94+1)). The object- part relationship is viewpoint-invarian applied to the pose of part will have a similar effect on the pose of the object. Therefore, it can be encoded by learned transformation matrices W RPxx, allowing to then generate the votes following
and approximately constant; this means that a transformation
V W P
(I)
Note that the W!J, transformation matrices are learned discriminatively by back-propagation allow- ing the network to gradually construct transformation matrix for each capsule pair to encode the corresponding part-whole relationship. Finally, a non-linear routing procedure is required to pro- cess the votes Vi for all i E Ω then generate the pose of the object capsules in layer 2+ 1. In Trans-CapsNet,, we replace the recurrent clustering approach of iterative routing methods with our non-iterative SAR procedure.
3.3 SELF-ATTENTIOX ROUTING
Our SAR mechanism aggregates votes V! sent to the ju object capsule according to their agree- ment and then generates the output that best explains the pose of the higher-level object. Aggregating votes through self-attention mechanism is sensible the influence of each vote the final pose of the object capsule is not necessarily equal. However, if we were to compute the pair- wise simi- larities among the votes following the self-attention mechanism proposed by Vaswani et al. (201)). we would be limited by the quadratic time complexity of the operation O(z") for n. part capsules, making it memory intensive for multiple part capsules. routing mechanism should also be permu- tation invariant since the order of the part capsules does not contribute to our understanding of the object capsules. Therefore, we took inspiration from Set Transformers (Lee et al. 2019), in which generate similarity measures among a set of unordered part capsule votes through interactions with a set of trainable vectors, allowing us to retain linear time complexity (JJ) where n and J are the number of part and object capsules, respectively.
We now formally describe our SAR mechanism. We use the following naming convention: N₁ corresponds to the number of heads, while de and d, are the respective number of dimensions for the key and value vectors. We also assume that the multi-head attention mechanism evenly divides the da and d, dimensional vectors into di and d dimensional key and value vectors per attention head. The operation begins by flattening the vote matrices ViJ and projecting them onto N₂, different
4

---

Under review as a conference paper at ICLR 2020
de and d dimensional key (k()) and value (v vectors, respectively. This is done using set of learnable transformation matrices A = { & wh j where W E R x and W E R R We set d = de = de throughout the rest of the paper for brevity, unless otherwise specified. Inspired by the inducing point vectors in Set Transformers, we use an inducing mixture of Gaussian (IMoG) distributions to compute the part capsule agreements, The IMoG is composed of one Gaussian component per object capsule and is parameterized as
p(œ) Σ o,N(al En) where
Σ
0 =1
Note that Φj. Mj and Σj are the learnable weights, means and covariance matrices associated to the jth component, respectively. Overall, a trainable IMoG with components can be seen as independent memory slots occupied by encoded templates that represent the average appearance of the corresponding objects. The IMoG templates are global query vectors accessed through our attention-based routing procedure to measure the agreement among the part capsule votes. This ef- fectively reduces the quadratic time complexity of key-query self-attention to linear time complexity OJJn) (where J is typically small) and allows the self-attention mechanism properly scale with input size and the number of capsules. Given our use of IMoG, we replace the standard dot-product attention in Transformers with our log- likelihhodd attention mechanism, which utilizes Gaussians to encode the second order interactions among points. Assuming a mixture of isotropic Gaussians with Σ, = diag ( ..., z. the similarity matrix S R from / part capsules to object capsules computes the log ligelikeooood of the key vectors knj R with respect to the components of the IMoG as
A = log p(kk 05 pkkyiii H4. Ej) = log Φj 0.5 (og(T - log(o + 12 20
(kiy)² h.d
(3)
We consider the vector 8, [stylen R' as the similarity vector associated the jth object capsule. The output of the SAR mechanism for the jth object capsule from a single head h can then be formulated as
of = ui h
+
0 o where at softmax 8 VV)
(4)
The similarity vectors s scaled by 1/Vd avoid the vanishing gradient problem (as was discussed by Vaswani et al.] (2017)), and then sof tmax normalized to sum to one over all part capsules, yielding the vector of attention coefficients a². We emphasize that the of outputs the sum of a static component ud) which represents the typical appearance of object j. and a dynamic
component E'= . While the static component input independent, the dynamic component is function of the input image, and therefore allows the output to account for deformations and variations in the objects' shape and appearance. The raw output linear transformation of the concatenation of all attention head outputs given by
0 = Norm (concat o W.
(5)
where Norm can be either Batch Nacch-Normalization (BN) (loffe & Szegedy 2015) or Layer Normal- ization (LN) (Ba et al. 2016). The final pose of the 7th object capsule computed
Pl Norm O, + rFF(O,))
(6)
where rFF is any feed-forward layer with ReLU activation that processes each instance indepen- dently and identically. As for the Norm function, we selected BN as we found it to be empirically superior to LN in improving the routing operation's convergence speed. BN also enables us train
5

---

Under review as a conference paper at ICLR 2020
Table 1: Key differences between Trans-Caps and the most prominent CapsNets.
DRCCop Sabour et al 2017
ExECapp Himmo et S01s
SRCCaps (Hain at ar 2019
Iranc-Capp Cani--appeee (ours)
Loss Function
Mangin
Speead
Negative LogeLienioo
Cross kittroyy
Pose, Activation
Vector, Scalar (Langth of Pose)
Marnx, Scalar
Maini, Scalar
Maltis, NIA
Routing
lientive
Agreement Mensure
Cosine Dntance (yymmiic
Euchiran Distance (Dynaiic)
Learnel weights (Suaie)
Log-Bkell hood Atlention (Dynamic)
Non-Lincarity
Normalization &
Squash
Sigmoid (for Activation)
Sigmoid (for Activation)
BN (foe Pose)
BN (fur Pose)
our model with the same optimizer and learning rate as the baseline CNN and CapsNet models, which improves the quality of our comparisons.
4 EXPERIMENTS
In our experiments, we evaluate the performance of the proposed SAR mechanism and compare to baseline CNNs, CapsNets with the four most prominent iterative approaches (Dynamic routing (Sabour et al. 2017), EM routing (Sabour al. 2017). IDPA-Caps (T'ai al., 2020). and Varia- tional Bayes Capsule Routing (VB (VB-Caps) (Ribeiro et al., 2020),, ) and a CapsNet with non-iterative self-routing mechanism (Hahn et al., 2019). Note that in all experiments, we use the same backbone CNN for all CapsNets for fair comparison. We finally evaluate the generalization and robustness of the proposed model to viewpoint changes following the key motivation of CapsNets. Our full source-code is publicly available ***.
4.1 TRAINING SETTINGS
The model parameters are updated using a Stochastic Gradient Descent optimizer with a learning rate of 0.1. In each experiment and for all models, we decay the learning rate by factor of 10 after set number of training epochs. We experimented with Spread (Hinton et al. 2018), Margin (Sabour al., (201)) and cross-entropy loss functions and achieved the best performance using cross-entropy for multi-class classification. Table summarizes the key differences of our proposed model when compared to various CapsNet architectures.
4.2 DATASETS
We evaluate our proposed model on various image classification tasks (CIFAR-10 10 (Krizhevsky et al., 2009), SVHN (Netzer et al., 201T). CIFAR-100 (Krizhevsky et al. 2009). SmallNORB (LeCun et al. 2004 Tiny-ImageNet, and ImageNet (Deng 2000g). The Tiny-Im dataset is composed of a subset of 200 object classes of the original 1000 ImageNet dataset classes, containing 500 and 50 images for training and evaluation, respectively. These datasets of natural images were selected to evaluate the performance of our proposed architecture on complex data tasks. Following Hinton al. 20188) we select the SmallNORB dataset evaluate the generalization performance of our model to viewpoint changes. Note that in all the experiments, 10 percent of the training data sampled randomly generate the validation set. We perform 5-fold cross-validation and report the average std) error rate over the trained models. Data pre-processing and augmentation details are provided in section appendix A.I
4.3 IMAGE CLASSTFICATION RESULTS
To ensure that our results are consistent with the literature, we follow the CapsNets configurations used in prominent publications (Hinton et al. 2018, Hahn et al,, 2019 Tsai al.. 2020). Model architectures for all datasets and their respective output sizes are described in detail in appendix A.4 Figure illlstrrae the convergence speed of all four CapsNets architectures on CIFAR- and the computational cost associated with each model on both CIFAR-10 and C00. Trans-Caps has significantly lower memory requirements and has a lower computational overhead (measured in FLOPs) than all other evaluated architectures. We also note that Trans-Capa has nearly the same number of trainable parameters as EM-Caps. We report the error rate of our model across each dataset of interest and compare this result with baseline CapsNets and CNNs in Table 2 The extent

---

Under review as a conference paper at ICLR 2020
CIFAR-10
DR-Caps
EM-Capp
SR-Caps
Trane-oppe (Oim)
90
CIFAR-
70
Acuuracy
50
CIAAR
100
200
300
400
params, (M)
Memory (GB)
Epochs
Performance analysis for CapsNets on CIFAR-IO and CIFAR-100. Left: Convergence plots for different routing mechanisms CIFAR-10 validation sets. Right: number of trainable parameters, memory usage, and number of FLOPs on CIFAR-10 (top) and CIFAR- 100 (bottom). For fair comparison, performance is benchmarked using the same GPU with batch size of 16.
Figure 2:
of improvement between Trans-Caps and the other CapsNets architectures increases with the com- plexity and variability of the dataset. We note that DR-Caps and EM-Caps degrade substantially as complexity increases, performing poorly on CIFAR-10 100 and failing to converge on Tiny- Imt SR-Caps performs similarly to Trans-Caps on SVHN but fails match its performance across the other datasets, and is vastly outperformed on Tiny-ImageNe (20.4% and 36.1% higher top-1 and top-5 error, respectively). Finally, we evaluated the performance of Trans Trans-Caps on full-scale Ima- geNet; the model architecture is described in detail in appendix A.4 We achieved a 65.13% and 86.81% % top-1 and top-5 classification accuracy, respectively.
Table The test error rates between different CapsNet and CNN models on various tasks.
SVHN
CIFAR-10
CIFAR-100
Tiny-Imagehe ImageNet
top-1
top-5
ResNet-20 He et al.
2016
3.58:0.11
8.28+0.24
29.10=1.221
17.47=10.27 39.81=0.32 38.68-0.30
ResNet-32
ct al. 2016
3.25:0016
7.5110.18
29.86:0.17
16.83:1026
SE-ResNet Hu
al. 2018
2.85=0.17
6.3210.20
24.83+0026
AA-ResNet Bello ct al.
2019
2.68=0.11
4.7610..1
21.43:0.14
32.48 0.27
13.66110.3
DR-Caps (Sabour al. 2017)
3,44-0.28
7.28+0.15
47.89-0.42
EM-Caps (Hinton et al. 2018
4.15:10.7
14.63-0.43
59.03-10.58
VB-Caps (Kibeıro ct al. 2020
3.81=:021
11.48=0.22
37.28:1044
IDPA-Caps [Tsai al. 2020
2.99-0.18
6.81+0.31
27.01
42.19-0.48
26.2910.37
SR-Caps (Háhn et al.
2019
3.12-0.13
7.86+0.11
28.53+0028
57.86-0.50
51.4240.43
Trans-Caps (ours)
3.02+0014
6.56+0.16
25.17+0.21
37.46-0.33
15.35+0.29
Table 3: The error rates of various models on familiar and novel SmallNORB viewpoints.
Azimuth
Elevation
# Params. (K)
Familiar
Novel
Familiar
Novel
CNN
897.2
8,42+0.48
22.43+1.32
7.82+0.63
18.97+0.92
DR-Caps
1406.9
8.28+0.50
19.33+1.02
7.57+0.46
17.18+0.88
EM-Caps
98.1
7.25++0.8
14.11+0.98
6.39-+0.81
12.73+0.80
VB-Caps
108.2
7.2111048
11.74_1.73
7.1210.58
12.04410.8
SR-Caps
989.2
7.85+0.61
18.47-1.97
6,89+10722
16.62+1.29
Trans- Trans-Caps (ours)
103.5
7.28
8.34.10.1
6.43-10.60
9.48-10.77

---

Under review as a conference paper at ICLR 2020
plane
bustruck
Asimath (familiar) 10 (novel) 20 (lovel)
Elevation lovels
(familar) (soovll)
Figure 3: The learned IMoG overlayed with the familiar and novel outputs of the Cl ClassCaps layer for the novel azimuth (top) and elevation (bottom) experiments on SmalINORB. Each shade of the Gaussians represents one standard deviation, while each axis represents one of the 16 total poses.
4.4 GENERALIZATION TO NOVEL VIEWPOINTS
We evaluate the performance of the proposed CaspsNet architecture the SmalINORB dataset demonstrate its generalization strength to novel viewpoints. The SmalINORB dataset is composed of gray-scale 96 x 96 images of 5 classes of toys (four-legged animals, human figures, airplanes, trucks, and cars). There are 10 physical instances of each class (5 for train and 5 for test). each different azimuths, 9 elevations, and 6 lighting conditions (for total of
of which is pictured 48, 48,600 images). The tightly controlled aspects of the SmalINORB dataset make the classification of its images ideal shape recognition task, and allow us evaluate the ability of the network to generalize in three-dimenssioal space. We investigate the ability of the architecture to generalize to both familiar and novel viewpoints. The familiar experiments are performed by training and testing all viewpoints, while the novel experiments are performed by holding out the most unique azimuths (from 6 to 28) and clevations (from 3 to 8), following what was described in Hinton et al.) (2018). We use single convolution layer backbone with a kernel size of 5 and stride of 2 encode the input image into 64 feature maps due to the set size and simplicity of the data. The architecture described in detail in Table 5 of appendix A.4 To compare performance, we evaluate baseline CapsNet architectures using the same structure and a simple CNN (see Table I1 of appendix (A.4).
We report the performance of all architectures in Table 3 for the familiar and novel viewpoint tasks. We note that EM-Caps, VB-Caps, and Trans-Caps significantly outperform the other architectures on both of the familiar viewpoint tasks. In the novel viewpoint tasks, Trans- Trans-Caps dramatically outperforms the other architectures, outperforming EM-Caps by 5.77% and 3.25% on the novel azimuth and elevation tasks, respectively. To understand the relationship between the familiar and novel viewpoints in the space of the Gaussians, we plot the learned IMoGs overlayed with the familiar and novel outputs O, from the ClassCaps layer, as shown in Figure 3 We note that the familiar viewpoint outputs (circles) are often grouped adjacent to the center of their respective Gaussians. Novel viewpoint outputs (triangles and squares) tend to cluster with their respective azimuths and elevations, and therefore typically shifted within the area of their class Gaussians. This implies that the network has learned to represent the input objects in three- dimensional space, allowing to effectively adapt novel viewpoints.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
In this work, we propose novel CapsNet architecture named Trans-Caps which employs non- iterative self-attention routing mechanism to address the computational complexity and numeri- cal instability of CapsNets. The proposed routing mechanism uses learnable, inducing mixture of Gaussians to estimate the agreement (or attention) between capsules. Our experimental results showed that Trans- Caps can effectively scale up to much larger datasets and outperform the baseline SOTA CNNs and CapsNets several image recognition tasks with significantly fewer parameters and reduced computational cost. Our observations show that the network is able to effectively struct a linee-oimensional understanding of the geometry of objects, which is indicative of properly encoded instantiation parameters.
REFERENCES
Karim Ahmed and Lorenzo Torresani. Star-caps: Capsule networks with straight-through attentive routing. In Advances in Neural Information Processing Systems, pp. 9101-9110, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv:1607.06450, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 1rXiv::409.0444 2014.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In Proceedings of the IEEE International Conference Computer Vi- sion, pp. 1286-3295, 2019.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention antention-based models for speech recognition. In Advances in neural information processing systems, pp. 577-585, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scabl scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /tti:/881..0000 1800004805 2018.
Ashley Daniel Gritzman. Avoiding implementation pitfalls of "matrix capsules with em routing" by hinton et al. In International Workshop on Human Brain and Artificial Intelligence, pp. 224-234. Springer, 2019.
Taeyoung Hahn, Myeongjang Pyeon, and Gunhee Kim. Self-routing capsule networks. In Advances in Neural Information Processing Systems, pp. 7658-7667, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto -etto-eec.drss.... In Interna- tional conference on artificial neural networks, pp. 44-51. Springer, 2011.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with routing. In Interna- tional conference on learning representations, 2018.
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In Proceedings of the IEEE International Conference Computer Vision, pp. 3464-3473, 2019.
Jie Hu, Shen, and Gang Sun. Squezze-and suueezzeannd-xxiitatiin excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.

---

Under review as a conference paper at ICLR 2020
Cheng- Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, lan Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv: 04288090042,,,. 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03: 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. I1-104, IEEE, 2004.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariaal neural networks. In Interna- tional Conference on Machine Learning, pp. 3744-3753, PMLR, 2019.
Hongyang Li, Xiaoyang Guo, Bo DaiWanli Ouyang, and Xiaogang Wang. Neural network encap- sulation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 252-267, 2018.
Ancheng Lin, Jun Li, and Zhenyuan Ma. On learning and learned representation by capsule net- works. arXiv preprint arXiv: 1810.04041, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
David Peer, Sebastian Stabinger, and Antonio Rodriguez-Sancher Training deep capsule networks. arXiv preprint arXiv: (XX2:1812009700) 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188):9, 2019.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: 1999600050000 2019.
Fabio De Sousa Ribeiro, Georgios Leontidis, and Stefanos D Kollias, Capsule routing via variational bayes. In AAAL, pp. 3749-3756 3756, 2020.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules, In Advances in neural information processing systems, pp. 1856-3866 2017.
Yao-Hung Hubert Tsai, Nitish Srivastava, Hanlin Goh, and Ruslan Salakhutdinov. Capsules with inverted dot-product attention routing. arXiv preprint arXiv:2002.04766 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang. and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164, 2017.
Sanghyun Woo, Jongchan Park, Joon- Joon-Young Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
10

---

Under review as a conference paper at ICLR 2020
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007. 14062, 2020.
Suofei Zhang, Quan Zhou, and Xiaofu Wu. Fast dynamic routing based on weighted kernel density estimation. In International Symposium on Artificial Intelligence and Robotics, pp. 301-309. Springer, 2018.
A APPENDIX
A.1 DATA PRO-PROCSSIING AND AUGMENTATION
In all our experiments, consider the same data augmentation for all networks (i.e. CNNs and CapsNets with different routing mechanisms). For SVHN, CIFAR-10, and CIFAR-I00 training, we first pad pixels with four zeros and randomly crop the image to 32 x 32. For Tiny- Imagee,,,,,et, images are rotated randomly by up to 20 degrees. For ImageNet, all training images are resized to 256 256, then randomly cropped to yield an input training patch of 224 x 224. For all datasets except SVHN, training images are also horizontally flipped with probability 0.5. Finally, all images are normalized to mean and unit standard deviation. During evaluation, do not perform data augmentation for all models.
For the SmaBINORB dataset and following the training protocol of Hinton al. (2018), all images are resized to 48 x 48. During training, each image is randomly cropped to yield an input training patch of 32 x 32. Then, we horizontally flip the image with probability 0.5, and randomly jitter the brightness and contrast of the images. During inference, we crop the center of the test image to yield 32 32 patch.
A.2 EFFECT OF THE INDUCING MIXTURE OF GAUSSIANS (IMoGs)
Here we evaluate the role of the IMoG performance by comparing the performance of our ar- chitecture (i.e. IMoG with log-likellng log-likelihood attention) to a CapsNet which adapts the inducing points (IPs) with dot-product attention from Set Transformers. We see that the increased complexity of the task positively correlated with the performance gains from the IMoG.
Table Comparison of the mean test classification error rates (+-std.) of the proposed Trans-Caps equipped with IMoGs with log-likelihood attention to IPs with dot-product attention.
SVHN
CIFAR-10
CIFAR-100
IMoGs with log- ligeliikl attention
3.02.10.14
6.56.10.16
25.1710.21
IPs with dot-prote dot-product attention
3.08.10.18
7.1310.21
28.7510.32
A.3 EFFECT OF NUMBER OF ATTENTION HEADS
We evaluate the effect of the number of attention heads on the performance of the proposed archi- tecture (see T Our results show that varying the number of heads may alter performance due to the changes in the capsule type's 's encoded features.
Table 5: The mean test classification error rates (d:std.) of the proposed Trans-Caps with SAR as function of the number of attention heads.
# of heads
SVHN
CIFAR-10
CIFAR-100
1
3.024:0.14
6.68+0021
25.32+4.24
2
3.1311013
6.56110.16
25.1710.21
4
3.2310.17
6.6910.19
25.42:1025
8
3.1210014
6.76100.2
25.38
16
3.1540.13
6.76440.20
25.30
11

---

Under review as a conference paper at ICLR 2020
A.4 MODEL CONFIGURATIONS
The following tables depict the architectures and parameter setups for the trained models discussed in the paper. Note that ConvCaps and FCCaps layers represent the convolutional capsule layers and fully-connected capsule layers, respectively.
Table 6: Architecture of the proposed CapsNet with SAR for SVHN and CIFAR-10.
Name
Operation
Output size 8x8x
Backbone 64)
ResNet- 20 (impul
duiput_
Conv, input dim=64, output outpur_dim=5122 stridc=1, padding=1
paddin=,
Pr PrimaryCap (B = 32)
8 x 32 4 x
reshape
4-dim cupsules
ConvCaps (C' 32)
3 ConvCaps SAR to 32, x 4-dim capsules, stride=2, padlingg=
4x4x 32 4
ClassCaps
FCCaps SAR to 10. 4 x 4-dim. capsules
10 1x
Classifier
input.dim= 16, output colput.dim=1, linear
10 1
Table 7: Architecture of the proposed CapsNet with SAR for CIFAR-100.
Name
Operation
Output size 8x8x
Backbone 128)
ResNet-32 (nputt_dim=3, output_dim==
3 Conv, input dim= 18. output dim=512, strid==1, padding=1
Pr PrimaryCaps (B 32)
8 4x4
reshape 32, 4-dim capsules
ConvCape1 (C 32)
3 ConvCaps SAR 32, 4 x 4-dim capsules, stride=2, padding=1
4 4 32 4
ConyCapa2 (D> 32)
4 x ConyCaps SAR 32, 4 x 4-dim capsules, stride=2, padding=1 FCCaps SAR to 100, 4 x 4-dim. capsules
32 x 4
C1 ClassCaps
100 x
Classifier
coput_dim=I1ee 16, curpur_dim=),, lincar
100 x
12

---

Under review as a conference paper at ICLR 2020
Table 8: Architecture of the proposed CapsNet with SAR for Ina-lmmaeNee.
Name
Operation
Output
Backhone (4 128)
ResNet-32 (input chm=3, output dim= 128)
16 16 128
3 x 3 Conv, impu dipu- 128, ouipui_dhm=3 d1m=512, stride= pudding=1
PrimaryCapa (B 32)
reshupe to 32, 4 4-dim capsules
x
ConvCapal (C 32)
ConvCaps SAR 32, x 4-dim capsules, stride=2. padding=1
8 4
ConvCaps2 (D) 64)
ConyCaps 64, 4-dim capsules, sinde=2, pudding=1
ClassCaps
FCCaps SAR 200, 4 4-dim. capsules
2000 x
Classifier
inpur_dim=16, output_dimel linear
200
Table 9: Architecture of the proposed CapsNet with SAR for ImageNet.
Name
Operation
Output size
Backbone (A 1024)
ResNet-50 (inpul_thm=3 output_ 1024)
14 1024
Conv, input.dim= dimn 1024, wuiput.dmm=lle 1024, stride= pudding=0
PrimaryCaps (B 64)
reshape to 64, 4 4-dim capsules
14 x 14 x 64 x4x4
ConvCaps1 (C = 128)
3 x ConvCaps SAR to 128, x 4-dim capsules, stride=2, padding=1
7 x 128 x 4x4
ConvCape2 (D 128)
3 ConvCaps SAR to 128, 4-dim capsules, strid=el, padding=0)
5x x 128 xxx
ClasaCapa
FCCaps 1000, 4 x 4-dim. capsules
1000 x
Classifier
input dim= output dim=1, linear
1000
Table 10: Architecture of the proposed CapsNet with SAR for SmalINORB.
Name
Operation
Output size
Backbone (1 64)
5 x 5 Cony, inpul dim=1, oulpul diim=64, stride=2, padding=1 Cony, reshape
15 15 64
output dim= stridc=1, padding=0
PrimaryCaps (B- 8)
15 x 15 x x
4-dim capsules
ConvCapal (C' 16)
3 ConyCaps SAR 16, 4-dim cupsules. stride=2, padding=0
7 16x4x4
ConvCapa2 (D) 16)
3 x ConvCaps SAR 4 4-dim. capsules, stride=l, padding=0
5 5
ClaseCaps
FCCaps SAR to 5, 4 4-dim, capsules
5
Classifier
impur_dhm=16 16, output dimml,, linear
x
Table II: Architecture of the baseline CNN used in SmallNORB experiments.
Name
Operation
Output size
Coav1
5 Conv, input dim=1, output dim=64, stride=2, pudding=1
ReLU
x 15 64
Conv2
1 x Cony, input dim=64, output dim= stride=1, padding=0+ BN ReLU
15
128
Conv3
3 Cony, input 128m output dim=256, stride=2, padding=0 BN ReLU
output.dim=266,
7x 256
Conv4
3 3 Cony, input 256. cutput_dim=2566 stride=1, padding=0 BN ReLU
5 256
AvgPool
5 glohal average pooling + flatten
256
Classifier
dim=t_di ourput_dim=5, linear
13


Reviews:
[{"review_id": "wIM8uB9fim3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to swap the typical routing mechanisms in capsules for a more standard attention mechanism. The attention mechanism is based on computing similarity scores using gaussians instead of dot-products . The authors show that this leads to better downstream performance of more natural tasks while preserving robustness to viewpoint changes, one of the main strengths of capsules.\n\nMy main concern is that the proposed solution is not a good fit for the general framework of capsules. One of the key ideas behind capsuels is that objects are made up of parts which are uniquely assigned to exactly one object. This assumption is broken in this paper (if I understood the paper correctly). That is, the attention mechanism doesn't enforce any kind of competition between objects for individual parts. Hence, this model should be thought of as a stacked set transformer instead. The results indicate that this is not necessary for downstream performance (it actually helps) and the models still remain somewhat robust to view point changes on synthetic data (NORB).\n\n== Detailed Comments and Questions ==\n\nWhere is the \"self\" in self-attention? IIUC there is no attention between individual states of one layer, not even indirectly. There is only attention between some learnable \"object\" states (inducing points) to their individual parts.\n\nWhy is it necessary to model the likelihood of a part state in this model, when all we care about is the similarity between an objects inducing vector and a part representation. Why not using just simple dot-product attention to compute the similarities? The paper says it is used to \"encode the second order interactions among points\". What does that mean? Actually, the log probability used as similarity score can be written as a dot-product (ignoring the norms of the vectors) + some bias over j which doesn't matter when computing the softmax later on.\n\nI don't understand the difference to self-routing capsules (Hahn et al.). The paper says they use stationary routing weights to specific locations, but I don't see how this approach differs from that? The routing weight in Hahn et al. basically computes dot products between inducing points (the rows in the learnable routing weight matrix) and the output of the part capsule which are used as similarity scores. The only difference here is that gaussians are used to model similarity, which, as I explained above is pretty much the same as a dot-product. Something that's different is the use of multiple heads and using a softmax over parts instead of over objects, which brings me to my biggest concern.\n\nI am also not sure that this architecture fits well into the Capsules framework. CapsNets make the assumption that each part belongs to a single object. That's also the reason why typically there needs to be some iterative procedure to compute probable assignments. Here, however, parts can be part of multiple objects, because the softmax is taken over the parts and there are multiple heads. This kind of defeats the purpose of capsules.\n\nUltimately, the architecture is basically a stacked version of the set transformer with some potentially interesting deviations, and it should be presented as such with the necessary ablations. I think it is interesting to see that \"competition\" between objects for the individual parts is not necessary to achieve similar or better performance to capsnets, to achieve good performance on down stream tasks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "t8vwhjndEG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use the self-attention to find the agreement among the capsules of consecutive layers of a capsule network instead of iterative routing procedure. To reduce the computational and memory complexity of self-transformer capsules of each layer are considered as a set and follow set-transformer and apply inducing mixture of Gaussian distributions to compute the agreement among capsules of layer L and L+1. \nThe proposed method have been evaluated on multiple dataset including large scale datasets like ImageNet and improved the performance compared to the baselines (except for  ImageNet)\npros:\n1- The idea is interesting and the paper is well written and easy to follow.\n2- The proposed method seems to be scalable to the large datasets. \n3- It achieves significant improvement compared to the baselines on the novel view point of SmallNorb.\n\ncons(clarification)\n1- There are related papers that are missing in the comparison section for instance \n-Capsule routing via variational Bayes.\n2- Why is the performance the proposed method on the ImageNet is even less than the baseline Resnet50. \n3- It seems that increasing the number of attention head negatively affect the performance, what is your justification for that?\n4- Results in the table 3 is counterintuitive. The proposed method has a significant improvement compare to the baseline on the novel view point of SmallNorb while it is on par with other method in familiar viewpoint.\n\nPost Rebuttal\n\nThanks for the author(s)' responses. The rebuttal addressed some of my questions. I have a couple of  suggestions : \n1- Your proposed capsule network is not the first one that is applicable on large scale datasets like ImageNet, there are other capsule networks that are applicable on real world scenarios and also ImageNet dataset with improvements over the baseline \n- Dual Directed Capsule Network for Very Low Resolution Image Recognition (ICCV 2019)\n-  Subspace Capsule Network (AAAI 2020)\nplease refer to them and also give intuitions about why your proposed Trans-Caps is not performing well on ImageNet. The intuition and analysis is valuable to the community.  \n\n2- To support the generalizability claim of Trans-Caps, I highly recommend reporting results on Multi-MNIST and also affNIST. Specially when you train the model on MNIST and test it on these two datasets. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TGnULJ4afQ9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a novel Capsule connection which is inspired by set transformer and induced points. They introduce log-likelihood based attention based on Gaussians centered at trainable fixed queries. They calculate the routing factors (attention) using the probability of a key projection of votes under trainable gaussians. Then they add the values weighted by the routing factor to the fixed mean. Their choice of modifying the mean based on the input rather than replacing it is quite interesting. It resonates with the concept of momentum as well. I am curious how much it affects the stability and convergence of their technique. \nAfterwards they linearly transform and add a skip connection + relu to get the output capsule parameters.\nIn this work the activation probability of the capsules are not calculated alongside the pose matrices. For the sake of classification (which needs the activation probabilities) they have an extra fully connected layer + cross entropy.\nTheir method surprisingly can generalize to new viewpoints, backed by experiments on smallNorb azimuth and elevation generalization much better than the CNN and previous Capsule Networks. \nAlso by removing the iterative routing and replacing it with trainable parameters they are able to achieve competitive results on Cifar10-Cifar100-tiny imagenet and imageNet. \n\nPros:\nThe paper is very well written and easy to follow. They provide a convincing set of experiments on reasonable datasets. Their method is novel and intuitive.\nCons:\nLack of ablation study to show the importance of their novel method IMoG. One baseline is just the attention used by set transformer. Essentially what is the effect of having a Gaussian (standard deviation). Is it necessary to modify the mean vs replacing it. \nIt would be more convincing to add attention based vision models (bello et al 2019) to the tables for cifar10-cifar100.\n\n\n\n------------------------Post Author Response\n\n\nThank you for adding the ablation study and the attention based models. I enjoyed reading your work and it has answered some of the questions we wanted to explore in Capsule Networks. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kb2GvJRoWqF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission details a novel technique to learn the routing in capsule networks for image classification tasks. Connecting capsules in such architectures typically requires iterative approaches which are computationally expensive. The main idea in this submission is to leverage a non-iterative attention mechanism to learn this routing and thus decreases computational cost. Furthermore, the experiments indicate that the proposed architecture leads to higher accuracy on a number of image classification tasks and datasets.\n\nOverall, the submission is well written and easy to follow. The discussion of related work is thorough and to my best knowledge appears to be complete.\n\n---\n## Pros:\n\n   * The paper clearly presents the proposed method, and it is easy to follow. \n   * The results show significant improvements over the baselines, especially on the Tiny-ImageNet dataset and the SmallNORB dataset in the case of novel poses.\n   * The proposed learning algorithm does not require expensive iteration and thus allows for better scaling.  \n   * The good results across several datasets suggest that the proposed model can be an important contribution to the field.\n\n--- \n## Cons:\n\n   * The authors discussion of related work mentions several papers that combine CapsNets with an attention mechanisms to address the issue of routing. \n   * However, the authors only provide a comparison to Hahn et al. (2019). The authors state that other methods are memory intensive and require knowledge of the number of concurrent iterations as an additional hyperparameter, which is probably why they do not compare to these methods. \n   * Although an important property, it would be beneficial to provide a comparison to other attention-based CapsNets. \n   * Furthermore, it would be interesting to compare to state-of-the-art image classification methods that are not based on the capsules concept. Adding these additional experiments would help readers to have a better overview of the task and where the proposed approach stands in the broader context of classification approaches. \n   * In other areas of deep-learning and representation learning in particular, use of GMMs in the architecture, e.g., in latent space, (as opposed to usage as a probabilistic output model) can lead to code-book collapse. I.e., it is hard to balance during pure back prop learning how to pick the correct gaussian to sample from and to shape it  (or in other words: should one update the parameters of a mixture component or penalize the selection of that component if the action leads to a high loss in the forward pass). It would be interesting to see more details on this aspect of the architecture and training. I.e. how was the number of mixtures selected, how does this influence performance? etc. \n\n--- \nSummary:\nOverall, there is much to like about this submission but some questions remain (see cons). I'm looking forward to the authors' response.  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new variant of capsule networks, where iterative routing is replaced by an attention-based procedure inspired by Induced Set Attention from Set Transformers. The method is competitive on several classification benchmarks and improves generalization to unseen views on SmallNORB.

The reviewers note that the method is presented well (R2, R3, R4), is more scalable than other capsules variants (R3, R4), and the results are good (R1, R2, R3, R4). However, the reviewers also point out missing relevant baselines (R2, R3, R4), limited amount of generalization experiments (R3), and issues with the positioning of the method and the details of the formulation (R1). In particular, R1 did a very thorough job at reading the paper and discussing with the authors.  The issue with missing baselines has been satisfactorily addressed in the updated version of the paper.

Considering all this feedback and after reading the paper myself, I would summarize the pros and cons of the paper as follows.

Pros:
1. Good presentation
2. The method is more scalable than prior capsule-based models
3. Competitive results on several small- to mid-scale classification datasets
4. Good results on viewpoint generalization on SmallNORB 

Cons:
1. Classification results on all datasets are worse than non-capsules models (SE-ResNet, AA-ResNet). I could not find a discussion of this fact either in the paper, or in the authors’ responses. Given this fact, superior generalization (or some other nice properties) would be a potential advantage of the proposed model. Which leads to the next point.
2. Generalization results on SmallNORB are encouraging, but it is just a single dataset. If these results are key to showing the benefit of the method (as argued in the previous point), it is crucial to demonstrate this generalization in more settings, e.g. at least on MultiMNIST and AffNIST, as suggested by R3.
3. Scalability of the method is only studied in limited detail (I do appreciate Figure 2). The best indication in the direction of scalability is that the model can be trained on ImageNet (which is great), but it performs worse than the ResNet-50 used as a backbone and it is not explained why (even after one of the reviewers asked about it) and how expensive computationally the model is.
4. I share the concerns of R1 regarding the use of the term “MoG”. It is a mathematical term, so one would expect mathematical precision when using it.
  4a. It is unclear how the mixing probabilities \phi are learned (IIUC they get no gradient, as described by R1) and if they are in some way actually learned, it is unclear how it is guaranteed that they sum to one.
  4b. MoG usually comes with the standard procedure of fitting it to data (EM), which IIUC the authors are not following here. This should be clearly explained. 
5. A relatively more minor concern: again, as pointed out by R1, the use of “self-” in “self-attention” does not seem accurate. Self-attention assumes inputs to the attention procedure attend to themselves in some sense. As one consequence, the output sequence has the same length as the input sequence. ISAB from Set Transformer can be seen as a factorized version of self-attention where first inducing points attend to the inputs and then the inputs attend to the inducing points, so the output of the whole block is still the same length as the input. But in the proposed model this second step of going back to the inputs is absent and the length of the output sequence is generally different from the length of the input sequence.

Note:
I partially share the doubts R1 raised on the positioning of the method as “capsules” as opposed to “attention”, but I believe it is not the authors’ fault that the definition of what capsules are is historically vague and that this term has been used in many different ways in the past. I would strongly recommend to discuss this point in the updated version of the paper and I hope the capsules community manages to get more clarity on what exactly capsules are. But I do not count this point as a weakness here.

Based on all this evidence, I recommend rejection at this point. The paper has its merit, but it has unfortunate gaps both on the experimental and the presentation sides, as listed above. Some of these have been mentioned during the discussion phase, but the authors have not quite addressed them. There is no mechanism to ensure these are fixed in the final version, so resubmission to a different venue is the only option.

Decision: Reject