Building Deep Equivariant Capsule Networks
Authors: Sai Raam Venkataraman
S. Balasubramanian
R. Raghunatha Sarma
Venue:  (2020)
Keywords: Capsule networks
equivariance

Abstract: Capsule networks are constrained by the parameter-expensive nature of their layers, and the general lack of provable equivariance guarantees. We present a variation of capsule networks that aims to remedy this. We identify that learning all pair-wise part-whole relationships between capsules of successive layers is inefficient. Further, we also realise that the choice of prediction networks and the routing mechanism are both key to equivariance. Based on these, we propose an alternative framework for capsule networks that learns to projectively encode the manifold of pose-variations, termed the space-of-variation (SOV), for every capsule-type of each layer. This is done using a trainable, equivariant function defined over a grid of group-transformations. Thus, the prediction-phase of routing involves projection into the SOV of a deeper capsule using the corresponding function. As a specific instantiation of this idea, and also in order to reap the benefits of increased parameter-sharing, we use type-homogeneous group-equivariant convolutions of shallower capsules in this phase. We also introduce an equivariant routing mechanism based on degree-centrality. We show that this particular instance of our general model is equivariant, and hence preserves the compositional representation of an input under transformations. We conduct several experiments on standard object-classification datasets that showcase the increased transformation-robustness, as well as general performance, of our model to several capsule baselines.

Published as a conference paper at ICLR 2020
BUILDING DEEP EQUIVARIANT CAPSULE NETWORKS
Sai Raam Venkataraman, S. Balasubramanian R. Raghunatha Sarma
Department of Mathematics and Computer Science Sri Sathya Sai Institute of Higher Learning (vsairaam, sbalasubraaaaaaa www.aaaubaaaanaa rraghunathasamma @sssihl edu. in
ABSTRACT
Capsule networks are constrained by the parameter-expensivee nature of their layers, and the general lack of provable equivariance guarantees. We present variation of capsule networks that aims to remedy this. We identify that learning all pair- wise part-whole relationships between capsules of successive layers inefficient. Further, we also realise that the choice of prediction networks and the routing mechanism are both key to equivariance. Based on these, we propose an alternative framework for capsule networks that learns to projectively encode the manifold pose-variations, termed the opacitof-variation (SOV), for every capsule-type of each layer. This is done using trainable, equivariant function defined over a grid of troup-tranfformtiion.. Thus, the prediction phaase of routing involves projection into the SOV of deeper capsule using the corresponding function. As specific instantiation of this idea, and also in order to reap the benefits of increased parameter-sharing we use type-homogeneous group-equivariant convolutions of shallower capsules in this phase. We also introduce an equivariant routing mech- anism based on degree- centralcen....ity.. We show that this particular instance of our general model is equivariant, and hence preserves the compositional representation an input under transformations. We conduct several experiments on standard object-classificatio datasets that showcase the increased transformation-robussii as well as general performance, of our model to several capsule baselines.
INTRODUCTION
The hierarchical component -struutcnttttttuureeee of visual objects motivates their description as instances of class-dependent spatial grammars. The production-rules of such grammars specify this structure by laying out valid type-combinations for components of an object, their inter-geometry, as well as the behaviour of these with respect to transformations on the input. A system that aims to truly understand a visual scene must accurately learn such grammars for all constituent objects - effect, learning their aggregational structures. One means of doing so is to have the internal representation of a model serve as component praponnnt-pprsing of an input across several semantic resolutions. Further, in order to mimic latent compositionallties in objects, such representation must be reflective of detected strengths of possible spatial relationships. A natural structure for such a representation is parse-tree whose nodes denote components, and whose weighted parent-child edges denote the strengths of detected aggregational relationships.
Capsule networks (Hinton et 2011), (Sabour et al. 2017) a family of deep neural networks that aim to build such distributed, spatial spatially-aware ly-aware representations in a multi-class setting. Each layer of capsule network represents and detects instances of set of components (of visual scene) at a particular semantic resolution. It does this by using vector-valued activations. termed 'capsules', : Each capsule is meant to be interpreted as being representative of set of generalised corrdioodsssses for a visual object. Each layer consists of capsules of several types that may be instantiated at all spatial locations depending on the nature of the image. Thus, given an image, capsule network provides description of its components order that this distributed representation across layers be an accurate caaponent-parsinng of a visual scene, and capture meaningful and inherent spatial relationships, deeper capsules constructed from shallower capsules using a mechanism that combines backpropagation-ben based learning, and consensus-based heuristics.
various 'levels' of semantics. In

---

Published as a conference paper at ICLR 2020
Briefly, the mechanism of creating deeper capsules from set of shallower capsules is follows. Each deeper capsule of a particular type receives a set of predictions for its pose from a local pool of shallower capsules. This happens by using set of trainable neural networks that the shallower capsules are given as input into. These networks can be interpreted as aiming to capture possible part-whole relationships between the corresponding deeper and shallower capsules. The predictions thus obtained are then combined in a manner that ensures that the result reflects agreement among them. This is so that capsules are activated only when their component-cassiee relationship to form an instance of the object-type it represents. The agreement- based aggregation described just now termed 'routing'. Multiple routing algorithms exist, for example dynamic routing (Sabour et al. 2017), EM-routing (Hinton et al. 2018). SVD-based routing (Bahadori 2018 and routing based on a clustering-like objective function (Wang & Liu 2018
are in the right spatial
Based on their explicit learning of compositional structures, capsule networks can be seen as an alternative (to CNNs) for better learning of compositional representations. Indeed, CNN-based models do not have an inherent mechanism to explicitly learn or use spatial relationships in a visual scene. Further, the common use of layers that enforce local transformation intiriinneiiri,eeeee, such as pooling, further limit their ability to accurately detect compositional structures by allowing for relaxations in otherwise striet spatial relations (Hinton et al. (011). Thus, despite some manner of hierarchical learning- as seen in their layers capturing simpler to more complex features as a function of depth CNNs do not form the ideal representational model we seek. It is our belief that capsulc-based models may serve us better in this regard.
This much said, research in capsule networks is still in its infancy, and several issues have to be overcome before capsule networks can become universally applicable like CNNs. We focus on two of these that we consider as fundamental to building better capsule network models. First, most capsule-network models, their current form, do not scale well to deep architectures. A significant factor is the fact that all pair-wise relationships between capsules of two layers (upto local pool) are explicitly modelled by a unique neural network. Thus, for a convolutional capsule' layer- - the number of trainable neural networks depends on the product of the spatial extent of the windowing and the product of the number of capsule-types of each the two layers. We argue that this design is not only expensive, but also inefficient. Given two successive capsule-layers, not all pairs of capsule-types have significant relationships. This is due to them either representing object-component that are part of different classes, or being just incompatible in compositional structures. The consequences of this inefficiency go beyond poor scalability. For example, due to the large number of in this design, only simple functions- often just matrices are used to model part-whole relationships. While building deep capsule networks, such a linear inductive bias can be inaccurate in layers where complex objects are represented. Thus, for the purpose of building deeper architectures, as well as more expressive layers, this inefficiency in the prediction phase must be handled.
The second issue with capsule networks is more theoretical, but nonetheless has implications in practice. This is the lack, in general, of theoretical guarantees on equivariance. Most capsule networks only use intuitive heuristics to learn transformation-rrbuus spatial relations among components. This is acceptable, but not ideal. A capsule network model that can detect compositionalitiss in a provably- invariant manner are more useful, and more in line with the basic motivations for capsules.
Both of the above issues are remedied in the following description of our model. First, instead of learning pair-wise relationships among capsules, we learn to projectively encode a description of each capsule-type for every layer. This we do by associating each capsule-type with vector- valuud function, given by a trainable neural network. This network assumes the role of the prediction mechanism capsule networks. We interpret the role of this network as a means of encoding the manifold of legal pose-variations for its associated capsule-type. It is expected that, given proper training, shallower capsules that have no relationship with a particular capsule-type will project themselves to a vector of low activation (for example, 2-norm), when input to the corresponding network. As an aside, it is this mechanism that gives the name to our model. We term this manifold the space-of-vare variiaiio of a capsule-type. Since, we attempt to learn such spaces at each layer, name our model oppcce-f--variaton networks (SOVNET). In this design, the number of trainable networks for given layer depend on the number of capsule-types of that layer.
2

---

Published as a conference paper at ICLR 2020
As mentioned earlier, the choice of prediction networks and routing algorithm is important to having guarantees on learning transformation ianiforrmationnnnaiiaan invariant compositional relationships. Thus, in order to ensure equivariance, which we show is sufficient for the above, (GCNN) (Cohen & Welling, 2016) in the prediction phase. Thus, shallower capsules of a fixed type input to GCNN associated with deeper capsule-type obtain predictions for it. Apart from ensuring equivariance to transformations, GCNNs also allow for greater parameter-sharing (across a of transformations), resulting in greater awareness of local object-structures. We argue that this could potentially improve the quality of predictions when compared to isolated predictions made by convolutional capsule layers, such those of (Hinton et al. 2018).
group-equivariat convolutions
The last contribution of this paper is an equivariant degree-centrality based routing algorithm. The main idea of this method to treat each prediction for a capsule as a vertex of graph, whose weighted edges are given by a similarity measure on the predictions themselves. Our method uses the softmaxed values of the degree scores of the affinity matrix of this graph as a set of weights for aggregating predictions. The key idea being that predictions that agree with a majority of other predictions for the same capsule get a larger weight- following the principle of routing-by wuuting-by-agreeme -ggggeement While this method only heuristic in the sense of optimality, provably equivariant and preserves the capsule-decompositio of an input. We summarise the contributions of this paper in the following:
1. A general framework for scalable capsule-network model.
2. A particular instantiation of this model that uses equivariant convolutions, and an equivariant, degree-centrality-boe
routing algorithm.
3. A graph- based framework for studying the representation of capsule network, and the proof of the sufficiency of equivariance for the (qualified) preservation of this representation under transformations of the input.
4. A set of proof-of-concept, evaluative experiments affinely transformed variations of MNIST, FASHIONMNIST, and CIFAR1O, as well as separate experiments on KMNIST and SVHN that showcase the superior adapatability of SOVNET architectures to train and test-time geometric perturbations of the data, as well as their general performance.
2 SOVNET, EQUIVARIANCE, AND COMPOSITIONALITY
We begin with essential definitions for a layer of SOVNET, and the properties we wish to guarantee. Given a group (G, o), we formally describe the 7th layer of SOVNET architecture as the set of function-tur tuples {(f) a!) 0 < is Ni 1; f G + ; G + [o, 1]}. Here, Ni denotes the number of capsule-types at the 7th layer, f is a functional description of the d'-dimensional pose- vectors of instances of the 7th capsule-type. and at is a functional description of the corresponding activations.
We model each capsule-type function over group of transformations so to allow for formal guarantees on wnwafformatioeequivvriraa to representation-spaan the group in question. This is similar in approach to (Cohen & Welling 2016). We wish for each capsule-type, both pose and activation-wise to display equivariance. We present formal definition of this notion.
Thus, we also model images as function from a group The main assumption being that the translation- grruup is a subgroup of
Consider a group (G, o) and vector spaces V. W. Let I' and T"' be two group-representations for elements of G over V and W. respectively. Φ: V + W is said to be equivariant with respect to T and T' if Vig E G, Vx E V, ((TTx) (z).
This definition translates something that allows no loss of information in compositional structures. As in (Cohen & Welling 2016), we restrict the notion of equivariance in our model by using the operator Lg in place of the rroup-representatioo.. Lg is given by [xr)((r) f(g x). Thus, if & denotes operation between two functions, we require ([L,fl f] &1) = [L, & v (z). The operator & describes the change in representation space, and is dependent on the nature of the deep learning model. In the case of
preservation transformations in the input-space to the output-space
3

---

Published as a conference paper at ICLR 2020
capsule networks (and SOVNET), this change is given by routing among capsules described in subsection 2.
2.1 SOVNET LAYER
We define the capsule-types a particular layer as an output of an agreement- based aggregation of predictions made by the preceding layer. A recursive application of this definition is enough to define a SOVNET architecture, given an initial set of capsules. means of obtaining this initial set is given in section 3] We provide general framework for the summation-basee family of routing procedures in Algorithm
Algorithm A general summation-basedd routing algorithm for SOVNET.
Input: {(A d')li E (O...., N₁ 1. G R : [0
Output: {( aaj t/+))) {0, No+1 1),
Rd a :G
H+]
-
(0,1]
Trainable Functions: (1)+1, ..) projection networks that use operator
(g) ((f),a) / / v Vg E
(co [+1 (g) ..... Ni (
1+1
GetWeights
(g). sd+
f (g) = E c (0)) Vg
G
(g) = Agreemend((5t (4), Stt (g) s4+1
(
The weighted-sum family of routing algorithms builds deeper capsules using a weighted sum of predictions made for them by shallower capsules. To ensure that the predictions are combined in meaningful manner, different methods can be used to obtain the weights. The role of the function GetW eights is represent any such mechanism. The activation of a capsule, representative of the probability of existence of the object it represents, is determined by the extent of the consensus principle of capsule networks. The
among predictions. This is based on the Agreement function represents any means of evaluating such consensus.
We instantiate the above algorithm to a specific model, as given in Algorithm 2 In this model, the V are group-equivarian convolutional filters, and the operator is the corresponding group- equivariant correlation operator *. The weights <3 J+1 (g) are, in this routing method, the softmaxed degree-scores of the affinities among predictions for the same deeper capsule. Further, like in dynamic routing (Sabour et al. (017), we also assume that the activation of a capsule is given by its 2-norm. To ensure that this value is [0, 1}, we use the 'squash' function of dynamic routing. Thus, we do not mention it explicitly. Note that have used the subscript notation to also denote that variable is part of vector, for example Stjp (g) denotes the nd element of the d'+1 -dimensionl s!+1 vector is (g). This new routing algorithm meant serve as an alternative to existing iterative
routing strategies such dynamic routing. An important strength of our method being that there is no hyperparameter, like that of the number of iterations in dynamic routing or EM routing.
2.2 EQUIVARIANCE, COMPOSITIONALITT AND SOVNET
The SOVNET layer we introduced Algorithm 2 is group-equivariant with respect to the group action Lg. where g G - the set of transformations
which the group-convolution is defined. For notational convenience, we define & to be an operator that encapsulates the degree- degree-routing procedure with prediction networks w/++1 Thus, the jth capsule-type of the 1+ th layer is functionally depicted ,|+1 (F* & 1 where Fl (f): (k, fk-1)) The formal statement of this result is given below; the proof is presented in the appendix.
Theorem 2.1. The SOVNET layer defined in Algorithm 2 and denoted by the operator & as given above, satisfies (LLFF') @ ///+1 = (L, [F* & w where belongs to the underlying group of the equivariant convolution.
Proof. The proof is given in the appendix.
4

---

Published as a conference paper at ICLR 2020
Algorithm 2 The degree- based routing algorithm for SOVNET.
Input: {ffli {0, ..., Ni 1), - R
Output: U (0,..., Ni+1 - G
Trainable Functions: (4 +).0 j S M+1 -1 1,- a set of d group-equivariant convolutional
filters (per capsule-type) that use the group-equivariat correlation operator *
sip (g) = (f: * * 3 ()) = Eneg Ed-
(t p {0...,, d - -1} 1.-15(9) (g)) DegrreSScree S
(g) .... d+1
(S) (g), S S -,-,,,,));;;;; (g)); Sj N1+1 1,
f/t (g) Ei 6/ (g) (g) 0A j <
I1 (g)+=
f)+1(() (g) = Squash
1+ 1 (0)l f3 (0):00 11
procedure DEGREESCORE(S bj
( (,-,(())
(o). (0)
(9 |st [a)l 211 (a)lla
S i,k S Ni
Degree, (g) - >i - (A
cij(1) E:- exp (a)
exp(Degree)(a)
0
return (g) 40 MiSN-1
Equivariance is widely considered a desirable inductive bias for variety of reasons. First, equivariance mirrors natural lahel-invariance under transformations. Second, it lends predictability to the output of network under (fixed) transformations of the input. These, of course, lead a greater robustness in handling transformations of the data, We aim at adding to this list by showing that equivariance guarantees the preservation of detected compositionalities in a SOVNET architecture. This is of course quite unsurprising, and has been a significant undercurrent of the capsule-network idea. Our work completes this intuition with a formal result.
We begin by first defining the notion of a capsule-decomppsition graph. This graph formed from the activations and the routing weights of a SOVNET. Specifically, given an input to a SOVNET model, each capsule of every type is a vertex in this graph. We construct an edge between capsules that are connected by routing, with the direction from the shallower capsule to the deeper capsule. Each of these edges are weighted by the corresponding routing coefficient. Capsules not related to each other by routing are not connected by an edge. This graph is a direct formalisation of the various detected compositionalities with their strengths.
What should the ideal behaviour of this graph be under the Shange-of-vieepping of an input? The answer to this lies in the expected behaviour of natural compositionalities. Thus, while the pose of objects, and their components, is changed under transformations of the input, the relative geometry is constant. Thus, it is desirable that the capsule-decomposition graphs of a particular input (and its transformed variations) be isomorphic each other. We show that SOVNET model that is equivariant with respect to set of transformations satisfies the above property for that set. A more formal description of the capsule-decompposition graph, and the statement for the above theorem are given below.
Consider an L-layer SOVNET model, whose routing procedure belongs to the family of methods given by Algorithm []] Let us consider a fixed input x: G - RR. We define the
graph of such model, for this input x, as G(x)
((),) Here, V(x) and E(x) denote the
N; 1,0 < 1, E G}. where fl()) (g,i, fl al()))((())) 9 G. < N₁ E(x) = (02) Gj (g2)) (gz) 91 E + (g2)). Poot (oz) denotes the pool of grid-positions at layer / that route to the deeper capsule of type j of layer /+ at 92. A more formal definition is given the appendix. We also use the notation La. f. (g) to denote (h-1 o g, i, f (1 g), d/(h-1 o g))-
vertex-set and the edge-set, respectively. V(x)
Theorem 2.2. Consider an L-layer SOVNET whose activations are routed according to a procedure
belonging to the family given by Algorithm 7 Further, assume that this routing procedure is
5

---

Published as a conference paper at ICLR 2020
equivariant with respect the group G. Then, given input and Vg G. G(æ) and G((ga)) are isomorphic.
Proof. The proof is given the appendix.
Based on above theorem, and the fact that saggree-centality based routing is equivariant, the above result applies SOVNET models that use Algorithm 2
EXPERIMENTS AND RESULTS
This section presents a description of the experiments we performed. We conducted two sets of experiments; the first to compare SOVNET architectures other capsule network baselines with respect to transformation robustness on classification, and the second to compare SOVNET to certain capsule as well as convolutional baselines based on classification performance. Before we present the details of these experiments, we briefly describe some details of the SOVNET architecture we used. We only present an outline the complete details, both architecture- wise and about the training, can be found in the anonymised github repository https:: /ttpps:/giithbb. com/ sai r amVenkat raman/ SOVNET
The first detail of the architecture pertains to the construction the first layer of capsules. While many approaches are possible, we used the following methodology that is similar in spirit other capsule network models. The first layer of the SOVNET architectures we constructed use modified residual block that uses the SELU activation, along with group-equivariant convolutions. This is so as to allow meaningful set of equivariant feature maps to be used for the creation of the first set of capsules. Intuition and some literature, for example Rosario et al.] (2019), suggest that the construction of primary capsules plays a significant role in the performance of the capsule network. Thus, necessary build a sufficiently expressive layer that yields the first set of meaningful supsule-activations. To this end, each capsule-type in the primary capsule layer is associated with a group-convolution layer followed by a modified residual block. The convolutional feature- naatr from the preceding layer passes through each of these sub neb-newworss to yield the primary capsules. No routing is performed in this layer.
We now describe the SOVNET blocks. Since the design of SOVNET significantly reduces the number of prediction networks, and thereby the number of trainable parameters, we are able to build architectures whose each layer uses more expressive prediction mechanisms than simple matrix. Specifically, each hidden layer of the SOVNET architectures we consider uses a (group-equivarian)) modified residual block as the prediction mechanism. We use a SOVNET architecture that uses hidden layers for MNIST, FashionMNIST, KMNIST, and SVHN, and a model that uses hidden layers for CIFAR-10. Unlike DeepCaps- another capsule network whose predictions use (regular) convolution, each of the hidden layers our SOVNET models use degree-routing. The hidden layers of DeepCaps (excepting the last), in contrast, are not strictly capsule-based being just convolutions whose outputs are reshaped to a capsule-form.
The output capsule-layer of SOVNET is designed similar to the hidden capsule-layers, with the difference that the mediction-mechaniie connected layer. In order to make a prediction for the class of an input, the maximum across the rotational (and reflectional) positions of the two-norm of the capsule-activations of this layer are taken for each class-type. This is an equivariant operation, as it corresponds to the subgroup- poolinn of Cohen Welling (2016). The predictions that this layer yields is the type of the capsule with the maximum 2-norm.
is group-convolutional implementation of fully-
In order to guarantee the robustness translations and rotations, used the p4-convolutions (Cohen & Welling 2016) for the prediction mechanism in all the networks used in the first set of experiments. For the second set, used the p4m-convolution (Cohen & Welling, 2016), that is equivariant to rotations, translations and reflections for greater ability to learn from augmentations. The architectures, however are identical but for this difference.

---

Published as a conference paper at ICLR 2020
As in (Sabour et al. 2017]. we used a margin loss and regularising reconstruction loss train the networks. The positive and negative margins for half of the training epochs were set to 0.9 and 0.1, respectively. Further, the negative margin-loss was weighted by 0.5, as in (Sabour et al. 2017). These values were used for the first half of the training epochs. In order to facilitate better predictions, these values were changed to 0.95, 0.05, and 0.8, respectively for the second half of the training. We adopt this from (Rajasegaran et al. 2019). The reconstruction loss was computed by masking the incorrect classes, and by feeding the 'true' class-capsule to a series of transposed convolutions to reconstruct the image. The mean square loss was computed for the reconstruction and original image. The main idea being that this loss guides the capsule network to build meaningful capsules. This loss was weighed by 0.0005 as in (Sabour et al.) 2017). We used the Adam optimiser and an exponential learning rate scheduler that reduced the learning rate by a factor of 0.9 each epoch.
With this outline of the architecture and details of the training, we now describe the first set of experiments we conducted on SOVNET. The preservation of detected compositionalities under transformations in SOVNET leads us to the expectation that SOVNET models, when properly trained, will display greater robustness to changes in viewpoint of the input. Apart from handling test-time transformations, is the commonly held notion of transformation robustness, robust model must also effectively learn from train-time perturbations of the data. Based on these ideas, we designed a set of experiments that compare SOVNET architectures to other capsule networks on their ability to handle train and test-time affine transformations of the data.
Specifically, we perform experiments on MNIST (LeCun & Cortes 2010), FashionMNIST (Xiao iet al. all (017), and CIFAR-10 10 (Krizhevsky & Hinton 2009). For each of these datasets, we created variations of the train and test-splits by randomly transforming data according to the extents of the transformations given in Table [ We train given model on each transformed version of the training-split, and test each model on each of the versions of the test-split. Thus we obtain, for a single model, 25 accuracies per dataset- - each corresponding to pair of train and test-splits. There is a single modification to these transformations for the case of CIFAR-10 10, In order to compare SOVNET against the closest competitor DeepCaps, we use their strategy of first resizing CIFAR-10 images to 64x64, followed by translations and rotations.
We tested SOVNET against four capsule network baselines, namely Capsnet (Sabour et al. 2017 EMcaps (Hinton et al. 2018). DeepCaps Rajasegaran et al. (2019), and GCaps (Lenssen et al. 2018). The results of these experiments are given in Tables 2 4 In the majority of the cases, SOVNET
obtains the highest accuracy showing that it is more robust to transformations of the data. Note that we had to conduct these experiments as such a robustness study was not done in the original papers for the baselines. We used, and modified, code from the following github sources for the
implementation of the baselines: (Li) 2019) for CAPSNET; (Yang) 2019) for EMCAPS; (Rajasegaran 2019) and (Hopefullational) 2019) for DeepCaps, and (Lenssen 2019) for GCaps. We also tested against a group-equivariant convolution network (GCNN).
The second set of experiments we conducted, tested SOVNET against several capsule as well as convolutional baselines. We trained and tested SOVNET on KMNIST (Clanuwat et al., 2018)) and SVHN (Netzer et al. 201 With fairly standard augmentation mild translations (and resizing for SVHN to 64x64) the SOVNET architecture with pomm-oonollttions was able to achieve on-par, or above, comparative performance. The results of this experiment in Table In order to compare the performance of SOVNET architectures against more sophisticated RNN-baselines, we also trained ResNet- 18, ResNet-34 on the most extreme transformation translation by up to + 2 pixels, and rotation by up to + 180 The results of these experiments are presented in the appendix.
4 DISCUSSION AND RELATED WORK
A number of insights can be drawn from an observation of the accuracies obtained from the experi- ments. First, the most obvious, is that SOVNET is significantly more robust to train and test-time geometric transformations of the input. Indeed, SOVNET learns to use even extreme transformations of the training data and generalises better to test-time transformations in majority of the cases. However, in certain splits, some baselines perform better than SOVNET. These cases are briefly discussed below.
7

---

Published as a conference paper at ICLR 2020
Table 1: List of the extents for the affine transformations.
S.no.
Translational extent
Rotational extent
pixels
0
2
2 pixels
30
2 pixels
60
pixels
90
pixels
180
Table 2: Experiments on MNIST.
Results on
Training on
Intransformed MNIST
Results on Training on MNIST
Transformed by (2.30")
Method
(0,0")
(2.30')
(2,649)
(2,90")
(2,180")
Method
(0.0")
(2,30")
(2,60) )
(2,90")
(2,180')
Capsnct
99.35%
91.37%
72.10%
53.27%
42.58%
Capsnet
99.60%
99.39%
95 65%
79.53%
59.58%
EMcaps
99.09%
92.23%
72.83%
56.66%
42.95%
EMcaps
99.36%
99.03%
94.91%
79.12%
59.03%
G-Caps
97.83%
82,59%
66.27%
56.63%
54.52%
G-Caps
98.12%
96.17%
90.87%
81.34%
77.13%
DeepCaps
99.56%
94.61%
74.44%
57.24%
45.43%
DeepCaps
99.62%
99.57%
97.50%
84.16%
62.75%
GCNN
99.61%
93.96%
75.53%
58.91%
46.07&
GCNN
99.67%
99 29.46%
97.11%
84.5%
63.74%
SOVNET
99.68%
96.15%
80.53%
64.55%
51.02%
SOVNET
99.77%
99.70% %
98.86%
90.63%
69.26%
Results on Training on MNIST Transformed by (2.60")
Training
MNIST
Transformed by (2.90"))
Method
(0,0")
(2.30")
(2,64")
(2,90')
(2,180")
Method
(0.0")
(2,30')
(2.60)
(2,90")
(2,180')
Capsnet
99.39%
99.12%
98.99%
95.33%
72.00%
Capsnet
99.17%
98.77%
98.73%
98.29%
79.18%
EMcaps
98 84%
98.79%
98.55%
94.03%
70.03%
EMcaps
98.83%
38.
98.42%
97.86%
77.47%
G-Caps
97.44%
96.31%
96.01%
93.18%
81.70%
G-Caps
97.67%
96.53%
96.33%
95.52%
83.76%
DeepCaps
99.54%
99 49%
99.425 99%42%
97. 27%
73.61%
DeepCaps
99.44%
99. 16%
99.03%
98.64%
77.54%
GCNN
99.52%
99.38%
99.37%
02.
74.98%
GCNN
89.34%
89. 89.16%
89.13%
88.86%
75.53%
SOVNET
99.70%
99.65%
99.63%
98.56 %
79.59%
SOVNET
99.68 %.
99.64 %
99.54%
99.5% %
87.76
Results Training
MNIST
Transformed by (,T)
Method
(0.0")
(2.30")
(2,60")
(2,90%)
(2,180")
Capenet
97.52%
96.09%
EMcaps
95.89%
95.22%
95.42%
95.42%
95.09%
G-Caps
95.24%;
93.67%
93.83%
93.79%
93.76%
DaepCaps
98.17%
97.84%
97.89%
98.11 1
98.01%
GCNN
87.8%
87.51%
87.47%
87.41%
87.45%
SOVNET
98.34%
98.10%
98.11%
98.08%
98.06
On the CIFAR-10 experiments, DeepCaps performs significantly better than SOVNET on the untransformed case generalising test-time transformations better. However, SOVNET learns from train-time transformations better than DeepCaps outperforming it in a large majority of the other cases. We hypothesize that the first observation is due the increased (almost double) number of parameters of DeepCaps that allows it to learn features that generalise better to transformations. Further, as p4-convolutions (the ceediction-mechanisnn used) are equivariant only to rotations in multiples of 90", its performance is significantly lower for test-time transformations of and 60% the untransformed case. However, the equivariance of SOVNET allows it to learn better from train- geometric transforms than DeepCaps, explaining the second observation.
The second case is that GCaps outperforms SOVNET on generalising to extreme transformations on (mainly) MNIST, and once on FashionMNIST, under mild train-time conditions. However, is unable to sustain this under more extreme train-time perturbations. We infer that this is caused largely by the explicit geometric parameterisation of capsules in G-Caps. While under mild-to- moderate train-time conditions, and on simple datasets, this approach could yield better results, this parameterisation, especially with very simple prediction-mechanismt can prove detrimental. Thus, the convolutional nature of the and also the greater depth of SOVNET allows it to learn better from more complex training scenarios. This makes the case for deeper models with more expressive and equivariant prediction-mecchaiisms mechanisms.
which can capture more complex features,
A related point of interest that G-Caps performs very poorly on the CIFAR-10 dataset - achieving the least accuracy on most cases on this dataset despite provable guarantees equivariance. We argue that this is significantly due to the nature of the capsules of this model itself. In GCaps, each capsule is explicitly modelled geometric information, and use only this information for routing. In contrast, other capsule models have such parameterisation. In the case of CIFAR-10, where non-geometric features such as
element of a Lie group. Thus, capsules capture exclusively
8

---

Published as a conference paper at ICLR 2020
Table 3: Experiments on FashionMNIST.
Resalts Training on Untransformed MashionMNIST
Resalis on Training on MNshionMNISS Transformed by (2,30')
Method
(0,0")
(2.30")
(2,60')
(2,90)
(2.180")
Method
(0,0")
(2,30')
(2.60)
(2.90')
(2.180')
Capsnet
91.23%
57.15%
37.98%
28.33%
22.38%
Capsnet
91.22%
89.57%
69.58%
50.17%
35.16%
EMcaps
90.05%
59.75%
40.26%
30.17%
23.82%
EMcups
90.17%
89.47%
68.39%
49.23%
37.02%
G-Caps
86.56%
50.05%
35.05%
29.93%
27.10%
G-Caps
83.28%
80.12%
64.86%
53.71%
52.54%
DaepCaps
93.27%
57.85%
37.06%
27.63%
21. 86.
DeepCups
93.71%
93.40%
75.32%
53.35%
36.30%
GCNN
84.63%
6. 1..391%
0.2862%
21.58%
GCNN
92.25%
90.95%
72.17%
51.93%
37.125%
SOVNET
94.72%
61.58%
41.01%
34.07%
27.63%
SOVNET
94.99%
94.36% %
77.19%
58.59 %
43.84%
Results on Tuining on FashionMNIST Transformed by (2.60")
Results on Training
FashionMINIST
Transformed by (2,90")
Method
(0,0")
(2.30")
(2,60")
(2,90")
(2,180")
Method
(0,0")
(2,30%)
(2,60') )
(2,90')
(2.180")
Capenet
88.55%
85.155%
72.81%
89%
Capsnet
88.78%
87.18%
87.13%
16.1
59.59%
EMcaps
88.24%
87.30%
87.04%
71.72%
48.14%
EMcups
86,43%
85.85%
85.82%
85.63%
61.15%
G-Caps
82.04%
80.12%
78.94%
68. 68.05%
59.25%
G-Caps
80.71
79.55%
79.17%
79.21%
72.11%
CappCups
93.36%
93.06%
92.84%
80.76%
49.90%
DeepCups
93.07%
92.93%
92.75%
92.51%
62.50%
GCNN
90.78%
89.82%
89.67%
76.69%
49.97&
GCNN
90.31%
89.46%
89.42%
89.22%
64..44%
SOVNET
94.49%
94.08%
94.20%
90.23%
73.48%
SOVNET
94.41 %
94.03% %
93.93%
93.98%
91.42% %
Resalts on Training on FashionMNIST Transformed by (2.180')
Method
(0.0")
(2.30")
(2,60")
(2,50")
(2.180")
Capsnet
86. 56.%%
84.34%
84.93%
84.75%
82.7
EMcaps
82.99%
82.67%
82.18%
82.32%
82.18%
G-Caps
80.65%
79.66%
79.46%
79.47%
79.37%
DeepCaps
92.07%
91.71%
91.70%
91.76%
91.66%
GCNN
89.7%
88. 68.65%
88 68%
88 62%
88 6%
SOVNET
94.11%
93.77%
93.56%
93.57 %
93.60 %
Table 4: Experiments Cifar-10.
Results Training Untransformed Cifar 10
Results Training Cifar
Transformed (2,30")
Mathod
(0.0))
(2,30")
(2,64")
(2,90")
(2,180")
Method
(0,0")
(2,30))
(2,60=)
(2,90")
(2, (80))
Capsnet
68.28%
55.57%
43.55%
37.48%
30.89%
Capsnet
73.45
69.87%
61.17%
52.29%
42.58%
EMcaps
62.85%
49.28%.
41.37%
34.73%
29.90%
EMcaps
70.24%
66.63%
59.10%
50.97%
42.26%%
G-Caps
49.54%
38.45%
31.89%
30.88%
27.70%
G-Caps
49.50%
48.88%
45.78%
42.93%
38.74%
Capp
76.76%
67.97%
53.56%
45.22
DoepCaps
84.24%
82.54%
74.63%
63.54%
48.63%
GCNN
80.26%
60.94%
47.93%
40.85%
34.02%
GCNN
82.13%
78.94%
70.51%
59.80%
47.81%
SOVNET
88.34%
47.57%
42.24%
43.75%
43.52%
SOVNET
86.58 %
85.35 %
82.51 %
79.14%
69.64% %
Results on Training on Citar- 10 Transformed by (2.60")
Results on Traiiing on Cifar- Transformed by (2.90")
Method
(0,0%)
(2,30")
(2,60))
(2,90%)
(2,180")
Method
(0,0%)
(2,30%)
(2,60-)
(2,90%)
(2. 180")
Capsnet
70.26%
67.69%%
66.62%%
60.04%%
47.993%
Capsnet
67.81%
65.64%
65 45%
64.35%
79%
EMcaps
66.53%
65.09%
63.21%
58.04%
47.61%
EMcaps
64.33%
63.00%
62.70%
61.42%
52 $2.08%
G-Caps
49.63%
50.31%
48.84%
47.43%
43.11%
G-Caps
49.98%
51, $1.24%
50.63%
49. 49.95%
46.59%
DeepCaps
83.92%
83.63%
82.79%
78.09%
60.02%
DeepCaps
82.91%
82.78% %
82.66%
82.62%
68.34%
GCNN
82.13%
78.94%
70.51%
59. 80%
47.81
GCNN
77.56%
75.79%
75.57%
75.14%
64.23%
SOVNET
83.86%
83.63%
83.57%
83.06%
80.89
SOVNET
83.33%
82.76% %
82.58% %
82.79 %
82.22 %
Results
Training on
Transformed by
80")
Method
(0,0")
(2.30")
(2,607)
(2,90")
(2,180")
Capsnet
61.08%
39,33%
60.04%%
59.835
30,009%
EMcaps
57.57%
55.89%
56.85%
56.35%
55,20%%
G-Caps
39.09%
41.03%
41.43%
41.25%
41.08%
DeepCaps
81.12%
80.81%
80.64%
81.05%
80.92%
GCNN
73.61%
72.50%
72.12%
72.24%
72.25%
SOVNET
82.50%
81.80 %
81.78% %
81.95% %
81.82%
texture are important, we see that purely spatio-geometric based routing is not effective. This observation allows us to make more general hypothesis that could deal with the fundamentals of capsule networks. We propose a trade-off in capsule networks, based on the notion of equivariance. To appreciate this, some background is necessary both equivariance and capsule networks.
As the body of literature concerning equivariance is quite vast, we only mention a relevant selection of papers. Equívariance can seen as a desirable, if not fundamental, inductive bias for neural networks used in computer vision. Indeed, the fact that AlexNet (Krizhevsky et al.) 2012) automatically learns representation that are equivariant to flips, rotation and scaling shows the importance of equivariance as well as its natural necessity (Lenc & Vedaldi 2015). Thus, neural network model that can formally guarantee this property is essential. An early work in this regard is the group-equivarian convolution proposed in (Cohen & Welling) 2016). There, the authors proposed a generalisation of the 2-D spatial convolution operation to act on a general group of symmetry transforms increasing the parameter-sharing and, thereby, improving performance. Since then,

---

Published as a conference paper at ICLR 2020
Table 5: Results augmented SVHN and KMNIST.
Method
SVHN
KMNIST
Sabour et al. (017))
95.7%
Deliège al. (2018)
94.50%
Rajasegaran et al.
97.16% %
89.18%
(2019)
Phaye al. (2018)
96.90%
Clanuwat et al.
98.83%
(2018)
Tissera et al. (2019)
96.8%
99.05%
SOVNET
97.03%
99.03% %
several other models exhibiting equivariance to certain groups transformations have been proposed, for example (Cohen al. 2018b), where spherical correlation operator that exhibits rotation- equivariance was introduced; (Carlos Esteves & Danilidis 2017). where a network equivariant to rotation and scale, but invariant translations a model equivariant to translations and 3D right-angled rotations was developed. A general theory of equivariant CNNs was developed in (Cohen al. (01 8ah. In their paper, they show that convolutions with equivariant kernels are the most general class of equivariant maps between feature spaces.
presented, and Worrall & Brostow (018),, where
A fundamental issue with group-equivariant convolutional networks is the fact that the grid the convolution works with increases exponentially with the type of the transformations considered. This was pointed out in (Sabour al., 2017); capsules were proposed as an efficient alternative. general capsule network model, each capsule is supposed to represent the pose-coordinates of an object-component. Thus, to increase the scope of equivariance, only linear increase in the dimension of each capsule is necessary. This was however not formalised in most capsule architectures, which focused on other aspects such as routing (Hinton al., 2018), (Bahadori] 2018), 8), (Wang Liu 2018);
general architecture (Rajasegaran et al. (2019), (Deliège et al. 2018). (Rawlinson et al. 2018 Jeong ct al. (2019), (Phaye et al. 2018). Rosario et al. (2019); ; or application Afshar et al. (2018).
2018) that this idea of efficient equivari- ance was formalised. Indeed, in that paper, equivariance changed from preserving the action of a group vector space preserving the group-transformatien scale well to larger transformation groups in the sense of preserving equivariance guarantees, we argue that they cannot efficiently handle compositionalities that involve more than spatial geometry. The direct use of capsules as geometric pose-coordine corrdiaodsss could lead to exponential representational inefficiencies in the number of capsules. This is the tradeoff we referred to. We do not attempt formalisation of this, and instead make the observation given next. While SOVNET (using GCNNs) lacks transformational efficiency, the use of convolutions allows it to capture non -on-meemetric structures well. Further, SOVNET still retains the advantage of learning compositional structures better than CNN models due to the use of routing, placing it in a favourable position between two extremes.
It was only in group-equivariant capsules Lenssen et al.
on an element. While such models
5 CONCLUSION
We presented a scalable, equivariant model for capsule networks that uses group-equivariant convolu- tions and degree-centrality routing. We proved that the model preserves detected compositionalities under transformations. We presented the results of experiments on affine variations of various classification datasets, and showed that our model performs better than several capsule network baselines. A second set of experiments showed that our model performs comparably to convolutional baselines on two other datasets. We also discussed a possible tradeoff between efficiency in the transformational sense and efficiency in the representation of non-geometric compositional relations. As future work, we aim at understanding the role of the routing algorithm in the optimality of the decomps-dcoomooiitionn graph, and various other properties of interest based on it. We also note that SOVNET allows other equivariant prediction mechanisms each of which could result a wider application of SOVNET to different domains.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Parnian Afshar, Arash Mohammadi, and Konstantinos N Plataniotis. Brain tumor type classification via capsule networks. In 20/8 25th IEEE International Conference on Image Processing (ICIP), pp. 3139-3133. IEEE, 2018.
Mohammad Taha Bahadori. Spectral capsule networks. 2018.
Xiaowei Zhou Carlos Esteves, Christine Blenn-Blanchette and Kostas Daniilidis, Polar transformer networks. CoRR, abs/1709.01 2017. /tttpp::://rx
www///ss///// 01889
Jaewoong Choi, Hyun Seo, Suii Im, and Myungjoo Kang. Attention routing between capsules. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:18/2.01718, 2018.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pp. 2990-2999, 2016.
Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems(Accepted paper), 2018a.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801.10130, 2018b.
Adrien Deliège, Anthony Cioppa, and Marc Van Droogenbroeck, Hitnet: a neural network with capsules embedded in hit-or-miss layer, extended with hybrid data augmentation and ghost capsules. arXiv preprint arXiv:1806.06519 2018.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International Conference on Artificial Neural Networks, pp. 44-51. Springer, 2011.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. 2018.
HopefulRational. PyTorch Implementation of "DeepCaps: Going Deeper with Capsule Net- works" " by Jathushan Rajasegaran et al.: Hopetulkkiioon//eepCpsss-yPyyyoo September 2019. original-date:
URL httptttss///giie thu.thhhhbb.... com/ hwtp://ooooeuuuaattooao///opooaa 2019-07-311T1::12224.
Taewon Jeong, Youngmin Lee, and Heeyoung Kim. Ladder capsule network. International Conference on Machine Learning, pp. 3071-3079, 2019.
Lorenzo Torresani Karim Ahmed. Star-caps: Capsule networks with straight-through attentive routing. Advances in neural information processing systems (Accepted paper), 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http::/yann lecun. cwm/exdb st/
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 991-999, 2015.
Jan Eric Lenssen. Pytorch implementation of Group Equivariant Capsule Networks: jel/group_equivariaatt equivariant capsules pytorch, July 2019. URR/https://qtt com/ mrjelr group_equivar ant caannuuaaaaalllaaaaaa eeppytrch_ptt original-date: 2018-10405T11::3::402
11

---

Published as a conference paper at ICLR 2020
Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. arXiv preprint arXiv:1806.05000 05086, 2018.
Ethan Li. Empirical studies Capsule Network representation and improvements implemented with ethanlectCappsNet, September 2019. URL httts:://qtthut com/ http:///a/cc.ccom//ettattea///// ethanleet/
PyTorch.:
CapsNet original-date: 2018-04-30T23:44:262
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Sai Samarth R Phaye, Apoorva Sikka, Abhinav Dhall, and Deepti Bathula. Dense and diverse capsule networks: Making the capsules learn better. arXiv preprint arXiv: N80::1805004001,,, 2018.
Jathushan Rajasegaran. Official Implementation of "DeepCaps: Going Deeper with Capsule Net-
works" paper (CVPR 2019).: brjathu/deepcaps, September 2019. URL httts:///itth:t
com/br coom/brjathu//deepaapp original-date: 2019-03-15T00:44255
Jathushan Rajasegaran, Vinoj Jayasundara, Sandaru Jayasekara, Hirunima Jayasekara, Suranga Seneviratne, and Ranga Rodrigo. Deepcaps: Going deeper with capsule networks. arXiv preprint arXiv:1904.09544 09546, 2019.
David Rawlinson, Abdelrahman Ahmed, and Gideon Kowadlo. Sparse unsupervised capsules generalize better. arXiv preprint arXiv: /804. 000906 2018.
Vanderson Martins do Rosario, Edson Borin, and Mauricio Breternitz Jr. The multi-lane capsule network (mlen). arXiv preprint arXiv: 1922:0900000441 2019.
Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems, pp. 3856-3866, 2017.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant transformer networks. arXiv preprint arXiv:1901.11399, 2019.
Dumindu Tissera, Kumara Kahatapitiya, Rukshan Wijesinghe, Subha Fernando, and Ranga Rodrigo. Context-aware multipath networks. arXiv preprint 1rX.v.190.111599, 2019.
Dilin Wang and Qiang Liu. An optimization view on dynamic routing between capsules. 2018.
Daniel Worrall and Gabriel Brostow. Cubenet: Equivariance to 3d rotation and translation. In
Proceedings of the European Conference Computer Vision (ECCV), pp. 567-584, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist; a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 17X77:1708.07747, 2017.
Lei Yang. A PyTorch Implementation of Matrix Capsules with EM Routing: y1-1993/Matrix Capsules-EM-PyTorchh September 2019. URL htttp:://iithb. com 1993/
Matrix--aapslles-BB--yyBBee original-date: 2018-04-21100::3:472
APPENDIX
ACKNOWLEDCEMENTS
We thank Vineeth Balasubramanian for the discussions have had, and his useful comments. We also thank Darshan Gera, Rishi Rao and Ankit Anand for their suggestions and comments.
A DEFINITIONS
We present formal definitions for the following concepts.
12

---

Published as a conference paper at ICLR 2020
A.1 GROUP
A tuple (G, o). where G is non-empty set and defines binary operation on G, is said to form group if the following properties are satisfied:
Closure: 91, E G. 91 92 E G.
Associativity: V 91, 92. E G. (91 o (2) - (02 9s). Existence of the identity element: Jee G G
Existence of an inverse: V E G. G. E G GGa
A.2 GROUP ACTION AND GROUP REPRESENTATION
Given group (G, and a vector space V, group action is function f: G V + satisfying the following properties.
1: VaeV, f(e,a))aa
2: v yg e and V ((),a)) = [(h g.a.
A group representation is a group action by invertible linear maps. More formally, a group representation of group (G, with respect to vector space V is homomorphism from G to GV( the set linear, invertible maps from
A.3 POOL
Consider one-layer GCNN-convolutional prediction network T/ for SOVNET layer 7 and for the d'+1, dimensional jth capsule-type. Intuitively, Pool ++ (g) is defined by the extent of the support g-transformed filter 4/+ More formally,
Poot +1 (g) = {h G w/t h) } +0}. 0}.
(1)
For general L-layer GCNN prediction- network, Poot (g) is defined by recursively applying the above definition through all the layers of the prediction network.
A.4 2-NORM
The 2-norm of a vector x (Io, ... In R and denoted by | is defined as ||=||| (DI 2
B PROOFS OF THEOREMS
We present proofs for the theorems mentioned in the main body.
Theorem B.1. The SOVNET layer defined in Algorithm and denoted by the operator & as given above, satisfies ([,,F") @ w/+1) = (Ly(E" & " where belongs to the underlying group of the equivariant convolution.
Proof. For the theorem to be true, we must show that each step of Algorithm 2 is equivariant. We do this step-wise.
The predictions S4+1 made in the first step are group -roup-qquivariant. This follows from the fact that sl+1 (g) = (fl * ()), and that ([L,f)) + w/tPP) L, (f} * w/+1PP) proved in (Cohen &
Welling 2016
13

---

Published as a conference paper at ICLR 2020
We now show that the DegreeScore procedure is equivariant. We that Degreed (9) Stt Stj (x)
En'' [ (0)l...1 2 (9) S x Nr 1. Each S (g) (g) = Ep= (
V w (g). From the equivariance of *, Ep-o (1 [T (
of
(g) Ep= [n v (g) (g) Σp p=0 U * x
+1
1 (h (1
(1 1pp ( 1) ( (G))
Ln 4n Ep-o ( (
Moreover, the 2-norm of an equivariant map is also equivariant from the equivariance of the post-composition of non-linearities over equivariant maps (Cohen & Welling 2016). Also, the division of two (non-zero) equivariant maps also equivariant. Thus, obtaining the degree-scores is equivariant. Again, the softmax function preserves the equivariance as it is point-wise non- non-linearies oon-iinear,
The proof is concluded by pointing that the product and of equivariant maps is also equivariant.
Theorem B.2. Consider an L-layer SOVNET whose activations are routed according to a procedure belonging to the family given by Algorithm Further, assune that this routing procedure is equivariant with respect to the group G. Then, given an input x and Vg E G. G(z) and G((Lg))) isomorphic.
Proof. Consider fixed L-layer SOVNET that is equivariant to transformations from a group G, and an input x G + R Let G(x) be the cepsule-decomppsstion graph corresponding to x. Then G (L,z)) denotes the the capsule-decomposition graph of the transformed input LA
We show that the map f(()) f - o gg) is an isomorphism from G(x) to G G(,,z). First, we note that (g) + (h o g) is bijection from V(x) to V(L,)). This is from the definition of the vertex set of a
graph and the fact that the map g + h g is a bijection.
We now show that (fH())), J, g2), Cit (h o g2)) E E(L,x).
E F(x) and only if (f o g1). g.).
First, let us assume (f(())) J3+ (12), ci((1)) E E(x)). Thus, (o)) is routed to f) (12) with coeffnieee Csy (92). However, due the assumed equivariance of the model, a+ (h-
91) is routed to J (h- 92) with routing-coefficiet Gzy(h o gz). This, of course, implies (h o g1), /j (h g2), cis(h 92)) E E(L,z).
The converse of this result is proved in the same way by considering E(L,x), noting that ( = E(x), and applying the above result to E(Lxx) and ((Lp-1Lat)
C FURTHER EXPERIMENTS
C.I EMPIRICAL VALIDATION OF CAPSULE DECOMPOSITION ISOMORPIIIM
We performed two experiments to verify that the capsule Decomposition-grapppe of the transformed and untransformed images isomorphic.
For the first of these, we trained a c4-coovolution based SOVNET architecture on untransformed images of MNIST and FashionMNIIT.. We then considered four variations of the two test-datasets untransformed, and three versions rotated exactly by multiples 90 degrees: 90, 180, and 270. Our experiment verifies that the mapping defined in the proof of Theorem 2.2 is indeed an isomorphism.
14

---

Published as a conference paper at ICLR 2020
To this end, considered the supsule-activation all the capsule-layers, for each image of all the variations of the test split of the corresponding dataset. We then mapped the activations and the degree-scoree for the untransformed images by the aforesaid mapping for each the transformations. This corresponds to 'rotating' the activations and degree-scores by each transformation. We then computed the squared error of these with each of the activations and degree-scores obtained from the correspondingly transformed image, respectively. A successful verification would result in zero error (up to machine precision). The results Table show that this happens.
well the degree-scone obtained across
The second of our experiments is empirical verification that the test-accuracies remain unchanged under transformations for which SOVNET exhibits equivariance. We use the same trained architecture above, and verify that the accuracy remains unchanged under exact transformations of the images. The results are presented Table / The accuracies presented below are only for the purpose of veryfying the isomorphism of the of the graph.
Table 6: Empirical validation of isomorphism: mean squared error of capsule activations and degree- scores.
MNIST
Rotation
Mean squared-error
Mean squared-error
for capsules
for degree-scores
90
6. 13
3.3087e-15
180
6.2821e-15
3.3606e-15 15
270
6.1911
3.3138e-15
FashionMNIST Mean squared-error for capsules
Rotation
Mean squared-error for degree-scores
90
2.5678e-13 13
1.9576e-13
180
2.6306e-13 13
1.9981e-13
270
2.5869e-13
1. 1.9662e-13
CIFARIO
Rotation
Mean squared-error
Mean squared-error for degree-scores
for capsules
90
6.4583e-13
2.1735e-13
180
6.3866c-13
2.1635e-13
270
6.4624e-13
2.1774e-13
Table 7: Empirical validation of isomorphism: accuracies under transformation.
Accuracies on MNIST
Accuracies on FashionMNIST
00
90
180
270
0º
90
180
270
99.52%
99.52%
99.52%
99.52%
92.23%
92.23%
92.23%
92.23%
Accuracies on CIFAR10
0
90"
180
270
77.19%
77.19%
77.19%
77.19%
C.2 RESULTS ON TESTING UNSEEN TRANSFORMS: AFENIST
We trained SOVNET architecture MNIST images that are padded to size 40x40 the size of AFENIST images. We augment these images by translation, as is the standard approach. Note that the changed size of the images necessitates different architecture. The result of this experiment is given in Table We see that our SOVNET architecture obtains the highest accuracy when compared other recent capsule network models.
15

---

Published as a conference paper at ICLR 2020
Table Accuracy AFFNIST.
Method
Accuracy
(Sabour
79.0%
ct al. 2017)
Hinton
93.1%
al. (
Lenssen
89.10%
et al. 20118)
eJong al.
87.8%
2019)
(Choi et al. 2019)
91.6%
SOVNET
97.01%
We also trained the above SOVNET architecture on MNIST with translations in the range of [-6,6] pixels and rotations from [-30 ,30] degrees. While this increases the extent of train-time augmentation, there are several test-time transformations that are unseen. With this scheme, we achieve state-of-the-art accuracy of 99.20%. This improves over the best, to our knowledge, accuracy of 98.3% obtained by (Tai et al. 2019).
C.3 CAPSNET WITH SHARED PARAMETERS
We considered an implementation of the CAPSNET model (Sabour et al. 2017). Unlike (Sabour jet al., (017), that uses one prediction-network per connection between capsules, this model uses one prediction-nettorkk per class-capsule. The result of this model augmented versions of MNIST and FashionMNIST are presented Table D. with corresponding accuracies of capsnet.
Table 9: Results on MNIST and NIST.
Method
MNIST
FashionMNIST
Capsnet
99.75%
Sabour al.
93.62% Rajasegaran
2017)
et al. 2019)
Shared Capsnet
99.47%
91.57%
C.4 RESULTS OF SOVNET ON CIFAR-
We have trained SOVNET architecture on CIFARI00. Our model has achieved accuracy of 71.55% an almost percentage improvement over recent capsule network model STARCAPS (Karm Ahmed 2019 which achieved 67.66%.
C.5 COMPARISON AGAINST RESNET ARCHITECTURES
In order to compare SOVNET with more sophisticated CNN models, we performed limited set of experiments on MNIST and FashionMNIST. We trained ResNer18 and ResNet34 on the train split MNIST and FashionMNIST transformed by random translations of up to pixels, and random rotations of up + 180". The models were tested various transformed versions of the test-splits. The results of these experiments are given in Table I0 As can be seen in the table, SOVNET compares with the two much deeper CNN models. More testing as well as deeper SOVNET models must be done, however, to obtain a better understanding of the relative performance of these two kinds of models.
complex datasets,
D DISCUSSION ON GENERAL WEIGHTED-SUMMATUUE ROUTING
Consider Algorithm | which given below for convenience. The role of the GetW GetWeights and Agreement procedures to evaluate the relative importances of predictions for deeper capsule,
16

---

Published as a conference paper at ICLR 2020
Table 10: Results ResNet18 and ResNet34.
Results on Training on MNIST
Transtormed by (2.180")
Kesulis on Training on FashionMNISTT Transformed by (2.180")
Method
(0,0%)
(2,30")
(2.609")
(2,90")
(2.180")
Method
(0.07)
12.30')
(2,60'
(2,90%
(2.180'
ResNet18
98.60%
98.30%
98.21%
98.15%
98.02%
ResNet18
94.21%
93.35%
91.24%
93.30%
93.45%
ResNe134
98.53%
98. 88.26%
98.21%
98. 12%
98.01%
ResNer34
94.38
93.75%
98.78%
93.78%
93.73%
SOVNET
98.34%
98.10%
98.11%
98.08%
98.06%
SOVNET
94.11%
93.77%
93.56%
93.57%
93.60%
and the extent of consensus among them, respectively, The second of these is interpreted as measure of the activation of the corresponding deeper capsule.
A formalisation of these concepts to general framework for even summation- based routing as to cover all possible notions of relative importance, and consensus is not within the scope of this paper. Indeed, to the best of our knowledge, such formalisation has not been successfully completed.
Thus, instead of formal description of a general routing procedure, we provide examples to better understand the role of these two functions. We first explain GetW eights, and then Agreement.
Algorithm A general weighted routing algorithm for SOVNET.
Input: {(A, a)ll E (O...., .... N; 1h-1),,f, 11 G Rd at G
[0, 1]} Ratti
Output: (U/+1 .
( )| (O...., Ni+1 - 11).
a aat GG - [0,1]]
Trainable Functions: (u'+1, ..) projection networks that operator
(g) (g) = (ff., a²) / (g) Vi.j. V
sdt1 (o) (g) oN 15(()) GetW Weghtights (g), ( (1)
[+1
f (++ E 2 (g) ()
Vj E
st+1 (j (g) = Af (o), (g), S (1)))(
The first example of GetW eights we provide from the proposed degree-centrality based routing. The algorithm is given below, again. In this case, GetW eights is instantiated by the SegreeScore procedure, which assigns weights to predictions based on their normalised degree centrality scores. Thus, a prediction that agrees with a significant number of its peers obtains higher importance than one that does not. This scheme follows the principle of deeper capsule only when its predicting shallower, component-capsule are an acceptable spatial configuration (Hinton et al. 2011).
byaagremmnt,, that aims to activate a
The above form for the summation based routing procedure generalises for several existing routing algorithms. As an example, we present the dynamic routing algorithm of (Sabour et al. 2017 7). This differs with our proposed algorithm in that it is a "attention-base" routing algorithm. That is, the relative importance of a prediction with respect fixed deeper capsule is not a direct measure the extent its consensus with its peers, but rather a measure of the relative attention offers to the deeper capsule.
rather than "agreement-
Thus, the weight associated with prediction for a fixed deeper capsule by a fixed shallower capsule depends on other deeper capsules. In order to accomodate such methods into a general procedure, we modify our formalism by having GetW eights take all the predictions as parameters, and return all the routing weights. This modified general procedure is given in Algorithm S
Consider the dynamic routing algorithm of (Sabour et al. 2017). given in Algorithm 6 modified to our notation and also the use group-equivariant convolutions. The procedure Routnic is the instantiation for GetW eights. Note that the weights (z)(g) depend on the routing weights for the deeper capsules.
Due to the formulation of capsules in paper, in (Sabour et al.) 2017), we use the 2-norm of a capsule to denote its activation. Thus, our degree-centrality based procedure, and also dynamic routing, do not use separate value for this. However, examples of algorithms that use separate
17

---

Published as a conference paper at ICLR 2020
Algorithm The degree-centrality based routing algorithm for SOVNET.
Input: Wili E (O.... N: - 1 G - RT)
:G Output: (ff "J E (0, (O. N4 - : :G-
Trainable Functions: (v//-1,)) 0 j < N4+1 1, a set of d group-equivariant convolutioalll
filters (per capsule-type) that use the group-equivariant correlation operator* *
(t) (g) (JL x / 1
a E ( o h): E (0.-..,'++11 -
(c) (g)...., 1+1 N1 DegreeScore(Sin gt+ (g..... S ( vo S Ni+1 1
(g) = - of
(g) 0A I S M41 - 1.YgEE
= | (g)l2
(/) (g) = Squash
1 (0::o N
1 1
procedure DEGREESCORE (o) (t)
(g) [g). (g)),...5x-(((0)))
Agx(()) | (a)l 2- 11 (0)ll
0 i S N₂ -1
Degree, (g)= -x E ()):
0 -1
ci)(g) ΣN exp( Degree) (0)
exp(Degree
0SiSN-1
return Gi)(0) 44 S i S -
Algorithm 5 A general weighted- routing algorithm for SOVNET.
Input: {UA, a))i E (O...., Nr 11-1),,, 11 G G : [0,1]
Output: {U oj ( E (0...., No+1 - - Rd
, _att G - [0,1]}
Trainable Functions: (V/+1, ( projection networks that operator
(t)( ((fl, a 4 (g) i...
(c (g)..., ..., N.- 1N (g)) GetWeights
eights ( (g)) ..., St. IN -1 Ag E
f (g) = Ein 4 (1)) (g) V.j Vg E
1+1 (g) = A d++1 (g), (g) ..... S (o) o
st+1
Algorithm 6 The dynamic routing algorithm
Input: {fili E (0. ..., Nr - N G -> T2d
Output: {$]+ Ij (0...., - 13,5/t1
el+1 G R Rdd
Trainable Functions: (V//1, +)) jj S N1+1 - 1, set of t group-equivariant
convolutiooalll
filters (per capsule-type) that use the group-equivariant correlation operator *
Sip (t) (f. // (o) = Enea>> En' (1) (g- o h): E 1
(g) (g). .... NN- (9)) DynamicRouting( ( (g).... st+1 s
(9));
1/+ = EM + (1))++)))))) S 1, V
(ا)و2 1 (9: N41 1,
15 (1+1(n) Squash = I+ 1+,
procedure DYNAMICROUTING st+1 (4),..,
s++
N - 1N (g)
(g)(g) + 0; YO s N: 1, v0
رj < N1+1 -1
for r iterations do:
ci(9)) M exp(Dxx()))
(x)
0 < 1 < Ni 1,40 < N411 1
[1+ (10))=X E 9j 1+1 ()) N1+1
f/*((g) (Q)) 1/+ 1||
(g) S i N44 -1 1
//4411
return
18

---

Published as a conference paper at ICLR 2020
activation value exist; for example, spectral routing (Bahadori 2018) computes the activation score from the sigmoid of the first singular value of the matrix of stacked predictions.
E GENERALISATION TO OTHER GROUPS
Our theoretical results and algorithms admit a generalisation to other groups as long as an appropriate group cooup-convolution is defined. The equivariance and the preservation of detected compositionality is preserved under the condition that the group-convolution equivariant.
As an example, consider the discrete translation group 2 and the regular correlation operation defined for an input with channels by (f (z) equivariance of this operation is proved in (Cohen & Welling 2016).
= Euzza > (t) - t). The translation-
The general n-dimensional correlation defined on n is given by (ʃ * (x) Etega Edn (t) - t). This operation is equivariant to translations in n-dimensions. The proof for this is given below.
Theorem E.1. The n--dimensional correlation operator equivariant with respect z" and the group representation L.
Proof. Consider 2, y, Z". and / 2 + +Rd Then, ([1y * v))))) Eiezo Ed= fall 3) (x t) ELEz fe (t)(t) (x( - -- :) ElEz Ek= f (t u)) [f
(z)(). Thus, ([4,)) * (x) = (J (x)
Our degree-centrality based algorithm, with its use discrete convolutions, can be used in its current form with the above convolution. The proof of equivariance and the preservation of compositionality holds from a direct application the above result to Theorem 2.1 and Theorem 2.2, using the underlying group Z".
For continuous groups such as SO(n), the degrec-centrality based algorithm must use equivariant convolutions defined over remain equivariant. We consider the specific case of SO(3) below.
The correlation of two functions f, V +0(3) - IRd is given by:
(f ()) [S Σ V
This correlation equivariant to transformations in SO(3). with respect to the group representation
Ln defined by [Q)] f(R-'(())))) as proved in (Cohen et al. 2018b). It is to be noted that due to approximations introduced by the sampling of continuous functions in implementations, exact equivariance is not preserved.
However, our routing algorithm can still be used with such convolutions and does not contribute to any reduction of equivariance by itself. This is due the equivariance of the dot-product and the post-composition operators. The equivariance of the post-composition operator was proved in (Cohen & Welling 2016). We formally prove the equivariance of dot-product for the SO(3) group.
Theorem E.2. The dot-product between with respect to the group representation L. That is, [Lnf].[[n]] = Lz(ff-]]
funciionss f.g: g: SO(3) - Rd is equivariant
Proof. [[ng]( Ei= ( (R |E fags (R f-g(R-IQ)) LR(f-g)-
The proof for the preservation of compositionality also holds by considering the infinite graph G(z). The definition for this is the same as before. The proof follows by using the same mapping between vertices, and from the equivariance of the routing procedure.
19


Reviews:
[{"review_id": "HyeH-bq9FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, a method was proposed to train capsule network by projectively encoding the manifold of pose-variations, termed the space-of-variation (SOV), for every capsule-type of each layer. Thereby, the proposed method aims to improve equivariance of capsule nets with respect to translation (rotation and scaling).\n\nThe proposed method is interesting and the initial results are promising. However, there are various major and minor problems with the work:\n\n- There are various undefined functions and mathematical notation, such as the following:\n\n- Please give formal and precise definitions of groups and group representations for readers who are not familiar with mathematical groups.\n\n- What are GetWeights and Agreement used in Algorithm 1?\n\n- Please define “routing among capsules” more precisely.\n\n- How do you calculate Pool() more precisely?\n\n- In the paper, the results are given for a general class of groups. However, it is not clear how these results generalize even for some popularly employed groups, such as Z^n, S_n, SO(n), SE(n) etc., with different symmetry properties, base space, and field type.\n\n- Please check the following paper for a detailed discussion on group equivariant CNNs with different group structures, and elaborate the theoretical results for particular groups (e.g. at least for p4m used in experiments) :\n\nT.S. Cohen, M. Geiger, M. Weiler, A General Theory of Equivariant CNNs on Homogeneous Spaces, NeurIPS 2019\n\n- Please define the norms used in Algorithm 2.\n\n- How do you calculate accuracy of models? Are these numbers calculated for a single run, or for an average of multiple runs? If it is the former, please repeat the results for multiple runs, and provide average accuracy with variance/standard deviation. If it is the latter, please provide the variance/standard deviation as well.\n\n- Have you performed analyses using larger datasets such as Cifar 100 or Imagenet? It would be great to provide some results for larger datasets to explore their scalability. \n\n- Please define accuracy given in tables more precisely, use dot \".\" at the end of sentences in captions.\n\n- There are several typo/grammatical errors, such as the following:\n\n-- Homegenous -> homogeneous\n\n--  for with prediction networks\n\n-- Please proof-read the paper in detail and fix the typo etc.\n\nAfter the discussions:\n\nMost of my questions were addressed and the paper was improved in the discussion period. Therefore, I increase my rating.\n\nHowever, some parts of the paper still need to be clarified. For instance;\n\n- GetWeights: To ensure that the predictions are combined in a meaningful manner, different methods can be used to obtain the weights. The role of GetWeights is to represent any such mechanism. \n\n-> Please define these methods in detail and more precisely, at least in the Supp. mat.\n\n- Agreement : The Agreement function represents any means of evaluating such consensus.\n\n-> This is also a very general concept, which should be more precisely defined. \n\n- Our theoretical results (Theorem 2.1 and Theorem 2.2) hold for all general groups, and the particular group representation  defined in the main paper. \n\n-> Could you please give a concrete discussion on generalization of these results and the proposed algorithms for discrete and continues groups? For instance, how do these algorithms and results generalize with Z^n and SO(n)?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJx4P6rAFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines CapsuleNetworks and GCNNs with a novel formulation. First they modify the CapsNet formulation by replacing the linear transformation between two capsule layers with a group convolution. Second they share the group equivarient convolution filters per all capsules of the lower layer.  Third, they change the similarity metric from a lower-upper similarity into a pairwise lower similarity and aggregation which makes it keep the equivarience. Since the cij does not depend on upper capsule anymore they only perform 1 routing iteration (no modification of the routing factors).\n\nOne assumption in CapsNets is that each part belongs to one whole. Therefore, the normalization in Alg.2 usually is division by degree^k_i. The proposed normalization formula for c_ij seems to encourage that each upper capsule only receives one part. Is this a typo or is there a justification for this?\n\nThe discussion on ideal graph on page 5 is interesting. But the points made are not used later on. I expected the results to have an analysis or at least a show case that indeed if you transform the resultant graphs stay isomorphic. \n\nOne goal for CapsuleNetworks vs GCNNs is the hope for handling different transformations and not only rotations that one can grid with group convolutions. But, the experiments only report on rotation, translation as a transformation. Reporting results by training on MNIST, testing on AFFNIST could shed light on this aspect of SOVNETs.\n\nConditioned that the last two points will be addressed in the rebuttal I vote for accepting this paper since they suggest a novel formulation that brings some measures of rotation equivarience guarantee into CapsNets. Also their results suggest that there is no need for per Capsule filter bank and several refinements to get rotation robustness (it would be interesting to check the performance of a simple capsnet with shared parameters).  In the appendix there is a comparison with GCNNs on fashion MNIST which shows they have better performance than GCNNs. I would advise reporting GCNNs for all the experiments in the main paper. \n\n\n------------------------------------------\nThank you for updating and expanding the paper. The extra experiments, isomorphism analysis and their response regarding the attention vs part-whole makes the paper much stronger. Therefore, I am increasing my score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper combine recent ideas from capsule networks and group-equivariant neural networks to form equivariant capsules, which is a great idea. The exposition is clear and the experiments provide a very interesting analysis and results. I believe this work will be very well received by the ICLR community.
Decision: Accept (Talk)