Momentum Capsule Networks
Authors: Josef Gugglberger
Antonio Rodriguez-sanchez
David Peer
Venue: Accepted by TMLR ()

Abstract: Capsule networks are a class of neural networks that aim at solving some limiting factors of Convolutional Neural Networks. However, baseline capsule networks have failed to reach state-of-the-art results on more complex datasets due to the high computation and memory requirements. We tackle this problem by proposing a new network architecture, called Momentum Capsule Network (MoCapsNet). MoCapsNets are inspired by Momentum ResNets, a type of network that applies reversible residual building blocks. Reversible networks allow for recalculating activations of the forward pass in the backpropagation algorithm, so those memory requirements can be drastically reduced. In this paper, we provide a framework on how invertible residual building blocks can be applied to capsule networks. We will show that MoCapsNet beats the accuracy of baseline capsule networks on MNIST, SVHN, CIFAR-10 and CIFAR-100 while using considerably less memory. The source code is available on https://github.com/moejoe95/MoCapsNet. 

Published in Transactions on Machine Learning Research (08/2022)
Momentum Capsule Networks
Josef Gugglberger Deportment of Computer Science University of Innsbruck, Austria
j. Beggglberryrrggmmi.. com com
David Peer
duwid..perrBbeeeoppiiiioo
DeepOpinion
Antonio Rodriguez-Sanchez Department of Computer Science University of Innsbruck, Austria
ontonio. wwwp://ooo.ogieeesooooeeebbb.bbb
Reviewed on OpenReview: https: // openreview. formf id- Ld290520knyy
Abstract
Capsule networks are a class of neural networks that aim al solving some limitinn factors of Convolutional Neural Networks. However, baseline capsule networks have failed reach state-of-the-art results on more complex datasets due the high computation and memory requirements. We tackle this problem by proposing new network architecture, called Mo- mentum Capsule Network (MoCapsNeet). MoCapsNets are inspired by Momentum ResNets, a type of network that applies reversible residual building blocks. Reversible networks al- low for recalculating activations of the forward pass in the backpropagation algorithm, so those memory requirements can be drastically reduced. this paper, we provide a frame- work on how invertible residual building blocks can be applied to capsule networks. We will show that MoCapsNNet beats the accuracy of baseline capsule networks on MNIST, SVHN, CIFAR-10 and CIFAR-100 while using considerably less memory. The source code is available https:// / /i thub. com/mooejoe95/Mocapsii
1 Introduction
Deep neural networks have been able to achieve impressive results on many computer vision tasks. Neural networks, like AlexNet (Krizhevsky et al., 2012), VGG nets (Simonyan & Zisserman, 2014) or ResNets (He al., 2016), consist mostly of a combination of many convolutional and pooling layers. One limiting factor of CNNs is their inability to learn viewpoint invariant features. A CNN aimed at image classification is likely to misclassify image of upside-down flipped object if the training data only contained images of the object from other orientations. Capsule networks were designed to overcome this drawback (Sabour et al., 2017). Neurons in a capsule define the properties of an object, while its length relates to the presence of such object. Instead of the pooling operations used in most well-known CNNs, capsule networks make use of dynamic routing algorithms. This routing procedure is superior to (max-) pooling because pooling throws away valuable information by only taking the most active fcature detector into account (Sabour et al., 2017).
Unfortunately, capsule networks suffer from their own limitations. First, capsule networks contain large number of trainable parameters, because it is necessary for each capsule in layer I to compute vote for cach capsule in layer 1+1, 1, which Icads to memory consumption that grows quadratically with the number of capsules per layer. Therefore, such capsule networks are very resource intensive, and their memory requirements are a bottleneck that prevents possibilities for training deeper architectures, and as such, limit their usability. Socond, deep capsule networks suffer from instabilitics, which can be overcome by training deep capsule networks using identity shortcut connections between capsule layers, allowing capsule networks with depth of up to 16 capsule layers (Gugglberger et al., 2021).

---

Published in Transactions on Machine Learning Research (08/2022)
For CNNs, the depth of the network is of great importance when maximizing the performance of such neural networks. The bottleneck for training deeper networks is memory, memory requirements are proport ional to the number of weights in a neural network. One possible way to overcome this bottleneck is to trade memory for computation by turning blocks of residual networks into reversible functions (Sander et al., 2021). We introduce in this work a way to include invertible residual blocks for the case of capsule networks. The proposed approach also compatible with different routing algorithms, as we will demonstrate in the experimental section. Similar to Momentum ResNets, we use a momentum term that enables us to convert any existing residual capsule block into its reversible counterpart. Our architecture, called MoCapsNNet, drastically reduces the memory consumption of deep capsule networks, such that we can train capsule networks at almost any arbitrarily deep configuration. A classical block of a residual capsule network increases the memory footprint of a model by around 185 MB, while our block adds just around 4 MB. We further show that MoCapsNNet outperforms the performance baseline capsule networks on MNIST, SVHN, CIFAR-10 and CIFAR-100 with accuracies of 99.54% 93.00%, 72.18% and 43.48% compared to the baseline values of 99.36%, 92.13%, 71.49% and 1%. Our model is also able overcome the instabilities deep capsule networks face during training to some degree. We show this by training deep capsule network with 40 layers.
2 Related Work
2.1 Capsule Networks
The key idea of capsule networks is derived from computer graphics, where during the rendering process, images are generated from data structures. Capsule networks are constructed to invert rendering, mcaning that they perform a conversion from an image to certain instantiation parameters. During inference, the parameters of a capsule represent an object or part of an object and therefore, much like an inverse computer graphies approach. These parameters are organized in vector form, called a capsule, where each element would correspond to a neuron in the capsule neural network. Capsules of a lower layer vote for the orientation of capsules in the upper layer by multiplying their representing vectors with a transformation matrix. Such transformation matrices are obtained through training, and they encode riewpoint-invarianl part-whole relationships. Routing algorithms are aimed at computing the agreement between two capsules of subsequent layers, where a high value represents strongly agreeing capsules. The computed agreement is a scalar value that weights the votes from the lower level capsules and docides where to route the output of a capsule. Sabour al. (2017) proposed the first capsule network with dynamic routing, reaching stale-of-the-art results on the MNIST dataset. Later, Hinton et al. (2018) published Matriz Capsules, capsule network with a more powerful routing algorithm based on popectatioo-mmxiiiaate
and reported new state-of- the-art performance on the Small-NORB dataset. Many variants of capsule networks have been proposed in recent years (Ribeiro et al., 2020; Xiang et al., 2018; Yang et al., 2020; Chang & Liu, 2020; Ding et al., 2019; Sun et al., 2021; Peer et al., 2018), but for many tasks (i.e., CIFAR-10, CIFAR-100 or ImageNet) even the most recent approaches (Ribciro et al., 2020) provide much higher error rates the than state-of- thott- CNN approaches (i.e., 9% on CIFAR-10, while CNNs provide errors of just 3% (Wistuba al., 2019)).
2.2 Reversible Architectures
A network is called reversible if one can recalculate all its activations the backward pass. One advantage of invertible networks is that they can perform backpropagation without saving the activations from the forward pass, which largely decreases the memory footprint of models where such strategy is applied. Recently, many reversible partly reversible architectures have been proposed, like RevNet (Gomez et al., 2017) or the three reversible alternatives to ResNets (Chang et al., 2018). Sander et al. (2021) introduced a way of turning any existing ResNet into a Moment ResNet. without the need to change the network by making the building blocks of residual neural networks reversible. Momentum ResNets change the forward rule of a classic residual network by inscrting a momentum term (7). Momentum ResNots would then be a generalization of classical ResNets (for y = 0) and RevNets (where 7 = 1). Additionally, ResNets using Momentum have higher degree of expressivity than classical residual networks.
2

---

Published in Transactions on Machine Learning Research (08/2022)
2.3 Residual Networks
Very deep convolutional neural networks have shown impressive results in many computer vision lasks. For example, VGG-net (Simonyan & Zisserman, 2014) provided an impressive error rate of just 23.7% the ILSVRC-2014 competition, compare this result with the 38. 1% error of the more shallow Alex (Krizhevsky et al., 2012), even though both networks consisted of the same building blocks. Unfortunately, as networks get deeper, the vanishing gradient problem (Hochreiter, 1991) becomes increasingly prevalent, which means that at some point stacking up more layers will lead to a drop in training accuracy. The vanishing gradient problem can be compensated to some degree by normalization techniques, such as batch normalization (IofTe & Szegedy, 2015), or special weight initialization (Glorot & Bengio, 2010). Another way to improve gradient flow through the network is to add shortcut connections, connecting the output of layer to the output of a layer that is deeper into the network (He et al., 2016), this approach is known the deep residual learning framework. The possibility to skip layers stabilizes training and allows the training networks with more than 1000 layers (He et al., 2016). Residual learning can be applied to capsule networks as well (Gugglberger et al., 2021) such that the training of deep capsule networks can be stabilized by using identity shortcut connections between capsule layers, which allows training capsule networks with a depth of up to 11 layers when RBA (Sabour et al., 2017) is the routing algorithm and up 16 layers for SDA (Peer et al., 2018) and EM (Hinton et al., 2018) routing.
3 Momentum Capsule Networks
Exprcssivity grows exponentially with network depth in the case of CNNs, making it one of its most impor- lant hyperparameters (Raghu et al., 2017). One limiting factor that needs to be addressed is the memory bottleneck: common machine learning frameworks such as PyTorch (Paszke et al, 2019) implement auto- matic reverse mode differentiation algorithms to compute the gradient. For non-invertible intermediate layers, it is necessary to cache all output values during forward propagation in order to compute the gradient correctly (Gomez et al., 2017). As capsules are multidimensional, they require significantly more memory during training when compared to classical neural networks or CNNs.
Reversible architectures reduce memory consumption by allowing to recalculate neuron activations in the backward pass instead of saving them in the forward pass (Gomez et al., 2017). In this work, we introduce the Momentum Capsule Network, or MoCapsNet for short. We will show how we can design capsule networks, such that a part of the network can be inverted. Following previous work on Momentum ResNets (Section 2.2), we will use a momentum term to make the blocks of a residual network invertible. In our NoCapsNNt, a block will contain capsule layers instcad of convolutional layers, which involves additional complexity when compared to Mo ResNets.
An overview of the architecture is shown in Figure 1. MoCapsNet is implemented as follows: After the first convolutional layer, we perform another convolution and reshape it into capsules to form the PrimaryCapsules layer. Between CapsLayer 1 and CapsLayer 2, the moment um residual capsule blocks (gray box in Figure 1) are inserted, each consisting of two capsule layers and shortcut connections, where the residual capsule blocks come with a modification of the forward rule. The smallest MoCapsNets consist of one residual block, as each block is composed of 2 hidden capsule layers (Figure 2) and these are added to the classical Capsule Networks. The smallest MoCapsNet will consist. of four fully connected capsule layers: First capsule layer, block (= 2 hidden capsule layers) and output capsule layer (as in Figure 1). We will explain the details on the momentun residual capsule blocks and the forward rule in Section 3.1. In order to implement the shorteut connections, an element- wise addition over the capsule votes after the squashing non- non-linearity is performed. As we add the capsule votes from a previous layer to the votes of the current layer, the addition of the shorteut connections happens after the routing process is already completed, and as such, it is totally independent of the number of routing iterations. The dimensions between the capsule layers do not change, the shortcut connection does not contain any learnable parameters. In the diagram of Figure 1 CapsLayer 2, has one capsule for cach class. Between capsule layers, we perform dynamic routing for 3 routing iterations. The reconstruction network is made up of three dense layers, where the first two layers use ReLu, and
Automatic Differentiation Package, http::///gpoott...oooaccc//ttttt html, Accessed 01/2022
3

---

Published in Transactions on Machine Learning Research (08/2022)
Input
Conv + ReLu
PrimaryCapsules
CapsLayer
n Momentum Residual blocks
CapsLayer
Reconstruction Net.
Norm
Class. OutputRecon.
Output
Figure 1: A high-level overview of our capsule network. Momentum residual blocks are shown in Figure 2b.
the last layer implements a sigmoid activation function. The reconstruction network is used to compute the reconstruction loss (Sabour et al., 2017) in the training procedure of the capsule network. The reconstruction loss corresponds to the sum squared differences between the input pixels of the image and the output from the roconstruction network. The overall loss the sum of the margin loss Imargin (Sabour al., 2017) and the reconstruction loss Lrecon, which is weighted by scalar factor A:
XLreccn + Limaria
(1)
The margin loss ensures that the vector representing capsule is large, if and only if the object that the capsule should represent is present in the input image. In our experiments, we set = 5 10-4
3.1 Momentum Residual blocks
The residual building block for capsule networks (Gugglberger et al., 2021) can be seen in Figure 2a. The design of the block is very similar to a common residual building block used in ResNet architectures, but such blocks need to be adapted in order to handle capsule layers. This is because they represent capsules (not neurons), which perform dynamic routing with upper level capsules. The layers Seape1 and feaps2 shown in Figure 2a and Figure 2b are fully connected capsule layers with dynamic routing, introduced by Sabour al. (2017). To insert shortcut connection, the input of the residual block is added to the output of the first and second capsule layers by an clement-wise addition on the capsule votes. Different from the classical ResNet architecture, the addition is performed after the non-linearity (Figure 2a), which is a squashing function (Sabour et al., 2017).
Momentum residual blocks (Figure 2b) follow from Momentum ResNets (Sander et al., 2021), which are a modification of classical residual networks. To be able to use momentum, the forward rule of the network needs to be changed in order to make the residual blocks invertible. Thanks to this new formulation,

---

Published in Transactions on Machine Learning Research (08/2022)
7
(2n Mn)
feapal On.) 71
[1
Tn++1
1 1 B
Senpua (Fo+ Outi)
Zx42
78n+2
(b)
(a)
Figure Residual building blocks for ResCapsNet (a) and MoCapsNet (b). The gray-colored elements correspond to capsule layers, and the orange-colored elements represent the addition of the shortcut connec- tions.
are able to recalculate the activations of block in the backward pass instead of having to save them into memory. The forward rule of classical residual building block is:
In+1 = In + f(zn 0n),
(2)
where f is function parameterized by 6R, and 2n is the output of the previous layer. Invertibility is achieved by changing this forward rule into velocity formulation (vw). which introduces a momentum term 7 E [0. 1]. defined as:
Un+1 = 7un + (1 (In, (x),
(3)
Making use of this velocity, the new forward rule for Momentum capsule networks is defined as:
In + Un+1-
(4)
Both forward rules can be seen in Figure where the original forward rule used in residual capsule networks (Figure 2a) can be compared to the forward rule used in momentum capsule networks (Figure 2b).
Equations 3 and 4 can now be inverted, the latter is inverted as follows:
En = Intl - Un+11
(5)
while equation 3 is inverted

---

Published in Transactions on Machine Learning Research (08/2022)
1 (Un+1 (1- (zn) 0,)).
1/
(6)
For the sake of clarity, we describe the general forms of the forward and backward steps in algorithis 1 and respectively. In the forward pass, the first step is initialize the velocity tensor v, for example by setting all values to zero (line of Algorithm 1). Next, iterate over the residual layers and compute the velocity and the output of each layer (Algorithm lines 4-6). To recompute the activations in the backward pass, we only need the velocity and activations of the very last layer by saving their corresponding tensors (line 7 of Algorithm 1).
For the backward pass, we load the output and the velocity of the last layer (line of Algorithm 2). line we initialize the variable for the gradient of the velocity. We then iterate over the residual layers re-compute the activations and velocities backwards from the last the first layer (Algorithm 2, lines 4-9). Finally, the gradients computed in line 10 of Algorithm 2. Because activations, do not need to save them in the forward pass. This process makes the memory requirements for deep residual networks much reduced, which allows the training of very deep capsule networks. Our experimental evaluation also shows that the reduction of memory consumption shrinks at much larger rates than the increase in training time, which further motivates the advantage of the proposed approach.
can re-compute the neuron
Algorithum High level Python code of the modified forward step of a momentuun capsule nelwork. The current inputs along with the residual layers are passed into the forward function below.
def forward (x, layers, gamma=0.9):
2
initialize ()
3
for layer in layers:
4
#= gamma
5
v = -1- gamma) * (xyer(x)
6
x = x +
7
save (x, v) # save for backward step
8
return X
Algorithm 2: High level Python code of the modified backward step a momentum capsule network. The current gradient along with the residual layers are arguments passed to the backward function.
def backward (x ggaadd, layers, gamma=0.9):
x, load () # load from forward slep
3
v_grad initialize
4
for layer in reversed (layers):
5
6
eval = layer (x)
7
v - -(1 gamma) * eval
8
v / gamma
grad = x grad+v grad
10
x xgradd, v y_grad compute_grai (f_eval, x, v, grad)
11
return x grad, v grad
4 Experimental evaluation
We evaluated MoCapsNet on four different datasets and compared it other capsule network models. In Section 4.1, we give detailed description of our setup and the used datasets, and we finalize summarizing results.

---

Published in Transactions on Machine Learning Research (08/2022)
Table I: Test accuracy capsule networks with increasing depth (1-8 residual blocks) on MNIST and SVHN. One block consists of two capsule layers. Shown are the average values and standard deviation over three runs. We highlight the best accuracies for each dataset in bold font.
MNIST
SVHN
Blocks
ResCapsNet
MoCapsNet
ResCapsNNe
MoCapsNet
99.41 0.01
99.42 0.04
92.32 + 0.52
92.68 + 0.23
2
99.28 0.05
99.25 = 0.11
92.57 + 0.12
92.54 = 0.43
3
99.30 + 0.08
99.31 + 0.04
92.13 + 0.59
93.00 + 0.65
4
99.38 = 0.03
99.42 = 0.05
91.87 = 0.95
92.03 = 1.12
5
99.30 + 0.03
99.27 + 0.06
92.58 + 0.42
92.78 = 0.81
6
99.35 + 0.02
99.38 + 0.02
92.37 + 0.16
92.50 = 1.60
7
99.36 = 0.02
99.54 = 0.23
92.52 = 0.23
91.35 = 1.60
99.34 = 0.06 8
99.37 = 0.06
92.63 + 0.34
91.20 = 1.58
4.1 Setup& & Hyperparameters
Our implementation is implemented in Pytorch (Paszke et al., 2019) and is publicly available on GitHub 2. The weight of the Momentum term was set to 7=0.9. We initialize the weights of the transformation matrices at random from normal distribution with mean 0 and standard deviation 0.01. The batch size for training was 128 and we trained each model for 30 (MNIST) or 60 (SVHN, CIFAR-10/100) epochs. We optimized our network weights with ADAM (Kingma & Ba, 2014), using an initial learning rate of 10 and an exponential decay of 0.96. We use 32 capsules in each capsule layer that is located inside a residual block. The shallowest ResCapsNet and MoCapsNet consist of one residual block, in either case each block is composed of 2 hidden capsule layers (Figure 2). Residual blocks are added to the classical Capsule Network, such that the shallowest ResCapsNet and MoCapsNet will consist of four capsule layers: First capsule layer, block (= 2 hidden capsule layers) and output capsule layer (Figure 1).
In the PrisaryCapsule layer, we do a convolution with a kernel size of 9 and a stride of 2. Afterwards, we do a reshaping into capsule form. All following capsule layers (CapsLayer 1, capsule layers inside residual blocks, CapsLayer 2) share the same setup. Each layer callod with the same routing algorithm, the same number of routing iterations, and has the same length. All layers have 32 capsules, except layer CapsLayer 2, which has the same number of capsules as there are classes in the dataset.
We evaluated our model on four different, popular datasets: MNIST (LeCun et al., 2010), SVHN (Netzer et al., 2012), CIFAR-10 and CIFAR-100 (Krizhevsky, 2009). CIFAR-100 contains images of 100 different classes, the other datasets contain ten classes each. We preprocess images by padding two pixels along all borders and take random crops of size 28 x 28 (MNIST) or 32 x 32 (SVHN, CIFAR-10/100) After cropping, we normalized per image to have zero mean and a variance of 1. We did not apply any data augmentation techniques.
4.2 Results
Table shows the test accuracies of ResCapsNet and MoCapsNet at various depths for MNIST and SVHN, averaged over three runs. Likewise, table 2 shows test accuracies for CIFAR-10 and CIFAR-100. CapsNet and ResCapsNet use the same model architecture, with the only difference being that ResCapsNNet uses shortcut connections between capsule layers (see Figure 2a), while CapsNet does not use residual learning. We do not include CapsNet in such table since CapsNet did not achieve better results than chance any case at any depth (Gugglberger et al., 2021). On the other hand, we obtained a consistently good performance for deeper configurations (more blocks) when using residual shortcut connections, be either ResCapsNet (Gugglberger et al., 2021) or MoCapsNet. The exception was for CIFAR-100, where only MoCapsNet provided good results, and what is even more interesting, the deeper the capsule network, the better the results. Each block of ResCapsNet and MoCapsNet consists of 2 hidden layers added between the first capsule layer and
Phttps:/rredarten

---

Published in Transactions on Machine Learning Research (08/2022)
Table 2: Test accuracy of capsule networks with increasing depth (1-8 residual blocks) on CIFAR-10 and CIFAR-100. One block consists of two capsule layers. Shown are the average values and standard deviation over three runs. We highlight the best accuracies for each dataset in bold font.
C1FAR-10
CIFAR- 100
Blocks
ResCapsNet
MoCapsNNel
ResCapsNNe
MoCapsNel
71.49 + 0.39
72.18 0.62
1.00
9.53 + 0.04
70.59 + 0.90
71.65 0.62
1.00
26.05 + 0.12
71.09 + 0.86
71.08 0.71
1.00
33.24 + 0.29
71.50 = 0.57
70.29 = 0.06
~ 1.00
39.57 = 0.11
5
71.94 + 0.34
71.74 + 0.74
2 1.00
42.54 + 0.16
71.22 + 0.70
71.17 + 0.32
R 1.00
42.79 0.13
71.85 = 0.91
71.50 0.93
~ 1.00
43.22 = 0.17
70.90 + 1.02
70.48 + 0.69
1.00
43.48 = 0.06
the output layer (Figure 1). Deeper configurations led to better results in MNIST (7 blocks 14 hidden layers) and SVHN (3 blocks 6 hidden layers). The accuracy of MoCapsNNet and CesCapsNet are almost the same for MNIST. For the case of SVHN and CIFAR-10, MoCapsNet achieves the best accuracy (72.18 %) for CIFAR-10 with residual block, as well as for SVHN (93.00 %) with 3 residual blocks. On CIFAR-100 our deepest model (8 residual blocks) performed best, with an accuracy of 43.48 %. On the other hand, with this datasot much more complex than simplifted version CIFAR-10 ResCapsNet was not able learn the 100-class recognition task all. We can see here the real benefit of using deeper capsule networks with momentum, where the other capsule network models seem to have collapsed. With CIFAR-100 we also see the large increase in performance when using deeper models, compare the case of 8 blocks (43.48 %) opposed to only one (9.53 %) or two (26.05 %) blocks.
For the sake of completeness, we compare ResCapsNNts and MoCapsNets with the CapsNet implementations from Xi et al. (2017) in Table This implementation reaches an accuracy of 68.93% on CIFAR-10, but thanks to two incremental improvements - compare CapsNet, CapsNet (2-Conv) and CapsNet (2-Conv + 4-ensemble) , an accuracy of 71.50% can be reached. These improvements consist first of the use of two convolutional layers in front of the capsule networks, and second through training 4-model ensemble. Even so, our model MoCapsNet (using the same optimizer and routing) with one residual block beats (72.18%) the best CapsNet (71.50%) without the need for an ensemble model. ResCapsNee'ss accuracy was similar to the ensemble CapsNet model. However, Sabour et al. (2017) reached an error rate of 10.6 % with an 7-model ensemble, but due to resource limitations we were not able to reproduce those results.
We chose CIFAR-10 for this analysis since it was the only dataset (Table 2) where we did not obtain higher accuracies when increasing depth by more than one block (i.e. two added hidden capsule layers between the first capsule layer and output layer). For this case, we further analyzed if there can be accuracy gains when increasing the depth of deeper (more blocks) capsule networks under different routing and optimization. Table 3 compares updates of CapsNet (Xi et al., 2017) with ResCapsNet and MoCapsNet on CIFAR-10. There are two ways we can improve MoCapsNet for better behavior on deeper architectures: Changing the optimizer and the routing algorithm. If we switch our optimizer from ADAM (Kingma Ba, 2014) to the state-of-the-art optimizer Ranger21 (Wright & Demoure, 2021), the latter handles better depth, reaching 75.15% accuracy on CIFAR-10 with three blocks, improving the results over ResCapsNet (74.24%) and MoCapsNet with one block (74.7%%). Secondly, switch our routing algorithm from Nouting-by-agreene (RBA) (Sabour et al., 2017) to designed for a better encoding of part/whole relationships. SDA routing again increased the performance, leading to our best performing model on 10,,R-10, consisting of 5 blocks and providing a 75.46% accuracy, compared with the accuracy of 73.67% for one block using the same routing and optimizer. Table 3 shows the benefit of using deeper capsule networks also in CIFAR-10 when using more novel optimizer or a more recent routing algorithm. We can also see that the change of the forward rule done in the MoCapsNet improves the models' performance when compare it against the CesCapsNet architecture in all cases.
(SDA) (Peer et al., 2018), routing algorithm
8

---

Published in Transactions on Machine Learning Research (08/2022)
Table 3: Comparison of our best performing models with a baseline capsule networks on CIFAR-10. NA
does Not Apply, as CapsNets do not include Residual blocks. Values from Xi et al. (2017)
Model
Blocks
Optimizer
Routing
Accuracy
CapsNet
NA
Adam
RBA
68.93 %
CapsNet (2-Conv)
NA
Adam
RBA
69.34 %
CapsNet (2-Conv 4-ensemble)
NA
Adam
RBA
71.50 %
ResCapsNet
Adam
RBA
71.49 %
MoCapsNe
Adam
RBA
72.18 %
ResCaps
Ranger21
RBA
73.90 %
McCapsNet
Ranger21
RBA
74.78 %
Res CascapsNet
Adam
SDA
73.16 %
MoCa Mete
Adam
SDA
73.67 %
Res CassapsNet
Ranger21
SDA
73.94 %
McCapsNet
Ranger21
SDA
72.87 %
Res CapsNet
Adam
RBA
71.09 %
MoCapsNee
Adam
RBA
71.08 %
Res CetsNet
Ranger21
RBA
74.24 %
MoCa Nept
Ranger21
RBA
75.15 %
Res Caps
Adam
SDA
74.02 %
MoCaps Net
Adam
SDA
74.91 %
Res CapsNet
Ranger21
SDA
73.90 %
MoCa Net
Ranger21
SDA
73.53 %
Res Ces
Adam
RBA
71.94 %
MoCa Net
Adam
RBA
71.74 %
Res ResCapsNet
Ranger21
RBA
74.49 %
MoCapsNet
Ranger21
RBA
73.47 %
Res CapCapsNet
Adam
SDA
74.16 %
MoCapsNet
Adam
SDA
75.46 %
CasCapsNet
Ranger21
SDA
69.07 %
NoCapsNet
Ranger21
SDA
72.63 %
Table 4: Memory consumption MB of capsule networks with increasing depth (1-5 residual blocks) when training on MNIST, CIFAR-10 and SVHN. One block contains two capsule layers. CapsNet are the results for both, CapsNet and ResCapsNet.
Dataset
Method
5
MNIST
(Res) CapsNet
4462
4612
4858
5199
5204
5502
5572
5786
MoCapsNet
4320
4324
4328
4332
4336
4546
4552
4554
CIFAR-10
(Res) CapsNet
7605
7865
8127
8197
8139
8399
8437
9179
MoCapsNet
7653
7657
7660
7663
7667
7671
7675
7679
SVHN
(Res) CapsNet
7326
7586
7763
7918
7860
8120
8158
8900
MoCapsNet
7371
7375
7379
7381
7385
7389
7393
7397
The main goal of our MoCapsNet is the ability to train deeper capsule networks, which is hindered by the large memory requirements of capsule networks (section 1). We show the memory consumption in Table 4 (and figs. 3a, 3d and 3g). Values shown are the average over three runs. The memory consumption for MoCapsNet increases by only MB when adding a new residual block. On the other hand, adding a residual block to ResCapsNet increases the memory footprint by around 185 MB, which is 45 times the memory increase MoCapsNet. While our MoCapsNeet needs roughly the same amount of memory as ResCapsNet when using one residual block, if make the model deeper, for example, with 8 residual blocks, MoCapsNet requires 500 MB less for CIFAR-10 and 1500 MB less for SVHN as its non-reversible counterpart CesCapsNet.

---

Published in Transactions on Machine Learning Research (08/2022)
Memory Consumption
Training Time
Inferance Time
5753
140
350
Casuter Matiaute:
134
Cscdett Nicksei
70
214
+700
100
4000
40
number blocks
num
number of
(b)
Memory Consumption ton
Training Time
Inferinc Time
14.
15.
NuKasata
00.0
15.
m-nnoyy
14.
14.
30:0
1400
114
125
number ofblocks
number of blocks
number blocks
(d)
Mamory Consumption
Training Time
Inferen Time
3504
Cscdett Wiclssit
-spolet ScCapole
240
2608
namber blocks
number
namber blocks
(h)
Figure 3: Comparison of memory consumption (first column: (a), (d), (g)), training time (second columm: (b), (c), (b)), and inference time (third column: (c), (f), (i)) between CapsNet (bluc), ResCapsNet (red), and MoCapsNet (green) with increasing nelwork depth on MNIST (first row, (a) to (c)), CIFAR-10 (second row, (d) to (f)), and SVHN (third row, (g) to (i)). As training and inference time does not vary much across epochs, we report values of the last epoch only.
Note that the memory consumption of CapsNet and ResCapsNet are the same because residual connections do not contain learnable parameters. For the sake of completeness, we also trained a very deep MoCapsNet with 20 blocks (=40 layers) on SVHN and got stable accuracy of 91.67%.
The price pay for saving memory be network has a steeper increase (roughly by factor of 1.8) when using the partly invertible architecture of our MoCapsNet due to re-computations of neuron activations in the backward pass. Inference time is on par, even though it was slightly higher when using momentum (just 0.5s in the worst case, i..., approx. 5% higher), as figures 3b, 3c, and 3h show, which is the result of the new forward rule. This new forward rule requires some small additional computation due to multiplying the momentum term y with the velocity of the previous layer and the activations of the current layer (see lines 4-5 in Algorithm 1). For the sake of
figures 3b, 3e, 3h. The required time for training the
10

---

Published in Transactions on Machine Learning Research (08/2022)
Memory Consumption
Training Time
9200
CapsNet: f(x) 174.2 X 7459
50
CapsNet: fix) 11.031 X 34.37
ResCapsNet: T(x) 174.2 7459
ResCapsNet: ((x) = 1.092 x 35.06
9000
MoCapsNet: f(x) 3.643 7649
MoCapsNet: f(x) 1.882 35.17
training time [s]
memory [MB]
46
8600
8400
42
8200
40
7800
36
7600
2
3
6
number of blocks
number of blocks (b)
(a)
Figure 4: Comparison of memory and training time on CapsNet, ResCapsNet and MoCapsNet for CIFAR- with an increasing number of residual blocks (one block consists of two layers). The legend shows the memory consumption (a) or training time (b) as a lincar function of the number of blocks.
completeness, we analyze deeper the effect on training time compared to memory savings in Figure 4. The memory requirements as the capsule network increases in depth are much flatter in the case of MoCapsNet than CapsNet or ResCapsNet, the steepness of memory as a function of the nunber of blocks (1 block = layers) for the last two is 174.2, while for MoCapsNet just 3.643, that is, memory increase over depth is almost 48 times less for MoCapsNet than for CapsNet or ResCapsNet. Even though the training time was steeper in the case of MoCapsNet, the difference with CapsNet and ResCapsNet was small when considering the memory differences. Training time function of depth is shown in Figure 4b, adding momentum meant just 1.8 times the computational load (steepness of 1.882 for MoCapsNet vs 1.031 for CapsNet).
We finalize our evaluation by analyzing different hyperparameter values which deviate from the usual setup (section 4.1). Our basic model has capsule types in each capsule layer that located inside just one residual block. Our evaluation includes different batch sizes, learning rates and the number of capsules the residual blocks. In order lo evaluate the effect of momentum under those hyperparameler variations, our baseline model here for comparison is residual capsule network (ResCapsNet) with two residual blocks, batch size of 128, an initial learning rate of 0.01, optimized with Ranger21 (Wright & Demoure, 2021), and trained over 100 epochs.
We compare ResCapsNet and MoCapsNet in Table 5. Neither a higher or lower batch size nor a higher or lower lcarning rate has big impact on the accuracy of the model. MoCapsNet achieved better results than ResCapsNet all cases except for the case of higher learning rate of 0.1, which had a higher negative impact in MoCapsNet than in ResCapsNet. It is also worth noting that increasing the number of capsules in the layers of the residual blocks to 64 has caused some performance gain in ResCapsNNt, but also it has very high memory consumption of about 14.6 GB, whereas ResCapsNet with 32 capsules per layer needs only 7.9 GB.
5 Conclusions & Future Work
We have introduced in this paper Momentum Capsule Networks (MoCapsNet). new capsule network architecture that implements residual blocks using capsule layers, which can be used to construct reversible building blocks. Through the use of reversible subnetworks, we have obtained a network that has much smaller memory footprint than its non-invertible counterpart. This fact would allow for the training of capsule networks at much deeper configurations than current. setups, thanks to much more reduced memory
11

---

Published in Transactions on Machine Learning Research (08/2022)
Table 5: Comparison of ResCapsNet and MoCapsNet on CIFAR-10 for different learning rates (0.01, 0.001
and 0.1), number of capsules (32 and 64) in the residual blocks and batch sizes (64, 128 and 256).
Model
learning rate number capsules
batch size
accuracy
ResCapsNet
0.01
32
128
74.85 %
MoCapsNet
0.01
32
128
75.26 %
ResCapsNet
0.01
32
64
72.02 %
MoCapsNet
0.01
32
64
73.85 %
ResCapsNet
0.01
32
256
74.03 %
MoCapsNet
0.01
32
256
74.13 %
ResCapsNet
0.01
64
128
74.83 %
MoCapsNet
0.01
64
128
75.52 %
ResCapsNet
0.001
32
128
72.70 %
MoCapsNet
0.001
32
128
73.14 %
ResCapsNet 0.1
32
128
70.84 %
MoCapsNet
0.1
32
128
68.98 %
consumption. The trade-off of reversible architectures is an increased training time. In spite of that, our experimental evaluation shows that the reduction of memory consumption shrinks at much larger rates than the increase in training times, which added similar inference time, shows the advantage of MoCapsNet over capsule networks not using reversible building blocks. Our results show that the memory requirements as the capsule network increases in depth are much flatter (Figure 4a) in the case of MoCapsNet than for CapsNet or ResCapsNet, the steepness of memory 2 layers) for MoCapsNet is 48 times less steep than for CapsNet or ResCapsNet. On the other hand, the added computation time when compared with CapsNet and ResCapsNet was quite small (Figure 4b) if we consider such great memory savings. This clear benefit of MoCapsNet over non-reversible architectures is what allows for the training of deeper capsule networks where the current limiting factor is memory. In terms of performance, we have shown experimentally that the modification of the forward rule leads to much improved results of the capsule network. MoCapsNet provides even better results in terms of accuracy on MNIST, SVHN, and CIFAR-10 than recent improved CapsNets consisting of ensembles of networks (Table 3).
function of the number of blocks (1 block
Similarly to what happens for CNNs, deeper capsule nelworks configurations performed better in most cases than flat ones. Recent optimizers such as Ranger21 (Wright & Demeure, 2021) led to even better results on deeper MoCapsNets.. Similarly, routing algorithms designed for deeper architectures such as scaled-distance- agreement (SDA) provided higher accuracy values than using the classical routing- by agreement (RBA). In fact,, our best performing model on the quite gegsuue-networr-chllennngii 5 residual blocks deep and using SDA routing, with 75.46 % accuracy. For completeness, we included hyperparameter analysis that evaluated different values of learning rales, number of capsules and batch sizes, increasing the number of capsules had a positive effect on MoCapsNet, while changing the batch size or learning rate did not have a big influence or worsened the results.
dataset CIFAR-10 consisted of
For future work, it would of interest lo analyze replacing the fully connected capsule layers with convo- lutional capsule layers, which would enable save even more memory. Such modification could allow for the application of capsule networks to more complex tasks and/or datasets, such as ImageNet.
References
Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architoc- tures for arbitrarily deep residual neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Siwei Chang and Jin Liu. Multi-lane capsule network for classifying images with complex background. JEEE Access, 8:79876 79886, 2020.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Xinpeng Ding, Nannan Wang, Xinbo Gao, Jie Li, and Xiaoyu Wang. Group reconstruction and max-pooling residual capsule network. In IJCAT, pp.
2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249 256. JMLR Workshop and Conference Proceedings, 2010.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Proceedings of the 31st International Conference Newral Information Processing Systems, pp. 2211-2221, 2017.
Josef Gugglberger, David Peer, and Antonio Rodríguez-Sánchez. Training deep capsule networks with resid- ual connections. In Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter (eds.), Artificial Neural Networks and Machine Learning ICANN 2021, pp. 541-552, Cham, 2021. Springer International Publishing. ISBN 978-3-030-86362-3.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 770 778, 2016.
Geolfrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In International conference on learning representations, 2018.
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universitäl München, (1(,), 1991.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelorating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprinl arXiv:1412.6980 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Πyya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25:1097 1105, 2012.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. MNIST handwritten digits. http: /ann. Lecun. com/ endb/ mist// 2010.
Yuval Netzer; Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew V. Ng. Tearning methods for generic object recognition with invariance to pose and lighting, Reading Digits in Natural Images with Unsupervised Feature Learning NIPS Workshop on Deep Learning and Unsupervised Fealure Learning, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http:///apeet nourips.cc/paperr. cc/paper/ wnt.toordaaar-mmeaatiieee--tttte httppppceeeietttteeiieeorrrrrrrr wwwporrraaanccddeeereeee -iwaamiinnn-ibrrry..mooa wwwrrrrrbrrr.r... pddf.
David Peer, Sebastian Stabinger, and Antonio Sänccez Increasing the adversarial robustness and explainability capsule networks with gamma-capsules. arXiv preprint arXiw:1812.09707 2018
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In international conference on machine learning. pp. 2847-2854. PMLR, 2017.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Fabio De Sousa Ribeiro, Georgios Leontidis, and Stefanos Kollias. Capsule routing via variational bayes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 3749-3756 3756, 2020.
Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 3859-3869) 2017.
Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Momentum residual neural networks. arXiv preprint arXiv:2102.0787 078700 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arxii:14053 1556, 2014.
Kun Sun, Xianbin Wen, Liming Yuan, and Haixia Xu. Dense capsule networks with fewer parameters. Soft Computing, 25100)6997-6645 2021.
Martin Wistuba, Ambrish Rawat, and Tejaswini Pedapati. survey on neural architecture search. arXiv preprinl ar Xiv: 01392, 2019.
Less Wright and Nestor Demeure. Ranger21: synergistic deep learning optimizer, 2021.
Edgar Xi, Selina Bing, and Yang Jin. Capsule network performance on complex data. ar preprint arXiv:1712.08.4 03480, 2017.
Canqun Xiang, Lu Zhang, Yi Tang, Wenbin Zou, and Chen Xu. Ms-capsnet: novel multi-scale capsule network. IEEE Signal Processing Letters, 25(12):1850 1854, 2018.
Shnai Yang, Feifei Lee, Ran Miao, Jiawei Cai, Lu Chen, Wei Yao, Koji Kotani, and Qiu Chen. Rs-capsnet: an advanced capsule network. IEEE Access, 8:85007-8501 2020.
14


Reviews:
[{"review_id": "GnRIoCt4Sm", "reviewer": "Reviewer_xYkD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- The proposed method was able to reduce memory consumption of training capsule networks, which is the main motivation for this submission\n- The proposed method shows performance improvement over the capsule network\n\n## Weakness\n\n- Proposed method is not novel.\n- Paper could be organized in better form and writing needs to be improved. I provide suggestions in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new architecture, Momentum Capsule Networks, to address the high memory consumption required for training Capsule Networks. The key contribution is to borrow techniques in reversible networks to recompute activations in the backward pass, so that activations do not need to be saved. \n\nThis submission evaluates the proposed network on three standard image classification benchmarks, MNIST, SVHN, and CIFAR10. The proposed network improves the performance of Capsule Networks and reduce the training memory consumption. However, the proposed network was not shown to reach state of the art results (in either the line of capsule network research or considering other convolutional network architecture). \n\nOverall, I find the submission to be a direct application of existing techniques (reversible network) to Capsule network, which means that the submission is less novel. I find the presentation to be confusing and not well-organized at times and provide suggestions in the following sections. In summary, I recommend rejection of the submission.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "## Requested Changes\n\n- The first paragraph of the introduction should be about the main claim of this paper. Currently, it described many low-level details about the capsule network, which makes this paragraph very long and hard to read\n- The second paragraph of the introduction mentioned three problems of the capsule networks but this paper really only touches upon the last one. I suggest to cut the first two and focus on what this paper is trying to solve.\n- The second paragraph of the introduction conflates two challenges in training deeper capsule networks: instability and memory consumption. This paper only aims at solving the latter. I suggest the authors to remove the discussion on instability.\n- The third paragraph of the introduction provides an overview of the results. I recommend the authors to cite concrete numbers to showcase the benefits of the proposed methods.\n- Section 2.1, “capsule networks require vast amounts of memory.” is not supported by the following example “… consisting of 14.36 million parameters”. The claim is about memory consumption during training and the example is about the parameter count. These are two different concepts\n- Section 2.3, unclear how this line of work is relevant to the submission\n- Section 3, first paragraph, the claim about “requires significant more memory during training” needs to be supported by actual numbers.\n- I suggest to reorganize Section 3 into subsections: 3.1 background on capsule network; 3.2 background on reversible network; 3.3. proposed method.\n- Section 3.1, “The design of the block is very similar to a common residual building block used in ResNet architectures, but such blocks need to be adapted in order to handle fully connected capsule layers.” The paper should explain the reason for why there is such a “need to be adapted”.\n- Section 3.1, “Differently from the classical ResNet architecture,” → “Different from”.\n- I suggest to formalize the descriptions in text and figure 1 into equations.;\n- Algorithm 1 and 2 are now named “Listing” in captions.\n- Section 4.2, “However, Sabour et al. (2017) reached an error rate of 10.6 % with an 7-model ensemble, but due to resource limitations we were not able to reproduce those results.” This is very unsatisfying. Can the author explain what kind of resource would be required to reproduce such results?\n- Figure 3 is very confusing. The authors should 1. show legends 2. mark different plots with the network depth. Also the plots on different rows look very similar and I am not sure there is much value in duplicating them many times.\n- “The required time for training the network has a steeper increase when using the partly invertible architecture of our MoCapsNet due to re-computations of neuron activations in the backward pass.” please provide actual numbers describing “stepper increase”.", "broader_impact_concerns": "I do not have ethical concern with the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GZNqtCdOvg", "reviewer": "Reviewer_QSyL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper did a thorough investigation comparing CapsNet, ResCapsNet and MoCapsNet on several datasets using different hyperparameter settings. \n\nWeakness:\n- The combination of momentum residual connection and capsule network might not be particularly interesting to investigate. On one hand, the reversibility of momentum residual networks can give memory saving benefits for many network architectures. On the other hand, capsule networks do not seem to be particularly memory intensive. In Figure 3, all of the memory consumption curves are well under 10GB, which can be handled by modern GPU hardware.\n- The paper did a good job explaining the momentum residual connections, but for completeness it should also describe what is inside f_caps.\n- The differences between ResCapsNet and MoCapsNet is small (I don’t think it should be the focus of the paper anyways since the performance is expected to be comparable from the momentum residual net paper).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Momentum Capsule network, a new network architecture design that combines the capsule network with momentum residual modules. In particular they insert momentum residual modules in between the capsule layers. This results in memory saving benefits since the momentum residual modules are reversible. The resulting MoCapNet (using momentum residual connections) is also marginally better than ResCapNet (using residual connections).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- The main concern of the paper is that the memory saving story isn't particularly strong. If the author could show that with this technique they can train a very deep CapsNet with much stronger performance, then it would be more impactful. However, Table 5 suggests that the performance already plateaued with 5 blocks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sDBOxWHuIB", "reviewer": "Reviewer_fsDs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\n- The paper clearly demonstrates that MoCapsNet allows to reduce the memory requirement of capsule networks at the expense of  computation.\n- MoCapsNet performs better than other capsule network baselines (ResCapsNet and CapsNet).\n- Authors say that they will release the code to ease reproducibility of the results.\n\n*Weaknesses:*\n- Imprecise claim: Authors claim that they can \"train capsule networks at almost any arbitrarily deep configuration\" and argue that \"the bottleneck for training deeper networks is memory\".  However they only considers MoCaspNet of 8 blocks max in the experiments. Depth usually introduces optimization difficulty. It is unclear if MoCapsNet eases the optimization of very deep capsule network. \n\n- Clarity could be improved:  What is the parametrization of PrimaryCaspules and CapsLayer 1/2 in Figure 1? In particular,  it was unclear that the paper was focusing on fully connected capsule from my first read. Similarly,  what is margin loss in equation (1) ? Could you extend on the sentence that \"the shortcut connection is inserted after the routing process is completed,\n\n- The memory/compute tradeoff of reversible architectures has been explored as highlighted in the related work section.  The specific application of reversible architecture to capsule network seems novel.\n\nQuestions:\n- Why do you report the training time only of the last epoch in Figure 3?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims at tackling the high-memory requirement of capsule network. It proposes MoCapsNet, a neural network parameterization that combines  capsule and momentum residual network. Momentum residual network is a reversible architecture that allows to recompute the network activations during the backward pass and discards the need of storing them. \n\n The momentum capsule network is evaluated on CIFAR10, SVHN and MNIST datasets. Empirical evaluation shows that MoCapsNet reduces the memory requirements at the expense of computation. MoCapsNet also outperforms previous capsule network baselines such as CapsNet and ResCaspNet.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I would encourage the authors to either add experiments that supports better the claims made in the introduction or to precise those claims, specifically regarding that MoCapsNet allows the training of almost any arbitrarily deep configuration.", "broader_impact_concerns": "No specific concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work addresses the memory bottleneck that comes with deep capsule networks, by investigating an architectural variant based on reversible residual blocks, or `Momentum ResNets'. The authors carry out experiments on several mid-scale benchmarks, demonstrating improvements in memory consumption and performance relative to other capsule-based models. 

The three experts reviewers who assessed this work agreed that this work does provide memory and performance improvements over other capsule networks, enabling training of deeper architectures. They also noted that, in its current form, this paper does not fully capitalize on this observation, and that the submission would be much more convincing if the authors could leverage the memory savings to train a large-scale capsule network with strong empirical performance on a larger-scale dataset. The AE agrees with this assessment. In conclusion, according to the TMLR guidelines we are happy to accept this manuscript, while strongly encouraging the authors to make a final effort to include these experiments. 
Decision: Accept (poster)