Title: A QUANTIFIABLE TESTING OF GLOBAL TRANSLA-TIONAL INVARIANCE IN CONVOLUTIONAL AND CAP-SULE NETWORKS / CONFERENCE SUBMISSIONS

Abstract: We design simple and quantifiable testing of global translation-invariance in deep learning models trained on the MNIST dataset. Experiments on convolutional and capsules neural networks show that both models have poor performance in dealing with global translation-invariance; however, the performance improved by using data augmentation. Although the capsule network is better on the MNIST testing dataset, the convolutional neural network generally has better performance on the translation-invariance.

1. INTRODUCTION:

Convolutional neural networks (CNN) have achieved state-of-the-art performance than the human being on many computer vision tasks Krizhevsky et al. (2012) ; He et al. (2015) . The deep learning community trend to believe that the success of CNN mainly due to two key features in CNN, reduced computation cost with weight sharing in convolutional layers and generalization with local invariance in subsampling layers Lecun et al. (1998) ; Lenc & Vedaldi (2014) . Due to convolutional layers are 'place-coded' equivariant and max-pooling layers are local invariant Goodfellow et al. (2016) , CNN has to learn different models for different viewpoints which need big data and expensive cost.

More Generalization model should be able to train on a limited range of viewpoints and getting good performance on a much more wider range. Capsule network is robust in dealing with different viewpoints Hinton et al. (2011); Sabour et al. (2017) ; Hinton et al. (2018) . Capsules are a group of neurons which includes the pose, colour, lighting and deformation of the visual entity. Capsule network aims for 'rate-coded' equivariance because it's the weights that code viewpoint-invariant knowledge, not the neural activities. Viewpoint changes in capsule network are linear effects on the pose matrices of the parts and the whole between different capsules layers. However, it still unclear whether capsule networks be able to generalize for global translation invariance.

Visualize and Quantify the translation-invariance in deep learning model are essential for understanding the architectural choices and helpful for developing Generalization model that is invariant to viewpoint changes. An analysis using translation-sensitivity map for MNIST digit dataset has been used to investigate translation invariance in CNN Kauderer-Abrams (2017) . In this paper, we introduce a simple method to test the performance of global translation-invariance in convolutional and capsule neural network models trained on the MNIST dataset.

2. GLOBAL TRANSLATIONAL INVARIANT TESTING:

Global translational invariance (GTI) of a deep learning model trained on the MINST data which can test by using a simple testing dataset. All images in the GTI testing dataset generated by shifting the centre of mass of a Helvetica font digit from top left to bottom right one pixel each time as shown in Figure 1 . The TI testing images size is 28 × 28 which the same size as MNIST images. Locations of the centre of mass for each digit has 18 columns and 14 rows. In total there are 18 × 14 × 10 = 2520 testing images, which cover all the possible cases of translational translations. We train all deep learning models using MNIST training dataset which includes 60000 samples, and testing on both MNIST testing dataset with 10000 samples and GTI testing dataset with 2520 samples. Nearly all the images in the MNIST dataset are located at the centre of canvas while the GTI dataset distributes uniformly on the canvas.

Compare to translation-sensitivity maps method based on average on numerous images of each class in MNIST testing dataset and shift it to all possible translations Kauderer-Abrams (2017), our way is robust to overcome random noise due to miss labelling in the MNIST testing dataset and our GTI training dataset is much smaller. Since the GTI training dataset is identical for testing all different models, it can capture the tiny difference in those models. Another advantage of the GTI dataset is it natural to quantify the global invariance because the accuracy of model predictions on the GTI testing dataset reflects the ability of the model to dealing with global translational invariance. 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 2 9 9 9 9 9 9 0 0 0 0 0 0 4 4 4 4 4 4 9 9 9 9 0 0 0 0 0 0 0 0 1 1 1 4 2 8 9 9 9 9 9 0 0 0 0 0 0 0 2 1 1 4 0 0 0 9 9 9 0 0 0 0 0 0 0 0 1 1 1 8 0 0 0 0 0 0 0 0 0 0 0 0 0 6 1 1 1 8 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 5 0 0 0 0 0 0 0 0 0 0 0 6 6 1 1 1 1 5 7 0 0 0 0 0 0 0 0 0 6 6 6 6 1 1 1 5 5 5 8 0 5 0 0 0 6 6 6 6 6 6 1 7 7 5 5 5 2 2 2 2 2 6 6 6 6 6 6 6 6 7 7 5 5 5 2 2 2 2 2 2 6 6 6 6 6 1 7 7 7 To improve the performance of CNN on the GTI dataset, we train the MNIST with data augmentation by shifting the image from the centre in x and y-direction. The accuracy on GTI testing dataset increase to 98.05% by randomly moving the centre of an MNIST training image in x and y-direction up to 30% of width or hight. Data Augmentation of training dataset to improve the performance imply place-code' equivariance in CNN, because those neurons at the corner of feature maps are activated when the model start to see training sample with objects at the edge.

4

Without Augmentation Shift 20% in training data:



3.2. CAPSULE NETWORK:

We test GTI dataset on CapsNet with the same architecture in Sabour et al. (2017) bashed on the Keras Chollet (2015) . The CapsNet has 8.2M parameters that are about 23 times larger than the CNN. We trained the model with Adam optimizer and using exponential decay of learning rate with decay parameter 0.9. We use margin lass with the same parameters in Sabour et al. (2017) . We also add reconstruction loss but scale down it by 0.0005.

Capsule network is robust in viewpoints invariance of object recognition; however, our experiment implies capsule network's performance on global invariance has yet to be improved. As we can see CNN's performance on GTI dataset generally better than the CapsNet as shown in Figure 5 . The accuracy on CNN is always better than the CapsNet with different amount of shifting, even the convolutional layers in CapsNet using wider receptive fields. Since we remove the max-pooling layers in the CapsNet and the convolutional layers in the CapsNet is 'place-coded' equivalent, which could be the reason that the CapsNet has lower performance on the GTI dataset. We believe there is much room to improve for the CapsNet to handling translational invariance.

4. CONCLUSION:

We introduce a simple GTI testing dataset for deep learning models trained on MNIST dataset. The goal is to get a better understanding of the ability of CNN and CapsNet to dealing with global translational invariance. Although the current version of CapsNet could not handle global translational invariance without data augmentation, we still believe CapsNet architecture potentially better than CNN on dealing with global translational invariance because capsules could train to learn all viewpoint no matter it receives the information for the centre or the edge. Our testing method is sample and quantifiable, and it easy to implement for other datasets of computer vision tasks by taking a clear and correct labelled image from each class and apply the translational shifting to cover all possible cases.

:

EXPERIMENTS ON GTI TESTING DATASET 3.1 CNN The CNN model has nine layers as shown in Figure 2. Layer 1, 2,

4, 5 are convolutional layers with 32, 32, 64, and 64 channels respectively. Each layer has 3 × 3 filters and a stride of 1. Layer 3 and 6 are max-pooling layers, and layer 7, 8, 9 are fully connected layers with size 256, 128 and 10 respectively. Dropout with the drop rate of 0.5 applies to the first max-pooling layer and all the fully connected layers. The total number of parameters is 361578 which is about 23 times smaller than the Capsule networks. Except the last layer's activation function is softmax, all the other layers use ReLU. The optimizer is Adam with default parameters in Keras, and the objective function is cross entropy loss.The results of CNN shown in Figure3and Table1. The CNN models trained on MNIST data, thus it achieves very high accuracy on the MNIST testing set. However, the model trained on the MNIST training dataset without any data augmentation has only 42.16% accuracy on GTI testing dataset, which implies CNN's performance on dealing with global translational invariance is abysmal. As we can see in the left picture of Figure3, images with the digit's centre of mass around the centre of the canvas predicted correctly, and images with the number at the corner assigned to an incorrect class. Those images around the centre are classified correctly due to the max-pooling layers preserve the local invariance in feature maps. Since only MNIST images used to train the model, and the model could not accurately predict those images shift toward the corner in GTI dataset, which strongly suggests that CNN is 'place-code' equivariant.

Figures and Tables:

Figure 1: Global translational invariant (GTI) testing dataset. The image shows a helvetica font digit 0 located at the top left. The 18 by 14 lattice are locations of center mass of the digit if we shift the image toward right or bottom one pixel each time.

Figure 3: Prediction of CNN models on digit 0's images in GTI testing set. The location of each number in the above pictures corresponding to the centre of mass lattice shown in Figure 1. (Left) The model trained on the original MNIST data without any data augmentation; (Right) the model trained on MNIST data with randomly shifting 20% width or height of the centre of mass. Correct predictions coloured by red, and black numbers are wrong predictions.

Figure 4: Generated images and prediction of CapsNet models on digit 0's images in GTI testing set. The location of each picture in the above pictures corresponding to the centre of mass lattice shown in Figure 1. (Left) The model trained on the original MNIST data without any data augmentation; (Right) the model trained on MNIST data with randomly shifting 20% width or height of the centre of mass. Inside brackets at the top of each image are ground true classes and prediction of CapsNet.

Figure 5: GTI dataset accuracy of models trained on CNN and CapsNet with different amount of random shifting in MNIST training dataset.

CNN performance on MNIST and GTI testing datasets.

Table content: <table><tr><td>0.45 0.01</td><td>57.84 0.98</td><td>0</td><td>0</td></tr><tr><td>0.43 0.06</td><td>44.38 0.84</td><td>5</td><td>5</td></tr><tr><td>0.45 0.03</td><td>24.63 0.40</td><td>10</td><td>10</td></tr><tr><td>0.55 0.04</td><td>10.05 2.34</td><td>15</td><td>15</td></tr><tr><td>0.55 0.07</td><td>3.81 1.07</td><td>20</td><td>20</td></tr><tr><td>0.56 0.06</td><td>1.95 0.46</td><td>30</td><td>30</td></tr><tr><td colspan="2">Without Augmentation</td><td colspan="2">Shift 20% in training data</td></tr></table>

CapsNet performance on MNIST and GTI testing datasets.

Table content: <table><tr><td colspan="4">MNIST error (%) TI error (%) Shift x (%) Shift y (%)</td></tr><tr><td>0.47 0.02</td><td>67.46 0.54</td><td>0</td><td>0</td></tr><tr><td>0.43 0.03</td><td>53.41 0.62</td><td>5</td><td>5</td></tr><tr><td>0.36 0.02</td><td>32.54 0.35</td><td>10</td><td>10</td></tr><tr><td>0.39 0.04</td><td>16.07 1.12</td><td>15</td><td>15</td></tr><tr><td>0.50 0.06</td><td>7.62 1.21</td><td>20</td><td>20</td></tr><tr><td>0.75 0.08</td><td>6.55 0.69</td><td>30</td><td>30</td></tr></table>

Peer review (iclr2019):
 The authors of this paper compare the robustness of CNN and CapsNet to global translation on the MNIST dataset. Both models were trained on the standard training set of MNIST, and then tested on a set with digits shifting from the upper left corner to the lower right corner. The results of both models were poor. To improve it, the authors add some shifted digits to the training set, and the performances of both models were significantly enhanced. Moreover, the performance of CNN was better than that of CapsNet in the experiments. Generally speaking, the work presented in this paper is clear and straightforward. However, the work is not significant enough to publish as a ICLR paper. Below is my major comments. 1. There are lots of typos and grammatical errors everywhere in the paper. Thus, the manuscript was not well prepared. 2. It is unclear which CapsNet and what settings were used in the experiment. 3. It is well-known that convolutional networks are good at capturing local patterns from the images, while capsule networks enhance it to consider global configurations of the local patterns, and robust to affine transformation. Obviously, the experiments presented in this manuscript is too simple. Lots of work should be done in the investigation. For example,o on the training set and shifted test set, the authors can enlarge the background and keep the digits in the original size to make it as a local pattern in the image. Will it be detected by CNNs with larger receptive fields for the images? How is it compared with CapsNets? 4. How are both models compared on other (perhaps more complicated and larger) datasets? In summary, the work presented here is interesting, but lots of work should be done in order to make it publishable.