# Simplified Routing Mechanism for Capsule Networks
## 6 Results  
In this work, the network architecture presented in Section 4 has been designed in three different ways and trained separately on the datasets presented in Section 5. The difference between the three networks is the routing algorithm used: the first is the original capsule network by [PERSON] et al., the second is our modified capsule network with some improvements, and the third is the efficient vector routing by [PERSON] [20]. Capsule networks are trained separately on the six presented dataset. For the implementation, we used Python 3.9.16 [21] programming language with PyTorch 1.12.1 [22] machine learning framework and CUDA toolkit 11.6 platform. The capsule networks are trained on the Paperspace [23] online artificial intelligence platform with an Nvidia Quadro RTX4000 series graphical processing unit. We trained all networks for 35 epochs with the [PERSON] [24] optimizer algorithm where the train and test batch size are both 128. We also attempted to train the networks over many more epochs, but found that the difference between the three solutions does not change significantly after 35 epochs. In this study, we used \(5\times 10^{-4}\) initial learning rate. In each epoch, we reduced the learning rate as follows:  
\[lr_{i}=lr_{init}\times 0.97^{i} \tag{13}\]  
where \(lr_{init}\) is the initial learning rate and \(lr_{i}\) is the learning rate in the \(i\)-th epoch. We used \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\) hyperparameter values to control the exponential decay, and \(\epsilon=1\times 10^{-8}\) to prevent any division by zero in the implementation. In the training process, we used the same loss function as proposed by [PERSON] et al. \[\mathcal{L}=T_{k}\times\max\bigl{(}0,\ m^{+}-p\bigr{)}^{2}+\lambda\times(1-T_{ k})\times\max\bigl{(}0,\ p-m^{-}\bigr{)}^{2} \tag{14}\]  
Figure 12: Sample data from GTSRB dataset. where  
\[T_{k}=\left\{ \begin{array}{ll}1,&\text{if object of class $k$ present}\\ 0,&\text{otherwise}\end{array} \right. \tag{15}\]  
\(m^{+}\), \(m^{-}\) and \(\lambda\) are hyperparameters. In the present work, we used the same values for these three hyperparameters as proposed by [PERSON] et al., in this case, \(m^{+}=0.9\), \(m^{-}=0.1\) and \(\lambda=0.5\). The original study also used reconstruction in the training process; however, we did not apply this in our work. During training without reconstruction, the efficiency of the capsule network is reduced. In the long term, we want to translate our results in the field of capsule networks into real-world applications. In this respect, reconstruction of the input image is a rarely necessary step. Therefore, we explicitly investigated the ability of capsule networks without reconstruction. Figures 13-18 show the accuracy of the training processes for the test sets of the six presented datasets with different numbers of routings. The value of r indicates the number of iterations which the routing algorithm has optimized the coefficients. Based on [PERSON] et al. 's experiment, the r = 3 is a good choice; however, we showed the efficiency with \(\texttt{r}=1\) and \(\texttt{r}=10\). This makes the difference more visible between the routing methods. As can be seen, for all six datasets, we have achieved efficiency gains compared to the two other capsule network solutions. The difference in efficiency between our and [PERSON] et al. 's solutions is minimal for about the first 10 epochs; however, after that, there is a noticeable difference in the learning curve. Although [PERSON]'s solution also proves to be effective in most cases; its performance is slightly lower than the other two solutions. It is also noticeable that changing the number of iterations has a much larger impact on [PERSON] et al. 's solution and that of [PERSON]. Our proposed solution is less sensitive to the iteration value chosen during the optimization. Figure 14: Classification test accuracy on Fashion-MNIST dataset. Figure 13: Classification test accuracy on MNIST dataset. Figure 16: Classification test accuracy on CIFAR10 dataset.