# DE-CapsNet: A Diverse Enhanced Capsule Network with Disperse Dynamic Routing
## 3 Diverse Enhanced Capsule Network
### Disperse Dynamic Routing  
The inputs to the digit capsules (DigitCaps) are the \"prediction vectors\" \(\hat{u}_{ji}\) produced by learned transformation weight matrices and the outputs of the primary capsule layer [4]. The routing algorithm calculates the \"digit capsules\" \(v_{j}\) from \(\hat{u}_{ji}\), which is kept fixed throughout the procedure. The dynamic routing procedure from reference [4] is given as follows (Algorithm 1).  
```
1:Input to Routing Procedure: \((\hat{u}_{ji},r,l)\)
2:for capsule \(i\) in layer \(l\) and capsule \(j\) in layer \((l+1)\): \(b_{ij}\gets 0\)
3:for\(r\) iterations:
4:for capsule \(i\) in layer \(l\) : \(c_{ij}\gets Softmax\left(b_{ij}\right)\)
5:for capsule \(j\) in layer \((l+1)\): \(s_{j}\leftarrow\sum\limits_{i}c_{ij}\hat{u}_{ji}\)
6:for capsule \(j\) in layer \((l+1)\): \(v_{j}\gets Squash\left(s_{j}\right)\)
7:for capsule \(i\) in layer \(l\) and capsule \(j\) in layer \((l+1)\): \(b_{\bar{i}}\leftarrow\ b_{\bar{i}}+\ \hat{u}_{\bar{i}\bar{j}}\cdot v_{j}\)
8:Return\(v_{j}\)
```  
**Algorithm 1**Softmax Routing Procedure  
\(\hat{u}_{ji}\) is the prediction vector that means feature \(i\) belongs to final digit capsule \(j\). Digit capsule \(j\) matches one class of images. The coupling coefficient \(c_{ij}\) represents the relative strength between primary capsule \(i\) and digit capsule \(j\), which are refined as Equation (2) is iterated. \(s_{j}\) is the input of digit capsule \(j\) and is calculated via Equation (2). In the digit capsule layer, we restrict the vector \(s_{j}\) to \(1\) by the squashing function as shown in Equation (8) [4].  
\[v_{j}=\ \frac{\left\|s_{j}\right\|^{2}}{1+\left\|s_{j}\right\|^{2}}\frac{s_{j}}{ \left\|s_{j}\right\|} \tag{8}\]The function shrinks short vectors to almost zero and long vectors to a length below 1. In reference [4], there are 10 classes represented by 10 capsules in the digit capsule layer. However, the distribution of coefficients is concentrated in the interval from 0.09 to 1.09, as demonstrated in Figure 2, which makes the \"sum\" s\({}_{\text{j}}\) of prediction vectors poorly distinguishing. In other words, the probabilities of the features sent to digit capsules are nearly equal. As a result, the lengths of each vector \(v_{j}\) in the final digit capsule layer are close to each other, which may produce a wrong class.  
Therefore, we calculated the coefficients by using the Sigmoid function instead of the Softmax function. Our Sigmoid routing procedure is almost the same as the Softmax routing procedure in reference [4], but we replace the Softmax function with the Sigmoid function as shown in Equation (9). \(c_{ij}\) no longer stands for the allocation probabilities toward the final capsules, but the correlation strength between the primary capsules and the final capsules. As can be seen from Figure 3, the distribution interval of the logarithm value of \(c_{\text{ij}}\) to base 10, as indicated in Equation (9), is much better distributed. The difference between the minimum and maximum coefficients is even bigger. The important prediction vectors are multiplied with larger coupling coefficients to make the significant features more decisive, while unrelated features get smaller ones. Besides this, it increases the difference between the lengths of the vectors in the final capsule layer. The correct digit capsule then exceeds all the other digit capsules in length. We adopted the Sigmoid routing in our model and the performance was better than that of the model using Softmax routing, which is shown as follows (Algorithm 2).  
\[c_{\text{ij}}=\ \frac{1}{1+\exp\bigl{(}b_{\text{ij}}\bigr{)}} \tag{9}\]  
Figure 2: Distribution of routing coefficients for the same training image from the CIFAR-10 dataset for network training using Softmax. The horizontal axis represents the logarithm of the number of coefficients to base 10; the vertical axis represents the interval of coefficients.