# DE-CapsNet: A Diverse Enhanced Capsule Network with Disperse Dynamic Routing
## 3 Diverse Enhanced Capsule Network
### DE-CapsNet Architecture  
Figure 4 demonstrates the training process of the CIFAR-10 dataset using our model. An enhanced capsule residual convolutional subnetwork was used to build the capsules based on the residual basic block [2] for complex datasets. These layers can copy other layers from the learned shallower model to reduce gradient loss [2] based on the connections from the initial layers to the later layers.  
Figure 3: Distribution of routing coefficients for the same training image from the CIFAR-10 dataset for network training using Sigmoid. The horizontal axis represents the logarithm of the number of coefficients to base 10; the vertical axis represents the interval of coefficients.  
Going deeper improves the learning for capturing diversified features. The architecture contains two levels of primary capsule layers. Each primary capsule represents a small area of input, and different levels of capsules carry different scales of the image. The first-level primary capsules were established with two residual convolutional subnetworks and two quasi-primary enhanced capsule convolutional layers. The second-level primary capsules were established with one residual convolutional subnetwork and one quasi-primary enhanced-capsule convolutional layer. One example of the residual convolutional subnetwork and the quasi-primary enhanced capsules is shown in Figure 5. We used a \(1\times 1\) convolutional layer before every subnetwork to implement cross-channel information combination and to add nonlinear features. The feature maps of subnetworks act as grouped capsules called quasi-primary capsules. Compared to the model in reference [4], our model adopts one convolutional layer with \(5\times 5\) kernels using a stride of \(2\) and \(3\times 3\) kernels using a stride of \(1\) to build primary capsules, rather than \(9\times 9\) kernels. This significantly reduces the number of parameters that need to be calculated. The first quasi-primary capsule layer consists of \(32\) groups of capsules and the second quasi-primary capsule layer consists of \(8\) groups of capsules. The features from the quasi-primary capsules are reshaped by the squash function to form the primary capsule layer. Carrying various scales of images at different levels of primary capsules provides macro and local entities of features as shown in Figures 6 and 7. The output from the two-level primary capsule layers is passed into the squash activation layer by disperse dynamic routing to generate two digit capsule (DigitCaps) layers. Besides this, one more DigitCaps output layer is created by routing the concatenation of the two-level primary capsule layers to learn features from the various scales of images. The DE-CapsNet performs better than simple stacking of DigitCaps and joint back-propagation. Finally, the DigitCaps layers are concatenated and squashed to create a \(32\)-dimension (\(32\)D) capsule for each of the \(10\) classes.  
Margin loss [4] was adopted as the loss function in the proposed architecture to enhance the probability of the true class and restrain the others. The loss function is defined as shown in Equation (10) [4]:  
\[L_{k} =T_{k}\;\text{max}\Big{(}0,m^{+}-\|v_{k}\|\Big{)}^{2}+\lambda(1-T_{k} \;)\;\text{max}(0,\|v_{k}\|-m^{-})^{2}. \tag{10}\]  
In Equation (10), \(T_{k}\) = 1 if class k is true and \(T_{k}\) = 0 otherwise. \(\lambda\), \(m^{+}\), and \(m^{-}\) are hyper parameters. We set \(m^{+}\) = 0.9, \(m^{-}\) = 0.1, and \(\lambda\) = 0.5 before training. \(\lambda\) is used to control the effect of gradient backpropagation at the initial learning [4]. The losses of the two layers are backpropagated separately.  
Figure 4: Pipeline of two-level DE-CapsNet for CIFAR-10.