# DE-CapsNet: A Diverse Enhanced Capsule Network with Disperse Dynamic Routing
## 2 Related Work  
Increasing the depth of layers in networks promotes the performance of deep networks and stimulates the innovation of architectures. Highway Networks [10] is a deep feedforward network that provides an effective way to train networks with more than 100 layers by using bypassing paths [10]. ResNets further explores the effect of pure identity mapping by using it as the bypassing path, with deep layers which can achieve excellent performance in many challenging benchmark datasets [2]. Increasing the width of a network can help to train deeper networks. Feature maps operated by kernels with different sizes are concentrated using the \"inception module\" in GoogLeNet [1]. [PERSON] et al. [3]proposed a novel architecture called DenseNets that provides dense connections between all layers. It allows better gradient flow across deeper networks. The current CapsNet [4] consists of one convolution layer, one primary capsule layer, and one digit capsule layer [4]. The input image is operated by a convolution layer with 256 9 \(\times\) 9 kernels using a stride of 1 to extract features and then activated by the Rectified Linear Unit (ReLU) function. The output of the ReLU function is a feature map tensor. The primary capsule (PrimaryCaps) layer adopts a second convolutional layer with 9 \(\times\) 9 kernels using a stride of 2 to deal with the feature map tensor. The output of the PrimaryCaps is also activated by the ReLU function. Every group of 8 scalars in the feature map tensor constitutes the primary capsule \(i\). Capsules use feature vectors to represent the properties of entities which can capture position, size, texture, and other information. \(u_{i}\) is the output of primary capsule \(i\). \(\hat{u}_{i}\) is the prediction vector which is the input of final digit capsule \(j\). \(W_{ij}\) is the weight matrix. \(\hat{u}_{i}\) is calculated by Equation (1) [4]. \[\hat{u}_{ji}=W_{ij}u_{i} \tag{1}\]  
Routing-by-agreement will send the output of the primary capsules to the final capsules by increasing or decreasing the connection strength between the primary capsules and the digit capsules (DigitCaps) instead of pooling operation and keeping the spatial relations between object parts. It can be seen as a prediction which sends the output of the primary capsule \(i\) to the final digit capsule \(j\). The coupling coefficient between the two capsules will increase when the output matches. \(b_{ij}\) is a logit of the Softmax function, which defines the coupling coefficient \(c_{ij}\) between capsule \(i\) in the layer above and capsule j in the layer below, as given by Equation (2) [4]. \[c_{ij}=\frac{\exp\left(\text{b}_{\text{ij}}\right)}{\sum_{\mathbf{k}}\exp \left(\text{b}_{\text{ik}}\right)},\ s_{j}=\ \sum_{i}c_{ij}\hat{u}_{ji} \tag{2}\]  
The length of these vectors \(s_{j}\) is the input of the digit capsule layer and is restricted to 1 by the squash function [4] to get the output \(v_{j}\). The inner product of \(v_{j}\) and \(\hat{u}_{ji}\) updates the log probabilities \(b_{ij}\). The more similar the two vectors, the longer the vector \(v_{j}\) will be. After several iterations, the one with the largest vector length in the final capsule layer corresponds to the true class. Based on this, [PERSON] et al. [11] proposed matrix capsules using logistic units to represent the presence of entities and a pose matrix to represent the poses of the entities with the Expectation-Maximization (EM) routing algorithm. HitNet [12] uses centripetal function loss to train the Hit-or-Miss layers of capsules. The capsule corresponding to the true class makes a hit in its target space, while the others make misses. [PERSON] et al. [13] showed that the scale-invariant Max-Min function can promote the performance of CapsNet [4]. An optimization of the routing strategy and a new routing approach proposed in reference [14] outperformed the dynamic routing method in reference [4].