## 3 Attentive Octave Convolutional Capsule Network
### Capsule Layer  
The two commonly used capsule networks are CapsNet with dynamic routing and matrix CapsNet with EM routing. In CapsNet with dynamic routing, the capsule vector is constructed by stacking the neurons with scalar values. In CapsNet with EM routing, the capsule contains a pose matrix and an activation.  
The output feature maps of the AOC-Layer are first reshaped into a series of capsules \(s_{j},j=1,2,\cdots\). For one capsule \(s_{j}\), its input is the weighted sum of all the prediction vectors \(\hat{u}_{j|i}\) generated by the previous layers. It can be defined as Formulas (5) and (6).  
\[s_{j}=\sum_{i}c_{ij}\hat{u}_{j|i} \tag{5}\]  
\[\hat{u}_{j|i}=W_{ij}u_{i} \tag{6}\]  
where \(c_{ij}\) is the routing coefficient, \(W_{ij}\) is the matrix used for voting, and \(u_{i}\) is the output vector of the previous capsule layer. The routing coefficients should be computed in dynamic routing [12] or EM routing [13]. In dynamic routing, \(u_{i}\) is a vector.  
The process of dynamic routing is shown as follows:  
1. The prior probability \(b_{ij}\) between capsule \(j\) and capsule \(i\) in the previous layer is initialized to be 0;
2. The routing coefficients \(c_{ij}\) can be computed through the softmax function \(c_{ij}=\frac{exp(b_{ij}))}{\sum_{k}exp(b_{ik})}\);
3. The input to capsule \(j\) is computed by Formula (5) and then it is squeezed by \(v_{j}=\frac{\left\|s_{j}\right\|^{2}}{1+\left\|s_{j}\right\|^{2}}\frac{s_{j}}{ \left\|s_{j}\right\|}\);
4. The \(b_{ij}\) is updated by \(b_{ij}\gets b_{ij}+\hat{u}_{j|i}\cdot v_{j}\);
5. Repeat steps 2 to 4 \(r\) times. The value of \(r\) is set empirically, usually from 1 to 3.  
Different from capsule networks with dynamic routing, capsules in the matrix capsule with EM routing consist of a pose matrix and an activation. A pose matrix defines the translation and the rotation of the objects. The aim of the EM algorithm is to cluster datapoints into different Gaussian distributions. Suppose the pose matrix is a 4 \(\times\) 4 matrix, i.e., 16 components. Let \(v_{ij}\) be the vote from capsule \(i\) to capsule \(j\), and \(v_{ij}^{h}\) be its h-th component. The probability density function of a Gaussian is defined as Formula (7):  
\[P(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-y)^{2}}{2\sigma^{2}}} \tag{7}\]  
It can be applied to compute the probability of \(v_{ij}^{h}\) belonging to the capsule \(j\)'s Gaussian model:  
\[P_{i|j}^{h}=\frac{1}{\sqrt{2\pi\left(\sigma_{j}^{h}\right)^{2}}}e^{-\frac{ \left(\frac{v_{ij}^{h}-h}{v_{ij}^{h}}\right)^{2}}{2\left(\sigma_{j}^{h}\right) ^{2}}} \tag{8}\]  
Let \(cost_{ij}^{h}=-ln\left(P_{i|j}^{h}\right)\) be the cost to activate the h-th component of capsule \(j\) by the h-th component of capsule \(i\), where  
\[ln\left(P_{ij}^{h}\right)=-ln(\sigma_{j}^{h}))-\frac{ln(2\pi)}{2}-\frac{ \left(v_{ij}^{h}-u_{j}^{h}\right)^{2}}{2\left(\sigma_{j}^{h}\right)^{2}} \tag{9}\]  
Whether the capsule \(j\) is activated is determined by the following equation:  
\[a_{j}=sigmoid(\lambda(b_{j}-\sum_{h}cost_{j}^{h})) \tag{10}\]  
where  
\[cost_{j}^{h}=\sum_{i}r_{ij}cost_{ij}^{h} \tag{11}\]  
\(r_{ij}\) is the assignment probability of each datapoint to a capsule. \(r_{ij}\), \(\mu\), \(\sigma\), and \(a_{j}\) are computed iteratively using the EM algorithm.