# Retrieval of Chemical Oxygen Demand through Modified Capsule Network Based on Hyperspectral Data
## 2 Materials and Methods
### The Architecture of Capsule Networks  
Figure 4 shows the architecture of the capsule network, designed for COD estimation. The model needed to handle 1-D hyperspectral data with a vector length of 984. There are three layers in capsule networks. Each layer was modified to suit our objective. In the first layer, the original 2-D convolution kernel was replaced by a 1-D convolution kernel to fit the 1-D input data. The model was extended by one channel dimension so that the input data met the requirement of the CONV1D function in Python. Moreover, it was found that a small kernel produced better accuracy than a larger one. Hence, a kernel with a size of one was used in our model instead of a kernel of \(9\times 9\), and only 3 channels were used in our first layer to replace the 256 channels in the original model. It was demonstrated that these modifications reduced the parameters without a loss of accuracy. The second layer is called the primary capsule layer. In this layer, the capsule dimension of the input was extended. One fast and convenient way to manipulate the method was to apply the convolution kernels to the input and then reshape the output to the designated number of dimensions. For example, in our model, 24 1-D convolution kernels were applied to the input and then reshaped the output to \(3\times 8\), reflecting 3 channels and an 8-unit length capsule vector. A small kernel size was used with fewer vectors in the capsule dimension compared with the original network. The third layer is called the digit capsule layer, which is the key part of the capsule network. For all but the first part of the digit capsule layer, the total input to a capsule \(s_{j}\) a weighted sum over-all \(\hat{u}_{j\bar{j}}\) and \(\hat{u}_{j\bar{j}}\) is produced by multiplying the output \(u_{i}\) of a capsule by a weight matrix \(W_{ij}\):  
\[s_{j}=\sum_{i}c_{ij}\hat{u}_{j\bar{j}},\quad\hat{u}_{j\bar{j}}=\mathrm{W}_{ij} u_{i} \tag{1}\] \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Group** & **Min** & **Max** & **Mean** & **Median** & **SD** \\ \hline Training set & 10 & 27 & 17.26 & 17 & 4.49 \\ Testing set & 13 & 26 & 18.68 & 19 & 3.85 \\ Entire dataset & 10 & 27 & 17.69 & 17 & 4.36 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Statistical information for the chemical oxygen demand (COD) index for the study area. Figure 4: Schematic showing the capsule network architecture and the modifications used in this study. where \(c_{ij}\) are the coupling coefficients determined by the iterative dynamic routing process. The formula for these \(c_{ij}\) is given by:  
\[c_{ij}=\frac{\exp\!\left(b_{ij}\right)}{\sum\limits_{k}\exp\!\left(b_{ik}\right)} \tag{2}\]  
where \(b_{ij}\) is initially set to zero and upgraded by \(b_{ij}=b_{ij}+\mathbf{u}\mathbf{j}\|i\cdot\mathbf{v}_{j}\). In this case, \(\mathbf{v}_{j}\) is determined in ach iteration in a dynamic routing process. The length of the output vector represents the probability that the entity is contained and exists in the input data. To represent such properties, a non-linear \"squashing\" activation function, shown in Equation (3) is introduced, which ensures that the short vectors are shrunk to near zero and long vectors are shrunk close to one, leaving their orientation unchanged. \[\mathbf{v}_{j}=\frac{\|s_{j}\|^{2}}{0.5+\|\mathbf{s}_{j}\|^{2}}\frac{\mathbf{ s}_{j}}{\|\mathbf{s}_{j}\|} \tag{3}\]  
The total routing algorithm is given in Table 3.  
The most important aspect of this routing procedure is that it uses dynamic routing to update the weights \(c_{i}\) instead of using back-propagation. The original model had to be modified because the range of the output in the digit capsule layer was not compatible with the label. The original model is supposed to output a probability that is used for classification, which has a range from zero to one. However, this study is dealing with a regression problem. In this case, the true label does not have a constraint on its range. Here, two options were outlined to solve this problem.