# Attention Neural Network for Water Image Classification under IoT Environment
## 3 Methods
### Design of Channel-wise Attention Gate  
We firstly discuss theoretical fundament of soft attention mechanism. Afterwards, we build channel-wise attention gate based on soft attention mechanism, which essentially acts as a lightweight gating mechanism.  
The proposed attention gate can describe channel-wise relationships among feature maps, which is utilized to construct attention neural network at local and global levels. Compared with traditional manually assigned weights, the proposed attention model can automatically select informative channels to perform classification tasks based on inherent characteristics of collected data, which is more flexible for different application scenarios.  
**Theoretical Fundamental of Soft Attention Mechanism.** Constructing soft attention model consists of two steps, namely computing weights according to similarity of input and trained signal, and re-scaling the feature map with weights.  
To compute similarity between input and trained signal, researchers usually adopt MLP for implicit calculation, which can be defined as  
\[sim(Q,W_{i})=MLP(Q,W_{i}) \tag{3}\]  
After computing similarity, Softmax function is often utilized to complete multi-tasks, i.e., normalization and assigning large values on important parts:  
\[\alpha_{i}=softmax(sim(Q,W_{i}))=\frac{e^{sim(Q,W_{i})}}{\sum_{j=1}^{L}e^{sim (Q,W_{i})}}; \tag{4}\]  
where \(L\) is number of trained signals.  
After calculating new weight for input signal, we can sum weighted parts to form new signal \(Atten\):  
\[Atten=\sum_{i=1}^{L}\alpha_{i}\cdot v_{i} \tag{5}\]  
where \(v_{i}\) refers to part of input signal.  
**Construction of Channel-wise Attention Gate.** We build the structure of the proposed attention gate shown in Figure 4, which is further utilized for description of local and global attention information. We firstly represent a feature map as \(U=[u_{i};i=1,\dots,c]\), where \(u_{i}\) refers to \(i\)th channel and \(c\) is the size of channels. In fact, each filter operates on pixels to compute features within a local receptive field, which fails to exploit contextual information outside its receptive field [31]. We thus collect channel-wise information to form a characteristics descriptor \(M\) by mean pooling  
\[\left\{\begin{array}{l}M=[m_{i};i=1,\dots,c]\\ m_{i}=\frac{1}{W\times H}\sum_{j=1}^{W}\sum_{k=1}^{H}u_{i}(j,k))\end{array}\right. \tag{6}\]  
where \(M\) consists of \(m_{i}\), \(W\) is the width of the \(i\)th channel, and \(H\) refers to its height.  
Afterwards, we construct weighting scheme to assign informativeness values to different channels. Considering that the relationship between channels is a highly nonlinear function of feature map, we construct a weighting scheme with two fully-connected layers to calculate weight vector \(G\) as  
\[\left\{\begin{array}{l}G=[g_{i};i=1,\ldots,c]\\ g_{i}=Nor(sig(W_{2}\eta(W_{1}m_{i}+b_{1})+b_{2})),\end{array}\right. \tag{7}\]  
where function \(sig()\) is sigmoid; \(\eta()\) is ReLU; \(W_{1}\), \(W_{2}\), \(b_{1}\), and \(b_{2}\) are parameters to be trained; and function \(Nor()\) re-scales the output to be the form of a one-hot activation.  
As shown in Figure 4, \(G\) assigns weights on different channels of input feature map \(U\) to emphasize informative feature channel, which could be represented as  
\[\tilde{u_{i}}=g_{i}\otimes u_{i} \tag{8}\]  
where \(\otimes\) refers to element-wise multiplication.