# Real-Time Multi-Label Upper Gastrointestinal Anatomy Recognition from Gastroscope Videos
## 2 Materials and Methods
### LSTM Structure  
After the above structure is trained to process video frames based on static images, the final prediction results may fluctuate due to the presence of some poor quality frames in the video. Due to the continuity of video data, temporal information provides background information for each frame identification. At the same time, individual frames may have similar appearance under the same endoscopic anatomy and scene, or they may be slightly blurred, making it difficult to distinguish them purely by their visual appearance. In contrast, the phase identification of the current frame would be more accurate if we could take into account the dependence of the current frame on the adjacent past frames. Therefore, time series information is introduced in this study to improve the stability of the model.  
Temporal information modeling. In our GL-Net, we input the image features extracted from the ResNet backbone network into the LSTM network, and use the memory units of the LSTM network to correlate current frame and past frame information for improved identification using temporal dependence.  
Figure 5 demonstrates the fundamental LSTM [48] units used in GL-Net. Each LSTM cell is equipped with three gates: \(i_{t}\) denotes input gate, \(f_{t}\) denotes forget gate and \(o_{t}\) denotes output gate. Three units are used to regulate the interaction between memory cells \(c_{t}\). At timestep \(t\), given input \(x_{t}\), hidden state before \(h_{t-1}\), and memory cell before \(c_{t-1}\), LSTM structural units are learned and updated in the following manner:  
\[i_{t  
In order to fully exploit both the label association and temporal information, we propose a new recursive convolutional network, GL-Net, as shown in Figure 6. GL-Net integrates ResNet-GCN for visual descriptor extraction with label-dependent association, and the LSTM network for temporal dynamic modeling. It outperforms existing methods for independent learning of visual and temporal features. We train GL-Net end-to-end, where the parameters of the ResNet structure and the LSTM structure are co-optimized to achieve better anatomical location recognition.  
Figure 5: The structure of LSTM storage unit [49]. The arrows indicate the path of data forward propagation.  
Figure 6: Overall framework of GL-Net model.  
In detail, to identify the frames at time \(t\), we extract the video clip containing a set of current frames. The sequence of frames in the video clip is represented by \(x=\{x_{t^{\prime}},\dots,x_{t-1},x_{t}\}\). We use \(f_{j}\) to denote the representative image features of each single frame \(x_{j}\). The image features \(f=\{f_{t^{\prime}},\dots,f_{t-1},f_{t}\}\) of the video clips are sequentially put into an LSTM network, which is denoted by \(U_{\theta}\) with parameters \(\theta\). With the input \(x_{t}\) and the previous hidden state \(h_{t-1}\), the LSTM calculates the output \(o_{t}\) and the updated current hidden state \(h_{t}\) as \(o_{t}=h_{t}=U_{\theta}(x_{t},h_{t-1})\). Finally, the prediction of frame \(x_{t}\) is generated by feeding the output \(o_{t}\) into the softmax:  
\[\hat{p}_{t}^{i}=sigmoid(W_{z}o_{t}+b_{z}), \tag{10}\]  
where \(W_{z}\) and \(b_{z}\) respectively denote the weight and bias term, \(\hat{P}_{t}\
otin\mathbb{R}^{C}\) is the predicted vector and \(C\) denotes the number of classes.  
Let \(\hat{P}_{t}^{i}\) be the \(i\)-th element of \(\hat{P}_{t}\), which denotes the prediction probability of frame \(x_{t}\) and it belongs to the class \(i\), \(l_{t}\) denotes the ground truth of frame \(x_{t}\), the negative log-likelihood of the frame of time \(t\) can be caculated as:  
\[\ell(x_{t})=-\log\hat{p}_{t}^{i=l_{t}}(U_{\theta}(x)). \tag{11}\]