# Real-Time Multi-Label Upper Gastrointestinal Anatomy Recognition from Gastroscope Videos
## 1 Introduction
[PERSON] et al. [25] divided the anatomical structure of endoscopy to 11 sites, and achieved 91.11% accuracy by using DenseNet121 [26]. The model was used to assist physicians in avoiding examination blind spots during examinations and to achieve comprehensive coverage of endoscopy. Despite the good results of the above-mentioned studies on quality control of gastrointestinal endoscopy, some problems and challenges remain. First, all the above-mentioned work on anatomical location identification models is based on single-label multi-class classification, which deviates from the reality of actual clinical examinations. Multiple related anatomical sites are usually present simultaneously in the same image. When the ratio of multiple anatomical locations in the field of view is equal, a single label is not sufficient to accurately describe the currently examined location, which, in turn, increases the bias of model feature learning. Multi-label classification learning is more accurate in this application compared to multi-class image single-label classification [27], but it is challenging to further exploit this a priori relationship to improve the model accuracy due to the spatial correlation between anatomical locations, which leads to dependencies between labels. Second, all of the above work is based on anatomical location recognition models, which are trained based on static image data rather than real-time video data. It is not sufficient to identify anatomical locations under videos based on static image datasets alone. While consecutive video frames are highly similar, the dynamics of the scene cannot be expressed in static images, and this dynamically changing data is important for the application of the model in real video scenes. Although the dynamic ones can produce severe scene blurring [28] and thus affect judgment due to camera motion and gases generated during surgery, etc., the impact of such blurred data can be mitigated by some means. In conclusion, spatial and temporal factors are strong priors for the anatomical relationships within the endoscope and between consecutive frames, and are key to further improving the performance of the recognition model. In this paper, we present a novel combined structure of deep learning models to process EGD videos to accurately identify anatomical structures of the gastrointestinal tract on real-time white-light upper gastrointestinal tract. The task consists of classifying each single frame of an EGD image sequence into a number of anatomical structures in 25 sites. Our model is built on a combination of a graph convolutional network (GCN) and a long short-term memory (LSTM) network, where the GCN is used to capture label dependencies, and the LSTM is used to extract inter-frame temporal dependencies. Specifically, we train them jointly in an end-to-end way to relate coded label interdependencies and extract high-level features of visual and temporal information of consecutive video frames. The combined features learned by our method can correlate different anatomical structures under endoscopy and are sensitive to camera movements in the video, allowing accurate identification of all anatomical structures contained in each frame of a continuous video, especially the transition frames between different anatomical locations. The main contributions of this paper are summarized as follows: (1) Unlike previous single-label multi-class studies, we define anatomical recognition as a multi-label classification task. This setting is more in line with clinical needs and real-time video-based examination. (2) GCN-based multi-label classification algorithm. In this paper, graph structure is introduced to learn domain prior knowledge, i.e., topological interdependencies between anatomical structure labels. A ResNet-GCN model is then constructed to implement multi-label classification. (3) Fusion of ResNet-GCN and LSTM modules. Due to the complexity of EGD endoscopy scenes, it is very difficult to classify the anatomical structures of each frame accurately.