<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Automated Summarization of Gastrointestinal Endoscopy Video

B. Sushma<fnref n="1" />(✉) and P. Aparna<fnref n="2" />

sushma.b@cmrit.ac.in

p.aparnadinesh@nitk.edu.in

**Abstract.** Gastrointestinal (GI) endoscopy enables many minimally invasive procedures for diagnosing diseases such as esophagitis, ulcer, polyps and cancers. Guided by the endoscope's video sequence, a physician can diagnose the diseases and administer the treatment. Unfortunately, due to the huge amount of data generated, physicians are currently discarding procedural video and rely on a small number of carefully chosen images to record a procedure. In addition, when a patient seeks a second opinion, the assessment of lesions in a huge video stream necessitates a thorough examination, which is a time-consuming process that demands much attention. To reduce the length of the video stream, an automated method to generate the summary of endoscopy video recordings consisting only of abnormal frames by using deep convolutional neural networks trained to classify normal, abnormal and uninformative frames is proposed. Results show that our method can efficiently detect abnormal frames and is robust to the variations in the frames. The proposed CNN architecture outperforms the other classification models with an accuracy of 0.9698 with less number of parameters.

**Keywords:** Convolutional Neural Networks · Deep Neural Network · Endoscopy · Video Summarization

## 1 Introduction

In recent days, GI tract endoscopy is considered as an active research area in detecting many GI tract abnormalities. A gastroenterologist can explore the GI tract using the GI endoscopic video sequence as a guide to diagnose GI disorders or collect tissue sample for biopsy. The GI tract is illuminated by the endoscope's light source, and video frames of the illuminated GI tract surfaces are collected by an embedded camera. The video stream provides detailed structural and morphological information about the surfaces of the GI tract, as well as a record

<footnote n="1">¹ Department of Electronics and Communication Engineering, CMR Institute of Technology, Bengaluru 560037, India</footnote>

<footnote n="2">² Department of Electronics and Communication Engineering, National Institute of Technology Karnataka, Surathkal, Mangalore 575025, Karnataka, India</footnote>
of the anatomical regions examined during endoscopy. The video is captured for 20 to 40 min at 30 frames per second with a frame resolution of 1920 × 1080. Due to the huge amount of video data generated, the current practice discards the procedural video and only few frames are selected for recording to avoid the high memory consumption. Another drawback of the current procedure is that it is difficult to diagnose the condition with only a few frames when the patient seeks a second opinion. Suppose the complete video is taken into account for the evaluation, the physician will have to devote a significant amount of time reviewing the entire video, which is also a tedious process.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Endoscopy video summary generation method consisting of abnormal frames</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. Proposed CNN architecture for GI endoscopy frame classification</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Fig. 3. Main building blocks of proposed CNN architecture</figcaption></figure>

In literature, many works have been proposed to detect the frames with one or two kinds of GI anomalies such as polyps, ulcers, Crohn's disease and gastrointestinal cancers [4, 9, 10, 15]. A deep learning method to detect and segment the GI anomalies is in [1]. A new tool to detect the celiac disease using principal component analysis is implemented in [12]. Segmentation of anomaly regions in endoscopy frames is proposed in [11], which can detect one anomaly in a frame. This model cannot detect multiple anomalies in a single frame. The gastroenterologist must still carefully review the majority of the frames with other GI anomalies. In the work proposed in [3, 14], to reduce the number of redundant frames for the review, video is segmented into shots and from each shot the most representative frames are extracted. This method extracts frames consisting normal, abnormal and uninformative frames. Therefore a method to identify and remove the normal and uninformative frames is required. To address all of the aforementioned issues, a method that generates a summary comprising solely of abnormal frames is necessary.

This work proposes an automated method for generating a summary of GI endoscopy procedural recordings comprising solely on anomalous frames to overcome the problem of storage and review time. The normal frames with out anomalies and frames captured during the movement of the endoscopic instrument that are uninformative are eliminated. This method reduces the number of frames required for storage as well as the number of frames required for review. The proposed method extracts a frame from the procedural video recording and determines whether it is normal, abnormal, or uninformative using a deep convolutional neural network (deep CNN). If an abnormal frame is detected, it is saved in a frame buffer; otherwise ignored as represented in Fig. 1. A deep CNN is trained to detect whether the input frame is abnormal or not. The workload
of gastroenterologists during second examinations could be reduced using a proposed approach. It could also assist inexperienced gastroenterologists to make better decisions. Furthermore, the suggested method's automatic summary generation can reduce the physician's workload, allowing them to focus on significant frames and improve diagnostic performance.

In GI endoscopy, sometimes a limited number of features of an image are sufficient to provide the diagnosis. The usage of complex CNN may require more resources for implementation and training. Furthermore, a complicated CNN takes longer to predict and may not give real-time performance when anomalous frames must be saved during an endoscopic procedure. In this work, CNN with fewer layers is proposed which gives the similar performance of deep standard CNN classification architectures requiring less number of resources and prediction time.

The remaining part of the paper consists three sections. The methodology in Sect. 2, which describes the proposed CNN architecture, dataset, training and implementation details. Section 3 presents the results obtained by experimental evaluation of the proposed and comparison with existing standard CNN architectures. The last section consists the discussion and conclusive summary of the work.

# 2 Methodology

## 2.1 CNN Model

The proposed CNN architecture aims at the detection of abnormal frames from GI Endoscopy video. It is implemented using five main blocks, as displayed in Fig. 2. The model accepts an input image of size 256 x 256 consisting three channels. The features of the input images are extracted by stack of convolution layers. The first CRBC block shown in Fig. 3a consists of a 3 x 3 convolution layer succeeded with batch normalization (BN) layer, ReLU activation operation and a 3 x 3 convolution layer. The next main component is BRCBRC block shown in Fig. 3b. BRCBRC consists two convolution layers. The extracted features are given to classification-head consisting fully-connected layers and a softmax activation layer. Each convolution layer is described in Table 1 in regard to the number of filters, the size of the filters, and the output dimensions.

## 2.2 Description of the Dataset

Dataset for classifying the input frame into normal, abnormal and uninformative is created using the HyperKvasir dataset consisting images and videos collected during real GI endoscopy and colonoscopy procedures. The dataset contains 374 videos of normal and various pathological findings which are labelled by GI-endoscopists [2]. These videos are captured at 30 fps. From each video sequence, frames are extracted at a frame rate of 1 fps. The extracted frames are cropped to remove the non-image part consisting the text. Finally, images are resized to
<table><caption>**Table 1.** Parameter details of the proposed CNN architecture</caption><thead><tr><th>Block</th><th>Layer-type</th><th>No. of Filters</th><th>Filter Size</th><th>Output Size</th></tr></thead><tbody><tr><td>–</td><td>Input</td><td>–</td><td>–</td><td>256 × 256 × 4</td></tr><tr><td rowspan="2">CBRC</td><td>Conv-1</td><td>16</td><td>3 × 3</td><td>256 × 256 × 16</td></tr><tr><td>Conv-2</td><td>16</td><td>3 × 3</td><td>256 × 256 × 16</td></tr><tr><td rowspan="3">BRCBRC-1</td><td>Maxpooling</td><td>–</td><td>–</td><td>128 × 128 × 16</td></tr><tr><td>Conv-3</td><td>32</td><td>3 × 3</td><td>128 × 128 × 32</td></tr><tr><td>Conv-4</td><td>32</td><td>3 × 3</td><td>128 × 128 × 32</td></tr><tr><td rowspan="3">BRCBRC-2</td><td>Maxpooling</td><td>–</td><td>–</td><td>64 × 64 × 32</td></tr><tr><td>Conv-5</td><td>64</td><td>3 × 3</td><td>64 × 64 × 64</td></tr><tr><td>Conv-6</td><td>64</td><td>3 × 3</td><td>64 × 64 × 64</td></tr><tr><td rowspan="3">BRCBRC-3</td><td>Maxpooling</td><td>–</td><td>–</td><td>32 × 32 × 64</td></tr><tr><td>Conv-7</td><td>128</td><td>3 × 3</td><td>32 × 32 × 128</td></tr><tr><td>Conv-8</td><td>128</td><td>3 × 3</td><td>32 × 32 × 128</td></tr><tr><td rowspan="3">BRCBRC-4</td><td>Maxpooling</td><td>–</td><td>–</td><td>16 × 16 × 128</td></tr><tr><td>Conv-9</td><td>256</td><td>3 × 3</td><td>16 × 16 × 256</td></tr><tr><td>Conv-10</td><td>256</td><td>3 × 3</td><td>16 × 16 × 256</td></tr><tr><td></td><td>FC Layer-1</td><td>–</td><td>–</td><td>1 × 65536</td></tr><tr><td></td><td>FC Layer-1</td><td>–</td><td>–</td><td>1 × 256</td></tr><tr><td></td><td>FC layer-3</td><td>–</td><td>–</td><td>1 × 3</td></tr></tbody></table>

256 × 256 pixel resolution. Abnormal and normal class images are the frames extracted from videos of abnormal and normal mucosa findings. Uninformative frames results due to the movement of endoscopy instrument are selected from both normal and abnormal videos. The example image belongs to normal, abnormal and uninformative frames are shown in Fig. 4, Fig. 5 and Fig. 6 respectively. The dataset details used for training the CNN models is given in Table 2 and Table 3.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Typical frames with normal findings</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Fig. 5. Frames with abnormalities</figcaption></figure>

## 2.3 Model Training Details

The model developed is trained by using Keras, a deep learning API using Tensortflow back-end on an NVIDIA-TESLA-P100 GPU with 16 GB RAM. The initial learning-rate ($lr$) is set to 0.0001 with $decay = \frac{lr}{epochs}$. Network parameters are initialized using Xavier initialization method. Optimum model parameters are found using adam optimizer during back propagation. Network is trained for 50 epochs with the batch size of 8. The model is trained with sparse categorical cross entropy loss function. The learning rate curves are shown in Fig. 7.

<figure><img src="image_7.png" /><figcaption>Fig. 6. Uninformative Frames</figcaption></figure>

<table><caption>Table 2. Dataset partition</caption><thead><tr><td>Class</td><td>Train</td><td>Validation</td><td>Test</td></tr></thead><tbody><tr><td>Abnormal</td><td>3600</td><td>720</td><td>480</td></tr><tr><td>Normal</td><td>3606</td><td>730</td><td>489</td></tr><tr><td>Uninformative</td><td>3600</td><td>720</td><td>480</td></tr></tbody></table>
<table><caption>**Table 3.** Frames of various abnormalities in abnormal class</caption><thead><tr><td>Abnormality</td><td>Train</td><td>Validation</td><td>Test</td></tr></thead><tbody><tr><td>Esophagal-Cancer</td><td>570</td><td>115</td><td>75</td></tr><tr><td>Stomach Cancer</td><td>555</td><td>115</td><td>75</td></tr><tr><td>Esophagitis</td><td>765</td><td>150</td><td>80</td></tr><tr><td>Ulcer</td><td>420</td><td>95</td><td>95</td></tr><tr><td>Colon Polyps</td><td>675</td><td>95</td><td>95</td></tr><tr><td>Colorectal-Cancer</td><td>699</td><td>150</td><td>60</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>**Fig. 7.** Train and validation learning curve with decaying learning rate</figcaption></figure>

<table><caption>**Table 4.** Proposed model performance comparison with standard CNN architectures</caption><thead><tr><td>Model</td><td>Precision</td><td>Recall</td><td>Accuracy</td><td>F1-Score</td><td>Model size</td><td>TPE</td></tr></thead><tbody><tr><td>AlexNet</td><td>0.91 ± 0.02</td><td>0.93 ± 0.04</td><td>0.9224 ± 0.0135</td><td>0.9198 ± 0.0266</td><td>60</td><td>11.55 s</td></tr><tr><td>DenseNet-121</td><td>0.77 ± 0.01</td><td>0.73 ± 0.04</td><td>0.7523 ± 0.0198</td><td>0.7494 ± 0.0162</td><td>0.4</td><td>66.44 s</td></tr><tr><td>Inception-V4</td><td>0.97 ± 0.01</td><td>0.95 ± 0.01</td><td>0.9621 ± 0.01</td><td>0.9598 ± 0.01</td><td>25</td><td>74.28 s</td></tr><tr><td>Resnet-50</td><td>0.96 ± 0.01</td><td>0.97 ± 0.01</td><td>0.9686 ± 0.01</td><td>0.9649 ± 0.01</td><td>25</td><td>32.82 s</td></tr><tr><td>SqueezeNet</td><td>0.83 ± 0.02</td><td>0.85 ± 0.02</td><td>0.8426 ± 0.015</td><td>0.8398 ± 0.021</td><td>1.24</td><td>11.45 s</td></tr><tr><td>VGG-19</td><td>0.97 ± 0.01</td><td>0.95 ± 0.03</td><td>0.9631 ± 0.02</td><td>0.9598 ± 0.015</td><td>144</td><td>69.23 s</td></tr><tr><td>Proposed</td><td>0.96 ± 0.01</td><td>0.97 ± 0.03</td><td>0.9698 ± 0.015</td><td>0.9649 ± 0.02</td><td>12</td><td>13 s</td></tr></tbody></table>

Size: Model size in million parameters, TPE: Training time per epoch in seconds

# 3 Experimental Results

This section provides the comprehensive analysis of the proposed model and the other deep learning methods proposed for classification. To quantitative evaluation of the models is done by using precision (P), recall (R), F1-Score (F1) and accuracy (ACC) computed using (1) to (4).
$$R = \frac{TP}{TP + FN} \qquad (1)$$

$$P = \frac{TP}{TP + FP} \qquad (2)$$

$$ACC = \frac{TP + TN}{TP + TN + FP + FN} \qquad (3)$$

$$F1 = 2 \frac{(P \times R)}{P + R} \qquad (4)$$

In the above equations TP, TN, FP and FN represent true positives, true negatives, false positives and false negatives respectively. The model's performance is systematically evaluated by comparing with the standard CNN architectures ResNet [6], DenseNet [7], VGG [13], SqueezeNet [8], Inception v4 [16] and AlexNet [5] trained with 50 epochs. Table 4 provides the detailed comparative results. For medical data recall should be high compared to precision which is achieved by the proposed with high accuracy. The proposed CNN architecture outperforms the other models with an accuracy of 0.9698 and better recall with 12 million parameters. The model is saved in h5 file which is the form of hierarchical data format. The size of the model after training is 90MB. The performance of the model is close to Inception-v4, Resnet-50 and VGG-19 which is achieved using reduced number of training parameters and training time. Also in the process of training these models, the over-fitting problem was observed. The performance of the denseNet and squeezenet is poor compared to the proposed model. These learning rate curves of the proposed model are represented in Fig. 7 demonstrates how a learning rate drops that is appropriate for the problem and selected model architecture can lead to final weights that are skilled and stable at convergence, which is a desirable quality in a final model at the conclusion of a training run.

# 4 Conclusion

The generation of GI endoscopy video summary consisting of only abnormal frames is presented in this work. The proposed method reduces the video data and allows the hospitals to save the endoscopy procedural videos. Also, it allows the physicians to have a quick review of the video, when the patient goes for the second opinion. In this work a deep CNN is trained to classify the input frames into normal, abnormal and uninformative frames. Abnormal frames are saved and considered for review when the patient seeks a second opinion. In the work, it is also demonstrated that complexity and depth of the deep CNN is not necessary to achieve better performance. The proposed model provided better results compared to Inception, Resnet-50 and VGG-19 which are very deep CNNs at the reduced complexity. In future a model can be integrated to detect and segment the region of anomalies in each frame.
References

1. Ali, S., et al.: Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy. Med. Image Anal. **70**, 102002 (2021)

2. Borgli, H., et al.: Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Sci. Data **7**(1), 1–14 (2020)

3. Chen, J., Zou, Y., Wang, Y.: Wireless capsule endoscopy video summarization: a learning approach based on Siamese neural network and support vector machine. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 1303–1308. IEEE (2016)

4. El-Nakeep, S., El-Nakeep, M.: Artificial intelligence for cancer detection in upper gastrointestinal endoscopy, current status, and future aspirations. Artif. Intell. Gastroenterol. **2**(5), 124–132 (2021)

5. ul Hassan, M.: Alexnet imagenet classification with deep convolutional neural networks (2018)

6. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

7. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

8. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: alexnet-level accuracy with 50x fewer parameters and 0.5 mb model size, 1(10). arXiv preprint arXiv:1602.07360 (2016)

9. Jia, X., et al.: Automatic polyp recognition in colonoscopy images using deep learning and two-stage pyramidal feature prediction. IEEE Trans. Autom. Sci. Eng. **17**(3), 1570–1584 (2020)

10. Klang, E., et al.: Deep learning algorithms for automated detection of Crohn's disease ulcers by video capsule endoscopy. Gastrointest. Endosc. **91**(3), 606–613 (2020)

11. Kundu, A.K., Fattah, S.A., Wahid, K.A.: Multiple linear discriminant models for extracting salient characteristic patterns in capsule endoscopy images for multi-disease detection. IEEE J. Transl. Eng. Health Med. **8**, 1–11 (2020)

12. Li, B.N., et al.: Celiac disease detection from videocapsule endoscopy images using strip principal component analysis. IEEE/ACM Trans. Comput. Biol. Bioinf. **18**(4), 1396–1404 (2019)

13. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

14. Sushma, B., Aparna, P.: Summarization of wireless capsule endoscopy video using deep feature matching and motion analysis. IEEE Access **9**, 13691–13703 (2020)

15. Sutton, R.T., Zaiane, O.R., Goebel, R., Baumgart, D.C.: Artificial intelligence enabled automated diagnosis and grading of ulcerative colitis endoscopy images. Sci. Rep. **12**(1), 1–10 (2022)

16. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)