Patent Description:
At present, the colon cancer ranks in the top five among high-occurrence malignant tumors in China. However, the incidence of the colon cancer in North America and Europe is also high. The colon cancer is a malignant digestive tract tumor that often occurs in colon. Generally speaking, <NUM>% of patients with advanced colon cancer die of recurrence and metastasis, and nearly <NUM>% of patients with early colon cancer may be completely cured. Therefore, it is necessary to prevent and cure the colon cancer. However, the early colon cancer cannot be predicted by clinical symptoms.

In the related art, when identifying colon polyps, a method of sliding window is usually used for detecting a polyp image (the sliding window means sliding an image block from top to bottom first, and then from left to right in an endoscopic video image frame), or the position of the polyp is manually marked. After the position of the polyp is determined, by using a computer vision extraction method, an identification result is outputted through classification.

In the foregoing solutions provided by the related art, in the sliding window method, it is calculated whether every image block includes a polyp by using a sliding window in an endoscopic video image frame. Due to a large amount of image blocks, the amount of calculation is large and the real-time performance cannot meet requirements. When the endoscope is controlled to move, an identification result of an image acquired in real time cannot be outputted in real time. The real-time performance of the manual marking method cannot meet requirements. When the endoscope is controlled to move, an identification result of an image acquired in real time cannot be outputted in real time. D1 (<CIT>) relates to methods, systems, and media for simultaneously monitoring colonoscopic video quality and detecting polyps in colonoscopy; D2 (<CIT>) relates to systems and method for processing optical images; D3 (XP036336378) relates to efficient disease detection in gastrointestinal videos; D4 (XP033429337) relates to localisation of colorectal polyps by convolutional neural network features learnt from white light and narrow band endoscopic images of multiple databases.

Embodiments of this application provide a colon polyp image processing method and apparatus, and a system, to detect a position of a polyp in real time and determine a property of the polyp, thereby improving the processing efficiency of a polyp image.

The embodiments of this application provide the following solutions:.

According to one aspect, an embodiment of this application provides a colon polyp image processing method, including:.

According to another aspect, an embodiment of this application further provides a colon polyp image processing apparatus, including:.

In the foregoing aspect, the composition modules of the colon polyp image processing apparatus may further perform steps described in the foregoing aspect and various possible implementations. For details, refer to the foregoing descriptions of the foregoing aspect and various possible implementations.

According to another aspect, an embodiment of this application further provides a medical system, including an endoscope apparatus and a colon polyp image processing apparatus, a communication connection being established between the endoscope apparatus and the colon polyp image processing apparatus;.

According to another aspect, an embodiment of this application provides a colon polyp image processing apparatus, including: a processor and a memory, the memory being configured to store an instruction, and the processor being configured to execute the instruction in the memory, to cause the colon polyp image processing apparatus to perform the method according to any one of the foregoing aspects.

According to another aspect, an embodiment of this application provides a computer-readable storage medium, the computer-readable storage medium storing an instruction, the instruction, when run on a computer, causing the computer to perform the method according to the foregoing aspects.

In an embodiment of this application, a position of a polyp in an endoscopic image is detected by using a polyp positioning model first, and a polyp image block is positioned in the endoscopic image, where the polyp image block includes: a position region of the polyp in the endoscopic image. Finally, a polyp type classification detection is performed on the polyp image block by using a polyp property identification model, and an identification result is outputted. In the embodiments of this application, because the position of the polyp is detected by using the polyp positioning model, the polyp image block may be directly positioned in the endoscopic image. The classification detection for the polyp type is also performed on the polyp image block, and does not need to be performed on the entire endoscopic image. Therefore, the real-time performance meets requirements. When the endoscope is controlled to move, an identification result of an image acquired in real time can be outputted in real time, thereby improving processing efficiency of the polyp image.

To describe the technical solutions in the embodiments of this application more clearly, the following briefly introduces the accompanying drawings required for describing the embodiments.

To make the inventive objectives, features, and advantages of the embodiments of this application clearer and more comprehensible, the following clearly describes the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application.

In the specification, the claims, and the foregoing accompanying drawings of this application, the terms "include", "have", and any other variations are meant to cover the non-exclusive inclusion, so that a process, method, system, product, or device that includes a list of units is not necessarily limited to those listed units, but may include other units not expressly listed or inherent to such a process, method, product, or device.

An embodiment of the colon polyp image processing method in this application may be specifically applied to a scene of processing a colon polyp image in an endoscopic video stream. An identification result may be output after the colon polyp image is processed according to this embodiment of this application. The identification result may be used for helping a doctor discover a polyp in real time and determine a property of the polyp during an endoscopic examination, and guiding the doctor to perform a next operation.

An embodiment of this application further provides a medical system. As shown in <FIG>, the medical system <NUM> includes: an endoscope apparatus <NUM> and a colon polyp image processing apparatus <NUM>, a communication connection being established between the endoscope apparatus <NUM> and the colon polyp image processing apparatus <NUM>.

The endoscope apparatus <NUM> is configured to generate an endoscopic video stream; and transmit the generated endoscopic video stream to the colon polyp image processing apparatus <NUM>.

The colon polyp image processing apparatus <NUM> is configured to receive the endoscopic video stream from the endoscope apparatus <NUM>; obtain a to-be-processed endoscopic image from the endoscopic video stream; detect a position of a polyp in the to-be-processed endoscopic image by using a polyp positioning model, and position a polyp image block in the endoscopic image, where the polyp image block includes: a position region of the polyp in the endoscopic image; and perform a polyp type classification detection on the polyp image block by using a polyp property identification model, and output an identification result.

The medical system provided by this embodiment of this application includes an endoscope apparatus and a colon polyp image processing apparatus. The endoscopic video stream may be transmitted between the endoscope apparatus and the colon polyp image processing apparatus in a wired or wireless manner. The endoscope apparatus may take images of the colon in a patient through the endoscope, to generate the endoscopic video stream. The colon polyp image processing apparatus detects the position of the polyp by using the polyp positioning model, so that the polyp image block may be directly positioned in the endoscopic image. The polyp type classification detection is also performed on the polyp image block, but does not need to be performed on the entire endoscopic image, so that the real-time performance meets requirements. When the endoscope is controlled to move, an identification result of an image acquired in real time may be outputted in real time, thereby improving processing efficiency of the polyp image.

Referring to <FIG>, a colon polyp image processing method provided in an embodiment of this application may include the following steps:
<NUM>. A colon polyp image processing apparatus detects a position of a polyp in a to-be-processed endoscopic image by using a polyp positioning model, and positions a polyp image block in the endoscopic image, where the polyp image block includes: a position region of the polyp in the endoscopic image.

In this embodiment of this application, the to-be-processed endoscopic image may be a single frame of endoscopic image obtained from the endoscopic video stream by the colon polyp image processing apparatus, or may be a single frame of endoscopic image received from an endoscope apparatus by the colon polyp image processing apparatus. After obtaining the single frame of endoscopic image, a position of a polyp position in the endoscopic image is detected by using a polyp positioning model trained in advance. The polyp positioning model includes network parameters that have been trained, and it may be detected, by using the network parameters of the polyp positioning model, which image regions in the endoscopic image meet polyp features, thereby determining the position region that meets the polyps features as the polyp image block circled in the endoscopic image in this embodiment of this application. <FIG> is a schematic diagram of a polyp image block circled in an endoscopic image according to an embodiment of this application. The polyp image block includes: the position region of the polyp in the endoscopic image. The completed polyp positioning model trained in advance is used in this embodiment of this application, and the polyp image block may be quickly circled through model detection, ensuring that the polyp image block may be determined in real time after the endoscopic video stream is generated, and ensuring that the polyp type classification detection may be performed in real time.

The endoscopic image is classified as a white light type picture or an NBI type picture according to different picture types. Therefore, the polyp positioning model trained in advance also needs to be divided into a white light polyp positioning model and an NBI polyp positioning model.

The white light polyp positioning model is obtained in the following manner: the colon polyp image processing apparatus performs polyp position training on the original polyp positioning model through white light type picture training data by using a neural network algorithm.

The NBI polyp positioning model is obtained in the following manner: the colon polyp image processing apparatus performs polyp position training on the original polyp positioning model through NBI type picture training data by using the neural network algorithm.

In this embodiment of this application, first, training data for the white light type and the NBI type are obtained in advance, that is, the white light type picture training data and the NBI type picture training data are obtained. A polyp positioning model is obtained in advance through training using a neural network algorithm. The polyp positioning model may be trained by using a plurality of machine learning algorithms. For example, the polyp positioning model may be a deep neural network model, a cyclic neural network model or the like. For example, the polyp positioning model may be trained by using a YOLOv2 algorithm.

In some embodiments of this application, in an implementation scene where the polyp positioning model is divided into the white light polyp positioning model and the NBI polyp positioning model, the foregoing step <NUM> that a colon polyp image processing apparatus detects a position of a polyp in a to-be-processed endoscopic image by using a polyp positioning model, and positions a polyp image block in the endoscopic image includes:.

In this embodiment of this application, it is necessary to determine the specific position of the polyp in the endoscopic image, to provide input data for the next operation of polyp property identification. Considering the requirements on real-time performance, in this embodiment of this application, the position of the polyp is detected by using the YOLOv2 algorithm. A principle and an implementation of YOLOv2 are described below. The YOLOv2 is a joint training method for detection and classification. A YOLO9000 model is trained based on a COCO detection data set and an ImageNet classification data set by using the joint training method, and the model can detect more than <NUM> types of objects. YOLOv2 are improved in many aspects compared with YOLOv1, so that performance of YOLOv2 is remarkably improved, and the speed of YOLOv2 is still very fast. The YOLOv2 algorithm is an upgraded version of a YOLO algorithm, and is an end-to-end real-time target detection and recognition algorithm. By using a single neural network, the algorithm transforms a target detection problem into extraction of bounding boxes in images and a regression problem of category probabilities. Compared with YOLO, the YOLOv2 algorithm uses a multi-scale training method and borrows the concept of Faster RCNN anchor box, thus not only ensuring a detection speed, but also greatly improving the accuracy and generalization ability of model detection.

The YOLOv2 algorithm is applied to a polyp positioning task in this embodiment of this application, a detection target is a colon polyp, and a size of the anchor box is obtained through clustering according to built-in polyp training data. A transfer learning technology is used in algorithm training. Transfer learning refers to applying mature knowledge in a field to other scenes, and in terms of a neural network, it means transferring a weight of each node in network layers from a trained network to a brand new network instead of starting from scratch, and it is unnecessary to train a neural network for each specific task. Parameters trained by using an open-source, large-scale, labeled data set are used for initialization. For example, the data set may be Imagenet data. The Imagenet data is an open source data set related to image classification and target detection in the field of computer vision. The Imagenet data covers tens of thousands of categories, and has a data volume of more than one million. Using model initialization parameters trained by a large-scale data set may better allow a model to converge to a global optimal solution.

In an image classification model, white light type pictures and NBI type pictures are distinguished. The two types of images differ greatly in terms of polyp appearances. A flow direction of a blood vessel may be observed in the NBI type picture, and the color of the blood vessel is black in the NBI type picture. Therefore, it is necessary to train respective polyp positioning models for the white light picture data and the NBI picture data, which are referred to as a white light polyp positioning model and an NBI polyp positioning model. The two polyp positioning models are both trained by using the method described above, and the only difference is training data of the models. The training data of the white light polyp positioning model is white light type pictures, and the training data of the NBI polyp positioning model is NBI type pictures. In a process of the algorithm, when a previous module determines an image as a white light type picture, the white light polyp positioning model is called to position the polyp; otherwise, the NBI polyp positioning model is called to position the polyp. The circled polyp image block is outputted in a case that the polyp is positioned, to be used as an input of a polyp property identification model.

Before the foregoing step <NUM>, the colon polyp image processing method provided in this embodiment of this application may further include the following step <NUM>.

In step100, the colon polyp image processing apparatus obtains the to-be-processed endoscopic image from an endoscopic video stream.

In this embodiment of this application, when a doctor operates an endoscope to examine the colon, the endoscope apparatus may generate an endoscopic video stream, where the endoscopic video stream includes successive frames of endoscopic images. After the endoscope apparatus generates the endoscopic video stream, the endoscopic video stream may be transmitted to the colon polyp image processing apparatus. The colon polyp image processing apparatus may receive the endoscopic video stream from the endoscope apparatus, and obtain a single frame of endoscopic image from the endoscopic video stream. For each frame of endoscopic image, the polyp position and polyp type may be identified according to the method provided in this embodiment of this application, so that the property of the colon polyp in the endoscopic video stream may be identified in real time. When the doctor operates the endoscope to examine the colon, the position of the colon polyp in the video stream may be positioned in real time and the property of the polyp may be determined. If the polyp is identified as a non-adenomatous polyp, the doctor does not need to remove the polyp for pathological examination. Processing the endoscopic image of each frame according to this embodiment of this application may help the doctor to find the polyp in real time and prevent missed diagnosis of the polyp, and may also help the doctor to determine the property of the polyp, so that the doctor determines the polyp more accurately. In the subsequent steps, image processing may be performed on the endoscope image in the single frame to output an identification result. For the processing of endoscope images in other frames in the endoscopic video stream, refer to the foregoing processing procedure, which is only explained herein.

<FIG> is a schematic diagram of an endoscopic image according to an embodiment of this application. After the endoscopic video stream is generated, one frame of endoscopic image is extracted from the endoscopic video stream. In the picture shown in <FIG>, the endoscopic image is a colon image shown in the box, parameters of the endoscope are shown on the left side of the endoscopic image, and parameter values of the endoscope may be set according to an actual scene. The parameters of the endoscope have nothing to do with the image processing. Therefore, only a colon image region may be reserved after the endoscopic video stream is acquired.

In algorithms designed in the related art, it is necessary to manually filter out low-quality noise data. However, the algorithms in the related art cannot be used in an actual production environment. Because the low-quality noise data is filtered out manually, the designed algorithms have a good effect in an ideal environment, but cannot be used in an actual scene. In order to resolve this problem, in some embodiments of this application, after step <NUM> of obtaining the to-be-processed endoscopic image from the endoscopic video stream, the method provided in this embodiment of this application further includes the following steps:.

The colon polyp image processing apparatus extracts a color feature, a gradient variation feature and an abnormal brightness feature from the endoscopic image.

The colon polyp image processing apparatus determines whether the endoscopic image is a low-quality picture according to the color feature, the gradient variation feature and the abnormal brightness feature, where the low-quality picture includes: a blurred picture, an overexposed/underexposed picture with abnormal tone, and a low-resolution picture.

The following step <NUM> is triggered in a case that the endoscopic image is not the low-quality picture: The colon polyp image processing apparatus detects the position of the polyp position in the to-be-processed endoscopic image by using the polyp positioning model.

The low-quality picture may also be referred to as a low-quality picture. For the endoscopic image in a single frame in the input video stream, it is determined whether the endoscopic image is a low-quality picture; if the endoscopic image is the low-quality picture, the endoscopic image is directly filtered out and the subsequent module identification is skipped. In an actual production environment, there are a large number of blurred pictures and fecal water pictures caused by underprepared intestinal, which affect the subsequent polyp positioning and an algorithm effect of a property identification module. Therefore, in this embodiment of this application, the color feature, the gradient variation feature and the abnormal brightness feature may be extracted to detect, based on the three extracted features, whether the endoscopic image is the low-quality picture.

The low-quality picture defined in this embodiments of this application includes three categories: blurred picture, overexposed/underexposed picture with abnormal tone, and low-resolution picture. <FIG> is a schematic diagram of endoscopic images that are a qualified pictures according to an embodiment of this application. The two pictures on the left and right shown in <FIG> are both qualified pictures. The qualified pictures refer to pictures other than the blurred picture, the overexposed/underexposed picture with abnormal tone, and the low-resolution picture. <FIG> is a schematic diagram of endoscopic images that are overexposed/underexposed pictures with abnormal tone according to an embodiment of this application. Abnormal colors occur in the both left and right pictures shown in <FIG>, and therefore, the pictures are unqualified pictures. <FIG> is a schematic diagram of endoscopic images that are blurred pictures according to an embodiment of this application. Both the left and right pictures shown in <FIG> are blurred, and therefore are unqualified pictures. Specific identification processes of a blurred picture, an overexposed/underexposed picture with abnormal tone, and a low-resolution picture will be illustrated respectively below with examples.

Identification of a low-resolution picture may be achieved by calculating an effective pixel area in the picture. The effective pixel area refers to an area after black borders on upper, lower, left and right sides of the picture are removed through cropping, as shown by the area enclosed by a white box in <FIG>. A black border cropping algorithm is mainly to collect statistics about gray-scale value distribution of pixel values in each row or each column. If a ratio of gray or black pixel values in a row or column is greater than a certain value, it is considered that the row or column needs to be removed through cropping. If the effective area after the black borders are removed through cropping is less than a certain threshold, the picture is considered to be a low-resolution picture, where the threshold may be customized according to actual applications.

A detection algorithm for a blurred picture can be performed as follows:.

Finally, whether the endoscopic image is a blurred picture may be determined according to the similarity between G_P and G_R.

In a detection algorithm for an overexposed/underexposed picture with abnormal tone, there are numerous abnormal types, which can hardly be exhausted. Therefore, a standard library file for qualified tones and normal shooting is created. A detection algorithm can be performed as follows:.

An endoscopic image that meets the foregoing target tone matching result may be determined as an overexposed/underexposed picture with abnormal tone.

In some embodiments of this application, the endoscopic video stream may be generated in a plurality of shooting methods. Therefore, the endoscopic images in the endoscopic video stream may include a plurality of picture types according to different shooting methods. Different polyp positioning models need to be used for different picture types during the polyp position detection, and details are described in the subsequent embodiments.

After step <NUM> of obtaining the to-be-processed endoscopic image from the endoscopic video stream, the method provided in this embodiment of this application further includes the following step:.

The colon polyp image processing apparatus identifies a picture type of the endoscopic image, and determines that the endoscopic image is a white light type picture or an NBI type picture.

According to different shooting methods used for the endoscopic video stream, the endoscopic image extracted from the endoscopic video stream may also have different picture types. For example, the endoscopic image may be a white light type picture or an NBI type picture. <FIG> is a schematic diagram of an endoscopic image that is a white light type picture according to an embodiment of this application. The white light type picture refers to a red, green and blue (RGB) image that is imaged by an ordinary light source. <FIG> is a schematic diagram of an endoscopic image that is an NBI type picture according to an embodiment of this application. In the NBI type picture, a broad band spectrum of RGB light waves emitted by the endoscope light source is removed by a filter, and only a narrow band spectrum is reserved for diagnosing various digestive tract diseases.

The identifying a picture type of the endoscopic image, and determining that the endoscopic image is a white light type picture or an NBI type picture includes:.

In this embodiment of this application, first, training data for a white light type and an NBI type are obtained respectively, that is, white light type picture training data and NBI type picture training data are obtained. An image classification model is trained in advance by using a neural network algorithm, and the image classification model may be trained by using a plurality of machine learning algorithms. For example, the image classification model specifically may be a deep neural network (DNN) model, or a cyclic neural network model. For example, the deep neural network model may be densely connected convolutional networks (DenseNet). After the white light type picture training data and the NBI type picture training data are collected in advance model training is performed through the white light type picture training data and the NBI type picture training data, a trained image classification model is outputted.

After the training of the image classification model is completed, a blood vessel color feature is extracted from the endoscopic image by using the trained image classification model, and the blood vessels color feature is a basis for classification of the endoscopic image. Finally, a value of the blood vessels color feature is classified by using the trained image classification model to obtain that the endoscope image is the white light type picture or the NBI type picture.

In this embodiment of this application, an input of the image classification model is a qualified single frame of endoscopic image, and the image classification model outputs a result indicating whether the endoscopic image is a white light type picture or an NBI type picture. When a doctor actually operates an endoscope to examine the colon, if a suspected polyp is found, a pathological type of the current polyp is generally diagnosed in an NBI mode. A picture in the NBI mode may show a direction of the blood vessel more clearly. <FIG> shows a white light type picture and <FIG> shows an NBI type picture. For example, the image classification model in this embodiment of this application may classify and detect a picture type by using DenseNet. Certainly, other picture classification networks may also be used in this embodiment of this application to achieve similar functions; however, the identification effect may differ in some degree, which is not limited herein.

Execution of the image classification model may be converted into an image classification problem. An image classification algorithm used is the DenseNet. A size of an input image of the networks is <NUM>*<NUM>. Therefore, an inputted original picture is scaled to a fixed size of <NUM>*<NUM> first. Considering that a task of the image classification model prefers lower-level feature combinations, for example, blood vessel color and the like, a wider and shallower mode is used when the combination of depth and width of the DenseNet structure is designed. The final network structure used is DenseNet-<NUM>, where <NUM> refers to the number of network layers. A growth-rate is set to <NUM> through network parameter optimization, and a compression ratio of features through a transition layer is <NUM>, thereby achieving an optimal effect. A model structure is shown in the following Table <NUM>:.

In the embodiment shown in the foregoing Table <NUM>, the function implementation and execution process of each layer in DenseNet-<NUM> may be determined according to scenes. In addition, conv in the network layers includes three operations: batch normalization (batchnorm), activation layer (ReLU) and a convolution layer.

In step <NUM>, the colon polyp image processing apparatus performs a polyp type classification detection on the polyp image block by using a polyp property identification model, and outputs an identification result.

In this embodiment of this application, after the polyp image block is circled in the endoscopic image, next, it is only necessary to perform a polyp type classification detection on the polyp image block by using the polyp property identification model trained in advance, and output the identification result. The identification result may output a polyp type with a maximum probability, and may also output polyp types under various confidence conditions, where the confidence is a credibility of the polyp image block including various polyp types after a prediction is performed based on the polyp property identification model.

In this embodiment of this application, the polyp property identification model may perform a polyp property discrimination task, which is implemented, for example, through an image classification task, and an input is picture data of a positioning box outputted by the polyp positioning model. As shown in <FIG>, the polyp image block circled in the endoscopic image is the polyp detected by the polyp positioning model, and is used as input data of the polyp property identification model. A module output may be four class values (<NUM>, <NUM>, <NUM>, <NUM>), where <NUM> means that the region has no polyps and is normal, <NUM> represents a non-adenomatous polyp, <NUM> represents an adenomatous polyp, and <NUM> represents an adenocarcinoma. In addition, respective confidence conditions may be set for a normal region, a non-adenomatous, an adenomatous, and an adenocarcinoma. If the output is <NUM>, a determining result of the polyp positioning model is corrected, this region has no polyps and is a normal region.

In some embodiments of this application, step <NUM> that the colon polyp image processing apparatus performs a polyp type classification detection on the polyp image block by using a polyp property identification model, and outputs an identification result includes:.

In this embodiment of this application, the polyp picture training data of different polyp types is obtained first. The polyp property identification model is obtained in advance through training using the neural network algorithm, and the polyp property identification model may be trained by using a plurality of machine learning algorithms. For example, the polyp property identification model may be a deep neural network model or a cyclic neural network model. For example, the deep neural network model may be DenseNet. After the polyp picture training data of different polyp types is collected in advance and model training is performed through the polyp picture training data of different polyp types, the trained polyp property identification model is outputted.

After the training of the polyp property identification model is completed, the polyp type feature is extracted from the polyp image block by using the trained polyp property identification model, and the polyp type feature is a basis for classification of the polyp image block. Finally, the value of the polyp type feature is classified by using the trained polyp property identification model, to obtain the identification result.

In some embodiments of this application, after step <NUM> that the colon polyp image processing apparatus positions the polyp image block in the endoscopic image, the method provided in this embodiment of this application further includes the following steps:.

The colon polyp image processing apparatus expands a polyp region occupied by the polyp image block in the endoscopic image upwards, downwards, leftwards and rightwards according to a preset image expansion ratio, to obtain an expanded polyp image block.

The colon polyp image processing apparatus inputs the expanded polyp image block into the polyp property identification model.

In this embodiment of this application, the polyp property classification task of the polyp property identification model may be implemented by using a DenseNet algorithm. The algorithm requires input images to have the same size. However, polyp positions outputted by the polyp positioning model has different sizes. During construction of algorithm input data, the method used in this embodiment of this application is as follows: for the polyp image block outputted by the polyp positioning model, expanding the region by <NUM>% upwards, downwards, leftwards and rightwards to ensure the framed region has context semantic information, to help the subsequent polyp property identification model to extract features. The expanded region is directly normalized to an input size of <NUM> * <NUM> required by the model. Considering the complexity of the task, deeper DenseNet may be used. The final network structure used is DenseNet-<NUM>. A growth-rate is set to <NUM> through the network parameter optimization, and a compression ratio of features through a transition layer is <NUM>, thereby achieving an optimal effect. A model structure is shown in the following Table <NUM>:.

Finally, according to the polyp image processing method provided in this embodiment of this application, it takes about <NUM> milliseconds (ms) to process each frame of endoscopic image, which meets requirements on real-time performance. Compared with doctors of different levels, the algorithm effect is equivalent to the level of top-notch doctors. When deployed in primary hospitals, the method may help doctors to find and identify polyps in real time.

In this embodiment of this application, the method may help a doctor to find a polyp and determine a property of the polyp in real time when the doctor conducts an endoscopic examination. The method may prevent the doctor from missing diagnosis of the polyp, and help the doctor to improve the accuracy of polyp property identification. If the polyp is identified a non-adenomatous polyp with high confidence, the doctor does not need to remove the polyp for pathological examination, which may reduce the operation time of the doctor, thereby further reducing a high complication risk of the patient and diagnosis cost of the patient, and reducing the burden of an endoscopist and a pathologist.

As can be learned from the description of the foregoing embodiments of this application, a position of a polyp in an endoscopic image is first detected by using a polyp positioning model, and a polyp image block is positioned in the endoscopic image, where the polyp image block includes: a position region of the polyp in the endoscopic image. Finally, a polyp type classification detection is performed on the polyp image block by using a polyp property identification model, and an identification result is outputted. In the embodiments of this application, because the position of the polyp is detected by using the polyp positioning model, the polyp image block may be directly positioned in the endoscopic image. The classification detection for the polyp type is also performed on the polyp image block, and does not need to be performed on the entire endoscopic image. Therefore, the real-time performance meets requirements. When the endoscope is controlled to move, an identification result of an image acquired in real time can be outputted in real time, thereby improving processing efficiency of the polyp image.

For the convenience of better implementation of the foregoing solutions of the embodiments of this application, the following further provides a related apparatus configured to implement the foregoing solutions.

Referring to <FIG>, an embodiment of this application provides a colon polyp image processing apparatus <NUM>. The apparatus may include one or more processors and one or more memories storing a program unit, the program unit being executed by the processor. The program unit includes: a position detection module <NUM> and a polyp classification module <NUM>.

The position detection module <NUM> is configured to detect a position of a polyp in a to-be-processed endoscopic image by using a polyp positioning model, and position a polyp image block in the endoscopic image, the polyp image block including: a position region of the polyp in the endoscopic image.

The polyp classification module <NUM> is configured to perform a polyp type classification detection on the polyp image block by using a polyp property identification model, and output an identification result.

In some embodiments of this application, as shown in <FIG>, the colon polyp image processing apparatus <NUM> may further include: an image obtaining module <NUM>.

The image obtaining module <NUM> is configured to obtain the to-be-processed endoscopic image from an endoscopic video stream.

In some embodiments of this application, as shown in <FIG>, the colon polyp image processing apparatus <NUM> further includes:
a low-quality picture identification module <NUM>, configured to extract a color feature, a gradient variation feature and an abnormal brightness feature from the endoscopic image before the position detection module <NUM> detects the position of the polyp in the to-be-processed endoscopic image by using the polyp positioning model; determine whether the endoscopic image is a low-quality picture according to the color feature, the gradient variation feature and the abnormal brightness feature, where the low-quality picture includes: a blurred picture, an overexposed/underexposed picture with abnormal tone, and a low-resolution picture; and trigger the position detection module in a case that the endoscopic image is not the low-quality picture.

Referring <NUM>. to <FIG>, the colon polyp image processing apparatus <NUM> further includes:
a picture type identification module <NUM>, configured to identify a picture type of the endoscopic image before the position detection module <NUM> detects the position of the polyp in the to-be-processed endoscopic image by using the polyp positioning model, and determine that the endoscopic image is a white light type picture or an endoscope NBI type picture.

Referring to <FIG>, the picture type identification module <NUM> includes:.

In some embodiments of this application, the polyp positioning model includes: a white light polyp positioning model and an NBI polyp positioning model;.

In some embodiments of this application, the position detection module <NUM> is specifically configured to perform polyp positioning by using the white light polyp positioning model in a case that the endoscopic image is the white light type picture, to position a white light polyp image block in the endoscopic image; and perform polyp positioning by using the NBI polyp positioning model in a case that the endoscopic image is the NBI type picture, to position an NBI polyp image block in the endoscopic image.

In some embodiments of this application, referring to <FIG>, the polyp classification module <NUM> includes:.

An embodiment of this application further provides another terminal. As shown in <FIG>, for ease of description, only parts related to the embodiments of this application are shown. For specific technical details that are not disclosed, refer to the method part in the embodiments of this application. The terminal may be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), an on-board computer and the like, and the terminal being a mobile phone is used as an example:.

<FIG> is a block diagram of a partial structure of a mobile phone related to a terminal according to an embodiment of this application. Referring to <FIG>, the mobile phone includes components such as a radio frequency (RF) circuit <NUM>, a memory <NUM>, an input unit <NUM>, a display unit <NUM>, a sensor <NUM>, an audio circuit <NUM>, a wireless fidelity (Wi-Fi) module <NUM>, a processor <NUM>, and a power supply <NUM>. A person skilled in the art may understand that the structure of the mobile phone shown in <FIG> does not constitute a limitation to the mobile phone, and the mobile phone may include more components or fewer components than those shown in the figure, or some components may be combined, or a different component deployment may be used.

The components of the mobile phone are described in detail below with reference to <FIG>:.

The RF circuit <NUM> may be configured to receive and transmit signals during an information receiving and transmitting process or a call process. Specifically, the RF circuit receives downlink information from a base station, then delivers the downlink information to the processor <NUM> for processing, and transmits designed uplink data to the base station. Usually, the RF circuit <NUM> includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), and a duplexer. In addition, the RF circuit <NUM> may also communicate with a network and another device through wireless communication. The wireless communication may use any communications standard or protocol, including, but not limited to a global system of mobile communication (GSM), a general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), Long Term Evolution (LTE), an email, a short messaging service (SMS), and the like.

The memory <NUM> may be configured to store a software program and module. The processor <NUM> runs the software program and module stored in the memory <NUM>, to implement various functional applications of the mobile phone and data processing. The memory <NUM> may mainly include a program storage area and a data storage area. The program storage area may store an operating system, an application program required for at least one function (such as an audio playing function, an image playing function and the like). The data storage area may store data (such as audio data, a phone book and the like) created according to use of the mobile phone. In addition, the memory <NUM> may include a high speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk memory device, a flash memory device, or other non-volatile solid state memory devices.

The input unit <NUM> may be configured to receive an entered numeral or character information, and generate key signal input related to user setting and function control of the mobile phone. Specifically, the input unit <NUM> may include a touch panel <NUM> and other input devices <NUM>. The touch panel <NUM>, also referred to as a touchscreen, may collect a touch operation performed by a user on or near the touch panel (such as an operation performed by a user on the touch panel <NUM> or near the touch panel <NUM> by using any proper object or accessory, such as a finger or a stylus), and drive a corresponding connecting apparatus according to a preset program. Optionally, the touch panel <NUM> may include two parts: a touch detection apparatus and a touch controller. The touch detection apparatus detects a touch position of a user, detects a signal generated by the touch operation, and transfers the signal to the touch controller. The touch controller receives the touch information from the touch detection apparatus, converts the touch information into touch point coordinates, and transmits the touch point coordinates to the processor <NUM>. Moreover, the touch controller can receive and execute a command sent from the processor <NUM>. In addition, the touch panel <NUM> may be a touch panel of a resistive, capacitive, infrared, or surface acoustic wave type. In addition to the touch panel <NUM>, the input unit <NUM> may further include another input device <NUM>. Specifically, the another input device <NUM> may include but is not limited to one or more of a physical keyboard, a function key (such as a volume control key or a power on/off key), a trackball, a mouse, a joystick, and the like.

The display unit <NUM> may be configured to display information entered by a user or information provided for the user, and various menus of the mobile phone. The display unit <NUM> may include a display panel <NUM>. Optionally, the display panel <NUM> may be configured by using a liquid crystal display (LCD), an organic light-emitting diode (OLED), and the like. Further, the touch panel <NUM> may cover the display panel <NUM>. After detecting a touch operation on or near the touch panel <NUM>, the touch panel <NUM> transfers the touch operation to the processor <NUM>, to determine a type of a touch event. Then, the processor <NUM> provides a corresponding visual output on the display panel <NUM> according to the type of the touch event. Although, in <FIG>, the touch panel <NUM> and the display panel <NUM> are used as two separate parts to implement input and output functions of the mobile phone, in some embodiments, the touch panel <NUM> and the display panel <NUM> may be integrated to implement the input and output functions of the mobile phone.

The mobile phone may further include at least one sensor <NUM> such as an optical sensor, a motion sensor, and other sensors. Specifically, the optical sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor may adjust luminance of the display panel <NUM> according to brightness of the ambient light. The proximity sensor may switch off the display panel <NUM> and/or backlight when the mobile phone is moved to the ear. As one type of motion sensor, an acceleration sensor can detect magnitude of accelerations in various directions (generally on three axes), may detect magnitude and a direction of the gravity when static, and may be applied to an application that recognizes the attitude of the mobile phone (for example, switching between landscape orientation and portrait orientation, a related game, and magnetometer attitude calibration), a function related to vibration recognition (such as a pedometer and a knock), and the like. Other sensors, such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which may be configured in the mobile phone, are not further described herein.

The audio circuit <NUM>, a speaker <NUM>, and a microphone <NUM> may provide audio interfaces between the user and the mobile phone. The audio circuit <NUM> may convert received audio data into an electrical signal and transmit the electrical signal to the speaker <NUM>. The speaker <NUM> converts the electrical signal into a sound signal for output. On the other hand, the microphone <NUM> converts a collected sound signal into an electrical signal. The audio circuit <NUM> receives the electrical signal, converts the electrical signal into audio data, and outputs the audio data to the processor <NUM> for processing. Then, the processor <NUM> transmits the audio data to, for example, another mobile phone by using the RF circuit <NUM>, or outputs the audio data to the memory <NUM> for further processing.

Wi-Fi belongs to a short distance wireless transmission technology. The mobile phone may help, by using the Wi-Fi module <NUM>, a user to receive and send an email, browse a web page, access stream media, and the like. This provides wireless broadband Internet access for the user. Although <FIG> shows the Wi-Fi module <NUM>, it may be understood that the Wi-Fi module <NUM> is not a necessary component of the mobile phone, and when required, the Wi-Fi module <NUM> may be omitted without changing the scope of the essence of the present disclosure.

As a control center of the mobile phone, the processor <NUM> is connected to all parts of the entire mobile phone by using various interfaces and lines, and performs various functions and data processing of the mobile phone by running or executing the software program and/or module stored in the memory <NUM> and invoking the data stored in the memory <NUM>, to perform overall monitoring on the mobile phone. Optionally, the processor <NUM> may include one or more processing units. Preferably, the processor <NUM> may integrate an application processor and a modem. The application processor mainly processes an operating system, a user interface, and an application program and the like, and the modem mainly processes wireless communication. It may be understood that the foregoing modem may alternatively not be integrated into the processor <NUM>.

The mobile phone further includes the power supply <NUM> (such as a battery) for supplying power to the components. Preferably, the power supply may be logically connected to the processor <NUM> by using a power management system, thereby implementing functions such as charging, discharging, and power consumption management by using the power management system.

Although not shown in the figure, the mobile phone may further include a camera, a Bluetooth module, and the like, which are not described herein.

In an embodiment of this application, the processor <NUM> included in the terminal further controls and performs a procedure of a colon polyp image processing method performed by the terminal.

<FIG> is a schematic structural diagram of a server according to an embodiment of this application. The server <NUM> may vary greatly due to different configurations or performance, and may include one or more central processing units (CPU) <NUM> (for example, one or more processors) and a memory <NUM>, and one or more storage medium <NUM> (for example, one or more mass storage devices) that store application programs <NUM> or data <NUM>. The memory <NUM> and the storage medium <NUM> may be transient storage or permanent storage. The program stored in the storage medium <NUM> may include one or more modules (not shown), and each module may include a series of instructions and operations for the server. Further, the CPU <NUM> may be set to communicate with the storage medium <NUM>, and perform, on the server <NUM>, the series of instruction operations in the storage medium <NUM>.

The server <NUM> may further include one or more power supplies <NUM>, one or more wired or wireless network interfaces <NUM>, one or more input/output interfaces <NUM>, and/or one or more operating systems <NUM>, for example, Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, or FreeBSDTM.

The steps of the colon polyp image processing method performed by the server in the foregoing embodiment may be based on the server structure shown in <FIG>.

In addition, the described apparatus embodiment is merely an example. Besides, in the accompanying drawings of the apparatus embodiments of this application, a connection relationship between modules indicates a communication connection between them, and can be specifically implemented as one or more communications buses or signal lines. A person of ordinary skill in the art may understand and implement the embodiments of this application without creative efforts.

According to the descriptions in the foregoing implementations, a person skilled in the art may clearly understand that the embodiments of this application may be implemented by software and necessary general hardware, and certainly can also be implemented by specific hardware including: an application-specific integrated circuit, a specific CPU, a specific memory, a specific component, and the like. Generally, any function implemented by a computer program can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same function may be various. The structures may be an analog circuit, a digital circuit, a specific circuit, or the like. However, for the embodiments of this application, the implementation by a software program is the better one in more cases. Based on such an understanding, the technical solutions in the embodiments of this application essentially or the part contributing to the related art may be implemented in a form of a software product. The computer software product is stored in a readable storage medium such as a floppy disk of a computer, a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform the method described in the embodiments of this application.

Claim 1:
A colon polyp image processing method, comprising:
detecting, by a colon polyp image processing apparatus, a position of a polyp in a to-be-processed endoscopic image by using a polyp positioning model, and positioning a polyp image block in the endoscopic image, the polyp image block having: a position region of the polyp in the endoscopic image (<NUM>); and
performing, by the colon polyp image processing apparatus, a polyp type classification detection on the polyp image block by using a polyp property identification model, and outputting an identification result (<NUM>);
characterized by
before the detecting, by a colon polyp image processing apparatus, a position of a polyp in a to-be-processed endoscopic image by using a polyp positioning model, the method further comprising: identifying, by the colon polyp image processing apparatus, a picture type of the endoscopic image, and determining that the endoscopic image is a white light type picture or an endoscope narrow band imaging, NBI, type picture;
wherein the identifying, by the colon polyp image processing apparatus, a picture type of the endoscopic image, and determining that the endoscopic image is a white light type picture or an endoscope NBI type picture comprises: performing classification training on an original image classification model through white light type picture training data and NBI type picture training data by using a neural network algorithm, to obtain a trained image classification model; extracting a blood vessel color feature from the endoscopic image by using the trained image classification model; and classifying a value of the blood vessel color feature by using the trained image classification model, to determine that the endoscopic image is the white light type picture or the NBI type picture.