Patent Description:
Device-assisted enteroscopy is an important exam for the diagnostic and therapeutic approach of the gastrointestinal tract, particularly of the small bowel. Enteroscopy allows in-depth exploration of the small bowel, enabling tissue sampling and the application of endoscopic therapy. The application of artificial intelligence (AI) algorithms to different endoscopic techniques has provided promising results. Convolutional Neuronal Networks (CNN) are a highly efficient multi-layered artificial intelligence architectures designed for image analysis. The application of these tools for automatic detection of lesions in device-assisted enteroscopy has not been thoroughly explored. The present invention discloses CNN-based method for the automatic detection and differentiation of clinical relevant lesions in the esophagus, stomach, small bowel and colon in enteroscopy exams.

Medical image visualization software allows clinicians to measure and report functional or anatomical characteristics on the medical image regions. Acquisition, processing, analysis, and medical image data storage play an essential role in diagnosing and treating patients. A medical imaging workflow and devices involved are configured, monitored, and updated throughout the operation of the medical imaging workflow and tools. Machine learning can help configure, monitor, and update the medical imaging workflow and devices.

Machine learning techniques can be used to classify an image. Deep learning uses algorithms to model high-level abstractions in data using a deep graph with multiple processing. Using a multilayered architecture, machines employing deep learning techniques process raw data to find groups of highly correlated values or distinctive themes.

Document <CIT>) shows a system to detect polyps in endoscopic procedures. Document <CIT>) presents a method that uses trained model for image classification in endoscopy. The present invention presents a method to train supervised classification models fast. The trained models can be applied, for example, in the processes of claim <NUM>. (ix) of the <CIT>).

Document <CIT>) discloses a system to detect anatomical landmarks in endoscopic videos using trained image classifiers models. The present invention apply a training method for image classification models and therefore a platform to deploy trained models applied on endoscopic image classification systems.

Document <CIT>) presents a system to detect lymphomas under enteroscopy in realtime. The method applied trained model of object detection and classification. The invention does not apply optimized training sessions for image classification and does not classify images according to the present invention. Gastrointestinal lesions such as blood or hematic traces, ulcers/erosions, vascular lesions and protruding lesions, are currently one of the most common diseases and very often, when not correctly detected and treated, these may evolve into cancer or to a unfavorable clinical outcome such as gastrointestinal bleeding or inflammatory bowel disease.

<NPL> describes a convolutional neural network for anomalies classification in the gastrointestinal tract using endoscopy images. The document discloses the usage of different network architectures and achieves remarkable accuracy in predicting eight-class anomalies.

<NPL> describes Multiple deep learning approaches to classification or segmentation of medical abdominal images. Features are extracted from the images using different convolutional architectures, but none specifically directed towards enteroscopic imaging of the gastrointestinal track.

<CIT> describes a deep learning method of determining a clinical value for an individual based on a tumor in an image by training a classifier to the lymphocyte distribution to identify the tumor.

Endoscopic images, due to the nature of their acquisition, often lack the light or other photographic conditions to allow the classification of esophagus, stomach, small bowel and colon straightforwardly executed. Within this context, machine learning techniques have been presented to automatically execute such task but up-to-date they have failed to present overall accuracy or false-negative rate that can be used in clinical practice and hence leads to inappropriate treatment.

The present invention discloses a method for deep learning based detection and classification of esophagus, stomach, small bowel and colon lesions such as blood or hematic residues, ulcers and erosions, pleomorphic protruding and vascular lesions in device-assisted enteroscopy images and exam input video examination images and indirectly measuring inflammatory activity. Indeed, the gold standard for assessing bowel inflammatory activity is the direct visualization of the bowel mucosa. The automatic identification of lesions, blood or hematic residues, small bowel ulcers and erosions, small bowel protruding, and vascular lesions is vital to detect esophageal, gastric, enteric and colonic lesions, crucial for diagnosis and treatment planning.

By using trained convolutional layers of different architectures on the ImageNet<NUM> dataset and further testing them using sample of the device-assisted enteroscopy image stack, the potential to detect lesions is shown. The disruptive clinical nature of the present invention is justified by the artificial intelligence system's ability to detect pleomorphic luminal and mucosal lesions, particularly blood or hematic residues, ulcers and erosions, protruding and vascular lesions. Indeed, this novel neuronal network AI based approach, capable to automatically identify lesions of subtle pleomorphic nature, is of the utmost importance in clinical practice, allowing a complete identification of esophageal, stomach, small bowel and colon lesions in images obtained with the device-assisted enteroscopy diagnosis. Furthermore, the specific application of a tailor-made artificial intelligence system to device-assisted enteroscopy is a relevant novelty introduced by this invention to the current state of the art. One of the most critical and frequent indications for performing device-assisted enteroscopy is inflammatory bowel disease and gastrointestinal bleeding. Correct assessment of blood or hematic residues, ulcers and erosions, protruding and vascular lesions in the device-assisted enteroscopy findings is vital for clinical follow-up management. Therefore, by accurately identifying blood or hematic residues, ulcers and erosions, protruding and vascular lesions in device-assisted enteroscopy, the present invention helps the clinical team better define the diagnostic and therapeutic management of the patient, which may translate into optimized clinical outcomes.

Indeed, the clinical relevance of the developed model lies in its ability to not only detect but classify esophageal, stomach, small bowel and colon lesions in device-assisted enteroscopy.

The present invention detects relevant blood or hematic residues, ulcers and erosions, protruding and vascular lesions in device-assisted enteroscopy images/videos. Ulcers and erosions identification in device-assisted enteroscopy is vital to assess inflammatory gastrointestinal activity. Furthermore, the invention uses transfer learning and semi-active learning. Transfer learning allows feature extraction and high-accuracy classification using reasonable datasets sizes. The semi-active implementation allows a continuous improvement in the classification system. A system such as this can embark a multitude of categories with clinical relevance. Furthermore, the invention preferably uses transfer learning for feature extraction of device-assisted enteroscopy images with overall accuracy reaching <NUM>% and employs a semi-active learning strategy for device-assisted enteroscopy images.

In one embodiment of the method splits the dataset into a number of stratified folds, where images relative to a given patient are included in one-fold only. Further, additionally or alternatively, such data is trained and validated with patient grouping to a random fold, i.e., images from an arbitrary patient are ensured to belong to one and one only fold.

Preferred is a method which uses the chosen training and validation sets to further train a series of network architectures, which include, among others, a feature extraction, and a classification component. The series of convolutional neuronal networks to train include but are not limited to: VGG16, InceptionV3, Xception EfficientNetB5, EfficientNetB7, Resnet50, and Resnet125. Preferably, their weights are frozen, with exception to the BatchNormalization layers, and are coupled with a classification component. The classification component comprises at least two dense layers, preferably of sizes <NUM> and <NUM>, and at least one dropout layer of preferably <NUM> in between them.

Alternatively, but not preferentially, the classification component can be used with more dense layers or with dense layers of different size. Alternatively, but not preferentially, the classification component can also be used without dropout layers.

Further, additionally, and preferably, the best performing architecture is chosen according to the overall accuracy f1-metrics and sensitivity. Performance metrics include but are not limited to the f1-metrics accuracy. Further, the method is not limited to two to four dense layers in sequence, starting with <NUM> and decreasing in half up to <NUM>. Between the final two layers there is a dropout layer of <NUM> drop rate.

Lastly, the best performing solution is trained using the complete dataset with patient grouping.

Further embodiments of the present invention may include similar classification networks, training weights and hyperparameters.

These may include the usage of any image classification network, new or not yet designed.

In general, the method includes two modules: prediction and output collector. Prediction reads videos and flags images with findings. Conversely, the output collector passes these images with findings for processing.

Examples of advantageous effects of the present invention include: training using parameters from machine learning results of cloud-based every-day increasing datasets; automatic prediction of the device-assisted enteroscopy image by using a deep learning method so that the esophageal, stomach, small bowel and colon lesions from image input of the device-assisted enteroscopy can be identified and classified into blood or hematic residues, ulcers and erosions, protruding and vascular lesions; the usage of transfer learning improves the image classification speed and corresponding classification accuracy.

The present invention discloses a method capable of identify and differentiate esophageal, stomach, small bowel and colon classifying in images/videos acquired during a device-assisted enteroscopy medical procedure.

Some preferential embodiments will be described in more detail with reference to the accompanying drawings, in which the embodiments of the present disclosure have been illustrated. However, the present disclosure can be implemented in various manners, and thus should not be construed to be limited to the embodiments disclosed herein.

The term "deep learning" is a machine learning technique that uses multiple data processing layers to classify the data sets with high accuracy. It can be a training network (model or device) that learns based on a plurality of inputs and outputs. A deep learning network can be a deployed network (model or device) generated from the training network and provides an output response to an input.

The term "supervised learning" is a deep learning training method in which the machine is provided with already classified data from human sources. In supervised learning, features are learned via labeled input.

The term "convolutional neuronal networks" or "CNNs" are networks that interconnect data used in deep learning to recognize objects and regions in datasets. CNNs evaluate raw data in a series of stages to assess learned features.

The term "transfer learning" is a machine storing the information learned when attempting to solve one problem to solve another problem of similar nature as the first.

The term "semi-active learning" is used as a process of machine learning. Before executing the next learning process, the training network appends a set of labeled data to the training dataset from a trusted external entity. For example, as a machine collects more samples from specialized staff steps, the less prone it is to mispredict images of identical characteristics.

The term "computer-aided diagnosis" refers to machines that analyze medical images to suggest a possible diagnosis.

The term "protruding lesions" refers to pleomorphic group of lesions, containing polyps, nodules, subepithelial lesions as well as gastrointestinal tumors.

The term "vascular lesions" refer to pleomorphic group of vascular lesions with different hemorrhagic potential, such as red spots, angioectasias and varices.

The terms "ulcers and erosions" refer to mucosal breaks in the mucosa of the gastrointestinal tract. These lesions are distinguished based on estimated size and depth of penetration. "Ulcers" were defined as a depressed loss of epithelial covering, with a whitish base and surrounding swollen mucosa with > <NUM> of diameter. Conversely, mucosal erosions were defined as a minimal loss of epithelial layering surrounded by normal mucosa.

The present invention relates to a method for deep learning-based method for the detection and classification of blood or hematic residues, ulcers and erosions, protruding and vascular lesions in device-assisted enteroscopy images/video (<FIG>). Often, embodiments of the present invention provide a visual understanding of the deep learning blood or hematic residues, ulcers and erosions, protruding and vascular lesions detection method. Automatic lesion classification of esophageal, stomach, small bowel and colon images/videos in device-assisted enteroscopy is a challenging task due to the pleomorphic nature of the lesions, and its hemorrhagic and neoplastic potential. Large variations in the gastrointestinal tract preparation before device assisted enteroscopy further complicate automated gastrointestinal lesion classification. Although the automatic training and classification times are fast (on average <NUM> seconds for a test dataset of <NUM> images), the output is not satisfactory for a fast diagnosis by the experts.

The method comprises an image acquisition module, a storage module, a training input module, a processing module, an exam input module, a training module, a prediction module, an output collector module and a display module.

The image acquisition module <NUM> receives exam input volumes from device-assisted enteroscopy providers. Images and corresponding labels are loaded onto the storage module <NUM>. The storage module <NUM> includes a multitude of classification network architectures <NUM>, trained convolutional network architectures <NUM> and hyperparameters for training. The storage module <NUM> can be a local or cloud server. The storage module contains training input labelled data from device-assisted enteroscopy images and the required metadata to run processing module <NUM>, training module <NUM>, prediction module <NUM>, a second prediction module <NUM>, and output collector module <NUM>. The input labelled data includes, but not exclusively, images and corresponding lesion classification. The input labelled data also includes anonymized tags of both provider and patient, and image size. The metadata includes a multitude of classification networks architectures <NUM> exemplified in <FIG>, a multitude of trained convolutional neuronal networks architectures <NUM>, training hyperparameters, training metrics, fully trained models, and selected fully trained models.

Images <NUM> and labelled data are processed at the processing module <NUM> before running the optimized training at the training module <NUM>. The processing module normalizes the images according to the deep model architecture, to be trained at the processing module <NUM> or evaluated at training module <NUM>. By manual or scheduled request, the processing module normalizes the images at the storage module <NUM> according to the deep model architectures that will run at training module <NUM>. All normalization procedure clean text annotation artifacts, center the region of interest, resize the image, and normalize the image pixels (between <NUM> and <NUM> or -<NUM> and <NUM>). Additionally, the processing module generates the data pointers to the storage module <NUM> to form the partial or full images and ground-truth labels required to run the training module <NUM>. To prepare each training session, a dataset is divided into folds, where patient-specific imagery is exclusive to one and one fold only, for training and testing. The training set is split for model training to generate the data pointers of all the images and ground-truth labels, required to run the training process <NUM>. K-fold is applied with stratified grouping by patient in the training set to generate the data pointers of the partial images and ground-truth labels, required to run the model verification process <NUM> of the training module <NUM>. The split ratios and number of folds are available at the metadata of the storage module. Operators include but are not limited to users, a convolutional neuronal network trained to optimize the k-fold or a mere computational routine. Merely as an example, the dataset is divided with patient split into <NUM>% for training and <NUM>% for testing. Optionally, images selected for training can be split into <NUM>% for training and <NUM>% for validation during training. A <NUM>-fold with stratified grouping by patient is applied in the images selected for training. By manual or scheduled request, the processing module normalizes the exam volume data <NUM> according to the deep model architecture to run at the prediction module <NUM>.

As seen in <FIG>, the training module <NUM> has a model verification process <NUM>, a model selection step <NUM> and a model training step <NUM>. The model verification part iteratively selects combinations of classification architectures <NUM> and convolutional networks <NUM> to train a deep model for esophageal, stomach, small bowel and colon lesion classification. The classification network <NUM> has Dense and Dropout layers to correctly classify esophageal, stomach, small bowel and colon lesions. A neuronal convolutional network <NUM> trained on large datasets is coupled to the said classification network <NUM> to train a deep model <NUM>. Partial training images <NUM> and ground-truth labels <NUM> train the said deep model <NUM>. The performance metrics of the trained deep model <NUM> are calculated using a plurality of partial training images <NUM> and ground-truth labels <NUM>. The model selection step <NUM> is based on the calculated performance metrics, such as f-<NUM>. The model training part <NUM> trains the selected deep model architecture <NUM>, at process <NUM>, using the entire data of training images <NUM> and ground-truth labels <NUM>. At the prediction module <NUM>, the trained deep model <NUM> outputs esophageal, stomach, small bowel and colon lesion classification <NUM> from a given evaluation image <NUM>. An exam volume of data <NUM> comprising the images from the device-assisted enteroscopy is the input of the prediction module <NUM>. The prediction module <NUM> classifies image volumes of the exam volume <NUM> using the best-performed trained deep model from <NUM> (see <FIG>). An output collector module <NUM> receives the classified volumes and load them to the storage module after validation by another neuronal network or any other computational system adapted to perform the validation task.

Merely as exemplificative, the invention comprises a server containing training images for architectures in which initial parameters were obtained from training in large cloud-based large datasets such as, but not only, ImageNet, ILSVRC, and JFT. The architecture variant include VGG, ResNet, Inception, Xception or Mobile, EfficientNets. All data and metadata can be stored in a cloud-based solution or on a local computer. Embodiments of the present invention also provide various approaches to make a faster deep model selection. <FIG> illustrates a method for deep learning esophageal, stomach, small bowel and colon lesion classification according to an embodiment of the present invention. The method of <FIG> includes a pretraining stage <NUM> and a training stage <NUM>. The training stage <NUM> is performed with early stopping on small subsets of data to select the best-performed deep neuronal network for esophageal, stomach, small bowel and colon lesion classification among multiple combinations of convolution and classification parts. For example, a classification network of two dense layers of size <NUM> is coupled with the Xception model to train on a random set resulting from k-fold cross validation with patient grouping. Another random set is selected as the test set.

The process of training <NUM> with early stopping and testing on random subsets is repeated in an optimization loop for combinations of (i) classification and transfer-learned deep neuronal networks; (ii) training hyperparameters. The image feature extraction component of the deep neuronal network is any architecture variant without the top layers accessible from the storage module. The layers of the feature extraction component remain frozen but are accessible at the time of training via the mentioned storage module. The BatchNormalization layers of the feature extraction component are unfrozen, so the system efficiently trains with device-assisted enteroscopy images presenting distinct features from the cloud images. The classification component has at least two blocks, each having, among others, a Dense layer followed by a Dropout layer. The final block of the classification component has a BatchNormalization layer followed by a Dense layer with the depth size equal to the number of lesions type one wants to classify.

The fitness of the optimization procedure is computed to (i) guarantee a minimum accuracy and sensitivity at all classes, defined by a threshold; (ii) minimize differences between training, validation, and test losses; (iii) maximize learning on the last convolutional layer. For example, if training shows evidence of overfitting, a combination of a shallow model is selected for evaluation.

The training stage <NUM> is applied on the best performed deep neuronal network using the whole dataset.

The fully trained deep model <NUM> can be deployed onto the prediction module <NUM>. Each evaluation image <NUM> is then classified to output a lesion classification <NUM>. The output collector module has means of communication to other systems to perform expert validation and confirmation on newly predict data volumes reaching <NUM>. Such means of communication include a display module for user input, a thoroughly trained neuronal network for decision making or any computational programmable process to execute such task. Validated classifications are loaded on the storage module to become part of the datasets needed to run the pipelines <NUM> and <NUM>, either by manual or schedule requests.

An embodiment of the classification network <NUM>, as seen in <FIG>, can classify according to the gastrointestinal lesions identification as N: normal; B: blood; PV: vascular lesions; PR: protruding lesions; PUE: ulcers or erosions; are shown and grouped accordingly. At a given iteration of method <NUM> (<FIG>, <FIG>, and <FIG>), the optimization pipeline described herein uses accuracy curves, ROC curves and AUC values and confusion matrix from training on a small subset of images and labelled data.

<FIG> illustrates exemplary ROC curves and AUC values obtained after training on a small subset of images and labelled data where <NUM> (N - AUC: <NUM>,<NUM>) and <NUM> represent the Random Guessing.

<FIG> illustrates an exemplary confusion matrix after training on a small subset of images and labelled data. Results used for model selection. Number of images of the small subset of data and respective class proportion between parentheses.

<FIG> shows examples of lesion classification according to an embodiment of the present invention, where in <NUM> there is no lesion; in <NUM> there is a ulcer/erosion; in <NUM> there is blood/hematic traces; in <NUM> there is a vascular lesion; in <NUM> there is a protruding lesion.

<FIG> shows a result of performing deep learning-based lesion classification on the data volume <NUM> and <NUM>, according to an embodiment of the present invention. The results of esophageal, stomach, small bowel and colon lesions classification, using the training method <NUM> of the present invention are significantly improved as compared to the results using the existing methods (without method <NUM>).

Claim 1:
A computer-implemented method capable of automatically detecting and differentiating esophageal, stomach, small bowel and colon lesions in device-assisted enteroscopy images by classifying the pixels as esophageal, stomach, small bowel and colon lesions, comprising selecting the architecture combination and fully training such architecture for esophageal, stomach, small bowel and colon lesions with means of output validation and storage capabilities wherein the method:
- selects a number of subsets of all endoscopic ultrasonography images/videos, each of said subsets considering only images from the same patient;
- selects another subset as validation set, wherein the subset does not overlap chosen images on the previously selected subsets;
- pre-trains (<NUM>) of each of the chosen subsets with one of a plurality of combinations of image feature extraction component, followed by a subsequent classification neural network component for pixel classification as esophageal, stomach, small bowel and colon lesions wherein said pre-training;
- early stops when the scores do not improved over a given number of epochs, namely three;
- evaluates the performance of each of the combinations;
- is repeated on new, different subsets, with another networks combination and training hyperparameters, wherein such new combination considers a higher number of dense layers if the f1-metric is low and fewer dense layers if the f1-metric suggests overfitting;
- selects (<NUM>) the architecture combination that performs best during pre-training;
- fully trains and validates (<NUM>) the selected architecture combination using the entire set of device-assisted enteroscopy images to obtain an optimized architecture combination for each classification;
- predicts (<NUM>) esophageal, stomach, small bowel and colon lesions using said optimized architecture combination for classification;
- receives the classification output (<NUM>) of the prediction module (<NUM>) by an output collect module (<NUM>) with means of communication to a third-party capable of performing validation of said network classification;
- stores the corrected prediction into the storage component (<NUM>).