Patent Description:
Colon polyps are growths on the inner lining of the colon and are very common. Colon polyps are significant because they may be or may become malignant (cancerous). They also are important because based on their size, number, and microscopic anatomy (histology), a clinician can predict which patients are more likely to develop more polyps and colon cancer.

Polyps may take on various shapes. For example, pedunculated polyps look like a mushroom, are attached to the lining of the colon by a thin stalk and flop around inside the intestine. Sessile polyps do not have a stalk and are attached to the lining by a broad base. Flat colon polyps are flat or even slightly depressed. These may be difficult to identify because they are not as prominent as polypoid or sessile polyps with the commonly-available methods of diagnosing polyps.

The most common type of polyp is the adenoma or adenomatous polyp. It is an important type of polyp to identify not only because it is the most common, but because it is the most common risk factor for colon cancer. The likelihood that an adenoma will develop into (or has already developed into) cancer is partially dependent on its type, shape and size; the larger the polyp, the more likely it is that the polyp is or will become malignant (e.g., concern about the potential malignancy increases with a polyp size greater than one centimeter in size).

It also matters if there is a single polyp or multiple polyps. Patients with multiple polyps, even if they are not malignant when examined under the microscope are more likely to develop additional polyps in the future that may become malignant. Concern about this increasing malignancy potential begins when there are three or more polyps.

Since there is a strong link between polyp size and the subsequent development of malignancy, it is essential that polyps are detected and sized accurately. The most common approach for sizing a polyp involves an endoscopist viewing a high definition screen during a procedure and visually estimating its size. Polyps removed during the procedure, e.g., using forceps or a snare, can thereafter be further measured by a pathologist. A recent study by the NIH however found significant discrepancies (e.g., an average difference of <NUM>) between the endoscopist's estimation and pathologist's measurement of colonic polyp size.

The results of the study suggest a wide variance in polyp size documentation. Neither clinician estimate nor pathologist measurement accurately reflects colonic polyp size. Inaccurate determination of polyp size can negatively impact patient outcomes involving advanced adenoma detection.

Patent document <CIT> describes a system and methods for detecting polyps using optical images acquired during colonoscopy. Patent document <CIT> describes analysis of data, images and video characterizing mammalian skin damage conditions using a mobile device as a data collection engine. al, "Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model" describes an AI model trained on endoscopic video that can differentiate diminutive adenomas from hyperplastic polyps with high accuracy. Patent document <CIT> describes a medical manipulator and a surgical support apparatus. al, "Use of a novel polyp 'ruler snare' improves estimation of colon polyp size" describes a prototype 'ruler snare' that includes graduated markings on the distal end of the sheath to facilitate polyp measurement.

There is provided an artificial intelligence (AI) platform for detecting and sizing a lesion in real time during a clinical procedure and a method for detecting and sizing a lesion in real time during a clinical procedure using AI in accordance with the independent claims. Other aspects of the invention are set forth in the dependent claims.

These and other features of this invention will be more readily understood from the following detailed description of the various aspects of the invention taken in conjunction with the accompanying drawings in which:.

The drawings are merely schematic representations, not intended to portray specific parameters of the invention. The drawings are intended to depict only typical embodiments of the invention, and therefore should not be considered as limiting the scope of the invention. In the drawings, like numbering represents like elements.

Referring to <FIG>, an artificial intelligence (AI) platform <NUM> is provided for detecting and sizing lesions during procedures in real-time. Note that while the embodiments are generally described with reference to detecting and sizing polyps, it is understood that the approach may be applied to any diseased tissue (i.e., lesions). AI platform <NUM> generally include: (<NUM>) a training system <NUM> that trains a detection classifier <NUM> based on a set of training images <NUM> and labels <NUM>; and (<NUM>) a real time video analysis system <NUM> that utilizes the detection classifier <NUM> to analyze a video feed <NUM> in real-time to provide polyp detection <NUM> and polyp sizing <NUM>. AI platform <NUM> may for example employ a neural network or other machine learning system.

In this illustrative embodiment, detection classifier <NUM> is trained with a deep learning system to detect both a lesion and a reference object, such as forceps, in the same image. (Alternatively, two detection classifiers <NUM> could be trained; one that detects lesions and one that detects reference objects. ) As described herein, the reference object is utilized to facilitate sizing of the lesion.

Training images <NUM> may for example be obtained from frames in captured videos. Images used for training include random objects along with the desired objects, in this case polyps and have a variety of backgrounds and lighting conditions. Additionally, in some training images <NUM> the desired object is partially obscured, overlapped with something else, only halfway in the picture, etc..

Once a set of training images <NUM> have been gathered, they are tagged with labels <NUM> to reference the desired objects in every picture. The desired object can be a polyp or any other lesions, or some other object that could appear in a video feed <NUM>, such as a finger, noise, shadows, etc. Any labeling system may be employed, such as LabelImg which is available as an open source on GitHub. <FIG> depicts an illustrative labeling tool <NUM> in which a user places a bounding box <NUM> around a lesion in a displayed training image <NUM> and selects a label for the type of lesion shown, e.g., a Pedunculated polyp, a Sessile polyp, a Flat polyp, etc. <FIG> depicts a further training image <NUM> in which a first bounding box <NUM> is placed around the lesion and a second bounding box <NUM> is placed around a reference object. In an illustrative embodiment, approximately <NUM> training images <NUM> are used, in which each has a resolution of 1280x1024 pixels.

Once all of the training images <NUM> are labeled, training system <NUM> can be employed to train the detection classifier <NUM>. Detection classifier <NUM> may for example comprise a neural network model into which the pixel data from each training image is processed. In one illustrative approach, an application programming interface (API) by TensorFlow™ can be used to construct the neural network model representative of a graph that includes nodes and edges. In this case, the model is mapped to underlying machine hardware. Nodes in the graph represent operations (e.g., machine learning functions, mathematical operations, etc.), and the edges represent the multidimensional data arrays also known as tensors communicated between the nodes. The unique edges, called control dependencies, can exist in the graph and denote that the source node must finish executing before the destination node starts executing. (TensorFlow provides a platform in which the designer's design algorithm flow and computation architecture is automatically optimized. ) Nodes are assigned to computational devices and execute asynchronously, and in parallel once all the tensors on their incoming edges become available.

The video processing required in the AI platform <NUM> can be very expensive in terms of CPU power. Accordingly, certain computer-vision sub-tasks are allocated to special-purpose hardware architectures, such as a GPU <NUM> (graphics processing unit), while others are allocated to the CPU <NUM>. The GPU <NUM>, for example, is an accelerator that is available not only on desktop computers but also on mobile devices such as smartphones and tablets. Accordingly, the model used herein has the built-in ability to configure GPU usage along with CPU usage to utilize machine resources most efficiently.

The AI platform <NUM> may for example utilize a NVIDIA CUDA® Deep Neural Network library (cuDNN), which is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides a highly tuned implementation of standard routines such as forward and backward convolution, pooling normalization, and activation layers. cuDNN provides high-performance GPU acceleration that automates low-level GPU performance tuning. AI platform <NUM> may also for example utilize a NVIDIA Quadro P5000 16GB - <NUM> CUDA CORES graphics card for development and testing. Anything above the NVIDIA GEFORCE GTX <NUM> - <NUM> CUDA CORES could likewise be utilized.

In the present case, the model is trained until the computed "loss" falls consistently below a threshold (e.g., <NUM> in the case of TensorFlow). Once training is completed, an inference graph representing the classification model is generated, which can then be deployed into the real-time video analysis system <NUM> (<FIG>).

Once deployed, images from a video feed <NUM> (e.g., generated during a procedure) can be analyzed by the real-time video analysis system <NUM> to provide polyp detection <NUM> and reference object detection <NUM> (to determine polyp sizing <NUM>). <FIG> depicts an illustrative. xml file for an image in the video feed <NUM> that includes an identified forceps and polyp.

In this two-step process, a lesion is first detected in real time while the clinician is doing the procedure. Next, the size of the lesion is calculated using the reference object, and then an indication of the lesion and size are output with a video feed. In a typical procedure involving polyps, forceps are used during a clinical procedure for intervention. Forceps comprise a slender flexible tool with movable cup-shaped jaws used to obtain biopsy specimens. Actual polyp size is determined based (<NUM>) on the measurement of the polyp size in pixels, (<NUM>) measurement of the size of the forceps in pixels, and (<NUM>) recalculation of the actual polyp size into a unit of length (e.g., millimeters) based on the known size of the forceps.

As shown in <FIG>, when the system identifies a polyp and the forceps in the same video frame, circumscribed rectangles around a polyp <NUM> and forceps <NUM> are defined and the coordinates of the corners of the rectangles get recorded. The area defined by the circumscribed rectangle around the polyp <NUM> gets cut out of the frame into a separate image <NUM> shown in <FIG>. A similar separate cut out image is created for the reference object (not shown).

<FIG> describes an illustrative method for processing the polyp image data. First, at S1, a rectangular cut out image <NUM> (<FIG>) containing the polyp is obtained (e.g., as an RGB image). Next, at S2, the RGB image gets split into three monochrome channels: Red, Green, and Blue and is converted into a grayscale image. At S3, the image gets smoothed by applying a Gaussian function to reduce image details and variation of brightness thus enhancing image structure. At S4, the polyp edges or contours are identified, e.g., using a Canny multi-stage edge detection algorithm, and at S5, multiple polyp contours are identified using an OpenCV function. The contours that fall below a threshold, e.g., contain less than <NUM> pixels, get filtered out. An example of this process is shown in <FIG> in which the image on right depicts a filtered contour image and the image on the left depicts an unfiltered contour image.

At S6, for all remaining contours, the minimum and maximum x and y-coordinates are determined (i.e., Xmin; Xmax; Ymin; Ymax). At S7, rectangle dimensions <NUM> for the polyp are recalculated. The new coordinates are defined as: <MAT>.

The rectangle dimensions are recalculated as: <MAT> <MAT>.

The larger of the two dimensions equals the polyp size in pixels (Dpx), e.g., 220x280 pixels. At S8, a circle is drawn around the polyp. The coordinates of the center of the circle denoted as: <MAT>.

In which the diameter of the circle Dc = Dpx.

<FIG> depicts an illustrative process for calculating the pixel size of the forceps. At S10, the area defined by a circumscribed rectangle around the forceps gets cut out of the frame into a separate image, and at S11, the color image is converted into a gray monochrome image. At S12, the image gets smoothed by applying, e.g., a Gaussian function to reduce image details and variation of brightness thus enhancing image structure, and at S13, the forceps edges are identified using, e.g., Canny multi-stage edge detection algorithm. At S14, straight lines are drawn along the forceps edges using a Hough transform technique, e.g., illustrative result shown in <FIG>.

At S15, edges are filtered and processed as follows. The number of lines gets reduced by filtering out short lines (e.g., less than <NUM> pixels). A straight line may be represented by equation: <MAT>.

For the remaining lines, the coefficients m and b and their angles of inclination to the x-axis are calculated. The lines with angles of inclination, e.g., less than <NUM>° and greater than <NUM>° are removed. The lines remaining on the image get extended to the image boundaries. The coordinates of intersection points (midpoints) of the lines with the horizontal boundaries of the image are calculated.

The groups of lines having a difference between their coefficients m < <NUM> or angles of inclination < <NUM>° and a difference between x coordinates of their intersection points with the upper horizontal boundary of the image < <NUM> pixels get replaced with a fewer number of midlines. A coefficient m of a midline for each group is calculated as follows: <MAT>.

A coefficient b for a midline is calculated as follows: <MAT>.

With several lines remaining on the image, the ones having an x-coordinate of the intersection points with the lower horizontal boundary of the image at a distance of <NUM> pixels or more from the left or right vertical image boundaries get removed.

Further line filtering leaves only two lines on the image: "left line", having a minimum x-coordinate ("x-left") of the intersection point with the upper horizontal image boundary and "right line" having a maximum x-coordinate ("x-right") of the intersection point with the upper horizontal image boundary, as shown in <FIG>. If the intersection point of the left line and the right line is located inside the image boundaries, the frame is considered not fit for the forceps size calculations. Another frame with a different forceps location must be taken. If the intersection point of the left line and the right line is located outside of the image boundaries, then the angles of inclination of the lines to the x-axis are evaluated.

As shown in <FIG>, if an angle of inclination of both lines less than or equal to <NUM>° then a perpendicular to the right line is drawn from the intersection point of the right line with the upper image boundary towards the left line until the perpendicular crosses the left line (left image). If an angle of inclination of both lines is greater than or equal to <NUM>° then a perpendicular to the left line is drawn from the intersection point of the left line with the upper image boundary towards the right line until the perpendicular crosses the right line (right image). A size of the forceps in pixels (Fpx) is the mean of the length of the perpendicular between the left and right lines and a distance between x-left and x-right points.

If an angle of inclination of the left line > <NUM>° and an angle of inclination of the right line < <NUM>°, then a size of the forceps in pixels (Fpx) is considered equal to the distance between x-left and x-right points (<FIG>).

Once the pixel sizes of the both the polyp and forceps are calculated as above, the actual size of the polyp, e.g., in millimeters (mm) is determined as follows. A size of the forceps in mm (Fmm) is available, e. g, based on the device model or actual measurement. A polyp size in mm (Dmm) is then calculated as follows and may be rounded to the whole mm: <MAT>.

Once a poly is detected during a medical procedure, a polyp size in mm is outputted, e.g., on a computer screen in real time with the video feed.

As described, the AI platform <NUM> is utilized for detecting and sizing lesions. Additionally, the described platform <NUM> can also be very useful in early detection other diseases as well. The classifier can be trained to detect diseases such as Inflammatory Bowel Disease (IBD). Inflammatory bowel disease is an umbrella term used to describe disorders that involve chronic inflammation of your digestive tract. Types of IBD include ulcerative colitis, which causes long-lasting inflammation and sores (ulcers) in the innermost lining of the large intestine (colon) and rectum. Another type of IBD is Crohn's disease, which is characterized by inflammation of the lining of the digestive tract, which often spreads deep into affected tissues. IBD can be debilitating and sometimes leads to life-threatening complications, and increases the risk of colon cancer.

The AI platform <NUM> can further be used in comparing the size (area) before and after the treatment of the inflammation of the lining of the digestive tract which may appear in patches in Crohn's disease or a continuous pattern in Ulcerative colitis as well as early detection of precancerous lesions in patients with Crohn's and Ulcerative colitis.

Additionally, a machine-learned model to automatically detect the location of the diseases is another way in which this AI platform <NUM> can be useful. Accurate detection of the location of the disorders can be beneficial for the surgeons who perform a surgical treatment of the diseases after once a clinical procedure has been performed.

Further, determining Gastroenterology (GI) quality metrics is another AI platform <NUM> application. This would allow automated measurement of several metrics that likely reflect the quality of a GI clinical procedure.

Further, the process can be used to analyze digitized image frames captured during colonoscopy procedure. Information like insertion time, withdrawal time, images at the time of maximal intubation, Cecal intubation time, and landmark identified, quality of bowel prep can be automatically measured with the described method. As these metrics can be obtained automatically, it will help to quantify health-care processes and can aid in providing high-quality health care.

The primary colonoscopy quality indicator is the adenoma detection rate (ADR), which is defined as the proportion of an endoscopist's screening colonoscopies in which one or more adenomas have been detected. To maximize adenoma detection, adequate mucosal inspection is required to ensure complete examination. Studies of tandem colonoscopies have revealed that physicians may miss adenomas larger than <NUM>. to ensure adequate adenoma detection, studies have suggested that a physician's withdrawal time, not including polyp resection, should be on average at least <NUM>-<NUM>. A paper published in <NUM> by the ASGE/ACG Taskforce on Quality in Endoscopy recommended that the withdrawal time should be measured as an aggregate of a physician's practice rather than based on an individual patient given the variation in anatomy such the prominence of colonic folds.

Another quality measure that affects adenoma detection is the quality of the bowel preparation. An adequate colon preparation is vital to ensure complete mucosal inspection. It has been reported that only three quarter of colonoscopies have an adequate colon preparation. High rates of missed adenomas and advanced neoplasia have been observed in patients with suboptimal colon preparations.

How should a physician ensure a quality colonoscopy for CRC screening in their practice? Despite some limitations, the ADR should be calculated for each endoscopist based on data from screening examinations. If a physician's ADR is lower than the benchmark of <NUM>%, quality improvement efforts are needed to increase this rate. Physicians should aim to achieve cecal intubation rates of <NUM>% or greater in screening colonoscopies. Techniques for mucosal examination with a focus on mean withdrawal time should be assessed. The quality of bowel preparation in the endoscopy practice should also be determined and optimized. Finally, in adequately prepped and carefully examined colons, compliance with screening and surveillance guidelines for future exams is strongly recommended. These steps will ensure that colonoscopy is maximally effective in preventing CRC.

<FIG> depicts an overview of an illustrative process of implementing a client-service model using the real time video analysis system <NUM> of <FIG>. On the client <NUM><NUM> (e.g., where the procedure is being performed), a video frame (image) is captured and is forwarded to a service <NUM> (e.g., a remote server). The service <NUM> receives the request, and searches for objects using a trained classifier. An XML result string is generated and is sent back to the client <NUM> in real time.

The client <NUM> receives the response and analyzes the XML string to determine if a polyp was found. If no, a next image is sent to the service <NUM>. If yes, the polyp is indicated as detected with the video feed (i.e., on a computer screen). Next a determination is made whether forceps were also found in the image. If no, a next image is sent to the service <NUM>. If yes, the size of the polyp is calculated and output with the video feed.

In a further embodiment, the AI platform may be employed in Video Capsule Endoscopy (VCE), which is a procedure that uses a tiny wireless camera to take pictures of the digestive tract. A capsule endoscopy camera sits inside a capsule which a patient swallows. As the capsule travels through the digestive tract, the camera takes thousands of pictures that get transmitted to a recorder that the patient wears, e.g., on a belt around their waist.

VCE helps doctors see inside the small intestine, an area that is hard to reach with more traditional endoscopy procedures. Traditional endoscopy involves passing a long, flexible tube equipped with a video camera down the throat or through the rectum.

The camera used in capsule endoscopy takes thousands of color photos as it passes through the digestive tract. The images saved on the recorder then gets transferred to a computer with special software that strings the images together to create a video.

Once the video gets created, the doctor watches the video to look for abnormalities within the digestive tract. The processes of watching the entire VCE of the small intestine takes about <NUM> minutes to <NUM> hour. In addition to using it in real time clinical procedure, the artificial intelligence (AI) platform can also be used to analyze these videos in order to detect lesions, which can then be evaluated by a clinician.

A method for detecting the lesion in a saved video, e.g., from a VCE procedure, using artificial intelligence (AI) may for example include the steps of: providing a trained classifier that includes a deep learning model trained to detect lesions; processing a video after it has been created using the transferred images from the recorder; and using the trained classifier to determine if a video frame from the video has a lesion.

It is understood that AI platform <NUM>, client <NUM>, and service <NUM> described herein may be implemented as a computer program product stored on a computer readable storage medium. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, and any suitable combination of the foregoing.

Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Python, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention.

A computing system may comprise any type of computing device and for example includes at least one processor, memory, an input/output (I/O) (e.g., one or more I/O interfaces and/or devices), and a communications pathway. In general, processor(s) execute program code which is at least partially fixed in memory. While executing program code, processor(s) can process data, which can result in reading and/or writing transformed data from/to memory and/or I/O for further processing. The pathway provides a communications link between each of the components in computing system. I/O can comprise one or more human I/O devices, which enable a user to interact with computing system. Computing system may also be implemented in a distributed manner such that different components reside in different physical locations.

Furthermore, it is understood that the AI platform <NUM> or relevant components thereof (such as an API component, agents, etc.) may also be automatically or semi-automatically deployed into a computer system by sending the components to a central server or a group of central servers. The components are then downloaded into a target computer that will execute the components. The components are then either detached to a directory or loaded into a directory that executes a program that detaches the components into a directory. Another alternative is to send the components directly to a directory on a client computer hard drive. When there are proxy servers, the process will select the proxy server code, determine on which computers to place the proxy servers' code, transmit the proxy server code, then install the proxy server code on the proxy computer. The components will be transmitted to the proxy server and then it will be stored on the proxy server.

Claim 1:
An artificial intelligence, AI, platform (<NUM>) for detecting and sizing a lesion in real time during a clinical procedure, comprising:
a trained classifier (<NUM>) that includes a deep learning model trained to detect lesions and reference objects in image data; and
a real time video analysis system (<NUM>) that receives a video feed (<NUM>) during a clinical procedure, uses the trained classifier (<NUM>) to determine if a video frame from the video feed (<NUM>) has both a lesion and a reference object, calculates an actual size of the lesion based on a pixel size of both the lesion and the reference object, and outputs an indication that the lesion was detected and the actual size of the lesion,
wherein the pixel size of the lesion is determined by:
capturing a rectangular cut out image containing the lesion;
enhancing the cut out image using image smoothing;
using an edge detection algorithm to detect contours of the lesion; and
calculating new rectangular coordinates of the lesion based on determined coordinates of the contours to provide a pixel size of the lesion, Dpx,
wherein the pixel size of the reference object is determined by:
capturing a rectangular cut out image containing the reference object;
enhancing the cut out image using image smoothing;
identifying edges of the reference object and imposing straight lines along the edges;
filtering the straight lines based on respective line lengths, line inclinations and coordinates of intersection of the straight lines with boundaries of the cut out image to identify a pair of intersecting lines; and
determining a pixel size of the reference object, Fpx, based on the pair of intersecting lines,
wherein the actual size of the lesion D is calculated as: <MAT> , wherein F is an actual size of the reference object.