MULTI-FRAME ANALYSIS FOR CLASSIFYING TARGET FEATURES IN MEDICAL VIDEOS

Methods, systems, and devices for classifying a target feature in a medical video are presented herein. Some methods may include the steps of: receiving a plurality of frames of the medical video, where the plurality of frames include the target feature; generating, by a first pretrained machine learning model, an embedding vector for each frame of the plurality of frames, each embedding vector having a predetermined number of values; and generating, by a second pretrained machine learning model, a classification of the target feature using the plurality of embedding vectors, where the second pretrained machine learning model analyzes the plurality of embedding vectors jointly.

TECHNICAL FIELD

The present disclosure relates generally to using deep learning models to classify target features in the medical videos.

BACKGROUND

Detecting and removing polyps in the colon is one of the most effective methods of preventing colon cancer. During a colonoscopy procedure, a physician will scan the colon for polyps. Upon finding a polyp, the physician must visually decide whether the polyp is at risk of becoming cancerous and should be removed. Certain types of polyps, including adenomas, have the potential to become cancer over time if allowed to grow while other types are unlikely to become cancer. Thus, correctly classifying these polyps is key to treating patients and preventing colon cancer.

By leveraging the power of artificial intelligence (AI), physicians may be able to identify and classify polyps more easily and accurately. AI is a powerful tool because it can analyze large amounts of data to learn how to make accurate predictions. However, to date, AI-driven algorithms have yet to meaningfully improve the ability of physicians to classify polyps. Therefore, improved AI-driven algorithms are needed to yield more accurate and useful classifications of polyps.

SUMMARY

Methods of classifying a target feature in a medical video by one or more computer systems are presented herein. The one or more computer systems may include a first pretrained machine learning model and a second pretrained learning model. Some methods may include the steps of receiving a plurality of frames of the medical video, where the plurality of frames includes the target feature; generating, by the first pretrained machine learning model, an embedding vector for each frame of the plurality of frames, each embedding vector having a predetermined number of values; and generating, by the second pretrained machine learning model, a classification of the target feature using the plurality of embedding vectors, where the second pretrained machine learning model analyzes the plurality of embedding vectors jointly.

In some embodiments, the first pretrained learning model may include a convolutional neural network and the second pretrained machine learning model may include a transformer. In some embodiments, the classification may include a score in a range of 0 to 1. In some embodiments, the classification may include one of: positive, negative, or uncertain. In some embodiments, the classification includes a textual representation. In some embodiments, the first pretrained machine learning model and the second pretrained machine learning model may be jointly trained. In some embodiments, the first pretrained machine learning model and the second pretrained machine learning model may be trained separately. In some embodiments, the medical video may be collected during a colonoscopy procedure using an endoscope and the target feature may be a polyp. In some embodiments, the classification may include one of: adenomatous and non-adenomatous. In some embodiments, the second pretrained machine learning model may analyze the plurality of embedding vectors without classifying each embedding vector individually.

Systems for classifying a target feature in a medical video are described herein. In some embodiments, the systems may include an input interface configured to receive a medical video, and a memory configured to store a plurality of processor-executable instructions. The memory may include an embedder based on a first pretrained machine learning model and a classifier based on a second pretrained machine learning model. The processor may be configured to execute the plurality of processor-executable instruction to perform operations including: receiving a plurality of frames of the medical video, where the plurality of frames includes the target feature; generating, with the embedder, an embedding vector for each frame of the plurality of frames, each embedding vector having a predetermined number of values; and generating, with the classifier, a classification of the target feature using the plurality of embedding vectors, where the classifier analyzes the plurality of embedding vectors jointly.

In some embodiments, the first pretrained machine learning model may include a convolutional neural network and the second pretrained machine learning model may include a transformer. In some embodiments, the classification may include a score in a range of 0 to 1. In some embodiments, the classification may include one of: positive, negative, or uncertain. In some embodiments, the classification may include a textual representation.

Non-transitory processor-readable storage mediums storing a plurality of processor-executable instructions for classifying a target feature in a medical video are described. The instructions may be executed by a processor to perform operations comprising: receiving a plurality of frames of the medical video, where the plurality of frames includes the target feature; generating, by a first pretrained machine learning model, an embedding vector for each frame of the plurality of frames, each embedding vector having a predetermined number of values; and generating, by a second pretrained machine learning model, a classification of the target feature using the plurality of embedding vectors, where the second pretrained machine learning model analyzes the plurality of embedding vectors jointly.

In some embodiments, the first pretrained machine learning model may include a convolutional neural network and the second pretrained machine learning model may comprise a transformer. In some embodiments, the classification may include a score in a range of 0 to 1. In some embodiments, the classification may include one of: positive, negative, or uncertain. In some embodiments, the classification may include a textual representation.

DETAILED DESCRIPTION

As used herein, the term “network” may comprise any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system and/or any training or learning models implemented thereon or therewith.

As used herein, the term “model” may comprise hardware or software-based framework that performs one or more functions. In some embodiments, the model may be implemented on one or more neural networks.

Many scientists, physicians, programmers and others have been working on harnessing the power of artificial intelligence (AI) to quickly and accurately diagnose diseases. AI has been used in a variety of different diagnostic applications including, for example, detecting the presence of polyps in colonoscopy videos. Some of the most promising ways of diagnosing diseases from medical videos include using a machine learning (ML) and, in particular, neural networks (NN). By inputting hundreds or thousands of frames of a target feature, ML programs can develop methods, equations, and/or patterns for determining how to classify the target feature in future frames. For example, if a ML program is fed thousands of frames where a physician has already classified the polyp, the ML program can use this labeled training data to learn what each type of polyp looks like and how to identify the types of polyps in future colonoscopy videos.

The present disclosure generally relates to improved methods, systems, and devices for classifying target features in frames of a medical video. In some embodiments, a target feature detector may be used to detect the target features in a medical video and identify a collection of frames in a time interval that includes each target feature. A joint classification model, including an embedder and a classifier, may then receive the frames of medical video and classify the target feature therein. The embedder may generate an embedding vector for each frame received by the joint classification model. The embedding vectors may be a computer-readable vector or matrix representing the frame. The classifier may then use the embedding vectors to generate a classification of the target feature. Preferably, the classifier may analyze all frames jointly and generate a single classification for all frames.

By jointly analyzing the frames, the classifier can leverage information in multiple frames to more accurately understand the target feature shown in the frames. For instance, when comparing all frames, there may be one or more frames that do not provide a good view or a high-quality picture of the target feature and in some cases may not show the target feature at all. Compared to other models which classify each frame individually and aggregate the individual classifications, the joint classification model is better able to recognize and give less weight to these low-quality frames or outliers. Therefore, the joint classification model may more accurately classify the target features than other classification models currently in use.

These descriptions are provided for example purposes only and should not be considered to limit the scope of the invention described herein. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.

FIG.1is a schematic diagram illustrating a computer system100for implementing a target feature detector140and a joint classification model150, according to some embodiments of the present disclosure. The computer system100includes a processor110coupled to a memory120. Although the computing device100is shown with only one processor110, it is understood that processor110may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and/or the like in the computing device100. The computing device100may be implemented as a stand-alone subsystem, as a board added to a computing device, and/or as a virtual machine. The memory120may be used to store software executed by computing device100and/or one or more data structures used during operation of computing device100. The memory120may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor (e.g., the processor110) or computer is adapted to read. In the present embodiments, for example, the memory120includes instructions suitable for training and/or using an image-to-image model130and/or a masking model140described herein.

The processor110and/or the memory120may be arranged in any suitable physical arrangement. In some embodiments, the processor110and/or the memory120are implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), and/or the like. In some embodiments, the processor110and/or the memory120include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, the processor110and/or the memory120may be located in one or more data centers and/or cloud computing facilities.

In some examples, the memory120may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., the processor110) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, the memory120includes instructions for a target feature detector140and a joint classification model150that may be used to implement and/or emulate the systems and models, and/or to implement any of the methods described further herein. In some embodiments, the target feature detector140may receive a medical video130and detect target features in one or more frames of the medical video130. In some embodiments, the target feature detector140identifies frames having a target feature and also identifies portions of the frames having the target feature. The joint classification model150may receive frames that include the detected target feature from the target feature detector140. The joint classification model150may include an embedder160and a classifier170. The embedder160may receive the frames of the detected target feature and generate an embedding vector for each frame, such that each frame has an associated embedding vector in one-to-one correspondence. The classifier170may then analyze the embedding vectors to classify the target feature and output the classification180.

FIG.2is a simplified diagram illustrating an example embodiment of a process200, according to one or more embodiments described herein. In the present embodiments, the process200describes aspects of using a target feature detector140and a joint classification model150incorporated in a computing device100for detecting and classifying target features of a medical video130. In the present disclosure, the medical video130may be a colonoscopy video collected using an endoscope. However, it is contemplated that the medical video130could be any other type of medical video including, for example, video captured during other endoscopic procedures, ultrasound procedures, magnetic resonance imaging (MRI) procedures, or any other medical procedure. The target feature detected in the medical videos130may be specific to that video. For example, the target feature may be a polyp in a colonoscopy video. In other examples, the target feature may be a cancerous tumor, a stenosis, or any other suitable target feature.

In the present embodiments, the medical video130is input into the target feature detector140. The target feature detector140may be configured to analyze the medical video130to detect target features. The target feature detector140may output frames210of the medical video130including one or more target features to the joint classification model150. In addition to outputting the frames210, the target feature detector140may also output a location of the target feature230to memory120or to a display. The embedder160may receive the frames210and generate embedding vectors220for each frame210. The classifier170may then receive the embedding vectors220from the embedder160and analyze the embedding vectors220to classify the target feature. The classifier170may then output the classification180.

The joint classification model150may include both the embedder160and the classifier170such that the models are jointly trained. However, the embedder160and classifier170may not be a joint classification model150and may instead be trained individually. In some embodiments, the embedder160may be jointly trained with the target feature detector140. In some embodiments, the medical video130may be input into the embedder160before it is input into the target feature detector140. The embedder160may then generate embedding vectors220for each frame of the medical video130. The target feature detector140may then receive embedding vectors220and detect target features therein. In these cases, the classifier170may receive embedding vectors220that include the target feature from the target feature detector140.

The target feature detector140may be implemented in any suitable way. In some embodiments, the target feature detector140may include a machine learning (ML) model and, in particular, may include a neural network (NN) model. For example, the target feature detector140may be an ML or NN based object detector. In some embodiments, the NN based target feature detector may be a two stage, proposal-driven mechanism such as a region-based convolutional neural network (R-CNN) framework. In some embodiments, the target feature detector140may use a RetinaNet architecture, as described in, for example, Lin et al.,Focal Loss for Dense Object Detection,arXiv: 1708.02002 (Feb. 7, 2018) or in U.S. Patent Publication No. 2021/0225511, the entirety of which are incorporated herein by reference.

The target feature detector140may output the location of the target features in any appropriate way. For example, the target feature detector140may output the location of the target feature. The location of the target feature may include coordinates. In some cases, the location of the target feature may be bounded by a box, circle or other object surrounding or highlighting the target features in the medical video130. The bounding box surrounding or highlighting the target feature is then combined with the medical video130such that, when displayed, the bounding box is displayed around target features in the medical video130.

Additionally, the target feature detector140may output frames210of the medical video130including the target feature. The frames210may include any number of frames. In some embodiments, the frames210including the target feature may be the total number of frames in the medical video130. In other embodiments, the frames210including the target feature may include less than the total number of frames in the medical video130. For example, the frames210including the target feature may include any number of frames in a range of 1 to 200. In particular embodiments, the frames210including the target feature may include 1, 5, 10, 15, 20, 25, 30, 35, 40, 45, or 50 frames.

The frames210including a target feature may be smaller than the frames of the medical video130. In some cases, the frames210including the target feature may be the portion of the frames of the medical video that are within a bounding box surrounding the target feature. In other cases, the frames210including the target feature may be the same size as the frames of the medical video130. In these cases, the joint classification detector150may only analyze the portion of the frames within the bounding box.

The embedder160receives the frames210including the target feature and generates an embedding vector220for each frame210. The embedding vector220may be a representation of the frame210that is computer readable. The embedder160may include an ML model such as a NN model. In some embodiments, the embedder160may use a convolutional NN (CNN).

The size of the embedding vectors220generated by the embedder160may be predetermined. The size of the embedding220may be determined in any suitable way. For example, the size may be determined through a hyper parameter search, which includes training several models, each with a different size, and choosing the size that produces the best outcomes. In other cases, the size of the embedding vector may be chosen based on other sizes known in the art that produce good outcomes. There may be a tradeoff when it comes to determining the size of the embedding vectors. As the vector size increases, the overall accuracy of the classification model is expected to increase. However, with vectors of a larger size, the models will also require more computing power and, thus, more time and cost. Therefore, the size that yields the best outcome may be a vector that is large enough to capture the necessary details for making accurate classifications while being small enough to minimize the computing power required. In some embodiments, the size of the vector may include 128 values.

The classifier170may receive the embedding vectors220from the embedder160. The classifier170may analyze each frame210individually. In this case, the classifier170generates a classification for each frame210then aggregates all of the classifications to generate an overall classification180for the frames. However, in some embodiments, the classifier170may jointly analyze all of the frames210including the target feature to generate a single classification180for the frames210. Analyzing multiple frames210jointly may be preferable to individually analyzing each frame210because when processing multiple frames210jointly leverages mutual information among the frames. Frames that are noisy outliers or are low-quality or include non-discriminative views of the target feature may generate an inaccurate classification (also known as a characterization) of the target feature. Thus, by jointly analyzing the frames210, frames with a low-quality rendering of the target feature (or with no target feature shown) can be compared to other frames with a better rendering of the target feature. The frames with a better rendering of the target feature can be given a higher weight and frames with a low-quality rendering of the target feature can be given a lower weight. On the contrary, when each frame is analyzed individually and the classifications are aggregated, the low-quality frames may be given an equal weight to the high-quality frames. Thus, this may generate a less accurate overall classification180. Therefore, analyzing all frames210jointly may generate more accurate classifications180than analyzing each frame210individually.

The classifier170may include an ML model such as a NN model. In some embodiments, the classifier170may include an attention model or a transformer. The transformer may be implemented in any suitable way. In some embodiments, the classifier170includes the self-attention based transformer as described in Vaswani et al.,Attention is All You Need,arXiv: 1706.03762 (Dec. 6, 2017) or Dosovitskiy et al.,An Image is Worth16×16Words: Transformers for Image Recognition at Scale,arXiv: 2010.11929 (Jun. 3, 2021), the entirety of which are incorporated herein by reference.

FIG.3is a simplified diagram illustrating an example transformer300architecture, according to one or more embodiments described herein. The transformer300includes multiple layers and sublayers, that analyze the embedding vectors220to generate a classification180of the target feature in the frames210. The transformer300may map a sequence of symbol representations (x1, . . . , xn) to a sequence of continuous representations z =(z1, . . . , zn). In some cases, each symbol representation x may be the embedding vector220for the current frame xior multiple embedding vectors220from the current and past frames (x1, . . . , xi). By processing multiple embedding vectors (corresponding to multiple frames) in parallel, the transformer300is able to leverage mutual information among frames. Then, the continuous representations may be mapped to an output that may represent a score reflecting the likelihood or probability of a polyp classification, such as whether a polyp is adenomatous or non-adenomatous, as described in more detail below. Each step of the transformer300may be auto-regressive such that the previously generated symbols for a frame are received as an input for generating symbols for the next frame. In some aspects, the transformer300may also be referred to as a transformer encoder in recognition that it is an encoder portion of some transformer architectures.

The transformer300may include any appropriate number of layers L. For example, the transformer300may include 2, 4, 6, 8, or 10 layers L. Each layer L may include two sublayers. The first sublayer330of the encoder layer L may be a multi-head self-attention mechanism, as described in more detail below. The second sublayer335of the encoder layer N may be a multilayer perceptron (MLP) such as a simple, position-wise fully connected feed-forward network, as described in more detail below. There may be a residual connection around each of the sublayers330,335followed by layer normalization. In some embodiments, the transformer300may have an MLP head that receives the output from the layers L. The input to the transformer300may be the embedding vector220for the current frame i or multiple embedding vectors from the current and past frames (x1, . . . , xi).

The output of each sublayer330,335may produce outputs of the same dimension dmodel. In some cases, embedding layers may be used before the transformer300. The output of the embedding layers may be the same dimension dmodelas the outputs of the sublayers330,335. In some cases, this dimension dmodelmay be 512.

The fully connected feed-forward network in sublayers335,350may be applied to each position separately and identically. In some embodiments, the feed-forward network may include two linear transformations with a ReLU activation between the linear transformations. The linear transformations may be the same across different positions, but may use different parameters from layer to layer.

FIG.4illustrates an example multi-head attention model400, according to some embodiments of the present disclosure. In some embodiments, the multi-head attention models400in sublayer330may be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output may be a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. In some embodiments, a single attention function with dmodel-dimensional keys, values, and queries may be performed. However, in some embodiments, it may be preferable to linearly project the keys, values, and queries a certain number of times (i.e. h times) with different, learned linear projections to dk, dv, and dkdimensions, respectively. On each projected version of keys, values, and queries, the attention function410may be performed in parallel, yielding dv-dimensional output values. These values are concatenated and linearly projected to yield the final output from the multi-head attention model400.

In some embodiments, the attention function410may be a scaled dot-product attention function500.FIG.5illustrates an example of scaled dot-product attention500, according to some embodiments of the present disclosure. The queries, keys, and values having dimensions dk, dk, and dv, respectively, may be input into the scaled dot-product attention function500. The dot product of the queries with all keys may be computed. The dot product may be scaled by dividing by √{square root over (dk)} and a softmax function may be applied to obtain the weights on the values.

In some embodiments, the attention function410may be applied to a set of queries Q simultaneously, which may be packed together into a matrix Q. The keys and values may also be packed together into a matrices K and V, respectively.

In other embodiments, the attention function410may be an unscaled dot-product attention function or an additive attention function. However, the scaled dot-product attention function500may be preferable because it can be implemented using highly optimized matrix multiplication code, which may be faster and more space-efficient.

The output of the classifier170may be a classification180indicating the type of target feature detected. In cases where the medical video130is a colonoscopy video and the target feature is a polyp, the classifier170may analyze the target feature to determine if it is adenomatous or non-adenomatous. If the polyp is adenomatous, it may be likely to become cancer in the future and thus may need to be removed. If the polyp is non-adenomatous, the polyp may not need to be removed. The classification180may be in any appropriate form. For example, when classifying polyps, the classification180may be a textual representation of the type of polyp for example the word “adenomatous” or “non-adenomatous.” The textual representation may include a suggestion of how to handle the polyp. Thus, the textual representation may be “remove” or “leave.” The textual representation may also include the word “uncertain” to indicate that an accurate prediction was not generated.

In another example, the classification180may be a score indicating whether the target feature detected is a certain type or is not a certain type. When the target feature is a polyp, the score may indicate whether the polyp is adenomatous or non-adenomatous. The score may be a value in a range of 0 to 1, where 0 indicates the polyp is non-adenomatous and 1 indicates the polyp is adenomatous. Values closer to 0 indicate the polyp is more likely to be non-adenomatous and values closer to 1 indicate that the polyp is more likely to be adenomatous. In some embodiments, the score values may only include 0 and 1 and may not include a range between 0 and 1. In some embodiments, both a score and a textual representation may be output from the classifier170. The textual representation may be based on a score, such that the score is compared to one or more threshold values to determine the textual representation. For example, if a score or value is less than a first threshold, the textual representation is one text string (e.g., “non-adenomatous”), and if a score or value is greater than a second threshold, the textual representation is a second text string (e.g., “adenomatous”), with the two thresholds between 0 and 1.

Although the above embodiments describe a target feature detector140being used in connection with the embedder160and the classifier170, in some embodiments, the embedder160and classifier170may be implemented without the target feature detector140. Instead, the embedder160may receive a set of frames including a target feature that were detected in any appropriate way. For instance, a physician may have identified frames that include a target feature and input only those frames into the embedder160. In some cases, the embedder160receives the medical video130directly and not a subset of frames including the target feature.

The disclosed method of implementing a target feature detector140, an embedder160, and a classifier170may be implemented using any appropriate hardware. For example,FIG.6shows a block diagram of a system600for implementing one or more of the methods described herein, according to some aspects of the present disclosure. The system600includes a medical device610, a computer system620, and a display630. The medical device610may be any medical device capable of collecting a medical video130. In some embodiments, the medical device610is an endoscope. The endoscope may be used during a colonoscopy to view the colon of a patient and collect a medical video130. The target features in the colon may be, for example, polyps.

The medical video130collected by the medical device610may be sent to a computer system620. In some embodiments, the medical device610may be coupled to the computer system620via a wire and the computer system620may receive the medical video130over the wire. In other cases, the medical device610may be separate from the computer system620and may be sent to the computer system620via a wireless network or wireless connection. The computer system620may be the computer system100shown and described in reference toFIG.1. The computer system620may be a single computer or may be multiple computers.

The computer system620may include a processor-readable set of instructions that can implement any of the methods described herein. For example, the computer system620may include instructions including one or more of a target feature detector140, an embedder160, and a classifier170, where the embedder160and classifier170may be implemented as a joint classification model150.

The computer system620may be coupled to a display630.FIG.7illustrates an example display, according to some embodiments of the present disclosure. In the illustrated embodiment, the medical video130is a colonoscopy video collected from an endoscope and the target feature is a polyp. However, any suitable medical video130may be used and any target feature may be detected therein.

The computer system620may output the medical video130received from the medical device610to the display630. In some cases, the medical device610may be coupled to or in communication with the display630such that the medical video130is output directly from the medical device610to the display630.

A target feature detector140implemented on the computer system620may output a bounding box710identifying a location of a detected target feature. In some embodiments, the computer system620may combine the bounding box710and the medical video130and output the medical video130including the bounding box710to the display630. Thus, the display630may show the medical video130with a bounding box710around a detected target feature so that the physician can see where a target feature may be located.

The target feature detector140may also output frames210including the target feature to the embedder160and the classifier170, which may be implemented as a joint classification model150. The joint classification model150may analyze the frames210to generate a classification180of the target feature, as described above. The classification180may be output to the display630. As described above, the classification180may be in any appropriate form including a textual representation and/or a score. When the classification180is displayed, the classification180may be different colors depending on the type of target feature. For example, when the target feature is a polyp, the classification180may be green if the polyp is likely non-adenomatous and may be red if the polyp is likely adenomatous. A sound may play when a classification180is made or when the type of target feature may require action on the part of the physician. For example, if the polyp is likely adenomatous and should be removed, a sound may play so that the physician knows that she may need to resect the polyp.

In some embodiments, the medical video130collected by the medical device610may be sent to the computer system620as it is collected. In other words, the medical video130analyzed by the computer system620and displayed on the display630may be a live medical video130taken during the medical procedure. Thus, the classification180can be generated and displayed in real-time so that the physician can view the information during the procedure and make decisions about treatment if necessary. In other embodiments, the medical video130is recorded by the medical device610and sent to or analyzed by the computer system620after the procedure is complete. Thus, the physician can review the classifications180generated at a later time. In some cases, the medical video130can be displayed and analyzed in real-time and can be stored for later viewing.

The target feature detector140, the embedder160, and classifier170may be trained in any suitable way. As described above, the embedder160and classifier170may be implemented as a joint classification model150such that the embedder160and classifier170are jointly trained. However, the embedder160and classifier170may not be a joint classification model150and may instead be trained individually. In some embodiments, the embedder160may be jointly trained with the target feature detector140. In some embodiments, the medical video130may be input into the embedder160before it is input into the target feature detector140. The embedder160may then generate embedding vectors220for each frame of the medical video130. The target feature detector140may then receive embedding vectors220and detect target features therein.

FIG.8is a flow diagram illustrating a method800of training the models, according to some aspects of the present disclosure. In the illustrated embodiment, the embedder160and the classifier170are implemented as a joint classification model150and, thus, are trained jointly. In the embodiments described herein, the target feature detector140is trained separately according to any suitable process known in the art. However, it is contemplated that the target feature detector140may be trained jointly with one or both of the embedder160or the classifier170. In other cases, a target feature detector140may not be implemented with the embedder160and classifier170.

Step802of the method800includes receiving a plurality of frames210of a medical video130comprising a target feature and classifications of each target feature by a physician. In some embodiments, the medical video130is a colonoscopy video and the target feature is a polyp. Thus, the physician classifying the polyp may be a gastroenterologist. In these cases, the gastroenterologist classifies the polyps as adenomatous and non-adenomatous based on a visual inspection of the medical video130. The gastroenterologist may also classify the polyp based on whether she would remove or leave the polyp. When a gastroenterologist classifies the polyp, the classification may not be a diagnosis. Instead, it may be a classification indicating the likelihood that the polyp is a certain type and whether the gastroenterologist determines that the polyp should be removed.

In some embodiments, the physician classifying the target feature in a medical video is a pathologist. In this case, the classification of the target feature is a diagnosis of that target feature. In some cases, the pathologist may classify the target feature based on a visual inspection of the medical video130. In other cases, the pathologist may receive a biopsy of the target feature in the medical video130and classify the target feature based on the biopsy. Thus, in cases where the target feature is a polyp, the pathologist may analyze the biopsy to diagnose the polyp in the colonoscopy video as adenomatous or non-adenomatous.

Step804of the method800may include generating an embedding vector220for each frame210of the medical video130using an embedder160. As described above, the embedder160of the joint classification model150may receive the frames210and generate embedding vectors220, where each embedding vector220is a computer-readable representation of the corresponding frame210.

Step806of the method800may include generating a classification180of the target feature based on the embedding vectors220using a classifier170. As described above, the classifier170may jointly analyze the embedding vectors220to generate a single classification180for the target feature in the frames210. The classifier170may be implemented in any suitable way and the classification180may be in any suitable form, as described above.

Step808of the method800may include comparing the classification180to the physician's classification of the target feature. The classification180may be a textual representation of the target feature indicating the type. For example, when the target feature is a polyp, the classification may be “adenomatous” or “non-adenomatous.” In the training data, the physician may indicate whether the polyp is adenomatous or non-adenomatous. Thus, the classification180of the target feature is the same classification as the physician or a different classification. The accuracy of the classification may include calculating the percentage of correct scores. In cases where the target feature is a polyp, the positive probability value (PPV) may be calculated, which corresponds to the error in classifying a polyp as adenomatous when the physician classified the polyp as non-adenomatous. The negative probability value (NPV) may be calculated, corresponding to classifying a polyp as non-adenomatous when the physician classified the polyp as adenomatous.

In some embodiments, the classification180may be a score indicating the likelihood that a target feature is one type or another. As described above, for cases where the target feature is a polyp, the polyp may be given a score in a range of 0 to 1 by the classifier170. A score of 1 may indicate the polyp is adenomatous and a score of 0 may indicate the polyp is non-adenomatous. In some embodiments, the Area Under the Receiver Operating Characteristic Curve (ROC AUC or AUC) may be calculated. The PPV and NPV may also be calculated for the scores generated by the classifier170. In some cases, the training data may include a note of the likelihood the physician would score the polyp on a scale of 0 to 1. Thus, the score generated by the classifier170can be compared to the numerical value determined by the physician. However, in other cases, the training data may simply indicate whether the polyp is adenomatous (1) or non-adenomatous (0). Thus, the score may be compared to this classification in several suitable ways. For example, the score can be marked as correct if the score is closer to the correct value than the incorrect value. In other words, if a physician marks the polyp as adenomatous and the score is 0.7, the classification180is viewed as correct because it is above 0.5. On the other hand, if a physician marks the polyp as non-adenomatous and the score is 0.7, the classification180is viewed as incorrect because it is not below 0.5. Thus, the error can be calculated similarly to if the classification180by the classifier170is not based on a score. In other embodiments, the error may be calculated based on how far the score was from a perfect score. For example, if a physician marks the polyp as adenomatous and the score is 0.7, the classification180is as being off by 0.3 because the correct score is a 1. On the other hand, if a physician marks the polyp as adenomatous and the score is 0.9, the classification180is viewed as being off by 0.1. In other words, the error may be calculated based on the difference between the score and the correct classification.

Step810of the method800includes updating the embedder160and the classifier170based on the comparison. The joint classification model150, including the embedder160and the classifier170, may then be updated in any suitable way to generate a classification180that approaches the classification by the physician. In some embodiments, the joint classification model150may be updated based on one or more of the error, accuracy, AUC, PPV, or NPV. Step810may be based on gradient based optimization to decrease the error, such as the one described in Kingma et al., Adam: A Method for Stochastic Optimization, arXiv: 1412.6980 (Jan. 30, 2017), the entirety of which is incorporated herein by reference. However, other optimization methods can be used.

FIG.9is a flow diagram illustrating a method of operating a target feature detector140, an embedder160, and a classifier170during the inference stage. Step902includes receiving a medical video130. As described above, the medical video130may be any suitable medical video and may include one or more target features. In some embodiments, the medical video130may be a colonoscopy video collected from an endoscope and the target feature may be a polyp.

Step904of method900may include detecting a target feature in the medical video130using a pretrained target feature detector140. The pretrained target feature detector140may receive the medical video130and detect the target features therein and may be implemented in any suitable way, as described above.

Step906of the method900may include generating a plurality of frames210comprising the target feature. As described above, the target feature detector140may generate a series of frames210that include the target feature. These frames210may include all of the medical video130or only some frames of the medical video130and may be the same size as the frames of the medical video130or may be a smaller size.

Step908of the method900may include generating an embedding vector220for each frame of the generated frames210using a pretrained embedder160. As described above, the embedder160of the joint classification model150may receive the frames210and generate embedding vectors220, where each embedding vector220is a computer-readable representation of the corresponding frame210. The embedder160may be trained in any suitable way, including the embodiments described in reference toFIG.8.

Step910of the method900may include generating a classification180of the target feature based on the embedding vectors220using a pretrained classifier170. As described above, the classifier170may jointly analyze the embedding vectors220to generate a single classification180for the target feature in the frames210. The classifier170may be implemented in any suitable way and the classification180may be in any suitable form, as described above.

Step912of the method900may include displaying the classification180of the target feature. The classification180may be displayed on a display630in any suitable way, as described above.

An experiment was conducted in which a joint classification model was implemented according to some embodiments of the present disclosure and three aggregation models were implemented according to different prior art schemes. All models were used to classify polyps in frames of colonoscopy videos. All models output a score in a range of 0 to 1, where 0 indicates the polyp is non-adenomatous and 1 indicates the polyp is adenomatous.

As described above, the joint classification model includes an embedder and a classifier, which are jointly trained. The classifier jointly analyzes the frames including the target feature to generate a single classification. The aggregation models generate a score for each frame individually, then aggregate the scores to calculate an overall classification score. The aggregation for the aggregation models was conducted in three different ways. First, the mean score aggregation model aggregates the classifications by calculating the mean value of the classifications. Second, the maximum score aggregation model aggregates the classifications by using the maximum score of the classifications as the overall classification score. Third, the minority voting aggregation model aggregates the classifications by minority voting.

In some embodiments, the aggregation models may use the same base embedder and classifiers as the joint classification model. However, for the aggregation models, the classifier classifies each frame individually unlike for the joint classification models where all frames are classified jointly.

FIGS.10-12show various graphs and charts comparing the performance of the joint classification model to the three different aggregation models.

FIG.10is a graph1000of the AUC versus the number of frames (or sequence length) for each model. As shown, the joint classification model1040has a higher AUC than any of the aggregation models1010,1020,1030for all numbers of frames. The minority voting aggregation model1030has the lowest AUC for all number of frames. The max score aggregation model1020has a slightly higher AUC than the mean score aggregation model1010. Thus, none of the aggregation models1010,1020,1030perform as well as the joint classification model1040.

FIG.11is a graph1100of the PPV versus the PPV for each model. The mean score aggregation model1110and the minority voting aggregation model1130have similarly PPV values across NPV values. The maximum score aggregation model1120has slightly lower PPV values across NPV values than the other aggregation models1110,1130. The PPV of the joint classification model1140is higher for all NPV values as compared to the three aggregation models1110,1120,1130. In particular, at high NPV values, the joint classification model1140significantly outperforms the aggregation models1110,1120,1130. This indicates that the joint classification model is particularly better at predicting that the polyp is non-adenomatous and is unlikely to develop cancer if left in the colon.

FIG.12is a chart illustrating why the joint classification model is better able to classify polyps. Each row of photos contains 10 frames of a colonoscopy video including a polyp in at least one frame. The score above each frame is a score calculated by the base model for the individual frame below it. The scores of the individual frames were aggregated and the aggregated score is shown on the left of the row. For the top row and the middle row, the individual scores were aggregated according to mean score aggregation. For the bottom row, the individual scores were aggregated by maximum score aggregation. The joint classification model score for the frames in the row is shown on the right of the row. The joint classification score is generated by jointly analyzing all frames in the row, as described herein. For the top row and the middle row, the joint classification score correctly classified the polyp as adenomatous and the aggregated score incorrectly classified the polyp as non-adenomatous. For the bottom row, the joint classification score correctly classified the polyp as non-adenomatous and the aggregated score incorrectly classified the polyp as adenomatous. Because the joint classification model compares the frames to each other, the joint classification model may give a lower weight to lower-quality or outlier frames that yield a less accurate result. On the other hand, the aggregation models may not identify the lower-quality or outlier frames and may weight these equally to higher-quality frames when aggregating the values. Thus, the joint classification model may generate a more accurate classification than aggregation models.

A number of variations are possible on the examples and embodiments described above. Accordingly, the logical operations making up the embodiments of the technology described herein are referred to variously as operations, steps, objects, elements, components, layers, modules, or otherwise. Furthermore, it should be understood that these may occur in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.

Generally, any creation, storage, processing, and/or exchange of user data associated with the method, apparatus, and/or system disclosed herein is configured to comply with a variety of privacy settings and security protocols and prevailing data regulations, consistent with treating confidentiality and integrity of user data as an important matter. For example, the apparatus and/or the system may include a module that implements information security controls to comply with a number of standards and/or other agreements. In some embodiments, the module receives a privacy setting selection from the user and implements controls to comply with the selected privacy setting. In some embodiments, the module identifies data that is considered sensitive, encrypts data according to any appropriate and well-known method in the art, replaces sensitive data with codes to pseudonymize the data, and otherwise ensures compliance with selected privacy settings and data security requirements and regulations.

In several example embodiments, the elements and teachings of the various illustrative example embodiments may be combined in whole or in part in some or all of the illustrative example embodiments. In addition, one or more of the elements and teachings of the various illustrative example embodiments may be omitted, at least in part, and/or combined, at least in part, with one or more of the other elements and teachings of the various illustrative embodiments.

Any spatial references such as, for example, “upper,” “lower,” “above,” “below,” “between,” “bottom,” “vertical,” “horizontal,” “angular,” “upwards,” “downwards,” “side-to-side,” “left-to-right,” “right-to-left,” “top-to-bottom,” “bottom-to-top,” “top,” “bottom,” “bottom-up,” “top-down,” etc., are for the purpose of illustration only and do not limit the specific orientation or location of the structure described above. Connection references, such as “attached,” “coupled,” “connected,” and “joined” are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily imply that two elements are directly connected and in fixed relation to each other. The term “or” shall be interpreted to mean “and/or” rather than “exclusive or.” Unless otherwise noted in the claims, stated values shall be interpreted as illustrative only and shall not be taken to be limiting.

Additionally, the phrase “at least one of A and B” should be understood to mean “A, B, or both A and B.” The phrase “one or more of the following: A, B, and C” should be understood to mean “A, B, C, A and B, B and C, A and C, or all three of A, B, and C.” The phrase “one or more of A, B, and C” should be understood to mean “A, B, C, A and B, B and C, A and C, or all three of A, B, and C.”

Although several example embodiments have been described in detail above, the embodiments described are examples only and are not limiting, and those skilled in the art will readily appreciate that many other modifications, changes, and/or substitutions are possible in the example embodiments without materially departing from the novel teachings and advantages of the present disclosure. Accordingly, all such modifications, changes, and/or substitutions are intended to be included within the scope of this disclosure as defined in the following claims.