Patent ID: 12211202

DETAILED DESCRIPTION

The following detailed description is merely illustrative and is not intended to limit embodiments and/or application or uses of embodiments. Furthermore, there is no intention to be bound by any expressed or implied information presented in the preceding Background section, Summary section or in the Detailed Description section.

The disclosed subject matter is directed to systems, computer-implemented methods, apparatus and/or computer program products that provide a self-supervised representation learning paradigm for medical images. In one or more embodiments, the disclosed techniques employ a self-supervised learning process to train a transformer network to automatically learn representations of medical images included in training set using paired similar image versions for each of the medical images. These paired alternate image versions are feed into parallel arms of a distillation model, wherein respective arms of the distillation model include teacher and student instances of the same transformer network. The result of this self-supervised learning process includes feature representations for each of the training images, as well as a trained transformer network capable of accurately generating new feature representations for new medical images included in training image domain/feature space. These training image feature representations can be stored and used for automating other tasks, including similar image retrieval, image classification, and other medical image processing tasks.

The disclosed techniques further provide novel mechanism to automatically generate or select the paired similar image versions (e.g., without manual involvement) for training the transformer network in a manner that is tailored based on domain knowledge associated with the medical images. In this regard, rather than generating an alternate version of a training medical image by randomly cropping, rotating, adjusting the resolution, etc., the disclosed techniques learn and define image augmentation schema that controls the alternate image generation/selection process in a manner that is specific to medical images, and/or specific to the types of the medical images, the anatomical region/regions depicted, the acquisition parameters/protocols used, the patient demography, the patient pathology, and clinical and non-clinical features associated with the medical images. In some embodiments, the disclosed methods further iteratively refine the medical image augmentation strategies based on the learned training image representations over the training epochs.

The terms “algorithm” and “model” are used herein interchangeably unless context warrants particular distinction amongst the terms. The term “clinical inferencing model” is used herein to refer to a AI/ML model configured to perform a clinical decision/processing on clinical data. The clinical decision/processing task can vary. For example, the clinical decision/processing tasks can include classification tasks (e.g., disease classification/diagnosis), disease progression/quantification tasks, organ segmentation tasks, anomaly detection tasks, image reconstruction tasks, and so on. The clinical inferencing models can employ various types of ML algorithms, including (but not limited to): deep learning models, neural network models, deep neural network models (DNNs), convolutional neural network models (CNNs), generative adversarial neural network models (GANs), long short-term memory models (LSTMs), attention-based models, transformers and the like.

As used herein, a “medical imaging inferencing model” refers to an image inferencing model that is tailored to perform an image processing/analysis task on one or more medical images. For example, the medical imaging processing/analysis task can include (but is not limited to): image reconstruction, image enhancement, scan series characteristic classification, disease/condition classification, disease region segmentation, organ segmentation, disease quantification, disease/condition staging, risk prediction, temporal analysis, anomaly detection, anatomical feature characterization, medical image reconstruction, and the like. The terms “medical image inferencing model,” “medical image processing model,” “medical image analysis model,” and the like are used herein interchangeably unless context warrants particular distinction amongst the terms.

The term “image-based inference output” is used herein to refer to the determination or prediction that an image processing model is configured to generate. For example, the image-based inference output can include a segmentation mask, a reconstructed image, an enhanced image, an adapted image, an annotated image, a classification, a value, or the like. The image-based inference output will vary based on the type of the model and the particular task that the model is configured to perform. The image-based inference output can include a data object that can be rendered (e.g., a visual data object), stored, used as input for another processing task, or the like. The terms “image-based inference output”, “inference output” “inference outcome,” “inference result” “inference”, “output”, “outcome,” “predication”, and the like, are used herein interchangeably unless context warrants particular distinction amongst the terms. The outputs can be in different formats, such as for example: a Digital Imaging and Communications in Medicine (DICOM) structured report (SR), a DICOM secondary capture, a DICOM parametric map, an image, text, and/or JavaScript Object Notation (JSON).

The types of medical images processed/analyzed by the disclosed transformer networks, feature matching algorithms, classification models, and other medical image inferencing models described herein can include images captured using various types of image capture modalities. For example, the medical images can include (but are not limited to): radiation therapy (RT) images, X-ray (XR) images, digital radiography (DX) X-ray images, X-ray angiography (XA) images, panoramic X-ray (PX) images, computerized tomography (CT) images, mammography (MG) images (including a tomosynthesis device), a magnetic resonance imaging (MRI) images, ultrasound (US) images, color flow doppler (CD) images, position emission tomography (PET) images, single-photon emissions computed tomography (SPECT) images, nuclear medicine (NM) images, and the like. The medical images can also include synthetic versions of native medical images such as augmented, modified or enhanced versions of native medical images, augmented versions of native medical images, and the like generated using one or more image processing techniques. In this regard, the term “native” image is used herein to refer to an image in its original capture form and/or its received form prior to processing by the disclosed systems. The term “synthetic” image is used herein to distinguish from native images and refers to an image generated or derived from a native image using one or more image augmentation processing techniques (e.g., cropping, rotating, and/or otherwise adjusting the visual appearance of the image). The medical imaging processing models disclosed herein can also be configured to process 3D images.

A “capture modality” as used herein refers to the specific technical mode in which an image or image data is captured using one or more machines or devices. In this regard, as applied to medical imaging, different capture modalities can include but are not limited to: a 2D capture modality, a 3D capture modality, an RT capture modality, a XR capture modality, a DX capture modality, a XA capture modality, a PX capture modality a CT, a MG capture modality, a MRI capture modality, a US capture modality, a CD capture modality, a PET capture modality, a SPECT capture modality, a NM capture modality, and the like.

One or more embodiments are now described with reference to the drawings, wherein like referenced numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a more thorough understanding of the one or more embodiments. It is evident, however, in various cases, that the one or more embodiments can be practiced without these specific details.

FIG.1presents an example system100that facilitates learning feature representations of medical images using a self-supervised learning paradigm in accordance with one or more embodiments of the disclosed subject matter. Embodiments of systems described herein can include one or more machine-executable components embodied within one or more machines (e.g., embodied in one or more computer-readable storage media associated with one or more machines). Such components, when executed by the one or more machines (e.g., processors, computers, computing devices, virtual machines, etc.) can cause the one or more machines to perform the operations described.

In this regard, system100includes a feature learning module102that corresponds to and includes several computer executable components. These computer executable components include paired view generation component104, selection component106, augmentation component108, region of interest component110, training component114, transformer network116and indexing component118. These computer/machine executable components (and other described herein) can be stored in memory associated with the one or more machines. The memory can further be operatively coupled to at least one processor, such that the components can be executed by the at least one processor to perform the operations described. For example, in some embodiments, these computer/machine executable components can be stored in memory122which can be coupled to processing unit120for execution thereof. Examples of said and memory and processor/processing unit as well as other suitable computer or computing-based elements, can be found with reference toFIG.26, and can be used in connection with implementing one or more of the systems or components shown and described in connection withFIG.1or other figures disclosed herein.

System100further includes various data structures (e.g., data stores, databases, data files, and the like) that provide information used by the feature learning module102and/or aggregate information generated by the feature learning module102. This information includes medical image data128, medical image domain knowledge information130, paired view generation schema112and indexed image-feature data134. The type of data structures and/or data sources in which this information is stored and located can vary. The feature learning module102can be operatively and communicatively coupled to these data structures either directly or via one or more wired or wireless communication networks (e.g., the Internet, an Intranet, etc.).

The deployment architecture of system100(and other systems described herein) can vary. In some embodiments, the feature learning module102can be deployed at and executed by a single computing device (e.g., real or virtual) operatively coupled to the processing unit120and the memory122. With these embodiments, one or more of the various data structures including the medical image data128, the medical image domain knowledge information130, the paired view generation schema112and the indexed image-feature data134may also be stored locally on the same computing device and/or remotely at another device and accessed by the computing device via a wired or wireless communication network. In other embodiments, one or more components of the feature learning models104can be deployed at two or more separate communicatively coupled computing devices operating in a distributed computing environment. The separate computing devices can be communicatively coupled via one or more wired or wireless communication networks. With these embodiments, one or more of the various data structures including the medical image data128, the medical image domain knowledge information130, the paired view generation schema112and the indexed image-feature data134may also be stored locally on the same computing device and/or remotely at another device and accessed by the components of the feature learning module102via a wired or wireless communication network. Various alternative deployment architecture variations can also be used.

The feature learning module102provides tools for automatically learning feature representations for a set of medical images using a self-supervised machine learning process. In the embodiment shown, this set of medical images is represented as training images126that are provided in the medical image data128. In this regard, the medical image data128may include one or more data stores or databases that include a repository of medical images. The type or types of medical images included in the training images126dataset can vary. For example, the medical images in the set can include images depicting same or different anatomical regions and patients, captured using same or different imaging modalities (e.g., XR, PT, MRI, CT, etc.), and/or captured using same or different acquisition parameters/protocols. The source of the medical images can also vary. For instance, a data store comprising the medical image data128can include or correspond to a local or remote medical image data store, such as for example, internal databases, third party databases, workstations, Picture Archiving and Communication Systems (PACS) and consoles, and so forth. The set of medical images can also be extracted from a plurality of different data sources and/or medical image provider systems (e.g., hospital systems, imaging systems, and so on). In this regard, the medical image data128can be centrally represented while providing the ability to move across the organization between various types of endpoints and storages that can be implemented in a seamless manner.

The medical images included in the medical image data128may also include or be associated with a variety of rich metadata (e.g., in the form of metadata tags of the like) describing known attributes associated with the respective images. The specific metadata that may or may not be associated with the images can vary. For example, the metadata may also include information identifying or indicating the type of imaging study that was performed (e.g., modality, anatomical region or regions scanned), the image series to which each image belongs, the timing of the study, the location of the study and so on. The acquisition parameters/protocol information can vary depending on the modality of the imaging study performed. Some example acquisition parameters/protocol information may include (but is not limited to): contrast phase, imaging frequency, reconstruction kernel size, slice thickness, radiation dose, view of an XR image, MR sequence, capture resolution, voxel size, scan prescription plane, scan region, bounding box, and scan time. The metadata may also include information describing known visual properties of the medical images such as image quality, image resolution, signal noise, pixel spacing, imaging depth, image capture position/orientation (e.g., including the relative scan position to other scan images generated for a medical image series), relative position/dimension of anatomical landmarks, and so on. The metadata may also include information describing known attributes of the patient/subject represented in the medical image (e.g., patient identifiers, patient demographic information, body mass index (BMI), medical history, pathology, etc.). The medical image data128can further structure the metadata into a standardized format (e.g., a Digital Imaging and Communications in Medicine (DICOM) standard format) to facilitate searchable results.

The feature representations learned/generated for the training images126by the feature learning module102are represented in system100as training image feature representations132. These feature representations can correspond to feature vectors or digital fingerprints for each of the training images that provide a reduced dimensionality representation of the most important image pixel features included in each of the images. Each feature representation thus uniquely captures the semantic content of the respective training images. These training image feature representations132can be stored in suitable indexed data structure along with their corresponding training images126and/or information identifying their corresponding training images (e.g., an image identifier and/or file source) and used to facilitate downstream tasks such as image retrieval, image classification and other tasks, as described in greater detail with reference toFIGS.9-25).

As noted above, the feature learning module102can automatically learn and generate the training image feature representations132with no or minimal manual involvement using a self-supervised representation learning process performed by the training component114. This self-supervised representation learning process involves training a transformer network116to learn the feature representation for each training image included in the set of training images126using one or more alternate versions of the training images to drive the learning. One piece of particular importance that drives the automation and accuracy of this this self-supervised training process as applied to medical images is the ability of the automatically generate these alternate image versions in a meaningful way that ensures the relevant semantic content of the native training image is represented in the one or more alternate image versions. As described in greater detail below, to facilitate this end, the paired view generation component104can employ one or more image augmentation techniques and/or one or more image selection techniques tailored based on domain knowledge (e.g., provided by the medical image domain knowledge information130) associated with the respective medical images to generate these meaningful alternate images.

The transformer network116comprises a deep learning neural network model architecture generally referred to as a transformer, that has been adapted for generating image feature representations. In this regard, the transformer network116can include or correspond to a vision transformer. Transformers have recently emerged as an alternative to convolutional neural networks (CNNs or convnets) for visual recognition. Their adoption has been coupled with a training strategy inspired by natural language processing (NLP), that is, pretraining on large quantities of data and finetuning on the target dataset. The transformer network116adopts the mechanism of attention, differentially weighing the significance of each part of the input data. Like recurrent neural networks (RNNs), transformers are designed to handle sequential input data, such as natural language, for tasks such as translation and text summarization. However, unlike RNNs, transformers do not necessarily process the data in order. Rather, the attention mechanism provides the context for any position in the input sequence.

The transformer network116adopts an encoder-decoder architecture. The encoder consists of encoding layers that process the input iteratively one layer after another, while the decoder consists of decoding layers that do the same thing to the encoder's output. The function of each encoder layer is to generate encodings that contain information about which parts of the inputs are relevant to each other. It passes its encodings to the next encoder layer as inputs. Each decoder layer does the opposite, taking all the encodings and using their incorporated contextual information to generate an output sequence. To achieve this, each encoder and decoder layer makes use of an attention mechanism. For each input, attention weighs the relevance of every other input and draws from them to produce the output. Each decoder layer has an additional attention mechanism that draws information from the outputs of previous decoders, before the decoder layer draws information from the encodings. Both the encoder and decoder layers have a feed-forward neural network for additional processing of the outputs and contains residual connections and layer normalization steps.

FIG.2presents an example vision transformer (ViT) network200that can be used as the transformer network116to generate feature representations for medical images. The ViT network200takes an input image202and image is cut into several different patches204. These patches are linear projections that contain different portions of the input image202. The patches and the relative positions of the patches are fed into the transformer encoder208which learns patch feature representations210for each of the patches as well as an intermediate a representation for the entire input image. To facilitate this end, the ViT employs a plurality of different attention heads206that respectively identify the regions in the respective patches that contain the most important/influential features to be fed into the transformer encoder208. In this regard, the attention heads206employ a mechanism to identify the regions in the patches204that are most influential in getting the relevant features to generate the patch feature representations210. These patch features representations and the intermediate representation for the entire image are then normalized by a normalization layer212to generate a final feature representation214for the input image202. The final feature representation214is thus a concatenation of the features extracted by each of the attention heads206. This final feature representation214can be stored and can be used for different tasks, such as image retrieval, image classification, and other tasks.

FIG.3illustrates an example self-supervised representation learning process300for medical images using parallel transformer networks in accordance with one or more embodiments of the disclosed subject matter. With reference toFIGS.1and3, process300provides an example self-supervised training process that can be performed by the training component14and the paired view generation component104to learn and generate the training image feature representations132. In accordance with processes300, the training component114trains the transformer network116to generate the feature representations for each of the training images126by creating two instances of the transformer network116, one referred to as the student transformer network308and another referred to as the teacher transformer network310. The student transformer network308and the teacher transformer network310employ identical network architectures. In various embodiments, both the student and the teacher transformer networks correspond to the ViT network200illustrated inFIG.2.

In accordance with the self-supervised training process300, the training component114trains the network with the objective of presenting two different views of the same input image302image with the assumption that they should have similar feature representations. To facilitate this end, at304, the paired view generation component104generates paired alternate image versions305of the input image302. As described in greater detail below, the alternate image or images for the input image302can include augmented versions of the input image302generated by adapting the appearance of the input image using one or more defined augmentation mechanisms tailored based on domain knowledge associated with the input image. These augmentation mechanisms can include for example, cropping, changing visual properties (e.g., resolution, brightness, etc.), changing the orientation of the input image, and so on, to create a synthetic version of the input image302. Additionally, or alternatively, these augmentation mechanisms can include selecting a separate native medical image from the medical image data128that is similar to the input image302using metadata associated with the respective images. For example, in implementations in which the input image includes a scan image included in a series (e.g., a CT scan series, an MRI scan series, etc.), the separate image may include another image in the same scan series that neighbors the input image302(e.g., captured at a different point along the same acquisition axis in the z-direction or another direction).

In this example, the paired alternate image versions305include one alternate image version306which is paired with the input image302in its original form. In other implementation, both of the paired alternate image versions305can include different versions of the input image302. Each of the paired images302and306are then respectively fed into the student transformer network308and the teacher transformer network310, wherein the respective networks generate feature representations for each of the images in parallel (e.g., parallel arms of a distillation model). In this example, feature representation312corresponds to a feature representation generated by the student transformer network308for the alternat image302, and feature representation314corresponds to a feature representation generated by the teacher transformer network310for the input image302. The self-supervised aspect of this training process is based on comparing the feature representation outputs of the student and teacher networks, calculating loss based on agreement at316, and iteratively updating the parameters of both the student and teacher networks at318until the feature representations generated by each of the student and teacher for the different image versions converge on the same feature representation (e.g., to a defined degree of similarity). The loss function used at316can vary. In some implementations, the loss can be based on distillation loss. In other implementations, the loss can be based on contrastive loss. The parameter updating at318can involve updating the parameters of both models by exotically moving the average of both the student and teacher model parameters.

The final converged feature representation generated by the respective student and teacher transformer networks can be used as the representative feature representation for the input image302. In the embodiment shown, the converged feature representation for the input image is based on the feature representations learned for the input image302as paired with alternat image306. However, the number of different paired images processed for a single input image302to train the network can vary. For example, in some embodiments, at304the paired view generation component104can generate several (e.g., two, three, five, ten, etc.) different alternate image versions for the same input image302. The paired image generation component104can further group the alternate image versions and/or the input image into different pairs which the training component114can process through the parallel arms of the student and teacher networks to converge on intermediate feature representation for each pair. The training component114can further combined the intermediate feature representations generated for each pair into a final feature representation for the input image302.

The training component116can perform process300for each input image included in the set of training images126to train the student and teacher networks in parallel to generate a converged feature representation for each of the training images126, represented in system100as training feature representations132. The indexing component118can further store the training image feature representations132in an indexed data structure (e.g., indexed image-feature data134) with their corresponding training images and/or information identifying their corresponding training images (e.g., an image identifier and its stored network location for accessing the corresponding training image as stored in the medical image data128).

Upon completion of this self-supervised training process, either the student transform network308or the teacher transformer network310can be employed as the transformer network116. In this regard, because both the student and the teacher transformer networks are trained until they converge on the same output, the parameters of the respective networks will be the same or substantially the same. In this regard, once training has been completed, the transformer network116can correspond to the trained version of the student or the teacher, or a combined version of both the student and the teacher transformer networks.

As noted above, the self-supervised training process300is driven by the assumption that the paired alternate image versions305are similar enough to one another (and/or the input image302) such that their feature representations are similar to the that of the input image302in latent space. In natural image problems, typical augmentations include random adjustments to the input image to generate an alternate image version, such as random cropping, random brightness adjustments, random rotation, etc. However, for medical imaging problems, each image can have multiple anatomies and varying acquisition schema (among other complex visual properties). Accordingly, a randomly cropped or augmented medical image can have completely different semantic information than the original.

For example,FIG.4presents example randomly generated alternate images for a natural image and a medical image in accordance with one or more embodiments of the disclosed subject matter. In this example, the natural image includes an image of a horse, and the medical image includes a cardiac MRI image. As illustrated inFIG.4when the original horse image401is randomly augmented to generate different augmented versions402and403(e.g., via random cropping and rotation), a human could easily view the augmented images and know they depict a horse. However, when the original cardiac MRI image404is randomly augmented to generate different augmented versions405and406(e.g., via random cropping and rotation), the semantic content of the augmented versions can be lost. For example, although the randomly augmented image406appears similar to the original cardiac MRI image, the randomly augmented image405does not.

One mechanism for ensuring the alternate medical image versions are semantically similar could involve manually generating the alternate images for each of the training images (e.g., manual cropping, rotation, etc.). However, this would counteract the initiative of the self-learning process300, which is aimed at eliminating manual involvement and annotation to significantly enhance the efficiency of the feature learning process to allow for fast and automated of feature representations for any set of medical images. With this problem in mind, the disclosed techniques employ domain knowledge associated with the medical images to facilitate automatically generating meaningful alternate images versions for the medical images by the paired view generation component104.

In particular, with reference again toFIG.1, the paired view generation component104can include paired view generation schema112that defines one or more rules and/or protocols for generating the alternate medical image views for the training images126, wherein the one or more rules and/or protocols are based on medical image domain knowledge information130associated with the training images126. For example, the paired view generation schema112can define how the paired view generation component104should select or generate an alternate image for a given medical image based on known information about the medical image, such as the capture modality, the anatomical region or regions depicted, the type of the scan, and acquisition protocols/parameters used, patient attributes, and so on. This known information can be provided in metadata associated with the medical image, provided with the training images as auxiliary data, and/or learned by the paired view generation component104in association with analyzing the training images before augmentation. The level of granularity of the paired view generation schema112can vary. For example, in some implementations, the paired view generation schema can define the same alternate image generation protocols for all medical images, regardless of modality, anatomical region or regions depicted, acquisition protocols used, patient attributes, and so on. In other implementations, the paired view generation schema112can define different rules/protocols for different types of medical images, different types of modalities, different types of anatomical regions, different types of patient groups (e.g., grouped by demographic attributes and/or clinical attributes) and so on. For example, the paired view generation schema for knee MRI images may be different than the paired view generation schema for cardiac MRI images and/or spine MRI images. The paired view generation schema may also vary depending on capture modality (e.g., different for MRI relative to CT and/or XR), depending on whether a training medical image is included in a scan series (e.g., a CT scan series, and MRI scan series or the like), depending on whether a training medical image corresponds to a frame in a video (e.g., an ultrasound video scan recording), and so on.

In some embodiments, the paired view generation component104can include selection component106that selects one or more alternate images for a given training image from the medical image data128. With these embodiments, the selection component106can employ known attributes about the original image and the candidate alternate images in the medical image data128and defined similarity criteria for the attributes as provided by paired view generation schema112to select the similar alternate image or images. The known attributes can be extracted from metadata associated with the respective images and/or otherwise associated with the medical images. The defined similarity criteria can include defined similarity criteria for acquisition parameter attributes, patient attributes, visual property attributes, 3D scan position attributes, and the like. For example, in some implementations in which a training medical image is included in a series, such as a CT exam series, an MRI exam series or the like, the paired view generation schema112may define selection criteria for selecting alternate images for the training image from the same series that is based on the relative position of the training image in the series. For instance, the selection criteria may require the alternate image to have a scan position that is spaced a defined distanced relative to the training image and captured along the same acquisition axis. In other implementations in which the training medical image corresponds to a frame in sequential frames of a video sequence (e.g., an ultrasound video sequence), the selection criteria may require the alternate image or images to include neighboring frames captured withing a defined time range relative to the training image frame (e.g., images from neighboring frames in a video sequences or 3D acquisitions or corresponding frames from different cardiac cycles).

Additionally, or alternatively, the defined similarity criteria can define similarity criteria for acquisition parameters and protocols (e.g., contrast phase, imaging frequency, reconstruction kernel size, slice thickness, radiation dose, image orientation/perspective, MR sequence, capture resolution, voxel size, scan prescription plane, scan region, bounding box, and scan time.), patient attributes (e.g., same patient, same patient demography, same pathology, same BMI, etc.), visual properties (e.g., pixel spacing, slice thickness, resolution, imaging depth, etc.) and so on. As noted above, the selection component106can extract/identify the relevant attributes from metadata tags associated with the respective images (e.g., DICOM tags or the like). For example, the selection criteria (defined by the paired view generation schema112) may instruct the selection component to select one or more similar images to the training image that were acquired of the same anatomical region with the same capture modality yet with varying (e.g., withing a defined degree of similarity) acquisition parameters.

Additionally, or alternatively, the augmentation component108can generate the alternate image version or versions for a given training image by creating an augmented version or version (e.g., a synthetically generated edited version) of the training image using one or more defined augmentation techniques that are tailored based on the domain associated with the training image. With these embodiments, the paired view generation schema112can define the specific image augmentation rules for a given training image based on the training image being a medical image and/or based on other known attributes about the training image. For example, the augmentation rules can be based on a type of the training image, a modality of the training image, acquisition parameters used to generate the training image, anatomical region or regions depicted in the training image, patient attributes associated with the training image, and/or known visual properties of the training image.

For example, in some implementations in which the acquisition parameters used to generate the training image are known (e.g., included in metadata associated with the image or can otherwise be extracted from the image), the paired view generation schema112can define how rules for generating the alternate image version that are based on adjusting the acquisition parameters of the original image in association with generating a synthetic alternative image that ensure the semantic content is similar. With these implementations, the augmentation component108can generate one or more similar versions of the original training image by adjusting one or more of the acquisition parameters as defined by the paired view generation schema112.

For example,FIG.5presents example alternate images for a medical image generated by adapting domain specific acquisition protocols. In this example, the original training image comprises a cardiac MM image captured at an original imaging depth and signal frequency (e.g., depth=30 centimeters and frequency=1.4 megahertz (MHz)). The augmented images in this example are generated (e.g., by the augmentation component108) by adjusting the imaging depth and/or the signal frequency. It should be appreciated that the type of acquisition parameters adjusted and the manner in which they are adjusted to generate the alternate image version or versions for a given medical image can vary based on the capture modality, the anatomical region or regions scanned, and the acquisition protocols used.

With reference again toFIG.1, in addition to and/or alternative to adjusting acquisition parameters, the augmentation component108can employ other defined augmentation mechanisms to create alternate image versions for a give training image, which can be tailored based on the type of the image (e.g., modality and/or anatomical region scanned) and/or other known attributes about the medical image. For example, in some implementations, augmentation schema for can define cropping bounds or cropping regions for the original image that are tailored based on the type of medical image and/or the anatomical region or regions scanned. For instance, in ultrasound images, acquisitions are generally focused on one or more objects of interest. According to this example, the augmentation schema for ultrasound images can instruct the augmentation component108to generate the alternate image version or versions by cropping the image a defined distance relative to the center point (e.g., a center crop).

In other embodiments, the paired view generation schema112can instruct the augmentation component108to generate one or more alternate versions for a given training image by cropping and/or rotating the image relative to one or more defined regions of interest included in the training image. For example, the defined region of interests can be based on anatomical landmarks depicted in the original medical image, tissue properties (e.g., bone verses soft tissue), scan lines, and the like. In some implementations of these embodiments, information identifying or indicating the defined regions of interest for a particular original image may be provided in metadata associated with the input image. For example, the metadata may indicate the scan range, the bounding box (e.g., for 3D acquisitions), and/or the organ or organs of interest included in the medical image. With these implementations, the augmentation component108can crop the original image relative to the identified or indicated region or regions of interest to generate the alternate versions.

In other embodiments, the paired view generation component104can include regions of interest component110to facilitate automatically identifying the region or regions of interest in the original image for which to base the cropping and/or rotating. In some implementations of these embodiments, the region of interest component110can determine the region or regions of interest relative to which the augmentation component108should perform the cropping based on feedback provided by the transformer network attention heads in association with one or more passes of the original image and/or previously generated alternate image versions through the encoder network. With these embodiments, the region of interest component110can examine attention head maps derived from the attention heads for different image patches to identify the particular region or regions that the attention heads consistently map to throughout the different patches, as illustrated inFIG.6. The augmentation component108can further generate one or more alternate images for the original image by cropping the original image relative to the mapped attention head regions to ensure the mapped attention head regions are included in the cropped versions.

For example,FIG.6illustrates example attention head maps for two different medical images in accordance with one or more embodiments of the disclosed subject matter. In this example, both of the medical images included axial MM images of the knee. Image set601corresponds to one image while image set602corresponds to the second image. The upper row of images in each set corresponds to the attention head map while the lower row of images depicts the attention head maps overlaid onto the original input image. Both images were processed by the ViT network200to generate the corresponding attention head maps which are overlaid onto the original input images. As illustrated inFIG.6, for both input images, the ViT network200attention head maps consistently point to similar anatomical regions of interest across the different images, which in this case include the bone region of the knee. These regions identified by the attention head maps can be used in guiding the view generation and driving the self-supervised learning process by retaining important landmarks in various view of the same image.

For example, with reference again toFIG.3, in some implementations, prior to generating the alternate image version or versions for the original input image302at304, the training component116can pass the input image302through either the student transformer network308or the teacher transformer network310to generate a baseline feature representation for the input image302. The region of interest component110can further derive the attention head maps for the different attention heads and identify the region or regions in the original input image302that the attention heads focused their feature extraction. Thereafter, the augmentation component108can generate the one or more cropped versions of the original input image302for training the network based on the identified region or regions of interest. For example, the augmentation component108can crop the original input image302to include different views that include the region or regions of interest yet exclude other non-relevant portions of the input image.

Additionally, or alternatively, the region of interest component110can derive the attention head maps during the training process and the augmentation component108can use the attention head maps to iteratively generate new alternate images for the original input image in association with iteratively training the student and teacher transformer networks, as illustrated inFIG.7

In this regard,FIG.7illustrates another example self-supervised representation learning process700for medical images using parallel transformer networks in accordance with one or more embodiments of the disclosed subject matter. Process700is similar to process300with the addition of iterative refinement of the paired view generation at304based on the attention head maps. Repetitive description of like elements is omitted for sake of brevity.

In accordance with process700, the student and transformer networks can process one or more first pairs of alternate image versions305for a given training image302as described with reference to process300to generate the corresponding feature representations312and314for the respective images. As described with reference to processes300this self-supervised learning process involves iteratively updating the parameters of the respective teacher and student networks at318based on the agreement loss between the respective feature vectors312and314. In accordance with process700, prior to convergence of the teacher and transformer networks for a particular pair of alternate images305, the model training can be paused and the region of interest component110can generate and examine the attention head maps derived from the attention heads of the respective student and/or teacher transformer networks to identify the region or regions of interest in the input images. In the embodiment shown, two attention head maps702are illustrated for exemplary purposes corresponding to attention head1and attention head2of the respective teacher and student transformer networks. It should be appreciated that the number of attention head maps can vary based on the number of attention heads deployed in the corresponding teacher and transformer networks. The regions of interest reflected in the attention head maps702are indicated by the outlined boxes. These regions of interest can be automatically identified and extracted by the region of interest component110. Additionally, or alternatively, these regions of interest can be manually identified.

At704, the regions of interest identified in the attention head maps702can further be employed to iteratively refine the alternate image view generation process at304. In this regard, once the region or regions of interest in the attention head maps have been identified, the augmentation component108can generate one or more new cropped versions of the original input image302for continued training of the network based on the identified region or regions of interest. For example, the augmentation component108can crop the original input image302to include different views that include the region or regions of interest yet exclude other non-relevant portions of the input image and generate new alternative versions for pairing and processing by the parallel arms of distillation model. These new alternative versions are thus guided based on the outputs of the respective student and teacher transformer networks prior to completion of the training process.

FIG.8presents a high-level flow diagram of an example computer-implemented process800for learning feature representations of medical images using a self-supervised learning paradigm in accordance with one or more embodiments of the disclosed subject matter. Process800provides a high-level flow diagram of the some of the primary features and functionality of system100. Repetitive description of like elements employed in respective embodiments is omitted for sake of brevity.

In accordance with process800, at802, a system operatively coupled to a processor (e.g., system100) can generate alternate view images for respective medical images included in set of training images (e.g., training images126) using one or more image augmentation techniques (e.g., performed by the augmentation component108) and/or one or more image selection techniques (e.g., performed by the selection component106) tailored based on domain knowledge associated with the respective medical images (e.g., as defined based on the paired view generation schema112). At804, the system can train a transformer network (e.g., transformer network116, ViT200or the like), to learn reference feature representations (e.g., training feature representations132) for the respective medical images using their alternate view images and a self-supervised training process (e.g., process300, process700or the like). At806, the system can store the reference feature representations in an indexed data structure (e.g., indexed image-feature data134) with information identifying the respective medical images that correspond to the reference feature representations.

FIG.9illustrates an example system900for employing the medical image feature representations learned using the self-supervised learning techniques described above for automating downstream tasks such as image retrieval, image classification and other medical image processing tasks, in accordance with one or more embodiments of the disclosed subject matter. As described above, these learned feature representations can include the training image feature representations132learned and generated for a set of training images126and stored in the indexed image-feature data134. System900can include same or similar elements, components as system100with the replacement of the feature learning module102with the task module902. Although not shown, in some embodiments, system900can also include or be operatively coupled to the feature learning module102. In this regard, in some embodiments, system100and system900can be combined into a same system. Repetitive description of like elements employed in respective embodiments is omitted for sake of brevity.

The task module902corresponds to and includes several computer executable components that facilitate employing the learned training image feature representations132and the transformer network116once trained in the manner described above to perform various tasks, including image retrieval, image classification and other medical image processing tasks. These computer executable components feature generator component904, matching component906, retrieval component908, classification component910, clustering component912and annotation component914. These computer/machine executable components (and other described herein) can be stored in memory associated with the one or more machines. The memory can further be operatively coupled to at least one processor, such that the components can be executed by the at least one processor to perform the operations described. For example, in some embodiments, these computer/machine executable components can be stored in memory122which can be coupled to processing unit120for execution thereof. Examples of said and memory and processor/processing unit as well as other suitable computer or computing-based elements, can be found with reference toFIG.26, and can be used in connection with implementing one or more of the systems or components shown and described in connection withFIG.9or other figures disclosed herein.

The deployment architecture of system900(and other systems described herein) can vary. In some embodiments, the task module902can be deployed at and executed by a single computing device (e.g., real or virtual) operatively coupled to the processing unit120and the memory122. With these embodiments, one or more of the various data structures including the medical image data128, the medical image domain knowledge information130, and the indexed image-feature data134may also be stored locally on the same computing device and/or remotely at another device and accessed by the computing device via a wired or wireless communication network. In other embodiments, one or more components of the task module902can be deployed at two or more separate communicatively coupled computing devices operating in a distributed computing environment. The separate computing devices can be communicatively coupled via one or more wired or wireless communication networks. With these embodiments, one or more of the various data structures including the medical image data128, the medical image domain knowledge information130, and the indexed image-feature data134may also be stored locally on the same computing device and/or remotely at another device and accessed by the components of the task module902via a wired or wireless communication network. Various alternative deployment architecture variations can also be used.

The feature generator module904can employ the transformer network116once trained on the training images126in the manner described above to generate feature representations for new medical images. In particular, the feature generator module904can apply the transformer network to a new medical image to generate a corresponding feature representation for the new medical image. In accordance with system900, these new medical images can be provided by the medical image data128. However, the source of the new medical images can vary. The type or types of new medical images that can be processed by the transformer network116once trained can include essentially any medical image that was represented in the training data set. In this regard, the scope of the transformer network116can vary depending on the scope of the medical images included in the training data set. For example, in embodiments in which the transformer network116was trained on a training dataset including only knee MRI images, the scope of the new images capable of being accurately processed by the transformer network116will be limited to knee MRI images. It should be appreciated that the scope of the transformer network116can thus vary. In some embodiments, a plurality of different transformer networks can be developed using the techniques described above with reference to system100, wherein each of the different transformer networks are tailored to a specific type of medical images. For example, the specific type of medical image may be based on anatomical region scanned (e.g., different transformer networks116for different anatomical regions), modality, or another factor.

The matching component906can further employ the feature representation for the new medical image and the training image feature representations to identify one or more of the training medical images that are similar to the new medical image based on matching the new medical image feature representation with the one or more training image feature representations. In this regard, the training image feature representations can be used as reference feature representations that can be used by the matching component906to match new medical images with one or more of the training medical images. Because the feature representations for the training images can be learned autonomously (e.g., without manual supervision) using the self-supervised techniques described above, the number and variation of the reference medical images is essentially unlimited. In this regard, the using the techniques described with reference to system100, the feature learning module102can be used to autonomously learn and generate feature representations for a plethora of medical image datasets, making the images in these datasets searchable and usable for downstream tasks including similar image identification and retrieval. For example, medical image providers can supply large datasets of medical images to the feature learning module102to automatically learn and generate feature representations for all of their medical images.

The techniques used by the matching component906to perform the matching can vary. In general, the matching component906can select the matching training image or images to the new image based on the feature representations for the matching training images having a defined degree of similarity to the feature representation for the new medical image. In some embodiments, the matching component906can employ a k-nearest neighbors (K-NN) algorithm to perform the matching. Other feature matching algorithms are envisioned.

In some embodiments, the retrieval component908can further retrieve the matching training image or images for the new medical image. For example, in embodiments in which the training medical images are stored in the medical image data128, the retrieval component908can retrieve the matching training images from the medical image data128. In other implementations in which instances of the training images are stored in the indexed image-feature data134along with their learned feature representations, the retrieval component can retrieve the matching training medical images from the indexed image-feature data134.

This new image to training image matching function provided by the task module can be used for variety of different use cases. For example, in some embodiments, this matching function can be used in the clinical context to find similar medical images for a given image to facilitate comparative analysis and longitudinal studies. For instance, a clinician may want to exam other medical images for same or different patients that are similar to a particular medical image to facilitate their clinical review and understanding of the particular medical image. With these implementations, the clinician can submit the new medical image for processing by the task module in association with a request to find similar images included in a database of reference medical images, wherein the reference medical images include the training images whose feature representations were previously learned. The retrieval component908can further retrieve and provide the matching image or images (if found) to the clinician for viewing. For example, the retrieval component908can provide the matching images to the clinician's device, PACs workstation, cloud-based storage system, or the like. In other embodiments, the new image to training image matching function can be used to find other medical images that match a given image in association with curating a training dataset for training a particular task model (e.g., a clinical inferencing model or the like). For example, the new medical image may correspond to an underrepresented class in training dataset used for the task model and the matching function can be used to find other medical images to balance the class in the training dataset.

FIG.10illustrates an example process1000for preforming similar medical image retrieval in accordance with one or more embodiments of the disclosed subject matter. In this regard, process1000provides an example process that can be performed by system900using the feature generator component904, the matching component906and the retrieval component908.

In accordance with process1000, the feature generator component904can apply the transformer network116once trained in the manner described above to a new medical image1002to generate a feature representation1004for the new medical image. At1006, the matching component906can employ the training image feature representations stored in the indexed-image feature data134to perform feature representation matching between the feature representation1004and the training image feature representations132(e.g., using a K-NN matcher or the like). The result of the matching process at1006will include the matching feature representations1008of the matching training image or images. In this example, the matching component906can be configured to return the top N closest matches. The number N can vary. At1010, the retrieval component908can retrieve the corresponding training images for the top N matches (e.g., from the medical image data128or another location whether they are stored). The retrieved matching training images1012can further be provided to the appropriate entity or entities for usage thereof.

Process1000can also be used to determine whether a given new image such as image1002is outside the scope of the training images126based on an inability of the matching component906to find any similar images. In this regard, depending on the degree of similarity scrutiny employed by the matching component906to perform the feature representation matching, the matching component906may determine that none of the training images match the new image1002. With these implementations, rather than returning the matching image or images, the matching component906can indicate to the requesting entity (e.g., the entity supplying the new image1002in association with a request to find matching training images) with information indicating that no matches were found.

FIGS.11-16present example medical image retrieval results facilitated by self-supervised learned medical image feature representations in accordance with one or more embodiments of the disclosed subject matter. The results illustrated inFIGS.11-16were achieved using a ViT corresponding to ViT200trained on a plurality of training medical images including a variety of different knee MRI images, cardiac MM images and spine MM images and the techniques described above with reference to system100. The features learned for each of these training medical images were then stored. Once trained, the ViT was applied to the test images in each of these categories (e.g., knee, cardiac and spine MRI images) to generate their corresponding feature representations. The feature representations for the test images were then used to find the top three most similar training images whose feature representations where most similar to the test image feature representations.FIGS.11and12present the similar image retrieval results for different knee MM test images.FIGS.13and14present the similar image retrieval results for different cardiac MM test images, andFIGS.15and16present the similar image retrieval results for different spine MM images. As can be seen by the similar image retrieval results, the top three similar training images for all of the test images are highly similar to their test images, demonstrating the efficacy of the disclosed techniques for similar image retrieval. In this regard, the similar images significantly resemble their test images in terms of structure, texture and contrast. The ViT trained using the self-supervised technique disclosed herein even learned to accurately detect and represent screws and metal artifacts, as illustrated in the spine MRI images shown inFIG.16.

FIG.17presents a high-level flow diagram of an example computer-implemented process1700for preforming similar medical image retrieval in accordance with one or more embodiments of the disclosed subject matter. Repetitive description of like elements employed in respective embodiments is omitted for sake of brevity.

In accordance with process1700, at7102, a system operatively coupled to a processor (e.g., a combination of system100and system900) can generate alternate view images for respective medical images included in set of training images (e.g., training images126) using one or more image augmentation techniques (e.g., performed by the augmentation component108) and/or one or more image selection techniques (e.g., performed by the selection component106) tailored based on domain knowledge associated with the respective medical images (e.g., as defined based on the paired view generation schema112). At1704, the system can train a transformer network (e.g., transformer network116, ViT200or the like), to learn reference feature representations (e.g., training feature representations132) for the respective medical images using their alternate view images and a self-supervised training process (e.g., process300, process700or the like). At1706, the system can apply the transformer network (once trained) to a new medical image to generate a feature representation for the new medical image (e.g., using feature generator component904). At1708, the system can employ the feature representation and the reference feature representations to identify similar medical images of the respective medical images whose feature representations have a defined degree of similarity to the feature representation (e.g., using matching component906and retrieval component908).

With reference again toFIG.9, in addition to similar image retrieval, the autonomously learned feature representations for a given medical image dataset (e.g., the training images126) can also be used for image classification tasks. For example, in some embodiments, the indexing component118can associate class labels (and/or other known attributes) with each of the training image feature representations in the indexed image-feature data134. The class labels can identify a known semantic classification of the respective training images corresponding to each of their feature representations. For example, the class labels can indicate a type of the medical image, a sub-type of the medical image, and/or other known attributes about the medical image (e.g., capture modality, orientation, contrast phase depicted, anatomical region or regions depicted, and so on). The number and type of annotation labels (e.g., class labels, attribute labels, etc.) associated with each of the training image feature representations can vary.FIG.18provides example knee magnetic MRI image classes that can be applied to different knee MRI images andFIG.20provides example cardiac MM image classes that can be applied to different cardiac MRI images.

With reference again toFIG.9, in some embodiments in which the indexed image-feature data134includes class labels (and/or other annotation labels) for the training image feature representations, the feature generator component904can employ the trained transformer network116to generate a feature representation for a new medical image using the techniques described above. The matching component906can further employ the feature representation to identify one or more similar training image feature representations stored in the indexed image-feature data134. However, rather than returning the similar training images, the classification component910can employ the class labels (and/or other annotations) associated with the similar training image feature representations to determine classification information for the new image. For example, in some implementations, the classification component910can provide all of the class labels and attributes associated with each of the top N matching training images as classification results for the new image. In other implementations, the classification component910can aggregate the class labels and attributes associated with each of the top N matching training images as classification results for the new image. For example, assuming all of the top N matching training image feature representations have the same classification label (e.g., coronal knee MM), the classification component910can determine with a high degree of confidence that the new image also belongs to the same classification label. Additionally, or alternatively, the training component114can train classification model (e.g., generally referred to as a classifier) to automatically classify the training feature representations based on their corresponding classification labels (and/or other annotation labels) provided in the indexed image feature data134. With these embodiments, once the classifier has been trained, the classification component can directly apply the classifier to the feature representation generated for the new image to output a corresponding classification for the new image.

The classification techniques described above rely on the existence of classification label and/or attribute annotations associated with the training image representations in the indexed image-feature data134. In some embodiments, the indexing component118can automatically determine and apply these annotations based on information included in metadata associated with the respective training images. Additionally, or alternatively, these annotations can be manually provided. Although manual annotation ensure accurate and correct labeling of the training image classes, manual annotation of medical images can be extremely tedious, expensive and time-consuming.

To minimize the degree of manual annotation needed, in one or more additional embodiments, the training component114can train a classification model to automatically classify the training images based on their feature representations using a select subset of the training images/feature representations with manually applied annotations as the ground truth exemplars. With these embodiments, the clustering component912can facilitate selecting the subset of training images for annotation based on clustering of their corresponding feature representations. In particular, the clustering component912can employ one or more existing clustering algorithms to cluster the training image feature representations into different groups or clusters based on similarities and differences between their feature representations. This technique was proven to be effective at separating the semantically different classes of both knee MRI images (e.g., the different knee MRI classes shown inFIG.18) and cardiac MRI images (e.g., the different cardiac MRI classes shown inFIG.20) based on their learned feature representations alone, as illustrated inFIGS.19and21respectively.

In this regard,FIG.19presents an example t-SNE plot1900of learned feature representations for a variety of different knee MRI images, andFIG.21presents an example t-SNE plot2100of learned feature representations for a variety of different cardiac MRI images. In various embodiments, the clustering component912can generate these t-SNE plots or a similar t-SNE plot for all of the learned feature representations for the training images. In the embodiments shown inFIGS.19and20, the semantic classes of each of the respective feature representations are indicated to demonstrate the ability of the clustering component to accurately group/cluster the learned feature representations into their semantic classes. As illustrated inFIG.19, the learned feature representations for the different knee MM images can automatically be well separated into their respective semantic classes using existing clustering algorithms. The irrelevant class has a mixture of images from all classes—femur, tibia, coronal and sagittal and thus is seen intermixed with all other classes. These images include those that correspond to the irreverent knee MRI image group shown inFIG.18. Similarly, as illustrated inFIG.21, the learned feature representations for the different cardiac MRI images can also be automatically separated into their respective semantic classes using existing clustering algorithms. For example, the subcostal class is clearly well separated from the 2CH and 4CH classes. Although some intermixing is observed for the cardiac classes, this can be expected due to the relative similarities between the images in the intermixed classes (e.g., 4CH, 5CH and 3CH for example). However, even if points are intermixed, there are local clusters of images from the same class.

With reference again toFIG.9in view ofFIGS.19and21, in various embodiments, the annotation component914can employ the feature representation cluster information (e.g., the tSNR plots or the information used to generate the tSNR plots) generated by the clustering component912to select a subset of representative images from the training images for manually annotating with classification labels and/or other annotation labels for downstream tasks. In some embodiments, the annotation component914can be configured to select one or more representative images for each cluster based on the relative positions of their feature representations within or around the distinct clusters, as illustrated inFIG.22. In this regard,FIG.22illustrates representative image selection based on learned feature representation clustering using tSNR plot2100for the cardiac MM images. As illustrated inFIG.22, in some implementations, the annotation component914can be configured to select one or more representative images for each cluster that corresponds to the feature representations nearest to the cluster center point. In this regard, those feature representations that are nearest the cluster centers can be assumed to most strongly correspond to the distinct class represented by each cluster. The annotation component914can also be configured to select representative images for those uncertain feature representations that are farthest away from the cluster centers and/or do not fall into a distinct cluster.

In some embodiments, based on selection of the representative images the annotation component914can provide the representative training images to one or more annotators for manual annotation (e.g., with class labels and/or other attribute labels). For example, the annotation component914can send the representative images to devices and/or workstations employed by one or more expert annotators via which the annotators can apply the annotations and return the applied annotations back to the task module902. The training component114can further use the representative images with the applied class label annotations as the few ground truth exemplars in association with training a classification model to classify the training images based on their feature representations. Once trained, the classification component910can apply the classification model to a feature representation generated for a new image (e.g., by the feature generator component904) to automatically generate classification information for the new image (e.g., to determine a class, sub-class, etc., of the new image).

FIG.23presents diagram of an example processes2300for employing medical image feature representations learned using a self-supervised learning paradigm for automating downstream tasks such as image classification, in accordance with one or more embodiments of the disclosed subject matter. Repetitive description of like elements employed in respective embodiments is omitted for sake of brevity.

In accordance with process2300, at2302, the feature learning module102can learn feature representations for respective images included in the set of training images126using the self-supervised transformer network training techniques described with reference toFIGS.1-8. As previously described, the particular alternate image views generated for the respective training images126can be tailored based on domain knowledge associated with the training images126and defined paired view generation schema112associated with the training image domain. For example, the specific augmented image version and/or selected alternate image versions can be tailored based on the type, modality, anatomical region depicted and/or other known information about the training images126. The result of the self-supervised feature learning process includes the training image feature representations132which can be stored in the indexed image-feature data134along with their corresponding training images and/or information identifying their corresponding training images.

At2304, the clustering component912can cluster the feature representations into one or more distinct clusters based on similarities and differences between their feature representation using one or more existing clustering algorithms (e.g., k-means or the like). As described above, the clustering results in the respective clusters corresponding to different semantic classes of the corresponding training images. At2306, the annotation component914can select representative images for class labeling (or other forms of annotation) based on the clustering of the training image feature representations132. For example, in some implementations, the annotation component914can select one or more representative images for each cluster (and thus each semantic class) corresponding to the feature representations at or nearest the cluster center points. The annotation component914may also select representative images corresponding to the outlier feature representations that do not clearly belong to a distinct cluster. The annotation component914can retrieve the representative images2308as stored in the indexed image feature data134and/or at the source location in the medical image data128. At2310, the annotation component can provide the representative images to an annotator2312for class labeling. At2316, the task module902can receive the annotated representative image2314from the annotator and at2318, the indexing component118can add the annotations to the indexed image-feature data for the representative images.

At2320the classification component910can thereafter employ the annotated representative images/feature representations as the “few” ground truth exemplars along with the unlabeled images/feature representations (e.g., the remaining training images126that were not labeled/annotated) to train a classification model2322to classify the images based on their feature representations.

FIG.24presents a diagram of an example processes2400for performing medical image classification in accordance with one or more embodiments of the disclosed subject matter. Process2400demonstrates an example process that can be performed by the task module902using the classification model2322after training using the techniques described above. Repetitive description of like elements employed in respective embodiments is omitted for sake of brevity.

In accordance with process2400, the task module902can receive a new image2402included in the same domain of the medical images used to train the transformer network116and the classification model. The feature generator component904can then apply the transformer network to the new image to generate a feature representation2404for the new image. Thereafter, the classification component910can apply the classification model2322to the feature representation2404to generate image classification information2406for the new image2402. In this example, the classification model correctly classifies the new image as an axial tibial knee MRI based solely on its feature representation2402.

FIG.25presents a high-level flow diagram of an example computer-implemented process2500for employing medical image feature representations learned using a self-supervised learning paradigm for automating downstream tasks in accordance with one or more embodiments of the disclosed subject matter. Repetitive description of like elements employed in respective embodiments is omitted for sake of brevity.

In accordance with process2500, at2502a system operatively coupled to a processor (e.g., a combination of system100and system900) can generate alternate view images for respective medical images included in set of training images (e.g., training images126) using one or more image augmentation techniques (e.g., performed by the augmentation component108) and/or one or more image selection techniques (e.g., performed by the selection component106) tailored based on domain knowledge associated with the respective medical images (e.g., as defined based on the paired view generation schema112). At2504, the system can train a transformer network (e.g., transformer network116, ViT200or the like), to learn reference feature representations (e.g., training feature representations132) for the respective medical images using their alternate view images and a self-supervised training process (e.g., process300, process700or the like). At2506, the system can select a subset of the representative medical images of the respective medical images for annotation (e.g., using the clustering component912and the annotation component914). At2508, the system can receive the annotations for the subset based on the selecting. At2510, the system can employ the subset of labeled representative images as ground truth exemplars in association with training a task model (e.g., classification model2322or another task model).

Example Operating Environment

One or more embodiments can be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions for carrying out operations of the present invention can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, procedural programming languages, such as the “C” programming language or similar programming languages, and machine-learning programming languages such as like CUDA, Python, Tensorflow, PyTorch, and the like. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server using suitable processing hardware. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In various embodiments involving machine-learning programming instructions, the processing hardware can include one or more graphics processing units (GPUs), central processing units (CPUs), and the like. For example, one or more of the disclosed machine-learning models (e.g., the transformer network116, the ViT200, the classification model2322and the like) may be written in a suitable machine-learning programming language and executed via one or more GPUs, CPUs or combinations thereof. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It can be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions can be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

In connection withFIG.26, the systems and processes described below can be embodied within hardware, such as a single integrated circuit (IC) chip, multiple ICs, an application specific integrated circuit (ASIC), or the like. Further, the order in which some or all of the process blocks appear in each process should not be deemed limiting. Rather, it should be understood that some of the process blocks can be executed in a variety of orders, not all of which can be explicitly illustrated herein.

With reference toFIG.26, an example environment2600for implementing various aspects of the claimed subject matter includes a computer2602. The computer2602includes a processing unit2604, a system memory2606, a codec2635, and a system bus2608. The system bus2608couples system components including, but not limited to, the system memory2606to the processing unit2604. The processing unit2604can be any of various available processors. Dual microprocessors, one or more GPUs, CPUs, and other multiprocessor architectures also can be employed as the processing unit2604.

The system bus2608can be any of several types of bus structure(s) including the memory bus or memory controller, a peripheral bus or external bus, or a local bus using any variety of available bus architectures including, but not limited to, Industrial Standard Architecture (ISA), Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Firewire (IEEE 1394), and Small Computer Systems Interface (SCSI).

The system memory2606includes volatile memory2610and non-volatile memory2612, which can employ one or more of the disclosed memory architectures, in various embodiments. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer2602, such as during start-up, is stored in non-volatile memory2612. In addition, according to present innovations, codec2635can include at least one of an encoder or decoder, wherein the at least one of an encoder or decoder can consist of hardware, software, or a combination of hardware and software. Although, codec2635is depicted as a separate component, codec2635can be contained within non-volatile memory2612. By way of illustration, and not limitation, non-volatile memory2612can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), Flash memory, 3D Flash memory, or resistive memory such as resistive random access memory (RRAM). Non-volatile memory2612can employ one or more of the disclosed memory devices, in at least some embodiments. Moreover, non-volatile memory2612can be computer memory (e.g., physically integrated with computer2602or a mainboard thereof), or removable memory. Examples of suitable removable memory with which disclosed embodiments can be implemented can include a secure digital (SD) card, a compact Flash (CF) card, a universal serial bus (USB) memory stick, or the like. Volatile memory2610includes random access memory (RAM), which acts as external cache memory, and can also employ one or more disclosed memory devices in various embodiments. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM) and so forth.

Computer2602can also include removable/non-removable, volatile/non-volatile computer storage medium.FIG.26illustrates, for example, disk storage2614. Disk storage2614includes, but is not limited to, devices like a magnetic disk drive, solid state disk (SSD), flash memory card, or memory stick. In addition, disk storage2614can include storage medium separately or in combination with other storage medium including, but not limited to, an optical disk drive such as a compact disk ROM device (CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RW Drive) or a digital versatile disk ROM drive (DVD-ROM). To facilitate connection of the disk storage2614to the system bus2608, a removable or non-removable interface is typically used, such as interface2616. It is appreciated that disk storage2614can store information related to a user. Such information might be stored at or provided to a server or to an application running on a user device. In one embodiment, the user can be notified (e.g., by way of output device(s)2636) of the types of information that are stored to disk storage2614or transmitted to the server or application. The user can be provided the opportunity to opt-in or opt-out of having such information collected or shared with the server or application (e.g., by way of input from input device(s)2628).

It is to be appreciated thatFIG.26describes software that acts as an intermediary between users and the basic computer resources described in the suitable operating environment2600. Such software includes an operating system2618. Operating system2618, which can be stored on disk storage2614, acts to control and allocate resources of the computer2602. Applications2620take advantage of the management of resources by operating system2618through program modules2624, and program data2626, such as the boot/shutdown transaction table and the like, stored either in system memory2606or on disk storage2614. It is to be appreciated that the claimed subject matter can be implemented with various operating systems or combinations of operating systems.

A user enters commands or information into the computer2602through input device(s)2628. Input devices2628include, but are not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, web camera, and the like. These and other input devices connect to the processing unit2604through the system bus2608via interface port(s)2630. Interface port(s)2630include, for example, a serial port, a parallel port, a game port, and a universal serial bus (USB). Output device(s)2636use some of the same type of ports as input device(s)2628. Thus, for example, a USB port can be used to provide input to computer2602and to output information from computer2602to an output device2636. Output adapter2634is provided to illustrate that there are some output devices2636like monitors, speakers, and printers, among other output devices2636, which require special adapters. The output adapters2634include, by way of illustration and not limitation, video and sound cards that provide a means of connection between the output device2636and the system bus2608. It should be noted that other devices or systems of devices provide both input and output capabilities such as remote computer(s)2638.

Computer2602can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s)2638. The remote computer(s)2638can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device, a smart phone, a tablet, or other network node, and typically includes many of the elements described relative to computer2602. For purposes of brevity, only a memory storage device2640is illustrated with remote computer(s)2638. Remote computer(s)2638is logically connected to computer2602through a network interface2642and then connected via communication connection(s)2644. Network interface2642encompasses wire or wireless communication networks such as local-area networks (LAN) and wide-area networks (WAN) and cellular networks. LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ring and the like. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks like Integrated Services Digital Networks (ISDN) and variations thereon, packet switching networks, and Digital Subscriber Lines (DSL).

Communication connection(s)2644refers to the hardware/software employed to connect the network interface2642to the bus2608. While communication connection2644is shown for illustrative clarity inside computer2602, it can also be external to computer2602. The hardware/software necessary for connection to the network interface2642includes, for exemplary purposes only, internal and external technologies such as, modems including regular telephone grade modems, cable modems and DSL modems, ISDN adapters, and wired and wireless Ethernet cards, hubs, and routers.

While the subject matter has been described above in the general context of computer-executable instructions of a computer program product that runs on a computer and/or computers, those skilled in the art will recognize that this disclosure also can or can be implemented in combination with other program modules. Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks and/or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the inventive computer-implemented methods can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, mini-computing devices, mainframe computers, as well as computers, hand-held computing devices (e.g., PDA, phone), microprocessor-based or programmable consumer or industrial electronics, and the like. The illustrated aspects can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. However, some, if not all aspects of this disclosure can be practiced on stand-alone computers. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

As used in this application, the terms “component,” “system,” “platform,” “interface,” and the like, can refer to and/or can include a computer-related entity or an entity related to an operational machine with one or more specific functionalities. The entities disclosed herein can be either hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between two or more computers. In another example, respective components can execute from various computer readable media having various data structures stored thereon. The components can communicate via local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and/or across a network such as the Internet with other systems via the signal). As another example, a component can be an apparatus with specific functionality provided by mechanical parts operated by electric or electronic circuitry, which is operated by a software or firmware application executed by a processor. In such a case, the processor can be internal or external to the apparatus and can execute at least a part of the software or firmware application. As yet another example, a component can be an apparatus that provides specific functionality through electronic components without mechanical parts, wherein the electronic components can include a processor or other means to execute software or firmware that confers at least in part the functionality of the electronic components. In an aspect, a component can emulate an electronic component via a virtual machine, e.g., within a cloud computing system.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. Moreover, articles “a” and “an” as used in the subject specification and annexed drawings should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form. As used herein, the terms “example” and/or “exemplary” are utilized to mean serving as an example, instance, or illustration and are intended to be non-limiting. For the avoidance of doubt, the subject matter disclosed herein is not limited by such examples. In addition, any aspect or design described herein as an “example” and/or “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs, nor is it meant to preclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

As it is employed in the subject specification, the term “processor” can refer to substantially any computing processing unit or device comprising, but not limited to, single-core processors; single-processors with software multithread execution capability; multi-core processors; multi-core processors with software multithread execution capability; multi-core processors with hardware multithread technology; parallel platforms; and parallel platforms with distributed shared memory. Additionally, a processor can refer to an integrated circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Further, processors can exploit nano-scale architectures such as, but not limited to, molecular and quantum-dot based transistors, switches and gates, in order to optimize space usage or enhance performance of user equipment. A processor can also be implemented as a combination of computing processing units. In this disclosure, terms such as “store,” “storage,” “data store,” data storage,” “database,” and substantially any other information storage component relevant to operation and functionality of a component are utilized to refer to “memory components,” entities embodied in a “memory,” or components comprising a memory. It is to be appreciated that memory and/or memory components described herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of illustration, and not limitation, nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or nonvolatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM). Volatile memory can include RAM, which can act as external cache memory, for example. By way of illustration and not limitation, RAM is available in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). Additionally, the disclosed memory components of systems or computer-implemented methods herein are intended to include, without being limited to including, these and any other suitable types of memory.

What has been described above include mere examples of systems and computer-implemented methods. It is, of course, not possible to describe every conceivable combination of components or computer-implemented methods for purposes of describing this disclosure, but one of ordinary skill in the art can recognize that many further combinations and permutations of this disclosure are possible. Furthermore, to the extent that the terms “includes,” “has,” “possesses,” and the like are used in the detailed description, claims, appendices and drawings such terms are intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim. The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations can be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.