Patent Description:
Traditional coregistration methods iteratively optimize an objective function on each new pair of images to be coregistered on, which is a computationally expensive process and can take hours to complete on a given image volume. Deep learning-based coregistration is capable of calculating the deformation without iteratively optimizing an objective function. When coupled with a graphics processing unit (GPU) as a processing unit, this results in a significantly reduced computational cost for computing the registration.

Traditional coregistration methods calculate displacement vector fields across all image pairings through a variety of iterative methods such as elastic-type modeling<NUM>, statistical parametric mapping<NUM>, and free-form deformation with b-splines<NUM>.

Frameworks for using deep convolutional neural networks (CNNs) to perform variants of coregistration on medical imaging are beginning to emerge. The majority of these methods are focused on creating deformation fields that minimize the difference between a pair of images. in particular proposed a weakly supervised method for registering magnetic resonance (MR) images onto intraoperative transrectal ultrasound prostate images<NUM>. Their method learns both affine transformation for global alignment of one image onto another as well as dense deformation fields (DDFs) of one image onto another. However, the method described in Hu et al. requires anatomical landmark points for training the model, the collection of which is time consuming and expensive. Balakrishnan et al. proposed a fully unsupervised CNN for coregistration of 3D MRI brain datasets where the loss function is purely based on the raw image data<NUM>. The approach of Balakrishnan et al. only learns the DDF of two images and accounts for affine transformations by feeding the DDF through a spatial transformation layer. <NPL>, pertains to a machine learning system that receives learning data comprising a plurality of batches of unlabeled image sets, wherein each image set comprises a source image and target image that each represents a medical image scan of at least one patient; trains one or more convolutional neural networks (CNNs) models, based on the learning data, to learn one or more transformation functions between the plurality of unlabeled images that allow for coregistration of a target image onto a source image.

The invention is defined in the appended independent claims. Embodiments of the invention are defined in the appended dependent claims.

The implementation described herein is a novel framework for unsupervised coregistration using CNNs, which is referred to herein as DeformationNet. DeformationNet takes a fully unsupervised approach to image coregistration. Advantageously, DeformationNet also explicitly stabilizes images or transfers contour masks across images. For the architecture of DeformationNet, global alignment is learned via affine deformations in addition to the DDF, and an unsupervised loss function is maintained. The use of an unsupervised loss function obviates the need for explicit human-derived annotations on the data, which is advantageous since acquisition of those annotations is one of the major challenges for supervised and semi-supervised CNNs. DeformationNet is also unique in that, in at least some implementations, it applies an additional spatial transformation layer at the end of each transformation step, which provides the ability to "fine-tune" the previously predicted transformation so that the network might correct previous transformation errors.

One implementation of the training phase of the DeformationNet system is shown in <FIG>. In at least some implementations, training of DeformationNet has two main processes:.

In at least some implementations, each pair of source and target images from a medical images database (<NUM>) represents two cardiac MR images from the same patient and possibly the same study. These cardiac MR series may include but are not limited to: Delayed Enhancement short axis (SAX) images, Perfusion SAX images, SSFP SAX images, T1/T2/T2* mapping SAX images, etc..

An affine transformation matrix with N or more affine transformation parameters, where N is an integer greater than or equal to <NUM>, is learned via a Global Network (<NUM>) wherein the input is a pair of images that includes a source image (<NUM>) and a target image (<NUM>). The learned affine transformation parameters are defined as those parameters which, when applied to the target image, align the target image with the source image. In at least some implementations, the target image is resized to match the size of the source image before the affine matrix is learned.

In at least some implementations, the Global Network (<NUM>) is a regression network. A version of the Global Network (<NUM>) includes <NUM> initial convolutional filters. At least some implementations downsample using strides in the convolutional layers and there are <NUM> convolutional layers with kernel size <NUM>, a batch normalization layer with a momentum rate, a dropout layer, and a ReLU nonlinearity layer before each downsampling operation. In at least some implementations, the last layer of the Global Network (<NUM>) is a dense layer mapping to the desired number of affine parameters.

In at least some implementations, the affine parameter outputs of the Global Network (<NUM>) are used as input to another affine spatial transformation layer that is bounded by different scaling factors for rotation, scaling, and zooming. The scaling factors control the amount of affine deformations that can be made to the target image. In at least some implementations, the affine spatial transformation matrix output by the affine spatial transformation layer includes a regularization operation that is implemented in the form of a bending energy loss function. A gradient energy loss function for regularization of the affine spatial transformation matrix may also be used, for example. This regularization further prevents the learned affine spatial transformation matrix from generating unrealistically large transformations.

In at least some implementations, a DDF is learned via a Local Network (<NUM>) wherein the input is a pair that includes a source image (<NUM>) and a target image (<NUM>). In some implementations, the target image (<NUM>) has first been warped onto the source image coordinates via an affine transformation matrix learned in the global network(<NUM>), providing a warped target image (<NUM>) to be input into the Local Network (<NUM>).

In at least some implementations, the Local Network (<NUM>) is a neural network architecture that includes a downsampling path and then an upsampling path. A version of such Local Network includes <NUM> initial convolutional filters and skip connections between the corresponding downsampling and upsampling layers. At least some implementations downsample using strides in the convolutional layers and there are <NUM> convolutional layers with kernel size <NUM>, a batch normalization layer with a momentum rate, a dropout layer, and a ReLU nonlinearity layer before each downsampling or upsampling operation. This upsampling allows the DDF to be the same size as the inputted source and target images provided that padding was used.

In at least some implementations, the learned DDF output of the Local Network (<NUM>) goes through a freeform similarity spatial transformation layer. As an example, this freeform similarity spatial transformation layer can include affine transformations or dense freeform deformation field warpings<NUM>, or both. If affine transformations are used, they may be scaled to control the amount of deformations that can be made to the target images. In at least some implementations, the DDF also includes a regularization operation that is implemented in the form of a bending energy loss functions. A gradient energy loss function may also be used to regularize the DDF. This regularization prevents the learned DDF from generating deformations that are unrealistically large.

In at least some implementations, the CNN models may be updated via backpropagation with an adam optimizer and a mutual information loss function between the source image and the target image that has been warped by the DDF (i.e., warped target image <NUM>). Adam optimizer adjusts its learning rate through training using both the first and second moments of the backpropagated gradients. Other non-limiting examples of optimizers that may be used include stochastic gradient descent, minibatch gradient descent, adagrad, and root mean squared propagation. Other non-limiting examples of loss functions may include root mean squared error, L2 loss, L2 loss with center weighting, and cross correlation loss<NUM> between the source image and the DDF that has been applied to the target image. These loss functions only depend on the raw input data and what the DeformationNet learns from that raw data. Advantageously, the absence of any dependence on explicit hand-annotations allows for this system to be fully unsupervised.

Weights of the trained Global Network (<NUM>) and Local Network (<NUM>) can be stored in storage devices including hard disks and solid state drives to be used later for image stabilization or segmentation mask transferring.

<FIG> illustrates an implementation of performing inference on a trained DeformationNet for image stabilization. In this implementation, the input to DeformationNet includes a source image (<NUM>), and a target image (<NUM>) to be stabilized by warping the target image onto the source image. These image pairings may be selected from a database of medical images (<NUM>). Using the trained DeformationNet (<NUM>), discussed above, a DDF (<NUM>) with respect to the source image (<NUM>) is inferred. This DDF (<NUM>) is applied to the target image (<NUM>), creating a warped target image (<NUM>) that is stabilized with respect to the source image (<NUM>). The newly stabilized target image (<NUM>) may be displayed to the user via a display (<NUM>) and stored in a warped images database (<NUM>) including hard disks and solid state drives.

Image pairings that may be used for image stabilization inference include but are not limited to: images from the same slice of a cardiac MR image volume but captured at different time points; images from the same time point of a cardiac MR image volume but different slices; images from any image of the same MR image volume; images from distinct MR image volumes; images from other medical imaging that involve a time series such as breast, liver, or prostate DCE-MRI (dynamic contrast enhancement MRI); or images from fluoroscopy imaging.

<FIG> illustrates one implementation of performing inference with a trained DeformationNet for transferring segmentation masks from one image to another. In at least some of the implementations, the input to DeformationNet is a pair of 2D cardiac SAX MR images (source image <NUM> and target image <NUM>) from a database of medical images (<NUM>), where one of the images has a corresponding segmentation mask (<NUM>) of ventricular contours, for instance, to include the left ventricular endocardium (LV endo), left ventricular epicardium (LV epi), and/or right ventricular endocardium (RV endo), for example. In at least some of the implementations, the segmentation mask (<NUM>) may correspond to the target image (<NUM>). Using the trained DeformationNet (<NUM>), a DDF (<NUM>) with respect to the source image is inferred. This DDF (<NUM>) is applied to the segmentation mask (<NUM>) corresponding to the target image (<NUM>) creating a warped segmentation mask (<NUM>) that has been warped onto the source image. The newly warped segmentation mask (<NUM>) can be displayed to the user via a display (<NUM>) and stored in a warped segmentation masks database (<NUM>) including but not limited to hard disks and solid state drives.

Implementations of attaining the segmentations masks (<NUM>) shown in <FIG> include, but are not limited to: having a user manually create the segmentation mask; and using a heuristic involving a previously trained CNN model to automatically create the segmentation mask.

<FIG> illustrates one implementation of using a heuristic and previously trained CNN to select a segmentation mask to transfer to other images. In this implementation, a group of 2D cardiac SAX MR images (<NUM>) is chosen for which segmentations are needed. Those images (<NUM>) are used as input to a previously trained CNN (<NUM>), as discussed above. In at least some implementations, the CNN (<NUM>) was previously trained to segment masks for the LV epi, LV endo, and RV endo in 2D SSFP MRs images. In those implementations, the output of the CNN (<NUM>) is a segmentation probability map (<NUM>) on a per-pixel basis for each 2D image.

The CNN (<NUM>) may not be able to accurately predict segmentations for every image, so it may be important to choose images with good quality segmentation masks as the target image for (<NUM>) (<FIG>). The segmentation probability maps (<NUM>) that are outputted from the previously trained CNN (<NUM>) are used to compute foreground map scores (<NUM>) and background map scores (<NUM>) for the given image. The map scores (<NUM>) and (<NUM>) are computed per pixel. The foreground mask scores (<NUM>) represent the probability that an image pixel belongs to one of the ventricular masks, and the background mask scores (<NUM>) represent the probability that the image pixel does not belong to one of the ventricular masks. The foreground map score (<NUM>) is calculated by taking the average of all probability map values above <NUM>. The background map score (<NUM>) is calculated by taking the distance from <NUM> of all the probability map values below <NUM>. A mask quality score (<NUM>) for that given slice prediction is then calculated by multiplying the background mask score (<NUM>) with the foreground mask score (<NUM>).

The general actions of the above described possible heuristic implementation are explained in the following example pseudocode:.

In at least some implementations, the image with the segmentation probability mask corresponding to the highest quality score across the group of 2D images will be treated as the single target image (<NUM>) and some or all of the other images will be treated as source images to which the target image's segmentation mask (<NUM>) will be warped.

<FIG> shows an example of how the heuristic described above may work in practice. Images (<NUM>) and (<NUM>) are examples of 2D SAX MR images that are to be fed into the CNN (<NUM>) (<FIG>). Images (<NUM>) and (<NUM>) are the probability map outputs of the CNN (<NUM>) for the LV epi of the images (<NUM>) and (<NUM>), respectively, represented as contour maps. The image (<NUM>) represents a good probability map. It has a clear boundary of high probability (represented by the black line of <NUM>) around the LV epi and the probability drops quickly outside of the LV epi area. The image (<NUM>) represents a bad probability map. The contours around the LV epi are overall fairly low, there is only high probability at the very center of the LV epi. Additionally, there is a change in probability far outside of the LV epi area. Foreground and background maps for the images (<NUM>) and (<NUM>) are represented as contours in images (<NUM>) and (<NUM>), respectively. The black contours represent the foreground map values as calculated by act <NUM>. b in the pseudocode above and the white contours represent the background map values as calculated by act <NUM>. d in the pseudocode. Image (<NUM>) has high probability for the foreground map and background map, which would give it a high quality score. Image (<NUM>) has high probability for the background map but low for the foreground map, which would give it a low quality score and it would likely not be used as the segmentation mask to transfer across images.

<FIG> shows a processor-based device <NUM> suitable for implementing the various functionality described herein. Although not required, some portion of the implementations will be described in the general context of processor-executable instructions or logic, such as program application modules, objects, or macros being executed by one or more processors. Those skilled in the relevant art will appreciate that the described implementations, as well as other implementations, can be practiced with various processor-based system configurations, including handheld devices, such as smartphones and tablet computers, wearable devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, personal computers ("PCs"), network PCs, minicomputers, mainframe computers, and the like.

The processor-based device <NUM> may include one or more processors <NUM>, a system memory <NUM> and a system bus <NUM> that couples various system components including the system memory <NUM> to the processor(s) <NUM>. The processor-based device <NUM> will at times be referred to in the singular herein, but this is not intended to limit the implementations to a single system, since in certain implementations, there will be more than one system or other networked computing device involved. Non-limiting examples of commercially available systems include, but are not limited to, ARM processors from a variety of manufactures, Core microprocessors from Intel Corporation, U. , PowerPC microprocessor from IBM, Sparc microprocessors from Sun Microsystems, Inc. , PA-RISC series microprocessors from Hewlett-Packard Company, 68xxx series microprocessors from Motorola Corporation.

The processor(s) <NUM> may be any logic processing unit, such as one or more central processing units (CPUs), microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Unless described otherwise, the construction and operation of the various blocks shown in <FIG> are of conventional design. As a result, such blocks need not be described in further detail herein, as they will be understood by those skilled in the relevant art.

The system bus <NUM> can employ any known bus structures or architectures, including a memory bus with memory controller, a peripheral bus, and a local bus. The system memory <NUM> includes read-only memory ("ROM") <NUM> and random access memory ("RAM") <NUM>. A basic input/output system ("BIOS") <NUM>, which can form part of the ROM <NUM>, contains basic routines that help transfer information between elements within processor-based device <NUM>, such as during start-up. Some implementations may employ separate buses for data, instructions and power.

The processor-based device <NUM> may also include one or more solid state memories, for instance Flash memory or solid state drive (SSD) <NUM>, which provides nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the processor-based device <NUM>. Although not depicted, the processor-based device <NUM> can employ other nontransitory computer- or processor-readable media, for example a hard disk drive, an optical disk drive, or memory card media drive.

Program modules can be stored in the system memory <NUM>, such as an operating system <NUM>, one or more application programs <NUM>, other programs or modules <NUM>, drivers <NUM> and program data <NUM>.

The application programs <NUM> may, for example, include panning / scrolling 632a. Such panning / scrolling logic may include, but is not limited to logic that determines when and/or where a pointer (e.g., finger, stylus, cursor) enters a user interface element that includes a region having a central portion and at least one margin. Such panning / scrolling logic may include, but is not limited to logic that determines a direction and a rate at which at least one element of the user interface element should appear to move, and causes updating of a display to cause the at least one element to appear to move in the determined direction at the determined rate. The panning / scrolling logic 632a may, for example, be stored as one or more executable instructions. The panning / scrolling logic 632a may include processor and/or machine executable logic or instructions to generate user interface objects using data that characterizes movement of a pointer, for example data from a touch-sensitive display or from a computer mouse or trackball, or other user interface device.

The system memory <NUM> may also include communications programs <NUM>, for example a server and/or a Web client or browser for permitting the processor-based device <NUM> to access and exchange data with other systems such as user computing systems, Web sites on the Internet, corporate intranets, or other networks as described below. The communications programs <NUM> in the depicted implementation is markup language based, such as Hypertext Markup Language (HTML), Extensible Markup Language (XML) or Wireless Markup Language (WML), and operates with markup languages that use syntactically delimited characters added to the data of a document to represent the structure of the document. A number of servers and/or Web clients or browsers are commercially available such as those from Mozilla Corporation of California and Microsoft of Washington.

While shown in <FIG> as being stored in the system memory <NUM>, the operating system <NUM>, application programs <NUM>, other programs/modules <NUM>, drivers <NUM>, program data <NUM> and server and/or browser <NUM> can be stored on any other of a large variety of nontransitory processor-readable media (e.g., hard disk drive, optical disk drive, SSD and/or flash memory).

A user can enter commands and information via a pointer, for example through input devices such as a touch screen <NUM> via a finger 644a, stylus 644b, or via a computer mouse or trackball 644c which controls a cursor. Other input devices can include a microphone, joystick, game pad, tablet, scanner, biometric scanning device, etc. These and other input devices (i.e., "I/O devices") are connected to the processor(s) <NUM> through an interface <NUM> such as touch-screen controller and/or a universal serial bus ("USB") interface that couples user input to the system bus <NUM>, although other interfaces such as a parallel port, a game port or a wireless interface or a serial port may be used. The touch screen <NUM> can be coupled to the system bus <NUM> via a video interface <NUM>, such as a video adapter to receive image data or image information for display via the touch screen <NUM>. Although not shown, the processor-based device <NUM> can include other output devices, such as speakers, vibrator, haptic actuator, etc..

The processor-based device <NUM> may operate in a networked environment using one or more of the logical connections to communicate with one or more remote computers, servers and/or devices via one or more communications channels, for example, one or more networks 614a, 614b. These logical connections may facilitate any known method of permitting computers to communicate, such as through one or more LANs and/or WANs, such as the Internet, and/or cellular communications networks. Such networking environments are well known in wired and wireless enterprise-wide computer networks, intranets, extranets, the Internet, and other types of communication networks including telecommunications networks, cellular networks, paging networks, and other mobile networks.

When used in a networking environment, the processor-based device <NUM> may include one or more wired or wireless communications interfaces 614a, 614b (e.g., cellular radios, WI-FI radios, Bluetooth radios) for establishing communications over the network, for instance the Internet 614a or cellular network.

In a networked environment, program modules, application programs, or data, or portions thereof, can be stored in a server computing system (not shown). Those skilled in the relevant art will recognize that the network connections shown in <FIG> are only some examples of ways of establishing communications between computers, and other connections may be used, including wirelessly.

Claim 1:
A machine learning system, comprising:
at least one nontransitory processor-readable storage medium that stores at least one of processor-executable instructions or data; and
at least one processor communicably coupled to the at least one nontransitory processor-readable storage medium, in operation the at least one processor:
receives learning data comprising a plurality of batches of unlabelled image sets, wherein each image set comprises a source image and target image that each represents a medical image scan of at least one patient;
trains one or more convolutional neural networks (CNNs) models, based on the learning data, to learn one or more transformation functions for coregistration of a target image onto a source image,
wherein the one or more CNNs models include a global network component that learns an affine transformation matrix and a local network component that learns a dense deformation field, and
wherein the global network component precedes the local network component in the one or more CNNs models; and
stores the one or more trained CNN models in the at least one nontransitory processor-readable storage medium of the machine learning system.