Generating synthetic electron density images from magnetic resonance images

A conversion device (20) is operable to perform a learning-based method of generating a synthetic electron density image (sCT) of an anatomical portion based on one or more magnetic resonance, MR, images. The method is processing-efficient and capable of producing highly accurate sCT images irrespective of misalignment in the underlying training set. The conversion device (20) receives and installs a machine-learning model (22) trained to predict coefficients of an image transfer function (24). The conversion device (20) then receives a current set of MR images (MRI) of the anatomical portion, computes current coefficients ([C]) of the image transfer function (24) by operating the machine-learning model (22) on the current set of MR images (MRI), and computes a current sCT image of the anatomical portion by operating the current coefficients ([C]), in accordance with the image transfer function (24), on the current set of MR images (MRI).

TECHNICAL FIELD

The present invention relates generally to the field of medical imaging and, more particularly, to a technique for generating synthetic electron density images from magnetic resonance images, e.g. for use in planning of external radiotherapy treatment.

BACKGROUND

In traditional planning of radiotherapy treatment, e.g. of cancer, a computed tomography (CT) image is acquired of an anatomical portion of the patient and used as basis for optimizing the treatment plan. CT or “CT scan” refers to a medical imaging technique that measures attenuation of radiation, typically x-rays, when passing through an object from different angles, and that reconstructs 2D or 3D images (“tomograms”) of the object based on the measured attenuation. The values in the resulting CT image thereby represents how the object responds to the radiation and may be directly converted to electron densities, e.g. so-called Hounsfield units (HU), and used for radiation dose calculation.

When creating the treatment plan, it is also necessary to delineate the target area for the radiotherapy, e.g. one or more tumors with some added margin. Often, it is also necessary to delineate so-called organs at risk (OARs), which are sensitive and vital organs that should be exposed to a minimum of radiation. A drawback of CT is that soft tissue structures are difficult to separate in CT images. Delineating using only CT images therefore entails guesswork and large margins. To alleviate this problem, modern radiotherapy often also involves an additional MRI scan. MRI (Magnetic Resonance Imaging) refers to an imaging technique that uses powerful magnetic fields, magnetic field gradients, and radio waves to generate tomographic 2D or 3D images of organs and structures in the body. MRI is free of ionizing radiation and offers superior soft tissue contrast that allows more accurate target and structure delineation. However, since MR images do not contain electron density information, a combined CT/MRI workflow is conventionally used in radiotherapy planning. The combined CT/MRI workflow is expensive and time consuming and also adds additional discomfort to the patient. Further, CT itself adds a significant radiation dose to the patient, which limits its repeated use on a patient.

To overcome these drawbacks, MRI-only workflows have been proposed, in which MRI data is processed algorithmically to synthetically generate electron density images, known as “pseudo CT”, “substitute CT” or “synthetic CT” (sCT), thereby obviating the need for CT scans. One major challenge in this context is the lack of correspondence between the respective pixel intensity in an MR image and the associated radiation attenuation property of the tissue. MRI intensities rather correlate with tissue proton density and magnetic relaxation. This leads to ambiguity between tissues in the MR image. For example, bone and air both appear dark in MR images although they have very different radiation attenuation properties.

Known techniques for sCT may be broadly divided into three categories: atlas-based approach, segmentation-based approach, and learning-based approach.

The atlas-based approach uses a database of pre-matched MR and CT images, denoted MR and CT atlases or templates. MR atlases from the database are then registered by deformation with a new incoming MR image, and the same deformation is then applied to CT atlases which are combined to generate an sCT image. Atlas-based approaches are generally quite time consuming.

The segmentation-based approach is exemplified in the article “A dual model HU conversion from MRI intensity values within and outside a bone segment for MRI-based radiotherapy treatment planning of prostate cancer”, by Korhonen et al, published inMed. Phys.41, 011704 (2014). A conversion model for soft tissue and bony tissue, respectively, is determined by comparing, e.g. by regression analysis, bulk intensity values within well-defined regions-of-interest (ROIs) in MR images and standard CT images. A new incoming MR image is then manually segmented and the respective conversion model is applied to the respective segment to transform the MR image into an sCT image. This approach relies on tissue segmentation, which may be difficult to perform automatically on MR images with high accuracy. Further, dividing the anatomy into few tissue types provides a coarse approximation of the actual anatomy.

The learning-based approach involves training and using machine-learning methods for direct conversion from a first to a second medical imaging modality. A technique of training a convolutional neural network (CNN) to convert MR images into sCT images is disclosed in the article “MR-based synthetic CT generation using a deep convolutional neural network method”, by Xiao Han, published inMed. Phys.,44(4), pp 1408-1419 (2017). A similar disclosure of direct image-to-image mapping is found in WO2018/048507. In fact, this technique was known in the scientific community well before 2017. For example, a technique of training a convolutional neural network (CNN) to convert MR images into PET images is disclosed in the article “Deep Learning Based Imaging Data Completion for Improved Brain Disease Diagnosis”, by Li et al, published inMed Image Comput Comput Assist Intern.2014; 17(0 3): 305-312. Further, methods of using machine-learning methods to convert MR images into estimated CT images are disclosed in the articles “Estimating CT Image from MRI Data Using Structured Random Forest and Auto-context Model”, by Huynh et al, published inIEEE Trans Med Imaging.2016January;35(1): 174-183, and “Generation of brain pseudo-CTs using an undersampled, single-acquisition UTE-mDixon pulse sequence and unsupervised clustering”, by Su et al, published inMed. Phys.42 (8), August 2015.

The learning-based approach generally employs a large training set of matched MR and CT images, by which an artificial neural network (ANN) or the like is trained to generate sCT images based on incoming MR images. In the training, the ANN will fit an unstructured non-linear function to each pixel based on a finite neighborhood (receptive field) around the pixel, based on the entire training set. This means that fine details in the sCT are lost if there are small errors in the matching between MR and CT images in the training set.

SUMMARY

It is an objective of the invention to at least partly overcome one or more limitations of the prior art.

A further objective is to provide a technique for generating synthetic electron density images based on MR images of an anatomical portion.

A yet further objective is to provide such a technique capable of being automated.

Another objective is to provide a technique capable of generating the synthetic electron density images with high level of detail and accuracy.

One or more of these objectives, as well as further objectives that may appear from the description below, are at least partly achieved by a computer-implemented methods and a device for generating a synthetic electron density image of an anatomical portion, a machine-learning model, computer-implemented methods of providing a trained machine-learning model, and computer-readable media according to the independent claims, embodiments thereof being defined by the dependent claims.

Still other objectives, as well as features, aspects and technical effects of the present invention will appear from the following detailed description, the attached claims and the drawings.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Also, it will be understood that, where possible, any of the advantages, features, functions, devices, and/or operational aspects of any of the embodiments of the present invention described and/or contemplated herein may be included in any of the other embodiments of the present invention described and/or contemplated herein, and/or vice versa. In addition, where possible, any terms expressed in the singular form herein are meant to also include the plural form and/or vice versa, unless explicitly stated otherwise. As used herein, “at least one” shall mean “one or more” and these phrases are intended to be interchangeable. Accordingly, the terms “a” and/or “an” shall mean “at least one” or “one or more”, even though the phrase “one or more” or “at least one” is also used herein. As used herein, except where the context requires otherwise owing to express language or necessary implication, the word “comprise” or variations such as “comprises” or “comprising” is used in an inclusive sense, that is, to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention. As used herein, a “set” of items is intended to imply a provision of one or more items.

As used herein, “synthetic electron density image” refers to any type of image that is computationally generated to contain signal values (“intensity values”) directly related to electron density. Such images may represent, replace or supplement electron density images generated by computed tomography (CT) using radiation in any wavelength range, including but not limited to x-ray radiation. In the description to follow, a synthetic electron density image is denoted “sCT image” for brevity.

As used herein, “MR image” refers to any type of image generated by an apparatus configured to detect nuclear magnetic resonance (NMR) signals, e.g. an MR scanner of any configuration. In the drawings, an MR image is designated “MRI”.

As used herein, an “image” may be two-dimensional (2D) or three-dimensional (3D). A 3D image corresponds to a stack of spatially adjacent 2D images and is commonly denoted “image stack” in the field of tomographic imaging. Each 2D image corresponds to a cross-sectional slice through the object that is being imaged.

As used herein, a “pixel” is an image element and is associated with a pixel value. In 2D images, the location of a pixel may be defined in a 2D regular grid with a fixed location in relation to the respective 2D image, e.g. by (x,y) values. In 3D images, pixels are also known as “voxels”, and the location of the pixel may be defined in a 3D regular grid with a fixed location in relation to the respective 3D image, e.g. by (x,y,z) values. As used herein, a pixel may correspond to the smallest individual element in the image, or a group of such elements.

As used herein, a “matrix” is an array of any dimension. The array comprises elements that may be populated by a respective value. For example, a matrix may have two or three dimensions. The term matrix may also be denoted “tensor” in the context of the present disclosure.

As used herein, a “machine-learning model” refers to any type of predictive model that may be trained to predict numeric values, including but not limited to artificial neural networks (ANN), such as deep learning networks, recurrent neural networks, etc., as well as support vector machines (SVM) and bayesian networks, etc.

In the description to follow, the machine-learning model is designated MLM, where a trained MLM is indicated by subscript T (MLMT) and a non-trained MLM is indicated by an asterisk (MLM*). In this context, the process of operating a trained MLM on new input data to generate a result is commonly denoted “inference”.

Embodiments of the present invention relate, in general, to the field of generating synthetic electron density information from one or more MR images. The forming of synthetic electron density information may be particularly useful for MR based radiation treatment planning. However, it should be realized that it may be used in other applications as well. For instance, the synthetic electron density information may be used as attenuation information for a positron emission tomography (PET) camera or a single-photon emission computed tomography (SPECT) camera. The synthetic electron density may also constitute proton stopping power information for use in proton radiotherapy planning.

Embodiments are based on the insight that it would be possible to improve the level of detail and accuracy in the synthetic electron density image compared to existing techniques, by operating an “optimized” image transfer function on the one or more MR images. Embodiments are also based on the insight that a properly trained machine-learning model may provide such an “optimized” image transfer function, which thus may be trained to parameterize an image transfer function for an incoming set of MR images such that the synthetic electron density image is obtained by operating the parameterized image transfer function of the set of MR images.

Thus, in contrast to the conventional learning-based approach, the trained machine-learning model does not directly output the synthetic electron density image but rather parameterizes the image transfer function, which thereby is tailored to yield the synthetic electron density image when operated on the input to the machine-learning model. Further, as will be exemplified further below, the machine-learning model may be trained to parameterize the image transfer function without the need for segmentation of the incoming MR image(s), which increases the versality of the technique and allows the technique to be automated and thus performed without subjective human input.

FIG.1schematically depicts an MRI-only imaging system which comprises an MRI apparatus10configured to generate one or more MR images of an anatomical portion of a patient (not shown). The MRI apparatus10, commonly known as a scanner, is well-known in the art and will not be further described. A conversion device20is configured to receive and process the MR image(s) to generate and output an sCT image. As exemplified in the Background section, the sCT image may then be analyzed together with the MR image(s) for planning of radiotherapy treatment. Alternatively or additionally, the sCT image may be analyzed for any other purpose in which the sCT image replaces a CT image that would have been obtained by conventional CT equipment.

FIG.2is a more detailed view of an MRI-only imaging system in accordance with an embodiment. In the illustrated example, the conversion device20is configured to retrieve MR image(s) from a memory device or storage100, in which a plurality of MR images, [MRI], generated by the MR apparatus10are stored. The conversion device20comprises a coefficient generator21which operates a trained machine-learning model, MLMT,22on incoming MR image(s) to generate and output a current set of coefficients [C] of a predefined image transfer function, ITF. The ITF is a parametric function that takes one or more pixel values in the MR image(s) as input (independent variable) and produces a pixel value (dependent variable) in the sCT image. As used herein, a “coefficient” represents any unknown value that is needed to complete the ITF to enable computation of the dependent variable based on the independent variable(s). In other words, the coefficient(s) are applied to parameterize the ITF. In an example, the ITF is an affine function given by k·Ip+m, where Ipis the pixel value in an incoming MR image and k, m are coefficients. An affine ITF has been found to provide a reasonable compromise between computation complexity and accuracy of the resulting sCT image. However, generally, the ITF may be any suitable function, including but not limited to a polynomial function of any degree.

In the following examples, it is assumed that the same ITF is applied for all pixels in the incoming MR image(s), but that the coefficient(s) may vary between pixel locations in the MR image(s). Thus, the above-described affine ITF may be represented by K·MRI+M, where K and M are matrices that define coefficients to be applied to individual pixel values in an MR image, MRI. Thus, in this example, the set of coefficients [C] includes the K and M matrices.

It may be noted that the conversion device20may operate on more than one MR image to generate the sCT image. In one example, two or more MR images may be generated during one acquisition sequence of the MR apparatus10to emphasize different tissues of the patient, e.g. so-called Dixon images which separately emphasize fat and water. Such MR images are known as “multi-channel” MR images in the art. In another example, different acquisition sequences of the MR apparatus10may produce MR images with different image contrasts, e.g. so-called T1-weighted and T2-weighted images. Any two or more MR images from one or more acquisition sequences may be jointly processed by the conversion device20into the sCT image.

The conversion device20further comprises an image converter23which is configured to receive [C] from the coefficient generator21and to apply [C] to parameterize the ITF24. The image converter23is further configured to receive the current MR image(s), i.e. the same MR image(s) that were processed by the coefficient generator21, and to apply the currently parameterized ITF on the MR image(s) to generate the current sCT image. Generally, by the parameterized ITF, the image converter23may be seen to apply the current coefficients [C] in accordance with the ITF on the current MR image(s).

FIG.3Aillustrates a training system for generating the trained machine-learning model MLMTfor use in the conversion device20ofFIG.2. The training system comprises a training device30which is configured to retrieve matched MR and CT images from a memory device200for use in producing the trained machine-learning model, MLMT. As indicated, the memory device200stores a plurality of MR images, [MRI], acquired by an MRI apparatus10, and a corresponding plurality of real CT images, [CT], acquired by a conventional CT apparatus12. The images [MRI], [CT] form a so-called training set.FIG.3Cexemplifies such a training set210, which comprises N matched MR and CT images. However, as understood from the foregoing, it is also conceivable that more than one MR image, e.g. multi-channel MR images, is matched to the CT image in the training set and thus used for training the machine-learning model. Further, although the training set210is shown to comprise matched 2D images inFIG.3C, it is equally possible that the training set210comprises matched 3D images. In fact, it is currently believed that improved performance, in terms of level of detail and accuracy of the sCT images generated by the conversion device20(FIG.2), is achieved by training the machine-learning model on matched 3D images, since the machine-learning model is thereby trained to also take into account structures in the z direction of the MR and CT images. The improved performance of the conversion device20, when operating such a trained machine-learning model, is achieved for either 2D or 3D MR images.

The training device300may operate on a large number of matched MR and CT images acquired from one or more patients. Typically, the machine-learning model may be trained on tens or even hundreds, or more, of matched 2D or 3D images. The images in the training set may be acquired for a selected anatomical area on the patients, e.g. head, torso, abdomen, limb, etc., and thus the machine-learning model may be trained to operate specifically on MR images acquired from the selected anatomical area.

FIG.3Bis a more detailed view of the training device30in accordance with an embodiment. In the illustrated example, the training device30is configured to retrieve a current data set of matched MR and CT images, e.g. from the memory device200(FIG.3A). The training device30comprises a coefficient predictor31which comprises the machine-learning model MLM*32to be trained. The coefficient predictor31corresponds to the coefficient generator21but is configured to, in response to feedback data FB* (below), modify model parameters of the MLM* and output a predicted set of coefficients [C]*. The training device30further comprises an image converter33, which corresponds to the image converter23and is configured to receive and apply [C]* to parameterize the ITF34, which is identical to the ITF24in the conversion device20. The image converter33is further configured to receive the MR image(s) of the current data set and apply the currently parameterized ITF on the MR image(s) to generate a current predicted sCT image, designated sCT* inFIG.3B. A prediction evaluator35is configured to apply a correction algorithm to generate the feedback data FB*. In one embodiment, the correction algorithm operates to optimize a cost function, which takes the sCT* image and the CT image of the current data set as input. As will be exemplified further below, the cost function may operate on further input data, e.g. [C]* or part thereof.

In the embodiment ofFIG.3B, the training device30automatically trains MLM* based on the training set to learn parameters or weights of MLM*. The parameters may be learned iteratively according to an optimality criterion, e.g. defined by the cost function. Specifically, the training device30may be seen to train MLM* until a similarity criterion is fulfilled between the CT image, which may be regarded as a reference image, and the sCT* image. The training device30may then output the trained machine-learning model MLMTfor subsequent installation in the conversion device20(FIG.2).

In one non-limiting example, the machine-learning model is a neural network for deep learning, e.g. a convolutional neural network (CNN). Such a neural network includes a stack of distinct layers that are configured to transform an input into an output. The layers may differ in input size, output size, and the relationship between the input and the output for the layer. Each layer may be connected to one or more upstream and downstream layers in the stack of layers. For deep learning, the convolutional neural network typically has more than one stage of non-linear feature transformation.

FIGS.4A-4Cwill now be used for providing a qualitative and simplified explanation of some technical advantages of generating and using a parameterized ITF in accordance with the foregoing.

FIG.4Ais intended to represent the prior art technique of direct image-to-image mapping from an MR image into an sCT image by a trained neural network, e.g. as described in the Background section. The top ofFIG.4Aillustrates matched MR and CT images, each comprising respective pixels p1, p2with exemplifying signal values (“intensity values”). The MR and CT images both represent an inverted L structure, which yields pixel values of 1 against a background of pixel values of 0 in the MR image (top left) and pixel values of 1000 against a background of pixel values of 100 in the CT image (bottom left). A slight misalignment of one pixel is introduced between the MR and CT images in the horizontal direction. Assuming that the neural network is trained based on a large number of matched MR and CT images with a small misalignment, e.g. as represented by the three additional instances of matched MR and CT images inFIG.4A, the trained neural network may produce the sCT image shown at the bottom ofFIG.4Abased on the MR image shown at the top left. As seen from the signal values of the pixels p3in the sCT image and indicated by thicker dashed lines, the inverted L structure is significantly blurred by the misalignment.

FIGS.4B-4Cillustrate an embodiment in which a machine-learning model has been trained to generate coefficients K, M of ITF=K·MRI+M, based on a large number of matched MR and CT images of the type shown inFIG.4A, i.e. with a slight mutual misalignment. InFIG.4B, an MR image representing an inverted L structure is input to the trained machine-learning model MLMT, which outputs [C] in the form of coefficient matrices or maps, namely a K matrix and an M matrix. The respective matrix comprises elements pK, pM that hold coefficient values (“pixel coefficients”) and correspond to individual pixels p1in the MR image. Thus, MLMTgenerates K, M coefficients for each pixel p1in the incoming MR image. InFIG.4C, the ITF is parameterized by the K and M matrices and operated on the same MR image that was input inFIG.4B, resulting in the sCT image shown to the right. The pixel value of the respective pixel p3in the sCT image is computed by operating the coefficient values of the corresponding elements pK, pM in the K and M matrices on the corresponding pixel value p1in the MR image, in accordance with the ITF. As seen from the signal values of the pixels p3in the sCT image and indicated by thicker lines, the inverted L structure is reproduced without blurring in the sCT image. Clearly, the generation of sCT images is robust to misalignment between images in the training set. In other words, the parameterized ITF enforces the propagation of details from the incoming MR image to the sCT image.

The example inFIGS.4B-4Cis simplified, in that MLMTis trained to generate [C] with the same coefficient values for all pixels in the incoming MR image. The present applicant has found that vastly improved results may be obtained by allowing the coefficient values to vary spatially within the image, i.e. between adjacent pixels in [C]. This means that in training, the machine-learning model MLM* is allowed to adapt coefficient values of the ITF to corresponding pixel values in the matched MR and CT images. In order to further stimulate that the ITF strives to propagate details from the MR image(s) to the sCT image(s) it may be advantageous to apply a constraint for the local variability of the coefficient values within the respective coefficient matrix. If MLM* is trained without such a restriction, the resulting MLMTmight produce coefficient matrices with high variance, even to the extent that coefficient values between neighboring elements of the respective coefficient matrix are approximately independent. This is in contradiction to the physical environment where the MR images are taken and known properties of MR images. It is well-known that MR images may exhibit artifacts in the form of a slow-varying background component (“bias field”) of image inhomogeneity and a non-standardness of the MR image intensity gray scale. In a non-limiting example, the slowly-varying bias field may have a spatial scale of about 10 cm between maximum and minimum values. The artifacts may thus cause the same tissue to yield different intensity values in different parts of an MR image. By applying the local variability constraint in training, it is ensured that MLMTgenerates [C] so that the respective coefficient values are slowly varying between matrix elements in the respective coefficient matrix, in approximate correspondence with the image inhomogeneity between pixels p1in the incoming MR image caused by artifacts. The parameterized ITF may thereby at least partly compensate for the image inhomogeneity in the incoming MR image when generating the sCT image.

For example, the local variability constraint may be set such that the CV (coefficient of variation) of coefficient values in the respective coefficient matrix are significantly smaller than the CV of intensity values in a typical MR image, where the CV is determined in a local region of predefined extent around the respective element/pixel in the coefficient matrix/MR image. For example, the local region may include all pixels/elements within one pixel distance from the respective element/pixel, e.g. neighboring pixels in x, y, z directions.

The local variability constraint may be set to differ between coefficient matrices and may also differ within a coefficient matrix, i.e. the constraint may differ between matrix elements (pixel positions). Further, a local variability constraint need not be applied to all coefficient matrices.

Generally, the parameterized ITF may be seen as a regression function which has been adapted pixel for pixel between MR and CT images in the training set so as to yield individual pixel values of an sCT image based on corresponding pixel values in an incoming MR image.

The foregoing discussion demonstrates that embodiments of the invention are capable of reducing the impact of misalignment between MR and CT images in the training set and thereby achieving a high level of detail and accuracy in the resulting sCT images. Further, there is no need for preparatory segmentation of the images used for training MLM*, or of the MR images that are being input to MLMT, and thus both training and sCT generation may be fully automated without sacrificing accuracy. However, preparatory segmentation may be used, if desired.

A further technical advantage is that the sCT image may be generated with the same pixel resolution as the incoming MR image. In the prior art technique of using a neural network to perform a direct image-to-image mapping from an MR image to an sCT image, for the neural network to work efficiently, the neural network needs to be trained on MR and CT images that are resampled to a standardized pixel size. The standardized pixel size may be determined by the pixel size of the training set, or by hardware limitations, since a smaller standardized pixel size will require more computing resources which will limit the field-of-view and/or depth of the neural network. Thus, in the prior art technique, an incoming MR image may need to be resampled to the standardized pixel resolution before inference in the trained neural network, and the resulting sCT image is generated with the standardized pixel size. If the incoming MR image has smaller pixel size, i.e. more details, than the standardized pixel size, this additional level of detail will be irrevocably lost in the resampling process. In embodiments of the invention, the machine-learning model is dissociated from the resulting sCT image and instead generates the current [C]. This makes it possible to change the pixel size of the current [C], e.g. by interpolation, before the ITF is parametrized and applied to the incoming MR image, with no or only minor impact on the ability of the parameterized ITF to transfer details of the incoming MR image to the sCT image, especially if the parameterized ITF is a slowly varying function. In the above example of an affine ITF, the respective coefficient matrix K, M may be interpolated to increase the pixel resolution of the parameterized ITF, by adding one or more element values between existing elements in K, M. Any interpolation function may be used, including by not limited to a linear function. In the present disclosure, a process of increasing pixel size (and thus decreasing pixel resolution) is denoted “down-sampling”, and a process of decreasing pixel size (and thus increasing pixel resolution) is denoted “up-sampling”.

A corresponding embodiment is exemplified inFIG.5, in which it is assumed that the pixel resolution of the incoming MR image, MRI, is higher than the predefined pixel resolution of the trained machine-learning model MLMT. A down-sampling device21A is arranged upstream of the coefficient generator21to adapt the pixel resolution of the incoming MR image to the predefined pixel resolution. The coefficient generator21operates on the downsampled MR image to generate a set of coefficients of the predefined resolution. InFIG.5, the downsampled MR image and the corresponding set of coefficients are designated by MRIs and [C]s, respectively. An up-sampling device21B is arranged upstream of the image converter23to adapt the resolution of [C]s to the pixel resolution of the incoming MR image. The image converter23then operates on the up-sampled set of coefficients [C] to generate the sCT image with the same pixel resolution as the incoming MR image.

As understood from the foregoing, the set of coefficients generated by the coefficient generator21may be up-sampled or interpolated to any desired resolution before being input to the image converter23, essentially without penalty on the level of detail in the resulting sCT image. Thus, MLMTmay be configured to operate on a predefined or standardized pixel size that is significantly larger than the pixel size used in conventional direct image-to-image mapping by neural networks. The ability to train and operate the machine-learning model on larger pixels frees up computing resources. Thus, for given computing resources, embodiments of the invention enable deployment of machine-learning models with larger field-of-view and deeper layers, thereby increasing the performance of sCT image generation even further.

The technique exemplified inFIG.5may be similarly implemented in the training device30ofFIG.3B, e.g. by a down-sampling device being arranged upstream of the coefficient predictor31, and an up-sampling device being arranged upstream of the image converter33, which operates on the up-sampled [C]* to generate the sCT* image with the same pixel resolution as the incoming MR image.

It may be noted that it is equally possible to perform up-sampling of the incoming MR image, for example to match its resolution to the resolution of the MLMT, and to perform down-sampling of set of coefficients [C] to generate the sCT image with any desired resolution, for example the same pixel resolution as the incoming MR image. Generally, a first resampling may be performed on the incoming MR image and a second resampling may be performed on the set of coefficients, where the first and second resampling may differ, for example by changing the resolution in different directions.

FIG.6illustrates an exemplifying method of generating an sCT image in accordance with an embodiment. The method may be implemented, e.g., by the conversion device20inFIG.2orFIG.5. In step601, a trained machine-learning model, MLMT, is received and installed, e.g. in computer memory. In step602, a current MR image is received, e.g. from an MRI apparatus (cf.10inFIG.2) or from a storage (cf.100inFIG.2). The current MR image may be stored in computer memory. In an optional step603, the current MR image may be pre-processed. Such pre-processing is known in the art and may involve a normalization with respect to any one of alignment, resolution, and intensity value distribution. Alternatively or additionally, step603may involve the above-mentioned re-sampling of the MR image to a predefined pixel resolution. Step603may also involve removing or masking certain portions in the MR image. Step604inputs the current MR image, optionally pre-processed, to the trained machine-learning model, MLMT, which thereby computes the set of coefficients [C] for the ITF. Thus, with reference toFIGS.4B-4C, step604may populate the coefficient matrices K, M with current pixel coefficients at elements corresponding to pixels in the current MR image. The set of coefficients [C] may then be stored in computer memory.

In an optional step605, the set of coefficients [C] are post-processed, e.g. by the above-mentioned up-sampling and/or by low-pass filtering. In step606, the current sCT image is computed by applying the resulting set of coefficients [C], in accordance with the ITF (which is thereby parametrized), on the current MR image. In an optional step607, the current sCT image may be post-processed, e.g. by removal or masking of certain portions, signal enhancement, noise removal, etc. In step608, the current sCT image is output, e.g. for storage in a memory and/or for display, optionally together with the current MR image.

As already noted above, the current sCT may be computed based on two or more MR images, which may be generated for different settings of the MRI apparatus, e.g. different sequences or contrasts. As is known in the art, still further MR images may be generated by combining MR images, e.g. by computing the sum or difference between MR images. An illustrative example is given inFIG.7, in which a coefficient generator21comprises machine-learning model MLMTthat has been trained to compute a set of coefficients for four-channel MR images. InFIG.7, the MR images of the different channels are designated A1-A4and are illustrated as 3D images, i.e. a respective stack of 2D images. In one non-limiting example, A1images may be acquired to emphasize fat, A2images may be acquired to emphasize water, A3images may be computed as A1+A2, and A4may be computed as A1-A2. In the illustrated example, the coefficient generator21outputs one K matrix for each channel and one common M matrix for all channels. The K matrices of the different channels are designated K1-K4. Since the incoming MR images are 3D images, the resulting K and M matrices are also 3D matrices. As an alternative to a common M matrix, the coefficient generator21may be configured to output one M matrix for each channel. An image converter23then operates the K and M matrices on the images A1-A4in accordance with an affine ITF to generate the sCT image, which is also a 3D image.

FIG.7is merely given as a non-limiting example. For the example of an affine ITF, the trained machine-learning model MLMTmay be generally represented as a function that takes Nin incoming MR images to generate Nout coefficient matrices K1, . . . , KNout-1, M∈Nx×Ny×Nz, but only has to apply the coefficient matrices to a subset of the MR images. This means that MLMTmay represented as a function:
h(A1, . . . , ANin)=(K1, . . . , KNout-1,M)  (1)

For example, an affine ITF that operates on the coefficient matrices may be represented as:

where {tilde over (C)} is the sCT image, g(m) is a mapping function that defines the mapping between coefficient matrix and MR image, and ⊙ represents an element-wise multiplication. The skilled person is readily able to derive corresponding equations for non-linear ITFs.

FIG.8Aillustrates an exemplifying method of training a machine-learning model MLM* in accordance with an embodiment. The method may be implemented, e.g., by the training device30inFIGS.3A-3B. As understood from the foregoing, the method is performed to learn model parameters of MLM* for a specific ITF, based on training images comprising single- or multi-channel MR images of an anatomical area of a patient, and corresponding CT images acquired by a CT scan of the same or similar anatomical area of the patient. The output of the method is a trained machine-learning model MLMT.

In step801, a training set of matched MR and CT images (cf.210inFIG.3C) is input and stored in memory (cf.200inFIG.3A). In step802, MLM* is initialized, which may involve initializing an iteration index, starting parameters of MLM*, etc. Steps803-808may be performed iteratively until one or more predefined stopping criteria are met (e.g., the iterative process converges according to a predefined criterion). In step803, training images are selected from the training set, e.g., randomly, where the training images comprise matched MR and CT images. Step803may also involve pre-processing the training images. Step804computes a predicted set of coefficients [C]* according to the current model parameters of MLM* for the training images. Step805computes the sCT* image given by ITF when parameterized by [C]* and operated on the MR image(s) in the training images. Step806evaluates a cost function. For example, the cost function may determine a difference between the CT image in the training images and the sCT* image. Such a difference may, e.g., correspond to an error map indicating the differences in pixel values between the two images. The difference may be quantified by any known regression loss function, e.g. comprising calculation of mean absolute error (MAE), mean squared error (MSE), Huber loss, Log-cosh loss, etc. Step807determines or updates model parameters of the MLM* based on the evaluation in step806, e.g. by optimizing (minimizing or maximizing) the cost function with respect to the model parameters of the MLM*. In the example of neural networks, the model parameters may include weight and offset values, as is well known in the art. Step807may use any conventional update algorithm, including but not limited to backpropagation, stochastic gradient descent (SGD), Adaptive Moment Estimation (Adam), BFGS, Rprop, etc.

Step808determines whether a stopping criterion has been satisfied. Various stopping criteria may be used, such as a predetermined maximum number of iterations or a predetermined image quality measure given, e.g., by the cost function or another measure of the difference between the sCT* and CT images. If step808determines that the stopping criterion is not satisfied, it may return to step803. Otherwise, MLM* is considered trained, and the training method proceeds to step809which stores or outputs MLMTfor subsequent use by the generation method inFIG.6.

As noted above, it has been found beneficial to apply a variability constraint during training to suppress large pixel-by-pixel variations within the respective coefficient matrix generated by the trained machine-learning model MLMT. The constraint may be applied at different stages of the training. In one example, the constraint may be embedded in the MLM* and thus applied during step804. In another example, the constraint is applied as a low-pass filtration of [C]* in step804. In yet another example, the constraint is included in the cost function and thus applied in step806. It is realized that the constraint may take many different forms. In one example, the constraint operates to limit the variability within a local region of predefined extent around the respective pixel/element in the respective coefficient matrix. The predefined extent may be determined by testing.

Since training is an autonomous process, it is desirable to mitigate undesirable outcomes, which is achieved by applying the constraint. Without the constraint, there is a risk that the MLM* is trained to suppress details from the MR images in the SCT images that are generated by applying the parameterized ITF.

In one non-limiting example, the constraint is implemented as a total-variation (TV) penalty in the cost function. The TV penalty may be computed for each coefficient matrix and in each dimension. In the example that the local region extends one pixel in each dimension of 3D matrix B, the TV penalty may be defined as:

where yx, yyand yzare predefined weight factors and Nx, Ny, Nz designate the number of elements in the respective dimension x, y, z. The weighting factors may be tuned to offset differences in resolution between the dimensions, but also be set to enforce different amount of smoothness to each dimension. It should be noted that the enforced smoothness only affects [C]* generated by MLM* and does not result in a smoothing of the sCT* image. On the contrary, a smoothness that is enforced on [C]* during training will cause the MLMTto propagate details from the MR images to the sCT images.

In one example, MLM* may be trained for the affine ITF defined in Eq. 2 by including a penalty term ρ in the cost function:

where the weight matrices Wmand the coefficient weight values ym(m=0, . . . ,Nout-1) are optional. The weight matrices Wmmay be predefined to weight specific areas of the images to further enhance or relax the smoothness penalty, and the coefficient weight values ymmay be predefined to adjust the relative effect of the respective coefficient matrix on the penalty.

In the example that the cost function comprises a computation of MAE for the error map, Eq. 2 and Eq. 4 may be combined into a cost function L:

where ∥●∥ denotes a chosen norm.

An illustrative example of the training method ofFIG.8Ais given inFIG.8B, which depicts a training device that operates on single-channel 2D MR images and is configured in correspondence withFIG.3B. A coefficient predictor31operates on the MR image to generate [C]* in the form of predicted K* and M* matrices (step804). An image converter33operates the ITF, parameterized by K* and M*, on the MR image to generate a predicted sCT* image (step805). The prediction evaluator35inFIG.3Bcorresponds to a cost function calculator35A and a model parameter updater35B inFIG.8B. The cost function calculator35A computes the terms of the cost function, including the mean absolute error between the sCT* and CT images (MAE), the TV penalty of K* (TVK), and the TV penalty of M* (TVM), and combines MAE, TVKand TVMinto a current value of the cost function (step806). The model parameter updater35B computes updated values of model parameters of MLM* to reduce the current value of cost function in subsequent iterations (step807).

FIG.8Cillustrates an alternative embodiment of the training device30. This embodiment presumes that a target set of coefficients has been pre-computed, e.g. by operating another machine-learning model (not shown) on a target set of matched MR and CT images, or by another type of computation or estimation process. The target set of coefficients is designated [C]TARinFIG.8Cand may, e.g., comprise one or more target K matrices and a target M matrix. The target device30inFIG.8Coperates on a training set of MR images. In relation to the embodiment ofFIG.3B, the target device30comprises an identical coefficient predictor31, which computes a predicted set of coefficients [C]*, but lacks the image converter33. Further, the prediction evaluator35is arranged to evaluate a cost function that compares [C]* to [C]TARand to generate, based on the evaluated cost function, feedback data FB* that causes the coefficient predictor31to update the model parameters of MLM*. Compared to the method inFIG.8A, the training device30inFIG.8Cperforms a corresponding method in which step805is omitted and at least steps806-807are modified. The training device30inFIG.8Cmay also apply the above-mentioned constraint, e.g. in the cost function.

Generally, any of the methods described herein, or part thereof, may be implemented in a processing device by a combination of software and hardware circuitry, or exclusively by tailored hardware circuitry.FIG.9is a block diagram of an exemplifying structure of such a processing device90. InFIG.9, the processing device90comprises a control circuit91responsible for the overall operation of the processing device90. As shown, the control circuit91may include a processor92, which may include a central processing unit (CPU), graphics processing unit (GPU), microcontroller, microprocessor, ASIC, FPGA, or any other specific or general processing device, or any combination thereof. The processor92may execute instructions93stored in a separate memory, such as memory93, and/or in an internal memory (not shown) of the control circuit91, in order to control the operation of the processing device90. The instructions94when executed by the processor92may cause the processing device80to perform any of the methods described herein, or part thereof. As indicated inFIG.9, the memory93may also store data95for use by the processor92. Depending on implementation, the data95may comprise one or more of data items described above and/or depicted in the drawings, such as MLMT, MLM*, [MRI], [CT], [C]PRE, etc. The memory93may comprise one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In an exemplary arrangement, the memory93includes a non-volatile memory for long term data storage and a volatile memory that functions as system memory for the control circuit91. The memory93may exchange data with the control circuit91over a data bus. Accompanying control lines and an address bus between the memory93and the control circuit91also may be present. The memory93is considered a non-transitory computer readable medium.

As indicated inFIG.9, the instructions94may be supplied to the processing device90on a computer-readable medium90A, which may be a tangible (non-transitory) product (e.g. magnetic medium, optical disk, read-only memory, flash memory, etc.) or a propagating signal.

Further, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, parallel processing may be advantageous.

Further, all of the foregoing methods, devices, embodiments, examples and aspects are equally applicable for generating an sCT image based on an origin image acquired by a medical imaging apparatus using another imaging modality than MRI, e.g. Cone Beam Computed Tomography (CBCT), Positron Emission Tomography (PET), Single-Photon Emission Computed Tomography (SPECT), Magnetic Particle Imaging (MPI), etc. It is likewise conceivable to generate the sCT image based on two or more origin images, which may be acquired by at least two different imaging modalities. In one embodiment, a computer-implemented method for generating a synthetic electron density image of an anatomical portion may comprise: receiving a machine-learning model trained to predict coefficients of an image transfer function; receiving a current set of origin images of the anatomical portion; computing current coefficients of the image transfer function by operating the machine-learning model on the current set of origin images; and computing a current synthetic electron density image of the anatomical portion by operating the current coefficients, in accordance with the image transfer function, on the current set of origin images. At least one origin image in the set of origin images may be acquired by use of an imaging modality other than MRI, e.g. any one of CBCT, PET, SPECT and MPI.

In the following, a set of items are recited to summarize some aspects and embodiments of the invention as disclosed in the foregoing.

Item 1: A computer-implemented method for generating a synthetic electron density image (sCT) of an anatomical portion, said method comprising:

receiving (601) a machine-learning model (MLMT) trained to predict coefficients of an image transfer function;

receiving (602) a current set of magnetic resonance, MR, images of the anatomical portion;

computing (604) current coefficients ([C]) of the image transfer function by operating the machine-learning model (MLMT) on the current set of MR images; and

computing (606) a current synthetic electron density image (sCT) of the anatomical portion by operating the current coefficients ([C]), in accordance with the image transfer function, on the current set of MR images.

Item 2: The computer-implemented method of item 1, wherein the machine-learning model (MLMT) is trained to predict the coefficients to achieve a similarity criterion between reference images (CT) and synthetic electron density images (sCT) generated by operating the coefficients, in accordance with the image transfer function, on MR images that correspond to the reference images (CT).

Item 3: The computer-implemented method of item 1 or 2, wherein the current coefficients ([C]) comprise one or more current pixel coefficients for a respective pixel location in the current set of MR images.

Item 4: The computer-implemented method of item 3, wherein the machine-learning model (MLMT) is trained to allow the one or more current pixel coefficients to vary between pixel locations.

Item 5: The computer-implemented method of item 3 or 4, wherein the one or more current pixel coefficients vary between pixel locations.

Item 6: The computer-implemented method of any one of items 3-5, wherein the machine-learning model (MLMT) is trained to allow the one or more pixel coefficients to vary between the pixel locations while adhering to a variability constraint of pixel coefficients within a predefined region around the respective pixel location.

Item 7: The computer-implemented method of any one of items 3-6, wherein said computing (606) the current synthetic electron density image (sCT) comprises operating the one or more current pixel coefficients, in accordance with the image transfer function, on one or more pixel values at the respective pixel location in the current set of MR images to generate a corresponding pixel value of the current synthetic electron density image (sCT).

Item 8: The computer-implemented method of any one of items 3-7, wherein said computing (604) the current coefficients ([C]) comprises populating one or more coefficient matrices (K, M) with current pixel coefficients at matrix elements corresponding to pixels in the respective MR image.

Item 9: The computer-implemented method of item 8, wherein said computing (606) the current synthetic electron density image (sCT) comprises operating the one or more coefficient matrices (K, M), in accordance with the image transfer function and element-wise, on pixel values in the current set of MR images.

Item 10: The computer-implemented method of claim 9, wherein said computing (604) the current coefficients ([C]) comprises a first resampling of the current set of MR images before operating the machine-learning model (MLMT) on the current set of MR images, wherein said method further comprises a second resampling of the one or more coefficient matrices (K, M) before operating the one or more coefficient matrices (K, M) in accordance with the image transfer function and element-wise, on the pixel values in the current set of MR images as received.

Item 11: The computer-implemented method of item 10, wherein the first resampling changes a resolution of the current set of MR images to a resolution expected by the trained machine-learning model (MLMT), and wherein the second resampling changes a resolution of the one or more coefficient matrices (K, M) to a target resolution equal to the resolution of the current set of MR images.

Item 12: The computer-implemented method of item 10 or 11, wherein the first resampling is a down-sampling and the second resampling is an up-sampling.

Item 13: The computer-implemented method of any preceding item, wherein the image transfer function is a polynomial function.

Item 14: The computer-implemented method of any preceding item, wherein the image transfer function is an affine function.

Item 15: The computer-implemented method of any preceding item, wherein the machine-learning model (MLMT) comprises an artificial neural network, ANN.

Item 16: A computer-readable medium comprising computer instructions (94) which, when executed by a processor (92), cause the processor (92) to perform the method of any one of items 1-15.

Item 17: A device for generating a synthetic electron density image (sCT) of an anatomical portion, said device comprising a processor (92) which is configured to:

receive a machine-learning model (MLMT) trained to predict coefficients ([C]) of an image transfer function;

receive a current set of magnetic resonance, MR, images of the anatomical portion;

generate current coefficients ([C]) of the image transfer function (ITF) by operating the machine-learning model (MLMT) on the current set of MR images; and

generate a current synthetic electron density image (sCT) of the anatomical portion by operating the current coefficients ([C]), in accordance with the image transfer function, on the current set of MR images.

Item 18: A system comprising a magnetic resonance imaging apparatus (10) which is configured to generate current MR images of an anatomical portion, and a device in accordance with item 17 which is arranged to receive the current MR images.

Item 19: A machine-learning model which is trained to predict coefficients ([C]) of an image transfer function to achieve a similarity criterion between reference images (CT) and synthetic electron density images (sCT), which are generated by operating the coefficients ([C]), in accordance with the image transfer function, on MR images that correspond to the reference images (CT).

Item 20: The machine-learning model of item 19, wherein the coefficients ([C]) comprise one or more pixel coefficients for a respective pixel location in the MR images, and wherein the machine-learning model is trained to allow the one or more pixel coefficients to vary between pixel locations.

Item 21: The machine-learning model of item 20, which is further trained to allow the one or more pixel coefficients to vary between the pixel locations in the respective MR image while adhering to a variability constraint of pixel coefficients within a predefined region around the respective pixel location.

Item 22: A computer-implemented method of providing a trained machine-learning model (MLMT) to predict coefficients of an image transfer function for use in generating a synthetic electron density image (sCT) of an anatomical portion as a function of a set of magnetic resonance, MR, images of the anatomical portion, said method comprising the steps of:(a) initializing the machine-learning model (MLM*);(b) obtaining training data comprising one or more MR images and one or more reference images (CT);(c) operating the machine-learning model (MLM*) on the one or more MR images to generate predicted coefficients ([C]*) of the image transfer function;(d) operating the predicted coefficients ([C]*), in accordance with the image transfer function, on the one or more MR images to generate one or more predicted synthetic electron density images (sCT*);(e) evaluating a predefined cost function at least based on the reference images (CT) and the one or more predicted synthetic electron density images (sCT*);(f) determining model parameter values for the machine-learning model (MLM*); and(g) repeating steps (b)-(f) for the model parameter values until a predefined criterion is fulfilled; and(h) outputting the model parameter values for the machine-learning model (MLMT).

Item 23: A method of providing a trained machine-learning model (MLMT) to predict coefficients of an image transfer function for use in generating a synthetic electron density image (sCT) of an anatomical portion as a function of a set of magnetic resonance, MR, images of the anatomical portion, said method comprising the steps of:(a) initializing the machine-learning model (MLM*);(b) obtaining pre-computed target coefficients ([C]TAR) of the image transfer function;(c) operating the machine-learning model (MLM*) on the one or more MR images to generate predicted coefficients ([C]*) of the image transfer function;(d) evaluating a predefined cost function at least based on the predicted coefficients ([C]*) and the pre-computed target coefficients ([C]TAR);(e) determining model parameter values for the machine-learning model (MLM*); and(f) repeating steps (b)-(e) for the model parameter values until a predefined criterion is fulfilled; and(g) outputting the model parameter values for the machine-learning model (MLMT).

Item 24: The computer-implemented method of item 22 or 23, wherein the predicted coefficients ([C]*) comprise one or more predicted pixel coefficients for a respective pixel location in the one or more MR images, said method further comprising allowing the machine-learning model (MLM*) to vary the one or more pixel coefficients between the pixel locations.

Item 25: The computer-implemented method of item 24, further comprising applying a variability constraint of pixel coefficients within a predefined region around the respective pixel location.

Item 26: The computer-implemented method of item 25, wherein the variability constraint is applied by the machine-learning model (MLM*) or by the predefined cost function, or by processing the predicted coefficients ([C]*).

Item 27: A computer-readable medium comprising computer instructions (94) which, when executed by a processor (92), cause the processor (92) to perform the computer-implemented method of any one of items 22-26.