Patent Description:
Since its introduction, CT has been used widely in the medical diagnostic and therapeutic areas. Although CT technology has undergone numerous advances, its basic principle has been the same: it uses a rotating x-ray tube and a row of detectors placed in the gantry to measure x-ray attenuations by different tissues inside the body. Compared with other image modalities, CT has many advantages: fast scanning speed, high spatial resolution, and broad availability. Millions of CT examinations are performed annually, making CT one of the most important and widespread imaging modalities used for patient care.

Despite its remarkable success, CT technology has several limitations. One of the most substantial limitations is its low contrast resolution. It cannot reliably differentiate between the material with low inherent contrast, such as pathologic and healthy tissues. The low contrast resolution is due to the slight difference in x-ray attenuation between different tissues. For example, it is difficult to reliably assess noncalcified plaques because the differences in attenuation between lipid-rich and lipid-poor noncalcified plaques are minimal. It is also challenging to segment soft tissue structures such as cartilage from the keen CT scans due to the low contrast of the cartilage from the surrounding soft tissues.

In clinical imaging, contrast agents enhance the material contrast in CT scans. The contrast agents absorb external x-rays, resulting in decreased exposure on the x-ray detector. Contrast agents such as iodinated agents could cause kidney damage and trigger allergic reactions.

In the conventional CT, the attenuation value of each voxel is the combined attenuation of multiple materials. Dual-energy CT uses two separate x-ray photon energy spectra rather than the single energy technology used in conventional CT. It allows the interrogation of materials that have different attenuation properties at different energies. However, due to the limit of two energy bins, the tissue discrimination is still suboptimal. With more than two energies and narrow energy ranges, multi-energy CT can concurrently identify multiple materials with increased accuracy.

Photon-counting CT is an emerging technology that has been shown tremendous progress in the last decade. With photon-counting detectors, each photon of the incident x-rays hits the detector element and generates an electrical pulse with a height proportional to the energy deposited by the individual photon. Photon-counting CT inherently allows dual-energy or multi-energy acquisitions at a single source, a single tube, a single acquisition, a single detector, and a single filter. Moreover, the user-defined energy threshold selection allows the choice of suitable energy thresholds tailored to the specific energy diagnostic task. This task-driven energy-threshold selection helps resolve different tissue types with optimal imaging settings to achieve the best image quality or lowest radiation dose.

With either multi-energy CT or photon counting CT, the basic principle of material decomposition is the same: it determines the full energy dependence of the attenuation curve in every voxel of a scan. The assumption is that any human tissue is approximately equivalent to a combination of two or more basis materials, as far as x-ray attenuation properties are concerned. Although any materials can be employed as basis materials, water, calcium, iodine or fat are usually used as the basis materials. Consequently, material decomposition is also referred to as basis material decomposition. The general workflow is as follows. Using multi-energy CT or photon counting CT, the energy selective (or energy-specific) images are produced by the multi-energy bins. A set of basis material images is generated from the energy-selective images. Each basis material image represents the equivalent concentration of basis material for each voxel in the scan. The basis images can be used to obtain images of human tissues such as bone, muscle, and fat through a linear transformation of the basis images. To find the transformation formula for a piece of human tissue, the concentrations of each basis material is calculated.

Material decomposition methods have been developed. The simplest method is inversing the matrix that relates attenuation values to material concentrations. Other methods were also advanced, such as optimization with regularization. However, with the assumption of the type and numbers of basis materials, material decomposition is a non-linear ill-posed problem and inaccurate decomposition is a problem in current methods.

<CIT> discloses a method including obtaining information specifying a multi-modal statistical model including values of parameters of each of multiple components of the multi-modal statistical model, the multiple components including first and second encoders for processing input data for the first and second modalities, respectively, first and second modality embeddings, a joint-modality representation, and a predictor; obtaining first input data for the first data modality; providing the first input data to the first encoder to generate a first feature vector; identifying a second feature vector using the joint-modality representation, the first modality embedding and the first feature vector; and generating a prediction task prediction using the predictor and the second feature vector.

<CIT> discloses a method for image processing that includes acquiring multiple multi-energy spectral scan datasets and computing basis material images (including correlated noise) representative of multiple basis materials from the multi-energy spectral scan datasets, and jointly denoising the multiple basis material images in at least a spectral domain with a deep learning-based denoising network to generate multiple de-noised basis material images.

<CIT> discloses a system for performing data segmentation that includes memory storing computer-executable instructions defining a learning network that includes sequential encoder down-sampling blocks, a processor configured to execute the computer-executable instructions to receive a multi-dimensional input tensor including at least a first dimension, a second dimension and a plurality of channels and to process the multi-dimensional input tensor by passing the received multi-dimensional input tensor through the sequential encoder down-sampling blocks and, in response thereto, to generate an output tensor that includes at least one segmentation classification.

Recently, machine learning, especially deep learning methods, has shown promise in solving ill-posed problems such as image reconstruction, image resolution enhancement, and voice recognition. In this invention, a deep learning method and system is invented to present the mapping between the energy-selective images and material-specific images.

It is an object of the present invention to provide a method of generating material decomposition images from plural-energy x-ray based imaging.

According to a first aspect of the invention, there is provided a method for generating material decomposition images from a plurality of images obtained with plural-energy x-ray based imaging, the plurality of images corresponding to respective energies of the plural-energy x-ray based imaging, the method comprising:.

Spatial relationships and spectral relationships are respectively relationships between the spatial information (i.e. of the objects, materials and structures in the images) and spectral information (i.e. the different material attenuations arising from different photon energies).

It should be noted that the plurality of images obtained with plural-energy x-ray based imaging may be synthetic, in the sense that they may not have been obtained simultaneously or in a single scan, but instead compiled from a plurality of scans.

The one or more encoder branches that encode two or more images of the plurality of images in combination may receive the respective two or more images in combination, concatenated, etc, or combine, concatenate, etc, the respective two or more images before encoding them.

In an embodiment, each of two or more of the encoder branches encodes a respective different individual image of the plurality of images.

In some embodiments, a first encoder branch encodes a first combination of two or more images of the plurality of images and a second encoder branch encodes a second combination of two or more images of the plurality of images, wherein the first combination is different from the second combination (though the combinations may include common images).

The plural-energy x-ray based imaging may comprise, for example, cold cathode x-ray radiography, dual-energy radiography, multi-energy radiography, photon-counting radiography, cold cathode x-ray CT, dual-energy CT, multi-energy CT or photon-counting CT.

Advantageously, in some embodiments the encoder branches that encode a respective individual image encode in total all of the images that are encoded in total by the encoder branches that encode two or more images.

However, in some other embodiments, the encoder branches that encode a respective individual image receive in total fewer images (such as by omitting one or more low-energy images) than are encoded in total by the encoder branches that encode two or more images. This may be done, for example, to reduce computation time.

In still other embodiments, the encoder branches that encode a respective individual image encode in total more images than are encoded in total by the encoder branches that encode two or more images.

The encoder branches that encode a respective individual image may encode only images than are not encoded by any of the encoder branches that encode two or more images. However, more advantageously, the encoder branches that encode a respective individual image encode in total at least one image than is also encoded by at least one of the encoder branches that encode two or more images.

In one implementation of the invention, the combination of all of the images (referred to as the 'energy images', as each corresponds to a respective x-ray energy bin or energy threshold) is used as input to a first encoder branch, and each of the individual energy images is used as the input to a respective one of a plurality of further branches. However, in some implementations, not all of the energy images are used as input to the first encoder branch and/or as inputs to respective further branches: some energy images may be omitted. For example, if the targeted basis material images (i.e. those of interest) relate to soft tissues only, high energy images may be omitted. On the other hand, high energy images are useful for differentiating hard materials such as bone, so in implementations in which the basis material images of interest relate to hard tissues, low energy images may be omitted.

It may also be advantageous (such as to reduce computing overhead) in these or other implementations to omit one or more energy images so that the neural network is smaller and simpler, with fewer encoder branches.

Hence, in an embodiment, each of the one or more of the encoder branches respectively encodes an individual image corresponding to a low x-ray energy, and the material decomposition images correspond to one or more soft tissues. In an embodiment, each of the one or more of the encoder branches respectively encodes an individual image corresponding to a high x-ray energy, and the material decomposition images correspond to one or more hard tissues.

It is appreciated that 'low' and 'high' may be viewed as relative terms, but the appropriate low- or high-energy subset of the entire set of energy images can be readily selected by simple experimentation, balancing the quality of the results (measured in terms of resolution or completeness of material decomposition) against computing time or computing overhead.

However, in one example, the low x-ray energy images (of n-images obtained with plural-energy x-ray based imaging) comprise the n-<NUM>, n-<NUM> or n-<NUM> images of lowest energy. In another example, the low x-ray energy images comprise the one or two images of lowest energy.

In one example, the high x-ray energy images comprise the n-<NUM>, n-<NUM> or n-<NUM> images of highest energy. In another example, the high x-ray energy images comprise the one or two images of highest energy.

In an embodiment, the deep learning neural network is a trained neural network, trained with real or simulated training images obtained with real or simulated plural-energy x-ray based imaging and with basis material images. For example, the basis material images may comprise any one or more (i) HA (hydroxyapatite) images, (ii) calcium images, (iii) water images, (vi) fat images, (v) iodine images, and (vi) muscle images.

In certain embodiments, the method comprises generating any one or more of (i) a bone marrow decomposition image, (ii) a knee cartilage decomposition image, (iii) an iodine contrast decomposition image, (iv) a tumor decomposition image, (v) a muscle and fat decomposition image, (vi) a metal artefact reduction image, and (vii) a beam hardening reduction image.

The method may include training or retraining deep learning models using the neural network.

The method may include combining features extracted by the one or more encoder branches that encode two or more images in combination and features extracted by the one or more encoder branches that encode respective individual images using a concatenation layer at the end of or after an encoder network of the neural network.

In other embodiments, the method includes combining features extracted by the one or more encoder branches that encode two or more images in combination and features extracted by the one or more encoder branches that encode respective individual images using one or more concatenation operations at plural levels of an encoder network of the neural network.

In still other embodiments, the method includes combining features extracted by the one or more encoder branches that encode two or more images in combination and features extracted by the one or more encoder branches that encode respective individual images using concatenation operations that connect an encoder network of the neural network and an decoder network of the neural network at multiple levels.

In yet other embodiments, the method includes combining features extracted by the one or more encoder branches that encode two or more images in combination and features extracted by the one or more encoder branches that encode respective individual images, but an encoder network of the neural network and an decoder network of the neural network are not connected at multiple levels.

According to this aspect, there is also provided a material decomposition image, generated according to the method of this aspect (including any of its embodiments) from a plurality of images obtained with plural-energy x-ray based imaging.

According to a second aspect of the invention, there is provided a system for generating material decomposition images from a plurality of images obtained with plural-energy x-ray based imaging, the plurality of images corresponding to respective energies of the plural-energy x-ray based imaging, the system comprising:.

In an embodiment, each of two or more of the encoder branches is configured to encode a respective different image of the plurality of images.

In some embodiments, a first encoder branch is configured to encode a first combination of two or more images of the plurality of images as input and a second encoder branch is configured to encode a second combination of two or more images of the plurality of images as input, wherein the first combination is different from the second combination (though the combinations may include common images).

The plural-energy x-ray based imaging may comprise cold cathode x-ray radiography, dual-energy radiography, multi-energy radiography, photon-counting radiography, cold cathode x-ray CT, dual-energy CT, multi-energy CT or photon-counting CT.

Advantageously, in some embodiments the encoder branches configured to encode a respective individual image receive in total all of the images that are encoded in total by the encoder branches configured to encode two or more images.

However, in other embodiments, the encoder branches that encode a respective individual image are configured to encode in total fewer images (such as by omitting one or more low-energy images) than are encoded in total by the encoder branches that encode two or more images (such as to reduce computation time).

In still other embodiments, the encoder branches that encode a respective individual image are configured to encode in total more images than are encoded in total by the encoder branches that encode two or more images.

The deep learning neural network may be a trained neural network, trained with real or simulated training images obtained with real or simulated plural-energy x-ray based imaging and with basis material images. For example, the basis material images may comprise any one or more (i) HA (hydroxyapatite) images, (ii) calcium images, (iii) water images, (vi) fat images, (v) iodine images, and (iv) muscle images.

The system may be configured to generate any one or more of (i) a bone marrow decomposition image, (ii) a knee cartilage decomposition image, (iii) an iodine contrast decomposition image, (iv) a tumor decomposition image, (v) a muscle and fat decomposition image, (vi) a metal artefact reduction image, and (vii) a beam hardening reduction image.

In an embodiment, the system is configured:.

The system may include deep learning model trainer configured to train or retrain deep learning models using the neural network.

The system may be configured to combine features extracted by the one or more encoder branches that encode two or more images in combination and features extracted by the one or more encoder branches that encode respective individual images using a concatenation layer at the end of or after an encoder network of the neural network.

In other embodiments, the system may be configured to combine features extracted by the one or more encoder branches that encode two or more images in combination and features extracted by the one or more encoder branches that encode respective individual images using one or more concatenation operations at plural levels of an encoder network of the neural network.

In still other embodiments, the system may be configured to combine features extracted by the one or more encoder branches that encode two or more images in combination and features extracted by the one or more encoder branches that encode respective individual images using concatenation operations that connect an encoder network of the neural network and an decoder network of the neural network at multiple levels.

In yet other embodiments, the system may be configured to combine features extracted by the one or more encoder branches that encode two or more images in combination and features extracted by the one or more encoder branches that encode respective individual images, wherein an encoder network of the neural network and an decoder network of the neural network are not connected at multiple levels.

According to a third aspect of the invention, there is provided a computer program comprising program code configured, when executed by one of more computing devices, to implemented the method of the first aspect (and any of its embodiments). According to this aspect, there is also provided a computer-readable medium (which may be non-transient), comprising such a computer program.

It should be noted that any of the various individual features of each of the above aspects of the invention, and any of the various individual features of the embodiments described herein, including in the claims, can be combined as suitable and desired.

In order that the invention may be more clearly ascertained, embodiments will now be described by way of example with reference to the following drawing, in which:.

<FIG> is a schematic view of an image processing system <NUM> (of application in particular for processing medical images) according to an embodiment of the present invention.

System <NUM> includes an image processing controller <NUM> and a user interface <NUM> (including a GUI <NUM>). User interface <NUM> includes one or more displays (on one or more of which may be generated GUI <NUM>), a keyboard and a mouse, and optionally a printer.

Image processing controller <NUM> includes at least one processor <NUM> and a memory <NUM>. Instructions and data to control operation of processor <NUM> are stored in memory <NUM>.

System <NUM> may be implemented as, for example, a combination of software and hardware on a computer (such as a server, personal computer or mobile computing device) or as a dedicated image processing system. System may optionally be distributed; for example, some or all of the components of memory <NUM> may be located remotely from processor <NUM>; user interface <NUM> may be located remotely from memory <NUM> and/or from processor <NUM> and, indeed, may comprise a web browser or a mobile device application.

Memory <NUM> is in data communication with processor <NUM>, and typically comprises both volatile and non-volatile memory (and may include more than one of type of memory), including RAM (Random Access Memory), ROM and one or more mass storage devices.

As is discussed in greater detail below, processor <NUM> includes an image data processor <NUM>, which includes a basis material image generator <NUM>, a diagnostic/monitoring task image generator <NUM> (including a decomposer <NUM>), and an additional task-driven image generator <NUM>. Processor <NUM> further includes a deep learning model trainer <NUM> (which includes one or more deep learning neural networks <NUM>), an I/O interface <NUM> and an output in the form of a results output <NUM>. Deep learning model trainer <NUM> may be omitted in some implementations of this and other embodiments, as it is required only if system <NUM> is itself to train deep learning model(s) <NUM>, rather than access one or more suitable deep learning models from an external source.

Memory <NUM> includes program code <NUM>, image data store <NUM>, non-image data store <NUM>, training data store <NUM>, trained deep learning model(s) <NUM>, generated basis material image store <NUM> and generated material specific or material decomposition image store <NUM>. Image processing controller is implemented, at least in part, by processor <NUM> executing program code <NUM> from memory <NUM>.

In broad terms, the I/O interface <NUM> is configured to read or receive image data (such as in DICOM format) and non-image data, pertaining to-for example-subjects or patients, into image data store <NUM> and non-image data store <NUM> of memory <NUM>, respectively, for processing. The non-image data stored in non-image data store <NUM> comprises broad information such as energies, desired materials and desired tasks, and is accessible by image generators <NUM>, <NUM>, <NUM> for use in image generation.

Basis material image generator <NUM> of image data processor <NUM> generates one or more sets of basis material images with one or more machine learning models (drawn from deep learning model(s) <NUM>). Diagnostic/monitoring task image generator <NUM> uses decomposer <NUM> to generate one or more sets of material specific or material decomposition images (suitable for, for example, diagnostic or monitoring tasks) using the basis material images, and additional task-driven image generator <NUM> generates at least one further set of images (such as beam hardening or metal artefact reduced images). I/O interface <NUM> outputs the results of the processing to, for example, results output <NUM> and/or to GUI <NUM>.

System <NUM> employs one or more deep learning models to accurately and reproducibly generate the basis material images. The basis material images are then used for generating images of different tissues and materials, especially of low contrast tissues and materials, which can in turn be used in pathology or disease identification and monitoring (such as of disease progression). For example, cartilage segment images from the knee scan may be used for osteoarthritis or rheumatoid arthritis diagnosis and/or monitoring; bone marrow segment images from musculoskeletal scans may be used for related diseases diagnosis and monitoring of associated diseases or pathologies; pathological and normal tissue images from a scan of a patient may be used for diagnosis and monitoring of a tumor; simultaneous material decomposition of multiple contrast agents from a CT scan may be used for the diagnosis or identification, and staging, of renal abnormalities; muscle extracted images many be used for sarcopenia diagnosis and/or monitoring.

System <NUM> can also generate images for other tasks (using additional task-driven image generator <NUM>). For example, system <NUM> can generate beam-hardening or metal artefact reduced images based on the aforementioned basis material images for better image quality. Beam hardening or metal artefact effects occur when a polychromatic x-ray beam passes through an object, resulting in selective attenuation-principally affecting lower energy photons. As a result, higher energy photons solely or excessively contribute to the beam, thereby increasing the mean beam energy-an effect known as 'beam hardening. ' As the full energy-dependent attenuation is considered in material decomposition, it is thus desirable that the decomposed images be free of beam-hardening and metal artefact effects.

Thus, referring to <FIG>, system <NUM> is configured to receive two types of data pertaining to a subject or patient: image data and non-image data. The image data is on the form of plural-energy images based on or derived from x-ray imaging, such as may be generated in cold cathode x-ray radiography, dual-energy CT, multi-energy CT or photon-counting CT. The non-image data includes information about the plural-energy images, such as the energies at which the plural-energy images were generated, information about the desired basis material images, such as the type and number of the basis material images, and information about the desired analysis or analyses, such as disease diagnosis/identification/monitoring or additional tasks (e.g., beam hardening or metal artefact reduction). System <NUM> stores image data and non-image data in the image data store <NUM> and non-image data store <NUM>, respectively.

As mentioned above, image data processor <NUM> includes three components: basis material image generator <NUM>, diagnostic/monitoring task image generator <NUM>, and additional task-driven image generator <NUM>. The image data and non-image data are received by image data processor <NUM> from memory <NUM>. Based on the plural-energy images and the basis material in the non-image data, image data processor <NUM> selects one or more suitable deep learning models <NUM> to generate one or more sets of basis material images. Based on the task information, image data processor <NUM> generates images (e.g., human tissues, contrast agents images) for disease diagnosis/identification and/or monitoring, and images (e.g. beam hardening and metal artefact reduced images) for better image quality.

Deep learning model trainer40 pre-trains deep learning models <NUM> using training data (from training data store <NUM>) that includes labels or annotations that constitute the ground truth for machine learning. The training data is prepared so as to be suitable for training a deep-learning model for generating basis material images from the plural-energy images. The training data consists of both known plural-energy images and known basis material images. The labels indicate the energy bin of each energy image (that is, an image corresponding to a particular energy threshold or bin) and the material information (e.g., material name and material density) of the basis material images. The training data can be in the form of real clinical data, real phantom data, simulated data, or a mixture of two or more of these.

As mentioned above, deep learning model trainer <NUM> is configured to train one or more deep learning models (and to retrain or update train deep learning models) using neural network <NUM> and the training data, but in other embodiments machine learning model trainers may be configured or used only to retrain or update (i.e., re-train) one or more existing deep learning models.

Image data processor <NUM> selects one or more suitable deep learning models from deep learning model(s) <NUM>, based on the plural-energy images and the targeted basis material(s) (as identified in the non-image data). Basis material image generator <NUM> generates images of the targeted basis material. Diagnostic/monitoring task image generator <NUM> generates images according to the information concerning diagnosis/identification and/or monitoring tasks (as also identified in the non-image data), from the generated basis material images. Optionally, additional task-driven image generator <NUM> generates images according to the information of the additional tasks (as also identified in the non-image data), from the generated basis material images.

The basis material images, diagnostic/monitoring images, and/or additional task-driven images are outputted to user interface <NUM> via results output <NUM> and I/O interface <NUM>.

<FIG> is a flow diagram <NUM> of the general workflow of system <NUM> of <FIG>. Referring to <FIG>, at step <NUM> system <NUM> receives plural-energy images (generated by, for example, dual-energy, multi-energy or photon-counting CT or radiography) and reads the images into image data store <NUM>. At step <NUM>, system <NUM> receives associated non-image data and reads that data into non-image data store <NUM>.

Memory <NUM> is advantageously configured to allow high-speed access of data by system <NUM>. For example, if system <NUM> is implemented as a combination of software and hardware on a computer, the images are desirably read into RAM of memory <NUM>.

At step <NUM>, image data processor <NUM> selects one or more suitable deep learning models from the trained deep learning model(s) <NUM>. The deep learning model selection is based on the energy information characterizing the plural-energy images and the information concerning the targeted basis material, both contained in the non-image data. Any particular model is trained using the images of specific energies to generate a specific set of basis material images; hence, more than one suitable model may be trained and available. According to the energies and desired basic material specs, the corresponding model or models are is selected. If plural models are selected, they are used in parallel.

For example, one deep learning model may be selected for use with all loaded images for generating one set of basis material images. In another example, more than one deep learning model is chosen for use with all loaded images for generating several sets of basis material images. In another example, more than one deep learning model is selected to use with respective subsets of the loaded images, for generating one or more sets of basis material images.

The selected deep learning model or models include spatial relationships and spectral relationships learned from training data. At step <NUM>, basis material images generator <NUM> generates the basis material images from the loaded subject or patient images in image data store <NUM> using the one or more selected deep learning models and these spatial and spectral relationships, and saves the generated basis material images in generated basis material image store <NUM>.

At step <NUM>, diagnostic/monitoring task image generator <NUM> uses the generated basis material images to decompose the original subject or patient images in image data store <NUM> and thereby generate material specific or material decomposition images of, in this example, specific, different (e.g. human) tissues, suitable for disease identification, diagnosis and/or monitoring, and saves these material specific or decomposition images in generated material specific or material decomposition image store <NUM>.

At step <NUM>, image data processor <NUM> determines whether-according to the associated non-image data <NUM> indicating the desired task(s)-additional task-driven image generator <NUM> is required to generate any images. If not, processing ends. If so, at step <NUM>, additional task-driven image generator <NUM> generates the appropriate task-driven images, such as beam hardening reduced images and/or metal artefact reduced images. Processing then ends.

<FIG> is a schematic view of a deep learning neural network <NUM> (such as may be employed as neural network <NUM> of system <NUM>), for generating material decomposition images from plural-energy images according to an embodiment of the present invention. Neural network <NUM> is shown with input in the form of n plural-energy x-ray based images <NUM> (where n ≥ <NUM>) and output in the form of basis material images <NUM>. Neural network <NUM> is configured to generate basis material images <NUM> from the images <NUM>. That is, the functional mapping between the input images <NUM> and output basis material images <NUM> is approximated by neural network <NUM>, which is configured to predict material-specific images using the images <NUM> as input. Material decomposition images can then be generated from the generated basis material images <NUM>.

Neural network <NUM> comprises an encoder network <NUM> and a decoder network <NUM>. Encoder network <NUM> encrypts the structures of the input images (e.g. some or all of images <NUM>) into a feature representation at multiple different levels. Decoder network <NUM> projects the discriminative feature representation learnt by encoder network <NUM> into the pixel/voxel space to get a dense classification. In one example, the encoding performed by encoder network <NUM> includes convolution operations and down-sampling operations; the decoding performed by decoder network <NUM> includes convolution operations and up-sampling operations. In another example, the encoding performed by encoder network <NUM> and/or the decoding performed by decoder network <NUM> include concatenation operations.

Encoder network <NUM> has a plural-branch structure, with a first set <NUM><NUM> and a second set <NUM><NUM> of encoder branches (each set having one or more encoder branches). Each of the branches of the first set <NUM><NUM> of encoder branches encodes a plurality of images selected from images <NUM> (which may comprise all of images <NUM>) in concatenated form. (It should be noted that this or these pluralities of images selected from images <NUM> for processing in concatenated form may be inputted either in concatenated form or non-concatenated form. In the latter case, the encoder network first concatenates the images.

Each of the branches of the second set <NUM><NUM> of encoder branches encodes an individual image selected from images <NUM>. First set <NUM><NUM> and second set <NUM><NUM> may include, in total, the same or different numbers of images.

In the example of <FIG>, first set <NUM><NUM> of encoder branches includes, in this example, one encoder network branch <NUM><NUM> (comprising 'Encoder network <NUM>') for encoding a plurality of images <NUM> (in this example all of images <NUM>) in concatenated form. Second set <NUM><NUM> of encoder branches includes a plurality m of encoder network branches <NUM><NUM>, <NUM><NUM>,. <NUM>m (comprising respectively 'Encoder network <NUM>', 'Encoder network <NUM>',. 'Encoder network m') for encoding each of the respective, individual input images <NUM><NUM>, <NUM><NUM>,. , <NUM>m, where m ≤ n. (Note that images <NUM>, <NUM>,. , m need not be sequential or comprise the first m images of images <NUM>. Also, encoder network branch <NUM><NUM> may be configured to receive a plurality-but not all-of the images <NUM> in concatenated form. ) The individual images <NUM><NUM>, <NUM><NUM>,. , <NUM>m are generally of a conventional format for the respective imaging modality (e.g. DICOM, JPEG, TIFF or other imaging files), so are typically two- or three-dimensional images comprising pixels or voxels but, as they have an extra dimension indicative of energy threshold or bin, could be described as three- or four-dimensional. Likewise, the images <NUM> in concatenated or combined form have an extra dimension (indicative of energy threshold or bin), so may also be described as typically three- or four-dimensional.

Encoder network branch <NUM><NUM> of the first set <NUM><NUM> learns relationships among images <NUM> inputted into that branch and effectively combines them. Encoder network branch <NUM><NUM> of the second set <NUM><NUM> learn the features of each individual image <NUM><NUM>, <NUM><NUM>,. , <NUM>m independently. The feature representations learned by the first set <NUM><NUM> of network branches (viz. network branch <NUM><NUM>) and by the second set <NUM><NUM> of network branches <NUM><NUM>, <NUM><NUM>,. , <NUM>m are combined as the input of decoder network <NUM>.

In one example, the features extracted by first set <NUM><NUM> of encoder network branches <NUM><NUM> and by second set <NUM><NUM> of encoder network branches <NUM><NUM>, <NUM><NUM>,. , <NUM>m are combined using a concatenation layer (not shown) at the end of or after encoder network <NUM>. In another example (cf. the embodiment in <FIG>), the features extracted from first set <NUM><NUM> of branches and second set <NUM><NUM> of branches are combined using one or more concatenation operations at the plural levels of encoder network <NUM>.

In a further example (cf. the embodiment in <FIG>), concatenation operations connect the encoder network <NUM> and decoder network <NUM> at multiple levels. In still another example, encoder network <NUM> is not connected to decoder network <NUM> at multiple levels.

As mentioned above, all of images <NUM> may be concatenated to form the input (or concatenated image) for input into first branch <NUM><NUM>; alternatively, only some (but a plurality) of the input images <NUM><NUM>, <NUM><NUM>,. , <NUM>m may be concatenated to form the input (or concatenated image) for input into first set <NUM><NUM> of encoder branches (viz. encoder branch <NUM><NUM>). In one example, all of the images <NUM> are separately input into second set <NUM><NUM> of encoder branches but, in another example, some (i.e. one or more) of the images <NUM><NUM>, <NUM><NUM>,. , <NUM>n might not be encoded by second set <NUM><NUM> of encoder branches. In addition, it should be noted that the images that are input into the first and second sets <NUM><NUM>, <NUM><NUM> of encoder branches need not be the same, but are drawn from the same multi-energy images <NUM>.

Thus, deep learning neural network <NUM>, which may thus be described as a multi-branch encoder-decoder deep learning network, generates the basis material images <NUM> by inherently modelling spatial and spectral relationships among the plural-energy images <NUM>.

<FIG> is a schematic view of a deep learning neural network <NUM>' (such as may be employed as neural network <NUM> of system <NUM>), which is comparable to neural network <NUM> of <FIG> so like numerals have been used to indicate like features. Neural network <NUM>' is thus also adapted for generating material decomposition images from plural-energy images according to an embodiment of the present invention.

Neural network <NUM>' includes an encoder network <NUM>' that includes first and second sets <NUM><NUM>', <NUM><NUM>' of encoder branches. Neural network <NUM>' differs from neural network <NUM> of <FIG> in that the first set <NUM><NUM>' of encoder branches of neural network <NUM>' includes at least two encoder branches <NUM><NUM>', <NUM><NUM>' comprising encoder network <NUM>' and encoder network <NUM>', respectively, each of which is configured to receive a plurality of concatenated images (respectively images <NUM><NUM> and images <NUM><NUM>) selected from images <NUM>.

Images <NUM><NUM> and images <NUM><NUM> may comprise the same or a different numbers of images and, in either case, may constitute overlapping or non-overlapping sets of images.

<FIG> is a schematic view of a deep learning neural network <NUM> (such as may be employed as neural network <NUM> of system <NUM>), for generating basis material images from plural-energy images according to an embodiment of the present invention. Neural network <NUM> is shown with input in the form of a plurality (in this example, four) of plural-energy x-ray based images <NUM>.

Neural network <NUM> includes a multi-branch encoder network <NUM> and a decoder network <NUM>. In this embodiment, encoder network <NUM> has a first set of encoder branches comprising a single branch: a first branch <NUM> that receives the combination of all four images <NUM> as input. Encoder network <NUM> has a second set of encoder branches comprising, in this example, two branches: a second branch <NUM> that receives the first image <NUM><NUM> (being the first of plural-energy x-ray based images <NUM>) as input, and a third branch <NUM> that receives the third image <NUM><NUM> (being the third of plural-energy x-ray based images <NUM>) as input.

The encoder network structure of each of the three encoder branches <NUM>, <NUM>, <NUM> is identical, each encoder branch containing three stages defined by the size of its feature maps, with each stage containing the convolutions, batch normalization, and ReLU (Rectified Linear Unit) functions or operations. Thus, the first branch <NUM> comprises a first stage <NUM><NUM> that includes <NUM> channel first feature map <NUM><NUM>, which is the same width and height as the original combination of images <NUM> (which may also be regarded as a part of first stage <NUM><NUM> of the first branch). The second stage <NUM><NUM> includes <NUM> channel second feature map <NUM><NUM> and <NUM> channel third feature map <NUM><NUM>, while the third stage <NUM><NUM> includes <NUM> channel fourth feature map <NUM><NUM> and <NUM> channel fifth feature map <NUM><NUM>.

Likewise, the second branch <NUM> comprises a first stage <NUM><NUM> that includes <NUM> channel first feature map <NUM><NUM>, which is the same width and height as the first individual image <NUM><NUM> (which may also be regarded as a part of first stage <NUM><NUM> of the second branch). The second stage <NUM><NUM> includes <NUM> channel second feature map <NUM><NUM> and <NUM> channel third feature map <NUM><NUM>, while the third stage <NUM><NUM> includes <NUM> channel fourth feature map <NUM><NUM> and <NUM> channel fifth feature map124<NUM>.

The third branch <NUM> comprises a first stage <NUM><NUM> that includes <NUM> channel first feature map <NUM><NUM>, which is the same width and height as third individual image <NUM><NUM> (which may also be regarded as a part of the first stage <NUM><NUM> of the third branch). The second stage <NUM><NUM> includes <NUM> channel second feature map <NUM><NUM> and <NUM> channel third feature map <NUM><NUM>, while the third stage <NUM><NUM> includes <NUM> channel fourth feature map <NUM><NUM> and <NUM> channel fifth feature map <NUM><NUM>. The feature map <NUM><NUM>, <NUM><NUM>, <NUM><NUM> of each respective first stage <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, and the last feature map <NUM><NUM>, <NUM><NUM>, <NUM><NUM> of each respective second stage <NUM><NUM>, <NUM><NUM>, <NUM><NUM> undergoes max pooling, reducing the size of the feature maps and allowing encoder network <NUM> to find the global features of the respective input images <NUM>, <NUM><NUM>, <NUM><NUM>. (Note that the pooling operation is on the feature representations or maps between two stages, which is why there are two pooling operations for the three stages.

In this embodiment, decoder network <NUM> also contains three stages, with the last feature map <NUM><NUM> of the first stage of decoder network <NUM> also acting as the first stage of encoder network <NUM>. Each of the three stages <NUM><NUM>, <NUM><NUM>, <NUM><NUM> of decoder network <NUM> is defined by the size of its respective feature maps: first stage <NUM><NUM> includes <NUM> channel feature map <NUM><NUM>, second stage <NUM><NUM> includes <NUM> channel feature map <NUM><NUM> and <NUM> channel feature map <NUM><NUM>, and third stage <NUM><NUM> includes <NUM> channel feature map <NUM><NUM> and <NUM> channel feature map <NUM> (the latter being the outputted basis material image(s)). Each of these three stages <NUM><NUM>, <NUM><NUM>, <NUM><NUM> contains convolutions, batch normalization, and ReLU operations, and the feature maps of stages <NUM><NUM>, <NUM><NUM>, <NUM><NUM> undergo average pooling (i.e. a pooling operation is applied to the feature maps between stages <NUM><NUM> and <NUM><NUM>, and between stages <NUM><NUM> and <NUM><NUM>), bringing the feature map dimensions back to match those of input images <NUM>, <NUM><NUM>, <NUM><NUM>.

In this embodiment, the feature maps of each stage of the three branches <NUM>, <NUM>, <NUM> of encoder network <NUM> are concatenated (hence, respectively, feature maps <NUM><NUM>, <NUM><NUM>, <NUM><NUM>; feature maps <NUM><NUM>, <NUM><NUM>, <NUM><NUM>; and feature maps <NUM><NUM>, <NUM><NUM>, <NUM><NUM>), and then concatenated with the feature maps at the corresponding stage of decoder network <NUM> (hence, respectively, feature maps <NUM><NUM>, <NUM><NUM> and <NUM><NUM>). The connection at the multiple levels between multi-branch encoder network <NUM> and decoder <NUM> enables neural network <NUM> to learn the local details of input images <NUM>, <NUM><NUM>, <NUM><NUM>.

<FIG> is a flow diagram <NUM> of the training-by deep learning model trainer <NUM>-of the deep learning model or models stored ultimately in deep learning model(s) <NUM>. At step <NUM>, training data are prepared or sourced, the training data comprising plural-energy x-ray based images and basis material images. The training data may be real data, simulated data, or combinations thereof. In one example, the training data is generated using phantoms of different known materials. In another example, the training data is simulated based on the known characteristics of known materials under different x-ray energies. In some examples, the training data comprises real data only or simulated data only. In another example, the training data comprises some real data and some simulated data. Hence, preparing the training data may involve-for example-generating (or sourcing) real training data (see step 144a) and/or simulating (or sourcing simulated) training data (see step 144b).

The process optionally includes step <NUM>, where the training data is increased using a data augmentation method. This may entail the addition of Gaussian noise to the training data to improve the robustness of the model training, and/or dividing the training data into patches to increase the quantity of training data.

At step <NUM>, the training data are labelled with the appropriate, correct labels. Each 'energy image' (that is, an individual image corresponding to a single energy threshold or energy bin) is labelled with the relevant energy threshold or energy bin (see step 150a), and each basis material image is labelled with the relevant material (see step 150b). At step <NUM>, deep learning model trainer <NUM> trains one or more deep learning models, employing the correctly labelled energy images and basis material images. Step <NUM> may entail updating (or retraining) one or more trained deep learning models, if such models have previously been trained and the training data prepared or sourced at step <NUM> is new or additional training data.

At step <NUM>, the trained or retrained model or models are deployed for use, by being stored in machine learning model(s) <NUM>. Processing then ends, unless the process includes optional step <NUM> at which deep learning model trainer <NUM> determines whether retraining or further training is to be conducted. If not, processing ends, but if deep learning model trainer <NUM> determines that retraining or further training is to be conducted, processing returns to step <NUM>.

In use, system <NUM> inputs one or more plural-energy x-ray based images into one or more of the now trained deep learning models <NUM>, which process the images and outputs a set of basis material images.

<FIG> are schematic views of exemplary training data preparation techniques. <FIG> shows the preparation <NUM> of training data using real data. For example, one or more phantoms are used, each having an insert that contains a known material (e.g. HA (hydroxyapatite), iodine, calcium, blood or fat) or a mixture (e.g. iodine and blood) of some or all of the known materials.

The composition, concentration, size and location of each material insert are known. The phantom is scanned <NUM> using, for example, cold cathode x-ray radiography, dual-energy CT, multi-energy CT or photon-counting CT, such that the plural-energy images are generated <NUM> with two or more energy thresholds or energy bins. In this example, the aim is to generate three basis material-specific images: a HA image, an iodine image and a fat image. Each basis material-specific image is thus generated <NUM> with the known concentration, size and location of each material insert.

<FIG> shows the preparation <NUM> of training data using simulated data. For example, one or more phantoms are simulated <NUM>, again with inserts containing known materials (e.g., iodine, calcium, blood, and fat) and a mixture of some of the known materials. The concentration, size and location of each material insert are known. The plural-energy images are simulated <NUM> based on the known materials and specific energies, such as by creating those images by referring to the real scans, acquired by a plural-energy or photon-counting CT, etc, but with a different concentration of the material. For example, real scans may be acquired by scanning a real phantom with inserts comprising iodine with respective iodine concentrations of <NUM>, <NUM> and <NUM>/cc using a photon-counting CT. Simulated scans of <NUM>, <NUM>, <NUM>/cc iodine inserts can then be generated by applying a linear fitting to the CT numbers of the real scans, as photon-counting CT maintains a strong linear relationship between CT numbers and the concentrations of the same material. In another example, the energy images are created by mathematical simulation based on the known reaction of known material under different x-ray energies. The basis material-specific images are simulated <NUM> with the concentration, size and location of each material insert.

<FIG> illustrates an exemplary work flow <NUM> of system <NUM> of <FIG>, for the particular case of a patient that has been injected with an iodine contrast agent then scanned <NUM> using photon-counting CT. Five images are generated using respectively the five energy thresholds (that is, the energies above which x-rays are counted in the five corresponding energy bins), thereby producing a <NUM> keV threshold image <NUM>, a <NUM> keV threshold image <NUM>, a <NUM> keV threshold image <NUM>, a <NUM> keV threshold image <NUM> and a <NUM> keV threshold image <NUM>. From these images <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, the trained deep learning model or models generate <NUM> four basis material images: a calcium image <NUM>, a water image <NUM>, a fat image <NUM> and an iodine image <NUM>. From different linear combinations of the basis material images <NUM>, <NUM>, <NUM>, <NUM>, various functional images are generated for disease diagnostic/monitoring and/or other tasks. For example:.

It will be understood by persons skilled in the art of the invention that many modifications may be made without departing from the scope of the invention. In particular it will be apparent that certain features of embodiments of the invention can be employed to form further embodiments.

It is to be understood that, if any prior art is referred to herein, such reference does not constitute an admission that the prior art forms a part of the common general knowledge in the art in any country.

Claim 1:
A method for generating material decomposition images from a plurality of images (<NUM>; <NUM><NUM>) obtained with plural-energy x-ray based imaging, the plurality of images corresponding to respective energies of the plural-energy x-ray based imaging, the method comprising:
modelling spatial relationships and spectral relationships among the plurality of images by learning features from the plurality of images in combination and one or more of the plurality of images individually with a deep learning neural network (<NUM>; <NUM>');
generating (<NUM>) one or more basis material images (<NUM>) employing the spatial relationships and the spectral relationships; and
generating (<NUM>) one or more material specific or material decomposition images (<NUM>) from the basis material images (<NUM>); and
wherein the neural network (<NUM>; <NUM>') has an encoder-decoder structure and includes a plurality of encoder branches (<NUM>; <NUM>');
wherein
each of one or more encoder branches (<NUM><NUM>; <NUM><NUM>', <NUM><NUM>') of the plurality of encoder branches (<NUM>; <NUM>') encodes two or more images (<NUM>; <NUM><NUM>, <NUM><NUM>) of the plurality of images in combination; and
each of one or more other encoder branches (<NUM><NUM>, <NUM><NUM>,... <NUM>m) of the plurality of encoder branches (<NUM>; <NUM>') encodes a respective individual image (<NUM><NUM>, <NUM><NUM>,... <NUM>m) of the plurality of images.