Tomographic image analysis using artificial intelligence (AI) engines

Example methods and systems for tomographic data analysis are provided. One example method may comprise: obtaining first three-dimensional (3D) feature volume data and processing the first 3D feature volume data using an AI engine that includes multiple first processing layers, an interposing forward-projection module and multiple second processing layers. Example processing using the AI engine may involve: generating second 3D feature volume data by processing the first 3D feature volume data using the multiple first processing layers, transforming the second 3D volume data into 2D feature data using the forward-projection module and generating analysis output data by processing the 2D feature data using the multiple second processing layers.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is related in subject matter to U.S. patent application Ser. No. 16/722,004.

BACKGROUND

Computerized tomography (CT) involves the imaging of the internal structure of a target object (e.g., patient) by collecting projection data in a single scan operation (“scan”). CT is widely used in the medical field to view the internal structure of selected portions of the human body. In an ideal imaging system, rays of radiation travel along respective straight-line transmission paths from the radiation source, through the target object, and then to respective pixel detectors of the imaging system to produce volume data (e.g., volumetric image) without artifacts. Besides artifact reduction, radiotherapy treatment planning (e.g., segmentation) may be performed based on the resulting volume data. However, in practice, reconstructed volume data may contain artifacts, which in turn cause image degradation and affect subsequent diagnosis and radiotherapy treatment planning.

SUMMARY

According to a first aspect of the present disclosure, example methods and systems for tomographic image reconstruction are provided. One example method may comprise: obtaining two-dimensional (2D) projection data and processing the 2D projection data using the AI engine that includes multiple first processing layers, an interposing back-projection module and multiple second processing layers. Example processing using the AI engine may involve: generating 2D feature data by processing the 2D projection data using the multiple first processing layers, reconstructing first three-dimensional (3D) feature volume data from the 2D feature data using the back-projection module generating second 3D feature volume data by processing the first 3D feature volume data using the multiple second processing layers. During a training phase, the multiple first processing layers and multiple second processing layers, with the back-projection module interposed in between, may be trained together to learn respective first weight data and second weight data.

According to a second aspect of the present disclosure, example methods and systems for tomographic image analysis are provided. One example method may comprise: obtaining first three-dimensional (3D) feature volume data and processing the first 3D feature volume data using an AI engine that includes multiple first processing layers, an interposing forward-projection module and multiple second processing layers. Example processing using the AI engine may involve: generating second 3D feature volume data by processing the first 3D feature volume data using the multiple first processing layers, transforming the second 3D volume data into 2D feature data using the forward-projection module and generating analysis output data by processing the 2D feature data using the multiple second processing layers. During a training phase, the multiple first processing layers and the multiple second processing layers, with the forward-projection module interposed in between, may be trained together to learn respective first weight data and second weight data.

DETAILED DESCRIPTION

FIG. 1is a schematic diagram illustrating example process flow110for radiotherapy treatment. Example process110may include one or more operations, functions, or actions illustrated by one or more blocks. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. In the example inFIG. 1, radiotherapy treatment generally includes various stages, such as an imaging system performing image data acquisition for a patient (see110); a radiotherapy treatment planning system (see130) generating a suitable treatment plan (see156) for the patient; and a treatment delivery system (see160) delivering treatment according to the treatment plan.

In more detail, at110inFIG. 1, image data acquisition may be performed using an imaging system to capture image data120associated with a patient (particularly the patient's anatomy). Any suitable medical image modality or modalities may be used, such as computed tomography (CT), cone beam computed tomography (CBCT), positron emission tomography (PET), magnetic resonance imaging (MRI), magnetic resonance tomography (MRT), single photon emission computed tomography (SPECT), any combination thereof, etc. For example, when CT or MRI is used, image data120may include a series of two-dimensional (2D) images or slices, each representing a cross-sectional view of the patient's anatomy, or may include volumetric or three-dimensional (3D) images of the patient, or may include a time series of 2D or 3D images of the patient (e.g., four-dimensional (4D) CT or 4D CBCT).

At130inFIG. 1, radiotherapy treatment planning may be performed during a planning phase to generate treatment plan156based on image data120. Any suitable number of treatment planning tasks or steps may be performed, such as segmentation, dose prediction, projection data prediction, treatment plan generation, etc. For example, segmentation may be performed to generate structure data140identifying various segments or structures may from image data120. In practice, a three-dimensional (3D) volume of the patient's anatomy may be reconstructed from image data120. The 3D volume that will be subjected to radiation is known as a treatment or irradiated volume that may be divided into multiple smaller volume-pixels (voxels)142. Each voxel142represents a 3D element associated with location (i, j, k) within the treatment volume. Structure data140may be include any suitable data relating to the contour, shape, size and location of patient's anatomy144, target146, organ-at-risk (OAR)148, or any other structure of interest (e.g., tissue, bone). For example, using image segmentation, a line may be drawn around a section of an image and labelled as target146(e.g., tagged with label=“prostate”). Everything inside the line would be deemed as target146, while everything outside would not.

In another example, dose prediction may be performed to generate dose data150specifying radiation dose to be delivered to target146(denoted “DTAR” at152) and radiation dose for OAR148(denoted “DOAR” at154). In practice, target146may represent a malignant tumor (e.g., prostate tumor, etc.) requiring radiotherapy treatment, and OAR148a proximal healthy structure or non-target structure (e.g., rectum, bladder, etc.) that might be adversely affected by the treatment. Target146is also known as a planning target volume (PTV). Although an example is shown inFIG. 1, the treatment volume may include multiple targets146and OARs148with complex shapes and sizes. Further, although shown as having a regular shape (e.g., cube), voxel142may have any suitable shape (e.g., non-regular). Depending on the desired implementation, radiotherapy treatment planning at block130may be performed based on any additional and/or alternative data, such as prescription, disease staging, biologic or radiomic data, genetic data, assay data, biopsy data, past treatment or medical history, any combination thereof, etc.

Based on structure data140and dose data150, treatment plan156may be generated include 2D fluence map data for a set of beam orientations or angles. Each fluence map specifies the intensity and shape (e.g., as determined by a multileaf collimator (MLC)) of a radiation beam emitted from a radiation source at a particular beam orientation and at a particular time. For example, in practice, intensity modulated radiotherapy treatment (IMRT) or any other treatment technique(s) may involve varying the shape and intensity of the radiation beam while at a constant gantry and couch angle. Alternatively or additionally, treatment plan156may include machine control point data (e.g., jaw and leaf positions), volumetric modulated arc therapy (VMAT) trajectory data for controlling a treatment delivery system, etc. In practice, block130may be performed based on goal doses prescribed by a clinician (e.g., oncologist, dosimetrist, planner, etc.), such as based on the clinician's experience, the type and extent of the tumor, patient geometry and condition, etc.

At160inFIG. 1, treatment delivery is performed during a treatment phase to deliver radiation to the patient according to treatment plan156. For example, radiotherapy treatment delivery system160may include rotatable gantry164to which radiation source166is attached. During treatment delivery, gantry164is rotated around patient170supported on structure172(e.g., table) to emit radiation beam168at various beam orientations according to treatment plan156. Controller162may be used to retrieve treatment plan156and control gantry164, radiation source166and radiation beam168to deliver radiotherapy treatment according to treatment plan156.

It should be understood that any suitable radiotherapy treatment delivery system(s) may be used, such as mechanic-arm-based systems, tomotherapy type systems, brachy therapy, sirex spheres, any combination thereof, etc. Additionally, examples of the present disclosure may be applicable to particle delivery systems (e.g., proton, carbon ion, etc.). Such systems may employ either a scattered particle beam that is then shaped by a device akin to an MLC, or a scanning beam of adjustable energy, spot size and dwell time. Also, OAR segmentation might be performed, and automated segmentation of the applicators might be desirable.

FIG. 2is a schematic diagram illustrating example imaging system200. Although one example is shown, imaging system200may have alternative or additional components depending on the desired implementation in practice. In the exampleFIG. 2, imaging system200includes radiation source210; detector220having pixel detectors disposed opposite to radiation source210along a projection line (defined below; see285); first set of fan blades230disposed between radiation source210and detector220; and first fan-blade drive235to hold fan blades230and set their positions.

Imaging system200may further include second set of fan blades240disposed between radiation source210and detector220, and second fan-blade drive245that holds fan blades240and sets their positions. The edges of fan blades230-240may be oriented substantially perpendicular to scan axis280and substantially parallel with a trans-axial dimension of detector220. Fan blades230-240are generally disposed closer to the radiation source210than detector220. They may be kept wide open to enable the full extent of detector220to be exposed to radiation but may be partially closed in certain situations.

Imaging system200may further include gantry250that holds at least radiation source210, detector220, and fan-blade drives235and245in fixed or known spatial relationships to one another, mechanical drive255that rotates gantry250about target object205disposed between radiation source210and detector220, with target object205being disposed between fan blades230and240on the one hand, and detector220on the other hand. The term “gantry” may cover all configurations of one or more structural members that can hold the above-identified components in fixed or known (but possibly movable) spatial relationships. For the sake of visual simplicity in the figure, the gantry housing, gantry support, and fan-blade support are not shown.

Additionally, imaging system200may include controller260, user interface265, and computer system270. Controller260may be electrically coupled to radiation source210, mechanical drive255, fan-blade drives235and245, detector220, and user interface265. User interface265may be configured to enable a user to at least initiate a scan of target object205, and to collect measured projection data from detector220. User interface265may be configured to present graphic representations of the measured projection data. Computer system270may be configured to perform any suitable operations, such as tomographic image reconstruction and analysis according to examples of the present disclosure.

Gantry250may be configured to rotate about target object205during a scan such that radiation source210, fan blades230and240, fan-blade drives235and245, and detector220circle around target object205. More specifically, gantry250may rotate these components about scan axis280. As shown inFIG. 2, scan axis280intersects with projection lines285, and is typically perpendicular to projection line285. Target object205is generally aligned in a substantially fixed relationship to scan axis280. The construction provides a relative rotation between projection line285on one hand, and scan axis280and target object205aligned thereto on the other hand, with the relative rotation being measured by an angular displacement value θ.

Mechanical drive255may be coupled to the gantry250to provide rotation upon command by controller260. The array of pixel detectors on detector220may be periodically read to acquire the data of the radiographic projections (also referred to as “measured projection data” below). Detector220has X-axis290and Y-axis295, which are perpendicular to each other. X-axis290is perpendicular to a plane defined by scan axis280and projection line285, and Y-axis295is parallel to this same plane. Each pixel on detector220is assigned a discrete (x, y) coordinate along X-axis290and Y-axis295. A smaller number of pixels are shown in the figure for the sake of visual clarity. Detector220may be centered on projection line285to enable full-fan imaging of target object205, offset from projection line285to enable half-fan imaging of target object205, or movable with respect to projection line285to allow both full-fan and half-fan imaging of target object205.

Conventionally, the task of reconstructing 3D volume data (e.g., representing target object205) from 2D projection data is generally non-trivial. As used herein, the term “2D projection data” (used interchangeably with “2D projection image”) may refer generally to data representing properties of illuminating radiation rays transmitted through target object205using any suitable imaging system200. In practice, 2D projection data may be set(s) of line integrals as output from imaging system200. The 2D projection data may contain imaging artifacts and originate from different 3D configurations due to movement, etc. Any artifacts in 2D projection data may affect the quality of subsequent diagnosis and radiotherapy treatment planning.

Artificial Intelligence (AI) Engines

According to examples of the present disclosure, tomographic image reconstruction and analysis may be improved using AI engines. As used herein, the term “AI engine” may refer to any suitable hardware and/or software component(s) of a computer system that are capable of executing algorithms according to any suitable AI model(s). Depending on the desired implementation, “AI engine” may be a machine learning engine based on machine learning model(s), deep learning engine based on deep learning model(s), etc. In general, deep learning is a subset of machine learning in which multi-layered neural networks may be used for feature extraction as well as pattern analysis and/or classification. A deep learning engine may include a hierarchy of “processing layers” of nonlinear data processing that include an input layer, an output layer, and multiple (i.e., two or more) “hidden” layers between the input and output layers. Processing layers may be trained from end-to-end (e.g., from the input layer to the output layer) to extract feature(s) from an input and classify the feature(s) to produce an output (e.g., classification label or class).

Depending on the desired implementation, any suitable AI model(s) may be used, such as convolutional neural network, recurrent neural network, deep belief network, generative adversarial network (GAN), autoencoder(s), variational autoencoder(s), long short-term memory architecture for tracking purposes, or any combination thereof, etc. In practice, a neural network is generally formed using a network of processing elements (called “neurons,” “nodes,” etc.) that are interconnected via connections (called “synapses,” “weight data,” etc.). For example, convolutional neural networks may be implemented using any suitable architecture(s), such as UNet, LeNet, AlexNet, ResNet, VNet, DenseNet, OctNet, etc. A “processing layer” of a convolutional neural network may be a convolutional layer, pooling layer, un-pooling layer, rectified linear units (ReLU) layer, fully connected layer, loss layer, activation layer, dropout layer, transpose convolutional layer, concatenation layer, or any combination thereof, etc. Due to the substantially large amount of data associated with tomographic image data, non-uniform sampling of 3D volume data may be implemented, such as using OctNet, patch/block-wise processing, etc.

In more detail,FIG. 3is a schematic diagram illustrating example system300for tomographic image reconstruction and analysis using respective AI engines301-302. As used herein, the term “tomographic image” may refer generally to any suitable data generated by a process of computed tomography using imaging modality or modalities, such as CT, CBCT, PET, MRT, SPECT, etc. In practice, tomographic images may be 2D (e.g., slice image depicting a cross section of an object); 3D (e.g., volume data representing the object), or 4D (e.g., 3D volume data over time).

At301inFIG. 3(left pathway), a first AI engine may be trained to perform tomographic image reconstruction. First AI engine301may include first processing layers forming a first neural network labelled “A” (see311), an interposing back-projection module (see312) and second processing layers forming a second neural network labelled “B” (see313). Network “A”311includes multiple (N1>1) first processing layers denoted as A1, A2, . . . AN1, while network “B”313includes multiple (N2>1) second processing layers denoted as B1, B2, . . . BN2.

As will be described further usingFIG. 4andFIG. 5, first AI engine301may be trained to perform 2D-to-3D transformation by transforming input=2D projection image data (see310) into output=3D feature volume data (see340). During a training phase, first processing layers311and second processing layers313may be linked by back-projection module312and trained together. This way, during subsequent inference phase, first AI engine301may take advantage of data in both 2D projection space and 3D volume space during tomographic image reconstruction.

At302inFIG. 3(right pathway), a second AI engine may be trained to perform tomographic image analysis. Second AI engine302may include first processing layers forming a first neural network labelled “C” (see314), an interposing forward-projection module (see315) and second processing layers forming a second neural network labelled “D” (see316). Network “C”314includes multiple (M1>1) first processing layers denoted as C1, C2, . . . CM1, while network “D”316includes multiple (M2>1) second processing layers denoted as D1, D2, . . . DM2.

As will be described further usingFIG. 6andFIG. 7, second AI engine302may be trained to transform input=3D feature volume data (see340) to 2D feature data (see360) for analysis. During a training phase, first processing layers314(C1, C2, . . . CM1) and second processing layers316(D1, D2, . . . DM2) may be linked by forward-projection module315and trained together. This way, during subsequent inference phase, second AI engine302may take advantage of data in both 2D projection space and 3D volume space during tomographic image analysis. In practice, network “D”316may be trained to perform analysis based on both 2D feature data360and original 2D projection data310(see dashed line inFIG. 3).

According to examples of the present disclosure, AI engine301/302may learn from data in both 2D projection space and 3D volume space. This way, the transformation between the 2D projection space and the 3D volume space may be performed in a substantially lossless manner to reduce the likelihood of losing the necessary features compared to conventional reconstruction approaches. Using examples of the present disclosure, different building blocks for tomographic image reconstruction may be combined with neural networks (i.e., an example “AI engine”). Feasible fields of application may include automatic segmentation of 3D volume data or 2D projection data, object/feature detection, classification, data enhancement (e.g., completion, artifact reduction), any combination thereof, etc.

Unlike conventional approaches, examples of the present disclosure take advantage of both 3D space of volume data as well as the 2D space of projection data. Since the 2D projection data and 3D volume data are two representations of the same target object, it may be assumed that the analysis or processing may be beneficial in one or the other. In practice, output 3D volume data340/350may be a 3D/4D volume with CT (HU) values, dose data, segmentation/structure data, deformation vectors, 4D time-resolved volume data, any combination thereof, etc. Output 2D feature data360(projections) may be X-Ray intensity data, attenuation data (both potentially energy resolved), modifications thereof (removed objects), segments, any combination thereof, etc.

According to examples of the present disclosure, a first hypothesis is that raw data for tomographic images contains more information than the resulting 3D volume data. In practice, image reconstruction may be tweaked for different tasks, such as noise suppression, spatial resolution, edge enhancement, Hounsfield Units (HU) accuracy, any combination thereof, etc. These tweaks usually have tradeoffs, meaning that information that is potentially useful for any subsequent image analysis (e.g., segmentation) is lost. Other information (e.g., motion) may be suppressed by the reconstruction. In practice, once image reconstruction is performed, the 2D projection data is only reviewed in more detail after there are problems with seeing or understanding features in the 3D volume image data (e.g., metal or artifacts).

According to examples of the present disclosure, a second hypothesis is that the analysis of 2D projection data profits from knowledge about the 3D image domain. A classic example may involve a prior reconstruction with volume manipulation followed by forward projection for, for example, background subtraction and detection of a tumor. Unlike conventional approaches, the 2D-3D relationship may be intrinsic or integral part of the machine learning engine. Processing layers may learn any suitable information in 2D projection data and 3D volume data to fulfil the task.

Depending on the desired implementation, first AI engine301and second AI engine302may be trained and deployed independently (seeFIGS. 4-7). Alternatively, first AI engine301and second AI engine302may be trained and deployed in an integrated form (seeFIG. 8). First AI engine301and second AI engine302may be implemented using computer system270, or separate computer systems. The computer system(s) may be connected to controller260of imaging system200via a local network or wide area network (e.g., Internet). As will be described usingFIG. 10, computer system270may provide a planning-as-a-service (PaaS) for access by users (e.g., clinicians) to perform tomographic image reconstruction and/or analysis.

Tomographic Image Reconstruction

According to a first aspect of the present disclosure, first AI engine301inFIG. 3may be trained to perform tomographic image reconstruction. Some examples will be explained usingFIG. 4, which is a flowchart of example process400for a computer system to perform tomographic image reconstruction using first AI engine301. Example process400may include one or more operations, functions, or actions illustrated by one or more blocks, such as410to440. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. Example process400may be implemented using any suitable computer system(s), an example of which will be discussed usingFIG. 10.

At410inFIG. 4, 2D projection data310associated with a target object (e.g., patient's anatomy) may be obtained. Here, the term “obtain” may refer generally to receiving 2D or retrieving projection data310from a source (e.g., controller260, storage device, another computer system, etc.). As explained usingFIG. 2, 2D projection data310may be acquired using imaging system200by rotating radiation source210and detector220about target object205.

In practice, 2D projection data310may be raw data from controller260or pre-processed. Example pre-processing algorithms may include defect pixel correction, dark field correction, conversion from transmission integrals into attenuation integrals (e.g., log normalization with air norm), scatter correction, beam hardening correction, decimation, etc. 2D projection data310may be multi-channel projection data that includes various pre-processed instances and additional projections from the acquisition sequence. It should be understood that any suitable tomographic imaging modality or modalities may be used to capture 2D projection data310, such as X-ray tomography (e.g., CT, and CBCT), PET, SPECT, MRT, etc. Although digital tomosynthesis (DTS) imaging is a no direct tomography method, the same principle may be applicable. This is because DTS also uses the relative geometry between the projections to calculate a relative 3D reconstruction with limited (dependent on the scan arc angle) resolution in imaging direction.

At420inFIG. 4, 2D projection data310may be processed using first processing layers (A1, A2, . . . AN1) of pre-processing network “A”311to generate 2D feature data320. In practice, network “A”311may include a convolutional neural network with convolutional layer(s), pooling layer(s), etc. All projections in 2D projection data310may be processed using network “A”311, or several instances for different subsets of the projections.

At430inFIG. 4, first 3D feature volume data330may be reconstructed from 2D feature data320using back-projection module312. As used herein, “back projection” may refer generally to transformation from 2D projection space to 3D volume space. Any suitable reconstruction algorithm(s) may be implemented by back-projection module312, such as non-iterative reconstruction (e.g., filtered back-projection), iterative reconstruction (e.g., algebraic and statistical based reconstruction), etc. In practice, 2D feature data320may represent a multi-channel output of network “A”311. In this case, back-projection module312may perform multiple back-projection operations on respective channels to form the corresponding 3D feature volume data330with a multi-channel 3D representation.

At440inFIG. 4, first 3D feature volume data330may be processed using second processing layers (B1, B2, . . . , BN2) of network “B”313to generate second 3D feature volume data340. Depending on the desired implementation, network “B”313may be implemented based on a UNet architecture. Having a general “U” shape, the left path of UNet is known as an “encoding path” or “contracting path,” where high-order features are extracted at several down-sampled resolutions.

In one example, network “B”313may be configured to implement the encoding path of UNet, in which case second processing layers (B1, B2, . . . , BN2) may include convolution layer(s) and pooling layer(s) forming a volume processing chain. Network “B”313may be seen as a type of encoder that finds another representation of 2D projection data310. As will be discussed usingFIG. 6, the right path of UNet is known as a “decoding path” or “expansive path,” and may be implemented by network “C”314of second AI engine302.

(b) Training Phase

Depending on the desired implementation, network “A”311and network “B”313may be trained using a supervised learning approach. The aim of training phase501is to train AI engine301to map input training data=2D projection data510to output training data=3D feature volume data520, which represents the desired outcome or belonging volume. In practice, 3D feature volume data520represents labels for supervised learning, and annotations such as contours may be used as labels. For each iteration, a subset or all of 2D projection data510may be processed using network “A”311to generate 2D feature data530, back-projection module312to generate 3D feature volume data540and network “B”313to generate a predicted outcome (see550).

Training phase501inFIG. 5may be guided by estimating and minimizing a loss between predicted outcome550and desired outcome specified by output training data520. See comparison operation at560inFIG. 5. This way, first weight data (WA1, WA2, . . . , WAN) and second weight data (WB1, WB2, . . . , WBN) may be improved during training phase501, such as through backward propagation of loss, etc. A simple example of a loss function would be mean squared error between true and predicted outcome, but the loss function could have more complex formulas (e.g., dice loss, jaccard loss, focal loss, etc.). This loss can be estimated from the output of the model, or from any discrete point within the model.

Depending on the desired implementation, network “A”311may be trained to perform pre-processing on 2D projection data310, such as by applying convolution filter(s) on 2D projection data310, etc. In general, network “A”311may learn any suitable feature transformation that is necessary to enable network “B”313to generate its output (i.e., second 2D feature volume data350). Using the Feldkamp-Davis-Kress (FDK) reconstruction algorithm, for example, network “A”311may be trained to learn a convolution filter part of the FDK algorithm. In this case, network “B”313may be trained to generate second 2D feature volume data350that represents a 3D FDK reconstruction output. During training phase501, network “A”311may learn any suitable task(s) that may be best performed on the line integrals based on 2D projection data310in the 2D projection space.

Once trained and validated, first AI engine301may be deployed to perform tomographic image reconstruction for current patient(s) during inference phase502. As described usingFIG. 3andFIG. 4, first AI engine301may operate in both 2D projection space and 3D volume space to transform input=2D projection data310into output=feature volume data340using network “A”311, back-projection module312and network “B”313. Various examples associated with the 2D-to-3D transformation using first AI engine301have been discussed usingFIG. 3andFIG. 4and will not be repeated here for brevity.

Depending on the desired implementation, network “B”313inFIG. 3andFIG. 5may represent a combination of both encoding network “B”313and decoding network “C”314. These networks313-314collectively form an auto-encoding network to find a 3D representation of 2D projection data310in the form of output=3D feature volume data340/350. Once trained and validated, tomographic image reconstruction may be performed using network “A”311, back-projection module212, and combined network313-314. Depending on the desired implementation, the final output (i.e., feature volume data340/350) of combined network313-314may be used as an input to AI engine(s) or algorithm(s) for 3D analysis. The output of the combined network313-314may also include 4D time-resolved volume data or other suitable representation.

Tomographic Image Analysis

According to a second aspect of the present disclosure, second AI engine302inFIG. 3may be trained to perform tomographic image analysis. Some examples will be explained usingFIG. 6, which is a flowchart of example process600for a computer system to perform tomographic image analysis using second AI engine302. Example process600may include one or more operations, functions, or actions illustrated by one or more blocks, such as610to640. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. Example process600may be implemented using any suitable computer system(s), an example of which will be discussed usingFIG. 10. In the following, input feature volume data340and output feature volume data350will be used as example “first” and “second” 3D “feature volume data” from the perspective of network “C”314.

At610inFIG. 6, input=3D feature volume data340may be obtained. In one example, input 3D feature volume data340may be an output of first AI engine301, or algorithmic equivalent(s) thereof. In the latter case, any suitable algorithm(s) may be used, such as 3D or 4D reconstruction algorithm, etc. The term “obtain” may refer generally to receiving or retrieving 3D feature volume data340from a source (e.g., first AI engine301, storage device, another computer system, etc.). Input 3D feature volume data340may be generated based on 2D projection data210acquired using imaging system200by rotating radiation source210and detector220about target object205.

At620inFIG. 6, input 3D feature volume data340may be processed using first processing layers (C1, C2, . . . , CM1) of network “C”314to generate output 3D feature volume data350. In general, network “C”314is trained to prepare features that may be forward-projected by forward-projection module315and processed by network “D”316, such as to reproduce input projections or segments. Depending on the desired implementation, network “C”314may be implemented based on a UNet architecture. The right path of UNet is known as a “decoding path” or “expansive path,” where features at lower resolution are upsampled to a higher resolution.

In one example, network “C”314may be configured to implement a decoding path, in which case first processing layers (C1, C2, . . . , CM1) may include convolution layer(s) and un-pooling layer(s). When connected with network “B”313inFIG. 3, both networks313-314may be seen as a type of encoder-decoder by using network “B”313to encode and network “C”314to decode. In this case, output=3D feature volume data350may have a potentially higher (up-sampled) resolution or denoised, higher dimensional features compared to input=3D feature volume data340of network “C”314(i.e., output of network “B”313).

At630inFIG. 6, 3D feature volume data350(i.e., output of network “C”314) may be forward-projected or transformed into 2D feature data360using forward-projection module315. As used herein, “forward projection” may refer generally to a transformation from the 3D volume space to the 2D projection space. Forward projection (also known as synthesizing projection data) may include data such as attenuation path integrals (primary signal), Rayleigh scatter and Compton scatter. Forward projection module315may implement any suitable algorithm(s), such as monochromatic or polychromatic; source-driven or destination-drive; voxel-based or blob-based; and use Ray Tracing, Monte Carlo, finite element methods, etc.

At640inFIG. 6, 2D feature data360may be processed using second processing layers (D1, D2, . . . , DM2) of network “D”316to generate analysis output data. Depending on the analysis performed by network “D”, any suitable architecture may be used, such as UNet, LeNet, AlexNet, ResNet, V-net, DenseNet, etc. Example analysis performed by network “D”316will be discussed below.

(b) Training Phase

Depending on the desired implementation, network “C”314and network “D”316may be trained using a supervised learning approach. The aim of training phase701is to train AI engine302to map input training data=3D feature volume data710to output training data=analysis output data720, which represents the desired outcome. For each iteration, a subset of 3D feature volume data710may be processed using (a) network “C”314to generate decoded 3D feature volume data730, (b) forward-projection projection module315to generate 2D feature data740and (c) network “D”316to generate a predicted outcome (see750).

Similar to the example inFIG. 5, training phase701inFIG. 7may be guided by estimating and minimizing a loss between predicted outcome750and desired outcome specified by output training data720. See comparison operation at760inFIG. 7. This way, first weight data (WC1, WC2, . . . , WCM) and second weight data (WD1, WD2, . . . , WDM) may be improved during training phase701, such as through backward propagation of loss, etc. Again, a simple loss function (e.g., mean squared error) or more complex function(s) may be used.

Depending on the desired implementation, network “D”316may be trained using training data710-720to generate analysis output data associated with one or more of the following: automatic segmentation, object detection (e.g., organ or bone), feature detection (e.g., edge/contour of an organ, 3D small-scale structure located within bone(s) such as skull, etc.), image artifact suppression, image enhancement (e.g., resolution enhancement using super-resolution), de-truncation by learning volumetric image content (voxels), prediction of moving 2D segments, object or tissue removal (e.g., bone, patient's table or immobilization devices, etc.), any combination thereof, etc. These examples will also be discussed further below.

Once trained and validated, second AI engine302may be deployed to perform tomographic image analysis for current patient(s) during inference phase702. As described usingFIG. 3andFIG. 6, second AI engine302may operate in both 2D projection space and 3D volume space to transform input=3D feature volume data340into output=analysis output data370using network “C”314, forward-projection module315and network “D”316. Example details relating to tomographic image analysis using second AI engine302have been discussed usingFIG. 3andFIG. 6and will not be repeated here for brevity.

Integrated Tomographic Image Reconstruction and Analysis

According to a third aspect of the present disclosure, first and second AI engines301-302inFIG. 3may be trained together to perform integrated tomographic image reconstruction and analysis. Some examples will be discussed usingFIG. 8, which is a schematic diagram illustrating example training phase800of AI engines301-302for integrated tomographic image reconstruction and analysis.

In the example inFIG. 8, AI engines301-302may be connected to form an integrated AI engine, which includes networks “A”311and “B”313that are interposed with back-projection module312, followed by networks “C”314and “D”316that are interposed with forward-projection module315. The aim of training phase801is to train integrated AI engine301-302to map input training data=2D projection data810to output training data=analysis output data820, which represents the desired outcome. For each iteration, a subset of 2D projection data810may be processed using network “A”311, back-projection module312, network “B”313, network “C”314, forward-projection module315and network “D”316to generate a predicted outcome (see830).

Training phase801inFIG. 8may be guided by estimating and minimizing a loss between predicted outcome830and desired outcome specified by output training data820. Using an end-to-end training approach, weight data associated with respective networks311,313-314and316may be improved, such as through backward propagation of loss, etc. By embedding both back-projection module312and forward-projection module315, training phase901may be guided by end-to-end loss function(s) in 2D projection space and/or 3D volume space. See comparison860between output training data820and predicted outcome830inFIG. 8.

In the example inFIG. 8, an optional copy of data from first AI engine301may be transported to second AI engine302to “skip” processing layer(s) in between. This provides shortcuts for the data flow, such as to let high-frequency features skip or bypass lower levels of a neural network. In one example (see840), an optional copy of data from one processing layer (Ai) in network “A”311may be provided to another processing layer (Dj) in network “D”316. A practical scenario would be scatter data that is removed by network “A”311skips networks “B”313and “C”314and added again to reproduce the input projections or patient motion removed by network “B”313. This way, static image data may be generated and network “C”314may reproduce the input. In another example (see850), an optional copy of data from one processing layer (Bi) in network “B”313may be provided to another processing layer (Cj) in network “C”314. This skipping approach is one of the possibilities provided by convolution neural networks provide the possibility to skip layers.

Depending on the desired implementation, first AI engine301and/or second AI engine302inFIGS. 3-8may be implemented to facilitate at least one of the following:(a) Identifying imaging artifact(s) associated with 2D projection data310and/or 3D feature volume data340/350, such as when the exact source of artifact(s) is unknown. In one approach, an auto-encoding approach may be implemented using both AI engines301-302. The loss function used during training phase701may be used to ensure 2D projection data310and analysis output data370are substantially the same, and volume data330-350in between is of the desired quality (e.g., reduced noise or motion artifacts). Another approach is to provide an ideal reconstruction as label and train the model to predict substantially artifact-free volume data from reduced or deteriorated (e.g., simulated noise or scatter) projection data.(b) Identifying region(s) of movement associated with 2D projection data310and/or 3D feature volume data340/350. In this case, training data710-720may include 2D/3D information where motion occurs to train network “D”316to identify region(s) of movement.(c) Identifying region(s) with an artifact associated with 2D projection data310and/or 3D feature volume data340/350. In this case, training data710-720may include segmented artifacts to train network “D”316to identify region(s) with artifacts.(d) Identifying anatomical structure(s) and/or non-anatomical structure(s) from 2D projection data310and/or 3D feature volume data340/350. Through automatic segmentation, anatomical structure(s) such as tumor(s) and organ(s) may be identified. Non-anatomical structure(s) may include implant(s), fixation device(s) and other materials in 2D/3D image regions. In this case, training data710-720may include data identifying such anatomical structure(s) and/or non-anatomical structure(s).(e) Reducing noise associated with 2D projection data310and/or 3D feature volume data340/350. In practice, this may involve identifying a sequence of projections for further processing, such as marker tracking, soft tissue tracking.(f) Tracking movement of a patient's structure identifiable from 2D projection data310or 3D feature volume data340/350. A feasible output of first AI engine301and second AI engine302may be a set of projections where each pixel indicates the probability of identifying a fiducial (or any other structure) center point or segment. This would provide the position of the structure for each projection. Advantage of this approach is that occurrence probability may be combined in 3D volume space to make a dependent 2D prediction for each projection. Any suitable tracking approach may be used, such as using 3D volume data in the form of long short-term memory (LSTM), etc.(g) Binning 2D slices associated with 2D projection data310to different movement bins (or phases). By identifying the bins, network “D”316may be trained to use data belonging to certain bin.(h) Generating 4D image data with movement associated with 2D projection data310or 3D feature volume data340/350. In this case, network “D”316may be trained to compute one volume with several channels (4D) for different bins in (g) to resolve motion. Other possibilities include using a variational auto-encoder in 3D volume space (e.g., networks “B”313and “C”314) to learn a deformation model.

Automatic segmentation using AI engines301-302should be contrasted against conventional manual approaches. For example, it usually requires a team of highly skilled and trained oncologists and dosimetrists to manually delineate structures of interest by drawing contours or segmentations on image data120. These structures are manually reviewed by a physician, possibly requiring adjustment or re-drawing. In many cases, the segmentation of critical organs can be the most time-consuming part of radiation treatment planning. After the structures are agreed upon, there are additional labor-intensive steps to process the structures to generate a clinically-optimal treatment plan specifying treatment delivery data such as beam orientations and trajectories, as well as corresponding 2D fluence maps. These steps are often complicated by a lack of consensus among different physicians and/or clinical regions as to what constitutes “good” contours or segmentation. In practice, there might be a huge variation in the way structures or segments are drawn by different clinical experts. The variation may result in uncertainty in target volume size and shape, as well as the exact proximity, size and shape of OARs that should receive minimal radiation dose. Even for a particular expert, there might be variation in the way segments are drawn on different days.

Example Treatment Plan

FIG. 9is a schematic diagram of example treatment plan156/900generated or improved based on output data(s) of AI engine301/302inFIG. 3. Treatment plan156may be delivered using any suitable treatment delivery system that includes radiation source910to project radiation beam920onto treatment volume960representing the patient's anatomy at various beam angles930.

Although not shown inFIG. 9for simplicity, radiation source910may include a linear accelerator to accelerate radiation beam920and a collimator (e.g., MLC) to modify or modulate radiation beam920. In another example, radiation beam920may be modulated by scanning it across a target patient in a specific pattern with various energies and dwell times (e.g., as in proton therapy). A controller (e.g., computer system) may be used to control the operation of radiation source920according to treatment plan156.

During treatment delivery, radiation source910may be rotatable using a gantry around a patient, or the patient may be rotated (as in some proton radiotherapy solutions) to emit radiation beam920at various beam orientations or angles relative to the patient. For example, five equally-spaced beam angles930A-E (also labelled “A,” “B,” “C,” “D” and “E”) may be selected using a deep learning engine configured to perform treatment delivery data estimation. In practice, any suitable number of beam and/or table or chair angles930(e.g., five, seven, etc.) may be selected. At each beam angle, radiation beam920is associated with fluence plane940(also known as an intersection plane) situated outside the patient envelope along a beam axis extending from radiation source910to treatment volume960. As shown inFIG. 9, fluence plane940is generally at a known distance from the isocenter.

In addition to beam angles930A-E, fluence parameters of radiation beam920are required for treatment delivery. The term “fluence parameters” may refer generally to characteristics of radiation beam920, such as its intensity profile as represented using fluence maps (e.g.,950A-E for corresponding beam angles930A-E). Each fluence map (e.g.,950A) represents the intensity of radiation beam920at each point on fluence plane940at a particular beam angle (e.g.,930A). Treatment delivery may then be performed according to fluence maps950A-E, such as using IMRT, etc. The radiation dose deposited according to fluence maps950A-E should, as much as possible, correspond to the treatment plan generated according to examples of the present disclosure.

Computer System

Examples of the present disclosure may be deployed in any suitable manner, such as a standalone system, web-based planning-as-a-service (PaaS) system, etc. In the following, an example computer system (also known as a “planning system”) will be described usingFIG. 10, which is a schematic diagram illustrating example network environment1000in which tomographic image reconstruction and/or tomographic image analysis may be implemented. Depending on the desired implementation, network environment1000may include additional and/or alternative components than that shown inFIG. 10. Examples of the present disclosure may be implemented by hardware, software or firmware or a combination thereof.

Processor1020is to perform processes described herein with reference toFIG. 1toFIG. 9. Computer-readable storage medium1030may store computer-readable instructions1032which, in response to execution by processor1020, cause processor1020to perform various processes described herein. Computer-readable storage medium1030may further store any suitable data1034, such as data relating to AI engines, training data, weight data, 2D projection data, 3D volume data, analysis output data, etc. In the example inFIG. 10, computer system1010may be accessible by multiple user devices1041-1043via any suitable physical network (e.g., local area network, wide area network, etc.) In practice, user devices1041-1043may be operated by various users located at any suitable clinical site(s).

Computer system1010may be implemented using a multi-tier architecture that includes web-based user interface (UI) tier1021, application tier1022, and data tier1023. UI tier1021may be configured to provide any suitable interface(s) to interact with user devices1041-1043, such as graphical user interface (GUI), command-line interface (CLI), application programming interface (API) calls, any combination thereof, etc. Application tier1022may be configured to implement examples of the present disclosure. Data tier1023may be configured to facilitate data access to and from storage medium1030. By interacting with UI tier1021, user devices1041-1043may generate and send respective service requests1051-1053for processing by computer system1010. In response, computer system1010may perform examples of the present disclosure generate and send service responses1061-1063to respective user devices1041-1043.

Depending on the desired implementation, computer system1010may be deployed in a cloud computing environment, in which case multiple virtualized computing instances (e.g., virtual machines, containers) may be configured to implement various functionalities of tiers1021-1023. The cloud computing environment may be supported by on premise cloud infrastructure, public cloud infrastructure, or a combination of both. Computer system1010may be deployed in any suitable manner, including a service-type deployment in an on-premise cloud infrastructure, public cloud infrastructure, a combination thereof, etc. Computer system1010may represent a computation cluster that includes multiple computer systems among which various functionalities are distributed. Computer system1010may include any alternative and/or additional component(s) not shown inFIG. 10, such as graphics processing unit (GPU), message queues for communication, blob storage or databases, load balancer(s), specialized circuits, etc.

The foregoing detailed description has set forth various embodiments of the devices and/or processes via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. Throughout the present disclosure, the terms “first,” “second,” “third,” etc. do not denote any order of importance, but are rather used to distinguish one element from another.

Those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure.

Although the present disclosure has been described with reference to specific exemplary embodiments, it will be recognized that the disclosure is not limited to the embodiments described, but can be practiced with modification and alteration within the spirit and scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense.