Patent Description:
Radiotherapy is an important part of a treatment for reducing or eliminating unwanted tumors from patients. Unfortunately, applied radiation does not inherently discriminate between an unwanted tumor and any proximal healthy structures such as organs, etc. This necessitates careful administration to restrict the radiation to the tumor (i.e., target). Ideally, the goal is to deliver a lethal or curative radiation dose to the tumor, while maintaining an acceptable dose level in the proximal healthy structures. However, to achieve this goal, conventional radiotherapy treatment planning may be time and labor intensive.

<CIT> discloses machine learning algorithms for use in radiation therapy, while taking advantage of the principle of transfer learning.

Various examples will be discussed below, such as with reference to <FIG> and <FIG>.

According to examples of the present disclosure, methods and systems for radiotherapy treatment planning based on deep transfer learning are provided. Various examples will be discussed below, such as with reference to <FIG> and <FIG>.

The technical details set forth in the following description enable a person skilled in the art to implement one or more embodiments of the present disclosure.

<FIG> is a schematic diagram illustrating example process flow <NUM> for radiotherapy treatment. Example process <NUM> may include one or more operations, functions, or actions illustrated by one or more blocks. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. In the example in <FIG>, radiotherapy treatment generally includes various stages, such as an imaging system performing image data acquisition for a patient (see <NUM>); a radiotherapy treatment planning system (see <NUM>) generating a suitable treatment plan (see <NUM>) for the patient; and a treatment delivery system (see <NUM>) delivering treatment according to the treatment plan.

In more detail, at <NUM> in <FIG>, image data acquisition may be performed using an imaging system to capture image data <NUM> associated with a patient (particularly the patient's anatomy). Any suitable medical image modality or modalities may be used, such as computed tomography (CT), cone beam computed tomography (CBCT), positron emission tomography (PET), magnetic resonance imaging (MRI), single photon emission computed tomography (SPECT), any combination thereof, etc. For example, when CT or MRI is used, image data <NUM> may include a series of two-dimensional (2D) images or slices, each representing a cross-sectional view of the patient's anatomy, or may include volumetric or three-dimensional (3D) images of the patient, or may include a time series of 2D or 3D images of the patient (e.g., four-dimensional (4D) CT or 4D CBCT).

At <NUM> in <FIG>, radiotherapy treatment planning may be performed during a planning phase to generate treatment plan <NUM> based on image data <NUM>. Any suitable number of treatment planning tasks or steps may be performed, such as segmentation, dose prediction, projection data prediction, treatment plan generation, etc. For example, segmentation may be performed to generate structure data <NUM> identifying various segments or structures may from image data <NUM>. In practice, a three-dimensional (3D) volume of the patient's anatomy may be reconstructed from image data <NUM>. The 3D volume that will be subjected to radiation is known as a treatment or irradiated volume that may be divided into multiple smaller volume-pixels (voxels) <NUM>. Each voxel <NUM> represents a 3D element associated with location (i, j, k) within the treatment volume. Structure data <NUM> may be include any suitable data relating to the contour, shape, size and location of patient's anatomy <NUM>, target <NUM>, organ-at-risk (OAR) <NUM>, or any other structure of interest (e.g., tissue, bone). For example, using image segmentation, a line may be drawn around a section of an image and labelled as target <NUM> (e.g., tagged with label = "prostate"). Everything inside the line would be deemed as target <NUM>, while everything outside would not.

In another example, dose prediction may be performed to generate dose data <NUM> specifying radiation dose to be delivered to target <NUM> (denoted "DTAR" at <NUM>) and radiation dose for OAR <NUM> (denoted "DOAR" at <NUM>). In practice, target <NUM> may represent a malignant tumor (e.g., prostate tumor, etc.) requiring radiotherapy treatment, and OAR <NUM> a proximal healthy structure or non-target structure (e.g., rectum, bladder, etc.) that might be adversely affected by the treatment. Target <NUM> is also known as a planning target volume (PTV). Although an example is shown in <FIG>, the treatment volume may include multiple targets <NUM> and OARs <NUM> with complex shapes and sizes. Further, although shown as having a regular shape (e.g., cube), voxel <NUM> may have any suitable shape (e.g., non-regular). Depending on the desired implementation, radiotherapy treatment planning at block <NUM> may be performed based on any additional and/or alternative data, such as prescription, disease staging, biologic or radiomic data, genetic data, assay data, biopsy data, past treatment or medical history, any combination thereof, etc..

Based on structure data <NUM> and dose data <NUM>, treatment plan <NUM> may be generated include 2D fluence map data for a set of beam orientations or angles. Each fluence map specifies the intensity and shape (e.g., as determined by a multileaf collimator (MLC)) of a radiation beam emitted from a radiation source at a particular beam orientation and at a particular time. For example, in practice, intensity modulated radiotherapy treatment (IMRT) or any other treatment technique(s) may involve varying the shape and intensity of the radiation beam while at a constant gantry and couch angle. Alternatively or additionally, treatment plan <NUM> may include machine control point data (e.g., jaw and leaf positions), volumetric modulated arc therapy (VMAT) trajectory data for controlling a treatment delivery system, etc. In practice, block <NUM> may be performed based on goal doses prescribed by a clinician (e.g., oncologist, dosimetrist, planner, etc.), such as based on the clinician's experience, the type and extent of the tumor, patient geometry and condition, etc..

At <NUM> in <FIG>, treatment delivery is performed during a treatment phase to deliver radiation to the patient according to treatment plan <NUM>. For example, radiotherapy treatment delivery system <NUM> may include rotatable gantry <NUM> to which radiation source <NUM> is attached. During treatment delivery, gantry <NUM> is rotated around patient <NUM> supported on structure <NUM> (e.g., table) to emit radiation beam <NUM> at various beam orientations according to treatment plan <NUM>. Controller <NUM> may be used to retrieve treatment plan <NUM> and control gantry <NUM>, radiation source <NUM> and radiation beam <NUM> to deliver radiotherapy treatment according to treatment plan <NUM>.

It should be understood that any suitable radiotherapy treatment delivery system(s) may be used, such as mechanic-arm-based systems, tomotherapy type systems, brachy therapy, sirex spheres, any combination thereof, etc. Additionally, examples of the present disclosure may be applicable to particle delivery systems (e.g., proton, carbon ion, etc.). Such systems may employ either a scattered particle beam that is then shaped by a device akin to an MLC, or a scanning beam of adjustable energy, spot size and dwell time. Also, OAR segmentation might be performed, and automated segmentation of the applicators might be desirable.

Conventionally, radiotherapy treatment planning at block <NUM> in <FIG> is time and labor intensive. For example, it usually requires a team of highly skilled and trained oncologists and dosimetrists to manually delineate structures of interest by drawing contours or segmentations on image data <NUM>. These structures are manually reviewed by a physician, possibly requiring adjustment or re-drawing. In many cases, the segmentation of critical organs can be the most time-consuming part of radiation treatment planning. After the structures are agreed upon, there are additional labor-intensive steps to process the structures to generate a clinically-optimal treatment plan specifying treatment delivery data such as beam orientations and trajectories, as well as corresponding 2D fluence maps. These steps are often complicated by a lack of consensus among different physicians and/or clinical regions as to what constitutes "good" contours or segmentation. In practice, there might be a huge variation in the way structures or segments are drawn by different clinical experts. The variation may result in uncertainty in target volume size and shape, as well as the exact proximity, size and shape of OARs that should receive minimal radiation dose. Even for a particular expert, there might be variation in the way segments are drawn on different days.

According to examples of the present disclosure, artificial intelligence (Al) techniques may be applied to ameliorate various challenges associated with radiotherapy treatment planning. In particular, deep learning engine(s) may be used to automate radiotherapy treatment planning step(s). Throughout the present disclosure, the term "deep learning" may refer generally to a class of approaches that utilizes many layers or stages of nonlinear data processing for feature learning as well as pattern analysis and/or classification. The "deep learning model" may refer to a hierarchy of "layers" of nonlinear data processing that include an input layer, an output layer, and multiple (i.e., two or more) "hidden" layers between the input and output layers. These layers may be trained from end-to-end (e.g., from the input layer to the output layer) to extract feature(s) from an input and classify the feature(s) to produce an output (e.g., classification label or class).

Accordingly, the term "deep learning engine" may refer to any suitable hardware and/or software component(s) of a computer system that are capable of executing algorithms according to any suitable deep learning model(s). Depending on the desired implementation, any suitable deep learning model(s) may be used, such as convolutional neural network, recurrent neural network, deep belief network, generative adversarial network (GAN), or any combination thereof, etc. In practice, a neural network is generally formed using a network of processing elements (called "neurons," "nodes," etc.) that are interconnected via connections (called "synapses," "weights," etc.). For example, convolutional neural networks may be implemented using any suitable architecture(s), such as U-net, LeNet, AlexNet, ResNet, V-net, DenseNet, etc. In this case, a "layer" of a convolutional neural network may be a convolutional layer, pooling layer, rectified linear units (ReLU) layer, fully connected layer, loss layer, activation layer, dropout layer, transpose convolutional layer, concatenation layer, or any combination thereof, etc. In practice, the U-net architecture includes a contracting path (left side) and an expansive path (right side). The contracting path includes repeated application of convolutions, followed by a ReLU layer and max pooling layer. Each step in the expansive path may include upsampling or transpose convolutions of the feature map followed by convolutions, etc..

Deep learning approaches should be contrasted against machine learning approaches that have been applied to, for example, automatic segmentation. In general, these approaches involve extracting (hand-designed) feature vectors from images, such as for every voxel, etc. Then, the feature vectors may be used as input to a machine learning model that classifies which class each voxel belongs to. However, such machine learning approaches usually do not make use of complete image data and additional constraints may be required. Another challenge is that these approaches rely on a high dimension of hand-designed features in order to accurately predict the class label for each voxel. Solving a high-dimensional classification problem is computationally expensive and requires a large amount of memory. Some approaches use lower dimensional features (e.g., using dimensionality reduction techniques) but they may decrease the prediction accuracy.

Conventionally, there are many challenges associated with training deep learning engines for radiotherapy treatment planning. For example, it may be difficult to achieve desirable training results for deep learning engines without regularizing the training process and/or collecting a huge amount of curated training data. This may lead suboptimal results or, worse, failure to create any working deep learning engines. In another example, if inexperienced users (e.g., clinicians) are provided with a tool to train their own deep learning engines, it can be challenging to perform the training from scratch. For example, in relation to automatic segmentation, the computational cost of training a blank deep learning engine can be high, especially if a large number of anatomical structures need to be modelled.

While conceptually simple, training deep learning engines generally requires significant technical expertise relating to model architecture(s), optimization, convergence analysis, regularization, etc. When trained with a limited amount of data, a deep learning engine that starts with a blank state may diverge to a suboptimal form. Such complexity of the training process may deter users from training and using deep learning engines for radiotherapy treatment planning, which is undesirable.

According to examples of the present disclosure, deep transfer learning may be performed to improve the training process of deep learning engines for radiotherapy treatment planning. Instead of starting from scratch using a blank state with random weights, a pre-trained "base" deep learning engine may be used as a starting point to improve the training phase of a "target" deep learning engine, particularly with reduced resources in terms of both computing and personnel. The target deep learning engine may be adapted from the base learning engine to according to any suitable users' needs and specifications.

As used herein, the term "deep transfer learning" may refer generally to technique(s) where one deep learning engine is adapted or re-purposed (fully or partially) as a starting point for another deep learning engine. In practice, deep transfer learning represents an optimization strategy that facilitates faster progress or improved performance during the training process. By improving the efficiency of radiotherapy treatment planning using deep transfer learning, treatment outcome may also be improved for patients, such as increasing the tumor control probability and/or reducing the likelihood of health complications or death due to radiation overdose in the healthy structures, etc..

In more detail, <FIG> is a schematic diagram illustrating example deep transfer learning <NUM> for radiotherapy treatment planning. In particular, deep transfer learning may be performed from a base deep learning engine (see <NUM>) to a target deep learning engine (see <NUM>, <NUM> and <NUM>). Base deep learning engine <NUM> may represent a "pre-trained model" that is generated or trained during pre-training phase <NUM> based on base training data <NUM> (e.g., performed at a provider's site). Target deep learning engine <NUM>/<NUM>/<NUM> may be generated or trained based on base deep learning engine <NUM> and target training data that is available during subsequent training phase <NUM> (e.g., performed at a user's site).

Examples of the present disclosure may be implemented to ameliorate various challenges associated with training deep learning engines for radiotherapy treatment planning. In one example, users do not have to start training target deep learning engine <NUM>/<NUM>/<NUM> from scratch, especially when they only have a limited amount of target training data (e.g., limited in amount or variations) that may lead to suboptimal results. Instead, users may take advantage of the better quality base training data <NUM> (e.g., more data, availability of expert-curated data, more variations, etc.) used to train base deep learning engine <NUM>.

Further, the computational cost of training deep learning engine <NUM>/<NUM>/<NUM> for radiotherapy treatment planning may be reduced by taking advantage of the knowledge already learned by base deep learning engine <NUM>. This helps increase the rate of improvement and convergence of target deep learning engine <NUM>/<NUM>/<NUM> during training phase <NUM>. The risk of achieving suboptimal training results due to limited training data (e.g., limited in amount or variation) may also be reduced. Using examples of the present disclosure, it is not necessary for users (e.g., clinicians) to have extensive knowledge about deep learning model architecture(s), etc. For example, by providing substantially stable base deep learning engine <NUM>, the required knowledge of the clinicians related to technical issues such as convergence, local minima or poor weight initializations during training phase <NUM> may be reduced.

Some examples will be discussed using <FIG>, which is a flowchart of example process <NUM> for a computer system to deep transfer learning for radiotherapy treatment planning. Example process <NUM> may include one or more operations, functions, or actions illustrated by one or more blocks, such as <NUM> to <NUM>. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. Example process <NUM> may be implemented using any suitable computer system(s), an example of which will be discussed using <FIG>.

At <NUM> in <FIG>, base deep learning engine <NUM> that is pre-trained to perform a base radiotherapy treatment planning task may be obtained. Here, the term "obtain" may refer generally to a computer system accessing, or retrieving data and/or computer-readable instructions associated with, base deep learning engine <NUM> from any suitable source (e.g., another computer system), memory or datastore (e.g., local or remote), etc. Base deep learning engine <NUM> may include multiple (N) base layers (denoted as Bi, where i = <NUM>,. , N) to perform a base radiotherapy treatment planning task. In the case of N = <NUM> in <FIG>, base deep learning engine <NUM> includes four base layers <NUM>-<NUM>.

At <NUM> in <FIG>, based on base deep learning engine <NUM>, target deep learning engine <NUM>/<NUM>/<NUM> may be generated to perform a target radiotherapy treatment planning task. Target deep learning engine <NUM>/<NUM>/<NUM> may include multiple (M) target layers (denoted as Tj, where j = <NUM>,. It should be noted that M (i.e., the number of target layers) may be the same as, or different from, N (i.e., the number of base layers). As such, base deep learning engine <NUM> may be further trained to suit the users' needs, or fixed and used as is. Base layers <NUM>-<NUM> of base engine <NUM> may be configured to be variable (see <NUM> and <NUM>), invariable (see <NUM> and <NUM>), or a combination of both.

In a first example, based on configuration of a variable base layer among base layers <NUM>-<NUM>, a target layer may be generated by modifying the variable base layer according to <NUM>-<NUM> in <FIG>. Here, the term "variable" may refer to a layer whose weights are modifiable during training phase <NUM>. For example in <FIG>, base layers <NUM>-<NUM> may be configured to be variable to generate first target deep learning engine <NUM>. Based on configuration of variable base layers <NUM>-<NUM>, target layers <NUM>-<NUM> may be generated by modifying respective variable base layers <NUM>-<NUM>. See also <NUM> and symbol "→" indicating modification.

In a second example, based on configuration of an invariable base layer among base layers <NUM>-<NUM>, a target layer may be generated based on feature data generated using the invariable base layer according to <NUM>-<NUM> in <FIG>. Here, the term "invariable" may refer to a layer whose weights are fixed and not modified during training phase <NUM>. For example in <FIG>, base layers <NUM>-<NUM> may be configured to be invariable to generate second target deep learning <NUM>. This way, target layers <NUM>-<NUM> are trained based on feature data (denoted Fi, where i = <NUM>,. , N = <NUM>) generated by invariable base layers <NUM>-<NUM>.

In a third example, a combination of variable and invariable base layers may be used to generate third target deep learning engine <NUM> in <FIG>. In this case, base layers <NUM>-<NUM> may be configured to be invariable and remaining base layers <NUM>-<NUM> variable. The latter variable base layers <NUM>-<NUM> may then be modified to generate respective target layers <NUM>-<NUM>. Additional target layers <NUM>-<NUM> may be generated or trained based on feature data (denoted F<NUM> and F<NUM>) generated using respective invariable base layers <NUM>-<NUM>. As will be discussed using <FIG>, a continuum of target deep learning engines may be generated using different combinations of variable and invariable base layers.

Depending on the desired implementation, base deep learning engine <NUM> and target deep learning engine <NUM>/<NUM>/<NUM> may be trained to perform any suitable radiotherapy treatment planning task(s), such as automatic segmentation, dose prediction, treatment delivery data estimation, abnormal organ detection, treatment outcome prediction, or any combination thereof. In the case of automatic segmentation, engine <NUM>/<NUM>/<NUM>/<NUM> may be trained to generate output = structure data (e.g., <NUM> in <FIG>) based on input = image data (e.g., <NUM> in <FIG>). In the case of dose prediction, engine <NUM>/<NUM>/<NUM>/<NUM> may be trained to generate output = dose data (e.g., <NUM> in <FIG>) based on input = structure data and beam geometry data. In the case of treatment delivery data estimation, engine <NUM>/<NUM>/<NUM>/<NUM> may be trained to generate output = treatment delivery data (e.g., fluence map data, structure projection data, etc.) based on input = structure data and/or dose data, etc..

Further, base and target deep learning engines are each trained to perform radiotherapy treatment planning task(s) associated with any suitable anatomical site(s), rule(s), etc. Some examples are shown in <FIG>, which is a schematic diagram illustrating example deep transfer learning <NUM> according to the example in <FIG>. In <FIG>, base deep learning engine <NUM> is trained using base training data <NUM> to perform a base radiotherapy treatment planning task associated with a base anatomical site = X (e.g., bladder region) and a base rule = R<NUM> according to which the task is performed.

Based on base deep learning engine <NUM>, target deep learning engines <NUM>-<NUM> are generated to perform a target radiotherapy treatment planning task associated with a different anatomical site = Y (see <NUM>-<NUM>). Further, target deep learning engines <NUM>-<NUM> associated with the same rule = R<NUM> (see <NUM>, <NUM>) or a different rule = R<NUM> or R<NUM> (see <NUM>, <NUM>) may be generated. Here, the term "anatomical site" or "anatomical region" may refer generally to one part of a patient's anatomy that has been captured using imaging modality or modalities. The term "rule" may refer to any suitable clinical guideline(s), strategy and/or planning practice(s) relating to a particular radiotherapy treatment planning task. To repurpose base deep learning engine <NUM>, target training data <NUM>-<NUM> associated with the target anatomical site (e.g., X or Y) and/or rule (e.g., R<NUM>, R<NUM> or R<NUM>) may be used during training phase <NUM>. This way, base deep learning engine <NUM> may be adapted according to the users' needs and specifications.

In the following, various examples will be discussed below using <FIG>. In particular, deep transfer learning for automatic segmentation will be discussed using <FIG>, and dose prediction using <FIG>. Example inference phase will be explained using <FIG> and example computer system using <FIG>. For simplicity, "deep learning engine" will be referred to as "engine" (e.g., base engine and target engine), and "radiotherapy treatment planning task" as "task" (e.g., base task and target task).

<FIG> is a schematic diagram illustrating first example deep transfer learning <NUM> for automatic segmentation of image data, and <FIG> a schematic diagram illustrating second example deep transfer learning <NUM> for automatic segmentation of image data. In both examples, base engine <NUM>/<NUM> may be pre-trained to perform base task = automatic segmentation for a base anatomical site (e.g., X = breast region) according to a base segmentation rule (e.g., R<NUM> = segmentation margin of <NUM>). The knowledge learned by base engine <NUM>/<NUM> may be transferred to various target engines during subsequent training phase <NUM>/<NUM>.

In the examples in <FIG>, target engines <NUM>-<NUM> are trained to perform target task = automatic segmentation for the same anatomical site as base engine <NUM>, but according to a different target rule (e.g., R<NUM> = contouring margin of <NUM>). In the examples in <FIG>, target engines <NUM>-<NUM> are trained to perform target task = automatic segmentation for a different target anatomical site (e.g., Y = lung region) compared to base engine <NUM>. The examples will be discussed in turn below. In practice, the result of automatic segmentation may be used for abnormal organ detection, dose prediction, treatment delivery data estimation, etc..

Although exemplified using segmentation margin in the following examples, it should be understood that the base and target segmentation rules (e.g., R<NUM> and R<NUM>) may be configured based on any suitable segmentation or contouring guidelines, such as when a structure stops, etc. For example, the contouring guidelines specifying when to stop contouring a structure (e.g., breast) superiorly and inferiorly, or whether the contour ends at the skin boundary or extends to the fat tissue (e.g., in the case of pendulum breast). Any additional and/or base and target segmentation rules may be used in the following examples.

Referring first to in <FIG>, base engine <NUM> may be trained during pre-training phase <NUM> using any suitable base training data <NUM>-<NUM> associated with (site = X, rule R<NUM>). The aim of pre-training phase <NUM> is to train base engine <NUM> to perform automatic segmentation by mapping input training data = image data <NUM> to output training data = structure data <NUM>. In practice, image data <NUM> may include 2D or 3D images of the patient's anatomical site, and captured using any suitable imaging modality or modalities. Structure data <NUM> may identify any suitable contour, shape, size and/or location of structure(s) identifiable from image data <NUM>. Example structures may include target(s), OAR(s) or any other structure of interest (e.g., tissue, bone).

For example, in relation to prostate cancer, image data <NUM> may include image data of X = prostate region. In this case, structure data <NUM> may identify a target representing the patient's prostate, and OARs representing proximal healthy structures such as rectum and bladder. In relation to lung cancer treatment, image data <NUM> may include image data of X = lung region. In this case, structure data <NUM> may identify a target representing cancerous lung tissue, and an OAR representing proximal healthy lung tissue, esophagus, heart, etc. In relation to brain cancer, image data <NUM> may include image data of X = brain region, in which case structure data <NUM> may identify a target representing a brain tumor, and an OAR representing a proximal optic nerve, brain stem, etc. Depending on the desired implementation, structure data <NUM> may identify multiple targets and OARs of any suitable shapes and sizes.

In practice, base training data <NUM>-<NUM> may be user-generated through observations and experience to facilitate supervised learning. For example, base training data <NUM>-<NUM> may be extracted from past treatment plans developed for past patients. Base training data <NUM>-<NUM> may be pre-processed using any suitable data augmentation approach (e.g., rotation, flipping, translation, scaling, noise addition, cropping, any combination thereof, etc.) to produce a new dataset with modified properties to improve model generalization using ground truth. In practice, a 3D volume of the patient that will be subjected to radiation is known as a treatment volume, which may be divided into multiple smaller volume-pixels (voxels). In this case, structure data <NUM> may specify a class label (e.g., "target," "OAR," etc.) associated with each voxel in the 3D volume.

Any segmentation rule(s) or guideline(s) may be used. For example, rule = R<NUM> may specify a margin around an organ to be contoured. More advanced rule(s) may be used, such as segmentation of a particular structure at an anatomical site, selection of the cutting plane of a structure (e.g., spinal cord), application of different margins at different sides of an organ (e.g., more margin inferior than superior sides of an organ, etc. The segmentation rule(s) may be fine-tuned during subsequent training phase <NUM> to achieve a more desired classification outcome. As such, base engine <NUM> represents a pre-trained model that may not be exactly related to the desired classification outcome, but is useful as a starting point for training phase <NUM>. The anatomical site = X associated with base engine <NUM> may be generic for medical image data, or specific to a particular treatment.

Any suitable deep learning model(s) may be used. For example in <FIG>, base engine <NUM> includes multiple (N > <NUM>) processing blocks or layers that are each labelled as Bi, where i = <NUM>,. ,N (see <NUM>-51N). In this case, pre-training phase <NUM> may involve finding base weight data (denoted as wBi) that minimizes the training error between training structure data <NUM>, and estimated structure data (not shown for simplicity) generated by base engine <NUM>. The training process is guided by estimating losses associated with the classification error. A simple example of a loss function would be mean squared error between true and predicted outcome, but the loss function could have more complex formulas (e.g., dice loss, jaccard loss, focal loss, etc.). This loss can be estimated from the output of the model, or from any discrete point within the model.

Base weight data wBi for the ith base layer may be a scalar or multidimensional vector. In the case of convolutional neural networks, the ith base layer (Bi,) may be a convolutional layer that is configured to extract feature data (Fi) from training data <NUM> or the output of the (i - <NUM>)th base layer (Bi-<NUM>) using convolution operations. Alternatively and/or additionally, each ith base layer may be configured to perform other operation(s) relating to activation functions, dropout, concatenations, batch normalizations, etc. For example, the first base layer (B<NUM>) processes input image data <NUM> to generate first feature data (F<NUM>). The second base layer (B<NUM>) processes first feature data (B<NUM>) to generate second feature data (F<NUM>), and so on. Feature extraction at the ith base layer (Bi) may involve applying convolutional filter(s) or kemel(s) to overlapping sites of its input to learn corresponding base weight data wBi.

The feature data (Fi) generated by the ith base layer may include a 2D feature map for 2D image data, or a 3D feature map for 3D image data. Feature data (Fi) may specify any suitable anatomical feature(s), such as borders, distance to centroid, distance to midline, distance to skin, distance from bone, laterality, presence of vertical and/or horizontal lines, shape-related parameter(s), texture types, any combination thereof, etc. This automatic feature extraction approach should be distinguished from conventional approaches that rely on hand-designed features.

During training phase <NUM>, target engine <NUM>/<NUM> associated with (site = X, rule R<NUM>) may be generated based on base engine <NUM> associated with (site = X, rule R<NUM>) and target training data <NUM>. Compared to base training data <NUM> used during pre-training phase <NUM>, target training data <NUM> is also associated with site = X, but includes examples relating to rule = R<NUM> instead of R<NUM>. For example, rule = R<NUM> may include a general segmentation margin of <NUM>, but the margin is increased to <NUM> according to rule = R<NUM>. See also related <NUM> in <FIG>.

In practice, even when trained with more limited target training data <NUM> compared to base training data <NUM>, base engine <NUM> facilitates faster convergence of target engine <NUM>/<NUM>, thereby improving efficiency. Training phase <NUM> may involve finding target weight data (denoted as wTj, where j = <NUM>,. , M) that minimizes the training error between training structure data <NUM>, and estimated structure data (not shown for simplicity) generated by target engine <NUM>/<NUM>. Two example target engines <NUM>-<NUM> will be discussed below.

In a first example (related to <NUM> in <FIG>), base engine <NUM> may be modified (e.g., further trained or fine-tuned) to generate target engine <NUM>. This may involve configuring M ≤ N base layers (see <NUM>-<NUM>) of base engine <NUM> to be "variable" such that they may be further trained to generate respective target layers (see <NUM>-<NUM>) of target engine <NUM>. The modification may involve modifying base weight data wBi (i = <NUM>,. , M) associated with respective base layers <NUM>-<NUM> to obtain corresponding target weight data wTj (j = <NUM>,. Once training is completed (see <NUM>), each modified base layer (Bi) of base engine <NUM> serves as a corresponding target layer (Tj) of target engine <NUM>. During an inference phase (to be discussed using <FIG>), target engine <NUM> may be used to perform automatic segmentation.

In a second example (related to <NUM> in <FIG>), instead of modifying base engine <NUM>, target engine <NUM> is built on top of base engine <NUM>. In this case, K ≤ N base layers (see <NUM>-<NUM>) of base engine <NUM> may be configured to be "invariable" (i.e., fixed or frozen). This way, the K invariable base layers may be used to generate feature data (FK) based on target training data <NUM>. As indicated at <NUM> in <FIG>, the feature data (FK) generated by the Kth base layer is then fed into target engine <NUM> to achieve the desired classification outcome, i.e., automatic segmentation for anatomical site = X according to rule = R<NUM>. Target engine <NUM> may include any suitable number (M) of target layers (see <NUM>-<NUM>). During an inference phase (to be discussed using <FIG>), both base engine <NUM> and target engine <NUM> may be used in sequence to perform automatic segmentation.

In a third example (not shown in <FIG> for simplicity), a combination of variable and invariable base layers may be configured (related to <NUM> in <FIG>). In this case, base engine <NUM> may be configured to include K ≤ N invariable base layers, and at most N - K variable base layers. During training phase <NUM>, the K invariable base layers are fixed and used to process training data <NUM> to generate feature data (F<NUM>,. The feature data then serves as input to modify the variable base layers, and generate any subsequent target layers.

Referring now to in <FIG>, target engines associated with a different anatomical site and/or rule may be generated based on base engine <NUM>. Similar to the example in <FIG>, base engine <NUM> may be trained to perform automatic segmentation using base training data <NUM> associated with (site = X, rule R1). The aim of pre-training phase <NUM> is to train base engine <NUM> to map input = image data <NUM> to output = structure data <NUM>.

In a first example, base engine <NUM> may be modified to generate first target engine <NUM> to perform automatic segmentation for a different target anatomical site = Y according to target rule = R<NUM>. For example, X may represent the bladder region, and Y the lung region. In another example, one could obtain base engine <NUM>/<NUM> that is trained to perform base task = breast segmentation and train it further to generate a target engine to perform target task = breast and lymph node segmentation using example contours that include both the breast and lymph nodes.

This way, knowledge relating to structure(s) in base anatomical site = X may be transferred from base engine <NUM> to target engine <NUM> for automatic segmentation in target anatomical site = Y. During training phase <NUM>, M ≤ N variable base layers (see <NUM>-<NUM>) of base engine <NUM> may be modified to generate respective target layers (see <NUM>-<NUM>). The modification involves modifying base weight data wBi (i = <NUM>,. , M) associated with respective base layers <NUM>-<NUM> to obtain corresponding target weight data wTj (j = <NUM>,. Once training is completed (see <NUM>), each modified base layer (Bi) of base engine <NUM> serves as a corresponding target layer (Tj) of target engine <NUM>.

In a second example, a combination of invariable and variable base layers may be configured to generate second target engine <NUM> to perform automatic segmentation for target anatomical site = Y according to a different target rule = R<NUM>. For simplicity, consider four base layers <NUM>-<NUM> that have been repurposed for target engine <NUM>. Invariable base layers (see <NUM>-<NUM>) are fixed and used to generate feature data (e.g., F<NUM> and F<NUM>) based on target training data <NUM>. The remaining variable base layers are modified based on the feature data to generate respective target layers <NUM>-<NUM>. Feature data from the variable base layers (e.g., F<NUM> and F<NUM>) is then used to train additional target layers <NUM>-<NUM>. In practice, it should be understood that any suitable number of invariable and variable base layers may be configured.

In practice, training process <NUM>/<NUM> may include the validation of target engines <NUM>-<NUM> in <FIG> and <NUM>-<NUM> in <FIG> using any suitable approach. For example, quality metrics may be tracked for validation purposes, such as dice score, average surface distance (measuring the error of the contoured surface location relative to the ground truth), etc..

<FIG> is a schematic diagram illustrating example deep transfer learning <NUM> for dose prediction. During pre-training phase <NUM>, base engine <NUM> may be trained using any suitable base training data <NUM> to perform dose prediction for anatomical site = X according to prediction rule = D<NUM>. The aim is to train base engine <NUM> to map example input data = image data and structure data <NUM> (i.e., segmented image data), and example output data = dose data <NUM>. Any suitable constraint(s) may be used, such as limiting dose prediction to the vicinity of target(s) or certain dose levels only.

Dose data <NUM> (e.g., 3D dose data) may specify dose distributions for a target (denoted "DTAR") and an OAR (denoted "DOAR"). For example, in relation to prostate cancer, dose data <NUM> may specify dose distributions for a target representing the patient's prostate, and an OAR representing a proximal healthy structure such as rectum or bladder. In practice, dose data <NUM> may specify the dose distributions for the whole 3D volume, not just the target and OAR volumes. Depending on the desired implementation, dose data <NUM> may include spatial biological effect data (e.g., fractionation corrected dose) and/or cover only part of the treatment volume. Besides structure data <NUM>, additional input data may include beam geometry data associated with the treatment delivery system.

During training phase <NUM>, knowledge learned by base engine <NUM> may be transferred to target engines <NUM>-<NUM>. In a first example, target engine <NUM> associated with anatomical site = X according to rule = D<NUM> is generated. This involves modifying variable base layers <NUM>-<NUM> of base engine <NUM> based on target training data <NUM>-<NUM> to generate respective target layers <NUM>-<NUM>. In a second example, target engine <NUM> associated with anatomical site = Y according to rule = D<NUM> is generated. In particular, based on target training data <NUM>-<NUM>, invariable base layers <NUM>-<NUM> of base engine <NUM> may be used to generate feature data (see <NUM>) to train target layers <NUM>-<NUM>.

In practice, any suitable prediction rules D<NUM> and D<NUM> may be used, such as rules relating to organ sparing, target coverage (and dose prescription), and normal tissue dose. Additionally or alternatively, the prediction rule(s) may relate to treatment techniques (e.g., IMRT, VMAT, etc.), cancer type, machine specification (e.g., energy and field shape, clinical practices for field placements, etc. All these will have an impact on the predicted dose data. The prediction rule(s) may be learned implicitly from training data <NUM>/<NUM>/<NUM>, or optionally provided as input parameters for certain types of deep learning engines.

Besides automatic segmentation in <FIG> and dose prediction in <FIG>, deep transfer learning may be implemented for other radiotherapy treatment planning tasks, such as treatment delivery data estimation, treatment outcome prediction, etc. In relation to treatment delivery data estimation, the estimated treatment delivery data (i.e., output data) may include structure projection data, fluence map data, etc. For example, a target engine may be trained to perform structure projection data, such as based on image data, structure data, dose data, or any combination thereof. The structure projection data may include data relating to beam orientations and machine trajectories for a treatment delivery system (see <NUM> in <FIG> and <FIG>).

In another example, a target engine may be trained to perform fluence map estimation, such as 2D fluence maps for a set of beam orientations/trajectories, machine control point data (e.g., jaw and leaf positions, gantry and couch positions), etc. Fluence maps will be explained further using <FIG>. Any additional and/or alternative training data may be used, such as field geometry data, monitor units (amount of radiation counted by machine), quality of plan estimate (acceptable or not), daily dose prescription (output), field size or other machine parameters, couch positions parameters or isocenter position within patient, treatment strategy (use movement control mechanism or not, boost or no boost), treat or no treat decision.

Although exemplified using deep convolutional neural networks, it should be understood that any alternative and/or additional deep learning model(s) may be used to implement the base and target engines. For example, base engines and target engines described in the present disclosure may include multiple processing pathways described in related <CIT> (Attorney Docket No. <NUM>-012US01). The processing pathways may be configured to process input data (e.g., image data) at different resolution levels. This way, a larger receptive field may be achieved.

In practice, medical image data generally includes both local and global feature data of a patient's anatomy, where the terms "local" and "global" are relative in nature. For example, the local feature data may provide a microscopic view of the patient's anatomy, such as tissue texture, whether a structure has a limiting border, etc. In contrast, the global feature data may provide a relatively macroscopic view of the patient's anatomy, such as which region the anatomy is located (e.g., prostate, etc.), orientation (e.g., to the left, to the right, front, back), etc..

<FIG> is a schematic diagram illustrating example inference phase <NUM> for target engines generated according to the examples in <FIG>, <FIG> and <FIG>. Once trained, target engine <NUM>/<NUM>/<NUM>/<NUM> may be used by a clinician during inference phase <NUM> to perform automatic segmentation to generate output data = patient structure data <NUM> based on input data = image data <NUM> of a current patient. In one example, image data <NUM> is processed using target layers of target engine <NUM>/<NUM> generated by modifying variable base layers of base engine <NUM>/<NUM>. In another example, image data <NUM> is processed by both base engine <NUM>/<NUM> and target engine <NUM>/<NUM> arranged in a sequence. In practice, it should be noted that the "current patient" is one of multiple patients who are being processed at the same time during inference phase <NUM>. Also, automatic segmentation may be performed when image data <NUM> is captured and being transferred to a storage system (e.g., picture archiving and communication system (PACS, etc.).

Similarly, target engine <NUM>/<NUM> may be used by a clinician during inference phase <NUM> to perform automatic segmentation to generate output data = dose data <NUM> of a current patient based on input data = image data and structure data <NUM> associated with the patient. In one example, input data <NUM> is processed using target layers of target engine <NUM> generated by modifying variable base layers of base engine <NUM>/<NUM>. In another example, input data <NUM> is processed by both base engine <NUM> and target engine <NUM> arranged in a sequence. As discussed using <FIG>, any additional and/or alternative input data and output data may be used. During radiotherapy treatment planning, a treatment plan may be generated based on structure data <NUM> and dose data <NUM> generated during inference phase <NUM> in <FIG>.

<FIG> is a schematic diagram of example treatment plan <NUM>/<NUM> generated or improved based on output data in the examples in <FIG>. Treatment plan <NUM> may be delivered using any suitable treatment delivery system that includes radiation source <NUM> to project radiation beam <NUM> onto treatment volume <NUM> representing the patient's anatomy at various beam angles <NUM>.

Although not shown in <FIG> for simplicity, radiation source <NUM> may include a linear accelerator to accelerate radiation beam <NUM> and a collimator (e.g., MLC) to modify or modulate radiation beam <NUM>. In another example, radiation beam <NUM> may be modulated by scanning it across a target patient in a specific pattern with various energies and dwell times (e.g., as in proton therapy). A controller (e.g., computer system) may be used to control the operation of radiation source <NUM> according to treatment plan <NUM>.

During treatment delivery, radiation source <NUM> may be rotatable using a gantry around a patient, or the patient may be rotated (as in some proton radiotherapy solutions) to emit radiation beam <NUM> at various beam orientations or angles relative to the patient. For example, five equally-spaced beam angles 930A-E (also labelled "A," "B," "C," "D" and "E") may be selected using a deep learning engine configured to perform treatment delivery data estimation. In practice, any suitable number of beam and/or table or chair angles <NUM> (e.g., five, seven, etc.) may be selected. At each beam angle, radiation beam <NUM> is associated with fluence plane <NUM> (also known as an intersection plane) situated outside the patient envelope along a beam axis extending from radiation source <NUM> to treatment volume <NUM>. As shown in <FIG>, fluence plane <NUM> is generally at a known distance from the isocenter.

Examples of the present disclosure may be deployed in any suitable manner, such as a standalone system, web-based planning-as-a-service (PaaS) system, etc. In the following, an example planning system (also known as a "computer system") will be described using <FIG>, which is a schematic diagram illustrating example network environment <NUM> in which deep transfer learning for radiotherapy treatment planning may be implemented. Depending on the desired implementation, network environment <NUM> may include additional and/or alternative components than that shown in <FIG>.

In the example in <FIG>, network environment <NUM> includes planning system <NUM> that is accessible by multiple user devices <NUM>-<NUM> via any suitable physical network (e.g., local area network, wide area network, etc.) In practice, user devices <NUM>-<NUM> may be operated by various users located at a particular clinical site, or different clinical sites. Planning system <NUM> may be implemented using a multi-tier architecture that includes web-based user interface (Ul) tier <NUM> to interact with user devices <NUM>-<NUM>; training infrastructure <NUM> (also known as an application tier) to perform deep transfer learning; and data tier <NUM> to facilitate data access to and from datastore <NUM>. Depending on the desired implementation, planning system <NUM> may be deployed in a cloud computing environment, in which case multiple virtualized computing instances (e.g., virtual machines, containers) may be configured to implement various functionalities of tiers <NUM>-<NUM>. The cloud computing environment may be supported by on premise cloud infrastructure, public cloud infrastructure, or a combination of both.

In more detail, Ul tier <NUM> may be configured to provide any suitable interface(s) to interact with user devices <NUM>-<NUM>, such as graphical user interface (GUI), command-line interface (CLI), application programming interface (API) calls, any combination thereof, etc. Training infrastructure may be configured to perform train base engines during a pre-training phase, and deep transfer learning (i.e., model adaptation) to generate target engines during a training and validation phase.

Data tier <NUM> may be configured to facilitate access (by UI tier <NUM> and training infrastructure <NUM>) to multiple base and target engines stored in datastore <NUM>. Using the examples discussed using <FIG>, datastore <NUM> may store base engine <NUM>/<NUM> associated with (site = X, segmentation rule = R<NUM>), and base engine <NUM> associated with (site = X, dose prediction rule = D<NUM>). This way, it is not necessary for user devices <NUM>-<NUM> to interact with backend training infrastructure <NUM> and data tier <NUM> directly, thereby providing abstraction and enhancing security.

<FIG> is a flowchart of an example process for deep transfer learning for radiotherapy treatment planning using example planning system in <FIG>. Example process <NUM> may include one or more operations, functions, or actions illustrated by one or more blocks, such as <NUM> to <NUM>. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. Example process <NUM> may be implemented using any suitable computer system(s), an example of which will be discussed using <FIG>.

At <NUM> in <FIG>, during a pre-training phase, multiple base engines may be trained to perform respective base tasks using base training data. Each base engine may be trained to perform a particular base task associated with a particular site (e.g., site X = bladder region) according to a particular rule (e.g., R<NUM> = segmentation margin of <NUM>). Each base engine may be trained to perform any suitable base task, such as segmentation (see <FIG>), dose prediction (see <FIG>), treatment delivery data estimation, treatment outcome prediction, etc. Once pre-training is performed, multiple base engines (see <NUM>-<NUM>) may be stored in datastore <NUM> for later access during training phase. Depending on the desired implementation, block <NUM> may be performed by training infrastructure <NUM> (as shown in <FIG>), user device <NUM>/<NUM>/<NUM>, an alternative system, or any combination thereof.

At <NUM> in <FIG>, planning system <NUM> may provide first Ul(s) supported by UI tier <NUM> for a user to send a request to generate a target engine to perform a target task. At <NUM>, the user may interact with the first Ul(s) to send a request to generate the target engine based on a base engine and target training data. The request may be generated using an interface (e.g., GUI, API, CLI) supported by UI tier <NUM>. For example in <FIG>, first user device <NUM> may send a first request (see <NUM>) to UI tier <NUM> by interacting with a GUI. Second user device <NUM> may send a second request (see <NUM>) by invoking API call(s), and third user device <NUM> may send a third request (see <NUM>) using CLI command(s).

Some example Ul(s) will be explained using <FIG> and <FIG>, each being a schematic diagram illustrating example interface <NUM> for target engine generation during training phase. To facilitate deep transfer learning, requests <NUM>-<NUM> may each include any suitable target training data (see <NUM> in <FIG> and <FIG>). Requests <NUM>-<NUM> may include any additional data for generating respective target engines, such as base engine selection parameter data (see <NUM> in <FIG> and <FIG>), training parameter data (see <NUM> and <FIG>), validation parameter data (see <NUM> and <FIG>), etc. In practice, training interface <NUM> may include any suitable "Ul elements" with which a user may interact, such as windows, panes, buttons, menus, text boxes, tabs, lists, application icons, menu bars, scroll bars, title bars, status bars, toolbars, dropdown lists, any combination thereof, etc..

At <NUM> in <FIG> (see "Target Engine" view), training interface <NUM> includes UI elements for a user to provide target training data associated with multiple past patients, or specify a link (e.g., a uniform resource locator (URL), network path, etc.) to the target training data. The target training data may include image data in any suitable format capable of being processed by training infrastructure <NUM>, such as Digital Imaging and Communications in Medicine (DICOM) images. Using the automated segmentation example in <FIG>, first request <NUM> may include target training data <NUM> to generate target engine <NUM>/<NUM> associated with (site = X, target rule = R<NUM>). Second request <NUM> may include target training data <NUM> in <FIG> to generate target engine <NUM>/<NUM> associated with (site = Y, target rule = R<NUM>). For dose prediction in <FIG>, third request <NUM> may include target training data <NUM> to generate target engine <NUM>/<NUM> associated with (site = X, target rule = D<NUM>).

At <NUM> in <FIG> (see "Base Engine" view), training interface <NUM> includes UI element(s) for a user to specify selection parameter data for selecting a base engine. On a left pane (see <NUM>) of training interface <NUM>, base engines may be searched or filtered based on task type (e.g., segmentation, dose prediction), anatomical site, structure, base engine provider (e.g., providers P<NUM> and P<NUM>), performance metric, keyword(s), etc. On a right pane (see <NUM>), a list of matching base engines are presented for selection, such as ID, anatomical site, base rule (e.g., segmentation margin = R<NUM>, dose prediction rule = D<NUM>), structure(s), performance metric(s), etc. In practice, a user may first select a planning task and a general anatomical site, before specifying more specific requirements. In response to receiving the base engine selection parameter data, UI tier <NUM> may retrieve a list of candidate base engines and present the list on interface <NUM> in <FIG> for user's selection.

At <NUM> in <FIG>, training interface <NUM> includes UI element(s) for a user to specify mapping data for use during the training phase. Based on the mapping data, training infrastructure <NUM> may map a target name (or target ID) associated with the target engine with a corresponding base name (or base ID) associated with a base engine. For example, a base engine for automated segmentation may provide a list of structures as its output. The structures may be referred to as "Breast Left," "Breast Right," and "Heart" by the base engine. In the target training data provided by the user, however, different (target) names may be used for the same structures, such as "Brust Links," "Brust Rechts" and "Herz" (in German), or "BR-L," "BR-R" and "H" (in shorthand specific to a clinical site). In this case, the mapping data may be provided to training architecture <NUM> to translate a base name (e.g., "Breast Right" with ID = <NUM>) to its corresponding target name (e.g., "BR-R" with ID = <NUM>). Where possible, the mapping may also be automatically handled by planning system <NUM> through base/target structure ID mapping. The mapping may also be based on structure IDs from any suitable clinical standards or reference ontology, such as the Foundational Model of Anatomy (FMA) Ontology, etc..

At <NUM> in <FIG> (see "Training" view), training interface <NUM> further includes UI element(s) for a user to configure training parameter data to guide training infrastructure <NUM> during training phase. Under "base layer configuration" (see <NUM>), a base engine may be configured to include variable base layer(s) and/or invariable base layer(s). Depending on the desired implementation, the number (or percentage) of variable and invariable base layers may be configured. Under "training hyperparameters" (see <NUM>), hyperparameters associated with deep transfer learning may be configured, such as dropout rate, activation function, number of training epochs, learning rate, number of hidden units, convolution kernel width, etc. In practice, configuration of base layers and training parameters may be performed automatically by training infrastructure <NUM> instead of relying on a user's input. Depending on the training hyperparameters changed, a full retraining may be required due to changes in an underlying graph associated with the base engine. For example, a full retraining may be required in response to changes to hyperparameters such as convolutional kernel width or activation function. However, a full retraining is generally not required when parameters such as loss functions, learning rates and number of epochs are updated.

At <NUM> in <FIG> (see "Validation" view), training interface <NUM> further includes Ul element(s) for a user to configure validation parameter data to guide training infrastructure <NUM> during validation phase. Under "validation criteria" (see <NUM>), for example, thresholds for validation parameter data may be configured, such as dice score, average surface difference, Hausdorff distance, etc. Although discussed using <FIG> and <FIG>, it should be understood that request <NUM>/<NUM> generated using API/CLI may similarly include target training data, base engine selection data, training parameter data, validation parameter data, or any combination thereof.

Referring to <FIG>, at <NUM>, planning system <NUM> (e.g., UI tier <NUM>) detects a request (e.g., <NUM>) from a user device (e.g., <NUM>) to generate a target engine. At <NUM> in <FIG>, in response to detecting the request, planning system <NUM> (e.g., training infrastructure <NUM>) may select a base engine from multiple base engines that have been pre-trained at block <NUM>. The selection at block <NUM> may be based on manual selection by the user, or automatic selection by training infrastructure <NUM>. In the case of manual selection, the request may specify (e.g., using model or engine ID) a particular base engine selected by the user, such as using training interface <NUM> in <FIG>. In the case of automatic selection, the request may specify selection parameter data for training infrastructure <NUM> to select the base engine automatically, such as an anatomical site, structure(s) or organ(s), performance metric(s), base engine provider, or any combination thereof.

At <NUM> in <FIG>, training infrastructure <NUM> configures the base engine selected at block <NUM> to include variable base layer(s), invariable base layer(s), or a combination of both. Block <NUM> may be performed based on the base layer configuration (see <NUM> in <FIG>) specified in a request, or default settings (where applicable). At <NUM> in <FIG>, training infrastructure <NUM> performs deep transfer learning to generate a target engine based on the base engine selected at block <NUM>. During the training phase, deep transfer learning may be performed based on mapping data (see <FIG>) and/or training parameter data (see <FIG>), etc. In practice, the base engine at block <NUM> may be represented using a set of weights (i.e., variable or invariable depending on their configuration) that will be used as a starting point for generating the target engine.

In a first example, a base engine (e.g., <NUM> in <FIG> and <NUM> in <FIG>) may be configured to include M ≤ N variable base layers. In this case, training infrastructure <NUM> may generate multiple target layers of the target engine (e.g., <NUM> in <FIG> and <NUM> in <FIG>) by modifying the variable base layers based on target training data (e.g., <NUM> in <FIG> and <NUM> in <FIG>). See <NUM> in <FIG>.

In a second example, a base engine (e.g., <NUM> in <FIG>) may be configured to include K ≤ N invariable base layers, which are not modified during training. In this case, training infrastructure <NUM> may generate multiple target layers of the target engine (e.g., <NUM> in <FIG>) based on feature data that is generated by the invariable base layers based on the target training data (e.g., <NUM> in <FIG>). See <NUM> in <FIG>.

In a third example, a base engine (e.g., <NUM> in <FIG>) may be configured to include a combination of variable and invariable base layers. In this case, training infrastructure <NUM> may generate multiple target layers of the target engine (e.g., <NUM> in <FIG>) by generating feature data using the invariable base layers (see <NUM>-<NUM>) based on the target training data. The remaining variable base layers are then modified based on the feature data to generate additional target layers.

Examples of deep transfer learning explained using <FIG> are applicable here, and will not be repeated in detail for brevity. In practice, the user's request may cause training infrastructure <NUM> to (a) train the target engine to meet specific requirements, or (b) tune a base engine to better match different features present in a particular patient population. For case (a), the request may specify guidelines such as additional margins, selection of the cutting plane of long organs (e.g., spinal cord), etc..

For case (b), retraining might be required if the anatomy of the patient population differs from the patient population used to train the original model (i.e., base engine). The differences in the patient population may include differences in body mass index and size, or even the shape of contoured organs (e.g., the breast size might differ between different geographies due to differences in the amount of fat tissue). The differences may be associated with the anatomical site, such as when a base engine developed for intact breasts may be retrained for patients with one breast removed. The base and target engines may be based on any suitable medical imaging modality, such as CT, CBCT, PET, MRI, or any combination thereof, etc..

At <NUM> in <FIG>, training infrastructure <NUM> performs validation of the target engine, such as based on validation parameter data configured using training interface <NUM> in <FIG>. Validation may be performed to improve results, such as to prevent overfitting or falling into a local minimum, provide an indication of convergence, etc. In practice, the target training data provided by the user may be divided into a training set for the training phase, and a validation set for the validation phase.

In response to determination a particular iteration of the target engine does not satisfy certain validation criteria, the training phase continues at block <NUM> at the next iteration. Otherwise, at <NUM>, a response (e.g., <NUM>/<NUM>/<NUM> in <FIG>) is sent to the user device (e.g., <NUM>/<NUM>/<NUM>) to facilitate access to the generated target engine and any associated data. The generated target engine (e.g., <NUM>/<NUM> in <FIG>) may be stored in datastore <NUM>.

Planning system <NUM> may generate statistics and visual inspection on how well the target engine has been adapted from the base engine. Some examples are shown in <FIG> (see "Result" view at <NUM>), which are each a schematic diagram illustrating training interface <NUM> for target engine generation during validation phase. At <NUM> and <NUM> in <FIG>, a validation metrics view of training interface <NUM> may present statistics or metrics associated with the generated target engine (e.g., ID = <NUM>). Example metrics include dice scores (or slice-wise dice scores), accuracy, sensitivity, average surface difference, Hausdorff distance, etc..

As a reference, the quality statistics could be provided also from applying the target engine on a larger set of target training data that the base engine was trained with. In practice, the validation criteria may be specific to a particular application, target task, anatomical site, etc. For large structures, for example, one typically gets higher dice scores compared to smaller structures. These scores may be used for tracking whether the target engine improves or not. In the example in <FIG>, it is desirable to have the "adapted" (target) models performs better than the "non-adapted" (base) models. Depending on the desired implementation, training interface <NUM> may provide a Ul that presents a how the target engine performs on the "base" validation cases (i.e., used for validating the base engine) and any additional validation cases (i.e., used for validating the target engine). This way, the user may distinguish between a decrease in generalizability and an increase in site-specific performance.

At <NUM> in <FIG>, a visual inspection view of training interface <NUM> may present how well the target engine has been adapted. For example, a specific patient may be selected (see <NUM>) to compare the ground truth against the output of the target engine for that patient. The visual inspection may be presented in multiple anatomical planes, including the transverse or axial plane, sagittal plane and coronal plane. The axial plane is parallel to the ground and divides the body into top and bottom parts. The sagittal or lateral plane divides the body into left and right halves. The coronal or frontal plane divides the body into front and back sections.

After the training phase, the generated target engine may be provided to the user for local application or for cloud-based planning application. At <NUM>-<NUM> in <FIG>, UI tier <NUM> is further configured to provide a planning interface for a user (e.g., clinician) to interact with. At <NUM>, through the interaction with the planning interface, the user may access and instruct a target engine to perform a target task for an individual patient. At <NUM>-<NUM>, planning system <NUM> performs the target task (e.g., automated segmentation) for the individual patient and responds to any suitable output data (e.g., structure data) and statistics, which will be accessible via interface(s) supported by UI tier <NUM>.

A first example is shown in <FIG>, which is a schematic diagram illustrating planning interface <NUM> for automated segmentation during inference phase. Planning interface <NUM> includes Ul elements for the user to provide input = image data associated with a patient (see <NUM>); and select and instruct a particular target engine (see <NUM>) trained to perform automated segmentation (see <NUM>). For example, target engine <NUM>/<NUM> in <FIG> may be selected to perform automated segmentation associated with (site X, rule R<NUM>), or target engine <NUM>/<NUM> in <FIG> for (site Y, rule R<NUM> or R<NUM>). Once automated segmentation is completed, planning interface <NUM> includes UI elements to present output = structure data associated with the patient (see <NUM>), and for the user to download the output structure data (see <NUM>). In practice, planning interface <NUM> may be integrated with any suitable system, such as treatment planning platform(s), patient database(s), etc. One example of the treatment planning platform is the Eclipse™ Treatment Planning Platform (available from Varian Medical Systems), etc..

A second example is shown in <FIG>, which is a schematic diagram illustrating planning interface <NUM> for dose prediction during inference phase. Planning interface <NUM> includes UI elements for the user to provide input = image data and structure data associated with a patient (see <NUM>); and select and instruct a particular target engine (see <NUM>) trained to perform dose prediction (see <NUM>). Using the examples in <FIG>, target engine <NUM>/<NUM> may be instructed to perform dose prediction associated with (site X, prediction rule D<NUM> or D<NUM>). Once dose prediction is completed, planning interface <NUM> includes UI elements to present output = dose data associated with the patient (see <NUM>), and for the user to download the output dose data (see <NUM>).

The above examples can be implemented by hardware, software or firmware or a combination thereof. <FIG> is a schematic diagram of example computer system <NUM> for deep transfer learning for radiotherapy treatment planning. In this example, computer system <NUM> (also known as a treatment planning system) may include processor <NUM>, computer-readable storage medium <NUM>, system interface <NUM> to interface with radiotherapy treatment delivery system <NUM>, and bus <NUM> that facilitates communication among these illustrated components and other components.

Processor <NUM> is to perform processes described herein with reference to <FIG>. Computer-readable storage medium <NUM> may store any suitable information <NUM>, such as information relating to training data, deep learning engines, image data, output data, etc. Computer-readable storage medium <NUM> may further store computer-readable instructions <NUM> which, in response to execution by processor <NUM>, cause processor <NUM> to perform processes described herein. Treatment may be delivered according to treatment plan <NUM> using treatment planning system <NUM> explained using <FIG>, the description of which will not be repeated here for brevity. In practice, computer system <NUM> may be part of a computation cluster that includes multiple computer systems. Computer system <NUM> may include any alternative and/or additional component(s), such as graphics processing unit (GPU), message queues for communication, blob storage or databases, load balancer(s), specialized circuits, etc. As discussed using <FIG>, computer system <NUM> may be deployed in any suitable manner, including a service-type deployment in an on-premise cloud infrastructure, public cloud infrastructure, a combination thereof, etc..

Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. Throughout the present disclosure, the terms "first," "second," "third," etc. do not denote any order of importance, but are rather used to distinguish one element from another.

Those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure.

Claim 1:
A method for a computer system to perform deep transfer learning for radiotherapy treatment planning, wherein the method comprises:
obtaining a base deep learning engine that is pre-trained, based on base training data, to perform a base radiotherapy treatment planning task, wherein the base deep learning engine includes multiple base layers;
based on the base deep learning engine and target training data, generating a target deep learning engine to perform a target radiotherapy treatment planning task by performing at least one of the following:
configuring a variable base layer among the multiple base layers of the base deep learning engine, and generating one of multiple target layers of the target deep learning engine by modifying the variable base layer, by modifying base weight data associated with respective base layers to obtain corresponding target weight data; and
configuring an invariable base layer among the multiple base layers of the base deep learning engine, and generating one of multiple target layers of the target deep learning engine based on feature data generated using the invariable base layer, by training target layers based on said feature data;
characterized in that
generating the target deep learning engine comprises:
based on the base deep learning engine that is pre-trained to perform the base radiotherapy treatment planning task associated with a base anatomical site, generating the target deep learning engine to perform the target radiotherapy treatment planning task associated with a target anatomical site different from the base anatomical site.