Patent Description:
Radiotherapy is an important part of a treatment for reducing or eliminating unwanted tumors from patients. Unfortunately, applied radiation does not inherently discriminate between an unwanted tumor and any proximal healthy structures such as organs, etc. This necessitates careful administration to restrict the radiation to the tumor (i.e., target). Ideally, the goal is to deliver a lethal or curative radiation dose to the tumor, while maintaining an acceptable dose level in the proximal healthy structures. However, to achieve this goal, conventional radiotherapy treatment planning may be time and labor intensive. <CIT> describes methods and systems for radiotherapy treatment planning comprising obtaining image data associated with a patient; and processing the image data to generate a treatment plan for the patient using an inferential chain that includes multiple Al engines that are trained separately to perform respective multiple treatment planning steps.

In one aspect the present invention provides a method for a computer system to perform quality-aware continuous learning for radiotherapy treatment planning, as defined in claim <NUM>. Optional features are specified in the claims dependent thereon.

In another aspect the present invention provides a non-transitory computer-readable storage medium that includes a set of instructions which, in response to execution by a processor of a computer system, cause the processor to perform a method of quality-aware continuous learning for radiotherapy treatment planning, as defined in claim <NUM>. Optional features are specified in the claims dependent thereon.

In a further aspect the present invention provides a computer system configured to perform quality-aware continuous learning for radiotherapy treatment planning, as defined in claim <NUM>.

According to examples of the present disclosure, methods and systems for quality-aware continuous learning for radiotherapy treatment planning are provided. In this case, one example method may comprise: obtaining an artificial intelligence (AI) engine that is trained to perform a radiotherapy treatment planning task. The method may also comprise: based on input data associated with a patient, performing the radiotherapy treatment planning task using the Al engine to generate output data associated with the patient; and obtaining modified output data that includes one or more modifications made by a treatment planner to the output data.

The example method may further comprise: performing quality evaluation based on (a) first quality indicator data associated with the modified output data, and/or (b) second quality indicator data associated with the treatment planner. In response to a decision to accept, a modified Al engine may be generated by re-training the Al engine based on the modified output data.

The technical details set forth in the following description enable a person skilled in the art to implement one or more embodiments of the present disclosure.

<FIG> is a schematic diagram illustrating example process flow <NUM> for radiotherapy treatment. Example process <NUM> may include one or more operations, functions, or actions illustrated by one or more blocks. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. In the example in <FIG>, radiotherapy treatment generally includes various stages, such as an imaging system performing image data acquisition for a patient (see <NUM>); a radiotherapy treatment planning system (see <NUM>) generating a suitable treatment plan (see <NUM>) for the patient; and a treatment delivery system (see <NUM>) delivering treatment according to the treatment plan.

In more detail, at <NUM> in <FIG>, image data acquisition may be performed using an imaging system to capture image data <NUM> associated with a patient (particularly the patient's anatomy). Any suitable medical image modality or modalities may be used, such as computed tomography (CT), cone beam computed tomography (CBCT), positron emission tomography (PET), magnetic resonance imaging (MRI), and/or single photon emission computed tomography (SPECT), or any combination thereof, etc. For example, when CT or MRI is used, image data <NUM> may include a series of two-dimensional (2D) images or slices, each representing a cross-sectional view of the patient's anatomy, or may include volumetric or three-dimensional (3D) images of the patient, or may include a time series of 2D or 3D images of the patient (e.g., four-dimensional (4D) CT or 4D CBCT).

At <NUM> in <FIG>, radiotherapy treatment planning may be performed during a planning phase to generate treatment plan <NUM> based on image data <NUM>. Any suitable number of treatment planning tasks or steps may be performed, such as segmentation, dose prediction, projection data prediction, and/or treatment plan generation, etc. For example, segmentation may be performed to generate structure data <NUM> identifying various segments or structures from image data <NUM>. In practice, a three-dimensional (3D) volume of the patient's anatomy may be reconstructed from image data <NUM>. The 3D volume that will be subjected to radiation is known as a treatment or irradiated volume that may be divided into multiple smaller volume-pixels (voxels) <NUM>. Each voxel <NUM> represents a 3D element associated with location (i, j, k) within the treatment volume. Structure data <NUM> may be include any suitable data relating to the contour, shape, size and location of patient's anatomy <NUM>, target <NUM>, and/or organ-at-risk (OAR) <NUM>, etc. In practice, OAR <NUM> may represent any suitable delineated organ, and/or non-target structure (e.g., bone, tissue, etc.), etc..

For example, using image segmentation, a line may be drawn around a section of an image and labelled as target <NUM> (e.g., tagged with label = "prostate"). Everything inside the line would be deemed as target <NUM>, while everything outside would not. In another example, dose prediction may be performed to generate dose data <NUM> specifying radiation dose to be delivered to target <NUM> (denoted "DTAR" at <NUM>) and radiation dose for OAR <NUM> (denoted "DOAR" at <NUM>). In practice, target <NUM> may represent a malignant tumor (e.g., prostate tumor, etc.) requiring radiotherapy treatment, and OAR <NUM> a proximal healthy structure or non-target structure (e.g., rectum, bladder, etc.) that might be adversely affected by the treatment. Target <NUM> is also known as a planning target volume (PTV). Although an example is shown in <FIG>, the treatment volume may include multiple targets <NUM> and OARs <NUM> with complex shapes and sizes. Further, although shown as having a regular shape (e.g., cube), voxel <NUM> may have any suitable shape (e.g., non-regular). Depending on the desired implementation, radiotherapy treatment planning at block <NUM> may be performed based on any additional and/or alternative data, such as prescription, disease staging, biologic or radiomic data, genetic data, assay data, biopsy data, and/or past treatment or medical history, or any combination thereof, etc..

Based on structure data <NUM> and dose data <NUM>, treatment plan <NUM> may be generated to include 2D fluence map data for a set of beam orientations or angles. Each fluence map specifies the intensity and shape (e.g., as determined by a multileaf collimator (MLC)) of a radiation beam emitted from a radiation source at a particular beam orientation and at a particular time. For example, in practice, intensity modulated radiotherapy treatment (IMRT) or any other treatment technique(s) may involve varying the shape and intensity of the radiation beam while at a constant gantry and couch angle. Alternatively or additionally, treatment plan <NUM> may include machine control point data (e.g., jaw and leaf positions), volumetric modulated arc therapy (VMAT) trajectory data for controlling a treatment delivery system, etc. In practice, block <NUM> may be performed based on goal doses prescribed by a clinician (e.g., oncologist, dosimetrist, planner, etc.), such as based on the clinician's experience, the type and extent of the tumor, and/or patient geometry and condition, etc..

At <NUM> in <FIG>, treatment delivery is performed during a treatment phase to deliver radiation to the patient according to treatment plan <NUM>. For example, radiotherapy treatment delivery system <NUM> may include rotatable gantry <NUM> to which radiation source <NUM> is attached. During treatment delivery, gantry <NUM> is rotated around patient <NUM> supported on structure <NUM> (e.g., table) to emit radiation beam <NUM> at various beam orientations according to treatment plan <NUM>. Controller <NUM> may be used to retrieve treatment plan <NUM> and control gantry <NUM>, radiation source <NUM> and radiation beam <NUM> to deliver radiotherapy treatment according to treatment plan <NUM>. The radiation may be designed to be curative, palliative, adjuvant, etc..

It should be understood that any suitable radiotherapy treatment delivery system(s) may be used, such as mechanic-arm-based systems, tomotherapy type systems, brachytherapy, SIR-spheres, and/or radiopharmaceuticals, or any combination thereof, etc. Additionally, examples of the present disclosure may be applicable to particle delivery systems (e.g., proton, and/or carbon ion, etc.). Such systems may employ either a scattered particle beam that is then shaped by a device akin to an MLC, or a scanning beam of adjustable energy, spot size and dwell time.

Conventionally, radiotherapy treatment planning at block <NUM> in <FIG> is time and labor intensive. For example, it usually requires a team of highly skilled and trained oncologists and dosimetrists to manually delineate structures of interest by drawing contours or segmentations on image data <NUM>. These structures are manually reviewed by a physician, possibly requiring adjustment or re-drawing. In many cases, the segmentation of critical organs can be the most time-consuming part of radiation treatment planning. After the structures are agreed upon, there are additional labor-intensive steps to process the structures to generate a clinically-optimal treatment plan specifying treatment delivery data such as beam orientations and trajectories, as well as corresponding 2D fluence maps.

Further, treatment planning is often complicated by a lack of consensus among different physicians and/or clinical regions as to what constitutes "good" contours or segmentation. In practice, there might be a huge variation in the way structures or segments are drawn by different clinical experts. The variation may result in uncertainty in target volume size and shape, as well as the exact proximity, size and shape of OARs that should receive minimal radiation dose. Even for a particular expert, there might be variation in the way segments are drawn on different days. Due to the lack of consistency, treatment planning might result in different clinical outcomes for patients and it is difficult to evaluate whether a final treatment plan is "good.

According to examples of the present disclosure, artificial intelligence (AI) techniques may be applied to ameliorate various challenges associated with radiotherapy treatment planning. Throughout the present disclosure, the term "Al engine" may refer generally to any suitable hardware and/or software component(s) of a computer system capable of executing algorithms according to any suitable Al model(s), such as deep learning model(s), etc. The term "deep learning" may refer generally to a class of approaches that utilizes many layers or stages of nonlinear data processing for feature learning as well as pattern analysis and/or classification. The "deep learning model" may refer to a hierarchy of "layers" of nonlinear data processing that include an input layer, an output layer, and multiple (i.e., two or more) "hidden" layers between the input and output layers. These layers may be trained from end-to-end (e.g., from the input layer to the output layer) to extract feature(s) from an input and classify the feature(s) to produce an output (e.g., classification label or class). The term "deep learning engine" may refer to any suitable hardware and/or software component(s) of a computer system capable of executing algorithms according to any suitable deep learning model(s).

Depending on the desired implementation, any suitable Al model(s) may be used, such as convolutional neural network, recurrent neural network, and/or deep belief network, or any combination thereof, etc. In practice, a neural network is generally formed using a network of processing elements (called "neurons," "nodes," etc.) that are interconnected via connections (called "synapses," "weights," etc.). For example, convolutional neural networks may be implemented using any suitable architecture(s), such as U-net, LeNet, AlexNet, ResNet, and/or V-net, DenseNet, etc. In this case, a "layer" of a convolutional neural network may be a convolutional layer, pooling layer, rectified linear units (ReLU) layer, fully connected layer, and/or loss layer, etc. In practice, the U-net architecture includes a contracting path (left side) and an expansive path (right side). The contracting path includes repeated application of convolutions, followed by a ReLU layer and max pooling layer. Each step in the expansive path may include upsampling of the feature map followed by convolutions, etc..

Deep learning approaches should be contrasted against machine learning approaches that have been applied to, for example, automatic segmentation. In general, these approaches involve extracting (hand-designed) feature vectors from images, such as for every voxel, etc. Then, the feature vectors may be used as input to a machine learning model that classifies which class each voxel belongs to. However, such machine learning approaches usually rely on a high dimension of hand-designed features in order to accurately predict the class label for each voxel. Solving a high-dimensional classification problem is computationally expensive and requires a large amount of memory. Some approaches use lower dimensional features (e.g., using dimensionality reduction techniques) but they may decrease the prediction accuracy.

Conventionally, there are many challenges associated with training Al engines (e.g., deep learning engines) for radiotherapy treatment planning. For example, different planners generally have different clinical practices in radiotherapy treatment planning. To train an Al engine according to a specific clinical practice, one option is to develop a specific in-house model. However, it may be difficult to achieve desirable training results without collecting a huge amount of carefully-curated training data. Also, while conceptually simple, training Al engines generally requires significant technical expertise relating to model architecture(s), optimization, convergence analysis, and/or regularization, etc. These challenges may lead to suboptimal results or, worse, failure to create any working Al engines. Such complexity may deter users from training and using Al engines for radiotherapy treatment planning, which is undesirable.

According to examples of the present disclosure, quality-aware continuous learning may be implemented to improve the performance of Al engines for radiotherapy treatment planning. As used herein, the term "continuous learning" (also known as "lifelong learning," "incremental learning" and "sequential learning") may refer generally to technique(s) where an Al engine is modified or improved throughout its operation based on additional training data. The term "quality-aware" may refer generally to a quality evaluation process for deciding whether continuous learning should be performed.

Using a quality-aware approach, a trained Al engine may be modified or improved over time based on training data that has been evaluated. By improving the quality and adaptability of Al engines, treatment planning outcome may also be improved for patients, such as increasing the tumor control probability and/or reducing the likelihood of health complications or death due to radiation overdose in the healthy structures, etc. Examples of the present disclosure may be deployed in any suitable manner, such as a standalone computer system, and/or web-based planning-as-a-service (PaaS) system, or any combination thereof, etc..

In more detail, <FIG> is a flowchart illustrating example process <NUM> for a computer system to perform quality-aware continuous learning for radiotherapy treatment planning. Example process <NUM> may include one or more operations, functions, or actions illustrated by one or more blocks, such as <NUM> to <NUM>. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. Example process <NUM> may be implemented using any suitable computer system(s), an example of which will be discussed using <FIG>. Some examples will be explained using <FIG>, which is a schematic diagram illustrating example quality-aware continuous learning for radiotherapy treatment planning according to the example in <FIG>.

At <NUM> in <FIG>, a treatment planning Al engine (see <NUM> in <FIG>) that is trained to perform a radiotherapy treatment planning task may be obtained. Here, the term "obtain" may refer generally to a computer system accessing, or retrieving data and/or computer-readable instructions associated with, Al engine <NUM> from any suitable source (e.g., another computer system), memory or datastore (e.g., local or remote), etc. Al engine <NUM> may be trained during training phase <NUM> based on first training data (see <NUM> in <FIG>) that includes treatment plans associated with multiple past patients. Note that "first training data" <NUM> may include synthetic training data, which are training cases derived from past patients.

At <NUM> in <FIG>, Al engine <NUM> may be used to perform the radiotherapy treatment planning task during inference phase <NUM>. For example, based on input data (see <NUM> in <FIG>) associated with a particular patient, Al engine <NUM> may perform the radiotherapy treatment planning task to generate output data (see <NUM> in <FIG>) associated with the patient. In practice, Al engine <NUM> may be trained to perform any suitable radiotherapy treatment planning task, such as automatic segmentation, dose prediction, treatment delivery data estimation, abnormal organ detection, and/or treatment outcome prediction, or any combination thereof.

In the case of automatic segmentation, Al engine <NUM> may be trained to generate output = structure data (e.g., <NUM> in <FIG>) based on input = image data (e.g., <NUM> in <FIG>). In the case of dose prediction, engine <NUM> may be trained to generate output = dose data (e.g., <NUM> in <FIG>) based on input = structure data and beam geometry data. In the case of treatment delivery data estimation, engine <NUM> may be trained to generate output = treatment delivery data (e.g., fluence map data, and/or structure projection data, etc.) based on input = structure data and/or dose data, etc..

At <NUM> in <FIG>, modified output data (see <NUM> in <FIG>) that includes modification(s) to output data <NUM> may be obtained. The term "modification" may refer generally to an addition, deletion, correction, change, movement, selection or alteration that may be made to output data. Modified output data <NUM> may be generated by a treatment planner (see <NUM> in <FIG>). Here, a "treatment planner" or "planner" may refer generally to an individual, a group of individuals, an institution, a clinical site or network, and/or a clinical region, or any combination thereof. For example, an individual may be a dosimetrist, clinician, physicist, medical personnel, etc. In some situations, a "treatment planner" may be another computerized algorithm.

In practice, the modification(s) may be made by a treatment planner <NUM> according to any suitable clinical guideline(s), planning strategy and/or planning practice(s) associated with the treatment planner. For example, in the case of automatic segmentation (to be discussed using <FIG>), modified output data <NUM> may include a modification made to segmentation margins associated with a structure (e.g., OAR or target). In relation dose prediction (to be discussed using <FIG>), modified output data <NUM> may include a modification made to OAR sparing, etc. Any alternative and/or additional modification(s) may be used.

At <NUM> in <FIG>, quality evaluation of modified output data <NUM> may be performed based on (a) first quality indicator data (see <NUM>/<NUM> in <FIG>) associated with modified output data <NUM> and/or (b) second quality indicator data (see <NUM>/<NUM> in <FIG>) associated with treatment planner <NUM>. As used herein, the term "quality indicator data" may refer generally to any qualitative and/or quantitative factor(s) or variable(s) that help inform a decision process as to whether to perform continuous learning based on modified output data <NUM>.

In the example in <FIG>, block <NUM> may include determining first quality indicator data in the form of statistical model parameter data (see <NUM> in <FIG>) by applying statistical model(s) on modified output data <NUM>. Additionally and/or alternatively, block <NUM> may include identifying the ith treatment planner from multiple planners, and determining second quality indicator data in the form of a credibility score C(i) assigned to the ith treatment planner (see <NUM> in <FIG>). The term "credibility score" (to be discussed further using <FIG>) may refer generally to any suitable quantitative measure that represents a reputation or trustworthiness of a particular treatment planner.

At <NUM> in <FIG>, a decision may be made as to whether to accept modified output data <NUM> for continuous learning based on the quality evaluation at block <NUM>. At <NUM> in <FIG>, in response to a decision to accept (see <NUM> in <FIG>), modified Al engine (see <NUM> in <FIG>) may be generated by re-training Al engine <NUM> based on modified output data <NUM> during continuous learning phase <NUM>. Otherwise, at <NUM> in <FIG>, continuous learning based on modified output data <NUM> will not be performed (see also <NUM> in <FIG>). In practice, the re-training process at block <NUM> may involve modifying or improving weight data associated with Al engine <NUM>. Further, block <NUM> may involve generating second training data (see <NUM> in <FIG>) based on modified output data <NUM> for the re-training process. A case weight may also be assigned to modified output data <NUM> based on (a) first quality indicator data <NUM> and/or (b) second quality indicator data <NUM> to influence the re-training process.

Depending on the desired implementation, the re-training process at block <NUM> may be performed for a batch of modified output data <NUM>. For example, the batch may include modification(s) made by multiple treatment planners over a period of time. For quality assurance purposes, the re-training process may be performed periodically so that re-trained or modified Al engine <NUM> may undergo some form of quality assurance checks prior to deployment.

According to examples of the present disclosure, continuous learning phase <NUM> may be improved using additional training data that has been evaluated for quality. If rejected, continuous learning will not be performed, thereby improving efficiency and reducing the negative impact of inferior or redundant training data. Various examples will be discussed below using <FIG>. In particular, an example automatic segmentation will be discussed using <FIG>, an example quality evaluation using <FIG>, an example credibility score assignment using <FIG>, an example dose prediction using <FIG>, an example multi-technique Al engine using <FIG>, various use cases for credibility scores using <FIG>, an example treatment plan using <FIG>, and an example computer system using <FIG>.

<FIG> is a schematic diagram illustrating example quality-aware continuous learning for automatic segmentation <NUM>. In this example, Al engine <NUM> (also referred to as "segmentation engine" below) may be trained using first training data <NUM> during training phase <NUM>; applied to perform automatic segmentation during inference phase <NUM>; and updated during continuous learning phase <NUM>. In practice, the output of automatic segmentation may be used for abnormal organ detection, dose prediction, and/or treatment delivery data estimation, etc..

During training phase <NUM>, segmentation engine <NUM> may be trained to map training image data <NUM> (i.e., input) to training structure data <NUM> (i.e., output). In practice, image data <NUM> may include 2D or 3D images of a patient's anatomy, and captured using any suitable imaging modality or modalities. Structure data <NUM> may identify any suitable contour, shape, size and/or location of structure(s) from image data <NUM>. Example structures may include target(s), OAR(s) or any other structure of interest (e.g., tissue, bone) of the anatomical site. Depending on the desired implementation, structure data <NUM> may identify multiple targets and OARs of any suitable shapes and sizes.

For example, in relation to prostate cancer, image data <NUM> may include images of site = prostate region. In this case, structure data <NUM> may identify a target representing each patient's prostate, and OARs representing proximal healthy structures such as rectum and bladder. In relation to lung cancer treatment, image data <NUM> may include images of a lung region. In this case, structure data <NUM> may identify a target representing cancerous lung tissue, and an OAR representing proximal healthy lung tissue, esophagus, heart, etc. In relation to brain cancer, image data <NUM> may include images of a brain region. Structure data <NUM> may identify a target representing a brain tumor, and an OAR representing a proximal optic nerve, brain stem, etc..

First training data <NUM> may be extracted or derived from past treatment plans developed for multiple past patients according to any desirable planning rule. First training data <NUM> may be pre-processed using any suitable data augmentation approach (e.g., rotation, flipping, translation, scaling, noise addition, and/or cropping, or any combination thereof, etc.) to produce a new dataset with modified properties to improve model generalization using ground truth. In practice, a 3D volume of the patient that will be subjected to radiation is known as a treatment volume, which may be divided into multiple smaller volume-pixels (voxels). In this case, structure data <NUM> may specify a class label (e.g., "target," "OAR," etc.) associated with each voxel in the 3D volume.

In one example in <FIG>, segmentation engine <NUM> includes multiple (N > <NUM>) processing blocks or layers that are each associated with a set of weight data. In this case, training phase <NUM> may involve finding weight data that minimizes a training error between training structure data <NUM>, and estimated structure data (not shown for simplicity) generated by segmentation engine <NUM>. The training process is guided by estimating losses associated with the classification error. A simple example of a loss function would be mean squared error between true and predicted outcome, but the loss function could have more complex formulas. This loss can be estimated from the output of the model, or from any discrete point within the model.

At <NUM> and <NUM> in <FIG>, trained segmentation engine <NUM> may be used to perform automatic segmentation for a particular patient during inference phase <NUM>. Input image data <NUM> associated with that patient is processed using segmentation engine <NUM> to generate output structure data <NUM>. For example, output structure data <NUM> may identify any suitable contour, shape, size and/or location of structure(s) in input image data <NUM>.

At <NUM> in <FIG>, output structure data <NUM> may be modified by the treatment planner <NUM> to achieve a preferred segmentation outcome. For example, modified output structure data <NUM> may include modification(s) made by planner <NUM> to the contour, edges, shape, size and/or location of structure(s) in output structure data <NUM>. In the case of automatic segmentation, a modification may refer to moving, adjusting or redrawing segmentation(s). For example, modified output structure data <NUM> may include different segmentation margin(s), identify an additional and/or alternative structure, etc..

At <NUM>-<NUM> and <NUM> in <FIG>, a quality evaluation may be performed based on any suitable quality indicator data to decide whether to accept modified output data <NUM> for continuous learning. As will be discussed further using <FIG>, the quality evaluation may be based on statistical parameter data <NUM> ("first quality indicator data") that is generated by identifying and applying statistical model(s) on modified output data <NUM>. Alternatively and/or additionally, credibility score <NUM> ("second quality indicator data") associated with treatment planner <NUM> may be obtained.

In one example, the quality evaluation may involve performing a first filtering of modified output data <NUM> based on statistical parameter data <NUM>. If a first threshold is satisfied (and credibility score <NUM> is available), a second filtering is then performed based on credibility score <NUM>. In this case, statistical parameter data <NUM> may be used to determine whether modification(s) made by treatment planner <NUM> provide any measurable improvement according to the statistical model(s). When combined with credibility score <NUM> associated with treatment planner <NUM> that made the modification, modified output data <NUM> may be categorized to be "high value" (i.e., decision = ACCEPT) or "low value" (i.e., decision = REJECT). Some examples will be discussed below using <FIG>.

At <NUM> and <NUM> in <FIG>, in response to a decision to accept (see <NUM>) modified output data <NUM> for continuous learning based on the quality evaluation, segmentation engine <NUM> may be updated based on second training data <NUM>. In practice, second training data <NUM> may include example input-output pair in the form of image data <NUM> processed by segmentation engine <NUM>, and modified output structure data <NUM> that includes modification(s) desired by planner <NUM>. Once continuous learning is performed, modified segmentation engine <NUM> may be deployed for use in the next iteration of inference phase <NUM>. If modification is made to subsequent output structure data generated by modified engine <NUM>, quality evaluation and continuous learning phase <NUM> may be repeated for further improvement.

<FIG> is a flowchart illustrating example process <NUM> for quality evaluation to facilitate quality-aware continuous learning for radiotherapy treatment planning. Example process <NUM> may include one or more operations, functions, or actions illustrated by one or more blocks, such as <NUM> to <NUM>. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. Example process <NUM> may be implemented using any suitable computer system(s), an example of which will be discussed using <FIG>.

In practice, any suitable quality indicator data denoted as Qk (where k = <NUM>,. , K) may be used for quality evaluation. As will be described further below, the quality indicator data may include statistical parameter data (see <NUM>), credibility score (see <NUM>), and/or expert review data (see <NUM>), etc. This way, at <NUM>, quality evaluation of modified output data <NUM> may be performed based on any combination of quality indicator data (Q<NUM>,.

At <NUM> in <FIG>, first quality indicator data (Q<NUM> for k = <NUM>) in the form of statistical parameter data <NUM> may be determined. Block <NUM> may involve identifying and applying statistical model(s) to the modified output data <NUM>. See <NUM>-<NUM> in <FIG>. In practice, the term "statistical model" may refer generally to a model for evaluating a probability of certain measurable quantity (also known as an attribute, feature or parameter) derivable from modified output data <NUM>. This way, statistical parameter data <NUM> may be used to indicate the reliability or validity of modified output data <NUM> associated with planner <NUM> in <FIG>.

Any suitable "statistical model" may be used, ranging from simple metric(s) to more complex model(s). The statistical model may return a single value (i.e., scalar) or multiple values (vector). Further, the "probability" evaluated using a statistical model may be unconditional (e.g., describes the statistics of the quantity over the whole set of training data) or conditional (e.g., describes the statistics of some quantity in condition that certain previous intermediate result has been obtained). For example, structure size could be used unconditionally (e.g., it has the same criteria regardless of the CT scan), or it could be conditional to some values derived from the CT scan (e.g., total area of non-zero region in the slice going through the center of the given organ).

In the case of automatic segmentation in <FIG>, statistical models may be applied to evaluate a probability of certain attribute associated with a patient's structure data <NUM>, such as its shape, size, texture, contour, principal component analysis, material density, geometric distribution, or Hounsfield Units (HU) distribution, and/or relative position of the structure to another structure, etc. For example, in relation to prostate cancer treatment, statistical parameter data <NUM> may be generated based on a statistical model for evaluating the sphericity of a prostate (i.e., target).

In the case of dose prediction, statistical models may be applied to evaluate certain attribute(s) associated with dose data, such as DVH, D20, D50, and/or dose fall-off gradient, etc. Other statistical models may be used for evaluating treatment delivery data, such as smoothness of 2D fluence maps; total motion of leaves in VMAT plan, beam orientations, and/or machine trajectories, any combination thereof, etc. Depending on the radiotherapy treatment planning task that an Al engine is trained to perform, any alternative and/or additional statistical model(s) that are known in the art may be used.

At <NUM> in <FIG>, second quality indicator data (Q2 for k = <NUM>) in the form of credibility score <NUM> may be determined. Block <NUM> may involve identifying a particular ith planner <NUM> responsible for the modification(s) in modified output data <NUM>, and obtaining a credibility score C(i) associated with the planner. See also <NUM>-<NUM> in <FIG>. In practice, the ith planner may be identified from multiple (P > <NUM>) planners who are each assigned with credibility score C(i), where i = <NUM>. Any suitable approach may be used for credibility score assignment. Some examples will be explained using <FIG>, which is a schematic diagram illustrating example process <NUM> for credibility score assignment to facilitate quality-aware continuous learning for radiotherapy treatment planning.

At <NUM> and <NUM>, multiple (N) treatment planning cases may be selected for a "planning contest" that involves multiple (P) treatment planners. At <NUM>, each ith planner-generated output data D(i,j) for each jth case may be obtained, where i = <NUM>,. , P and j = <NUM>,. For example, D(<NUM>, <NUM>) refers to a first planner's (i = <NUM>) output data for a third case (j = <NUM>), and D(<NUM>, N) to a fourth planner's (i = <NUM>) output data for the Nth case (j = N). As previously noted, a "planner" may represent an individual, a group of individual, an institution, a clinical site or network, etc..

In practice, treatment planning cases <NUM> may be new cases specifically selected for the planning contest, or historical cases (e.g., from historical planning contests). These cases may be "sprinkled" into each planner's daily workflow. In both cases, examples of the present disclosure may provide a built-in procedure for cross-calibration and consensus truth analysis for multiple patients. A given case (j) may be used in such a calibration process until adding the output data of new planners fails to substantially affect its corresponding consensus truth G(j). The calibration process may be presented as a training program for planners (e.g., human delineators) to help develop and maintain their skills over time.

In a first example (see <NUM> in <FIG>), credibility score C(i) may be assigned based on a comparison between planner-generated output data D(i,j) and consensus truth data associated with cases j = <NUM>,. For the jth case, its consensus truth data may be denoted as G(j) and determined based on corresponding all planner-generated output data D(i, j) for that case, where i = <NUM>,. In this case, G(j) may be an average or mean of output data D(i, j) for planners i = <NUM>,. Using a normal distribution for D(i, j) as an example, the consensus truth data for the jth case may be determined using G(j) = <NUM>/ <MAT>. Any other distribution may be used. A smaller deviation will lead to a higher credibility score, and a higher deviation to a lower credibility score.

In practice, the term "ground truth" or "absolute truth" may refer generally to the "ideal" or "optimal" output data for the jth case. Since the "ground truth" may not exist, the "consensus truth" G(j) for the jth case may be determined from output data D(i,j) produced by multiple planners to represent the "gold standard. " In this case, a credibility score may be assigned to a planner on the basis of their ability to consistently produce output data in accordance with the current consensus truth. As will be discussed further below, when multiple clusters of practice patterns are identified, the term "consensus truth" may refer to the mean or average of a particular cluster (cohort). For example, if there are M clusters, G(j, m) may represent the consensus truth of the mth cluster for the jth case, where m = <NUM>,.

In a second example (see <NUM> in <FIG>), credibility score C(i) may be assigned based on cluster analysis data <NUM> associated with D(i, j). Here, cluster analysis data <NUM> may identify multiple (M) clusters indicating different self-similar methodologies used by the planners. In practice, cluster analysis may be performed when D(i,j) for a particular jth case converges into separate clusters. In this case, the set of planners may be increased with additional planners (e.g., P + <NUM>, P + <NUM>, and so on). Output data D(i > P,j) generated by the additional planners may then be added until the clusters separates from each other "cleanly" based on any suitable threshold. In this case, the credibility score C(i) of the ith planner may be determined based on the deviation between the planner-generated output data D(i,j) and the consensus truth G(j, m) of a particular mth cluster. As practice patterns move over time (and the number of clusters changes), the continuous learning process may be led by the most credible planners who have higher case weights, or by masses of new practitioners who emerge as a new cluster.

In a third example (see <NUM> in <FIG>), credibility score C(i) may be assigned based on expert review data <NUM> associated with a review of output data generated by each planner, such as by a panel of human experts. In a first approach, the panel may review D(i,j) for the planning contest in <FIG> and/or each planner's historical plans to evaluate their plan quality, and/or segmentation accuracy, etc. In a second approach, each time a particular planner makes a substantial correction (e.g., adjusting a contour by more than <NUM>-<NUM> sigma in inter-observer variability from the result) using a treatment planning system, the correction may be automatically flagged in the system. The correction is then reviewed by a panel of individuals who are regarded as experts in their field. In this case, expert review data <NUM> may indicate the "value" of the correction or output data associated with each planner. Based on expert review data <NUM>, the credibility score C(i) for the planner may be increased, or decreased.

In a fourth example (see <NUM> in <FIG>), credibility score C(i) may be assigned based on algorithm comparison data <NUM> that evaluates a deviation between (i) planner-generated output data that includes D(i,j) and/or historical plans and (ii) algorithm-generated output data. For example, the selected algorithm may be designed to produce result that is close to the consensus truth. In this case, algorithm comparison data <NUM> may include parameter(s) that compare the planner- and algorithm-generated output data, such as similarity measure, mean deviation, and/or self-consistency measure, etc. If the deviation is high, a lower credibility score will be assigned. Otherwise, a higher credibility score will be assigned.

To evaluate internal consistency for a particular planner, one approach is to calculate a mean deviation between the (i) planner-generated output data and (ii) the algorithm-generated output data over many cases, and the offset each planner's result by the mean deviation. The average deviation from the offset (over many cases) may be used as a measure of self-consistency. High self-consistency and a notable mean deviation (offset) from the algorithm indicate a behavioral bias or cluster. Another approach to evaluate internal consistency is to perform a clustering analysis of a planner's historical plans. For example, historical plans may be divided into multiple tranches based on any suitable factor(s), such as time (AM or PM), month, year, and/or geographical location, etc. Multiple Al engines may be trained using training data associated with respective trances. Once trained, the Al engines may be used to process a set of test cases to evaluate whether they converge or diverge, such as based on a temporal factor, etc. A convergence would indicate high internal consistency, whereas a divergence would indicate low internal consistency.

At <NUM> in <FIG>, a credibility score C(i) may be assigned to each ith planner based on any combination of the following: consensus truth data <NUM>, cluster analysis data <NUM>, expert review data <NUM>, and/or algorithm comparison data <NUM>, etc. The examples in <FIG> may be repeated at any suitable frequency to update the credibility score assigned to each planner. Over time, good performers will have a higher credibility score compared to poor performers. Planners may be compared based on their respective credibility scores, such as how they stand in percentiles compared to others.

Referring to <FIG> again, at <NUM>, any additional and/or alternative quality indicator data may be determined, such as expert review data associated with a review of modified output data <NUM> by a panel of human experts. This way, at <NUM>, quality evaluation of modified output data <NUM> may be performed based on any combination of quality indicator data (Q<NUM>,. , QK) discussed above. For example, if the credibility score of a planner is lower than a particular threshold, the modification(s) made by the planner may be ignored during continuous learning for efficiency.

At <NUM>, <NUM> and <NUM> in <FIG>, in response to a decision to accept modified output data <NUM>, second training data <NUM> may be generated to facilitate quality-aware continuous learning. Depending on the desired implementation, a case weight (w) may be assigned to modified output structure data <NUM> to influence the continuous learning process. For example, the case weight may be assigned based on the credibility score C(i) associated with the ith planner <NUM>. A higher C(i) will lead to a higher case weight (w1) to indicate a relatively higher measure of reliability, whereas a lower C(i) will lead to a lower case weight (w<NUM> < w<NUM>) to reduce its influence in modified segmentation engine <NUM>. Additionally and/or alternatively, the case weight (w) may be assigned based on statistical parameter data <NUM> associated with modified output structure data <NUM>. See corresponding <NUM>-<NUM> in <FIG>.

In another example, a case weight (w) may be a function of several factors, including magnitude of change δ (i) made by the ith planner <NUM>, credibility score C(i) associated with the ith planner <NUM>, etc. For a planner with a substantially low credibility score, a small change may be ignored. A big change may be reviewed a panel of human experts. If accepted by the panel, the change may be assigned with a low case weight (w) before being added to the new training set. For a more credible planner with a substantially high credibility score, a small change made by the planner may be accepted. A big change may be reviewed by the panel, and assigned with a higher case weight (w) if accepted. This way, changes made by a more credible planner will potentially have more material influence on modified Al engine <NUM>. It should be understood that any suitable thresholds may be set to determine whether a planner's credibility score is "low" (e.g., C(i) ≤ Cthreshold) or "high" (e.g., C(i) > Cthreshold), and whether the corresponding magnitude of change is "small" (e.g., δ(i) ≤ δthreshold) or "big" (e.g., δ(i) > δthreshold).

Examples of the present disclosure may be implemented for other treatment planning tasks. <FIG> is a schematic diagram illustrating example quality-aware continuous learning for dose prediction <NUM>. In this example, dose prediction engine <NUM> may be trained using first training data <NUM> during training phase <NUM>; applied to perform dose prediction during inference phase <NUM>; and updated during continuous learning phase <NUM> based on quality evaluation.

During training phase (see <NUM> in <FIG>), first training data <NUM> may be used to train dose prediction engine <NUM>. First training data <NUM> may include image and structure data <NUM> (i.e., training input) and dose data <NUM> (i.e., training output) associated with multiple past patients. Dose data <NUM> (e.g., 3D dose data) may specify dose distributions for a target (denoted "DTAR") and an OAR (denoted "DOAR"). For example, in relation to prostate cancer, dose data <NUM> may specify dose distributions for a target representing the patient's prostate, and an OAR representing a proximal healthy structure such as rectum or bladder. In practice, dose data <NUM> may specify the dose distributions for the whole 3D volume, not just the target and OAR volumes. Dose data <NUM> may include spatial biological effect data (e.g., fractionation corrected dose) and/or cover only part of the treatment volume. Any additional input data may be used to train dose prediction engine <NUM>, such as beam geometry data associated with the treatment delivery system.

During inference phase (see <NUM> in <FIG>), dose prediction engine <NUM> may be used to generate output dose data <NUM> based on input image and structure data <NUM> associated with a particular patient. Dose data <NUM> may specify dose distributions for an OAR ("DOAR") and a target ( "DTAR"). Modification(s) may then be made by treatment planner <NUM> (e.g., dosimetrist) to generate modified output dose data <NUM> based on any suitable dose prediction practice(s) preferred by the planner. The modification(s) may be associated with OAR sparing, target coverage, target dose prescription, normal tissue dose, location of dose gradients, steepness of dose gradients, and/or orientation of dose gradients, etc..

During continuous learning phase (see <NUM> in <FIG>), quality evaluation may be performed based on statistical parameter <NUM> and/or credibility score <NUM> associated with planner <NUM> responsible for the modification(s). As discussed using <FIG>, example statistical models for evaluating certain attribute(s) associated with dose data <NUM>/<NUM> may be used, such as D20, D50, dose fall-off gradient, etc. Credibility score <NUM> may be generated according to the example in <FIG>, in which case planner-generated dose data D(i,j) may be used.

In response to determination to accept modified output dose data <NUM> for continuous learning based on the quality evaluation (see <NUM>-<NUM> in <FIG>), modified dose prediction engine <NUM> may be generated by re-training dose prediction engine <NUM> based on second training data <NUM>. Similar to the example in <FIG>, second training data <NUM> may include (input, output) pair in the form of input image and structure data <NUM> and modified output dose data <NUM>. Once validated and approved, modified dose prediction engine <NUM> may be deployed for use in the next iteration of inference phase <NUM>. If modification is made to output dose data generated by modified engine <NUM>, continuous learning phase <NUM> may be repeated for further improvement.

Besides automatic segmentation in <FIG> and dose prediction in <FIG>, quality-aware continuous learning may be implemented for other radiotherapy treatment planning tasks, such as treatment delivery data estimation, and/or treatment outcome prediction, etc. The estimated treatment delivery data (i.e., output data) may include structure projection data, and/or fluence map data, etc. For example, an Al engine may be trained to perform structure projection data, such as based on image data, structure data, and/or dose data, or any combination thereof. Structure projection data may include data relating to beam orientations and machine trajectories for a treatment delivery system.

In another example, an Al engine may be trained to perform fluence map estimation, such as 2D fluence maps for a set of beam orientations or trajectories, machine control point data (e.g., jaw and leaf positions, and/or gantry and couch positions), etc. Fluence maps will be explained further using <FIG>. Any additional and/or alternative training data may be used, such as field geometry data, monitor units (amount of radiation counted by machine), quality of plan estimate (acceptable or not), daily dose prescription (output), field size or other machine parameters, couch positions parameters or isocenter position within patient, treatment strategy (use movement control mechanism or not, boost or no boost), and/or treat or no treat decision, etc..

Examples of the present disclosure may be used to facilitate quality-aware continuous learning for multi-technique AI engines for radiotherapy treatment planning. <FIG> is a schematic diagram illustrating example quality-aware continuous learning <NUM> for radiotherapy treatment planning using a multi-technique Al engine. Similar to the example in <FIG>, dose prediction engine <NUM> in <FIG> is a multi-technique Al engine that is trained to perform dose prediction using multiple techniques. Here, the term "multi-technique AI engine" may refer generally to a single Al engine, or a group of multiple Al engines that are trained according to respective techniques.

During training phase (see <NUM> in <FIG>), first training data <NUM> may be used to train multi-technique dose prediction engine <NUM> to generate multiple sets of output data. First training data <NUM> may include image and structure data <NUM> (i.e., training input) and dose data <NUM> (i.e., training output) associated with multiple techniques denoted as (T<NUM>, T<NUM>, T<NUM>, T<NUM>). Example image data, structure data and dose data explained using <FIG> are also applicable here and will not be repeated for brevity.

During inference phase (see <NUM> in <FIG>), multi-technique dose prediction engine <NUM> may be used to generate multiple sets of output dose data <NUM>-<NUM> based on input image and structure data <NUM> associated with a particular patient. For each technique, dose data <NUM> may specify dose distributions for any suitable structures (e.g., OAR and target). For example, first set <NUM> may be generated based on a first technique (e.g., T<NUM> = <NUM> fields) for treatment delivery, second set <NUM> based on a second technique (e.g., T<NUM> = <NUM> fields), third set <NUM> based on a third technique (e.g., T3 = proton therapy), and fourth set <NUM> based on a fourth technique (e.g., T4 = VMAT). In practice, each set of output dose data may be evaluated based on any suitable factor(s) during inference phase <NUM>, such as deliverability, adherence to dose prescription, collision, OAR limits, and/or machine parameters, or any combination thereof, etc..

In the example in <FIG>, multiple sets <NUM>-<NUM> generated using different techniques (T<NUM>, T<NUM>, T<NUM>, T<NUM>) are then ranked using a cost function. In the case of dose prediction, the cost function may be based on time, complexity, and/or DVH, or any combination thereof, etc. In the case of segmentation, the cost function may be based on segmentation-related parameter(s), such as segmentation mean, etc. In practice, the cost function may be designed for a problem (representing a mathematical ground truth) in which an optimal solution may be found. This way, sets <NUM>-<NUM> may be processed using the cost function and ranked accordingly. Further, a generative adversarial network (GAN), or any other suitable generative model, may be set up to create new techniques to explore options for which no previously trained Al engine exists.

The ranked list (see <NUM>) may then be presented to treatment planner <NUM> for selection. Any suitable metric(s) may be presented along with each technique to guide the selection process, such as the cost function metrics discussed above. Selection may then be made by treatment planner <NUM> (e.g., dosimetrist) to generate modified output dose data <NUM>. For example, if the first technique is selected, first set <NUM> (i.e., based on T<NUM> = <NUM> fields) may be used as modified output dose data <NUM>. Treatment planner <NUM> may also make any additional modification(s) to first set <NUM>. The planner's selection made by used to improve or update the approach used for ranking multiple sets of output data in the next iteration (see arrow from <NUM> to <NUM> in <FIG>).

During continuous learning phase (see <NUM> in <FIG>), quality evaluation may be performed based on statistical parameter <NUM> and/or credibility score <NUM> associated with planner <NUM>. In response to a decision to accept modified output dose data <NUM> for continuous learning based on the quality evaluation (see <NUM>-<NUM> in <FIG>), modified engine <NUM> may be generated by updating or re-training multi-technique engine <NUM> based on second training data <NUM>. Similar to the example in <FIG>, second training data <NUM> may include (input, output) pair in the form of input data <NUM> and modified output dose data <NUM> (e.g., first set <NUM>). Once validated and approved, modified dose prediction engine <NUM> may be deployed for use in the next iteration of inference phase <NUM> to facilitate further improvement. Using the example in <FIG>, multi-technique comparisons may be performed using Al engines, which generally improves efficiency compared to brute force techniques.

According to examples of the present disclosure, a credibility score C(i) may be assigned to a planner to facilitate various aspects of quality-aware continuous learning. Some additional use cases will be discussed using <FIG>, which is a flowchart of example process <NUM> for request processing based on credibility score. Example process <NUM> may include one or more operations, functions, or actions illustrated by one or more blocks, such as <NUM> to <NUM>. The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated based upon the desired implementation. Example process <NUM> may be implemented using any suitable computer system(s), an example of which will be discussed using <FIG>. The computer system may be configured to perform example process <NUM> according to a user's request to perform any one of the following: expert panel selection, planning task assignment, and/or reward determination, etc. The user's request may be generated using any suitable user interface, such as application programming interface (API), graphical user interface (GUI), and/or command line interface (CLI), etc..

In relation to expert panel selection (see <NUM>-<NUM> in <FIG>), the computer system may be configured to select, from multiple treatment planners, a panel of "experts" to review modified output data (e.g., <NUM>/<NUM>/<NUM>) based on their credibility score C(i). This way, an expert panel may be assembled periodically to perform quality evaluation by reviewing modification(s) made by other planners, and deciding whether to accept the modification(s). Depending on the credibility score, the expert panel may be updated over time to reflect changing practices.

In one example, in response to receiving a request for expert panel selection, a planner with the relevant expertise may be identified and selected based on their credibility score C(i). For example, a selected planner may be one whose quality metric seems to have been violated the most by the modification(s) in the modified output data. Each expert (i.e., selected planner) is then requested to review the modified output data, such as by sending the expert an anonymized snapshot of the modification(s). After performing an offline review, each expert may submit their individual decision (e.g., vote) as to whether to accept or reject. A final decision may be made based on review decisions submitted by different experts. If accepted, the modified output data will be used as part of training data for continuous learning purposes.

In relation to task assignment (see <NUM>-<NUM> in <FIG>), certain planning tasks may be assigned to certain planners based on their credibility score. In this case, the computer system may be configured to select, from multiple planners, a particular planner to perform a particular planning task based on their credibility score C(i). In one example, in response to receiving a request for task assignment, planner(s) with relevant expertise relating to the planning task may be identified and selected. For example, a planning task relating to breast cancer might be assigned to a planner who has the highest credibility score and expertise relating to breast cancer treatment. Another planning task relating to prostate cancer might be assigned to a different planner who is most credible in prostate cancer treatment.

In relation to reward determination (see <NUM>-<NUM> in <FIG>), a reward (e.g., monetary reward, promotion, and/or award, etc.) may be determined for a planner based on their credibility score. For example, a reward R(i) for the ith planner may be proportional to the planner's C(i). A more credible planner should receive a better reward compared to a less credible planner. The reward may be used as an incentive for planners to improve their credibility score over time.

During radiotherapy treatment planning, treatment plan <NUM>/<NUM> may be generated based on structure data and/or dose data generated using treatment planning engines discussed above. For example, <FIG> is a schematic diagram of example treatment plan <NUM>/<NUM> generated or improved based on output data in the examples in <FIG>. Treatment plan <NUM> may be delivered using any suitable treatment delivery system that includes radiation source <NUM> to project radiation beam <NUM> onto treatment volume <NUM> representing the patient's anatomy at various beam angles <NUM>.

Although not shown in <FIG> for simplicity, radiation source <NUM> may include a linear accelerator to accelerate radiation beam <NUM> and a collimator (e.g., MLC) to modify or modulate radiation beam <NUM>. In another example, radiation beam <NUM> may be modulated by scanning it across a target patient in a specific pattern with various energies and dwell times (e.g., as in proton therapy). A controller (e.g., computer system) may be used to control the operation of radiation source <NUM> according to treatment plan <NUM>.

During treatment delivery, radiation source <NUM> may be rotatable using a gantry around a patient, or the patient may be rotated (as in some proton radiotherapy solutions) to emit radiation beam <NUM> at various beam orientations or angles relative to the patient. For example, five equally-spaced beam angles 1030A-E (also labelled "A," "B," "C," "D" and "E") may be selected using an Al engine configured to perform treatment delivery data estimation. In practice, any suitable number of beam and/or table or chair angles <NUM> (e.g., five, seven, etc.) may be selected. At each beam angle, radiation beam <NUM> is associated with fluence plane <NUM> (also known as an intersection plane) situated outside the patient envelope along a beam axis extending from radiation source <NUM> to treatment volume <NUM>. As shown in <FIG>, fluence plane <NUM> is generally at a known distance from the isocenter.

In addition to beam angles 1030A-E, fluence parameters of radiation beam <NUM> are required for treatment delivery. The term "fluence parameters" may refer generally to characteristics of radiation beam <NUM>, such as its intensity profile as represented using fluence maps (e.g., 1050A-E for corresponding beam angles 1030A-E). Each fluence map (e.g., 1050A) represents the intensity of radiation beam <NUM> at each point on fluence plane <NUM> at a particular beam angle (e.g., 1030A). Treatment delivery may then be performed according to fluence maps 1050A-E, such as using IMRT, etc. The radiation dose deposited according to fluence maps 1050A-E should, as much as possible, correspond to the treatment plan generated according to examples of the present disclosure.

The above examples can be implemented by hardware, software and/or firmware or a combination thereof. <FIG> is a schematic diagram of example computer system <NUM> for quality-aware continuous learning for radiotherapy treatment planning. In this example, computer system <NUM> (also known as a treatment planning system) may include processor <NUM>, computer-readable storage medium <NUM>, interface <NUM> to interface with radiotherapy treatment delivery system <NUM>, and bus <NUM> that facilitates communication among these illustrated components and other components.

Processor <NUM> is to perform processes described herein with reference to <FIG>. Computer-readable storage medium <NUM> may store any suitable information <NUM>, such as information relating to training data, Al engines, weight data, input data, and/or output data, etc. Computer-readable storage medium <NUM> may further store computer-readable instructions <NUM> which, in response to execution by processor <NUM>, cause processor <NUM> to perform processes described herein. Treatment may be delivered according to treatment plan <NUM> using treatment planning system <NUM> explained using <FIG>, the description of which will not be repeated here for brevity.

Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. Throughout the present disclosure, the terms "first," "second," "third," etc. do not denote any order of importance, but are rather used to distinguish one element from another.

Those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure.

Claim 1:
A method (<NUM>) for a computer system (<NUM>) to perform quality-aware continuous learning for radiotherapy treatment planning, wherein the method (<NUM>) comprises:
obtaining (<NUM>) an artificial intelligence, "Al", engine (<NUM>, <NUM>, <NUM>, <NUM>) that is trained to perform a radiotherapy treatment planning task; and
based on input data (<NUM>, <NUM>, <NUM>, <NUM>) associated with a patient (<NUM>), performing the radiotherapy treatment planning task using the Al engine (<NUM>, <NUM>, <NUM>, <NUM>) to generate output data (<NUM>, <NUM>, <NUM>, <NUM>-<NUM>) associated with the patient (<NUM>);
characterized by:
obtaining (<NUM>) modified output data (<NUM>, <NUM>, <NUM>, <NUM>) that includes one or more modifications made by a treatment planner (<NUM>, <NUM>, <NUM>, <NUM>) to the output data (<NUM>, <NUM>, <NUM>, <NUM>-<NUM>);
performing (<NUM>) quality evaluation based on at least one of (a) first quality indicator data (<NUM>, <NUM>, <NUM>, <NUM>) associated with the modified output data (<NUM>, <NUM>, <NUM>, <NUM>), and (b) second quality indicator data (<NUM>, <NUM>, <NUM>, <NUM>) associated with the treatment planner (<NUM>, <NUM>, <NUM>, <NUM>); and
in response to a decision (<NUM>) to accept the modified output data (<NUM>, <NUM>, <NUM>, <NUM>) based on the quality evaluation, generating a modified Al engine (<NUM>, <NUM>, <NUM>, <NUM>) by re-training the Al engine (<NUM>, <NUM>, <NUM>, <NUM>) based on the modified output data (<NUM>, <NUM>, <NUM>, <NUM>).