Patent Description:
Radiotherapy is used to treat cancers and other ailments in mammalian (e.g., human and animal) tissue. The direction and shape of the radiation beam should be accurately controlled to ensure the tumor receives the prescribed radiation, and the placement of the beam should be such as to minimize damage to the surrounding healthy tissue (often called the organ(s) at risk (OARs)). Treatment planning can be used to control radiation beam parameters, and a radiotherapy device effectuates a treatment by delivering a spatially varying dose distribution to the patient.

Traditionally, for each patient, a radiation therapy treatment plan ("treatment plan") may be created using an optimization technique based on clinical and dosimetric objectives and constraints (e.g., the maximum, minimum, and mean doses to the tumor and critical organs). The treatment planning procedure may include using a three-dimensional (3D) image of the patient to identify a target region (e.g., the tumor) and to identify critical organs near the tumor. Creation of a treatment plan can be a time-consuming process where a planner tries to comply with various treatment objectives or constraints (e.g., dose volume histogram (DVH) objectives), taking into account their individual importance (e.g., weighting) in order to produce a treatment plan which is clinically acceptable. This task can be a time-consuming trial-and-error process that is complicated by the various OARs, because as the number of OARs increases (e.g., <NUM> are commonly segmented in a head-and-neck treatment), so does the complexity of the process. OARs distant from a tumor may be easily spared from radiation, while OARs close to or overlapping a target tumor may be difficult to spare.

Segmentation may be performed to identify the OARs and the area to be treated (for example, a planning target volume (PTV)). After segmentation, a dose plan may be created for the patient indicating the desirable amount of radiation to be received by the PTV (e.g., target) and/or the OARs. The PTV may have an irregular volume and may be unique as to its size, shape, and position. A treatment plan can be calculated after optimizing a large number of plan parameters to ensure that enough dose is provided to the PTV while as low a dose as possible is provided to surrounding healthy tissue. Therefore, a radiation therapy treatment plan may be determined by balancing efficient control of the dose to treat the tumor against sparing any OAR. Typically, the quality of a radiation treatment plan may depend upon the level of experience of the planner. Further complications may be caused by anatomical variations between patients.

Machine learning can play a significant role in assisting with creation of radiotherapy treatment plans. Most machine learning models that can be used to create radiotherapy treatment plans are trained on sets of sensitive data (e.g., medical images) that come from other patients and hospitals. Such models fail to protect the privacy of the patients associated with the medical images used in training such machine learning models. Particularly, machine learning models fail to guarantee that it is not possible to infer whether a particular individual was part of the training set.

Document <CIT> discloses a machine learning process wherein patient images are used for radiotherapy treatment planning.

Further, <NPL>, discloses a machine learning process, using medical data, wherein a plurality of teacher machine learning models generates a plurality of parameters/predictions, which are then aggregated. Then, a student machine learning model learns to predict said parameters, based on the aggregated data. The data provided by the teachers to the student is perturbed to protect privacy of patients" data.

The invention is defined in the independent claims <NUM> and <NUM>. Particular embodiments are defined in the dependent claims.

In the drawings, which are not necessarily drawn to scale, like numerals describe substantially similar components throughout the several views. Like numerals having different letter suffixes represent different instances of substantially similar components. The drawings illustrate generally, by way of example but not by way of limitation, various embodiments discussed in the present document.

The present disclosure includes various techniques to generate radiotherapy treatment plans by using a student machine learning (ML) model that has been trained based on public datasets which have been labeled by teacher machine learning models in a way that maintains data privacy (e.g., in a way that satisfies privacy criteria). The technical benefits include reduced computing processing times to generate radiotherapy treatment plans and accompanying improvements in processing, memory, and network resources used to generate radiotherapy treatment plans. The technical benefits also include enhancements in data privacy guarantees for medical image and information, which increases the training data set size used to train ML models and thereby increases accuracy and reliability of ML models used to generate treatment plans. Particularly, because of the enhanced data privacy guarantees, there is an increased likelihood of hospital and patient participation in sharing sensitive medical information for increasing the training data set size. These radiotherapy treatment plans may be applicable to a variety of medical treatment and diagnostic settings or radiotherapy treatment equipment and devices. Accordingly, in addition to these technical benefits, the present techniques may also result in many apparent medical treatment benefits (including improved accuracy of radiotherapy treatment, reduced exposure to unintended radiation, and the like).

Prior approaches discuss ways for providing data privacy for training ML models that predict a single value. Particularly, such prior approaches discuss ways of training teacher ML models on sensitive data and applying such trained teacher models to estimate values for public data. These prior approaches also discuss using the estimated public data to train a student model to be applied to a new data set. Such approaches also consider adding privacy in the process of training the teacher models but do so on very low-dimensional data sets. This makes such approaches unsuitable for radiotherapy applications, such as those involving medical image analysis. This is because radiotherapy applications work with very high dimension data sets (e.g., a computed tomography (CT) volume may have <NUM><NUM> or approximately <NUM><NUM> voxels) which are very highly correlated. Applying the prior art techniques in such scenarios would be computationally prohibitive and inefficient. In addition to the computational resource challenge of applying prior approaches to radiotherapy applications, unreasonable noise levels would be required to provide adequate privacy guarantees because of the high correlation among the high-dimension data sets. For example, if labels of nearby voxels that are highly correlated are treated as a set of independent classification tasks, according to the prior approaches, each medial image voxel would require an unreasonable amount of noise to be added, making it computationally prohibitive, if not impossible, and would need unreasonably large storage resources for the training data to train a useful segmentation ML model.

The disclosed techniques address these challenges by leveraging a dimension adjustment ML model (e.g., a variational autoencoder (VAE)) to encode (reduce) dimensions of the radiotherapy application data which has been labeled by the teacher models before perturbing the data according to privacy criteria. This allows for low noise levels to be introduced in a computationally efficient manner. Subsequently, the dimension adjustment ML model is applied to the perturbed data to decode (increase) the dimension to restore the size of the radiotherapy application data for use in training the student model. Specifically, the disclosed techniques include receiving a medical image and processing the medical image with a student machine learning model to estimate radiotherapy plan parameters. The student machine learning model is trained to establish a relationship between a plurality of public training medical images and corresponding radiotherapy plan parameters. The radiotherapy plan parameters of the plurality of public training medical images are generated by aggregating a plurality of radiotherapy plan parameter estimates produced by: processing the plurality of public training medical images with a plurality of teacher machine learning models to generate sets of radiotherapy plan parameter estimates; and reducing respective dimensions of the sets of radiotherapy plan parameter estimates, the radiotherapy plan parameters of the plurality of public training medical images being perturbed in accordance with privacy criteria. The disclosed techniques generate a radiotherapy treatment plan based on the estimated radiotherapy plan parameters provided by the student model.

According to some embodiments, a VAE is used to compress segmentation maps that are predicted by the teacher models (e.g., teacher ML models that each estimate segmentation labels for CT images, magnetic resonance (MR) images, and PET images) to obtain one low-dimensional feature vector for each teacher model. Specifically, VAEs can learn highly dense non-linear compressions, and the decompressions of VAEs are robust to noisy perturbations of the feature vectors which allows for such an approach to data privacy.

In a specific example, at training time, sensitive (private) data is split into disjoint subsets of data, and a separate teacher segmentation ML model is trained on each subset. After training the teacher ML models on the sensitive data, for each teacher segmentation ML model, the teacher segmentation ML models are applied to unlabeled public datasets. Outputs or predictions on unlabeled public datasets are collected. A VAE is trained on separate, non-sensitive set of segmentation maps, and the teacher segmentation ML predictions on the unlabeled public datasets are compressed using the trained VAE. For each data point, the compressed feature vectors are aggregated across the teacher segmentation ML models using an average and/or a learned function (e.g., an aggregation ML model). The result of the aggregation is perturbed according to a privacy criteria (e.g., using differential privacy techniques) to add noise from a Normal (Gaussian) distribution appropriately scaled to the desired level of privacy. The perturbed predictions are decompressed using the VAE, and a student segmentation ML model is trained on the public dataset which has been labelled by the decompressed predictions. In an embodiment, the student and teacher ML models are implemented using separate implementations of the same ML architecture and processes.

The teacher and student ML models can be trained to estimate any one or more radiotherapy plan parameter. Specifically, in some embodiments, the teacher and student ML models are trained to estimate segmentations as the radiotherapy plan parameter and specifically are trained to segment a radiotherapy medical image, such as a CT image, an MR image, and/or an sCT image. As another example, in some embodiments, the teacher and student ML models are trained to estimate a three-dimensional (3D) model as the radiotherapy plan parameter and specifically are trained to estimate a 3D model of a radiotherapy medical image, such as a CT image, an MR image, and/or an sCT image. As another example, in some embodiments, the teacher and student ML models are trained to estimate a dose distribution as the radiotherapy plan parameter and specifically are trained to estimate a dose distribution based on one or more radiotherapy images. As another example, in some embodiments, the teacher and student ML models are trained to generate or estimate an sCT image as the radiotherapy plan parameter and specifically are trained to estimate an sCT image based on a CT or MR image. As another example, in some embodiments, the teacher and student ML models are trained to estimate radiotherapy device parameters (e.g., control points) as the radiotherapy plan parameter and specifically are trained to estimate control points based on one or more radiotherapy images and/or distance maps specifying (possibly signed) distances to regions of interest.

<FIG> illustrates an exemplary radiotherapy system <NUM> adapted to perform radiotherapy plan processing operations using one or more of the approaches discussed herein. These radiotherapy plan processing operations are performed to enable the radiotherapy system to provide radiation therapy to a patient based on specific aspects of captured medical imaging data and therapy dose calculations or radiotherapy machine configuration parameters. Specifically, the following processing operations may be implemented as part of the treatment processing logic <NUM>. It will be understood, however, that many variations and use cases of the following trained models and treatment processing logic <NUM> may be provided, including in data verification, visualization, and other medical evaluative and diagnostic settings.

The radiotherapy system <NUM> includes a radiotherapy processing computing system <NUM> which hosts treatment processing logic <NUM>. The radiotherapy processing computing system <NUM> may be connected to a network (not shown), and such network may be connected to the Internet. For instance, a network can connect the radiotherapy processing computing system <NUM> with one or more private and/or public medical information sources (e.g., a radiology information system (RIS), a medical record system (e.g., an electronic medical record (EMR)/ electronic health record (EHR) system), an oncology information system (OIS)), one or more image data sources <NUM>, an image acquisition device <NUM> (e.g., an imaging modality), a treatment device <NUM> (e.g., a radiation therapy device), and treatment data source(s) <NUM>. As an example, the radiotherapy processing computing system <NUM> can be configured to receive a treatment goal of a subject (e.g., from one or more MR images) and generate a radiotherapy treatment plan by executing instructions or data from the treatment processing logic <NUM>, as part of operations to generate treatment plans to be used by the treatment device <NUM> and/or for output on device <NUM>. In an embodiment, the treatment processing logic includes a student ML model that has been trained on public medical information to estimate one or more radiotherapy parameters. The public information used to train the student ML model is generated by a plurality of teacher ML models that are trained on sensitive or private medical information. The teacher ML models generate the public information (e.g., segmentation or labels for CT images) which is then compressed, aggregated, perturbed according to privacy criteria, and decompressed before being made available to the student ML model for training. In this way, the student ML model can be trained on data (e.g., segmentation or labels of CT images) that satisfies privacy criteria, which enhances and ensures data privacy and does not compromise any individual patient's or hospital's identity.

The radiotherapy processing computing system <NUM> may include processing circuitry <NUM>, memory <NUM>, a storage device <NUM>, and other hardware and software-operable features such as a user interface <NUM>, a communication interface (not shown), and the like. The storage device <NUM> may store transitory or non-transitory computer-executable instructions, such as an operating system, radiation therapy treatment plans, training data, software programs (e.g., image processing software, image or anatomical visualization software, artificial intelligence (AI) or ML implementations and algorithms such as provided by deep learning models, ML models, and neural networks (NNs), etc.), and any other computer-executable instructions to be executed by the processing circuitry <NUM>.

In an example, the processing circuitry <NUM> may include a processing device, such as one or more general-purpose processing devices such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), or the like. More particularly, the processing circuitry <NUM> may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processing circuitry <NUM> may also be implemented by one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a System on a Chip (SoC), or the like.

As would be appreciated by those skilled in the art, in some examples, the processing circuitry <NUM> may be a special-purpose processor, rather than a general-purpose processor. The processing circuitry <NUM> may include one or more known processing devices, such as a microprocessor from the Pentium™, Core™, Xeon™, or Itanium® family manufactured by Intel™, the Turion™, Athlon™, Sempron™, Opteron™, FX™, Phenom™ family manufactured by AMD™, or any of various processors manufactured by Sun Microsystems. The processing circuitry <NUM> may also include graphical processing units such as a GPU from the GeForce®, Quadro®, Tesla® family manufactured by Nvidia™, GMA, Iris™ family manufactured by Intel™, or the Radeon™ family manufactured by AMD™. The processing circuitry <NUM> may also include accelerated processing units such as the Xeon Phi™ family manufactured by Intel™. The disclosed embodiments are not limited to any type of processor(s) otherwise configured to meet the computing demands of identifying, analyzing, maintaining, generating, and/or providing large amounts of data or manipulating such data to perform the methods disclosed herein. In addition, the term "processor" may include more than one physical (circuitry based) or software-based processor (for example, a multi-core design or a plurality of processors each having a multi-core design). The processing circuitry <NUM> can execute sequences of transitory or non-transitory computer program instructions, stored in memory <NUM>, and accessed from the storage device <NUM>, to perform various operations, processes, and methods that will be explained in greater detail below. It should be understood that any component in system <NUM> may be implemented separately and operate as an independent device and may be coupled to any other component in system <NUM> to perform the techniques described in this disclosure.

The memory <NUM> may comprise read-only memory (ROM), a phase-change random access memory (PRAM), a static random access memory (SRAM), a flash memory, a random access memory (RAM), a dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), an electrically erasable programmable read-only memory (EEPROM), a static memory (e.g., flash memory, flash disk, static random access memory) as well as other types of random access memories, a cache, a register, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD) or other optical storage, a cassette tape, other magnetic storage device, or any other non-transitory medium that may be used to store information including images, training data, one or more ML model(s) or technique(s) parameters, data, or transitory or non-transitory computer executable instructions (e.g., stored in any format) capable of being accessed by the processing circuitry <NUM>, or any other type of computer device. For instance, the computer program instructions can be accessed by the processing circuitry <NUM>, read from the ROM, or any other suitable memory location, and loaded into the RAM for execution by the processing circuitry <NUM>.

The storage device <NUM> may constitute a drive unit that includes a transitory or non-transitory machine-readable medium on which is stored one or more sets of transitory or non-transitory instructions and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein (including, in various examples, the treatment processing logic <NUM> and the user interface <NUM>). The instructions may also reside, completely or at least partially, within the memory <NUM> and/or within the processing circuitry <NUM> during execution thereof by the radiotherapy processing computing system <NUM>, with the memory <NUM> and the processing circuitry <NUM> also constituting transitory or non-transitory machine-readable media.

The memory <NUM> and the storage device <NUM> may constitute a non-transitory computer-readable medium. For example, the memory <NUM> and the storage device <NUM> may store or load transitory or non-transitory instructions for one or more software applications on the computer-readable medium. Software applications stored or loaded with the memory <NUM> and the storage device <NUM> may include, for example, an operating system for common computer systems as well as for software-controlled devices. The radiotherapy processing computing system <NUM> may also operate a variety of software programs comprising software code for implementing the treatment processing logic <NUM> and the user interface <NUM>. Further, the memory <NUM> and the storage device <NUM> may store or load an entire software application, part of a software application, or code or data that is associated with a software application, which is executable by the processing circuitry <NUM>. In a further example, the memory <NUM> and the storage device <NUM> may store, load, and manipulate one or more radiation therapy treatment plans, imaging data, segmentation data, treatment visualizations, histograms or measurements, one or more AI model data (e.g., weights and parameters of teacher ML models, student ML models, aggregation ML models, and/or dimension adjustment models), training data, labels and mapping data, and the like. It is contemplated that software programs may be stored not only on the storage device <NUM> and the memory <NUM> but also on a removable computer medium, such as a hard drive, a computer disk, a CD-ROM, a DVD, a Blu-Ray DVD, USB flash drive, a SD card, a memory stick, or any other suitable medium; such software programs may also be communicated or received over a network.

Although not depicted, the radiotherapy processing computing system <NUM> may include a communication interface, network interface card, and communications circuitry. An example communication interface may include, for example, a network adaptor, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adaptor (e.g., such as fiber, USB <NUM>, thunderbolt, and the like), a wireless network adaptor (e.g., such as a IEEE <NUM>/Wi-Fi adapter), a telecommunication adapter (e.g., to communicate with <NUM>, <NUM>/LTE, and <NUM>, networks and the like), and the like. Such a communication interface may include one or more digital and/or analog communication devices that permit a machine to communicate with other machines and devices, such as remotely located components, via a network. The network may provide the functionality of a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a wide area network (WAN), and the like. For example, network may be a LAN or a WAN that may include other systems (including additional image processing computing systems or image-based components associated with medical imaging or radiotherapy operations).

In an example, the radiotherapy processing computing system <NUM> may obtain image data <NUM> from the image data source <NUM> (e.g., MR images) for hosting on the storage device <NUM> and the memory <NUM>. In yet another example, the software programs may substitute functions of the patient images such as signed distance functions or processed versions of the images that emphasize some aspect of the image information.

In an example, the radiotherapy processing computing system <NUM> may obtain or communicate image data <NUM> from or to image data source <NUM>. In further examples, the treatment data source <NUM> receives or updates the planning data as a result of a treatment plan generated by the treatment processing logic <NUM>. The image data source <NUM> may also provide or host the imaging data for use in the treatment processing logic <NUM>.

As referred to herein, "public" or "non-sensitive" data includes a collection of data that is publicly available via one or more public databases and which does not contain any data that is subject to data privacy (e.g., the data does not identify a particular individual or identifies an individual who has given express/implied consent). "Private" or "sensitive" data includes a collection of data that includes private hospital and/or patient medical information and is subject to data privacy regulations. In some cases, the private data is not publicly available but is available on a limited access basis to certain organizations and entities without concern for privacy. Such data should ideally not be shared outside of an organization without express permission by a patient or hospital and is maintained in secure, non-publicly accessible databases. In an embodiment, the parameters of the teacher ML models are maintained as private and sensitive data as such parameters can reveal identities of individuals if accessed by an adversary. Data can be received, by way of a query from an external source, and such data can be applied to the private teacher ML models to generate outputs of results. The outputs or results of the teacher ML models can be made publicly accessible as a response to the query, such as by a student ML model, after being subject to privacy criteria according to the disclosed embodiments. The parameters of the student ML model may be publicly accessible.

In an example, computing system <NUM> may communicate with treatment data source <NUM> and input device <NUM> to generate pairs of private prior patient radiotherapy treatment information, such as pairs of labels or segmentations of radiotherapy medical images (e.g., CT, MR and/or sCT images); pairs of a 3D model and one or more corresponding radiotherapy medical images; pairs of a dose distribution and one or more radiotherapy images; pairs of a sCT image and a CT image; pairs of control points and one or more radiotherapy images and/or distance maps; and pairs of individual radiotherapy plan parameter estimates and aggregated results of the plurality of training individual radiotherapy plan parameter estimates. In an example, computing system <NUM> may communicate with treatment data source <NUM> and input device <NUM> to generate pairs of public prior patient radiotherapy treatment information, such as pairs of segmentation maps and compressed dimension segmentation maps; pairs of compressed dimension segmentation maps and uncompressed segmentation maps. Computing system <NUM> may continue generating such pairs of training data until a threshold number of pairs is obtained.

The processing circuitry <NUM> may be communicatively coupled to the memory <NUM> and the storage device <NUM>, and the processing circuitry <NUM> may be configured to execute computer-executable instructions stored thereon from either the memory <NUM> or the storage device <NUM>. The processing circuitry <NUM> may execute instructions to cause medical images from the image data <NUM> to be received or obtained in memory <NUM> and processed using the treatment processing logic <NUM>. Particularly, treatment processing logic <NUM> implements a trained student ML model that is applied to a medical image to generate one or more radiotherapy parameters of a treatment plan. In an example, the student ML model segments a radiotherapy medical image, such as a CT image, an MR image, and/or an sCT image. As another example, in some embodiments, the student ML model estimates a 3D model of a radiotherapy medical image, such as a CT image, an MR image, and/or an sCT image. As another example, in some embodiments, the student ML model estimates a dose distribution based on one or more radiotherapy images. As another example, in some embodiments, the student ML model estimates an sCT image based on a CT image. As another example, in some embodiments, the student ML model estimates control points of a radiotherapy treatment device based on one or more radiotherapy images and/or MR scan distance maps.

In addition, the processing circuitry <NUM> may utilize software programs to generate intermediate data such as updated parameters to be used, for example, by a NN model, machine learning model, treatment processing logic <NUM>, or other aspects involved with generation of a treatment plan as discussed herein. Further, such software programs may utilize the treatment processing logic <NUM> to produce new or updated treatment plan parameters for deployment to the treatment data source <NUM> and/or presentation on output device <NUM>, using the techniques further discussed herein. The processing circuitry <NUM> may subsequently then transmit the new or updated realizable treatment plan parameters via a communication interface and the network to the treatment device <NUM>, where the radiation therapy plan will be used to treat a patient with radiation via the treatment device <NUM>, consistent with results of the trained student ML model implemented by the treatment processing logic <NUM> (e.g., according to the processes discussed below in connection with <FIG> and <FIG>).

In the examples herein, the processing circuitry <NUM> may execute software programs that invoke the treatment processing logic <NUM> to implement functions of ML, deep learning, NNs, and other aspects of artificial intelligence for treatment plan generation from an input radiotherapy medical information (e.g., CT image, MR image, and/or sCT image and/or dose information). For instance, the processing circuitry <NUM> may execute software programs that train, analyze, predict, evaluate, and generate a treatment plan parameter from received radiotherapy medical information as discussed herein.

In an example, the image data <NUM> may include one or more MRI image (e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric MRI, 4D cine MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), Computed Tomography (CT) images (e.g., 2D CT, 2D Cone beam CT, 3D CT, 3D CBCT, 4D CT, 4DCBCT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), Positron Emission Tomography (PET) images, X-ray images, fluoroscopic images, radiotherapy portal images, Single-Photo Emission Computed Tomography (SPECT) images, computer generated synthetic images (e.g., pseudo-CT images) and the like. Further, the image data <NUM> may also include or be associated with medical image processing data (for example, training images, and ground truth images, contoured images, and dose images). In other examples, an equivalent representation of an anatomical area may be represented in non-image formats (e.g., coordinates, mappings, etc.).

In an example, the image data <NUM> may be received from the image acquisition device <NUM> and stored in one or more of the image data sources <NUM> (e.g., a Picture Archiving and Communication System (PACS), a Vendor Neutral Archive (VNA), a medical record or information system, a data warehouse, etc.). Accordingly, the image acquisition device <NUM> may comprise a MRI imaging device, a CT imaging device, a PET imaging device, an ultrasound imaging device, a fluoroscopic device, a SPECT imaging device, an integrated Linear Accelerator and MRI imaging device, CBCT imaging device, or other medical imaging devices for obtaining the medical images of the patient. The image data <NUM> may be received and stored in any type of data or any type of format (e.g., in a Digital Imaging and Communications in Medicine (DICOM) format) that the image acquisition device <NUM> and the radiotherapy processing computing system <NUM> may use to perform operations consistent with the disclosed embodiments. Further, in some examples, the models discussed herein may be trained to process the original image data format or a derivation thereof.

In an example, the image acquisition device <NUM> may be integrated with the treatment device <NUM> as a single apparatus (e.g., a MRI device combined with a linear accelerator, also referred to as an "MRI-Linac"). Such an MRI-Linac can be used, for example, to determine a location of a target organ or a target tumor in the patient so as to direct radiation therapy accurately according to the radiation therapy treatment plan to a predetermined target. For instance, a radiation therapy treatment plan may provide information about a particular radiation dose to be applied to each patient. The radiation therapy treatment plan may also include other radiotherapy information, including control points of a radiotherapy treatment device, such as couch position, beam intensity, beam angles, dose-histogram-volume information, the number of radiation beams to be used during therapy, the dose per beam, and the like.

The radiotherapy processing computing system <NUM> may communicate with an external database through a network to send/receive a plurality of various types of data related to image processing and radiotherapy operations. For example, an external database may include machine data (including device constraints) that provides information associated with the treatment device <NUM>, the image acquisition device <NUM>, or other machines relevant to radiotherapy or medical procedures. Machine data information (e.g., control points) may include radiation beam size, arc placement, beam on and off time duration, machine parameters, segments, multi-leaf collimator (MLC) configuration, gantry speed, MRI pulse sequence, and the like. The external database may be a storage device and may be equipped with appropriate database administration software programs. Further, such databases or data sources may include a plurality of devices or systems located either in a central or a distributed manner.

The radiotherapy processing computing system <NUM> can collect and obtain data, and communicate with other systems, via a network using one or more communication interfaces, which are communicatively coupled to the processing circuitry <NUM> and the memory <NUM>. For instance, a communication interface may provide communication connections between the radiotherapy processing computing system <NUM> and radiotherapy system components (e.g., permitting the exchange of data with external devices). For instance, the communication interface may, in some examples, have appropriate interfacing circuitry from an output device <NUM> or an input device <NUM> to connect to the user interface <NUM>, which may be a hardware keyboard, a keypad, or a touch screen through which a user may input information into the radiotherapy system.

As an example, the output device <NUM> may include a display device that outputs a representation of the user interface <NUM> and one or more aspects, visualizations, or representations of the medical images, the treatment plans, and statuses of training, generation, verification, or implementation of such plans. The output device <NUM> may include one or more display screens that display medical images, interface information, treatment planning parameters (e.g., contours, dosages, beam angles, labels, maps, etc.), treatment plans, a target, localizing a target and/or tracking a target, or any related information to the user. The input device <NUM> connected to the user interface <NUM> may be a keyboard, a keypad, a touch screen or any type of device that a user may use to input information to the radiotherapy system <NUM>. Alternatively, the output device <NUM>, the input device <NUM>, and features of the user interface <NUM> may be integrated into a single device such as a smartphone or tablet computer (e.g., Apple iPad®, Lenovo Thinkpad®, Samsung Galaxy®, etc.).

Furthermore, any and all components of the radiotherapy system <NUM> may be implemented as a virtual machine (e.g., via VMWare, Hyper-V, and the like virtualization platforms) or independent devices. For instance, a virtual machine can be software that functions as hardware. Therefore, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that together function as hardware. For example, the radiotherapy processing computing system <NUM>, the image data sources <NUM>, or like components, may be implemented as a virtual machine or within a cloud-based virtualization environment.

The image acquisition device <NUM> can be configured to acquire one or more images of the patient's anatomy for a region of interest (e.g., a target organ, a target tumor or both). Each image, typically a 2D image or slice, can include one or more parameters (e.g., a 2D slice thickness, an orientation, and a location, etc.). In an example, the image acquisition device <NUM> can acquire a 2D slice in any orientation. For example, an orientation of the 2D slice can include a sagittal orientation, a coronal orientation, or an axial orientation. The processing circuitry <NUM> can adjust one or more parameters, such as the thickness and/or orientation of the 2D slice, to include the target organ and/or target tumor. In an example, 2D slices can be determined from information such as a 3D CBCT or CT or MRI volume. Such 2D slices can be acquired by the image acquisition device <NUM> in "near real time" while a patient is undergoing radiation therapy treatment (for example, when using the treatment device <NUM> (with "near real time" meaning acquiring the data in at least milliseconds or less)).

The treatment processing logic <NUM> in the radiotherapy processing computing system <NUM> implements a student ML model, which involves the use of a trained (learned) student ML model. This ML model may be provided by a NN trained as part of a NN model. One or more teacher ML models may be provided by a different entity or at an off-site facility relative to treatment processing logic <NUM> and is accessible by issuing one or more queries to the off-site facility. The teacher ML models may include architectures and processes that complement that of the student ML model. The teacher ML models are each implemented using a same or common set of ML parameters that are private, and the student ML model is implemented using a set of public ML parameters. The discussions pertaining to ML models are equally applicable to any ML model discussed herein.

Machine learning (ML) algorithms or ML models or techniques can be summarized as function approximation. Training data consisting of input-output pairs of some type (e.g., CT-images with segmentations) are acquired from, e.g., expert clinicians, and a function is "trained" to approximate this mapping. Some methods involve NNs. In these, a set of parametrized functions Aθ are selected, where θ is a set of parameters (e.g., convolution kernels and biases) that are selected by minimizing the average error over the training data. If the input-output pairs are denoted by (xm, ym), the function can be formalized by solving a minimization problem such as Equation <NUM>: <MAT>.

Once the network has been trained (e.g., θ has been selected), the function Aθ can be applied to any new input. For example, in the above setting of segmentation of CT images, a never before seen CT image can be fed into Ae, and a segmentation is estimated that matches what an expert clinician would find. In some cases, an autoencoder which is an unsupervised model can be trained by attempting to reconstruct the inputs, such as by setting y = x in Equation <NUM>.

Simple NNs consist of an input layer, a middle or hidden layer, and an output layer, each containing computational units or nodes. The hidden layer(s) nodes have input from all the input layer nodes and are connected to all nodes in the output layer. Such a network is termed "fully connected. " Each node communicates a signal to the output node depending on a nonlinear function of the sum of its inputs. For a classifier, the number of input layer nodes typically equals the number of features for each of a set of objects being sorted into classes, and the number of output layer nodes is equal to the number of classes. A network is trained by presenting it with the features of objects of known classes and adjusting the node weights to reduce the training error by an algorithm called backpropagation. Thus, the trained network can classify novel objects whose class is unknown.

Neural networks have the capacity to discover relationships between the data and classes or regression values, and under certain conditions, can emulate any function y = f(x) including non-linear functions. In ML, an assumption is that the training and test data are both generated by the same data-generating process, pdata, in which each {xi, yi} sample is identically and independently distributed (i. In ML, the goals are to minimize the training error and to make the difference between the training and test errors as small as possible. Underfitting occurs if the training error is too large; overfitting occurs when the train-test error gap is too large. Both types of performance deficiency are related to model capacity: large capacity may fit the training data very well but lead to overfitting, while small capacity may lead to underfitting.

<FIG> illustrates an exemplary image-guided radiotherapy device <NUM> that includes a radiation source, such as an X-ray source or a linear accelerator, a couch <NUM>, an imaging detector <NUM>, and a radiation therapy output <NUM>. The radiation therapy device <NUM> may be configured to emit a radiation therapy beam <NUM> to provide therapy to a patient. The radiation therapy output <NUM> can include one or more attenuators or collimators, such as a MLC.

As an example, a patient can be positioned in a region <NUM>, supported by the treatment couch <NUM>, to receive a radiation therapy dose according to a radiation therapy treatment plan. The radiation therapy output <NUM> can be mounted or attached to a gantry <NUM> or other mechanical support. One or more chassis motors (not shown) may rotate the gantry <NUM> and the radiation therapy output <NUM> around couch <NUM> when the couch <NUM> is inserted into the treatment area. In an example, gantry <NUM> may be continuously rotatable around couch <NUM> when the couch <NUM> is inserted into the treatment area. In another example, the gantry <NUM> may rotate to a predetermined position when the couch <NUM> is inserted into the treatment area. For example, the gantry <NUM> can be configured to rotate the therapy output <NUM> around an axis ("A"). Both the couch <NUM> and the radiation therapy output <NUM> can be independently moveable to other positions around the patient, such as moveable in transverse direction ("T"), moveable in a lateral direction ("L"), or as rotation about one or more other axes, such as rotation about a transverse axis (indicated as "R"). A controller communicatively connected to one or more actuators (not shown) may control the couch <NUM>'s movements or rotations in order to properly position the patient in or out of the radiation therapy beam <NUM>, according to a radiation therapy treatment plan. Both the couch <NUM> and the gantry <NUM> are independently moveable from one another in multiple degrees of freedom, which allows the patient to be positioned such that the radiation therapy beam <NUM> can precisely target the tumor.

The coordinate system (including axes A, T, and L) can have an origin located at an isocenter <NUM>. The isocenter <NUM> can be defined as a location where the central axis of the radiation therapy beam <NUM> intersects the origin of a coordinate axis, such as to deliver a prescribed radiation dose to a location on or within a patient. Alternatively, the isocenter <NUM> can be defined as a location where the central axis of the radiation therapy beam <NUM> intersects the patient for various rotational positions of the radiation therapy output <NUM> as positioned by the gantry <NUM> around the axis A.

Gantry <NUM> may also have an attached imaging detector <NUM>. The imaging detector <NUM> is preferably located opposite to the radiation source (output <NUM>) and, in an example, the imaging detector <NUM> can be located within a field of the therapy radiation beam <NUM>. The imaging detector <NUM> can be mounted on the gantry <NUM>, preferably opposite the radiation therapy output <NUM>, so as to maintain alignment with the radiation therapy beam <NUM>. The imaging detector <NUM> rotates about the rotational axis as the gantry <NUM> rotates. In an example, the imaging detector <NUM> can be a flat panel detector (e.g., a direct detector or a scintillator detector). In this manner, the imaging detector <NUM> can be used to monitor the radiation therapy beam <NUM>, or the imaging detector <NUM> can be used for imaging the patient's anatomy, such as portal imaging. The control circuitry of radiation therapy device <NUM> may be integrated within the radiotherapy system <NUM> or remote from it.

In an illustrative example, one or more of the couch <NUM>, the therapy output <NUM>, or the gantry <NUM> can be automatically positioned, and the therapy output <NUM> can establish the therapy radiation beam <NUM> according to a specified dose for a particular therapy delivery instance. A sequence of therapy deliveries can be specified according to a radiation therapy treatment plan, such as using one or more different orientations or locations of the gantry <NUM>, couch <NUM>, or therapy output <NUM>. The therapy deliveries can occur sequentially but can intersect in a desired therapy locus on or within the patient, such as at the isocenter <NUM>. A prescribed cumulative dose of radiation therapy can thereby be delivered to the therapy locus while damage to tissue nearby the therapy locus can be reduced or avoided.

Thus, <FIG> specifically illustrates an example of a radiation therapy device <NUM> operable to provide radiotherapy treatment to a patient consistent with or according to a radiotherapy treatment plan, with a configuration where a radiation therapy output can be rotated around a central axis (e.g., an axis "A"). Other radiation therapy output configurations can be used. For example, a radiation therapy output can be mounted to a robotic arm or manipulator having multiple degrees of freedom. In yet another example, the therapy output can be fixed, such as located in a region laterally separated from the patient, and a platform supporting the patient can be used to align a radiation therapy isocenter with a specified target locus within the patient. In another example, a radiation therapy device can be a combination of a linear accelerator and an image acquisition device. In some examples, the image acquisition device may be an MRI, an X-ray, a CT, a CBCT, a spiral CT, a PET, a SPECT, an optical tomography, a fluorescence imaging, ultrasound imaging, or radiotherapy portal imaging device, and the like, as would be recognized by one of ordinary skill in the art.

As discussed above, the training data used by treatment processing logic <NUM> may include a plurality of previous private or public estimated treatment plan parameters paired with prior private or public patient images that are stored in a memory <NUM>. For example, the stored training data may include diagnostic images, treatment images (dose maps), segmentation information, and the like, associated with one or more previous estimated treatment plans. The training data may include a plurality of training samples. Each training sample may comprise a feature vector and a corresponding output vector.

The feature vector may include one or more feature elements. Each feature element may indicate an observation of a medical image (e.g., provided by image acquisition device <NUM>) used in a past radiotherapy session. The observation may be a distance between a volume (e.g., a voxel) and an anatomical region, such as a target or the surface of the body part in the medical image. In another example, the observation may include spatial coordinates of an anatomical region or a probability that an anatomical region includes a particular tissue type. In another example, the feature element may include patient-specific information, responsible physician, organ or volume of interest segmentation data, functional organ modeling data (e.g., serial versus parallel organs, and appropriate dose response models), radiation dosage (e.g., also including DVH information), lab data (e.g., hemoglobin, platelets, cholesterol, triglycerides, creatinine, sodium, glucose, calcium, weight), vital signs (blood pressure, temperature, respiratory rate and the like), genomic data (e.g., genetic profiling), demographics (age, sex), other diseases affecting the patient (e.g., cardiovascular or respiratory disease, diabetes, radiation hypersensitivity syndromes and the like), medications and drug reactions, diet and lifestyle (e.g., smoking or non-smoking), environmental risk factors, tumor characteristics (histological type, tumor grade, hormone and other receptor status, tumor size, vascularity cell type, cancer staging, gleason score), previous treatments (e.g., surgeries, radiation, chemotherapy, hormone therapy), lymph node and distant metastases status, genetic/protein biomarkers (e.g., such as MYC, GADD45A, PPM1D, BBC3, CDKN1A, PLK3, XPC, AKT1, RELA, BCL2L1, PTEN, CDK1, XIAP, and the like), single nucleotide polymorphisms (SNP) analysis (e.g., XRCC1, XRCC3, APEX1, MDM2, TNFR, MTHFR, MTRR, VEGF, TGFβ, TNFα), and the like. The feature vector may include one or more such feature elements, regardless of whether these feature elements are related to each other or not.

The output vector may include one or more output elements. Each output element may indicate a corresponding estimated plan outcome or parameter in the past radiotherapy session based on the observation(s) included in the feature vector. For example, the output element may include the estimated dose applied or received at a particular spatial location (e.g., a voxel). In another example, the output element may include a patient survival time based on observations such as a treatment type, treatment parameters, patient history, and/or patient anatomy. Additional examples of output elements include, but not limited to, a normal tissue complication probability (NTCP), a region displacement probability during treatment, or a probability that a set of coordinates in a reference image is mapped to another set of coordinates in a target image. The output vector may include one or more such output elements, regardless of whether these output elements are related to each other or not.

As an example of an embodiment, an output element may include a dose to be applied to a voxel of a particular OAR. Further, a feature element may be used to determine the output element. The feature element may include a distance between the voxel in the OAR and the closest boundary voxel in a target tumor. Therefore, the feature element may include a signed distance x indicating the distance between a voxel in an OAR and the closest boundary voxel in a target for the radiation therapy. The output element may include a dose D in the voxel of the OAR from which x is measured. In some other embodiments, each training sample may correspond to a particular voxel in the target or OAR, such that multiple training samples within the training data correspond to the whole volume of the target or OAR and other anatomical portions subject to the radiotherapy treatment.

<FIG> illustrates an exemplary data flow for training and use of one or more private and public machine learning models to generate a radiotherapy treatment plan, according to some examples of the disclosure. The data flow includes training input <NUM>, ML model(s) (technique(s)) training <NUM>, and model(s) usage <NUM>.

Training input <NUM> includes model parameters <NUM> and training data <NUM> which may include paired training data sets <NUM> (e.g., input-output training pairs) and constraints <NUM>. Model parameters <NUM> stores or provides the parameters or coefficients of corresponding ones of machine learning models Âθ. Model parameters <NUM> may include private parameters for teacher ML models and aggregation models and public parameters for the student model and the dimension adjustment models. The model parameters <NUM> may be shared and the same for each teacher ML model Âθ. The model parameters <NUM> for the student ML model Âθ may differ from those of the teacher ML model even though the teacher and student ML models have the same architecture and processes. During training, these parameters <NUM> are adapted based on the input-output training pairs of the training data sets <NUM>. After the parameters <NUM> are adapted (after training), the parameters are used by trained models <NUM> to implement the respective one of the trained machine learning models Âθ (e.g., the trained student model Âθ, the trained teacher model Âθ, the trained aggregation model Âθ, and/or the trained dimension adjustment model Âθ) on a new set of data <NUM>.

Training data <NUM> includes constraints <NUM> which may define the physical constraints of a given radiotherapy device. The training data sets <NUM> may include sets of private and public input-output pairs, such as a pairs of private prior patient radiotherapy treatment information, such as pairs of labels or segmentations of radiotherapy medical images (e.g., CT, MR and/or sCT images); pairs of a 3D model and one or more corresponding radiotherapy medical images; pairs of a dose distribution and one or more radiotherapy images; pairs of a sCT image and a CT image; pairs of control points and one or more radiotherapy images and/or distance maps; and pairs of individual radiotherapy plan parameter estimates and aggregated results of the plurality of training individual radiotherapy plan parameter estimates; pairs of public prior patient radiotherapy treatment information, such as pairs of segmentation maps and compressed dimension segmentation maps; and pairs of compressed dimension segmentation maps and uncompressed segmentation maps. Some components of training input <NUM> may be stored at a different off-site facility or facilities than other components. For example, private parameters and private training data pairs may include sensitive or private information and should be restricted for access by only authorized parties or queries.

Machine learning model(s) training <NUM> trains one or more machine learning techniques Âθ based on the private and public sets of input-output pairs of training data sets <NUM>. For example, the model training <NUM> may train the student ML model parameters <NUM> by minimizing a first loss function based on public training patient input data and the corresponding radiotherapy plan parameters that have been generated using the teacher ML models and that satisfy privacy criteria. For example, the treatment model training <NUM> may train the teacher ML model parameters <NUM> by minimizing a second loss function based on private training patient input data and the corresponding private radiotherapy plan parameters. For example, the treatment model training <NUM> may train aggregation ML model parameters <NUM> by minimizing a third loss function based on individual radiotherapy plan parameter estimates and aggregated results of a plurality of training individual radiotherapy plan parameter estimates and an amount of perturbation needed to satisfy privacy criteria. For example, the treatment model training <NUM> may train the dimension adjustment ML model parameters <NUM> (e.g., a VAE encoder and decoder, an autoencoder, and/or principal component analysis) by minimizing a fourth loss function based on public segmentation maps. In some embodiments, the teacher ML model, the aggregation ML model and the dimension adjustment ML model are trained in parallel or sequentially before the student ML model is trained.

The result of minimizing these loss functions for multiple sets of training data trains, adapts, or optimizes the model parameters <NUM> of the corresponding ML models. Model training <NUM> may be performed in accordance with the process and dataflow described in connection with <FIG> and <FIG>. In this way, the student ML model is trained to establish a relationship between the un-labelled public radiotherapy medical information and estimated radiotherapy plan parameters that are provided and privatized by the teacher ML models.

In some embodiments, after each of the machine learning models Âθ is trained, new data <NUM> including one or more patient input parameters (e.g., an MR image, a medical image, segmentation information of an object of interest associated with the patient, or dose prescription information) may be received. The trained machine learning technique Âθ may be applied to the new data <NUM> to generate generated results <NUM> including one or more parameters of a radiotherapy treatment plan. For example, after being trained on sensitive private medical information, the trained teacher ML models <NUM> may be applied to public radiotherapy information (e.g., public medical images) to generate respective one or more radiotherapy parameters (e.g., labels of a CT image, an sCT image corresponding to a CT image, the medical images, processed versions of the medical images, a dose distribution for an image, and/or control points of a radiotherapy treatment device corresponding to the radiotherapy images and/or MR scan distance maps) of a treatment plan. The generated one or more radiotherapy parameters of the treatment plan are processed by the trained dimension adjustment ML model <NUM> to reduce a dimension of the one or more radiotherapy parameters of the treatment plan (e.g., by processing the plurality of radiotherapy plan parameters with a variational autoencoder, autoencoder, principal component analysis, or homomorphic compression). Then, the reduced dimension one or more radiotherapy parameters of the treatment plan are processed by the trained aggregation ML model <NUM> (e.g., which computes a mean, a trimmed mean, a median, or a generalized f-mean of the reduced dimension radiotherapy plan parameters) and are perturbed according to a privacy criteria (e.g., differential privacy, Rényi differential privacy, concentrated differential privacy, mutual information, conditional entropy, Fisher information, generative adversarial privacy, or k-anonymity).

In some embodiments, after each of the machine learning models Âθ is trained, new data <NUM> including one or more patient input parameters (e.g., an MR image, a medical image, segmentation information of an object of interest associated with the patient, or dose prescription information) may be received. The trained machine learning technique Âθ may be applied to the new data <NUM> to generate generated results <NUM> including one or more parameters of a radiotherapy treatment plan. For example, the new data <NUM> may include public medical images which are processed by the trained dimension adjustment ML model <NUM> to reduce a dimension of the public medical images in the new data <NUM> plan (e.g., by processing the medical images with a variational autoencoder, autoencoder, principal component analysis, or homomorphic compression). After being trained on sensitive private medical information, the trained teacher ML models <NUM> may be applied to reduced dimension public medical images) to generate respective one or more radiotherapy parameters (e.g., labels of a CT image, an sCT image corresponding to a CT image, the medical images, processed versions of the medical images, a dose distribution for an image, and/or control points of a radiotherapy treatment device corresponding to the radiotherapy images and/or MR scan distance maps) of a treatment plan. Then, the generated one or more radiotherapy parameters of the treatment plan are processed by the trained aggregation ML model <NUM> (e.g., which computes a mean, a trimmed mean, a median, or a generalized f-mean of the reduced dimension radiotherapy plan parameters) and are perturbed according to a privacy criteria (e.g., differential privacy, Rényi differential privacy, concentrated differential privacy, mutual information, conditional entropy, Fisher information, generative adversarial privacy, or k-anonymity) to be used to train the student ML model. Alternatively, the generated one or more radiotherapy parameters of the treatment plan are first perturbed according to a privacy criteria (e.g., differential privacy, Rényi differential privacy, concentrated differential privacy, mutual information, conditional entropy, Fisher information, generative adversarial privacy, or k-anonymity), processed by the trained aggregation ML model <NUM> (e.g., which computes a mean, a trimmed mean, a median, or a generalized f-mean of the reduced dimension radiotherapy plan parameters) and then the student ML model is then trained based on the output of the aggregation ML model <NUM>.

In an embodiment, the perturbation is performed by adding noise comprising samples from a Gaussian, Beta, Dirichlet, or Laplace distribution. Then, the perturbed aggregated one or more radiotherapy parameters of the treatment plan are processed by the trained dimension adjustment ML model <NUM> to restore their dimension (increase their dimension) (e.g., by processing the perturbed aggregated plurality of radiotherapy plan parameters with a variational autoencoder, autoencoder, principal component analysis, or homomorphic compression) and are then provided to train the student ML model <NUM>. After the student ML model <NUM> is trained, the trained student ML model <NUM> is applied to new radiotherapy information (e.g., a medical image) to generate one or more radiotherapy parameters (e.g., labels of a CT image, an sCT image corresponding to a CT image, a dose distribution for an image, the medical image, a processed version of the medical image, and/or control points of a radiotherapy treatment device corresponding to the radiotherapy images and/or MR scan distance maps).

<FIG> illustrates an exemplary data flow 400A for training and use of one or more private and public machine learning models to generate a radiotherapy treatment plan, according to some examples of the disclosure. The data flow 400A includes teacher ML model training portion <NUM>, dimension adjustment ML models 430A and 430B, aggregator ML model <NUM>, privacy criteria noise addition module <NUM>, student ML model <NUM>, and public training data <NUM>. The ML models discussed in connection with <FIG> and <FIG> may be trained and applied in a similar manner as discussed above in connection with <FIG>.

Initially, the teacher ML model training portion <NUM> operates on a set of private and sensitive training data <NUM> to train a plurality of teacher ML models <NUM>. Each of the plurality of teacher ML models <NUM> may be identical in implementation and leverage a common set of ML parameters <NUM>. In some implementations, the sensitive training data <NUM> is provided by one or more hospitals and/or patients. The sensitive training data <NUM> may include one or more pairs of labels or segmentations of radiotherapy medical images (e.g., CT, MR and/or sCT images); pairs of a 3D model and one or more corresponding radiotherapy medical images; pairs of a dose distribution and one or more radiotherapy images; pairs of a sCT image and a CT image; pairs of control points and one or more radiotherapy images and/or MR scan distance maps; and pairs of individual radiotherapy plan parameter estimates and aggregated results of the plurality of training individual radiotherapy plan parameter estimates. The pairs included among the sensitive training data <NUM> may depend on the type of teacher ML model <NUM> that is used. For example, when the teacher ML model <NUM> and its complement student ML model <NUM> are configured to generate labels or segment for CT images, the pairs of sensitive training data <NUM> may include pairs of CT images and corresponding labels. As another example, when the teacher ML model <NUM> and its complement student ML model <NUM> are configured to generate sCT images for a CT image, the pairs of sensitive training data <NUM> may include pairs of sCT images and corresponding CT images.

The sensitive training data <NUM> may be divided into disjoint datasets <NUM>. Each dataset <NUM> is provided to a respective instance of the teacher ML models <NUM>. In one example, the datasets <NUM> in one set may include a set of the sensitive training data <NUM> from one hospital and/or one collection of patients and the datasets <NUM> in another set may include a set of the sensitive training data <NUM> from another hospital and/or another collection of patients. The sensitive training data <NUM> may be inaccessible by unauthorized parties outside of the portion <NUM>. Namely, the sensitive training data <NUM> may be stored in one or more secure databases external to the public training data <NUM> and/or student model <NUM>.

As an example, when the teacher ML models <NUM> are configured to generate labels or segments for CT images, the teacher ML models <NUM> operate on various sets of the sensitive training data <NUM> to be trained to estimate labels or segment a CT image given the CT image. For example, the teacher ML models <NUM> may receive a respective training CT image and generate respective labels or segments for the training CT image. The generated respective labels or segments are compared with ground truth labels or segments in the paired training data. Based on a deviation between the generated respective labels or segments and ground truth labels or segments in the paired training data and a loss function associated with the teacher ML models, the teacher ML model parameters are updated until a threshold number of iterations of pairs of training data is processed and/or until a deviation reaches a threshold amount.

After being trained, the teacher models <NUM> process public training data <NUM> to generate radiotherapy treatment plan parameters for the public training data <NUM>. For example, the public training data <NUM> may include un-labeled unsegmented CT images. The teacher models <NUM> process the public training data <NUM> to generate labels or segment the CT images in the public training data <NUM>. The results are provided to a trained dimension adjustment ML model 430A. The trained dimension adjustment ML model 430A may include a variational autoencoder, an autoencoder, principal component analysis, or homomorphic compression trained based on public segmentation maps. The trained dimension adjustment ML model 430A may reduce a dimension of the labeled CT images generated by the teacher ML models <NUM>.

After reducing a dimension of the labeled CT images generated by the teacher ML models <NUM>, the output of the trained dimension adjustment ML model 430A is processed by an aggregator ML model <NUM>. In an embodiment, the aggregator ML model <NUM> computes a mean, a trimmed mean, a median, or a generalized f-mean of the individual reduced dimension labeled CT images generated by the teacher ML models <NUM>. In other embodiments, the aggregator ML model <NUM> processes the received individual reduced dimension labeled CT images and estimates an aggregation of the individual reduced dimension labeled CT images.

An output of the aggregator ML model <NUM> is provided to the privacy criteria noise addition module <NUM>. Privacy criteria noise addition module <NUM> adds noise to the aggregated information provided by the aggregator ML model <NUM> according to a selected privacy criteria level. The privacy criteria can include differential privacy, Rényi differential privacy, concentrated differential privacy, mutual information, conditional entropy, Fisher information, generative adversarial privacy, or k-anonymity. The privacy criteria noise addition module <NUM> adds noise to the aggregated information by adding samples from a Gaussian, Beta, Dirichlet, or Laplace distribution based on the privacy criteria.

Differential privacy offers a rigorous guarantee for database access mechanisms. It is based on the notion of dataset adjacency (also referred to as neighborhood): two datasets d, d' are defined to be adjacent if they differ in the presence of a single dataset record. Differential privacy then requires that the outputs of a mechanism be indistinguishable for adjacent inputs. A randomized mechanism M : D → R satisfies ( ∈, δ) - differential privacy if for any two adjacent inputs d, d' ∈ D and for any S ⊆ R, Pr[M(d) ∈ S] ≤ e∈ Pr[M(d') ∈ S] + δ.

According to some embodiments, a record is an image-segmentation pair and M is a randomized training algorithm. That is, M(d) is a random variable that represents the model parameters of the ML model, M, are trained on dataset d. In an implementation, the randomness comes from the training algorithm, not from the data. Differential privacy is agnostic towards the data distribution, thus d is treated as a constant.

The output of the privacy criteria noise addition module <NUM> is provided to the dimension adjustment model 430B. Dimension adjustment model 430B may perform the reverse operation of dimension adjustment model 430A. Namely, dimension adjustment model 430B may increase or restore the dimension of the now noise aggregated information that is provided by the privacy criteria noise addition module <NUM>. The output of the dimension adjustment model 430B is provided to the student ML model <NUM> to train the student ML model <NUM>. Namely, the student ML model <NUM> can now be trained based on the public training data <NUM> using the privatized noisy estimates of the radiotherapy treatment plan parameters provided by the trained teacher models <NUM>. The privacy parameters ∈ and δ also straight-forwardly translate to lower bounds on the false-positive and false-negative rate of any discriminator that tries to distinguish between d and d'. This may also be helpful in deciding which parameter values can be considered strong enough.

A randomized mechanism privacy criteria M: D → R satisfies (∈, δ) - differential privacy if and only if the following conditions are satisfied for any two adjacent inputs d, d' ∈ D and any rejection region S ⊆ R: <MAT> <MAT> where PFP and PFN are, respectively, the false-positive rate and false-negative rate for the classifier that outputs d if M E S and d' otherwise. There are a few basic mechanisms that add differential privacy to existing non-private functions. They are based the notion of the function's sensitivity.

The lp-sensitivity Sp(f) of a function f: D → Rn is defined as: <MAT> where the maximum is taken over adjacent d, d' ∈ D. The standard mechanisms then add noise to a function, calibrated to the sensitivity and privacy parameters. For Laplace mechanism privacy criteria, let f: D → Rn be a function and Lap(b) be the Laplace distribution with scale b. If <MAT> and α = (α<NUM>,. , αn)T then the mechanism M(d) = f (d) + α is (∈, <NUM>) is differentially private. For Gaussian mechanism privacy criteria, Let f : D → Rn be a function and N (µ, σ<NUM>) be the Normal distribution with mean µ and variance σ<NUM>. Let <MAT> with c<NUM> > <NUM> ln(<NUM>/δ). If α<NUM>,. , αn iid N(<NUM>, σ<NUM>) and α = (α<NUM>,. , αn)T then the mechanism M(d) = f (d) + α is (∈, δ) is differentially private. Rényi Differential Privacy is a generalization of differential privacy in the sense that every Rényi Differential Privacy mechanism satisfies differential privacy but not the other way around. Due to its stricter requirements, it allows for a sharper analysis of cumulative privacy loss and defines indistinguishability in terms of the more general Rényi divergence.

For Rényi divergence privacy criteria, let X ~ p and Y ~ q be random variables. Their Rényi divergence of order α is defined as <MAT> for any α > <NUM>. Rényi Differential Privacy requires the Rényi divergence of the outputs of a mechanism to be small when run on adjacent inputs. For Renyi Differential Privacy, a randomized mechanism M : D <NUM> → R satisfies (α, ∈) Renyi Differential Privacy if for any two adjacent inputs d,d' E D Dα(M(d)∥M(d')) ≤ ∈.

Another privacy criteria is k-anonymity, which is a property of databases. For a set of quasi identifiers recorded in the database - such as zip code or date of birth - k-anonymity demands that any combination of values of the quasi identifiers that is present in the database occurs at least k times.

Another privacy criterion is conditional entropy, for which X given Y is defined as: <MAT> and intuitively represents the additional number of bits needed to describe X after having observed Y. In a machine learning context, Y could refer to the model parameters and X to the training data. A high conditional entropy would then limit the ability of an adversary to reconstruct the training set from the published model parameters. Conditional entropy implies a lower bound on the expected estimation error.

Another privacy criterion is mutual information, in which the mutual information between X and Y is defined as: <MAT> and intuitively represents the amount of information shared between X and Y. Alternatively, one can describe mutual information as the amount of information gained about X by observing Y. While conditional entropy limits certainty of the reconstruction of the training data, mutual information instead limits the reduction of uncertainty. Conditional entropy and mutual information are directly related through entropy, which describes how uncertain an adversary is about X before observing Y. Mutual information is precisely the difference between entropy and conditional entropy.

Another privacy criterion is generative adversarial privacy, which is a privacy definition inspired by generative adversarial networks. In a constrained minimax game, an adversarial model is trained alongside a privacy-preserving generative model. The goal of the former is to predict private attributes from public ones while the latter minimizes the adversary's prediction performance. Thereby, the data holder implicitly learns a privatization scheme from the data. This scheme is data-dependent but does not require detailed knowledge of the data distributions, thus trying to combine the best of both worlds.

<FIG> illustrates an exemplary data flow 400B for training and use of one or more private and public machine learning models to generate a radiotherapy treatment plan, according to some examples of the disclosure. Data flow 400B is identical in operation to data flow 400A except that the aggregator ML model <NUM> is applied directly to the outputs of the trained teacher ML models <NUM> rather than the reduced dimension outputs. In data flow 400B, after the outputs of the trained teacher ML models <NUM> are aggregated in a similar manner as in data flow 400A, the aggregated information is provided to the dimension adjustment ML model 430A to reduce the dimension of the aggregated information. The reduced dimension aggregated information is provided by the dimension adjustment model 430A to the privacy criteria noise addition module <NUM>. And, as in data flow 400A, the output of the privacy criteria noise addition module <NUM> is provided to the student model <NUM> after passing through the dimension adjustment ML model 430B to restore or increase the dimension of the information.

<FIG> is a flowchart illustrating example operations of the treatment processing logic <NUM> in performing process <NUM>, according to example embodiments. The process <NUM> may be embodied in computer-readable instructions for execution by one or more processors such that the operations of the process <NUM> may be performed in part or in whole by the functional components of the treatment processing logic <NUM>; accordingly, the process <NUM> is described below by way of example with reference thereto. However, in other embodiments, at least some of the operations of the process <NUM> may be deployed on various other hardware configurations. The process <NUM> is therefore not intended to be limited to the treatment processing logic <NUM> and can be implemented in whole, or in part, by any other component. Some or all of the operations of process <NUM> can be in parallel, out of order, or entirely omitted.

At operation <NUM>, treatment processing logic <NUM> receives training data. For example, treatment processing logic <NUM> receives training data <NUM>, which may include paired training data sets <NUM> (e.g., input-output training pairs).

At operation <NUM>, treatment processing logic <NUM> receives constraints for training the model. For example, treatment processing logic <NUM> receives constraints <NUM>.

At operation <NUM>, treatment processing logic <NUM> performs training of the model.

At operation <NUM>, treatment processing logic <NUM> outputs the trained model. For example, treatment processing logic <NUM> outputs the trained model <NUM> to operate on a new set of input data <NUM>.

At operation <NUM>, treatment processing logic <NUM> utilizes the trained model <NUM> to generate a radiotherapy plan. For example, treatment processing logic <NUM> utilizes the trained model <NUM> to operate on a new set of input data <NUM>.

<FIG> is a flowchart illustrating example operations of the treatment processing logic <NUM> in performing a process <NUM>, according to example embodiments. The process <NUM> may be embodied in computer-readable instructions for execution by one or more processors such that the operations of the process <NUM> may be performed in part or in whole by the functional components of the treatment processing logic <NUM>; accordingly, the process <NUM> is described below by way of example with reference thereto. However, in other embodiments, at least some of the operations of the process <NUM> may be deployed on various other hardware configurations. The process <NUM> is therefore not intended to be limited to the treatment processing logic <NUM> and can be implemented in whole, or in part, by any other component. Some or all of the operations of process <NUM> can be in parallel, out of order, or entirely omitted.

At operation <NUM>, treatment processing logic <NUM> receives a medical image of a patient.

At operation <NUM>, treatment processing logic <NUM> processes the medical image with a student machine learning model to estimate one or more radiotherapy plan parameters, wherein the student machine learning model is trained to establish a relationship between a plurality of public training medical images and corresponding radiotherapy plan parameters of the public training medical images.

At operation <NUM>, treatment processing logic <NUM> processes the plurality of public training medical images with a plurality of teacher machine learning models to generate sets of radiotherapy plan parameter estimates.

At operation <NUM>, treatment processing logic <NUM> reduces respective dimensions of the sets of radiotherapy plan parameter estimates, wherein the radiotherapy plan parameters of the plurality of public training medical images are perturbed in accordance with privacy criteria.

At operation <NUM>, treatment processing logic <NUM> generates a radiotherapy treatment plan for the patient based on the estimated one or more radiotherapy plan parameters of the medical image of the patient.

At operation <NUM>, treatment processing logic <NUM> receives a public training medical image of a patient.

At operation <NUM>, treatment processing logic <NUM> trains a student machine learning model to estimate one or more radiotherapy plan parameters of the public medical image by establishing a relationship between a plurality of public training medical images and corresponding radiotherapy plan parameters of the plurality of public training medical images.

At operation <NUM>, treatment processing logic <NUM> processes the public training medical images with a plurality of teacher machine learning models to generate sets of radiotherapy plan parameter estimates.

As previously discussed, respective electronic computing systems or devices may implement one or more of the methods or functional operations as discussed herein. In one or more embodiments, the radiotherapy processing computing system <NUM> may be configured, adapted, or used to control or operate the image-guided radiation therapy device <NUM>, perform or implement the training or prediction operations from <FIG>, operate the trained model <NUM>, perform or implement the operations of the flowcharts for processes <NUM>-<NUM>, or perform any one or more of the other methodologies discussed herein (e.g., as part of treatment processing logic <NUM>). In various embodiments, such electronic computing systems or devices operate as standalone devices or may be connected (e.g., networked) to other machines. For instance, such computing systems or devices may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Features of computing systems or devices may be embodied by a personal computer (PC), a tablet PC, a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine.

As also indicated above, the functionality discussed above may be implemented by instructions, logic, or other information storage on a machine-readable medium. While the machine-readable medium may have been described in various examples with reference to be a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more transitory or non-transitory instructions or data structures. The term "machine-readable medium" shall also be taken to include any tangible medium that is capable of storing, encoding or carrying transitory or non-transitory instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure, or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions.

Claim 1:
A method for performing privacy based radiotherapy treatment planning, the method comprising:
receiving, by processor circuitry, a medical image of a patient;
processing, by the processor circuitry, the medical image with a student machine learning model to estimate one or more radiotherapy plan parameters, wherein the student machine learning model is trained to establish a relationship between a plurality of public training medical images and corresponding radiotherapy plan parameters of the public training medical images, wherein the radiotherapy plan parameters of the plurality of public training medical images are generated by:
aggregating a plurality of radiotherapy plan parameter estimates which have been produced by:
processing the plurality of public training medical images with a plurality of teacher machine learning models to generate sets of radiotherapy plan parameter estimates; and
reducing respective dimensions of the sets of radiotherapy plan parameter estimates or the plurality of public training medical images; and
perturbing the aggregated plurality of radiotherapy plan parameter estimates in accordance with a privacy criteria;
increasing a dimension of the perturbed aggregated reduced dimension radiotherapy plan parameter estimates to output the plurality of radiotherapy plan parameters; and
generating, by the processor circuitry, a radiotherapy treatment plan for the patient based on the estimated one or more radiotherapy plan parameters of the medical image of the patient.