Patent ID: 12243637

The features and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements.

DETAILED DESCRIPTION

It is contemplated that systems, devices, methods, and processes of the claimed invention encompass variations and adaptations developed using information from the embodiments described herein. Adaptation and/or modification of the systems, devices, methods, and processes described herein may be performed by those of ordinary skill in the relevant art.

Throughout the description, where articles, devices, and systems are described as having, including, or comprising specific components, or where processes and methods are described as having, including, or comprising specific steps, it is contemplated that, additionally, there are articles, devices, and systems of the present invention that consist essentially of, or consist of, the recited components, and that there are processes and methods according to the present invention that consist essentially of, or consist of, the recited processing steps.

It should be understood that the order of steps or order for performing certain action is immaterial so long as the invention remains operable. Moreover, two or more steps or actions may be conducted simultaneously.

The mention herein of any publication, for example, in the Background section, is not an admission that the publication serves as prior art with respect to any of the claims presented herein. The Background section is presented for purposes of clarity and is not meant as a description of prior art with respect to any claim.

Headers are provided for the convenience of the reader—the presence and/or placement of a header is not intended to limit the scope of the subject matter described herein.

A. Nuclear Medicine Images

Nuclear medicine images are obtained using a nuclear imaging modality such as bone scan imaging, Positron Emission Tomography (PET) imaging, and Single-Photon Emission Tomography (SPECT) imaging.

As used herein, an “image”—for example, a 3-D image of mammal—includes any visual representation, such as a photo, a video frame, streaming video, as well as any electronic, digital or mathematical analogue of a photo, video frame, or streaming video. Any apparatus described herein, in certain embodiments, includes a display for displaying an image or any other result produced by the processor. Any method described herein, in certain embodiments, includes a step of displaying an image or any other result produced via the method.

As used herein, “3-D” or “three-dimensional” with reference to an “image” means conveying information about three dimensions. A 3-D image may be rendered as a dataset in three dimensions and/or may be displayed as a set of two-dimensional representations, or as a three-dimensional representation.

In certain embodiments, nuclear medicine images use imaging agents comprising radiopharmaceuticals. Nuclear medicine images are obtained following administration of a radiopharmaceutical to a patient (e.g., a human subject), and provide information regarding the distribution of the radiopharmaceutical within the patient. Radiopharmaceuticals are compounds that comprise a radionuclide.

As used herein, “administering” an agent means introducing a substance (e.g., an imaging agent) into a subject. In general, any route of administration may be utilized including, for example, parenteral (e.g., intravenous), oral, topical, subcutaneous, peritoneal, intraarterial, inhalation, vaginal, rectal, nasal, introduction into the cerebrospinal fluid, or instillation into body compartments

As used herein, “radionuclide” refers to a moiety comprising a radioactive isotope of at least one element. Exemplary suitable radionuclides include but are not limited to those described herein. In some embodiments, a radionuclide is one used in positron emission tomography (PET). In some embodiments, a radionuclide is one used in single-photon emission computed tomography (SPECT). In some embodiments, a non-limiting list of radionuclides includes99mTc,111In,64Cu,67Ga,68Ga,186Re,188Re,153Sm,177Lu,67Cu,123I,124I,125I,126I,131I,11C,13N,15O,18F,153Sm,166Ho,177Lu,149Pm,90Y,213Bi,103Pd,109Pd,159Gd,140La,198Au,199Au,169Yb,175Yb,165Dy,166Dy,105Rh,111Ag,89Zr,225Ac,82Rb,75Br,76Br,77Br,80Br,80mBr,82Br,83Br,211At and192Ir.

As used herein, the term “radiopharmaceutical” refers to a compound comprising a radionuclide. In certain embodiments, radiopharmaceuticals are used for diagnostic and/or therapeutic purposes. In certain embodiments, radiopharmaceuticals include small molecules that are labeled with one or more radionuclide(s), antibodies that are labeled with one or more radionuclide(s), and antigen-binding portions of antibodies that are labeled with one or more radionuclide(s).

Nuclear medicine images (e.g., PET scans; e.g., SPECT scans; e.g., whole-body bone scans; e.g. composite PET-CT images; e.g., composite SPECT-CT images) detect radiation emitted from the radionuclides of radiopharmaceuticals to form an image. The distribution of a particular radiopharmaceutical within a patient may be determined by biological mechanisms such as blood flow or perfusion, as well as by specific enzymatic or receptor binding interactions. Different radiopharmaceuticals may be designed to take advantage of different biological mechanisms and/or particular specific enzymatic or receptor binding interactions and thus, when administered to a patient, selectively concentrate within particular types of tissue and/or regions within the patient. Greater amounts of radiation are emitted from regions within the patient that have higher concentrations of radiopharmaceutical than other regions, such that these regions appear brighter in nuclear medicine images. Accordingly, intensity variations within a nuclear medicine image can be used to map the distribution of radiopharmaceutical within the patient. This mapped distribution of radiopharmaceutical within the patient can be used to, for example, infer the presence of cancerous tissue within various regions of the patient's body.

For example, upon administration to a patient, technetium 99m methylenediphosphonate (99mTc MDP) selectively accumulates within the skeletal region of the patient, in particular at sites with abnormal osteogenesis associated with malignant bone lesions. The selective concentration of radiopharmaceutical at these sites produces identifiable hotspots—localized regions of high intensity in nuclear medicine images. Accordingly, presence of malignant bone lesions associated with metastatic prostate cancer can be inferred by identifying such hotspots within a whole-body scan of the patient. As described in the following, risk indices that correlate with patient overall survival and other prognostic metrics indicative of disease state, progression, treatment efficacy, and the like, can be computed based on automated analysis of intensity variations in whole-body scans obtained following administration of99mTc MDP to a patient. In certain embodiments, other radiopharmaceuticals can also be used in a similar fashion to99mTc MDP.

In certain embodiments, the particular radiopharmaceutical used depends on the particular nuclear medicine imaging modality used. For example 18F sodium fluoride (NaF) also accumulates in bone lesions, similar to99mTc MDP, but can be used with PET imaging. In certain embodiments, PET imaging may also utilize a radioactive form of the vitamin choline, which is readily absorbed by prostate cancer cells.

In certain embodiments, radiopharmaceuticals that selectively bind to particular proteins or receptors of interest—particularly those whose expression is increased in cancerous tissue may be used. Such proteins or receptors of interest include, but are not limited to tumor antigens, such as CEA, which is expressed in colorectal carcinomas, Her2/neu, which is expressed in multiple cancers, BRCA 1 and BRCA 2, expressed in breast and ovarian cancers; and TRP-1 and -2, expressed in melanoma.

For example, human prostate-specific membrane antigen (PSMA) is upregulated in prostate cancer, including metastatic disease. PSMA is expressed by virtually all prostate cancers and its expression is further increased in poorly differentiated, metastatic and hormone refractory carcinomas. Accordingly, radiopharmaceuticals corresponding to PSMA binding agents (e.g., compounds that a high affinity to PSMA) labelled with one or more radionuclide(s) can be used to obtain nuclear medicine images of a patient from which the presence and/or state of prostate cancer within a variety of regions (e.g., including, but not limited to skeletal regions) of the patient can be assessed. In certain embodiments, nuclear medicine images obtained using PSMA binding agents are used to identify the presence of cancerous tissue within the prostate, when the disease is in a localized state. In certain embodiments, nuclear medicine images obtained using radiopharmaceuticals comprising PSMA binding agents are used to identify the presence of cancerous tissue within a variety of regions that include not only the prostate, but also other organs and tissue regions such as lungs, lymph nodes, and bones, as is relevant when the disease is metastatic.

In particular, upon administration to a patient, radionuclide labelled PSMA binding agents selectively accumulate within cancerous tissue, based on their affinity to PSMA. In a similar manner to that described above with regard to99mTc MDP, the selective concentration of radionuclide labelled PSMA binding agents at particular sites within the patient produces detectable hotspots in nuclear medicine images. As PSMA binding agents concentrate within a variety of cancerous tissues and regions of the body expressing PSMA, localized cancer within a prostate of the patient and/or metastatic cancer in various regions of the patient's body can be detected, and evaluated. Risk indices that correlate with patient overall survival and other prognostic metrics indicative of disease state, progression, treatment efficacy, and the like, can be computed based on automated analysis of intensity variations in nuclear medicine images obtained following administration of a PSMA binding agent radiopharmaceutical to a patient.

A variety of radionuclide labelled PSMA binding agents may be used as radiopharmaceutical imaging agents for nuclear medicine imaging to detect and evaluate prostate cancer. In certain embodiments, the particular radionuclide labelled PSMA binding agent that is used depends on factors such as the particular imaging modality (e.g., PET; e.g., SPECT) and the particular regions (e.g., organs) of the patient to be imaged. For example, certain radionuclide labelled PSMA binding agents are suited for PET imaging, while others are suited for SPECT imaging. For example, certain radionuclide labelled PSMA binding agents facilitate imaging a prostate of the patient, and are used primarily when the disease is localized, while others facilitate imaging organs and regions throughout the patient's body, and are useful for evaluating metastatic prostate cancer.

A variety of PSMA binding agents and radionuclide labelled versions thereof are described in U.S. Pat. Nos. 8,778,305, 8,211,401, and 8,962,799, each of which are incorporated herein by reference in their entireties. Several PSMA binding agents and radionuclide labelled versions thereof are also described in PCT Application PCT/US2017/058418, filed Oct. 26, 2017 (PCT publication WO 2018/081354), the content of which is incorporated herein by reference in its entirety. Section C, below, describes several example PSMA binding agents and radionuclide labelled versions thereof, as well.

B. Automated Lesion Detection and Analysis

i. Automated Lesion Detection

In certain embodiments, the systems and methods described herein utilize machine learning techniques for automated image segmentation and detection of hotspots corresponding to and indicative of possible cancerous lesions within a subject.

In certain embodiments, the systems and methods described herein may be implemented in a cloud-based platform, for example as described in PCT/US2017/058418, filed Oct. 26, 2017 (PCT publication WO 2018/081354), the content of which is hereby incorporated by reference in its entirety.

In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like. In certain embodiments, machine learning modules implementing machine learning techniques are trained, for example using manually segmented and/or labeled images, to identify and/or classify portions of images. Such training may be used to determine various parameters of machine learning algorithms implemented by a machine learning module, such as weights associated with layers in neural networks. In certain embodiments, once a machine learning module is trained, e.g., to accomplish a specific task such as identifying certain target regions within images, values of determined parameters are fixed and the (e.g., unchanging, static) machine learning module is used to process new data (e.g., different from the training data) and accomplish its trained task without further updates to its parameters (e.g., the machine learning module does not receive feedback and/or update). In certain embodiments, machine learning modules may receive feedback, e.g., based on user review of accuracy, and such feedback may be used as additional training data, to dynamically update the machine learning module. In some embodiments, the trained machine learning module is a classification algorithm with adjustable and/or fixed (e.g., locked) parameters, e.g., a random forest classifier.

In certain embodiments, machine learning techniques are used to automatically segment anatomical structures in anatomical images, such as CT, MRI, ultra-sound, etc. images, in order to identify volumes of interest corresponding to specific target tissue regions such as specific organs (e.g., a prostate, lymph node regions, a kidney, a liver, a bladder, an aorta portion) as well as bones. In this manner, machine learning modules may be used to generate segmentation masks and/or segmentation maps (e.g., comprising a plurality of segmentation masks, each corresponding to and identifying a particular target tissue region) that can be mapped to (e.g., projected onto) functional images, such as PET or SPECT images, to provide anatomical context for evaluating intensity fluctuations therein. Approaches for segmenting images and using the obtained anatomical context for analysis of nuclear medicine images are described, for example, in further detail in PCT/US2019/012486, filed Jan. 7, 2019 (PCT publication WO 2019/136349) and PCT/EP2020/050132, filed Jan. 6, 2020 (PCT publication WO 2020/144134), the contents of each of which is hereby incorporated by reference in their entirety.

In certain embodiments, potential lesions are detected as regions of locally high intensity in functional images, such as PET images. These localized regions of elevated intensity, also referred to as hotspots, can be detected using image processing techniques not necessarily involving machine learning, such as filtering and thresholding, and segmented using approaches such as the fast marching method. Anatomical information established from the segmentation of anatomical images allows for anatomical labeling of detected hotspots representing potential lesions. Anatomical context may also be useful in allowing different detection and segmentation techniques to be used for hotspot detection in different anatomical regions, which can increase sensitivity and performance.

In certain embodiments, automatically detected hotspots may be presented to a user via an interactive graphical user interface (GUI). In certain embodiments, to account for target lesions detected by the user (e.g., physician), but that are missed or poorly segmented by the system, a manual segmentation tool is included in the GUI, allowing the user to manually “paint” regions of images that they perceive as corresponding to lesions of any shape and size. These manually segmented lesions may then be included, along with selected automatically detected target lesions, in subsequently generated reports.

ii. AI-Based Lesion Detection

In certain embodiments, the systems and methods described herein utilize one or more machine learning modules to analyze intensities of 3D functional images and detect hotspot representing potential lesions. For example, by collecting a dataset of PET/CT images in which hotspots that represent lesions have been manually detected and segmented, training material for AI-based lesion detection algorithms can be obtained. These manually labeled images can be used to train one or more machine learning algorithms to automatically analyze functional images (e.g., PET images) to accurately detect and segment hotspots corresponding to cancerous lesions.

FIG.1Ashows an example process100afor automated lesion detection and/or segmentation using machine learning modules that implement machine learning algorithms, such as ANNs, CNNs, and the like. As shown inFIG.1A, a 3D functional image102, such as a PET or SPECT image, is received106and used as input to a machine learning module110.FIG.1Ashows an example PET image, obtained using PyL™ as a radiopharmaceutical102a. The PET image102ais shown overlaid on a CT image (e.g., as a PET/CT image), but the machine learning module110may receive the PET (e.g., or other functional image) itself (e.g., not including the CT, or other anatomical image) as input. In certain embodiments, as described below, an anatomical image may also be received as input. The machine learning module automatically detects and/or segments hotspots120determined (by the machine learning module) to represent potential cancerous lesions. An example image showing hotspots appearing in a PET image120bis shown inFIG.1Aas well. Accordingly, the machine learning module generates, as output, one or both of (i) a hotspot list130and (ii) a hotspot map132. In certain embodiments, the hotspot list identifies locations (e.g., centers of mass) of the detected hotspots. In certain embodiments, the hotspot map is identifies 3D volumes and/or delineates 3D boundaries of detected hotspots, as determined via image segmentation performed by the machine learning module110. The hotspot list and/or hotspot map may be stored and/or provided (e.g., to other software modules) for display and/or further processing140.

In certain embodiments, machine learning-based lesion detection algorithms may be trained on, and utilize, not only functional image information (e.g., from a PET image), but also anatomical information. For example, in certain embodiments, one or more machine learning modules used for lesion detection and segmentation may be trained on, and receive as input, two channels—a first channel corresponding to a portion of a PET image, and a second channel corresponding to a portion of a CT image. In certain embodiments, information derived from an anatomical (e.g., CT) image may also be used as input to machine learning modules for lesion detection and/or segmentation. For example, in certain embodiments, 3D segmentation maps identifying various tissue regions within an anatomical and/or functional image can also be used to provide anatomical context.

FIG.1Bshows an example process100bin which both a 3D anatomical image104, such as a CT or MR image, and a 3D functional image102are received108and used as input to a machine learning module112that performs hotspot detection and/or segmentation122based on information (e.g., voxel intensities) from both the 3D anatomical image104and the 3D functional image102as described herein. A hotspot list130and/or hotspot map132may be generated as output from the machine learning module, and stored/provided for further processing (e.g., graphical rendering for display, subsequent operations by other software modules, etc.)140.

In certain embodiments, automated lesion detection and analysis (e.g., for inclusion in a report) includes three tasks: (i) detection of hotspots corresponding to lesions, (ii) segmentation of detected hotspots (e.g., to identify, within a functional image, a 3D volume corresponding to each lesion), and (iii) classification of detected hotspots as having high or low probability of corresponding to a true lesion within the subject (e.g., and thus appropriate for inclusion in a radiologist report or not). In certain embodiments, one or more machine learning modules may be used to accomplish these three tasks, e.g., one by one (e.g., in sequence) or in combination. For example, in certain embodiments, a first machine learning module is trained to detect hotspots and identify hotspot locations, a second machine learning module is trained to segment hotspots, and a third machine learning module is trained to classify detected hotspots, for example using information obtained from the other two machine learning modules.

For example, as shown in the example process100cofFIG.1C, a 3D functional image102may be received106and used as input to a first machine learning module114that performs automated hotspot detection. The first machine learning module114automatically detects one or more hotspots124in the 3D functional image and generates a hotspot list130as output. A second machine learning module116may receive the hotspot list130as input along with the 3D functional image, and perform automated hotspot segmentation,126to generate a hotspot map132. As previously described, the hotspot map132, as well as the hotspot list130, may be stored and/or provided for further processing140.

In certain embodiments, a single machine learning module is trained to directly segment hotspots within images (e.g., 3D functional images; e.g., to generate a 3D hotspot map identifying volumes corresponding to detected hotspots), thereby combining the first two steps of detection and segmentation of hotspots. A second machine learning module may then be used to classify detected hotspots, for example based on the segmented hotspots determined previously. In certain embodiments, a single machine learning module may be trained to accomplish all three tasks—detection, segmentation, and classification—in a single step.

iii. Lesion Index Values

In certain embodiments, lesion index values are calculated for detected hotspots to provide a measure of, for example, relative uptake within and/or size of the corresponding physical lesion. In certain embodiments, lesion index values are computed for a particular hotspot based (i) on a measure of intensity for the hotspot and (ii) reference values corresponding to measures of intensity within one or more reference volumes, each corresponding to a particular reference tissue region. For example, in certain embodiments, reference values include an aorta reference value that measures intensity within an aorta volume corresponding to a portion of an aorta and a liver reference value that measures intensity within a liver volume corresponding to a liver of the subject. In certain embodiments, intensities of voxels of a nuclear medicine image, for example a PET image, represent standard uptake values (SUVs) (e.g., having been calibrated for injected radiopharmaceutical dose and/or patient weight), and measures of hotspot intensity and/or measures reference values are SUV values. Use of such reference values in computing lesion index values is described in further detail, for example, in PCT/EP2020/050132, filed Jan. 6, 2020, the contents of which is hereby incorporated by reference in its entirety.

In certain embodiments, a segmentation mask is used to identify a particular reference volume in, for example a PET image. For a particular reference volume, a segmentation mask identifying the reference volume may be obtained via segmentation of an anatomical, e.g., CT, image. To identify voxels of the reference volume to be used for computation of the corresponding reference value, the mask may be eroded a fixed distance (e.g., at least one voxel), to create a reference organ mask that identifies a reference volume corresponding to a physical region entirely within the reference tissue region. For example, erosion distances of 3 mm and 9 mm have been used for aorta and liver reference volumes, respectively. Additional mask refinement may also be performed (e.g., to select a specific, desired, set of voxels for use in computing the reference value), for example as described below with respect to the liver reference volume.

Various measures of intensity within reference volumes may be used. For example, in certain embodiments, a robust average of voxels inside the reference volume (e.g., as defined by the reference volume segmentation mask, following erosion) may be determined as a mean of values in an interquartile range of voxel intensities (IQRmean). Other measures, such as a peak, a maximum, a median, etc. may also be determined. In certain embodiments, an aorta reference value is determined as a robust average of SUV from voxels inside the aorta mask. The robust average is computed as the mean of the values in the interquartile range, IQRmean.

In certain embodiments, a subset of voxels within a reference volume is selected in order to avoid impact from reference tissue regions that may have abnormally low radiopharmaceutical uptake. Although the automated segmentation techniques described and referenced herein can provide an accurate outline (e.g., identification) of regions of images corresponding to specific tissue regions, there are often areas of abnormally low uptake in the liver which should be excluded from the reference value calculation. For example, liver reference value (e.g., a liver SUV value) is computed so as to avoid impact from regions in the liver with very low tracer (radiopharmaceutical) activity, that might appear e.g., due to tumors without tracer uptake. In certain embodiments, to account for effects of abnormally low uptake in reference tissue regions the reference value calculation for the liver analyzes a histogram of intensities of voxels corresponding to the liver (e.g., voxels within an identified liver reference volume) and removes (e.g., excludes) intensities if they form a second histogram peak of lower intensities, thereby only including intensities associated with a higher intensity value peak.

For example, for the liver, the reference SUV may be computed as a mean SUV of a major mode in a two-component Gaussian Mixture Model fitted to a histogram of SUV's of voxels within the liver reference volume (e.g., as identified by a liver segmentation mask, e.g., following the above-described erosion procedure). In certain embodiments, if the minor component has a larger mean SUV than the major component, and the minor component has at least 0.33 of the weight, an error is thrown and no reference value for the liver is determined. In certain embodiments, if the minor component has a larger mean than the major peak, the liver reference mask is kept as it is. Otherwise a separation SUV threshold is computed, defined by that the probability to belong to the major component for a SUV that is at the threshold or is larger is the same as the probability to belong to the minor component for a SUV that is at the separation threshold or is smaller. The reference liver mask is then refined by removing voxels with SUV smaller than the separation threshold. A liver reference value may then be determined as a measure of intensity (e.g., SUV) values of voxels identified by the liver reference mask, for example as described herein with respect to the aorta reference.FIG.2Aillustrates an example liver reference computation, showing a histogram of liver SUV values with Gaussian mixture components shown in red and the separation threshold marked in green.

FIG.2Bshows the resulting portion of the liver volume used to calculate the liver reference value, with voxels corresponding to the lower value peak excluded from the reference value calculation. As shown in the figure, lower intensity areas towards the bottom of the liver have been excluded, as well as regions close to the liver edge.

FIG.2Cshows an example process200where a multi-component mixture model is used to avoid impact from regions with low tracer uptake, as described herein with respect to liver reference volume computation. The process shown inFIG.2Cand described herein with regard to the liver may also be applied, similarly, to computation of intensity measures of other organs and tissue regions of interest as well, such as an aorta (e.g., aorta portion, such as the thoracic aorta portion or abdominal aorta portion), a parotid gland, a gluteal muscle. As shown, inFIG.2Cand described herein, in a first step, a 3D functional image202is received, and a reference volume corresponding to a specific reference tissue region (e.g., liver, aorta, parotid gland) is identified therein208. A multi-component mixture model210is then fit to a distribution intensities (e.g., a histogram of intensities) of (e.g., within) the reference volume, and a major mode of the mixture model is identified212. A measure of intensities associated with the major mode (e.g., and excluding contributions from intensities associated with other, minor, modes) is determined214and used as the reference intensity value for the identified reference volume. In certain embodiments, hotspots are detected216and the reference intensity value determined in this manner can be used to determine lesion index values for the detected hotspots218, for example via approaches such as those described in PCT/US2019/012486, filed Jan. 7, 2019 and PCT/EP2020/050132, filed Jan. 6, 2020, the content of each of which is hereby incorporated by reference in its entirety.

iv. Suppression of Intensity Bleed Associated with Normal Uptake in High-Uptake Organs

In certain embodiments, intensities of voxels of a functional image are adjusted in order to suppress/correct for intensity bleed associated with certain organs in which high-uptake occurs under normal circumstances. This approach may be used, for example, for organs such as a kidney, a liver, and a urinary bladder. In certain embodiments, correcting for intensity bleed associated with multiple organs is performed one organ at a time, in a step-wise fashion. For example, in certain embodiments, first kidney uptake is suppressed, then liver uptake, then urinary bladder uptake. Accordingly, the input to liver suppression is an image where kidney uptake has been corrected for (e.g., and input to bladder suppression is an image wherein kidney and liver uptake have been corrected for).

FIG.3shows an example process300for correcting intensity blead from a high-uptake tissue region. As shown inFIG.3, a 3D functional image is received304and a high intensity volume corresponding to the high-uptake tissue region is identified306. In another step, a suppression volume outside the high-intensity volume is identified308. In certain embodiments, as described herein, the suppression volume may be determined as a volume enclosing regions outside of, but within a pre-determined distance from, the high-intensity volume. In another step, a background image is determined310, for example by assigning voxels within the high-intensity volume intensities determined based on intensities outside the high-intensity volume (e.g., within the suppression volume), e.g., via interpolation (e.g., using convolution). In another step, an estimation image is determined312by subtracting the background image from the 3D functional image (e.g., via a voxel-by-voxel intensity subtraction). In another step, a suppression map is determined314. As described herein, in certain embodiments, the suppression map is determined using the estimation image, by extrapolating intensity values of voxels within the high-intensity volume to locations outside the high intensity volume. In certain embodiments, intensities are only extrapolated to locations within the suppression volume, and intensities of voxels outside the suppression volume are set to 0. The suppression map is then used to adjust intensities of the 3D functional image316, for example by subtracting the suppression map from the 3D functional image (e.g., performing a voxel-by-voxel intensity subtraction).

An example approach for suppression/correction of intensity bleed from a particular organ (in certain embodiments, kidneys are treated together) for a PET/CT composite image is as follows:1. The projected CT organ mask segmentation is adjusted to high-intensity regions of the PET image, in order to handle PET/CT misalignment. If the PET-adjusted organ mask is less than 10 pixels, no suppression is made for this organ.2. A “background image” is computed, replacing all high uptake with interpolated background uptake within the decay distance from the PET-adjusted organ mask. This is done using convolution with Gaussian kernels.3. Intensities that should be accounted for when estimating suppression are computed as the difference between the input PET and the background image. This “estimation image” has high intensities inside the given organ and zero intensity at locations farther than the decay distance from the given organ.4. A suppression map is estimated from the estimation image using an exponential model. The suppression map is only non-zero in the region within the decay distance of the PET-adjusted organ segmentation.5. The suppression map is subtracted from the original PET image.

As described above, these five steps may be repeated, for each of a set of multiple organs, in a sequential fashion.

v. Anatomical Labeling of Detected Lesions

In certain embodiments, detected hotspots are (e.g., automatically) assigned anatomical labels that identify particular anatomical regions and/or groups of regions in which the lesions that they represent are determined to be located. For example, as shown in the example process400ofFIG.4, a 3D functional image may be received404an used to automatically detect hotspots406, for example via any of the approaches described herein. Once hotspots are detected, anatomical classifications for each hotspot can be automatically determined408and each hotspot labeled with the determined anatomical classification. Automated anatomical labeling may, for example, be performed using automatically determined locations of detected hotspots along with anatomical information provided by, for example, a 3D segmentation map identifying image regions corresponding to particular tissue regions and/or an anatomical image. The hotspots and anatomical labeling of each may be stored and/or provided for further processing410.

For example, detected hotspots may be automatically classified into one of five classes as follows:T (prostate tumor)N (pelvic lymph node)Ma (non-pelvic lymph)Mb (bone metastasis)Mc (soft tissue metastasis not situated in prostate or lymphe node)

Table 1, below, lists tissue regions associated with each of the five classes. Hotspots corresponding to locations within any of the tissue regions associated with a particular class may, accordingly, be automatically assigned to that class.

TABLE 1List of Tissue Regions Corresponding to Five Classesin a Lesion Anatomical Labeling ApproachPelvic lymphBoneLymph nodesnodesProstateSoft tissueMbMaNTMcSkullCervicalTemplate rightProstateBrainThoraxSupraclavicularTemplate leftNeckVertebraeAxillaryPresacralLunglumbarMediastinalOther, pelvicEsophagealVertebraeHilarLiverthoracicMesentericGallbladderPelvisElbowSpleenExtremitiesPoplitealPancreasPeri-/para-aorticAdrenalOther, non-KidneypelvicBladderSkinMuscleOther
vi. Graphical User Interface and Quality Control and Reporting

In certain embodiments, detected hotspots and associated information, such as computed lesion index values and anatomical labeling are displayed with an interactive graphical user interface (GUI) so as to allow for review by a medical professional, such as a physician, radiologist, technician, etc. Medical professionals may thus use the GUI to review and confirm accuracy of detected hotspots, as well as corresponding index values and/or anatomical labeling. In certain embodiments, the GUI may also allow users to identify, and segment (e.g., manually) additional hotspots within medical images, thereby allowing a medical professional to identify additional potential lesions that he/she believes the automated detection process may have missed. Once identified, lesion index values and/or anatomical labeling may also be determined for these manually identified and segmented lesions. For example, as indicated inFIG.3, the user may review locations determined for each hotspot, as well as anatomical labeling, such as a (e.g., automatically determined) miTNM classification. The miTNM classification scheme is described in further detail, for example, in Eiber et al., “Prostate Cancer Molecular Imaging Standardized Evaluation (PROMISE): Proposed miTNM Classification for the Interpretation of PSMA-Ligand PET/CT,”J. Nucl. Med., vol.59, pg. 469-78 (2018), the content of which is hereby incorporated by reference in its entirety. Once a user is satisfied with the set of detected hotspots and information computed therefrom, they may confirm their approval and generate a final, signed report that can be reviewed and used to discuss outcomes and diagnosis with a patient, and assess prognosis and treatment options.

For example, as shown inFIG.5A, in an example process500for interactive hotspot review and detection, a 3D functional image is received504and hotspots are automatically detected506, for example using any of the automated detection approaches described herein. The set of automated hotspots is represented and rendered graphically within an interactive GUI508for user review. The user may select at least a portion (e.g., up to all) of the automatically determined hotspots for inclusion in a final hotspot set510, which may then be used for further calculations512, e.g., to determine risk index values for the patient.

FIG.5Bshows an example workflow520for user review of detected lesions and lesion index values for quality control and reporting. The example workflow allows for user review of segmented lesions as well as liver and aorta segmentation used for calculation of lesion index values as described herein. For example, in a first step, a user reviews images (e.g., a CT image) for quality522and accuracy of automated segmentation used to obtain liver and blood pool (e.g., aorta) reference values524. As shown inFIGS.6A, and6Bthe GUI allows a user evaluates images and overlaid segmentation to ensure that the automated segmentation of the liver (purple color inFIG.6A) is within healthy liver tissue and that automated segmentation of blood pool (aorta portion, shown as salmon color inFIG.6Bis within the aorta and left ventricle.

In another step526, a user validates automatically detected hotspots and/or identifies additional hotspots, e.g., to create a final set of hotspots corresponding to lesions, for inclusion in a generated report. As shown inFIG.6C, a user may select an automatically identified hotspot by hovering over a graphical representation of the hotspot displayed within the GUI (e.g., as an overlay and/or marked region on a PET and/or CT image). To facilitate hotspot selection, the particular hotspot selected may be indicated to the user, via a color change (e.g., turning green). The user may then click on the hotspot to select it, which may be visually confirmed to the user via another color change. For example, as shown inFIG.4C, upon selection the hotspot turns pink. Upon user selection, quantitatively determined values, such as a lesion index and/or anatomical labeling may be displayed to the user, allowing them to verify the automatically determined values528.

In certain embodiments, the GUI allows a user to select hotspots from the set of (automatically) pre-identified hotspots to confirm they indeed represent lesions526aand also to identify additional hotspots562bcorresponding to lesions, not having been automatically detected.

As shown inFIG.6DandFIG.6E, the user may use GUI tools to draw on slices of images (e.g., PET images and/or CT images; e.g., a PET image overlaid on a CT image) to mark regions corresponding to a new, manually identified lesion. Quantitative information, such as a lesion index and/or anatomical labeling may be determined for the manually identified lesion automatically, or may be manually entered by the user.

In another step, e.g., once the user has selected and/or manually identified all lesions, the GUI displays a quality control checklist for the user to review530, as shown inFIG.6F. Once the user reviews and completes the checklist, they may click “Create Report” to sign and generate a final report532. An example of a generated report is shown inFIG.6G.

C. Imaging Agents

i. PET Imaging Radionuclide Labelled PSMA Binding Agents

In certain embodiments, the radionuclide labelled PSMA binding agent is a radionuclide labelled PSMA binding agent appropriate for PET imaging.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises [18F]DCFPyL (also referred to as PyL™; also referred to as DCFPyL-18F):

or a pharmaceutically acceptable salt thereof.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises [18F]DCFBC:

or a pharmaceutically acceptable salt thereof.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises68Ga-PSMA-HBED-CC (also referred to as68Ga-PSMA-11):

or a pharmaceutically acceptable salt thereof.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises PSMA-617:

or a pharmaceutically acceptable salt thereof. In certain embodiments, the radionuclide labelled PSMA binding agent comprises68Ga-PSMA-617, which is PSMA-617 labelled with68Ga, or a pharmaceutically acceptable salt thereof. In certain embodiments, the radionuclide labelled PSMA binding agent comprises177Lu-PSMA-617, which is PSMA-617 labelled with177Lu, or a pharmaceutically acceptable salt thereof.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises PSMA-I&T:

or a pharmaceutically acceptable salt thereof. In certain embodiments, the radionuclide labelled PSMA binding agent comprises68Ga-PSMA-I&T, which is PSMA-I&T labelled with68Ga, or a pharmaceutically acceptable salt thereof.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises PSMA-1007:

or a pharmaceutically acceptable salt thereof. In certain embodiments, the radionuclide labelled PSMA binding agent comprises18F-PSMA-1007, which is PSMA-1007 labelled with18F, or a pharmaceutically acceptable salt thereof.
ii. SPECT Imaging Radionuclide Labelled PSMA Binding Agents

In certain embodiments, the radionuclide labelled PSMA binding agent is a radionuclide labelled PSMA binding agent appropriate for SPECT imaging.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises 1404 (also referred to as MIP-1404):

or a pharmaceutically acceptable salt thereof.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises 1405 (also referred to as MIP-1405):

or a pharmaceutically acceptable salt thereof.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises 1427 (also referred to as MIP-1427):

or a pharmaceutically acceptable salt thereof.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises 1428 (also referred to as MIP-1428):

or a pharmaceutically acceptable salt thereof.

In certain embodiments, the PSMA binding agent is labelled with a radionuclide by chelating it to a radioisotope of a metal [e.g., a radioisotope of technetium (Tc) (e.g., technetium-99m (99mTc)); e.g., a radioisotope of rhenium (Re) (e.g., rhenium-188 (188Re); e.g., rhenium-186 (186Re)); e.g., a radioisotope of yttrium (Y) (e.g.,90Y); e.g., a radioisotope of lutetium (Lu)(e.g.,177Lu); e.g., a radioisotope of gallium (Ga) (e.g.,68Ga; e.g.,67Ga); e.g., a radioisotope of indium (e.g.,111In); e.g., a radioisotope of copper (Cu) (e.g.,67Cu)].

In certain embodiments, 1404 is labelled with a radionuclide (e.g., chelated to a radioisotope of a metal). In certain embodiments, the radionuclide labelled PSMA binding agent comprises99mTc-MIP-1404, which is 1404 labelled with (e.g., chelated to)99mTc:

or a pharmaceutically acceptable salt thereof. In certain embodiments, 1404 may be chelated to other metal radioisotopes [e.g., a radioisotope of rhenium (Re) (e.g., rhenium-188 (188Re); e.g., rhenium-186 (186Re)); e.g., a radioisotope of yttrium (Y) (e.g.,90Y); e.g., a radioisotope of lutetium (Lu)(e.g.,177Lu); e.g., a radioisotope of gallium (Ga) (e.g.,68Ga; e.g.,67Ga); e.g., a radioisotope of indium (e.g.,11In); e.g., a radioisotope of copper (Cu) (e.g.,67Cu)] to form a compound having a structure similar to the structure shown above for99mTc-MIP-1404, with the other metal radioisotope substituted for99mTc.

In certain embodiments, 1405 is labelled with a radionuclide (e.g., chelated to a radioisotope of a metal). In certain embodiments, the radionuclide labelled PSMA binding agent comprises99mTc-MIP-1405, which is 1405 labelled with (e.g., chelated to)99mTc:

or a pharmaceutically acceptable salt thereof. In certain embodiments, 1405 may be chelated to other metal radioisotopes [e.g., a radioisotope of rhenium (Re) (e.g., rhenium-188 (188Re); e.g., rhenium-186 (186Re)); e.g., a radioisotope of yttrium (Y) (e.g.,90Y); e.g., a radioisotope of lutetium (Lu)(e.g.,177Lu); e.g., a radioisotope of gallium (Ga) (e.g.,68Ga; e.g.,67Ga); e.g., a radioisotope of indium (e.g.,111In); e.g., a radioisotope of copper (Cu) (e.g.,67Cu)] to form a compound having a structure similar to the structure shown above for99mTc-MIP-1405, with the other metal radioisotope substituted for99mTc.

In certain embodiments, 1427 is labelled with (e.g., chelated to) a radioisotope of a metal, to form a compound according to the formula below:

or a pharmaceutically acceptable salt thereof, wherein M is a metal radioisotope [e.g., a radioisotope of technetium (Tc) (e.g., technetium-99m (99mTc)); e.g., a radioisotope of rhenium (Re) (e.g., rhenium-188 (188Re); e.g., rhenium-186 (186Re)); e.g., a radioisotope of yttrium (Y) (e.g.,90Y); e.g., a radioisotope of lutetium (Lu)(e.g.,177Lu); e.g., a radioisotope of gallium (Ga) (e.g.,68Ga; e.g.,67Ga); e.g., a radioisotope of indium (e.g.,111In); e.g., a radioisotope of copper (Cu) (e.g.,67Cu)] with which 1427 is labelled.

In certain embodiments, 1428 is labelled with (e.g., chelated to) a radioisotope of a metal, to form a compound according to the formula below:

or a pharmaceutically acceptable salt thereof, wherein M is a metal radioisotope [e.g., a radioisotope of technetium (Tc) (e.g., technetium-99m (99mTc)); e.g., a radioisotope of rhenium (Re) (e.g., rhenium-188 (188Re); e.g., rhenium-186 (186Re)); e.g., a radioisotope of yttrium (Y) (e.g.,90Y); e.g., a radioisotope of lutetium (Lu)(e.g.,177Lu); e.g., a radioisotope of gallium (Ga) (e.g.,68Ga; e.g.,67Ga); e.g., a radioisotope of indium (e.g.,111In); e.g., a radioisotope of copper (Cu) (e.g.,67Cu)] with which 1428 is labelled.

In certain embodiments, the radionuclide labelled PSMA binding agent comprises PSMA I&S:

or a pharmaceutically acceptable salt thereof. In certain embodiments, the radionuclide labelled PSMA binding agent comprises99mTc-PSMA I&S, which is PSMA I&S labelled with99mTc, or a pharmaceutically acceptable salt thereof.
D. Computer System and Network Architecture

As shown inFIG.7, an implementation of a network environment700for use in providing systems, methods, and architectures described herein is shown and described. In brief overview, referring now toFIG.7, a block diagram of an exemplary cloud computing environment700is shown and described. The cloud computing environment700may include one or more resource providers702a,702b,702c(collectively,702). Each resource provider702may include computing resources. In some implementations, computing resources may include any hardware and/or software used to process data. For example, computing resources may include hardware and/or software capable of executing algorithms, computer programs, and/or computer applications. In some implementations, exemplary computing resources may include application servers and/or databases with storage and retrieval capabilities. Each resource provider702may be connected to any other resource provider702in the cloud computing environment700. In some implementations, the resource providers702may be connected over a computer network708. Each resource provider702may be connected to one or more computing device704a,704b,704c(collectively,704), over the computer network708.

The cloud computing environment700may include a resource manager706. The resource manager706may be connected to the resource providers702and the computing devices704over the computer network708. In some implementations, the resource manager706may facilitate the provision of computing resources by one or more resource providers702to one or more computing devices704. The resource manager706may receive a request for a computing resource from a particular computing device704. The resource manager706may identify one or more resource providers702capable of providing the computing resource requested by the computing device704. The resource manager706may select a resource provider702to provide the computing resource. The resource manager706may facilitate a connection between the resource provider702and a particular computing device704. In some implementations, the resource manager706may establish a connection between a particular resource provider702and a particular computing device704. In some implementations, the resource manager706may redirect a particular computing device704to a particular resource provider702with the requested computing resource.

FIG.8shows an example of a computing device800and a mobile computing device850that can be used to implement the techniques described in this disclosure. The computing device800is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device850is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

The computing device800includes a processor802, a memory804, a storage device806, a high-speed interface808connecting to the memory804and multiple high-speed expansion ports810, and a low-speed interface812connecting to a low-speed expansion port814and the storage device806. Each of the processor802, the memory804, the storage device806, the high-speed interface808, the high-speed expansion ports810, and the low-speed interface812, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor802can process instructions for execution within the computing device800, including instructions stored in the memory804or on the storage device806to display graphical information for a GUI on an external input/output device, such as a display816coupled to the high-speed interface808. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). Thus, as the term is used herein, where a plurality of functions are described as being performed by “a processor”, this encompasses embodiments wherein the plurality of functions are performed by any number of processors (one or more) of any number of computing devices (one or more). Furthermore, where a function is described as being performed by “a processor”, this encompasses embodiments wherein the function is performed by any number of processors (one or more) of any number of computing devices (one or more) (e.g., in a distributed computing system).

The memory804stores information within the computing device800. In some implementations, the memory804is a volatile memory unit or units. In some implementations, the memory804is a non-volatile memory unit or units. The memory804may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device806is capable of providing mass storage for the computing device800. In some implementations, the storage device806may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor802), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine-readable mediums (for example, the memory804, the storage device806, or memory on the processor802).

The high-speed interface808manages bandwidth-intensive operations for the computing device800, while the low-speed interface812manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface808is coupled to the memory804, the display816(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports810, which may accept various expansion cards (not shown). In the implementation, the low-speed interface812is coupled to the storage device806and the low-speed expansion port814. The low-speed expansion port814, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device800may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server820, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer822. It may also be implemented as part of a rack server system824. Alternatively, components from the computing device800may be combined with other components in a mobile device (not shown), such as a mobile computing device850. Each of such devices may contain one or more of the computing device800and the mobile computing device850, and an entire system may be made up of multiple computing devices communicating with each other.

The mobile computing device850includes a processor852, a memory864, an input/output device such as a display854, a communication interface866, and a transceiver868, among other components. The mobile computing device850may also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor852, the memory864, the display854, the communication interface866, and the transceiver868, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor852can execute instructions within the mobile computing device850, including instructions stored in the memory864. The processor852may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor852may provide, for example, for coordination of the other components of the mobile computing device850, such as control of user interfaces, applications run by the mobile computing device850, and wireless communication by the mobile computing device850.

The processor852may communicate with a user through a control interface858and a display interface856coupled to the display854. The display854may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface856may comprise appropriate circuitry for driving the display854to present graphical and other information to a user. The control interface858may receive commands from a user and convert them for submission to the processor852. In addition, an external interface862may provide communication with the processor852, so as to enable near area communication of the mobile computing device850with other devices. The external interface862may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory864stores information within the mobile computing device850. The memory864can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory874may also be provided and connected to the mobile computing device850through an expansion interface872, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory874may provide extra storage space for the mobile computing device850, or may also store applications or other information for the mobile computing device850. Specifically, the expansion memory874may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory874may be provide as a security module for the mobile computing device850, and may be programmed with instructions that permit secure use of the mobile computing device850. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier. that the instructions, when executed by one or more processing devices (for example, processor852), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory864, the expansion memory874, or memory on the processor852). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver868or the external interface862.

The mobile computing device850may communicate wirelessly through the communication interface866, which may include digital signal processing circuitry where necessary. The communication interface866may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver868using a radio-frequency. In addition, short-range communication may occur, such as using a Bluetooth®, Wi-Fi™, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module870may provide additional navigation- and location-related wireless data to the mobile computing device850, which may be used as appropriate by applications running on the mobile computing device850.

The mobile computing device850may also communicate audibly using an audio codec860, which may receive spoken information from a user and convert it to usable digital information. The audio codec860may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device850. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on the mobile computing device850.

The mobile computing device850may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone880. It may also be implemented as part of a smart-phone882, personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

In some implementations, the various modules described herein can be separated, combined or incorporated into single or combined modules. The modules depicted in the figures are not intended to limit the systems described herein to the software architectures shown therein.

Elements of different implementations described herein may be combined to form other implementations not specifically set forth above. Elements may be left out of the processes, computer programs, databases, etc. described herein without adversely affecting their operation. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Various separate elements may be combined into one or more individual elements to perform the functions described herein.

Throughout the description, where apparatus and systems are described as having, including, or comprising specific components, or where processes and methods are described as having, including, or comprising specific steps, it is contemplated that, additionally, there are apparatus, and systems of the present invention that consist essentially of, or consist of, the recited components, and that there are processes and methods according to the present invention that consist essentially of, or consist of, the recited processing steps.

It should be understood that the order of steps or order for performing certain action is immaterial so long as the invention remains operable. Moreover, two or more steps or actions may be conducted simultaneously.

While the invention has been particularly shown and described with reference to specific preferred embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.