Abstract:
A medical imaging system ( 10 ) comprises one or more displays ( 66 ). A viewer device ( 86 ) generates an interactive user interface screen ( 80 ) on the display ( 66 ), which viewer device ( 86 ) enables a user to simultaneously inspect selected image data of multiple patients or multiple images.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application is a continuation of U.S. Ser. No. 11/424,147 filed Jun. 14, 2006, which is incorporated herein by reference. 
     
    
     BACKGROUND 
       [0002]    The present application relates to the medical imaging systems and methods. It finds particular application in conjunction with multi-modality systems, such as PET-CT systems. It will be appreciated that the invention is also applicable to the various combinations of SPECT, CT, ultrasound, MRI, fluoroscopy, and the like. 
         [0003]    In multi-modality tomographic systems, two or more different sensing modalities are used to locate or measure different constituents in the object space. In the PET-CT system, the PET creates images of high metabolic activity in the body, rather than creating images of surrounding anatomy. CT scans allow doctors to see the internal structures within the human body. Before having a PET-CT scan, the patient receives a dose of a radiopharmaceutical. The pharmaceutical is carried through the blood and concentrates in a particular organ or region and causes radiation to be emitted from the blood and this organ or region. During the scan, tracings of the emitted radiation are detected by the system creating an image of the distribution of the radiopharmaceutical in the patient. The image can show the circulatory system and/or the relative absorption of the radiopharmaceutical in various regions or organs. Integration of the anatomical data from the CT scan with the metabolic data from the PET scan in the PET-CT image gives physicians visual information to determine if disease is present, the location and extent of disease, and track how rapidly it is spreading. The PET-CT system is particularly helpful in difficult-to-treat regions (e.g. head and neck area, mediastinum, postsurgical abdomen) and localization of the treatment area for the patients receiving radiation therapy or chemotherapy. 
         [0004]    As each medical imaging modality may provide complementary information on the imaged subject, it is desirable to combine all available information for review. There is a growing demand for a medical imaging review system to be able to handle multiple patients and multiple modalities over a temporally spaced series of imaging sessions. However, the current approach for viewing of the multiple patients is to load patients one at a time, which is cumbersome from a workflow standard point of view and also renders patient comparison difficult if not impossible. 
         [0005]    Another problem arises in handling multiplicity of patients and modalities. One problem, or example, is the registration of images from multiple modalities or the same modality over multiple imaging sessions. Current methods allow handling of only few images with the assumption that the first volumetric image is fixed. Another problem is in providing support when conflicting requirements due to different needs exist. One approach is to provide customizable display protocols. However, the customizable display protocols make a tightly integrated viewing environment difficult to define and implement. 
         [0006]    The present application provides new and improved methods and apparatuses which overcome the above-referenced problems and others. 
       SUMMARY 
       [0007]    In accordance with one aspect, a medical imaging system which comprises one or more displays is disclosed. A viewer device generates an interactive user interface screen on the display, which viewer device enables a user to simultaneously inspect selected image data of multiple patients or multiple images. 
         [0008]    In accordance with another aspect, a medical imaging method is disclosed. An interactive user interface screen is generated on a display. Selected image data of multiple patients or multiple images is simultaneously inspected on the display. 
         [0009]    One advantage is that multiple images can be simultaneously reviewed in a side to side comparison. 
         [0010]    Another advantage is that unlimited support of customer display protocols is provided. 
         [0011]    Another advantage resides in simplicity of image alignment. 
         [0012]    Still further advantages of the present invention will be appreciated to those of ordinary skill in the art upon reading and understand the following detailed description. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]    The invention may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. 
           [0014]      FIG. 1  is a diagrammatic illustration of an imaging system; 
           [0015]      FIG. 2  is a diagrammatic illustration of a detailed portion of the imaging system; 
           [0016]      FIG. 3  is an image of a user interface screen showing a layout; 
           [0017]      FIG. 4  is an image of a user interface screen showing another layout; 
           [0018]      FIG. 5  is a diagrammatic illustration of another detailed portion of an imaging system; 
           [0019]      FIG. 6  is a flow chart of a localization process; 
           [0020]      FIG. 7  is a flow chart of a crude registration of comparison groups; 
           [0021]      FIG. 8  is a flow chart of precise registration of comparison groups; 
           [0022]      FIG. 9  is a flow chart of a registration reformatting which results in oblique reformatting; 
           [0023]      FIG. 10  is a flow chart of a layout definition; 
           [0024]      FIG. 11  is a flow chart of a layout selection; 
           [0025]      FIG. 12  is a flow chart of converting the selecting layout to a viewing network; 
           [0026]      FIG. 13  is a continuation of the flow chart of converting the selected layout to a viewing network; 
           [0027]      FIG. 14  is an image of one of the layouts; 
           [0028]      FIG. 15  is an image of a modified layout of  FIG. 14 ; 
           [0029]      FIG. 16  is an image showing the layout of  FIG. 15  and a newly saved icon for this layout; and 
           [0030]      FIG. 17  is a flow chart of control of oblique TSC display. 
       
    
    
     DETAILED DESCRIPTION 
       [0031]    With reference to  FIG. 1 , an operation of an imaging multi-modality system  10 , which, for example, include any combination of known imaging modalities, is controlled from an operator workstation  12  which is coupled to a hospital network  14 . The hospital network  14  includes associated hospital software  16  and a hospital records database  18 . The workstation  12  may be hardwired to the network  14  or may communicate with it wirelessly. In one embodiment, the workstation  12  communicates with other hospital workstations or computers, which are connected to the hospital network  14 , enabling the images and patient records to be forwarded to the appropriate hospital personnel and displayed on associated monitors. Of course, it is contemplated that imaging system  10  is a stand alone system without network connections. 
         [0032]    The multi-modality system  10  includes a first imaging system, preferably a functional modality, preferably, a nuclear imaging system  20 , and a second imaging system, such as a computed tomography (CT) scanner  22 . The CT scanner  22  includes a non-rotating gantry  24 . An x-ray tube  26  is mounted to a rotating gantry  28 . A bore  30  defines an examination region  32  of the CT scanner  22 . An array of radiation detectors  34  is disposed on the rotating gantry  28  to receive radiation from the x-ray tube  26  after the x-rays transverse the examination region  32 . Alternatively, the array of detectors  34  may be positioned on the non-rotating gantry  24 . Of course, other imaging systems are also contemplated. 
         [0033]    The functional or nuclear imaging system  20 , preferably, includes a positron emission tomography (PET) scanner  40  which is mounted on tracks  42 . Of course, SPECT and other imaging systems are also contemplated. The tracks  42  extend in parallel to a longitudinal axis of a subject support or couch  44 , thus enabling the CT scanner  22  and PET scanner  40  to form a closed system. A moving means  46 , such as a motor and a drive, is provided to move the PET scanner  40  in and out of the closed position. Detectors  48  are arranged around a bore  50  which defines an examination region  52 . In the PET system, the detectors  48  are preferably arranged in a stationery ring, although rotatable heads are also contemplated. In the SPECT system, the detectors  48  are preferably incorporated into individual heads, which are mounted for rotational and radial movement relative to the patient. A couch moving means  54 , such as a motor and a drive, provides a longitudinal movement and vertical adjustment of the couch  44  in the examination regions  32 ,  52 . 
         [0034]    With continuing reference to  FIG. 1 , an operator, user or imaging technician performs a scan using the workstation  12  which includes a CPU processor or hardware device  60  and a software component or means  62  for carrying out the image processing functions and operations. The workstation  12  preferably includes one or more input devices  64  (e.g., a computer keyboard, mouse, etc.), and one or more monitors or displays  66 . The user, via the input devices  64 , can selectively control the workstation  12  and/or the entire imaging system  10 . The couch  44 , which carries a subject, is moved by the couch moving means  54  into the examination region  52  for a PET image to be generated by the PET scanner  40 . Electronic data is reconstructed into a PET image by a PET reconstruction processor  70  and stored in a PET image memory  72 . The couch moving means  54  moves the couch  44  to position the couch carrying the subject in the CT scanner examination region  32 , where a CT image is taken. More particularly, the couch  44  with the subject is moved to the position in the CT examination  32  region that is geometrically and mechanically predicted as being the same as its imaged position in the PET imaging region  52 . Electronic data is reconstructed into a 3D CT image by a CT reconstruction processor  74  and stored in a CT image memory  76 . 
         [0035]    The PET and CT images are aligned, registered, fused or manipulated in any other appropriate way, after which the image data is appropriately formatted by a video processor  78  for display on the monitor  66 . 
         [0036]    The operator or user controls display of images by using an application interface screen or screens  80  which are incorporated into the workstation  12  and displayed on the monitor  66 . An interface component, processor, or means  82  controls the application interface  80 . The operator uses the input device, such as a keyboard or mouse  64 , to interact with an applications database  84  by navigating the application interface screens  80 . 
         [0037]    With continuing reference to  FIG. 1  and further reference to  FIG. 2 , a viewer algorithm, device or means  86  allows the user to view, navigate and compare images of multiple patients. For example, the user selects multiple patient data to review and/or compare patients via the interface screens  80 . Selected patient data  88  is passed to the viewer device  86  which then creates a patient viewer manager component  90  and passes the data to it. 
         [0038]    The patient viewer manager  90  maintains the selected image data for each individual patient. More specifically, the patient viewer manager  90  keeps track of which patients are currently being viewed, launches any new patient for viewing in any order, closes a patient in viewing, and tabs all patients in viewing or tiles all patients in viewing for direct side by side comparison. The patient viewer manager  90  can launch a patient by name, or launch a patient before or after a given patient. If the patient is already launched, it is brought up to the front. If the patient&#39;s record is not launched or opened, a new launch is created. 
         [0039]    A patient viewing component, algorithm, device or means  92  is responsible for a given patient viewing. The first patient on the list is chosen for viewing automatically at start-up time. More than two patients can be displayed in a tiled format which can be in a single display monitor or cross multiple monitors, for direct side by side comparison. 
         [0040]    The navigations among the patients in viewing are done through clicking on a patient tab or selecting that patient from a graphical user interface control (a list control). The patient data is loaded into the application on-demand to reduce the memory usage. However, it is also possible to load all data if the total data does not consume much memory or if the memory with a proper size is available. In this manner, multiple patients are viewed directly in a single integrated environment. 
         [0041]    With continuing reference to  FIG. 2  and further reference to  FIG. 3 , a layout manager, component, device, algorithm or means  94  manages layouts. More specifically, the layout manager  94  maintains a collection of layouts or layout pool  96 . The layouts are obtained from embedded resources for factory layouts  98  and files for user defined layouts  100 . Given a particular set of selected patient image data for reviewing, a predefined startup layout  102  is determined by consulting a preference setting. The user may switch to a different layout yet to be created or visited previously during a reviewing session. The user may modify a predefined layout as shown in  FIG. 4  and save the modified layout  104 . The user may also start from scratch, define his or her own layout, set it as the preferred one for a particular given set of image data, and use it in subsequent viewing sessions. For the currently displayed layout, the user may replace one image data with a different image data. The replacing image data can be displayed in the context of the replaced data, e.g. in the same layout. Alternatively, the best matching layout is found and launched. All images are displayed with the same viewing context. 
         [0042]    With continuing reference to  FIG. 2 , a registration component, device, algorithm, processor or means  108  selects a global reference image  110  and registers or aligns images  112  with one another with respect to the global reference image  110 . Any number of images can be managed and registered at a single time. Any image can participate in the registration as either a reference image or a floating image. More specifically, if a single image is selected for viewing, the image is viewed in its own coordinate system. When two or more images are selected for viewing, one image is selected as a reference image and the rest of the images is designated as floating image(s). The floating images are shifted and rotated to match the reference image. The individual relation of each floating image to the reference image is computed based on the relation of each image to the global reference as discussed in detail below. The initial registration relation is determined on the as needed basis. In the registration that involves multiple images, the registration relation is propagated to other images in a well-controlled manner 
         [0043]    For example, when a group of images (including the case where only one image is selected) is selected for viewing for the first time, the registration device  108  selects the first volumetric image in the group as the global reference  110  (once the global reference is determined, it will not change). A global reference coordinate system, which defines a common frame of reference, is stored in a global reference memory  114 . 
         [0044]    When an image  112  is selected for viewing, the registration device  108  determines and stores a registration matrix (or parameters) for this particular volumetric image with respect to the global reference and stores the registration matrix in a registration matrix memory  116 . 
         [0045]    In one embodiment, the registration device  108  compares the frame of reference of each image to the common frame of reference, e.g. frame of reference of the global reference image. If the image has the same frame of reference unique identification (FoR UID) as that of the global reference and there is no other information to overwrite that, the registration matrix is identity. If the image and the global reference have different FoR UID and the registration is not known a priori, the registration device  108  aligns the volume centers of the two images and propagates the resultant registration matrix to all other volumetric images that share the same FoR UID as the image under registration. As a result, registrations of such images to the global reference image do not need to be determined. If the registration between the image and the global reference is known a priori, the registration device  108  propagates the determined registration matrix to all other volumetric images that share the same FoR UID as the image under registration. The registrations of such images to the global reference image do not need to be determined In this manner, registrations of multiple images are simplified. 
         [0046]    The registration of the global reference can be changed. Any change is interpreted as a change with respect to its original coordinate system. The registration of any image with respect to the global reference is interpreted with respect to the original coordinate system of the global reference. 
         [0047]    As an alternative, the global reference can be duplicated or copied to be registered to other images as a floating image. As another alternative, the registration matrices of each image can be determined in a single step rather than on-demand basis. 
         [0048]    With continuing reference to  FIG. 2  and further reference to  FIG. 5 , the registration processor  108  computes a relative registration relation M r1r2  between first and second or reference and floating volumetric images  117 ,  118  as: 
         [0000]        M   r1r2   =M   r1   −1   M   r2 , where 
         [0000]    M r1  is the registration of the first image with respect to the global reference; and
 
M r2  is the registration of the second image with respect to the global reference.
 
         [0049]    The relative registration M r1r2  between the reference and floating volumetric images  117 ,  118  is stored in a registration memory  119 . 
         [0050]    When the relative registration M r1r2  of the floating image  118  with respect to the reference image  117  is changed due to any reasons (manual registration, automatic registration, etc), the equivalent change with respect to the global reference image  110  is computed and recorded. The same registration is propagated to other images in a controlled manner. 
         [0051]    The registration M r2  of the floating image  118  with respect to the global reference is: 
         [0000]      M r2 =M r1 M r1r2 . 
         [0052]    If the reference image and floating image have different FoR UID, the registration M r2  of the floating image with respect to the global reference is propagated to all other volumetric images that share the same FoR UID as the floating image under concern. If the FoR UID is not present, it is considered as a mismatch. 
         [0053]    If the reference image and floating image have the same FoR UID, to correct the motion between CT and PET acquisition, for example, the registration M r2  of the floating image with respect to the global reference is propagated to all other images that share the same FOR UID as the floating image under concern and have the same modality as the floating image. 
         [0054]    In this manner, a global reference volumetric image  110  is introduced and all other volumetric images are registered to such virtual global reference. There is no limit on the number of volumetric images the registration device  108  can handle and register at a single time. The coordinate system of the global reference is not fixed. The global reference can participate in the registration as a floating image with respect to other images (the coordinate system of the global reference before registration is used as global reference coordinate system and thus is virtual). The registration relation between any two volumetric images is computed from their respective relation with the global reference. The individual relation to the global reference can be decided inside the same application or inferred from outside information (same or different frame of reference unique identification (FoR UID), or DICOM spatial registration object). The global reference is intelligently selected. For example, in the case of mixed volumetric and non-volumetric images, the global reference is not set until a first volumetric image is selected for viewing. Likewise, the individual registration with respect to the global reference is not determined until the image is selected for viewing. If there is no pre-determined registration between the global reference and the image under concern, the two centers of the volumetric images are aligned and that alignment relation is set as the initial registration that can be overwritten later if necessary. Moreover, the so-determined initial registration is propagated to other volumetric images if their relations to the image under concern are known. 
         [0055]    When multiple floating images are registered to the same reference image, the registrations with respect to the reference image can be updated as a group with the option that each individual image can be either updated or not updated. 
         [0056]    With continuing reference to  FIG. 2 , a localization device, algorithm, processor or means  120  uses a maximum intensity projection (MaxIP) of the PET image to update other image planes such as transverse, sagittal, and/or coronal images. More specifically, the user selects a point of interest, for example, by clicking on a point in the image with the mouse. The localization device  120  computes the exact location of the selected point and uses this point&#39;s 3D coordinates to update all other image planes. The clicked point is computed in the original coordinate system and transformed to the common registered coordinate system. If there are more than two images which are registered and displayed, the localization device  120  allows the registered and fused images to be updated properly. Optionally, the MaxIP image, transverse, sagittal, and/or coronal images of other volumetric images can be displayed at the same time and can be fused with the corresponding transverse, sagittal, or coronal images. As a further option, the MaxIP image can be fused. In the case of MaxIP fusion, the user has the option to specify which MaxIP image from the correlation to be used from this addresses the case in which two different physical locations contribute to the same point in the MaxIP images. In the case that the clicked point is from a regular two-dimensional image plane (transverse/sagittal/coronal), a point in the MaxIP image is also identified where the maximum intensity pixel and the clicked point are in the same projection line. 
         [0057]    More specifically, with particular reference to  FIG. 6 , the localization device  120  performs a localization process  122  after the user selects a point on the MaxIP image. In step  122   a,  the position of the clicked point is retrieved in the screen coordinate system. In step  122   b,  the screen position is transformed into a point in patient coordinate system by taking into account the image position and image orientation vectors. In step  122   c,  the original contributing point in patient coordinate system is sought. Optionally, the information can be stored and no further computation is needed. In step  122   d,  the coordinates found in step  122   c  are transformed into a common coordinate system (i.e. of the global reference image). In step  122   e,  the coordinates in the common coordinate system are used to update all associated images. E.g., the transverse image at the same z position is displayed. 
         [0058]    With reference again to  FIGS. 3 and 4 , a selected point of interest  124  is illustrated by a cross. The selected point of interest  124  is propagated to all associated images to identify corresponding points of interest  126 . 
         [0059]    With reference again to  FIG. 5 , a comparator  130  compares temporal studies or groups  132 ,  134  of selected images. More specifically, the user selects different groups such as first and second groups of images  132 ,  134  for comparison via the interface screen  80 . For example, if two or more DICOM study objects are selected, the volumetric images in each study form a group. Volumetric images can also be directly selected to form groups. The user can start with one group and use combinations of keys and mouse or other devices to select more groups. A two-dimensional array is used to represent selected groups, in which each row is a group. One of the volumetric images, for example a first image in the group, is selected as the reference image  117 , for example, automatically or manually. The rest of the images of the group are designated as floating images  118 . All floating images  118  in the same group are registered to the reference image  117  in the same group. 
         [0060]    Within a group, the floating image has to be informed which image is the reference image; and the reference image has to be informed it is a reference image. 
         [0061]    After the comparison groups  132 ,  134  are formed, the comparison groups  132 ,  134  can be displayed independently. A group registration device, algorithm, means or processor  138  determines a linking relation between groups that is used in subsequent reviewing. The registration can be established in at least two different ways: (1) the crude registration and (2) the precise registration. 
         [0062]    With continuing reference to  FIG. 6  and further reference to  FIG. 7 , the group registration processor  138  determines the crude registration in a crude group registration process  140 . In step  142 , a point correlation (triangulation) is used to identify a common point (hot spot, internal mark, or external mark) in any volumetric image in the first group  132 . For example, the user clicks on the point of interest in one of the images of the first group  132 . Step  142  can be performed multiple times. Position of the identified point is stored  144  in a first group memory  146 . In step  148 , the point correlation is similarly used to identify the same common point in any volumetric image in the second group  134 . For example, the user clicks on the similar point in one of the images of the second group  134 . Step  148  can be performed multiple times. Position of the identified point is stored  150  in a second group memory  152 . Of course, step  148  can be performed on more than one group. 
         [0063]    Alternatively, several common points can be identified in each group  132 ,  134  to improve the registration accuracy. The positions of the points are stored in a corresponding first or second group memory  146 ,  152 . 
         [0064]    After the common points in all groups are identified, the group registration processor  138  retrieves the stored positions from the first and second group memories  146 ,  152  and the registration is computed  154 . The process  140  can be initialized with a clicking on a graphical (button) control. More hue can be provided once the relation is computed. Only the relation with respect to the first group is computed regardless of a number of groups selected. 
         [0065]    If there is only one point identified in each group, only translation relation between two groups is considered, which is given by the differences between their x/y/z coordinates. If two points are identified in each group, translation (or translation and one rotation) relation between two groups is considered. If more than three (inclusive) points are identified, the relation can be found by the technique used in conjugate-point based registration. It is not required to select the same number of points in each group. 
         [0066]    With continuing reference to  FIG. 6  and further reference to  FIG. 8 , the group registration processor  138  determines  160  the precise registration between groups by using registration between two studies (one volumetric image from each study group) and linking the groups by this precise relation. For example, a first image in each group is selected as the reference image. The registration M r1  of the reference image  117  of in the first group  132  with respect to the global reference  110  is retrieved  162 . The registration M r2  of the reference image of the second group  134  with respect to the global reference  110  is retrieved  164 . The registration M r1r2  between the reference image in the first group  132  and the reference image in the second group  134  is determined  166  as: 
         [0000]        M   r1r2   =M   r1   −1   M   r2 . 
         [0000]    where M r1  is the registration of the first reference image with respect to the global reference; and
 
M r2  is the registration of the second reference image with respect to the global reference.
 
         [0067]    The registration M r1r2  is used as the registration between the two groups  132 ,  134 . For other groups if any, only the relation with the first group  132  is maintained which is determined in the same manner Alternatively, all pairwise relations can be maintained and used. Once the group relation is established, the same relation is used to link the localization points in the reference images of each group. Whenever the localization point changes in any group, the corresponding point in other groups are computed using the linking relation established early and the computed point is set to their groups, which triggers the update within the group. 
         [0068]    With reference again to  FIG. 2  and further reference to  FIG. 9 , an oblique processor, device, algorithm or means  168  performs  170  a registration reformatting to do an oblique reformatting to one of the two images. For example, two volumetric images A and B are registered with respect to the global reference and the registration matrix  116  is available. Alternatively, images A, B share the same FoR UID. Alternatively, images A, B are registered  172 , either manually or automatically, inside or outside the same application and the registration matrix is available. Alternatively one can directly set the registration parameters using the provided registration numerical values. 
         [0069]    In step  174 , an oblique image is received. For example, at least one of the images A, B is oblique. The image can be oblique when the image is loaded from disk. Alternatively, the image is made oblique reformatted within the same application. As a further option, the image itself is not oblique directly, but the oblique parameters are set and used. 
         [0070]    In step  176 , an oblique plane is determined. For example, if the oblique images are loaded from the disk directly, the oblique plane is determined from the image orientation vectors. The cross product of those two vectors gives a third vector. The name of the oblique plane (oblique transverse/sagittal/coronal) can be determined by the information coded in the image if present; otherwise, it is determined by comparing the image orientation vectors with respect to the standard x/y/z axes, and if it is closest to the transverse (sagittal or coronal) plane, it is oblique transverse (sagittal or coronal). The comparison is done on the summation of squared inner products between normalized vectors and the larger, the closer. In the comparison, the directions of those vectors are ignored. Denote the three oblique vectors as u, v, and w, which are closest to the x, y, and z-axes respectively. Generally, the directions of the u, v, and w vectors point to the directions of positive x, y, and z-axes respectively; if not, they are negated. 
         [0071]    If the oblique images are not loaded from disk files, but generated, then the three oblique vectors can be decided in the same fashion. 
         [0072]    A set of geometric information is generated for oblique slice creation, which includes the image orientation vectors, the number of slices, the center of the generated slices, the physical extent of the generated slices, and the number of pixels and pixel physical sizes for the slices. The image orientation vectors are determined by the name of the oblique planes. For example, if an oblique transverse view is requested, two vectors u and v are used. If an oblique sagittal view is requested, y and w are used. If an oblique coronal view is requested, then w and x are used. 
         [0073]    In steps  178 ,  180 , the oblique image (Image B) is selected as a reference image  117  and an image to be oblique (Image A) is selected as a floating image  118 . The floating image  118  is automatically oblique reformatted  182  due to the registration reformatting. Particularly, the generated geometric information is used to create oblique slice for the image A. The geometric information can be used directly. As an alternative, the center of the generated slices, the physical extent of the generated slices, and the number of pixels and pixel physical sizes for the slices can vary. When generating the slices for a different image, the registration of this image with respect to the reference image, on which the geometric information (mainly orientation vectors and image position) is specified, is taken into account. This process is called registration reformatting. User can optionally save the secondary reformatted images in any oblique plane should it be available. 
         [0074]    In this manner, a new workflow concept utilizes the registration reformatting to do the oblique reformatting, which avoids the manual or automatic oblique reformatting operation on associated images. In the fusion display, the oblique planes of the oblique image (reference) are used rather than the conventional transverse/sagittal/coronal, i.e. the oblique transverse/sagittal/coronal image planes are used. Due to the registration reformatting, the other (floating) image is automatically oblique reformatted to have the same image planes as the oblique (reference) image. This way, the tedious manual oblique reformatting on the (floating) image can be avoided or the automatic oblique reformatting becomes unnecessary, which often generates a different set of oblique parameters. 
         [0075]    With continuing reference to  FIG. 2  and further reference to  FIG. 10 , the layout manager  94  defines and/or creates a layout definition  196  in step  198 , in which layout attributes  200 , such as data inputs  202  and viewers  204  are specified  206 , and data is associated with the viewers  204 . The layout definition  196  can be defined in any format of a text file or a data object as long as it can be easily manipulated. An exemplary XML format is: 
         [0000]    
       
         
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 &lt;name&gt;Axial-comparison-AllF&lt;/name&gt; 
               
               
                   
                 &lt;description&gt;AxialComparison&lt;/description&gt; 
               
               
                   
                 &lt;writable&gt;false&lt;/writable&gt; 
               
               
                   
                 &lt;favoriteRating&gt;101&lt;/favoriteRating&gt; 
               
               
                   
                 &lt;inputs&gt; 
               
             
          
           
               
                   
                 &lt;Characteristics&gt; 
               
             
          
           
               
                   
                 &lt;inputReference&gt;Input1&lt;/inputReference&gt; 
               
               
                   
                 &lt;modality&gt;Anatomical&lt;/modality&gt; 
               
             
          
           
               
                   
                 &lt;/Characteristics&gt; 
               
               
                   
                 &lt;Characteristics&gt; 
               
             
          
           
               
                   
                 &lt;inputReference&gt;Input2&lt;/inputReference&gt; 
               
               
                   
                 &lt;modality&gt;FunctionalAnatomical&lt;/modality&gt; 
               
             
          
           
               
                   
                 &lt;/Characteristics&gt; 
               
             
          
           
               
                   
                 &lt;/inputs&gt; 
               
               
                   
                 &lt;ViewerCfgs&gt; 
               
             
          
           
               
                   
                 &lt;ViewerCfg&gt; 
               
             
          
           
               
                   
                 &lt;width&gt;50&lt;/width&gt; 
               
               
                   
                 &lt;height&gt;100&lt;/height&gt; 
               
               
                   
                 &lt;x&gt;0&lt;/x&gt; 
               
               
                   
                 &lt;y&gt;0&lt;/y&gt; 
               
               
                   
                 &lt;viewerType&gt;MPR&lt;/viewerType&gt; 
               
             
          
           
               
                   
                 &lt;fusionState&gt;Underlay&lt;/fusionState&gt; 
               
             
          
           
               
                   
                 &lt;underlay&gt; 
               
             
          
           
               
                   
                 &lt;inputReference&gt;Input1&lt;/inputReference&gt; 
               
             
          
           
               
                   
                 &lt;/underlay&gt; 
               
               
                   
                 &lt;overlay&gt; 
               
             
          
           
               
                   
                 &lt;inputReference&gt;Input2&lt;/inputReference&gt; 
               
             
          
           
               
                   
                 &lt;/overlay&gt; 
               
               
                   
                 &lt;orientation&gt;Transverse&lt;/orientation&gt; 
               
               
                   
                 &lt;zoomFactor&gt;1.3&lt;/zoomFactor&gt; 
               
             
          
           
               
                   
                 &lt;/ViewerCfg&gt; 
               
               
                   
                 &lt;ViewerCfg&gt; 
               
             
          
           
               
                   
                 &lt;width&gt;50&lt;/width&gt; 
               
               
                   
                 &lt;height&gt;100&lt;/height&gt; 
               
               
                   
                 &lt;x&gt;50&lt;/x&gt; 
               
               
                   
                 &lt;y&gt;0&lt;/y&gt; 
               
               
                   
                 &lt;viewerType&gt;MPR&lt;/viewerType&gt; 
               
             
          
           
               
                   
                 &lt;fusionState&gt;Overlay&lt;/fusionState&gt; 
               
             
          
           
               
                   
                 &lt;underlay&gt; 
               
             
          
           
               
                   
                 &lt;inputReference&gt;Input1&lt;/inputReference&gt; 
               
             
          
           
               
                   
                 &lt;/underlay&gt; 
               
               
                   
                 &lt;overlay&gt; 
               
             
          
           
               
                   
                 &lt;inputReference&gt;Input2&lt;/inputReference&gt; 
               
             
          
           
               
                   
                 &lt;/overlay&gt; 
               
               
                   
                 &lt;orientation&gt;Transverse&lt;/orientation&gt; 
               
               
                   
                 &lt;enableCine&gt;false&lt;/enableCine&gt; 
               
               
                   
                 &lt;zoomFactor&gt;1.3&lt;/zoomFactor&gt; 
               
             
          
           
               
                   
                 &lt;/ViewerCfg&gt; 
               
             
          
           
               
                   
                 &lt;/ViewerCfgs&gt; 
               
             
          
           
               
                   
                 &lt;/Layout&gt; 
               
               
                   
                   
               
             
          
         
       
     
         [0076]    Such exemplary layout defines two inputs. A first input (“Input 1 ”) looks for any anatomical volume series. A second input (“Input 2 ”) looks for any supported volume series, which are defined in a modality field. The value of the modality field can also be very specific, for example, MR, CT, PET, SPECT, SC (secondary capture), etc. The exemplary layout next defines two viewers (“ViewerCfg”), which have fusion capability. The viewers show transverse image orientation: one viewer is showing underlay image and the other viewer is showing overlay image as the starting viewing option. The layout manager  94  can use this file as an input, in conjunction with the selected series and a set of built-in linking rules, to build a complete specification. The layout definition  196  is, of course, scalable to include multiple inputs, multiple viewers with different types, and different viewer configurations. Tables 1-3 below explain how to configure each attribute  200  of a layout definition. 
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 Layout 
               
             
          
           
               
                 Attribute 
                 Type 
                 Description 
               
               
                   
               
               
                 writable 
                 bool 
                 0: this layout is read only; 1: layout 
               
               
                   
                   
                 can be overwritten, and can be removed. 
               
               
                 name 
                 string 
                 Layout name is served as ID. The layout 
               
               
                   
                   
                 name is unique among all layouts defined 
               
               
                   
                   
                 by one physician. 
               
               
                   
                   
                 The layout name/layout icon name/layout 
               
               
                   
                   
                 file name are the same. 
               
               
                 description 
                 string 
                 Layout description is used as tool tip 
               
               
                   
                   
                 and item display on preference page. 
               
               
                 favoriteRating 
                 int 
                 A score. The higher the value, the better 
               
               
                   
                   
                 chance the layout will be selected when 
               
               
                   
                   
                 multiple layouts match a same query. 
               
               
                 ViewerCfgs 
                 IList 
                 A list of all viewer configurations. 
               
               
                 inputs 
                 Charac- 
                 Characterized data inputs. Each input forms 
               
               
                   
                 teritics[ ] 
                 a query to all selected data from application 
               
               
                   
                   
                 data model (a software component that is 
               
               
                   
                   
                 responsible for packaging and reformatting 
               
               
                   
                   
                 image data for display). 
               
               
                   
               
             
          
         
       
     
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 2 
               
             
             
               
                   
               
               
                 ViewerCfg 
               
             
          
           
               
                 Attribute 
                 Type 
                 Description 
               
               
                   
               
               
                 Name 
                 String 
                 Viewer name, which is normally 
               
               
                   
                   
                 generated by layout manager at runtime 
               
               
                   
                   
                 if this is not specified. It is used as a 
               
               
                   
                   
                 node name by a protocol engine to 
               
               
                   
                   
                 associate with a viewer. 
               
               
                 Width 
                 int 
                 Width of tab container holding the 
               
               
                   
                   
                 viewer 
               
               
                 Height 
                 int 
                 Height of tab container holding the 
               
               
                   
                   
                 viewer 
               
               
                 X 
                 int 
                 X location of the tab container holding 
               
               
                   
                   
                 the viewer 
               
               
                 Y 
                 int 
                 Y location of the tab container holding 
               
               
                   
                   
                 the viewer 
               
               
                 screenIndex 
                 int 
                 0—main monitor, 1—second monitor 
               
               
                 viewerType 
                 ViewerType 
                 The exemplary values include 
               
               
                   
                   
                 Basic2D 
               
               
                   
                   
                 MPR 
               
               
                   
                   
                 MIP 
               
               
                   
                   
                 CardiacOblique 
               
               
                   
                   
                 BrainOblique 
               
               
                   
                   
                 Different viewer type means different 
               
               
                   
                   
                 default viewer configuration. For 
               
               
                   
                   
                 example, for basic 2D viewing, we do 
               
               
                   
                   
                 not provide fusion capability for this 
               
               
                   
                   
                 viewer. For MIP (maximal intensity 
               
               
                   
                   
                 projection) viewer, we do not provide 
               
               
                   
                   
                 any ROI measurement tool. For 
               
               
                   
                   
                 CardiacOblique viewer and 
               
               
                   
                   
                 BrainOblique viewer, we provide some 
               
               
                   
                   
                 specialized image interactors. 
               
               
                 underly 
                 Characteristics 
                 Used to refer to which input source is 
               
               
                   
                   
                 used for the underlying data view. For 
               
               
                   
                   
                 example, it is a query among all layout 
               
               
                   
                   
                 inputs. 
               
               
                 overlay 
                 Characteristics 
                 Used to refer to which input source is 
               
               
                   
                   
                 used for the superimposed data view. For 
               
               
                   
                   
                 example, it is as a query among all 
               
               
                   
                   
                 layout inputs. 
               
               
                 orientation 
                 ImageOrientation 
                 Values include 
               
               
                   
                   
                 Any (Basic2D, MIP) 
               
               
                   
                   
                 Transverse (TSC display) 
               
               
                   
                   
                 Sagittal (TSC display) 
               
               
                   
                   
                 Coronal (TSC display) 
               
               
                 transformedStage 
                 TransformedStage 
                 Values include 
               
               
                   
                   
                 NotTransformed (default) 
               
               
                   
                   
                 Intermediate1 
               
               
                   
                   
                 Transformed 
               
               
                   
                   
                 Any values other than NotTransformed 
               
               
                   
                   
                 are only meaningful for oblique utility 
               
               
                   
                   
                 layouts. 
               
               
                 columns 
                 unit 
                 Rows of tiles in the viewer 
               
               
                 rows 
                 unit 
                 Columns of tiles in the viewer 
               
               
                 enableCine 
                 bool 
                 This flag indicates if cine should be 
               
               
                   
                   
                 enabled when the viewer is created. 
               
               
                 zoomFactor 
                 float 
                 Zoom factor with which the viewer 
               
               
                   
                   
                 should initially apply to images. If it is 
               
               
                   
                   
                 not specified, then default 1 will be used. 
               
               
                 viewerConfigurationClass 
                 string 
                 This attribute allows user to specify 
               
               
                   
                   
                 his/her own viewer configuration class, 
               
               
                   
                   
                 which is normally done by layout 
               
               
                   
                   
                 manager automatically based on viewer 
               
               
                   
                   
                 type. 
               
               
                 extraActions 
                 string[ ] 
                 This attributes allows user to instruct 
               
               
                   
                   
                 layout manager to execute canned 
               
               
                   
                   
                 actions specified here after a viewing 
               
               
                   
                   
                 network is established. 
               
               
                   
               
             
          
         
       
     
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 3 
               
             
             
               
                   
               
               
                 Characteristics 
               
             
          
           
               
                 Attribute 
                 Type 
                 Description 
               
               
                   
               
               
                 name 
                 string 
                 Unique string used to identify data view 
               
               
                   
                   
                 generated from this input. It is 
               
               
                   
                   
                 generated internally by protocol builder 
               
               
                 inputReference 
                 InputIndex 
                 Values include 
               
               
                   
                   
                 None 
               
               
                   
                   
                 Input1 
               
               
                   
                   
                 Input2 
               
               
                   
                   
                 Input3 
               
               
                   
                   
                 Input4 
               
               
                   
                   
                 Input5 
               
               
                   
                   
                 Input6 
               
               
                   
                   
                 --- 
               
               
                   
                   
                 ALL 
               
               
                   
                   
                 This attribute keeps tracking the original 
               
               
                   
                   
                 input index. With this macro, we can 
               
               
                   
                   
                 search all inputs available that meet the 
               
               
                   
                   
                 criterions presented in Characteristics 
               
               
                   
                   
                 data object. 
               
               
                 modality 
                 Modality 
                 Values include supported modality: 
               
               
                   
                   
                 Any—any modality including secondary 
               
               
                   
                   
                 capture, 
               
               
                   
                   
                 FunctionalAnatomical—any volume 
               
               
                   
                   
                 Functional—any functional volume 
               
               
                   
                   
                 such as PET and SPECT, 
               
               
                   
                   
                 Anatomical—any anatomical volume, 
               
               
                   
                   
                 PET, CT, MR, SPECT 
               
               
                 method 
                 Method 
                 Value include 
               
               
                   
                   
                 Any (default) 
               
               
                   
                   
                 Diagnostic (CT) 
               
               
                   
                   
                 LowDose (CT) 
               
               
                   
                   
                 CTAC (PET) 
               
               
                   
                   
                 NonAC (PET) 
               
               
                 contrastAgent 
                 ContrastAgent 
                 Enumeration type. Value include 
               
               
                   
                   
                 Either (default) 
               
               
                   
                   
                 NonContrast (not contrast) 
               
               
                   
                   
                 Contrast (contrast enhanced) 
               
               
                 groupIndex 
                 Int 
                 It is used in group comparison. It 
               
               
                   
                   
                 indicates which group this input belongs 
               
               
                   
                   
                 to. Default value is 0 if it is not 
               
               
                   
                   
                 specified. 
               
               
                 ImageType 
                 string[ ] 
                 Array of strings requiring all image 
               
               
                   
                   
                 types in this array to be present in 
               
               
                   
                   
                 order for an NM image to be matched. 
               
               
                   
               
             
          
         
       
     
         [0077]    In this manner, users define a balanced layout definition by editing a simple layout file, with which a complete display protocol can be built on the fly based on the current data selection, user preference, and built-in application linking rules. This allows the users to review and interact with the images in the way that the layout is specified. 
         [0078]    With continuing reference to  FIGS. 2 and 10  and further reference to  FIG. 11 , in step  208 , the layout device  94  performs data sorting and layout selection. In step  210 , a series characterization is performed. More specifically, some series&#39; properties are retrieved and saved in an object of characteristics class, which is then put into a hash table with the series as its key. Some of other information such as group index, which is not part of series&#39; attribute, is obtained from input groups. 
         [0079]    When the selected data  88  includes multi studies, the layout manager  94  performs a study sorting  212  based on studies sorting rules  214  and decides which study should be displayed first. For example, the studies sorting rules include a rule by which a newer study has a higher priority than an older study. 
         [0080]    In step  220 , a series role sorting is performed based on series role sorting rules  222  to select a series which can serve as a reference. For example, a resolution along the z direction is checked in all selected series to determine which one in its group is the best for reference. If the resolution fails to tell the difference, then the modality is used to differentiate them with the following order CT/MR/PET/SPECT. 
         [0081]    In step  230 , the series are sorted based on series sorting rules  232  to determine which series will be used first for data matching purpose. For example, the series are sorted into two groups such as anatomical and functional. For example, the series are sorted based on a modality with following order: PET/SPECT/CT/MR/SC. 
         [0082]    For example, functional series, e.g. images generated, for example, with the nuclear imaging system, are sorted based on following priorities:
       Registered over non-registered   CT Attenuation Corrected (CTAC) over non-attenuation corrected (NonAC)   Larger field of view (FOV) over smaller FOV   Longer Z coverage over shorter Z coverage (such as whole body over brain)   Static over dynamic   Reformatted over non-reformatted   Newer over older       
 
         [0090]    For example, anatomical series, e.g. images generated with, for example, the CT scanner, are sorted based on the following priorities:
       Higher resolution over lower resolution   Larger FOV over smaller FOV (such as low dose CT over diagnose CT)   Longer Z coverage over shorter Z coverage   Contrast enhanced over non-contrast enhanced Newer over older       
 
         [0095]    After the sorting, series are picked alternately from functional group and anatomical group if possible until limit is reached ( 3  series for a non-comparison case). For example, if one CT, one CTAC PET, one NonAC PET and one SC series are selected, then in the final list are CTAC PET/CT/NonAC series. 
         [0096]    In step  240 , all applicable matching layouts  242  among all available layouts are identified for data selection. For example, the layout identification algorithm is: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 Characterize input series 
               
               
                   
                 for each layout 
               
               
                   
                  get all required input characteristics 
               
               
                   
                  for each input characteristics 
               
               
                   
                   get a list of series matching this characteristics 
               
               
                   
                    if there isn&#39;t any series in the list which is not included 
               
               
                   
                     the layout is considered not matched 
               
               
                   
                      continue on next layout 
               
               
                   
                     else 
               
               
                   
                      include the new series and 
               
               
                   
                      continue on next required input characteristics 
               
               
                   
                   add this layout into matched layout list 
               
               
                   
                 sort all matched layouts 
               
               
                   
                   
               
             
          
         
       
     
         [0097]    The layouts are selected or sorted  250  based on input matching number, user favorite rating, and layout sorting rules  252  to decide which layout should be launched first. For example, a user selects one PET and one CT and there are one layout displaying PET and another displaying PET/CT. The layout sorting rules  252  include a rule by which the PET/CT layout gets higher priority regardless of favorite rating. Favorite rating starts to play a role when there are more than one layout matching the same number of inputs. Another example of the rule is the rule which gives a priority to the current layout if the current layout matches the current series selection. 
         [0098]    In this manner, the optimal layout is automatically selected based on the data selection. This is based on built-in study and series sorting algorithms, the list of available layouts and the user favorite rating, and the layout that is currently being displayed. 
         [0099]    With continuing reference to  FIGS. 2 ,  10  and  11  and further reference to  FIGS. 12 and 13 , the layout device  94  converts  260  the selected layout to a viewing network. The layout manager  94  assigns  262  monitors to the selected layout. For example, some layouts require multiple monitors for display. Effective viewing areas of each monitor are computed and use to configure a monitor configure object, which is used by the protocol engine. As a result, the viewers are displayed in monitors where the viewers are supposed to be displayed without crossing a monitor boundary. 
         [0100]    In step  264 , input macros are expanded. As described above, layout inputs are defined through characteristics objects, which may include some macro key words such as “All” for inputReference field. For example, all series available that meet the defined criterion, are searched. The input is expanded into several concrete inputs. For example, the layout input contains a macro “All” in the inputReference field: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 &lt;inputs&gt; 
               
               
                   
                   &lt;Characteristics&gt; 
               
               
                   
                     &lt;inputReference&gt;All&lt;/inputReference&gt; 
               
               
                   
                     &lt;modality&gt;PET&lt;/modality&gt; 
               
               
                   
                   &lt;/Characteristics&gt; 
               
               
                   
                 &lt;/inputs&gt; 
               
               
                   
                   
               
             
          
         
       
     
         [0101]    Assuming there are three series available, among which two series are PET image data and one series is SPECT image data. The input “All” is expanded into two input definitions, including “Input 1 ” for PET modality and “Input 2 ” for SPECT modality: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 &lt;inputs&gt; 
               
               
                   
                   &lt;Characteristics&gt; 
               
               
                   
                     &lt;inputReference&gt;Input1&lt;/inputReference&gt; 
               
               
                   
                     &lt;modality&gt;PET&lt;/modality&gt; 
               
               
                   
                   &lt;/Characteristics&gt; 
               
               
                   
                   &lt;Characteristics&gt; 
               
               
                   
                     &lt;inputReference&gt;Input2&lt;/inputReference&gt; 
               
               
                   
                     &lt;modality&gt;SPECT&lt;/modality&gt; 
               
               
                   
                   &lt;/Characteristics&gt; 
               
               
                   
                 &lt;/inputs&gt; 
               
               
                   
                   
               
             
          
         
       
     
         [0102]    After macro expansion, each series are assigned  270  to the layout input which characteristics matches the series. 
         [0103]    In step  272 , viewer macros are expanded. Generally, every viewer can have a maximum of two inputs. Each input can potentially have macros such as “All” for inputReference. A viewer&#39;s input characteristics defines a query. If there is more than one series that meet the criterion for one viewer input, the extra viewer(s) are spawned. For example, the extra viewers are tabbed together in the same area. The scope of the search is limited within all series the layout has been assigned to. 
         [0104]    In step  274 , viewer inputs are synched with layout inputs. This step makes sure that the viewer&#39;s input is assigned with the series. 
         [0105]    Once a layout definition is fully populated, a complete display protocol  278  is built  280  from the layout definition. The full display protocol describes what components are needed and how the components need to be connected. 
         [0106]    In step  282 , a node map is registered. Since data views (packed image data for viewers to display) are provided by application data model, this step helps to label them with proper names so that they can be connected properly. 
         [0107]    In step  284 , data view dimensions are matched for a possible fusion application. For example, when one viewer gets assigned two data views with different number of dimensions, this step will expand the data view with fewer dimensions to match the one with more dimensions. For example, a viewer has a CT data as its underlay, which is one-dimensional, e.g. it contains only one volume, and has a PET data as its overlay, which is two-dimensional, e.g. meaning it contains multiple volumes at the same location but from different times. The CT data is expanded to have the same number of volumes as the PET data by “duplicating” itself. No actual pixel data is duplicated but only the reference points are added. E.g., the user is allowed to navigate through all dimensions of a multiple dimensional image data and to fuse it with other modality data with or without the same dimensions. 
         [0108]    In step  288 , a viewing protocol is executed. A protocol engine  290  uses the complete display protocol  278  to instantiate all software objects and connect them to establish the viewing network. 
         [0109]    In step  292 , a viewing network is registered with a link controller  294  which automatically links viewers. Although the protocol engine  290  establishes an initial viewing network, in a fusion display, for example, color map links need to be changed based on each viewer&#39;s fusion state. The link controller  294  acts as a manager and administers the entire linking behavior, e.g. adding/removing links More specifically, the link controller  294  automatically links viewers to the associated image data based on a set of rules  300  including rules for a color map linking  302 , a window level linking  304 , triangulation  306  and a zoom/pan presentation linking  308  as described below. 
         [0110]    Examples of rules for a color map linking  302 :
       If same group (in case of comparison)
           If common modality
               If non-fused viewer, or same pin if fused viewer
                   Same color stripe (lookup table)   
                   
               
               
 
         [0115]    Example of rules for a window level linking  304 :
       If same group (in case of comparison)
           If common series
               Same upper/lower, window/center   
               
               
 
         [0119]    Example of rules for triangulation  306 :
       If same group (in case of comparison)
           Triangulation of all TSC views   
               
 
         [0122]    Examples of rules for zoom/pan linking  308 :
       If same group (in case of comparison)
           If common orientation (transverse, sagittal or coronal)
               Same zoom and pan   
               
               
 
         [0126]    In this manner, the viewers are automatically linked to the image data associated with the viewers based on a set of built-in linking rules for all layouts. The rules are automatically enforced even when some viewers are destroyed or series is replaced until the user specifically makes a change. 
         [0127]    In step  309 , the layout manager  94  applies application specific actions in an orderly manner. 
         [0128]    With continuing reference to  FIGS. 2 and 10 , for easy switching of layouts, the layout manager  94  preserves  310  the viewing context, including preservation of a color map  312 , window level  314 , hot spot  316 , zoom/pan factors  318 , image index offset  320 , start position  322  and free dimension  324 . Each viewing context is stored under a certain key, which is generated based on certain set of rules (in some cases, these are also the rules used to generate across viewer links) After layout is switched, the keys for every viewer and data views attached to it are re-generated. Using the keys, the viewing context is fetched from a common hash table. If such viewing context exists for a given key, it should be reapplied to the viewer and data view from which the key is generated. Of course, it is contemplated that the other features can be preserved such as region of interest (ROI), labels, tags, markers, and the like. 
         [0129]    To preserve the color map  312 , a color map of each data view is retrieved and saved in a hash table under a unique key string which uses namespace naming convention. The color map will be transferred to next layout&#39;s data view as long as both layouts share the same group index, the same modality and the same pin (for fused case). 
         [0130]    For example, for a fused case, a unique key string is: 
         [0131]    Identifier.Modality.Pin.GroupIndex,
   where Identifier is “Colormap”;   Modality is one of the following key words “PET”, “CT”, “MR” or “SPECT”;   Pin means “underlay” or “overlay” depending on where the data view is attached; and   GroupIndex indicates which group the color map is pulled from in case of group comparison (for non-comparison layout, the value is  0 ).   
 
         [0136]    As another example, for a non-fused case, a unique key string is: 
         [0137]    Identifier.Modality.GroupIndex,
   where Identifier is “Colormap”;   Modality is one of the following key words “PET”, “CT”, “MR” or “SPECT”; and   GroupIndex indicates which group the color map is pulled from in case of group comparison (for non-comparison layout, the value is 0).   
 
         [0141]    The window level is preserved  314  by using a following exemplary key string: 
         [0142]    Identifier.SeriesHashCode.RotationMIP.GroupIndex,
   where Identifier is “WindowLevel”;   SeriesHashCode is the hash code of the series; and   RotationMIP is a Boolean indicating if the window level is pulled from a MIP viewer (window level is not linked to/from an MIP viewer.)   
 
         [0146]    Since there is only one hot spot per group, the hot spot is preserved  316  by using an exemplary key string: 
         [0147]    Identifier.GroupIndex,
   where Identifier is “Geo”.   
 
         [0149]    Zoom and Pan factors are kept in displayed area window. Only displayed area window needs to be preserved. Zoom and Pan Factors are preserved  318  by using an exemplary key string: 
         [0150]    Identifier.SeriesHashCode.Orientation.ViewerType.Groupindex,
   where Identifier is “ZoomPan”;   SeriesHashCode is the hash code of the series;   Orientation is image orientation specified in viewer configuration; and   ViewerType is viewer type that is also specified in viewer configuration object.   
 
         [0155]    The image index offset, which is an attribute of viewer&#39;s control model, is preserved  320  by using an exemplary key string for a non fused case: 
         [0156]    Identifier.SeriesHashCode.Orientation.ViewerType.GroupIndex,
   where Identifier is “IndexOffset”;   SeriesHashCode is the hash code of the series;   Orientation is image orientation specified in viewer configuration; and   ViewerType is viewer type that is also specified in viewer configuration object.   
 
         [0161]    For a fused case, the image index offset is preserved  320  by using another exemplary key string: 
         [0162]    Identifier.UnderlaySeriesHashCode.OverlaySeriesHashCode.Orientation.ViewerType.Gro upindex,
   where Identifier is “IndexOffset”;   SeriesHashCode is the hash code of the series;   Orientation is image orientation specified in viewer configuration; and   ViewerType is viewer type that is also specified in viewer configuration object.   
 
         [0167]    The start position is preserved  322  by using the same key as used to preserve the image index offset. The start position is an attribute of indexer control model which itself is a part of viewer&#39;s control model. 
         [0168]    In this manner, viewing context is automatically and intelligently preserved when switching layouts. Viewing Context includes color maps for modalities of series being reviewed, window levels for series being reviewed, zoom/pan factor for different image orientations that are displayed, lesions that are located by the user, image index, and others. 
         [0169]    Free dimension is preserved  324  by using the same key as used to preserve the image index offset. The free dimension is an attribute of indexer control model which itself is a part of viewer&#39;s control model. 
         [0170]    With continuing reference to  FIG. 2 , a fusion controller  326  controls display of fused images via a toggle switch  328 . Generally, images are frequently displayed in a non-fused mode with a demand to occasionally review images in the fused fashion. In one embodiment, all viewers are defined as fused viewers, e.g. in the non-fused mode, the viewer status is changed to a non-fused. The invisible data view is deactivated so that it will not be affected by any changes occurred in the same viewer or other viewers. If the viewer status is changed to a fused mode, the link controller  294  first disconnects the viewer and its image data from its current color map link, and then looks for a suitable color map link which takes care of fused viewers. If such link exists, the link controller includes the new fused viewer along with its data views into the link. If such link does not exist, the link controller creates such a viewer. A similar process happens when a fusion is turned off. 
         [0171]    Different set of templates, that define what image information is and how it should be displayed along with image pixel data, is used to handle image information annotation. When the fusion is turned on, a fused annotation template is used. When the fusion is turned off, depending on which data view is visible, a corresponding non-fused annotation template is used. To avoid confusion, an ROI measurement reflects one image at a time. When the fusion status is changed, an ROI association is changed accordingly. 
         [0172]    In this manner, the single toggle  328  is provided to turn fusion on/off in any individual viewer within a layout with annotation, region of interest measurements and linking behavior automatically adjusted to the change of the fusion status so that consistent application behavior is maintained. 
         [0173]    With reference to  FIGS. 14 and 15 , the users are allowed to modify existing display and save the modified display as a new layout. For example, the user inspects the current display  330  and moves image  332  into position  334 . All viewers that are on screen are searched. Their position, size, fusion state, zoom factors, and monitor information are fetched. Layout inputs are populated when an associated layout is processed. 
         [0174]    At runtime, layout inputs are populated and they can also be replaced. A fully initialized layout contains no macros. For example, a layout has two inputs which are defined as: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 &lt;inputs&gt; 
               
               
                   
                 &lt;Characteristics&gt; 
               
               
                   
                     &lt;inputReference&gt;Input1&lt;/inputReference&gt; 
               
               
                   
                     &lt;modality&gt;Anatomical&lt;/modality&gt; 
               
               
                   
                   &lt;/Characteristics&gt; 
               
               
                   
                   &lt;Characteristics&gt; 
               
               
                   
                     &lt;inputReference&gt;Input2&lt;/inputReference&gt; 
               
               
                   
                     &lt;modality&gt;FunctionalAnatomical&lt;/modality&gt; 
               
               
                   
                   &lt;/Characteristics&gt; 
               
               
                   
                 &lt;/inputs&gt; 
               
               
                   
                   
               
             
          
         
       
     
         [0175]    After the layout is fully populated, the inputs of its working copy become, for example: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 &lt;inputs&gt; 
               
               
                   
                 &lt;Characteristics&gt; 
               
               
                   
                     &lt;inputReference&gt;Input1&lt;/inputReference&gt; 
               
               
                   
                     &lt;modality&gt;MR&lt;/modality&gt; 
               
               
                   
                   &lt;/Characteristics&gt; 
               
               
                   
                   &lt;Characteristics&gt; 
               
               
                   
                     &lt;inputReference&gt;Input2&lt;/inputReference&gt; 
               
               
                   
                     &lt;modality&gt;CT&lt;/modality&gt; 
               
               
                   
                   &lt;/Characteristics&gt; 
               
               
                   
                 &lt;/inputs&gt; 
               
               
                   
                   
               
             
          
         
       
     
         [0176]    The same level of modality specificity that the original layout supports is maintained when the layout is modified and saved. From the example above, when such layout is loaded, only MR &amp; CT can be handled. To overcome this, the following scheme can be used: 
         [0177]    Go through each input
       If the populated input supports a subset of data range that the original input does
           Then the original input will be used as the new layout input definition Otherwise   Promote the populated input to the same modality specificity as the original   
               
 
         [0181]    With reference to  FIG. 16 , a layout icon  344  is created on the fly. A screen capture of an entire image display area is performed. A thumbnail out of the capture is made and saved as a part of the new layout&#39;s icon. 
         [0182]    With reference to  FIG. 17 , the users handle the display  350  of the oblique images. In step  352 , an oblique angle is detected and measured. When a volume is reformatted, two pieces of oblique information are stored in DICOM file: (1) row/column vectors, and (2) view code sequence, from which the original oblique transformation can be uniquely constructed. Once the original oblique transformation is constructed, the original oblique rotation angles or the rotation of each axis from its original direction can be determined 
         [0183]    In step  354 , user sets an oblique TSC display state based on the oblique angle. When a volume is loaded, the user requests permission from the application to view the images at the oblique TSC instead of the normal TSC. The permission to use the oblique display is based on rules. The rules, for example, are based on the oblique angle measurement, the existence of the oblique angle in one of the images, and the layout configuration. For example, the permission is granted if the oblique angle is from about 0° to about 42.5°. If the volume is not tilted, the permission is denied. In step  356 , the user controls TSC display based on different oblique angles. The user can set any series as reference, so the actual oblique angle is derived from the reference image. For example, a first series has a 10° rotation along axial axis and a second series has a 20° rotation along axial axis. The user can display or fuse the first and second series by aligning the first or second series with either series. The series can be displayed in non-oblique and/or oblique fashion. 
         [0184]    In this manner, users can edit a currently displayed layout by re-positioning viewers (components that display images on screen), resizing viewers, changing fusion status of viewers, adjust zoom/pan factor of viewers, destroying viewers, changing viewing series, etc, and caching it in memory and saving it. This makes creating a viewing network with desired initializations, which used to be the task of software developers, as easy as saving a modified word processing document. 
         [0185]    The invention has been described with reference to the preferred embodiments. Modifications and alterations may occur to others upon reading and understanding the preceding detailed description. It is intended that the invention be constructed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.