Patent Publication Number: US-6708184-B2

Title: Method and apparatus for producing and accessing composite data using a device having a distributed communication controller interface

Description:
RELATED APPLICATIONS 
     The application is a continuation-in-part of U.S. patent application Ser. No. 09/382,594, filed Aug. 25, 1999 and issued as U.S. Pat. No. 6,253,210 on Jun. 26. 2001, which is herein incorporated by reference in its entirety, which is a continuation of U.S. patent application Ser. No. 08/832,688 filed on Apr. 11, 1997 and issued as U.S. Pat. No. 5,970,499 on Oct. 19, 1999. This continuation-in-part application also claims priority to U.S. Provisional Patent Application No. 60/135,057 filed on May 20, 1999, which is also herein incorporated by reference in its entirety. 
    
    
     BACKGROUND OF THE INVENTION 
     The present invention relates to information systems and methods, and more particularly to data fusion systems. 
     Many applications benefit from the contemporaneous assimilation of large amounts of data. In medical, military, and commercial applications, operators engage in procedures and make decisions based on data describing various subjects represented by, for example, images, recorded sound, and text. Current technology has limitations presenting personnel with a unified view of this subject data to allow them to use all available data to make informed decisions. 
     For example, a physician providing medical treatment reviews image data acquired in multiple modalities, such as magnetic resonance (“MR”), computed tomographic (“CT”), and X-ray images, medical journals describing procedures, video images, such as ultrasound, and atlases describing anatomical structures. A physician therefore consults several sources to review the data necessary to provide patient treatment. These sources may include multiple computer display terminals located in different parts of a hospital, hard copies of medical images printed on film archived among thousands of images in a hospital film library or remote storage site, and volumes of journals located in the stacks of a hospital library. Also, the sources of data consulted by treating physicians may include medical atlases containing thousands of MR and CT scans of a cadaver corresponding to photographic images of cross-sectional slices taken of various anatomical structures. 
     Usually data from these atlases and other sources are not correlated with each other. A cadaver image in an atlas does not usually have the same geometry as a patient receiving treatment, so a physician must mentally fuse the available data which requires correlating the data retrieved from the various sources to develop a treatment plan or to provide information during medical procedures. The difficulties of fusing available data increase if the physician must assimilate the various data types while rendering treatment. 
     The World Wide Web (“WWW”) has recently made vast amounts of data stored on local and remote computers easily accessible through a graphical computer interface. The WWW is a network of computers, connected by the Internet, sharing a common file structure and mark-up language for creating files. The two most prevalent languages used to create multimedia WWW files are the hypertext mark-up language (“HTML”) and the virtual reality mark-up language (“VRML”). HTML is best suited for creating files with text and two-dimensional image data, whereas VRML is designed for creating files containing images of three-dimensional objects. Both languages provide an easy way to combine image, text, and sound data in files accessible by “point-and-click,” computer mouse driven user interfaces called “browsers.” 
     A “browser” is a computer program that provides users access to files stored on the WWW. The browser displays files on a computer screen and can run programs, known as “applets,” indicating links to data in other files on the WWW by, for example, underlining text or highlighting areas of an image. By selecting the underlined text or a highlighted image, the browser retrieves the linked data, allowing a user to view data stored on computers in the WWW without needing to know where the information is physically stored. Files can be joined using these “hyperlinks,” which give the name of the file along with an address for a computer storing the file. For example, the text or an image in a file stored on a computer in Switzerland can contain an embedded link to data stored on a computer in the United States. The WWW browser automatically recognizes the linked file data type, so the linked file can be an image, an audio clip, a video, or even an executable computer program. For example, if the linked data is an audio clip, the browser will load a program that takes the audio clip and plays it through the speakers of the user&#39;s computer. A browser usually runs on a computer referred to as a “client,” while a computer known as a “server” hosts and produces WWW files requested by a client. 
     In particular, the WWW serves as a useful tool for navigating through two- and three-dimensional image data. For example, an image can be displayed by the browser, and different parts of the image can be linked to different files. But, for the most part, this WWW capability is primarily used for providing simple menus of uncorrelated data available on WWW computers. For example, a WWW computer will show an image of people, cars, and boats. By clicking on the image of people, a user can go to on-line chat sessions with people, or by clicking on a boat image, a user gets information about boats. 
     The current technology is limited because there does not exist an information system that exploits the data navigation capabilities of the WWW to correlate data retrieved from diverse sources and then assimilate the data into a useful form. For example, the tools available for information gathering in the WWW environment include database search engines and expert systems that assist a user in describing the information sought. However, these tools only retrieve files corresponding to a particular term or pertaining to certain designated subject matter. The retrieved files are not correlated with one another. 
     There is, therefore, a need for an information system that harnesses the power of the technology associated with the WWW and other similar image-based information retrieval systems to produce assimilated composite data in a form that operators can readily use. 
     SUMMARY OF THE INVENTION 
     The present invention is directed to a method and apparatus for producing and accessing composite data containing co-registered and subject data. Co-registered data is generated, for example, by registering data to a common coordinate system. The method for automatically producing composite data includes several steps, performed by a server computer. The steps include: creating a mapping relationship between the co-registered data and the subject data by mapping or deforming a template to fit the subject data; filtering the co-registered data; and producing composite data by mapping the filtered co-registered data to the subject data according to the mapping relationship. 
     A method consistent with this invention is also directed to steps, performed in a client computer, including: requesting composite data from a server computer; transmitting the subject data to the server computer; receiving the requested composite data from the server computer; presenting the received composite data to an operator; and monitoring the operator&#39;s use of composite data. 
     An apparatus consistent with this invention for automatically producing composite data containing co-registered data and subject data includes: structure for creating a mapping relationship between the co-registered data and the subject data by mapping or deforming a template to fit the subject data; structure for filtering the co-registered data; and structure for producing composite data by mapping the filtered co-registered data to the subject data according to the mapping relationship. 
     Another apparatus consistent with the present invention automatically presents an operator with composite data containing co-registered data and subject data. Such an apparatus includes: structure for requesting composite data from a server computer; structure for transmitting the subject data to the server computer; structure for receiving the requested composite data from the server computer; structure for presenting the received composite data to an operator; and structure for monitoring the operator&#39;s use of the received composite data. 
     Both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. 
    
    
     DESCRIPTION OF THE FIGURES 
     The accompanying drawings provide a further understanding of the invention. They illustrate embodiments of the invention and, together with the description, explain the principles of the invention. 
     FIG. 1 is a block diagram of an apparatus for producing composite data containing co-registered data and subject data consistent with the present invention; 
     FIG. 2 is a block diagram of another apparatus for producing composite medical data containing co-registered medical data and patient data consistent with the present invention; 
     FIG. 3 is flow diagram of a method for producing composite data containing co-registered data and subject data consistent with the present invention; 
     FIG. 4 is a schematic diagram of user interaction with an embodiment of the present invention consistent with the block diagram of FIG. 2; 
     FIG. 5 is an illustration of co-registered medical data used in an embodiment of the present invention consistent with the block diagram of FIG. 2; 
     FIG. 6 is a display produced in accordance with an embodiment of the present invention consistent with the block diagram of FIG. 2; and 
     FIG. 7 is a block diagram of a facility for providing composite data in accordance with an embodiment of the present invention. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Reference will now be made in detail to the embodiments consistent with the present invention, examples of which are illustrated in the accompanying drawings. 
     To illustrate the principles of this invention, FIG. 1 shows subject data  100 , template layer  102 , deformation engine  104 , mapping engine  106 , map  107 , filtered, co-registered data  108 , search and filter engine  110 , links  112 , and co-registered databases  114 . The term “layer” denotes a grouping of data types represented in, for example, a database on a server computer. To produce composite data  109 , deformation engine  104  deforms a template layer  102  to fit subject data  100  generating map  107 . When subject data  100  is multi-modal, multiple template layers can be utilized to correlate co-registered data  114  to subject data  100  as well as to correlate multi-modal subject data  100  to itself. A template layer contains reference data locations, or landmarks, used to correlate template data with elements of subject data. Examples of landmarks include data representing points, lines, surfaces, volumes, or other defining features in image data. 
     Generally, deformation is the process of mapping one image to another image where both images represent related structure but generally have different geometric proportions and orientations. During deformation, mathematical transforms are applied to the images that may perform the equivalent of bending, stretching, and/or rotating template data to match subject data. For example, after deformation, template data in the form of a volume image of a generalized model of the human brain is manipulated to relate the size, shape, and orientation of the anatomical structure in this model to the subject data, the actual anatomy of a patient receiving treatment. There are many techniques available for deforming one set of data to fit a target data set, including rule-based morphology, correlation of selected landmarks in each data set, and a technique fusing selected landmarks and image data. One example of such technique appears in U.S. Pat. No. 6,009,212 to Miller et al., which is herein incorporated by reference. Similar techniques are also suitable for generating co-registered data consistent with the present invention. 
     Once template layer  102  is deformed to fit subject data  100 , a mapping relationship, map  107 , is established whereby mapping engine  106  maps co-registered data  108  to subject data  100  producing composite data  109 . Co-registered data  108  represents a knowledge base providing supplemental information about structures contained in subject data  100 . The co-registered databases  114  and template layer  102  share a common coordinate system, so a data element representing a position in one co-registered database  114  is correlated with a data element representing that same position in each of the other co-registered databases  114 . In an embodiment consistent with the present invention, co-registered data  108  is co-registered using, for example, the deformation techniques described above. The mapping relationship obtained from deforming template layer  102  to fit subject data  100  correlates the co-registered database coordinate system with a subject data coordinate system. Implementing this mapping relationship, mapping engine  106  relates points in the subject data coordinate system to corresponding points in the co-registered database coordinate system, providing a dynamic connection between subject data  100  and co-registered databases  114 . 
     Search and filter engine  110  controls which elements of co-registered data  108  are mapped to subject data  100  and presented to an operator. Search and filter engine  110  can allow mapping engine  106  to map all or a subset of co-registered data  108  to subject data  100 . Links  112  specify relationships among data elements across co-registered databases  114  which are used by search and filter engine  110  to assimilate co-registered data  108  according to a service request by an operator including, for example, an indication of a region or regions of interest in subject data  100 . The links  112  may be formed using an appropriate database indexing strategy to assign key-word or concept search tags to associated data elements. 
     FIG. 2 shows a preferred embodiment of the present invention for producing composite patient data for medical treatment. Collected patient data  200 , includes, for example, MR image  236 , CT image  238 , and landmarks  240 . Template layer  202  includes corresponding MR image  230 , CT image  232 , and landmark  234  templates. This embodiment also includes a deformation engine  204 , mapping engine  206 , search and filter engine  210 , links  212 , and co-registered data database  214 . 
     Co-registered database  214  and associated co-registered data  208  includes a medical atlas  228 , text  226 , computer programs  225  (such as applets), labels and landmarks  224 , images  222 , audio and video clips  220 , links to computers located on the WWW  218 , and treatment plans  216 . Co-registered data  208  can also include co-registered subject data. The forgoing list of co-registered data types is only provided as an example of the types of data that are useful in practicing this invention in the context of providing medical treatment. Persons of ordinary skill will recognize that many other data types may also be useful. One of ordinary skill in the art will also recognize that two or more of the co-registered data types may reside in a single database. 
     FIG. 3 is a flow diagram of a method for producing composite medical data consistent with the invention. An operator of a system consistent with this invention wishing to have composite data for medical treatment first identifies and possibly collects patient data  200  using a client computer  245  (step  300 ). The collected patient data  200  must have a modality (e.g., CT, MR, or X-ray) and protocol (e.g., slice thickness and resolution) that is compatible with a template in template layer  202 . 
     The client computer  245  then generates request  246  for composite data  244  (step  304 ). Request  246  includes, for example, an operator identifier, security screening information, and treatment information. The client computer  245  also transmits or enables transmission of (from a radiological database, for example) collected patient data  200  and associated filtering context  242  to the server computer  243  (step  306 ). 
     Responding to client computer request  246  for composite data  244 , the server computer  243  selects a template  202  conforming with this request (step  308 ). Example template  202 , MR  230 , employed by the present invention when used for medical treatment includes a three-dimensional MR scan of the same slice thickness, resolution, and collection protocol as the patient MR dataset  236 . Associated with the selected template imagery is a set of landmarks  234  identifying anatomical structures in a region of interest for a particular surgical procedure. Next, deformation engine  204  fits the selected template  202  to patient data  200  received from the client computer  243 . The process of deforming the selected template  202  to fit the patient data  200  creates a mapping relationship (map  248 ) relating template data space to a patient data space coordinate system (step  310 ). Mapping engine  206  also uses map  248  to relate a point in the patient data space coordinate system to an element of co-registered data  214 . Once the mapping relationship is determined by deforming the selected template  202  all co-registered data  214  can be mapped to patient data  200 . Note that if multi-modal patient data  200  is used with multiple corresponding templates  202 , multiple maps  248  can be constructed that can then be used to correlate the multi-modal patient data with each other in addition to correlating co-registered data  208 . 
     Search and filter engine  210  controls how much of co-registered data  208  is included in the composite data  244 . One reason for such control is that certain data types and/or data elements in co-registered data  208  may not be relevant for some medical treatment. Search and filter engine  210  responds to filtering context  242  and selects co-registered data elements as appropriate for this filtering context  242  (step  312 ) using links  212  to identify related data elements. The filtering context  242  can be derived from data provided by the client computer  245  during the initial request for composite data  244  (step  304 ). A filtering context  242  can also be derived from previously stored profiles or histories. The server  243  then produces composite data  244  using mapping engine  206  and map  248  to combine patient data  200  and filtered, co-registered data  208  (step  314 ), producing composite data  244 . The server then transmits composite data  244  and map  248  to the client computer  245  for presentation to an operator (step  318 ). The operator navigates the composite data  244  by specifying a region of interest in the patient data  200  using a browser interface (step  320 ). The operator may also use the browser interface to select highlighted text or other specified segments of the composite data  244  activating a link to a particular region of interest in the patient data  200 . 
     Map  248  also allows an operator to access additional composite data  244  (step  322 ). The server  243  receives a request for additional co-registered data  208  and preferably an associated position in the patient data coordinates from the client computer  245  and subsequently retrieves and transmits additional co-registered data  208  using mapping engine  206 , map  248 , search and filter engine  210 , and links  212  to additional co-registered databases  214 . (repetition of steps  312 - 320 ). 
     An embodiment consistent with the present invention for use in the medical field links a radiologist&#39;s report to radiological imagery of a patient. Preferably, in an area of the radiologist&#39;s text report stating, for example, “in the left parietal-occipital region is a 1.2 cm hypodense lesion with an irregular border that does not enhance on contrast but is hyperintense on T 2  . . . ”, selecting the highlighted word “lesion” activates a link to the patient imagery which highlights the particular sub-region discussed in the report text. Likewise, if the operator selects the lesion site in the displayed patient imagery, the link will be activated to display the section or sections of the text report that discuss the selected region of interest. 
     Although the foregoing description of embodiments of the present invention specifically allocate certain operations to client  245  and server  243  computers, one of ordinary skill in the art will recognize that the distribution of specific tasks between client  245  and server  243  computers can vary based on application requirements. Moreover, embodiments of the present invention with several client  245  or server computers are within the scope of this invention. Furthermore, it is also consistent with the present invention that the client  245  and server  243  tasks can be performed in a single computer. 
     In an embodiment consistent with the present invention, a graphical user interface designed for browsing data presents composite data  244  to the operator. The interface can be executed on networked computers. Computer program code that can be adapted to perform this browsing function includes Internet browsers designed to navigate the WWW, such as Netscape&#39;s Navigator and Microsoft&#39;s Explorer, and equivalent programs that support links to data stored on networked computers. 
     Communication among the devices and with the Internet is controlled by the surgeon or other staff within the operating room using the web-like interface or browser. Thus, operating room staff have control over information allowed into and out of the operating room by the switch, to insure patient privacy and security. To provide this functionality, an embodiment consistent with the present invention connects devices within the operating room and can also communicate bidirectionally with the world beyond the operating room. Since devices in the operating room commonly generate large data streams, one implementation of the present invention utilizes a broadband network within the operating room. Since an implementation of the network infrastructure of the present invention can facilitate communication among information servers in addition to controlling devices such as robots which demand accurate timing, each operating room&#39;s network should be able to be isolated from stray network traffic that could interfere with communications within the operating room. An architecture suitable for providing such a networking infrastructure is described in U.S. Provisional Patent Application, Ser. No. 60/135,057 filed on May 20, 1999, by Richard D. Bucholz, entitled “Networking Infrastructure for an Operating Room,” which is incorporated by reference herein in its entirety. 
     Finally, networks as presently conceived tend to be static constructs, such as desktop computers connected in an office. This is often at odds with the work flow of an operating room. Rather than connecting a number of devices which stay connected for long periods, the operating room is continually in flux. Networked devices may be present for only a portion of a particular procedure, and the preferences of the surgeon and the demands of the procedure dictate which devices are employed. Therefore, an embodiment consistent with the present invention contemplates simplified connections wherein the network or the device initiate communications automatically and promptly upon connection. In addition, operating system(s) consistent with the present invention can tolerate disconnection without serious incident. Since these systems are, in many instances, life support devices, embodiments consistent with the present invention contemplate the components of the system operating and being controlled despite connection or disconnection of a particular device from the network. In addition, components of the system operate and are controlled whether networked or not. In short, the connections are robust and fault tolerant. 
     An embodiment consistent with of the present invention employs the Jini networking protocol (as developed by Sun Microsystems). The Jini network protocol allows a Jini compatible device to make and break network connections upon connection of the device to the network. Further, communications established in a Jini compatible network allow prompt sharing of information between and control of devices after connection. This control of networked devices is orchestrated through standard Internet and web technology such as hypertext transfer protocol (e.g., http over TCP/IP). 
     The suitability of the Jini networking protocol is made more apparent when the inherent organization of the operating room is taken into account. Hospitals often have operating suites with a number of separate operating rooms, each with substantial autonomy. Therefore, establishing a network within each operating room allows control within each room, and multiple operating rooms may be linked through an intelligent switch in each operating room so that selective bidirectional communications can occur between the linked operating rooms. In addition, an embodiment of the present invention facilitates selective bidirectional communication with the Internet. 
     By controlling each operating room&#39;s switch from within the operating room, all devices in the operating room can communicate with each other using a broad bandwidth network, and extraneous Internet network traffic can be selectively prevented from entering the room. Thus, the surgeon can exercise control of devices within the operating room and secure patient information, while gaining access to the Internet. Of course, embodiments consistent the present invention also contemplates compatibility with and the utilization of other network protocols. 
     According to an embodiment consistent with the present invention, networked devices in the operating room can have a distributed controller that is Jini compliant and capable of communication using standard Jini communication protocols over a local-area or wide-area network. For example, server  243  and client  245  in FIG. 2 could be equipped with a distributed communication controller interface using the Jini communication protocols. Devices are controlled locally using their own distributed controller that drives a display device that is also attached to the network. A client is modified by the addition of a controller that interfaces with the network and a touch sensitive flat panel display. The distributed controller has software for a “minibrowser” (scaled-down browser) which may be saved in read-only memory (ROM) along with control forms written in html. A control form is displayed on the display device upon startup of the client, and consists of virtual buttons that are actuated by touch. When the user touches a button to request a desired task, the browser activates the controller through an interface so that the controller controls the client to perform the desired task. Therefore, local control of the client through a user-friendly interface is achieved using a browser in the absence of any communication between the client and the network. Since this embodiment of the present invention uses a browser (a web-like interface), the control language of the device can be changed easily, and can have a variable complexity determined by the user. For example, a display for a nurse may differ from the surgeon&#39;s display allowing each different control capabilities. 
     The presence of the web-like interface enables remote control of the device over the network. Since the device is Jini compliant, it has, among other things, a Jini compliant controller. Therefore, communications within the operating room are established automatically upon plugging the device into the network. The web-like interface allows the device to be controlled by other devices in the operating room. When two devices are connected by the network, the display of each device is programmed to display the control form of all connected devices. For example, if an MR machine is plugged into the network along with the client, the browser of the MR machine will display the fact that other control forms are available to it over the network by displaying buttons for each controllable device. By pressing the button marked client, the MR machine&#39;s browser will display the control form for the client, and all functions of the client can be manipulated through the control form as displayed on the MR machine&#39;s browser. This bidirectional communication is established by plugging the device into a network jack located in the operating room, as orchestrated by the Jini network protocol and the device&#39;s embedded Jini-compliant controller. According to this embodiment, the functions embedded in the control form are html compliant and can therefore be of any form. 
     The present invention employs a wired local-area or wide-area network, or may alternatively employ a wireless, infrared, or other suitable network as long as the network has a bandwidth capable of transmitting the appropriate data streams. For a simple operation, infrared communication may be adequate. Alternatively, control of a surgical robot requires a network that is robust and resistant to noise, making presently available wireless networks inappropriate. Further, presently available wireless networks may allow crosstalk between operating rooms, creating potentially severe control problems. 
     According to another embodiment consistent with present the invention, the devices have a controller that is Jini compliant, but the devices are connected to each other rather than to a local-area or wide-area network. In this embodiment, the Jini protocol allows communications to be established between two devices without using a network. However, many procedures require more than two devices, and therefore a device allowing multiple connections is needed. If no communication with a network outside of the operating room is desired, then a repeater can be used to create the multiple connections when more than two devices are connected. 
     The present invention further contemplates use of the network infrastructure with a StealthStation as disclosed in U.S. Pat. Nos. 5,383,454, 5,871,445, 5,891,034 and 5,851,183, and International Publication Nos. WO 94/24933 and WO 96/11624, which are incorporated herein by reference. In the StealthStation embodiment, instead of one cart holding all equipment for the StealthStation, the StealthStation consists of two stand-alone modules. 
     A first module is a display unit having a high resolution touch panel on a pole extending from an electronics cluster located on casters. The network switch is located in the electronics cluster, along with the computer for a navigational system. A network jack panel is also located in the electronics cluster, and provides a connection to the Internet. An Internet connection is provided in each operating room, such as in the form of a telephone jack with a modem. 
     In the second module of the StealthStation embodiment of the present invention, at least one camera is attached to a long arm connected to an electronics cluster located on casters. The camera communicates with the display system through the network. Therefore, a network cable extends from the camera electronics cluster to the network switch in the display module, and another cable extends from the display module to the Internet wall jack. Any other devices used with the network are connected to the network switch located in the base of the display unit. Thus, the StealthStation display unit is the hub of the operating room network. Alternatively, for example, the network switch can be wall-mounted in the operating room so that the StealthStation need not contain the network switch. 
     Notably, new technology can be incorporated easily into the system by making the new technology Jini compliant. For example, a robot can be controlled by the networked system if its control mechanisms were programmed to accept an interface consistent with the present invention, such as, for example, the Jini interface standard. New display or control devices, ultrasound devices, or fluoroscopes can connect to the network and transmit their images, and be controlled, by other devices within the operating room. In this way, the network infrastructure of the present invention makes the StealthStation compatible with technological innovations, and fosters development of new technologies without need for reprogramming for each device. 
     FIG. 4 illustrates operator interaction associated with producing composite data in accordance with an embodiment of the present invention. FIG. 4 shows a database containing co-registered data  400  of several data types, including video  402 , text  404 , waveforms  406 , programs  407 , still images  408 , and segmentation labels  409 ; a mapping engine  410 ; a map  412 ; a set of links  414  among associated data elements across and within the co-registered databases; a patient data space coordinate system  416 ; and a universal (“atlas”) coordinate system  418  common to all data stored in co-registered database  400 . A physician using this embodiment of the present invention selects any point in patient data space coordinate system  416  to retrieve co-registered data  400  corresponding to the selected point. 
     In an example illustrating co-registered data in an embodiment of the present invention, still image database  408  contains MR images of a human head, database  406  contains recordings of waveforms produced by certain electrical signals in the brain, video database  402  contains recorded motion picture images of neurosurgical procedures or tutorials for these procedures, text database  404  contains short descriptive paragraphs or full journal articles describing regions of the brain and related surgical plans, database  407  contains programs for processing image data, and database  409  contains segmentation maps outlining brain structures. 
     The patient data space coordinate system  416  is a frame of reference for patient specific data. This coordinate system is provided by, for example, an MR of a patient&#39;s head or the surgical field surrounding the patient during operation. Deformation engine  204  computes a mapping relationship relating template layer data points in atlas coordinate system  418  to patient data points in patient data space coordinate system  416 . Mapping engine  410  uses this computed mapping relationship to transform co-registered data  400  mapped to atlas coordinate system  418  to patient data space coordinate system  416 . 
     After mapping, a physician has available during a surgical procedure composite data adapted to the patient&#39;s anatomy. This composite data is a representation of (1) a patient&#39;s anatomy comprising patient specific data acquired before or during a medical procedure, and (2) data from one or more of the co-registered databases  400 . 
     Map  412  provides a virtual grid overlaying the patient data space coordinate system  416  allowing an operator to position a pointing device in the patient data to retrieve co-registered data. Selecting a position in map  412  retrieves co-registered data correlated with the selected position by mapping engine  410  through links  414 . 
     In one embodiment of the invention, map  412  contains a number of positions corresponding to the number of positions in patient data space coordinate system  416  detectable by a surgical navigation system (see, e.g., U.S. Pat. No. 5,383,454). A map position is selected according to the location of a surgical probe in patient data space coordinate system  416  during a medical procedure. For example, during neurosurgery a surgeon placing the probe at a patient&#39;s ventricle activates a map position corresponding to the probe position in the ventricle. The activated map position is communicated to mapping engine  410  which queries co-registered databases  400  for data corresponding to the map position in the ventricle. This corresponding co-registered data is deformed to correlate to the patient&#39;s anatomy and combined with patient specific data giving the surgeon composite data related to the patient&#39;s ventricle, containing more information than the patient data alone. 
     A further illustration of the types of data a physician may require during neurosurgery or during surgical planning is shown in FIG.  5 . This figure contains illustrations of six data types available to a surgeon in an embodiment of the present invention including a cross-sectional image  502 , a medical journal article  504 , electroencephalograph waveforms  506 , computer programs  507 , video images of the brain  508 , and a segmentation map  510  identifying the different regions of the brain. Because these data sources have been co-registered to a common atlas coordinate system  418 , a point, such as point  500  in the cerebellum in brain image  502  from database  408 , has a corresponding point in each of the other data types in co-registered database  400 . For example, point  500  in text database  404  corresponds to article  504  on new surgical techniques involving the cerebellum. Point  500  in waveform database  406  corresponds to recorded waveforms  506  produced by the brain at this location. Point  500  in program database  407  corresponds to applet program  507 , which is used to provide enhanced visualization of brain image  502 . Point  500  in video database  402  corresponds to video clips  508  of the brain at this location. Point  500  in segmentation map database  409  corresponds to a point within segmentation map  510 . 
     Each of these examples of data need not be acquired from the patient currently receiving treatment. For example, the data may come from digital anatomical atlases, libraries of cadaver images, or research databases produced by projects such as the “Visible Human” research sponsored by the National Library of Medicine. Data available in a co-registered database would include, for example: 
     
       
         
           
               
               
             
               
                   
                   
               
             
            
               
                   
                 1. Anatomic 
               
               
                   
                 Magnetic Resonance Imaging 
               
               
                   
                 Computed Tomography 
               
               
                   
                 Magnetic Resonance Angiography 
               
               
                   
                 Ultra-Sound 
               
               
                   
                 Slice photographic images 
               
               
                   
                 Sulci/Gyri traces 
               
               
                   
                 2. Functional 
               
               
                   
                 Positron Emission Tomography 
               
               
                   
                 Single Photon Emission Computed Tomography 
               
               
                   
                 Functional Magnetic Resonance images 
               
               
                   
                 Electroencephalograph 
               
               
                   
                 Magnetoencephalography 
               
               
                   
                 3. Symbolic 
               
               
                   
                 Structure name 
               
               
                   
                 Structure size 
               
               
                   
                 Structure function 
               
               
                   
                 Structure related text cues 
               
               
                   
                 Structure related video cues 
               
               
                   
                 Structure related audio cues 
               
               
                   
                 Structure related labels 
               
               
                   
                 Histology 
               
               
                   
                 Morphological data 
               
               
                   
                 4. Multimedia 
               
               
                   
                 Video Footage of procedures 
               
               
                   
                 Training Videos 
               
               
                   
                 Conference, Journal Articles 
               
               
                   
                 Statistics 
               
               
                   
                 5. Computer Programs 
               
               
                   
                 Applets 
               
               
                   
                 Data Analysis 
               
               
                   
                 Automated Structural Segmentation 
               
               
                   
                 Image Enhancement and Visualization 
               
               
                   
                   
               
            
           
         
       
     
     These data need not be located in a single database. One of ordinary skill in the art will recognize that individual co-registered databases may be distributed among several databases accessible through, for example, local area computer networks or wide area computer networks connecting local and distributed computers. The combination of the different data components of the co-registered databases produces a generic data model of human anatomy where each data element is correlated through a common coordinate system with corresponding data elements of other types. When planning or providing medical treatment, the invention produces composite data linking co-registered data of a generic model to the specific anatomy of a patient receiving treatment. Examples of other embodiments consistent with the present invention for producing composite data for medical diagnosis, planning, and treatment include, but are not limited to, the following. 
     1. Diagnostic Radiology—co-registered patient magnetic resonance, X-ray, and/or computed tomography imagery are linked to text data such as radiologists&#39; reports, patient case history files, and relevant conference/journal articles. Sequential scans of a patient are co-registered for tracking the growth or reduction of lesions. Data analysis programs are linked to the composite data for computation of quantitative data measurements for planning and monitoring of treatment progress. Co-registered multi-modal patient image data and relevant co-registered data are presented in a common, easy-to-use presentation scheme. 
     2. Radiation Treatment Planning—Three-dimensional segmented atlases are mapped to patient data to produce an object-based model of lesions, targets, and major organs and other critical structures. The patient data with associated object information is utilized by a treatment planning program for computing optimized radiation delivery strategies from target and critical structure information. 
     3. Neurosurgical Targeting—cranial patient imagery is mapped to neurosurgical atlas information containing coordinates and shapes of surgical targets and surrounding neuroanatomic structures. Structural information is linked to audio files for use in-surgery with microrecording probes. Links to statistical databases provide information relating regions of interest to procedures and success rates. 
     FIG. 6 shows how one embodiment of the present invention presents composite data to an operator on a computer display having several windows. Window  604  contains patient data in the form of a three-dimensional MR brain scan. Windows  602 ,  606 , and  608  contain axial, sagittal, and coronal photographic section data, respectively, from the Visible Human data set, which are co-registered to the patient data by deformation engine  204 . Window  610  presents co-registered atlas data to an operator. By positioning cross-hairs  612  in Window  604  at a desired point  600 , the corresponding point  600  in each of the images in windows  602 ,  606 , and  608  is identified automatically by the location of cross-hairs in those windows. Here, for example, the operator selected point  600  in window  604  corresponding to a region of the brain known as the putamen. Atlas data containing views of the putamen are displayed in window  610  with highlighted pointers indicating the putamen in each of the atlas images. The operator can also choose to play a movie showing video images of the putamen by pushing “play movie” button  614 . Alternatively the operator may select the word “putamen” in window  610  and cross-hairs  612  will indicate the position of the putamen in patient data window  604  and the Visible Human “atlas” data windows  602 ,  606 ,  608 . 
     FIG. 7 is a block diagram of a facility consistent with the present invention for providing composite data across a computer network to customers under a service contract. In FIG. 7, solid lines indicate a path for both control and data flow and dotted lines indicate data flow only. Facility  700  is preferably connected to a wide area network, such as Internet  701 , through firewall  702 . Firewall  702  is a computer that monitors all data traffic into and out of facility  700  to prevent unauthorized access to the facility. World Wide Web page  704  provides a graphical user interface to access facility  700 . Facility  700  also includes customer account manager  710 , which controls functions available to customers with service contracts authorizing access to facility  700 . 
     User login authentication is performed by customer account manager  710  and control is passed to one of three processes, service request manager  706 , customer database manager  708 , or results manager  712  depending on the service that the customer chooses. Customers that wish to initiate a new request for composite data are passed to service request manager  706 . After successful completion of a composite data request, the customer&#39;s account is billed and the status of any pending requests is provided. Customers that wish to view the composite data generated in response to the request are passed to results manager  712 . Information pertaining to a customer&#39;s account (e.g., billing information, changing passwords, user preferences, etc.) may be obtained by submitting queries to customer database manager  708 . 
     Service request manager  706  initiates service requests and controls the computational operations required for anatomic mapping. The progress of a service request is tracked and reported to customer account manager  710 . Customer database manager  708  administers a database that contains customer account data (not shown). Customer database manager  708  is responsible for controlling and backing up the customer database and it also processes queries from customer account manager  710 , service request manager  706 , and results manager  712 . Results manager  712  integrates the results generated by service request manager  706  with context-specific medical knowledge. Results manager  712  receives information from search and filter engine  720  and mapping engine  722  specific to application requirements. Some results may be provided as visual representations (e.g., mapped segmented structures in a brain image) while others may be expressed in numeric form (e.g., coordinates of a neurosurgical target). 
     Preprocessor  714  checks patient data associated with a service request to make sure that the patient data is in the correct format and that the service request is appropriate. Support personnel  724  confirm the check performed by preprocessor  714 . Any inconsistencies or anomalies that are found are reported to service request manager  706 . Similarly, post processor  718  checks the results of a service request. Support personnel  724  confirm the check performed by post processor  718 . Any inconsistencies or anomalies that are found are reported to service request manager  706 . 
     Facility  700  also includes deformation engine  716 , search and filter engine  720 , and mapping engine  722 . Deformation engine  716  computes atlas-to-patient transformations requested by service request manager  706 . Search and filter engine  720  processes customer, patient, and procedure contextual data and integrates relevant atlas information according to application requirements. Both results manager  712  and service request manager  706  initiate search and filter engine  720  operations. Following a request from results manager  712 , mapping engine  722  applies the results of the deformation process mapping an atlas to the patient data. 
     Patient database manager  726  administers a database that contains patient data and corresponding transformations computed by deformation engine  716 . Patient database manager  726  serves queries from deformation engine  716 , search and filter engine  720 , and mapping engine  722  and is also responsible for controlling and backing up the patient database (not shown). 
     Atlas database manager  728  administers a database that contains the atlas data (not shown). Atlas database manager  728  serves queries from deformation engine  716 , search and filter engine  720 , and mapping engine  722  and is also responsible for controlling and backing up the atlas database. Atlas database manager  728  can also perform and/or manage the indexing of the co-registered data databases. 
     While there has been illustrated and described what are at present considered to be preferred embodiments and methods of the present invention, persons skilled in the art will understand that various changes and modifications may be made, and equivalents may be substituted without departing from the scope of the invention. 
     In addition, many modifications may be made to adapt a particular element, technique or implementation to the teachings of the present invention without departing from the central scope of the invention. For example, disclosed elements may be implemented in hardware, computer program code, or a combination of both hardware and computer program code. Moreover, elements depicted and described separately may be combined and implemented in a single element. Therefore, this invention is not limited to the particular embodiments and methods disclosed, but includes all embodiments falling within the scope of the appended claims.