Patent Publication Number: US-10332228-B2

Title: System and method for graphical processing of medical data

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims the benefit of, and priority to, U.S. Provisional Application Ser. No. 61/745,120, filed Dec. 21, 2012, the contents of which are incorporated by reference. 
    
    
     FIELD OF THE INVENTION 
     The invention generally relates to imaging systems in catheter labs and methods of processing data. 
     BACKGROUND 
     A catheterization laboratory, or cath lab, is an examination room in a hospital that provides the equipment to perform medical procedures that require the insertion of a catheter into a patient&#39;s arteries. Typical procedures in a cath lab include intravascular imaging, which can be used to detect vulnerable plaque in a patient&#39;s arteries before the onset of a stroke or heart attack. Cath labs also provide the equipment to treat hardened and narrowed arteries by coronary angiography—a procedure in which a doctor uses a catheter to deliver a stent or balloon to open up a narrowed artery and prevent a stroke or heart attack. 
     In all of these procedures, medical imaging equipment such as x-ray angiography, intravascular ultrasound (IVUS), or optical coherence tomography (OCT) systems can be used to help a doctor look at the affected arteries. Unfortunately, these imaging systems by their nature can impose some limits on the availability of cath labs. 
     Typical cath lab imaging systems generate large three-dimensional data sets that must be processed by high-powered computers to provide useful images. The amount of processing power required to work with the high resolution 3D images that enable plaque detection and angiographic intervention goes well beyond what is offered by typical desktop computers. Since each imaging system requires an expensive, high-powered computer, building a cath lab is a very expensive undertaking. Due to the expense required for a cath lab, some large hospitals may only build one or none, even where several are called for, and some smaller clinics may go without a cath lab entirety. 
     SUMMARY 
     The invention provides a computer server with a graphics processor that processes data sets from multiple cath labs simultaneously. The server includes a hypervisor that creates a dedicated virtual machine for each cath lab so that independent imaging or other medical system can access the resources they require, such as unique operating systems, applications, or APIs. The graphics processor provides capabilities that are needed by the imaging or other medical systems such as, for example, the ability to perform very large numbers of transformations in parallel (e.g., linear or non-linear transformations). Since the graphical processing hardware is well-suited to medical data processing, the virtual machines can efficiently handle the work demanded by imaging or other medical systems. Since the hypervisor can allocate resources as-needed to the virtual machine and re-capture the capacity of idle resources, the server can do the work that would otherwise require a large number of dedicated machines. Using high-speed networking technologies, the computer server and each of the cath labs can be in different parts of a building or different buildings. The efficiencies offered by using a hypervisor to share the resources of a graphical processor for medical imaging processing allows a greater number of cath labs to be built or operated for a given amount of resources. Thus, hospitals may have a greater number of cath labs and individual clinics can have a cath lab that could not have one otherwise. Since a greater number of cath labs can be made available, more patients can be diagnosed and treated for conditions such as arterial plaque prior to adverse events like heart attacks or strokes. 
     In certain aspects, the invention provides a medical imaging or other measurement or analysis system that uses a server with a graphics processor coupled to a memory. The server uses a hypervisor to define a plurality of virtual machines sharing the system and graphics processor. The system is operable to receive a data from a cath lab comprising a three-dimensional data set describing a patient&#39;s anatomy and perform, using the graphics processor, a plurality of transforms in parallel on the data set within one of the plurality of virtual machines. The data can be a three-dimensional data set, blood flow data, or other data. The system can then provide an analysis or a visualization image of the data set, for example, on a monitor or saved to disk. 
     The graphics processor includes one or more graphic processing units (GPUs) operably coupled together. The graphics processor is configured to perform massively parallel data processing. Additionally, the graphics processor may perform such operations as oversampling and interpolation. One or more of the GPUs may include one or more frame buffer, a hardware accelerator, or both. The graphics processor can include one or more microchip (e.g., on each GPU) operable to execute a kernel written using OpenCL, CUDA, or a similar programming language. Additionally, one or more of each GPU could include an integrated ARM CPU. In certain embodiments, the graphics processor includes a GPU from NVIDIA, AMD/ATI, S Graphics, Intel, or Matrox, a Many Integrated Cores (MIC) processor from Intel, or other similar massively parallel computational device. 
     Preferably, the server is communicably coupled to an imaging instrument in each of a plurality of cath labs, a plurality of imaging instruments of different modalities in any one cath lab, or a combination thereof. The server and the plurality of cath labs can be separated from one another, e.g., on a different floors of a building or in different buildings. 
     In related aspects, the invention provides a method of medical imaging or data analysis that includes using a server comprising a graphics processor coupled to a memory and a hypervisor to define a plurality of virtual machines. The server can be used for receiving from a cath lab a data set such as, for example, a series of images comprising a three-dimensional data set of a patient&#39;s anatomy and performing a plurality of transforms in parallel on the data set within one of the plurality of virtual machines. A visualization image of the data set may be provided by, for example, displaying it on a monitor or storing it on a disk. The server is communicably coupled to a plurality of imaging instruments, e.g., in each of a plurality of cath labs. The method can include storing transformed data in a frame buffer on the graphics processor, using a hardware accelerator on the graphics processor, performing a variety of algorithms (e.g., oversampling or interpolation) on the data set, or a combination thereof. 
     In other aspects, the invention provides a method of imaging tissue by capturing data from a patient using a medical imaging instrument, transferring the data to a server computer comprising a graphical processing unit, and using the graphical processing unit to perform various algorithms on the data in a virtual machine while the server computer simultaneously uses the graphical processing unit to perform various algorithms on other data in a second virtual machine. A visualization image of the data can then be viewed or stored based on the processing operations on the data. In some embodiments, the processing operations include performing a plurality of linear and non-linear transformations in parallel. In certain embodiments, the data includes an image of a patient&#39;s tissue and the visualization image provides an image of the patient&#39;s tissue. 
     Aspects of the invention provide a system for medical imaging that includes a server computer comprising a graphical processing unit; a hypervisor module operable to initiate the creation of a plurality of virtual machines in the server computer and to coordinate, in each of the virtual machines, a set of processing operations by the graphical processing unit on a set of data; and a tangible, non-transitory memory coupled to the graphical processing unit and operable to store processed image data. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  depicts a catheterization lab. 
         FIG. 2  shows a system for image processing. 
         FIG. 3  gives a diagram of an image processing architecture. 
         FIG. 4  illustrates an alternative architecture for image processing. 
         FIG. 5  shows a local network structure according to some embodiments. 
         FIG. 6  diagrams a method according to the invention. 
         FIG. 7  depicts an application of systems and methods of the invention. 
     
    
    
     DETAILED DESCRIPTION 
     The invention provides a computer server with a graphical processor that can process data from multiple medical imaging systems simultaneously. Data sets from cath labs or other imaging suites (x-ray, angiography, PET scans, MRI, etc.) can be provided by any suitable imaging system and a processing system of the invention allocates resources in the form of a virtual machine, processing power, operating system, applications, etc., as-needed. Embodiments of the invention are described as applied to a catheterization lab, or cath lab, and may find particular application with cath labs due to the particular processing requirements of typical cath lab systems such as intravascular ultrasound (IVUS), optical coherence tomography (OCT), functional measurement (FM), optical-acoustic imaging, and angiographic systems. However, the processing system of the invention further may be applied to any medical imaging modality. 
       FIG. 1  shows a diagram of a cath lab  101  according to certain embodiments of the invention. An operator uses control station  110  and optional navigational device  125  to operate catheter  112  via patient interface module (PIM)  105 . Here, control station  110  and PIM  105  will be described as for an IVUS system. However, the processing system of the invention is applicable to OCT, optical-acoustic imaging, FM, and other modalities, as well as IVUS. At a distal tip of catheter  112  is an ultrasound transducer  114  (in the case of IVUS). Imaging system  120  works with PIM  105  to coordinate imaging operations. Imaging operations proceed by rotating an imaging mechanism via catheter  112  while transmitting a series of electrical impulses to transducer  114  which results in sonic impulses being sent into the patient&#39;s tissue. Backscatter from the ultrasonic impulses is received by transducer  114  and interpreted to provide an image on monitor  103 . The IVUS system is operable for use during diagnostic ultrasound imaging of the peripheral and coronary vasculature of the patient. The IVUS instruments can be configured to automatically visualize boundary features, perform spectral analysis of vascular features, provide qualitative or quantitate blood flow data, or a combination thereof. Systems for IVUS suitable for use with the invention are discussed in U.S. Pat. No. 6,673,015; U.S. Pub. 2012/0265077; and U.S. RE40,608 E, the contents of which are incorporated by reference in their entirety for all purposes. Aspects of the invention are discussed in U.S. Provisional Patent Application No. 61/473,591, as well as progeny of that application, the contents of each of which are hereby incorporated by reference in their entirety. 
     Operations in cath lab  101  employ a sterile, single use intravascular ultrasound imaging catheter  112 . Catheter  112  is inserted into the coronary arteries and vessels of the peripheral vasculature under angiographic guidance. Catheters such as may be used for IVUS are described in U.S. Pat. Nos. 7,846,101; 5,771,895; 5,651,366; 5,176,141; U.S. Pub. 2012/0271170; U.S. Pub. 2012/0232400; U.S. Pub. 2012/0095340; U.S. Pub. 2009/0043191; U.S. Pub. 2004/0015065, the contents of which are incorporated by reference herein in their entirety for all purposes. Cath lab  101  may include industry standard input/output interfaces for hardware such as navigation device  125 , which can be a bedside mounted joystick. System  101  can include interfaces for one or more of an EKG system, exam room monitor, bedside rail mounted monitor, ceiling mounted exam room monitor, and server room computer hardware. 
     Catheter  112  and PIM  105  may be connected to the imaging instrument  120  and/or base station  110 , which may contain a type CF (intended for direct cardiac application) defibrillator proof isolation boundary. All other input/output interfaces within the patient environment may utilize both primary and secondary protective earth connections to limit enclosure leakage currents. The primary protective earth connection for controller  125  and control station  110  can be provided through the bedside rail mount. A secondary connection may be via a safety ground wire directly to the bedside protective earth system. Monitor  103  and an EKG interface can utilize the existing protective earth connections of the monitor and EKG system and a secondary protective earth connection from the bedside protective earth bus to the main chassis potential equalization post. Monitor  103  may be, for example, a standard SXGA (1280×1024) exam room monitor. System  101  includes control system  120  to coordinate operations. 
     Imaging instrument  120  may include one or more processor coupled to a memory. Any suitable processor can be included such as, for example, a general-purpose microprocessor, an application-specific integrated circuit, a massively parallel processing array, a field-programmable gate array, others, or a combination thereof. In some embodiments, imaging instrument  120  can include a high performance dual Xeon based system using an operating system such as Windows XP professional. Imaging instrument  120  may be provided as a single device (e.g., a desktop, laptop, or rack-mounted unit, or may include different machines coupled together (e.g., a Beowulf cluster, a network of servers, a server operating with a local client terminal, other arrangements, or a combination thereof). 
     Imaging instrument  120  may operate with different modality data sets in parallel, such as processing real time intravascular ultrasound imaging while simultaneously running a tissue classification algorithm referred to as virtual histology (VH). Instrument  120  may offload all or a portion of any processing to a graphic processor as described herein. The application software can include a DICOM3 compliant interface, a work list client interface, interfaces for connection to angiographic systems, or a combination thereof. Imaging instrument  120  may be located in a separate control room, the exam room, or in an equipment room and may be coupled to one or more of a custom control station, a second control station, a joystick controller, a PS2 keyboard with touchpad, a mouse, or any other computer control device. 
     Imaging instrument  120  will generally include memory coupled to the processor. Memory includes any one or more computer readable storage media. Memory preferably refers to tangible, non-transitory computer-readable media. Thus, any computer of the invention generally includes at least one processor (e.g., one or more silicon chip with one or more cores) coupled to at least one non-transitory memory. 
     Imaging instrument  120  may generally include one or more USB or similar interfaces for connecting peripheral equipment. Available USB devices for connection include the custom control stations, optional joystick  125 , and a color printer. In some embodiments, imaging instrument  120  includes one or more of a USB 2.0 high speed interface, a 10/100/1000 baseT Ethernet network interface, AC power jack, PS2 jack, Potential Equalization Post, 1 GigE Ethernet interface, microphone and line jacks, VGA video, DVI video interface, PIM interface, ECG interface, other connections, or a combination thereof. In certain embodiments, imaging instrument  120  operates as a proximal collector, and optional preprocessor, of data collected via PIM  105  and catheter  112 . In some embodiments, PIM  105  transmits data to a shared system without the benefit of an imaging instrument  120 , e.g., either directly or through control system  110 . 
       FIG. 2  shows a shared system  201  for processing images from a plurality of imaging systems  120   a ,  120   b ,  120   c , . . . ,  120   n . Each of the imaging systems  120  may send data via hub  213  to a shared graphics processor  205 . Processor  205  can use resources from, or be part of, networked resources  229 . In operation, processor  205  includes a hypervisor  219  that allocates processing power from one or more graphical processing unit (GPU)  213  to create a plurality of virtual machines  223   a ,  223   b ,  223   c , . . . ,  223   n . In some embodiments, each virtual machine  223  services one imaging system  120 . Additionally or alternatively, any imaging system  120  could request and receive more than one virtual machine  223 , and any virtual machine  223  could perform services for more than one imaging system  220 . Additionally, processor  205  will generally include a connection to storage  207 . 
     Shared system  201  virtualizes computers, operating systems, or both for the processing of images from a plurality of imaging systems  120  by means of hypervisor  219 . Any suitable virtual machine monitor may perform the role of hypervisor  219 . Platform virtualization is performed by system  201  (a control program), which creates a simulated computer environment, a virtual machine  223 , for its guest software. The guest software is not limited to user applications; it may allow the execution of complete operating systems. The guest software executes as if it were running directly on the physical hardware. The described architecture provides a number of benefits. The system operates at significantly lower energy consumption than a similar number of cath labs that do not share a graphic processor  205 . Processor  205  can be more easily maintained, inspected, updated, protected, and moved than a plurality of distributed computers. 
     In certain embodiments, one or more of the virtual machines  223  simulates enough hardware to allow an unmodified “guest” OS (one designed for the same instruction set) to be run in isolation. This may be allowed by including such tools as, for example, Parallels Workstation, Parallels Desktop for Mac, VirtualBox, Virtual Iron, Oracle VM, Virtual PC, Virtual Server, Hyper-V, VMware Workstation, VMware Server (formerly GSX Server), KVM, QEMU, Adeos, Mac-on-Linux, Win4BSD, Win4Lin Pro, and Egenera vBlade technology, Linux KVM, VMware Workstation, VMware Fusion, Microsoft Hyper-V, Microsoft Virtual PC, Xen, Parallels Desktop for Mac, Oracle VM Server for SPARC, VirtualBox and Parallels Workstation. 
     Due to the nature of image processing operations that are employed in medical imaging, processor  205  includes one or more of GPU  215 . GPU  215 , also occasionally called visual processing unit (VPU), provides a specialized electronic circuit to manipulate and alter memory to accelerate the building of images (e.g., within a frame buffer). GPU  215  is efficient at manipulating medical image data, and the highly parallel structure can make it more effective than general-purpose CPUs for algorithms where processing of large blocks of data is done in parallel. GPU  215  can include resources for 2D acceleration, 3D functionality, graphics-related application programming interfaces (APIs) such as OpenGL or DirectX, or general purpose GPU (GPGPU) development environments such as OpenCL or CUDA by NVIDIA. GPU  215  can include programmable shading to (e.g., each pixel can be processed by a short program that can include additional image textures as inputs; each geometric vertex can be processed by a short program; etc.). Such functionality can be offered by OpenGL API, DirectX, and the GeForce chips by NVIDIA. GPU  215  may further include support for generic stream processing. In certain embodiments, processor  205  includes a plurality of parallelized GPUs (e.g., each itself configured to perform parallel operations). Parallelized GPU computing can be implemented using any suitable platform such as, for example, products from NVIDIA, or OpenCL. OpenCL is an open standard defined by the Khronos Group. OpenCL solutions are supported by Intel, AMD, NVIDIA, and ARM. Processor  205  will include at least one GPU  215 . Any suitable GPU can be used, including, for example, those made by Intel, NVIDIA, AMD/ATI, S3 Graphics (owned by VIA Technologies), and Matrox. GPU can provide any suitable algorithm or processing function known in the art such as, for example, neural networks, decision trees, graph algorithms, tree-space searching, Markov chain Monte Carlo sampling of data sets, etc. GPU  215  can include a programmable shader or other resources to manipulate vertices and textures, perform oversampling and interpolation techniques to reduce aliasing, and very high-precision color spaces. In certain embodiments, GPU  215  is a GTX680 (GK104 core), GT640M (GK107 core), GTX 660 Ti (GK104 core), GTX 660 (GK106 core), GTX 650 (GK107 core), or GTX690 by NVIDIA or a Radeon by AMD. In some embodiments, GPU  215  includes an integrated ARM CPU of its own. GPU  215  may operate via OpeNVIDIA, OpenCL, or CUDA, an SDK and API that allows using the C programming language to code algorithms. GPU  215  can process many independent vertices and fragments in parallel. In this sense, GPU  215  is a stream processor and can operate in parallel by running one kernel on many records in a stream at once. 
     A stream includes a set of records that require similar computation. Streams provide data parallelism. Kernels are the functions that are applied to each element in the stream. In the GPUs, vertices and fragments are the elements in streams and vertex and fragment shaders are the kernels to be run on them. 
       FIG. 3  shows an exemplary relation of resources in processor  205 . Hypervisor  219  allows for the virtualization of the GPU  215 . Hypervisor  219  may be any suitable manager such as, for example, the NVIDIA VGX Hypervisor, which allows a virtual machine to interact directly with a GPU. Hypervisor  219  manages GPU resources to allow multiple medical imaging systems to share common hardware while improving user density. Each virtual machine  223  can provide a guest operating system or processing environment. The guest OS can provide applications, drivers, APIs, and remote protocol tools. Virtualization and data processing are discussed in U.S. Pat. Nos. 8,239,938; 7,672,790; 7,743,189; U.S. Pub. 2011/0274329; U.S. Pub. 2008/0143707; and U.S. Pub. 2004/0111552, the contents of each of which are incorporated by reference. Processor  205  may be onsite or off-site. 
       FIG. 4  illustrates an architecture  203  for image processing that is conducive to the use of an off-site processor  205 . Processor  205  includes the elements shown in  FIG. 4  (e.g., hypervisor  219 , VM  223 , etc.) and is connected to hub  213  over network resource  229 , which can include the Internet, a WAN or LAN, cellular telephone data networks, other methodology, or a combination thereof. Network  229  can also provide a connection for storage  207 . Via hub  213 , processor  205  is connected to a plurality of local imaging instruments  120 . It should be appreciated that a local connection can service a medical suite such as a cath lab, and can include connections to a plurality of different imaging instruments within a medical suite. 
       FIG. 5  shows a local network structure in which each medical suite has its own combination of imaging modalities, all connected to processor  205 . Here, hub  213   a  is connected to imaging suite  101   a  that includes angiographic, MRI, OCT, FM, and IVUS imaging instruments. Hub  213   b  is connected to cath lab  101   b  that provides angiographic, OCT, and IVUS services. Hub  213   n  connects to cath lab  101   n . It will be appreciated that any suitable number of cath lab  101 , each having any given combination of imaging modality instruments, may be connected to processor  205 . By sharing a graphical processor  205  with a plurality of different imaging instruments  120 , methods of the invention provide beneficial costs and qualities of image processing services. 
       FIG. 6  diagrams a method according to the invention. A data set is received  609  at processor  205  from a medical imaging instrument  120 . Hypervisor  219  allocates  615  a virtual machine  223  for the requesting instrument  120 . GPU  215  processes  621  the data set within virtual machine  223 . Where GPU  215  includes one or more optional frame buffer, the nascent processed data is stored  627  in the frame buffer. Processor  205  then operates to provide  631  the image data, which can include, for example, a 2D image for viewing on a monitor or an analytical result such as from a virtual histology analysis. When done processing  621  the data, hypervisor  219  recaptures  637  the capacity of GPU  215  that was allocated  615  to virtual machine  223 . This methodology according to the systems described herein may provide considerable savings in terms of efficient use of processing resources (e.g., in some embodiments, a shared GPU will provide service associated with a 10× reduction in demand for processing hardware). In certain embodiments, processor  205  coordinates pre-allocation. For pre-allocation, a cath lab indicates that it will perform an IVUS operation, and the server allocates and holds resources for that cath lab. When data flows, it flows through frame-by-frame in real-time for any and all sessions (e.g., IVUS sessions). When the lab session is done, the server can release the lock on those system resources that were held. Further, the server notifies labs or instruments of deficiencies in resources (e.g., if GPU is operating at full capacity and a request comes in, server can send a message to requestor saying so or giving an estimated delay). Additionally, the systems and methods herein allow for distributed medical imaging laboratories to each avail themselves of system processing power despite geographical separation. 
       FIG. 7  depicts a distributed system  701  for shared graphical processing of medical image data. Here, graphical processor  205  is operating in San Diego, Calif. Medical imaging lab  101   a  is operating in, for example, Ontario, Oreg. Lab  101   b  operates out of Creston, Iowa. Lab  101   n  is shown here in Calhoun, Ga. In each lab  101  an imaging instrument  120  operates while attached via a PIM  105  to a catheter  112  inserted into a patient&#39;s body. Data collected by transducer  114  is transferred via instrument  120  from Oregon, Iowa, and Georgia to processor  205  in California. A processing system  215  including one or more GPU in processor  205  invokes a virtual machine  223  for each lab. The data is processed in the respective virtual machine  223  in processor  205 . Results can be displayed, for example, on monitors  103  back in the respective labs  101 . While discussed with respect to  FIG. 7  as a distributed embodiment, it will be appreciated that systems and methods of the invention particularly provide embodiments in which processor  205  operates at a facility that includes the labs  101  (e.g., a plurality of cath labs  101  on a hospital campus, each networked to processor  205  in a server “closet”—generally, an air conditioned room with server racks). Further discussion of client server architecture for imaging may be found in U.S. Pub. 2012/0083696; U.S. Pub. 2011/0257545; U.S. Pub. 2011/0245669; U.S. Pub. 2011/0034801; U.S. Pub. 2008/0306766; and U.S. Pub. 2007/0043597, the contents of which are incorporated by reference in their entirety. 
     As used herein, the word “or” means “and or or”, sometimes seen or referred to as “and/or”, unless indicated otherwise. 
     INCORPORATION BY REFERENCE 
     References and citations to other documents, such as patents, patent applications, patent publications, journals, books, papers, web contents, have been made throughout this disclosure. All such documents are hereby incorporated herein by reference in their entirety for all purposes. 
     EQUIVALENTS 
     Various modifications of the invention and many further embodiments thereof, in addition to those shown and described herein, will become apparent to those skilled in the art from the full contents of this document, including references to the scientific and patent literature cited herein. The subject matter herein contains important information, exemplification and guidance that can be adapted to the practice of this invention in its various embodiments and equivalents thereof.