Method of and system for storing, communicating, and displaying image data

A system for storing, communicating, and displaying image or graphic data over a network. The system includes a client that is connectable to a server via a network. The server is configured to store an image file having image data, where the structure of the image file preferably includes submatrices. The submatrices allow the system to render the images using an adaptive rendering technique.

BACKGROUND OF THE INVENTION

The present invention relates to a method of and system for storing, communicating, and displaying image data and, particularly, storing the image data at a server, communicating at least a portion of the image data from the server to a client via a network, and displaying one or more images at the client using the communicated data.

In medical imaging, data is acquired by an imaging system (e.g., a computed-tomography system, a magnetic-resonance system, a computed-radiography system, etc.) as a series of two-dimensional (2D) image planes. The series of planes represent a three-dimensional (3D) or higher-dimensional image (e.g., a 3D image changing over time is considered to be a four-dimensional (4D) image). Prior systems stored the series of planes in memory and various storage media as either a “slice” model or a “matrix” model.

The slice model is a structure where the acquired data is stored as a series (sometimes referred to as a “stack”) of images. That is, each image plane is represented by a 2D array. Each 2D array includes a plurality of points represented by the structure (x, y), and each point has a value (e.g., an intensity, color, etc.).

The matrix model is a structure where the acquired data is stored as a single 3D array. The 3D array includes a plurality of points represented by the structure (x, y, z), and each point has a value (e.g., an intensity). For the matrix model, the 3D array includes all of the acquired points.

Medical image viewing has traditionally been film-based. A physician interested in viewing a multi-slice data set had no choice but to view the data as it was acquired, i.e., as a series of 2D planes along the direction of the scan. Both the slice model and matrix models are sufficient mechanisms to store data that is rendered in this fashion. However, these models are not adequate in other applications.

SUMMARY OF THE INVENTION

Four trends in current medical imaging have introduced limitations to the slice and matrix models for representing medical images. These trends are 1) the increasing dimensionality of the data sets (e.g., 4D imaging), 2) the increasing size of the data sets, 3) the growing practice of remote image viewing, and 4) the growing utility of generating 3D digital reconstructions of the data (e.g., multi-planar reconstruction (MPR), oblique-data slicing, surface rendering, volute rendering, etc.) The four trends have also introduced drawbacks to the slice and matrix models for storing and communicating data. The drawbacks include 1) inefficient retrieval of the data from memory and permanent media and 2) non-optimal management of network bandwidth.

A. Inefficient Retrieval of the Data From Memory and Other Storage Media.

When planes are accessed along a non-axial dimension, both the matrix and slice models force the entire data set to be loaded into memory before rendering can take place. This limitation means that the computer rendering the image must hold the entire data set in its memory and that the time required to render the first image cannot be faster than the time to retrieve the entire series from storage media.

An additional limitation of the matrix model is that consecutive points along the higher order dimensions (e.g., the third or fourth dimensions) are spaced far apart in computer memory. Points spaced far apart in memory are retrieved much less efficiently than points closer together. Rendering of coronal and sagittal slices is, therefore, much slower than rendering of transverse slices. This limitation is overcome with transposition of the data sets, but only at the cost of replicating the data in memory.

B. Non-optimal Management of Network Bandwidth.

When using a remote client to view slices along arbitrary planes of an image, the viewing application can either choose to render the image on the server followed by transmission to the client or to transmit the data set to the client followed by the client rendering the images locally. These two strategies are referred to herein as “server-based” and “client-based” rendering, respectively. Server-based rendering has the advantage that the entire data set need not be transferred to the client prior to rendering. This is important when rendered images are required very quickly at the onset of the session. Client-based rendering has high performance since the application does not need to retrieve the data over a network once the data is obtained. The limitations of client-based rendering, however, are that the client requires sufficient memory to store the image file and a latency period during which no rendering can take place while the client is downloading the data from the server.

If the client requires an image orthogonal to the acquisition plane, the matrix and slice models force prior imaging applications to choose between performing client-based or server-based rendering. That is, the application is pre-programmed to either render the data on the server or wait until the entire data set is transferred to the client before local rendering can take place.

The invention provides, in one embodiment, a system with two components: 1) a representation of D dimensional image data as a series of contiguous D dimensional matrix blocks (also referred to herein as “an array of matrices” or “submatrix representation”) and 2) an “adaptive” rendering component which switches between server-based and client-based rendering. The submatrix representation facilitates rendering of arbitrary planes through the data set (without transposition or replication of the data), compression of the image, and the efficient transmission of regions of the data set over a computer network. The adaptive rendering component enables a client viewer to enjoy the benefits from both server- and client-side rendering by optimally switching between the two modes based upon the session state, the speed of the network connection, and the computing power of the client.

In another embodiment, the invention provides a method of displaying images of a multidimensional item at a client. The multidimensional item is stored as a data file at a server, and the data file includes image data constructable to form the multidimensional item. The method includes requesting a first image and communicating a first portion of the image data to the client. The first portion of the image data includes data for displaying the first image. The method further includes storing the first portion of the image data at the client, the storing of the image data results in stored data; requesting a second image; and determining whether the stored data includes the data for displaying the second image. If the stored data does not include the data for displaying the second image, then the method further includes communicating a second portion of the image data to the client, and storing the second portion of the image data at the client. The storing of the second portion of the image data results in the stored data including the data for displaying the second image.

In yet another embodiment, the invention provides a method of rendering an image of a multidimensional item. The rendering occurs at one of a server and a client, which are connectable via a network. The method includes storing a data file at the server. The data file includes image data that is constructable to form the multidimensional item. The method further includes requesting an image and performing at least one of: 1) determining a communications speed for the network, 2) determining a memory size of a memory of the client, 3) determining an available memory size of a memory of the client, 4) determining a computing capability of the client, and determining a session state. The method also includes determining whether to render the image at the server or the client. The determination is based in part on at least one of 1) the communications speed of the network, the memory size of the memory, the available memory size of the memory, the computing capability of the second computer, and the session state. The method further includes rendering the image at the server when the determination is made to render the image at the server, and rendering the image at the client when the determination is made to render the image at the client.

In even yet another embodiment, the invention provides a method of storing an image file in memory. The method includes receiving a three-dimensional image file having image data representing a three-dimensional item, structuring the received image file using submatrices, and storing the structured image file.

Other features of the invention will become apparent by consideration of the detailed description and accompanying drawings.

DETAILED DESCRIPTION

A system100for storing, communicating, and displaying image or graphic data over a network is schematically shown inFIG. 1. The system generally includes a client105that is connectable to a server110via a network115. In some embodiments, the system100also includes an imaging system120and/or a storage device125, which may store a database126. Unless specified otherwise, the system100includes each of the elements105,110,115,120, and125. Further, while the system100is capable of storing, rendering, and communicating a variety of multidimensional image or graphic data over a network, the description below, unless specified otherwise, is in reference to multidimensional image data acquired by a medical imaging system.

The imaging system120may be any imaging or graphic generating system operable to generate an image file having image or graphic data. The image or graphic data is constructable to form images or graphics of items. In the embodiment shown, the imaging system is a medical imaging system (e.g., a computed-tomography system, a magnetic-resonance system, a computed-radiography system, etc.) that acquires a plurality of parallel 2D images of an object. The plurality of 2D images is stored by the imaging system120in an image file as either 1) an array of areas or planes, or 2) as a single matrix. The stored image data is constructable (e.g., by a computer) to form an item that represents the scanned object. However, the invention is not limited to medical imaging systems. Other imaging systems may be used to generate the image data or the imaging system may be a graphics generator that generates graphic data. Unless specified otherwise, the terms “image file,” “image data,” and “images” include a computer-generated graphic file, computer-generated graphic data, and computer-generated graphics, respectively. Further, the image file generated by the imaging system may be in other formats than the above-described array of planes or single matrix. For example and in some embodiments, the imaging system makes available a plurality of 3D matrix blocks (discussed further below) that form the 3D matrix. For the description herein, unless specified otherwise, the generated image file includes image data structured as either a plurality of 2D images or a 3D array, where the image data are constructable to form an item that represents an object.

For the embodiment shown inFIG. 1, the imaging system provides the image file to the server110. However, in other embodiments, the imaging system stores the images on a “permanent” storage medium (e.g., a medium at storage device125) and the server110obtains the image file from the storage medium.

The server110, in general, receives the image file, converts the image data of the image file into a series of D dimensional matrix blocks (if necessary), stores the matrix blocks in memory, renders images (if necessary), and communicates the matrix blocks and/or the rendered images to the client105via the network115. As used herein, the terms “computer” and “server” are not limited to a device with a single processor, but may encompass multiple computers linked in a system, computers with multiple processors, special purpose devices, computers or special purpose devices with various peripherals and input and output devices, software acting as a computer or server, and combinations of the above. For the embodiment shown inFIG. 1, the server110includes one or more processors130and a memory135. For the description below, it will be assumed that the server110has only one processor130. The processor130receives, interprets, and executes software instructions stored in the memory135; and receives, interprets, manages, and communicates information from and to the imaging system120, the storage device125, the client105, and the memory135. As used herein, the term “information” is broadly construed to include data, commands, and signals.

The memory135includes one or more memory devices (e.g., memory chips, memory modules, etc.) acting as a temporary memory (e.g., RAM), and is preferably relatively large or expandable. The memory135includes program storage140and data storage145. The program storage140includes a plurality of software modules such as a file-converter engine150and a renderer155. The file-converter engine150converts files having image data that are in an undesired format to an array of matrices. The renderer155renders images and determines the appropriate matrices corresponding to the rendered images. Operation of the file-converter engine150and the renderer155will become more apparent in the operation description below. The program storage140further includes other software modules160used by the server110that would be apparent to one skilled in the art. Examples of other software modules include an operating system, a communications module, a data manager, and other known modules.

The data storage145temporarily stores data, including 3D and 4D image files. For the embodiment shown, an image file is stored in the data storage140while a client is displaying an image of the image file. However, it is envisioned that in other embodiments the image files are maintained in the storage device125at all times. Operation of the server110is described in more detail below.

The storage device125includes one or more memory devices that provide storage for image files. For the embodiment shown, the storage device125includes one or more external storage devices (e.g., magnetic storage devices, optical storage devices, etc.) that maintain a database of 3D or 4D image files and is preferably expandable or substantially infinite. The image files are “permanently” maintained in the storage device125. When an image file is requested for viewing, the file is communicated to the server110, processed for viewing (if necessary), and stored in the data storage145until viewing is completed.

As shown inFIG. 1, the imaging system120and the storage device125are in communication with the server110by direct connections. However, it is envisioned that the communications between the server110, the imaging system120, and the storage device125may be by indirect connections.

The network115is not limited to a single network, but may encompass one or more interconnected networks. For example and in one embodiment, the network125is the Internet. However, the network125may be a private network or even a single connection between the server110and the client105. For the embodiment shown inFIG. 1, the communication protocols between the server110and the client105are primarily transmission control protocol/Internet protocol (“TCP/IP”) based. However, other communication protocols are possible.

The client105may be implemented using a variety of electronic devices operable to connect to the server110(e.g., via the network115), receive image data from the server110, and communicate the data to an operator (e.g., through a display). The client105generally includes one or more input devices165, a processor170, a memory175, and one or more output devices180. For the embodiment shown, the system100includes only one client105. However, the number of clients may vary, and the number of clients connectable to the server110is only limited by the capacity of the network115and the server110.

The one or more input devices165receive input or data and provide the received input or data to the processor170. Example input devices165include a keyboard or keypad, a pointing device (e.g., a mouse, a track ball, touch pad, etc.), a touch screen, a scanning device, microphone, or similar devices.

The processor170, which may include multiple processors, receives data or inputs from the one or more input devices165; receives, interprets, and executes software instructions from the memory175; receives, interprets, manages, and communicates information from the server120; and communicates data or outputs to the one or more output devices180.

The memory175may include one or more memory devices (e.g., memory chips, memory modules, etc.), and may be relatively large (i.e., is capable of storing large image files or multiple image files). However, in some embodiments, the memory175is limited in size (i.e., is incapable of storing large image files or multiple image files). The memory175includes program storage185and data storage190. The program storage185includes a plurality of software modules, including a renderer195. The renderer195renders desired images from matrices communicated to the client105from the server110. The program storage further includes other software modules200used by the server that would be apparent to one skilled in the art. Example software modules include an operating system, a communications module, a data manager, drivers for the input/output devices, and other known modules. The data storage190temporarily stores data, including communicated portions of the image file. Operation of the client110will become more apparent in the operation description below.

The one or more output devices180receive output or data from the processor170and perform an action (e.g., displaying, printing, storing, transmitting, etc.) with the received data. Example output devices include a visual display device (e.g., a monitor), a hard-copy device (e.g., a printer), a storage device (e.g., a magnetic storage device, an optical storage device), an audio device (e.g., a speaker), a communications device (e.g., an antenna), or similar devices. It is also envisioned that the input and output devices may be incorporated as one device (e.g., a touch screen).

Referring again toFIG. 1, the one or more input devices165, the housing205, and the one or more output device170are shown as separate components. However, the devices165and170may be incorporated within a single housing. Example electronic devices capable of being used as a client105include a computer, an Internet appliance, a personal data assistant (PDA), a handheld device, and similar devices.

Having described basic architectures of the system100, the operation of various embodiments will now be described. It should be understood that the system may include other components or elements that are not discussed above.

The system100is designed to overcome the inefficient retrieval of data from the memory135and/or the data storage145, and the non-optimal management of network bandwidth that occurs when multi-planar reconstructions of large data sets are requested by the client105. One method of operation for the system100is schematically shown inFIG. 2. At act300, the imaging system120acquires image data of an object. As used herein, the term “object” is the thing of concern. For example, if the imaging system120is a medical imaging system, then the medical imaging system acquires image data for an aspect of a patient (e.g., a patient's body or a portion of a patient's body).

While acquiring the image data, the imaging system120stores the image data in a file (act305). How the imaging system120acquires and stores the image varies depending on the imaging system used. For a typical medical imaging system, the system120acquires the image data of an object400(FIG. 3) as a series of 2D slices (only 4 slices are shown)405. Each slice405includes a plurality of points410having data values. The pluralities of points410and data values, when constructed, form an item (shown in Phantom as415) that functionally relates to or represents the object400. That is, the item415is an image of the object of interest400.

In various embodiments, the imaging system120stores the slices in a file as a 3D array (schematically represented as420) or as a series of 2D arrays (schematically represented as425). The 3D array420or the series of 2D arrays425are constructable to form the 3D image415. However, other variations of storing the image data are possible. For example, if the imaging system120acquires 4D images, then the image file may contain a series of 3D arrays. As another example and in other embodiments, the image data is structured as an array of matrices (discussed below).

At act310(FIG. 2), the imaging system120communicates the image file to the server110, which stores the image file in the storage device125. In other embodiments, the imaging system120communicates the image file to the storage device125. The storage device125, which may be a permanent storage device, stores the image file125for later use by the server110. In yet other embodiments, the imaging system120communicates the image file to the server110, which stores the image file in the data storage145.

At act315, the client makes a request to view a stored image file. For example and in one embodiment, an operator activates the client105, accesses the server110via the network115, and communicates a request to the server for viewing a specific file.

At act320, the server110obtains the requested image. For the embodiment shown, the server obtains the image file from the storage device125and places the image file in the data storage145. Depending on how the data is stored within the image file (act325), the server may apply the requested image to the file-converter engine150, which converts the data of the image file to an image file including matrices (act330).

For example,FIG. 3shows a graphical representation of an image file having an array of planes425. The file-converter engine150converts the array of planes425into a plurality of matrices (graphically represented as430, where only some of the matrices are shown). The plurality of matrices are stored in the data storage145as an array (q) of matrices (x, y, z) (graphically represented as435). Similarly, for embodiments where the image file stored in the storage device120as a 3D matrix420, the file-converter engine150also converts the large 3D matrix420into an array of matrices435. Each “sub-matrix” (graphically represented as a . . . f) stores a local 3D dimensional region of the data set. The smaller size of the sub-matrices, a . . . f, increases the chance that consecutive data points along any dimension are spaced closer together in computer memory as compared to the single matrix420or layer representations425. Additionally, the lossless compression capacity of the data blocks, a . . . f, increases because data within a local region are more likely to have a reduced dynamic range.

In one embodiment, the size of each submatrix block, a . . . f, is calculated according to the equation:

nd=2r⁢⁢o⁢⁢u⁢⁢n⁢⁢d⁡[log2⁡(Nd)2]1≤d≤D,(1)
where ndis the number of points along dimension d of the submatrix a . . . f, Ndis the number of points along dimension d of the full data set430, and D is the total number of dimensions in the data set430. Preferably, the value of ndis subject to the following constraint:

212≤Z≤214,w⁢⁢h⁢⁢e⁢⁢r⁢⁢e⁢⁢Z=∏d=1D⁢nd.(2)
If Z is above the limit of 214then the values of n1, n2, . . . , nd(i.e., a . . . h) are divided by 2 until the limit is no longer violated. If Z is below the limit of 212, the values of n1, n2, . . . , nDare multiplied by 2 until the limit is no longer violated. The inventor(s) established equations (1) and (2) based on the following considerations:1) Provide efficient access along any dimension by maintaining a constant ratio along each dimension between the length of the matrix along that dimension and the corresponding number of points in each submatrix block.2) Prevent excessive fragmentation of the data by enforcing a minimum submatrix size. Too small of a submatrix block increases the number of retrievals made to permanent media and thus slows down retrieval times.3) Prevent the size of the submatrix block from becoming too large and thus losing the benefits of the submatrix model versus the matrix model.

Thus, for the embodiment shown, if the image file obtained by the server110is not in the form of a 1D array of matrices that satisfy equations 1 and 2, then the file-converter engine150converts each 3D image of the image file into a 1D array of matrices that do satisfy equations (1) and (2). Once the file-converter engine150converts the image file, the server110stores the converted image file435in data storage145. For other embodiments, the server110stores the acquired image file within the data storage145and the file-converter engine150converts the image data as the data is being communicated to the client105.

Before proceeding further, it should be noted that an image file having a 4D image may be stored as a series of 3D images, where each 3D image has a plurality of submatrices, or as a 2D array of submatrices. The operation of the system100for a 4D image is similar to a 3D image except for the added dimension (usually time).

It should also be noted that when the server110receives an image from the imaging system120, the server110may perform the file conversion, if necessary, before storing the image file435in the storage device120. In yet other embodiments, if the structure of the image data within the image file is not in the format of an array of matrices, then the server110stores the converted image file435over, or in addition to, the original image file.

Once the image file435is properly stored within the data storage145, at act340, the server110communicates an initial image (also referred to herein as “view”) of the item to the client105via the network115. The initial image is an arbitrary image (e.g., an image plane440,FIG. 3) of the item and is communicated (e.g., displayed) at the client105to an operator (act345).

The client105and the server110then exchange subsequent requests and image data (acts350,355, and360). The exchanges allow the client to communicate multiple images of the item to the operator. In one embodiment and as shown inFIG. 4, the client105displays an initial image plane500and related orthogonal views505and510. The operator uses one of the input devices165(e.g., a pointing device) to activate one of the views (e.g., view500). The operator then uses a cursor (e.g., formed by lines515and520) to change the shown image. The lines515and520provide the geometry necessary to establish the desired plane of interest and the displayed image500varies as the user moves the cursor. Similarly, as the image500changes, the related orthogonal views505and510change. If the client105does not have the necessary image data for one of the images, then the client105communicates a request to the server110and the server110responds by providing the additional image data. The additional image data allows the client105to display the requested image.

The communicating and displaying of the multiple images preferably includes an “adaptive” rendering technique. For adaptive rendering, data is rendered either at the client105or at the server110, depending on what is optimal for a given environment. Considerations include the session state, the speed of the network115, the speed of the processor170, and the amount of memory in the data storage190(e.g., the amount of available memory for image data).

For example, if the network connection is slow, then downloading of data blocks will not keep up with requests for image rendering. This will result in a condition of the server rendering the image. That is, for low-bandwidth connections, the server110renders each requested image and communicates the rendered image to the client105for display.

If the network connection is fast, then a preference for rendering at the client results. The client105downloads the matrices having the requested necessary data for rendering the requested image. The downloading of the matrices will be ahead of requests for images.

In one embodiment, the system100determines the computing power of the client105at the beginning of the session and determines the speed of the network (e.g., using a “ping”) when a request is sent to the server110. If the data storage190has sufficient memory to hold the necessary matrices to render a requested image at the client105and if the network speed is sufficient, then the system100performs the rendering at the client105; otherwise the server renders the images.

Additionally, the memory available to the client105determines how many matrices are cached (i.e., stored in the data storage190) by the client105. For example, the system100accommodates “thin-client” devices by storing only those matrices used to render the current image. The system100accommodates “thick-client” devices by keeping a semi-permanent cache of the image data communicated to the client105. If a “thick-client” device runs out of memory, then the client105overwrites the oldest cached image data.

For client-based rendering using the data structure435, the server110communicates only the plurality of matrices that include the requested image. When a new image for a given orientation and position is requested, the client105determines whether its data buffer contains all of the matrices associated with that image. If not, the client105places a request to retrieve the requested image from the server110on a priority queue.

In one embodiment (schematically represented as600inFIG. 5), the server110renders and communicates the requested image plane440(represented by path605). When the client105receives the single image plane440from the server110, it is immediately displayed. This process takes approximately 20 frames/second for a 512×512 image over a 100K/sec network using a lossy compression ratio of around 30 and about 2 frames/second using a lossless compression method. While the image is being displayed, the server110communicates to the client105the submatrices (the shaded submatrices of data structure430) that contain the data for that view440(represented by path610). A subsequent request for data in that region likely results in a condition where the data storage190contains all of the requested data blocks and client-based rendering occurs. If not all of the data for rendering the image is local, then the client105places a request on a priority queue615for the image plane and the associated submatrices. Requests for image planes take priority over requests for data blocks. The result of the adaptive rendering technique is that the images are displayed in a substantially real-time basis.

In another embodiment, requests for images to the server result in the server immediately communicating to the client105the plurality of matrices that contain the image data. The client105then stores the plurality of matrices as it receives the matrices and renders the requested image when the necessary matrices are stored. In some embodiments, if the client105already includes a communicated matrix, then it does not store that matrix (i.e., the matrix was already received in a previous request).

In yet other embodiments, the client105includes a general knowledge of the data structure430such that, when an image is requested, the client105requests only the necessary matrices for the client105to render a desired image.

Thus, the invention provides, among other things, a new and useful method of and system for storing, communicating, and displaying image data. Various features and advantages of the invention are set forth in the following claims.