High quality embedded graphics for remote visualization

Apparatuses and methods for rendering and compositing two-dimensional images for remote visualization using a multi-pass rendering technique are provided. A server computer renders a three-dimensional volume separately from one or more three-dimensional graphical objects to be embedded in the rendered image of the volume. The server compresses and transmits the separately rendered images to a client computer. The client computer decompresses the rendered images and generates a composite image using the rendered images. If a user manipulates, adds or deletes an embedded object, the client computer generates a new composite image using the previously rendered volume image. In the case of a new or manipulated embedded graphical object, the server only renders the manipulated or new object.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent document claims the benefit of U.S. Patent Application No. 62/010,712, filed on Jun. 11, 2014, which is hereby incorporated by reference.

BACKGROUND

In many three-dimensional (3D) volume visualization applications, such as computerized tomography (CT) or magnetic resonance (MR) scanning, it is common to embed graphical objects within a 3D volume rendering to provide display points, line graphics, or opaque object surfaces at their true 3D locations within the 3D volume rendering. The embedded graphical objects are often manipulated using a mouse or other user interface. It is challenging to meet the speed and performance requirements to manipulate the embedded graphical objects within a 3D volume because the 3D volume and the graphical objects are completely re-rendered in order to generate a composite image with the correct depth embedding. This challenge is common in many advanced visualization application systems and is particularly limiting when the visualization application is accessed from a remote client computer, adding data transfer latency between the application client and the application server.

Many existing remote visualization applications workaround this challenge by providing only a two-dimensional (2D) overlay visualization graphics over a 3D volume rendering. Providing the 2D overlays prevents the visualization from providing accurate depth perception. Another workaround renders lower resolution 3D embedded graphics in order to speed up performance of the visualization application, compromising the image quality of the graphics. Alternatively, high performance computing application systems and high speed networks may compensate for the latency, but the systems and networks are not cost effective and may not be available. Thus, these workarounds fail to provide a high quality 3D visualization in a cost effective manner.

SUMMARY

The present embodiments relate to high quality remote visualization of embedded graphics. By way of introduction, the present embodiments described below include apparatuses and methods for rendering and compositing resulting two-dimensional images for remote visualization using a multi-pass rendering technique. A server computer separately renders one or more three-dimensional volumes and one or more three-dimensional graphical objects. The server computer compresses and transmits the separately rendered images to a client computer. The client computer decompresses the rendered images and generates a composite image using the decompressed images. If a user manipulates, adds or deletes an object, the client computer re-generates the composite image using the previously rendered volume image. In the case of a new or manipulated embedded graphical object, the server only renders and transmits the manipulated or new object to the client for compositing.

In a first aspect, a method for rendering embedded graphics for remote visualization is provided. A server renders a two-dimensional volume rendered image from three-dimensional data captured for a volume and renders a two-dimensional graphic rendered image to be embedded in the two-dimensional volume rendered image from three-dimensional data for a graphic. The server compresses the volume rendered image and the graphic rendered image and transmits the compressed volume rendered image and the compressed graphic rendered image to a workstation over a network.

In a second aspect, a system for rendering embedded graphics for remote visualization is provided. A server is configured to render a two-dimensional volume rendered image from three-dimensional data captured for a volume and to render a two-dimensional graphic rendered image to be embedded in the two-dimensional volume rendered image from three-dimensional data for a graphic. The server is also configured to compress the volume rendered image and the graphic rendered image and to transmit the compressed volume rendered image and compressed the graphic rendered image over a network. A workstation is configured to receive the compressed volume rendered image and the compressed graphic rendered image over the network and to decompress the compressed volume rendered image and the compressed graphic rendered image. The workstation is also configured to generate a composite rendered image from the volume rendered image and the graphic rendered image and to display the composite rendered image.

In a third aspect, a method for rendering high quality embedded graphics for remote is provided. A patient is scanned to capture three-dimensional volume data representing the patient. A two-dimensional volume rendered image is rendered from the three-dimensional volume data representing the patient and a plurality of two-dimensional graphic rendered images to be embedded in the two-dimensional volume rendered image are rendered from three-dimensional data for a plurality of graphics. The graphics include a display point, a line graphic, an opaque object or a combination thereof. A composite rendered image is generated from the volume rendered image and the plurality of graphic rendered images and the composite rendered image is displayed.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

This present embodiments provide a multi-pass rendering technique to preserve high quality embedded graphics in a volume rendering. The multi-pass rendering technique allows for real-time interaction when the graphical objects are manipulated within an application system. The multi-pass rendering technique also leverages the client side device of the application system to perform all or part of the compositing steps without loss of depth perception in the composite image generated and displayed. According to one embodiment, the multi-pass rendering technique is used to display full, high quality embedded graphics accurately in a volume rendering. The multi-pass rendering technique is not limited to client-server applications, and can be used in standalone applications to improve the performance of the generation and manipulation of the embedded graphics.

The multi-pass rendering technique may improve remote visualization of embedded graphics by rendering a volume image and one or more graphics images separately on a server, and compositing the rendered images on a client computer. The proposed embodiments may provide for full, high quality (e.g., available quality or resolution of the voxels from a patient scan) embedded graphics to be displayed on the client computer. The proposed embodiments may also provide for efficient graphics manipulation, including fast deletion or modification of graphics without the need to re-render the volume image on the server computer. Upon deletion or modification of a graphic, the client computer only re-generates and displays the rendered volume image or a new composite image. As a result of the present embodiments, better performance may be achieved.

FIG. 1illustrates an example of high quality embedded graphics for remote visualization.FIG. 1depicts a composite image generated from a volume rendering and three embedded graphical objects. The composite image is captured by a three-dimensional (3D) scanning application, such as a computerized tomography (CT) or magnetic resonance (MR) scanning application. The rendered volume depicted inFIG. 1is an image of a patient's torso. Other volumes may be depicted, such as another area of a patient's body, or another object. The three embedded graphical objects include a polyline103and two markers101and105. For example, the polyline103is used to take measurements in the volume, such as the size of a tumor or the size of organs and other tissue. As shown inFIG. 1, the length of the polyline103is provided (e.g., 463.96 mm) as part of the graphic. The embedded graphical objects101and103are display points. The display points provide 3D markers in the volume, such as used to identify objects or areas within the volume. Additional, different, or fewer embedded graphical objects may be provided. For example, opaque object surfaces may be provided, such as a mesh model of a surface extracted from the volume to show blood vessels in the volume. As another example, a target or other shape point marker is used. The embedded graphical objects may be manipulated using a mouse or other user interface.

FIG. 2illustrates an embodiment of a system for high quality embedded graphics for remote visualization. The system200includes a server computer201, a network203, a client computer205and a scanner207. Additional, different, or fewer components may be provided. For example, additional server computers201, networks203, client computers205and/or scanners207are used. In another example, the server computer201and the client computer205are directly connected, or implemented on a single computing device. In yet another example, server computer201, client computer205and scanner207are implemented on a single scanning device. As another example, the client computer205is part of the scanner207.

The server201is a server computer platform having hardware such as one or more central processing units (CPU), a system memory, a random access memory (RAM) and input/output (I/O) interface(s). The server201also includes a graphics processor unit (GPU) to accelerate image rendering. The server201is implemented on one or more server computers connected to network203. Additional, different or fewer components may be provided.

The server201is configured to execute an application to render two-dimensional (2D) rendered images from three-dimensional (3D) image data. For example, the server201executes the application to render a 2D volume rendered image from 3D data captured by the scanner207. In alternative embodiments, a memory separate from the scanner provides the 3D data or volume to the server201. Further, the server201executes the application to render a 2D graphic rendered image from 3D data for one or more graphics to be embedded in the 2D volume rendered image. The 3D data from the graphic is generated by the application, by an application executed by client computer205, and/or by scanner207.

The server201is configured to execute an application to render the 2D volume rendered image and the 2D graphic rendered image separately, according to a multi-pass methodology. According to the multi-pass methodology, the 2D volume rendered image is rendered in a first pass, then one or more 2D graphic rendered images are rendered in subsequent passes. Alternatively, the 2D volume rendered image and the 2D graphic rendered images are rendered separately and concurrently in parallel processes. Any rendering may be used, such as projection or surface rendering.

The server201is further configured to execute an application to compress the volume rendered image and the graphic rendered image. The rendered images are compressed using JPEG or other lossy or lossless compression. The server201transmits the compressed volume rendered image and compressed the graphic rendered image to the client computer205via the network203. In other embodiments, one or more of the images are not compressed, but instead transmitted without compression.

The system200includes a client computer205, such as a user workstation. The client computer205is a client computer platform having hardware, such as one or more central processing units (CPU), a system memory, a random access memory (RAM) and input/output (I/O) interface(s). The client computer205is implemented on one or more computers connected to network203, such as a tablet, personal computer, laptop, desktop, or other computer. Additional, different or fewer components may be provided, such as a graphics processing unit.

The client computer205is configured to execute an application configured to receive and to decompress the compressed volume rendered image and the compressed graphic rendered image from the server201over the network203. Further, the client computer205executes the application to generate and display a composite rendered image from the volume rendered image and the graphic rendered image. For example, the client computer205executes the application to replace pixels in the volume rendered image with pixels from graphic rendered image, embedding the graphic object image in the volume image resulting in a composite image. Only the pixels representing a graphic object are replaced. The composite image is displayed. An overlay process with partially transparent graphic image may be used to embed the graphic object image in the volume image.

The system200includes a scanner207. The scanner207is a three-dimensional scanner, such as a computerized tomography (CT) or magnetic resonance (MR) scanner. Other three-dimensional scanners may be used. The scanner207captures three-dimensional data for the volume and/or generates the embedded graphic used to render one or more two-dimensional images.

The system200includes a network203. The network203is a wired or wireless network, or a combination thereof. Network203is configured as a local area network (LAN), wide area network (WAN), intranet, Internet or other now known or later developed network configurations. Any network or combination of networks for communicating between the client computer205, the scanner207and the server201may be used.

FIG. 3illustrates a flowchart diagram of an embodiment of a method for generating high quality embedded graphics for remote visualization. The method is implemented by the system ofFIG. 2and/or a different system. Additional, different or fewer acts may be provided. The method is provided in the order shown. Other orders may be provided and acts may be repeated. Further, the acts may be performed concurrently as parallel acts.

At act301A, a patient or other object is scanned to capture three-dimensional volume data representing the patient or other object. At act3018, one or more graphics are input by a user or generated by a processor. For example, the processor of the scanner207generates the spline graphic shown inFIG. 1, and a user inputs the point selections. The graphics are distributed in 3D space.

At act303A, a two-dimensional volume image is rendered from the three-dimensional data captured for the patient or other object. Any rendering may be used, such as projection or surface. Other rendering processes may be applied, such as adding lighting effects. The three-dimensional volume rendering outputs a two-dimensional volume image, such as a graphics hardware frame buffer object (FBO). For example, the processor of server201renders a 2D image from the spline graphic generated by the processor of the scanner207. For example, using FBO(s), each image has a set of two-dimensional image data including a color image of the two-dimensional rendering results, a depth buffer and a stencil buffer.

At act303B, a two-dimensional graphic image is rendered from three-dimensional data for a graphic and/or object to be embedded in the two-dimensional volume rendered image. One or more graphics and/or objects are rendered. For example, one or more graphics are rendered separately. Alternatively, one or more graphics are rendered together, or the graphics are rendered in one or more sets of graphic images. Only one set of graphic image data is required (e.g., with only one graphic image), but the application may utilize more than one set of graphic image data, with each graphic image having a separate set of image data. In an example, several objects are rendered separately. By separating the graphic objects, performance associated with manipulating the various graphic objects may improve by reducing the size of the graphic object needing to be re-composited with the rendered volume image upon manipulation (e.g., reducing the non-empty image region in the rendered graphic image). For example, referring toFIG. 1, if each of the objects101,103and105are rendered separately, manipulating one of the three objects (101,103or105) only requires re-rendering the manipulated object.

The same or different type of rendering is used for the graphics as is used for the scan volume. For example, the scan volume is rendered using alpha blending or maximum intensity projection after segmentation. The graphics are rendered using the same projection rendering or a surface rendering. For example, the processor of server201renders one or more 2D images in the same manner as the 2D image rendered from the spline graphic.

The three-dimensional rendering outputs two-dimensional images, such as FBO(s), with the same image size and resolution for the rendered volume image and the rendered graphics images. For example, using frame buffer objects FBO(s), each image has a set of two-dimensional image data including a color image of the two-dimensional rendering results, a depth buffer and a stencil buffer. The color image includes pixel color information of the two-dimensional rendered image. The depth buffer provides a relative distance for each pixel of the image (e.g., the distance from the scanner or camera to the pixel). The stencil buffer is used to identify which pixels of the rendered graphic image intersect with pixels of the volume rendered image. In an example, the stencil buffer is used to identify so-called empty and non-empty pixels of a rendered graphic image. For example, non-empty pixels are pixels of the rendered graphic image that intersect with pixels of the rendered volume image (e.g., same location in the two-dimensional rendered area). Empty pixels are pixels of the rendered graphic image that do not intersect with pixels of the volume rendered image. By employing empty and non-empty pixels, the color image of the rendered graphic image will only contain pixels that cover, or are covered, by the rendered volume image.

At acts305A and305B, the rendered volume and the graphic images are compressed. For example, lossless compression is used to compress the rendered volume image and the graphic image(s) for transmission. In this example, various lossless compression techniques may be employed, such as, sparse index and color array or run-length encoding. Other lossless compression techniques are contemplated, such as encoding using a JPEG format. For example, the processor of server201compresses the rendered volume image and one or more graphic images.

At acts307A and307B, the compressed volume rendered image and the compressed graphic rendered image(s) are transmitted to the client computer over the network. For example, a volume rendered image and one or more graphic color images are transmitted by the server201to the client computer205via a network203. The compressed volume rendered image and one or more compressed the graphic rendered images are received by the client. In the client-server implementation, the compressed volume rendered and one or more graphic color images are received by the client computer205from the server201via the network203.

At act309, the compressed volume rendered image and one or more compressed graphic rendered images are decompressed. For example, in a remote client in a client-server implementation, the compressed volume rendered and one or more graphic color images are decoded on the client side by the processor of the client computer205.

At act311, a composite rendered image is generated from the volume rendered image and one or more graphic rendered images. For example, the processor of client computer205generates the composite rendered image. The volume rendered image and one or more graphic color images are composited or blended together by replacing pixels in the volume rendered image with the non-empty pixels from one or more graphics color images. By compositing the images together, one or more graphic rendered images are embedded into the volume rendered image. For example, each non-empty pixel from a graphic rendered image replaces each corresponding pixel of the volume rendered image. Alternatively, the depth buffer entry for each non-empty pixel from a graphic rendered image is compared with the depth buffer entry for each corresponding pixel for the volume rendered image. In this example, only pixels of the volume rendered image with smaller depth buffer values are replaced by non-empty pixels from the graphic rendered image. As such, if a pixel of the volume rendered image has a greater three-dimensional depth than a pixel of the graphic rendered image, the pixel is covered by the pixel of the graphic image and will be replaced in the composite image. In alternative embodiments, the depth buffer controls the level of transparency. Closer objects are rendered more opaque than further objects, but with transparency allowing viewing of the further objects (e.g., allowing a graphic behind an organ of the volume rendered image to be viewed).

At act311, the composite rendered image is displayed. For example, a display of client computer205displays the composite rendered image. Optionally, the compressed or decompressed volume rendered image and/or graphic rendered image(s) are maintained on the client computer such that unmodified images do not need to be re-rendered, re-compressed and/or re-transmitted by the server. For example, after displaying the composite image, a user may manipulate one or more of the embedded graphics images. In this example, the user requests that one or more of the embedded graphics images be deleted, such as deleting the spline ofFIG. 1. The client re-generates a composite rendered image from the volume rendered image and the remaining or none of the graphic rendered images. If all of the embedded graphics images are deleted, the volume rendered image is displayed without added graphics. Alternatively, the user requests an additional embedded image. In this example, the server renders, compresses and transmits the additional graphic rendered image to the client. The client re-generates a composite rendered image from the composite image and the additional graphic rendered image. As such, the disclosed embodiments allow for real-time interaction where the graphic objects can be manipulated within an application system.