Patent Description:
Multi-modality "fusion" imaging of ultrasound with a prior image (of the same or other imaging modality) can be enabled using electromagnetic (EM) tracking of an ultrasound probe and registration of the EM coordinate system with the coordinate system of the prior image. Automatic methods to establish the registration may be based on acquisition of an EM-tracked three-dimensional (3D) ultrasound (US) volume (called a baseline 3DUS), followed by manual or automatic image-based registration of the baseline 3DUS to the prior static image (e.g., a computed tomography (CT) image).

If internal organ motion occurs, e.g., due to respiration, the registration between the live ultrasound imaging and the prior image will no longer be accurate. In particular, if the operator is planning an intervention such as a needle insertion into a tumor, the operator will typically request a breath hold to interrupt the tumor motion. However, the position of the tumor during this breath hold typically differs from the position during baseline 3DUS acquisition. Thus, the fusion image with the prior static image may suffer from inaccuracies.

Image-based registration methods to re-register a current or "live" ultrasound image back to the baseline 3DUS have been attempted to compensate for organ motion. However, such registration methods are not robust or accurate if the overlap or similarity between the images to be registered is insufficient.

<CIT> shows an image-based registration method to register a current ultrasound image back to a baseline image that uses a tracking system that is registered to the current and baseline images. A quantitative analysis is performed on the current ultrasound image to determine whether to acquire additional ultrasound images.

<CIT> shows an ultrasound imaging system that provides a concurrent display of a reference contrast enhanced image and a current ultrasound image in either of a side-by-side presentation or with the ultrasound images overlaid over each other.

<CIT> shows a system for correcting misalignments or offsets between a series or sequence of medical images that determines an image offset between the images to enable placing the sequence of images into a common frame of reference. To determine the image offset, a search is performed for a transformation that derives a maximum of similarity for the pixels between the images.

<CIT> shows a calibration checking system that utilizes validation testing including a testing of an absolute differential between an image based volume motion (VMIB) and a tracking based volume motion (VMTB) relative to a calibration threshold (CT) to determine whether there are problems with the calibration. In a case wherein a problem with the calibration is detected, a warning sign may be raised by the system.

In accordance with the present principles, a system for fusing images to account for motion compensation according to claim <NUM> is disclosed.

Further, a method for fusing images to account for motion compensation according to claim <NUM> is disclosed.

Advantageous embodiments are disclosed in the dependent claims.

These and other objects, features and advantages of the present disclosure will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

This disclosure will present in detail the following description of preferred embodiments with reference to the following figures wherein:.

In accordance with the present principles, systems and methods provide feedback to an operator about success parameters of a current view, and may provide guidance for a user to achieve a view with optimal success parameters. Success parameters are parameters related to overlap or similarity between images (fields of view (FOV)) to be registered. Without motion compensation, fusion imaging becomes less accurate and less useful when organ motion occurs. Manual motion compensation is cumbersome, inaccurate, and user-dependent. Automatic image-based motion compensation is fast and accurate only if a live image to be used for registration has sufficient overlap and similarity ("success parameters") with the baseline image. Since it is not trivial for the operator to assess the success parameters of an ultrasound view, the operator may acquire 3DUS (three-dimensional ultrasound) images and attempt motion compensation by "trial and error", resulting in failed registration attempts, wasted time, and operator dissatisfaction.

The present principles provide feedback about the success parameters of the current view, before the image is acquired and motion compensation is attempted, and provide guidance to a view that has high or optimal success parameters. The present principles result in efficient motion compensation in multi-modality "fusion" imaging procedures. Motion is compensated using image-based registration of a "live" or "current" ultrasound volume relative to a prior "static" ultrasound volume. The static volume, in turn, may be pre-registered to another modality such as a computed tomography (CT) image. The registration-based motion compensation uses the live and static images to find sufficient similarity and overlap (success parameters).

In one embodiment, fields of view are compared to identify similar poses for imaging equipment to enable sufficient overlap (success parameters) between baseline and live images. An interface provides live feedback on such success parameters to guide the user to an acquisition of the live image that permits successful and accurate registration with the static image.

In one motion compensation embodiment, the operator acquires a live, electromagnetic (EM)-tracked 3D ultrasound (3DUS) image that can be registered with a prior static 3DUS. Based on the EM tracking and the known field-of-view (FOV) of the 3DUS, the relative pose and overlap between the current view and the static image can be computed. This information is provided to the operator to identify a view that is suited for motion compensation while also imaging a desired target area (e.g., a tumor). In an alternative embodiment, the operator inputs the desired target area, and the system computes one or several suitable views to image the target area with sufficient overlap and pose similarity to the static image to permit successful motion compensation. The system may then provide guidance to the operator to place an ultrasound probe in or near a pre-computed pose for 3DUS acquisition. In other embodiments, image similarity between baseline and current views may be employed to find an optimal field of view match.

It should be understood that the present invention will be described in terms of medical imaging instruments; however, the teachings of the present invention are much broader and are applicable to any imaging instruments where motion compensation is useful. In some embodiments, the present principles are employed in tracking or analyzing complex biological or mechanical systems. In particular, the present principles are applicable to internal tracking procedures of biological systems and in procedures in all areas of the body, such as the lungs, gastro-intestinal tract, excretory organs, blood vessels, etc. The elements depicted in the FIGS. may be implemented in various combinations of hardware and software and provide functions which may be combined in a single element or multiple elements.

The functions of the various elements shown in the FIGS. can be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which can be shared. Moreover, explicit use of the term "processor" or "controller" should not be construed to refer exclusively to hardware capable of executing software, and can implicitly include, without limitation, digital signal processor ("DSP") hardware, read-only memory ("ROM") for storing software, random access memory ("RAM"), non-volatile storage, etc..

It be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative system components and/or circuitry embodying the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams and the like represent various processes which may be substantially represented in computer readable storage media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

Furthermore, embodiments of the present invention can take the form of a computer program product accessible from a computer-usable or computer-readable storage medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable storage medium can be any apparatus that may include, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk - read only memory (CD-ROM), compact disk - read/write (CD-R/W), Blu-Ray™ and DVD.

Reference in the specification to "one embodiment" or "an embodiment" of the present principles, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment", as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

It is to be appreciated that the use of any of the following "/", "and/or", and "at least one of", for example, in the cases of "A/B", "A and/or B" and "at least one of A and B", is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of "A, B, and/or C" and "at least one of A, B, and C", such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.

It will also be understood that when an element, such as, e.g., an image, image region or overlay, is referred to as being "on" or "over" another element, it can be directly on the other element or intervening elements may also be present. In contrast, when an element is referred to as being "directly on" or "directly over" another element, there are no intervening elements present.

Referring now to the drawings in which like numerals represent the same or similar elements and initially to <FIG>, a system <NUM> for image fusion is illustratively shown in accordance with one embodiment. System <NUM> may include a computer or other workstation or console <NUM> from which a procedure is supervised and/or managed and/or imaging fusion is performed. Workstation <NUM> preferably includes one or more processors <NUM> and memory <NUM> for storing programs and applications. Memory <NUM> may store a plurality of modules, units, applications and/or programs configured to fuse images from same or different imaging modalities. It should be understood that the modules being described as stored in memory <NUM> may in fact include electronic circuits or other hardware components including connectors, wires, circuit boards, integrated circuit chips, memory devices, etc. in addition to or instead of software.

Memory <NUM> includes a pose analyzer unit <NUM>, which receives spatial pose tracking information from a baseline 3DUS stored in tracked baseline memory <NUM> and from live/current image <NUM>, which may be stored in tracked current memory <NUM> when acquired at a current/live position of an imaging probe <NUM>, to compute success parameters. The imaging probe <NUM> may include an ultrasound probe for live or real-time imaging of a subject <NUM>. The imaging probe <NUM> is tracked using electromagnetic (EM) tracking <NUM>, although other tracking technologies may be employed. Pose analyzer unit <NUM> passes the information (success parameters) to a display <NUM>, and optionally to a pose guidance unit <NUM>. The pose guidance unit <NUM> may receive user input (target area) via a graphical user interface on display <NUM> and/or interface <NUM>. While the pose analyzer unit <NUM> computes overlap and similarity (success parameters) for measuring field of view similarity, the pose guidance unit <NUM> provides direct information on how to reposition the probe <NUM> to obtain optimal fields of view for images. As an example, the pose analyzer unit <NUM> may provide an overlap percentage (e.g., <NUM>% overlap) as feedback, while the pose guidance unit <NUM> could provide instructions, e.g., "move probe to the left", to achieve a higher overlap percentage.

The user is provided with real time feedback and/or guidance to achieve a best pose for performing a task, replicating the baseline image, increasing the probability of a good registration with prior static images, etc..

Display <NUM> may also permit a user to interact with the workstation <NUM> and its components and functions, or any other element within the system <NUM>. This is further facilitated by the interface <NUM> which may include a keyboard, mouse, a joystick, a haptic device, microphone, speakers, lights or any other peripheral or control to permit user feedback from and interaction with the workstation <NUM>. The interface <NUM> may provide feedback and guidance that is displayed to the operator and is thus visible on the system's display <NUM>, audible if audio feedback is employed, vibrational if haptic feedback is employed, etc..

System <NUM> provides an ultrasound fusion imaging system based on spatial tracking (e.g., EM tracking using and EM tracking device <NUM> and EM probe tracking <NUM>) of the ultrasound probe <NUM>. It should be understood that other tracking technologies may also be employed, e.g., optical shape sensing, etc. A registration unit <NUM>, e.g., a multi-modality registration module, provides registration for different modalities with or to a prior static image stored in memory <NUM>. The prior static image <NUM> may be captured by any imaging modality <NUM>, such as, e.g., CT, magnetic resonance, X-ray, US, etc. The imaging modality <NUM> may be present during a procedure or the static images <NUM> may be supplied from a previous procedure or captured image. Registration unit <NUM> registers the prior static image from memory <NUM> with an acquired spatially tracked baseline 3DUS volume stored in memory <NUM>.

The information from live probe tracking <NUM> and a known field-of-view of the ultrasound images <NUM> from a live imaging modality <NUM> are employed to continuously compute and display overlap and pose similarity (e.g., relative rotations, etc. that provide success parameters) of a current view with the baseline 3DUS stored in memory <NUM> before a new image 3DUS <NUM> is acquired, stored and/or used for registration.

The success parameters may also measure image similarity (e.g., image contrast, brightness, information content or other features) extracted from live images. Note that the image-based success parameters (from live-images) need the acquisition and processing of 2D or preliminary images (<NUM>) (e.g., live and/or baseline images), whereas the other success parameters only need the pose tracking information (and knowledge of the ultrasound field of view - but no images needed).

While US is described, other imaging modalities may be employed for imaging modality <NUM>. Note that the image registration is less likely to succeed the smaller the overlap and the larger the relative rotation between the images. A motion compensation unit <NUM> accounts for motion between the new image 3DUS <NUM> and the baseline 3DUS image <NUM> and provides the motion compensation information to a registration unit <NUM>, which is employed to register an EM coordinate system of EM tracker <NUM> with the static image <NUM> and provide registration for the baseline image <NUM> registered to the static images <NUM> from the registration unit <NUM>.

The motion compensation registration unit <NUM> computes the differences in images between the baseline images <NUM> and the tracked current image <NUM>. This information is employed by the pose analyzer unit <NUM> to compute the success parameters for a current pose before acquiring a new image. It should be understood that while registration units and modules are described individually, registration of the various coordinate systems, computation of transforms and other registration functions may be performed by a single or multiple registration units, engines or programs.

In an alternative embodiment, the system <NUM> also computes and provides guidance to acquire an optimal view using the post guidance unit <NUM>. A computation of the optimal view may use operator input of a desired target area for imaging, such as a tumor, to find a view that images the tumor while maintaining sufficient success parameters for motion compensation.

The system <NUM> is operable on a tracking-based ultrasound fusion imaging system, such as the Philips® PercuNav® product, although the present principles may be applied to other devices and may include other imaging modalities. In particularly useful embodiments, the present principles are operable on a system that acquires 3DUS baseline images <NUM>, which, in turn, are registered by the registration unit <NUM> to the prior static image (e.g., CT) <NUM> that the live ultrasound image <NUM> is to be fused with. One goal is to provide an efficient workflow and method for compensation of organ motion that may have occurred since the time the 3DUS baseline image <NUM> was captured. During fusion imaging, the ultrasound system is typically in "live 2D" mode, and the live 2D images <NUM> from the ultrasound scanner are fused (via the current registration units <NUM>, <NUM>) with the prior static image (CT) <NUM>.

The operator is interested in fusion-imaging of a particular target area, such as a tumor, and will move the ultrasound probe <NUM> to explore different ways of visualizing the tumor. Different views of the tumor are possible, but only some may have sufficient overlap and pose similarity (success parameters) with the baseline 3DUS <NUM> for motion compensation.

The system <NUM> continuously computes and displays the success parameters, enabling the operator to identify a view of the tumor with sufficient success parameters. The operator optionally proceeds to obtain a breath hold from the patient, acquires a live 2D/3DUS image <NUM> of the tumor, which is stored in the current view memory <NUM> once acquired. The motion compensation unit <NUM> is triggered with the acquisition of the new 3DUS image (stored in memory <NUM>).

The system <NUM> carries out the motion compensation by registering the live 3DUS image <NUM> with the baseline 3DUS <NUM>, and using the registration result from registration unit <NUM> to update the displayed fusion image. The updated fusion image can now be used by the operator to visualize the tumor or to carry out an intervention (such as a needle insertion).

Referring to <FIG>, a method for tracking-based ultrasound fusion imaging is illustratively shown. In block <NUM>, the fusion imaging system is initialized. A baseline image is acquired and its pose is stored. The acquired image is registered with a prior image (e.g., a static image of the same volume). In particular, acquisition of a 3DUS baseline image is obtained, which in turn is registered to the prior static image (e.g., CT) that the live ultrasound is to be fused with. In block <NUM>, a live image of a target area is obtained. The live image may be, e.g., a 2D or 3DUS image. The ultrasound system is usually in "live 2D" mode, and the live 2D images from the ultrasound scanner are fused (via the current registration) with the prior static image (CT).

In block <NUM>, live tracking information (e.g., EM pose and/or image features) and the baseline pose and/or image are employed to compute and display success parameters. Optionally, the live 2D image from block <NUM> may also be used in the calculation of success parameters. Features of the live 2D images such as, e.g., image brightness, contrast, etc. may be employed to detect good imaging conditions for successful motion compensation. One aim is to provide an efficient workflow and method for compensation of organ motion that may have occurred since the time of the 3DUS baseline image. The system continuously computes and displays the success parameters, enabling the operator to identify a view of the tumor with sufficient success parameters.

In block <NUM>, a determination of the quality of the success parameters is performed. If the success parameters are sufficient, the path continues with block <NUM>. Otherwise, the path returns to block <NUM>. An operator is usually interested in fusion-imaging a particular target area, such as a tumor, and will move the ultrasound probe to explore different ways of visualizing the tumor. Different views of the tumor are possible, but only some may have sufficient overlap and pose similarity (success parameters) with the baseline 3DUS for motion compensation.

In block <NUM>, an optional breath hold may be requested of the patient. In block <NUM>, an acquisition of a new image (e.g., 3DUS) is triggered. This occurs when the success parameters are adequate. In block <NUM>, the newly acquired image (e.g., 3DUS) is registered onto the baseline image. The operator may proceed to obtain the breath hold from the patient, acquire a live 3DUS of the tumor in the current view, and trigger the motion compensation with the acquired 3DUS.

In block <NUM>, the registration result from block <NUM> is employed to update the registration with the prior US image (baseline or previously acquired US image). The system carries out the motion compensation by registering the live 3DUS with the baseline 3DUS, and using the registration result to update the fusion image. In block <NUM>, a determination of the quality of the fusion is made. If the quality is good, then the path ends, and the updated fusion image can now be used by the operator to visualize the tumor or to carry out an intervention (such as a needle insertion). Otherwise, the path returns to block <NUM> to reattempt to update the fusion image.

Referring to <FIG>, a diagram shows ultrasound imaging overlap to describe the computation of success parameters in the pose analyzer unit <NUM> of <FIG>. Image overlap and relative pose can be computed as success parameters (shown in <FIG> in 2D for simplicity). Ultrasonic probes <NUM>, <NUM> are each positioned to provide different views <NUM> and <NUM>, respectively. Using a pose of the baseline 3DUS (transform: TUS2EM_base (US to EM registration for a baseline image)) (view <NUM>) and of a current 3DUS view (transform: TUS2EM_current (US to EM registration for a current image)) (view <NUM>), a relative pose transform Tcurrent2base (current image to baseline image fusion) = inv(Tus2EM_base) · TUS2EM_current between two views <NUM> and <NUM> is computed. In this example, view <NUM> includes a baseline 3DUS pose and view <NUM> includes a current 3DUS pose. Known ultrasound fields of view (FOV) (sector images <NUM> and/or <NUM>) are employed to compute overlap of a hatched image area <NUM>. An image angle difference, α, is computed directly from the relative pose transform Tcurrent2base. The success parameters may include the angle (α) and the hatched area <NUM>. If the angle α = <NUM> and the hatched area <NUM> coincides with the views <NUM>, <NUM>, then there is no motion to be compensated for. Success parameters may also include parameters, such as, e.g., brightness, contrast or other image features, extracted from the 2D US images acquired in the different views.

Referring again to <FIG> with continued reference to <FIG>, the system <NUM> may provide feedback and guidance for optimizing the success parameters of the ultrasound view. To this end, the pose analyzer unit <NUM> will be connected to the pose guidance unit <NUM> to compute the ultrasound probe motion needed to increase or optimize the parameters. This information is passed onto the display <NUM> to show the instructions to the operator (e.g., "move left", "rotate clockwise"). The pose guidance unit <NUM> may use the prior static image <NUM> (e.g., CT) and information derived from it (e.g., skin surface) to determine ultrasound poses that are acceptable (e.g., ultrasound probe touching skin).

In another embodiment, the system <NUM> may provide feedback and guidance to optimize the success parameters of an ultrasound view that images the user-provided target area. To this end, the interface <NUM> will permit the operator to enter a target area in the 3DUS baseline <NUM> or prior static image <NUM>. The information is passed on to the pose guidance unit <NUM> to compute guidance toward a view that images that target area while maximizing the success parameters.

The pose optimization problem can be computationally solved by the pose analyzer unit <NUM> by considering the probe positions and rotations as input parameters (possibly constrained to positions on the patient's skin, as derived from the prior static image <NUM>) that are to be optimized, and defining a cost function, f, that is to be minimized, which is inversely related to the likelihood of motion compensation succeeding for a 3DUS acquired at the current pose. One suitable cost function includes: <MAT>.

For the embodiment using the user-provided target area, the cost function can be modified to reflect the requirement to image the target area at the same time. For example, <MAT> where T(pi) is a unit step function that is <NUM> if the target area is fully in the current field of view, and very small (e.g., 1e-<NUM>) otherwise, such that the "cost" becomes prohibitively large in f_B(pi).

Using the cost function or functions, the user may move the US probe and be given audio, visual, haptic, etc. feedback on the display <NUM> or from the interface <NUM> as the target area is approached to guide the operator to an optimal pose for motion compensation. The cost function may be configured to evaluate overlap and/or rotation between different image fields of view as well as image parameters in determining image similarity, e.g., from 2D/preliminary and/or baseline images. The pose guidance unit <NUM> may also employ the cost function or functions for optimal pose and image similarities to provide guidance commands to the user.

Referring to <FIG>, a method for fusing images to account for motion compensation is illustratively shown in accordance with the present principles. In block <NUM>, a baseline image is captured. In block <NUM>, live images of a target area are obtained. The live images may be 2D images or provide a preliminary image. In block <NUM>, an imaging instrument is tracked to obtain a pose for capturing the baseline image and to obtain a pose for a current view of the live images such that a tracking system has its coordinate system registered with the baseline image and the live images. In block <NUM>, a pose for a current view is analyzed to compare field of view differences between the pose for the baseline image and the pose for the current view using the tracking system to generate success parameters. In addition, parameters/image features from the live image and baseline image may be computed and compared as well. In block <NUM>, the success parameters are conveyed to provide feedback on optimal image acquisition for motion compensation between the baseline image and the current view. The success parameters measure field of view overlap and pose similarity between the baseline images and the current view image. The success parameters may include different parameters for achieving overlap, pose similarity and image similarity, e.g., angles, positions, areas, percentages, image contrast, brightness or other quantities or features.

In block <NUM>, if the success parameters are adequate, a new image at the pose of the current view is acquired. This may be a full blown 3D image as opposed to a 2D image or preliminary image. The adequacy of the success parameters may be determined by a user or may be set automatically or as a default. It is determined whether a pose provides a field of view for a current pose comparable to that of the pose of the baseline image to permit a user to replicate the baseline image field of view. For example, a threshold, say,. e.g., <NUM>% overlap, between a baseline field of view and a current field of view may be set to determine adequacy. Other criteria are also contemplated.

In block <NUM>, the baseline images and the current view image may be registered. In block <NUM>, static images (e.g., CT, MRI, etc.) may be registered with one or more of the baseline images and the current image(s). This registration may occur at any time and preferably occurs during an early stage (e.g., planning) with regards to the baseline image being registered to the static image.

In block <NUM>, direction is provided to a user to achieve a satisfactory pose for the current view image using a pose guidance unit. The pose guidance unit may include and compute a cost function to evaluate overlap between different image fields of view. The cost function may also consider image parameters between, e.g., a live image (e.g., a preliminary 2D image) and the baseline image. The image parameters may include contrast, brightness, content information, etc..

In block <NUM>, user actions are guided through feedback activity to achieve a satisfactory pose for the current view image. The feedback activity may include at least one of a visual signal (e.g., flash, images), text commands, audio signals (e.g., beeping, voice commands), haptic signals (e.g., vibration intensity, vibration changes), etc..

In interpreting the appended claims, it should be understood that:.

Claim 1:
An ultrasound fusion imaging system (<NUM>) for fusing images to account for motion compensation, comprising:
an ultrasound imaging probe (<NUM>) configured to capture a baseline image and live images of a target area;
a live tracking system (<NUM>) configured to track the ultrasound imaging probe (<NUM> ) during capture of the baseline image and the live images, the live tracking system having a coordinate system registered with the baseline image and the live images; and
a pose analyzer unit (<NUM>) configured to compare field of view differences between the pose of the probe (<NUM>) for the baseline image and the pose of the probe (<NUM>) for a current view of a live image determined using the live tracking system to generate success parameters representing a measure of overlap between the baseline image and the current view image and a similarity between the poses, and to convey the success parameters to an operator to provide feedback on image acquisition for motion compensation between the baseline image and the current view image,
the system further comprising a motion compensation registration unit (<NUM>) to register the baseline image with the current view image.