Patent Description:
Example aspects described herein relate to medical image processing, and, in particular, relate to a procedure, apparatus, system, and computer program, for facilitating interactive pre-operative assessments using manipulative, visually perceptible models of anatomical structures.

Liver tumor resection can be an efficient treatment method for addressing liver cancer. Before surgery, physicians need to carefully evaluate a hepatic lesion or tumor to be re-sectioned, the volume of the expected remaining liver segments, how a proposed resection is going to affect nearby vascular structures and corresponding blood supply/drainage regions, and how the resection will affect biliary systems, for example. Such comprehensive assessments can be helpful in determining an optimal operative plan.

Whether or not a resection will be suitable for a particular patient can depend on a number of factors, such as, for example, the location of a tumor, its size, and the quality of expected post-operative liver function. Pre-surgical planning is therefore important. During the planning stage, a surgeon has to understand the spatial relationships between tumors and surrounding vessel structures. To enable a surgeon to do so, tools and systems that are capable of assisting and enabling the surgeon to define a surface that separates tumors and corresponding diseased tissue from the rest of healthy liver tissue, are needed, as are tools for measuring the size and/or volume of the affected areas and/or distances between the affected areas and nearby structures. It has been a challenge in the past to provide tools having such capabilities, and which can be operated intuitively and in a user work-flow-friendly manner. As is commonly known, ease-of-use is an important factor in terms of whether a physician can effectively evaluate different scenarios to determine a best resection procedure based on various information from different sources.

Traditional, available commercial products in this area are mostly based primarily on 2D display technology that provides a user with a view having axial slices. The user can operate a user interface to scroll up and down within the display to view different structures in a sliced arrangement along an axial direction. Such conventional products also provide drawing tools that enable a user to draw separating curves in different slices. A corresponding 3D separating surface can then be interpolated computationally based on multiple curves drawn in different slices, but is not visible. With such a generated separating surface, the volume of the underlying target object or a virtual resection within the confines of the separating surface can be calculated. Because such a separating surface is not visible, the user is required to imagine in his/her mind, while making the 2D drawings, the possible outcome in 3D space. Thus the conventional 2D approach is not intuitive and is difficult to apply.

<NPL>] refers to a virtual reality environment that purportedly enables a user to perform some 3D operations directly in a 3D virtual reality space. However, the environment requires users to wear stereo goggles and to use special 3D tracking devices which are not generally available. Other prior art is <CIT>; <CIT>; <CIT>; <NPL>; <NPL>); <NPL>.

Existing limitations associated with the foregoing, and other limitations, can be overcome by a procedure for pre-operating assessment of one or more anatomical structures generated from medical images and provided in a rendered 3D space, and an imaging system, apparatus, and computer program, that operate in accordance with the procedure, wherein each has a 3D capability not limited by the above limitations. In one example embodiment, the procedure comprises providing one or more safety margin indicators in the rendered 3D space, each having a shape corresponding to that of a respective one of the anatomical structures within an organ and having a predetermined size of safety margin from the respective one of the anatomical structures. The procedure also comprises manipulating at least one of the shape and predetermined size of safety margin of at least one of the safety margin indicators in the rendered 3D space, and immediately providing a rendering in the 3D space of a manipulated version of the at least one safety margin indicator.

The safety margin indicator can be provided in a 2D view on the medical images, rendered in its original or processed form, and in a 3D view, and the manipulating can include adjusting at least one of the shape and location of the at least one safety margin indicator.

According to one exemplary embodiment, the procedure further comprises providing at least one numerical dimension of the safety margin indicator, and additionally comprises changing at least one of a color and an appearance of the 3D rendering of at least one of the anatomical structures, and/or by masking one or more structures.

A procedure according to another exemplary embodiment herein defines at least one cutting surface to resect one or more medical anatomical structures using an imaging system. The procedure comprises forming at least one cutting surface in one of a 2D view and a 3D view of a rendered 3D space in the imaging system, automatically providing a 3D rendering or a 2D rendering, respectively, of the at least one cutting surface, interactively manipulating the at least one cutting surface to adjust at least one of its orientation, shape, and location, using graphical handlers, and immediately providing a rendering in the 3D space of a manipulated version of the at least one cutting surface.

The procedure also can include identifying corresponding parts of the anatomical structures between the 2D and 3D views, and the forming can be performed using at least one of a plane and a template defining the cutting surface.

The forming forms at least one cutting surface in at least one of the anatomical structures, and the procedure also can vary an orientation of the at least one anatomical structure, wherein the forming forms at least one cutting surface in the anatomical structure, for each orientation.

As a further example, the procedure can comprise separating the at least one anatomical structure into at least two parts, along the at least one cutting surface, and/or re-coupling the at least two parts of the anatomical structure. At least one of a color and an appearance of at least one of the parts also can be changed.

Additionally, sub-structures (e.g., vascular branches or the like) located in an interior of the anatomical structure model, can be separated into portions, and an indicator can be provided of at least one dimensional characteristic of at least one of the sub-structures. In one example, the dimensional characteristic includes a diameter of a cross section of the at least one sub-structure, that cross section can be interactively selected at a location along the at least one sub-structure, and, in one example embodiment, the at least one sub-structure can be automatically rotated in response to the selecting.

The teachings claimed and/or described herein are further described in terms of exemplary embodiments. These exemplary embodiments are described in detail with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals represent similar structures throughout the several views of the drawings.

Exemplary aspects described herein relate to methods, systems, apparatuses, and computer programs, that facilitate pre-operative assessments in an intuitive, interactive, and real-time manner. These exemplary aspects can be used to facilitate hepatic lesion/tumor resection planning based upon quantitative assessment and evaluation of anatomic structures and other tissue, such as, for example (and without limitation), the liver, liver segments, vascular structures, vascular territory, biliary structures, and hepatic lesions, as well as the spatial relationships among them. These aspects also can provide an intuitive 3D working environment and can be used to fuse 3D and 2D information to perform efficient resection evaluation and analysis.

<FIG> depicts a construct of an exemplary medical imaging system <NUM> according to an exemplary embodiment herein, and which can be used to conduct, among other things, pre-operative assessments. The system <NUM> comprises one or more user interfaces such as a display device <NUM>, and also comprises a Graphical user interface (GUI) controller <NUM>, a 3D scene manipulating mechanism <NUM>, a 2D/3D transformation mechanism <NUM>, a 3D scene rendering mechanism <NUM>, a safety margin indicator manipulation mechanism <NUM>, a separating surface defining mechanism <NUM>, a cut spread out mechanism <NUM>, a volume/segmented object input module <NUM>, and a cross- sectional cut piece mechanism <NUM>. A user <NUM> may interact with the system <NUM> via at least one user interface, including an input user interface and an output user interface. In one example embodiment, the input user interface can include at least one of a user-operable slider control, a track ball, a keyboard, control stick, an input user interface provided by way of the display device <NUM>, and/or any other suitable type of user input mechanism, and the output user interface can be provided by way of the device <NUM> and/or any other suitable type of interface capable of providing a user-perceptible output. For convenience, the input user interface is identified in <FIG> by reference numeral <NUM> associated with a keyboard, although it should be noted that the user input interface can include any one or more of the foregoing types of input user interfaces, and is not limited to a keyboard.

When the user <NUM> operates the input user interface (also referred to interchangeably hereinafter as a "user interface") <NUM> to manipulate a scene, such as a 3D scene, the controller <NUM> (which can control interaction between the system <NUM> and user <NUM>) activates the 3D scene manipulating mechanism <NUM>, which, in tum, invokes a 3D scene rendering mechanism <NUM>. The system <NUM> can load in volume data and segmented 3D objects from volume/segmented object input module <NUM>, and render such objects via the 3D scene rendering mechanism <NUM>. The mechanism <NUM> operates based on outputs from components <NUM> and <NUM> to manipulate a 3D scene rendered on display device <NUM>. Data rendering may be carried out dynamically and data to be rendered may also change. For example, when user <NUM> rotates a 3D volume presented on display device <NUM>, the original rendering may need to be updated (and can be updated) to reflect the change in appearance of the data due to the rotation. Accordingly, due to the rotation, 2D cross sections to be rendered may also change.

The user <NUM> also can specify the performance of various other functions, including a function to define separating surfaces, safety margins, and the like, as will be described in more detail below. When the user <NUM> selects any such functions, such as in a 2D rendering presented on display device <NUM>, the system <NUM> converts the user's input signal using 2D/3D transformation mechanism <NUM> and sends the converted signal to one or more corresponding ones of a safety margin manipulator mechanism <NUM>, separating surface defining mechanism <NUM>, a cut spread out mechanism <NUM>, and a cross-sectional cut piece measurement mechanism <NUM>, depending on the specific function specified by the user <NUM>. If the user <NUM> specifies a resection function, for example, the user <NUM> may request the system <NUM> to load in volume data and segmented 3D objects from volume/segmented object input module <NUM>, and render such objects via the 3D scene rendering mechanism <NUM>, based on the information from the module <NUM> and outputs from one or more of the applicable modules <NUM>, <NUM>, <NUM>, <NUM>. The mechanism <NUM> operates based on outputs from components <NUM> and one or more of the components <NUM>, <NUM>, <NUM>, and <NUM>, to conduct the function specified by the user <NUM>.

In the illustrated embodiment, display device <NUM> is part of a data processing system that includes the device <NUM>, the user interface <NUM>, and at least one storage device (memory) <NUM>, each of which is bidirectionally coupled to a processor <NUM> of the data processing system. Processor <NUM> also can be coupled to external Input/Output (I/<NUM>) devices (not shown) and a communications interface device (not shown), at least one of which enables the data processing system to communicate with other elements external thereto, such as, for example, a network.

The storage device <NUM> has a computer-readable medium, and is used by the processor <NUM> to store and read/write data, as well as computer-readable program instructions used to implement the procedure(s) described herein and shown in the accompanying drawing(s). In operation, processor <NUM> executes the program instructions to perform any of the example procedure(s) described herein, for operating the overall system <NUM>.

The modules <NUM> and <NUM> to <NUM> of <FIG> each may include software, hardware, or a combination thereof, and at least some of those elements <NUM> and <NUM> to <NUM> (although represented outside of the device <NUM> for illustrative purposes), may be incorporated within the data processing system. As one example, those modules may be embodied, at least in part, in the computer program instructions stored in the storage device <NUM>, or can be hardware components located in or in association with the processor <NUM>.

Having described the system <NUM> of <FIG>, an aspect relating to the display functionality of the system <NUM> will now be described in conjunction with <FIG> and <FIG>. Resection medical procedures performed on actual human or animal patients are, of course, conducted in physical 3D space. The exemplary aspects described herein can model such activities in a 3D model working environment on the display of a computing device (e.g., such as the display device <NUM> of the system <NUM> represented in <FIG>). <FIG> depicts a computing device display <NUM>, which, in one example, is a further representation of the display device <NUM> of <FIG>. The device <NUM> of <FIG> includes a 3D display region <NUM> (also referred to as a "working environment") wherein a 3D volume can be rendered, and a separate display region that includes plural sub-regions <NUM> in which respective 2D cross-sectional views can be displayed, each corresponding to the volume represented in region <NUM>, as viewed from particular perspectives (i.e., at different angles). In one exemplary embodiment, the views shown in sub-regions <NUM> are taken from perspectives that are perpendicular to (and looking in general towards) respective ones of three axes represented in region <NUM>, and are orthogonal to one another, although in or embodiments, other, non-orthogonal views can be provided.

The volume that may be rendered in the region <NUM> can be any applicable object being viewed, and in the context of the present exemplary description, may include one or more anatomic structures or organs, such as, by example only, at least one or more of a liver, a segmented liver after liver resection, one or more related anatomical structures such as lesions (e.g., within the organ(s)), one or more hepatic veins, portal veins, hepatic arteries, biliary tracts, liver lobes, vascular territories, biliary territories and the like. The displayed inter-spatial relationships among the structures are clearly visible to a viewer. The anatomic structures in the view may be pre-segmented based on scanned volume data either automatically, or via an interactive manual method.

A user, such as user <NUM> (<FIG>), can operate the system <NUM> to rotate and/or zoom a 3D scene (on display device <NUM>) in order to visualize segmented objects from various perspectives to gain a more substantial understanding of the spatial relationships among different segmented structures. The appearance of a segmented object may also be adjusted by the user <NUM> to opaque, transparent, invisible or the like, using, e.g., associated object controls. In some embodiments, a user may show or hide certain objects to reveal areas that are occluded or see the objects within transparent objects. Examples of these aspects will be further described below.

An exemplary embodiment for enabling a user to define a safety margin will now be described, with reference to <FIG>, which shows a representation of a safety margin indicator <NUM> surrounding a lesion or tumor <NUM> of an anatomical structure <NUM> (e.g., a liver), depicted in 3D space (i.e., in region <NUM> of <FIG>). A safety margin can be useful to define an area for resection, for example, that includes both the lesion or tumor <NUM> and an additional buffer area around it for additional precautionary reasons. The safety margin indicator <NUM> shown in the example of <FIG> is circular in shape or an ovoid; however, it is within the scope of the present invention for the safety margin indicator <NUM> to have any other suitable shape. As a non-limiting and non-exclusive example, <FIG> shows a safety margin indicator <NUM> as has having a dilated shape, and one that follows the overall shape of the lesion or tumor <NUM>. It also should be understood, that in one example, the safety margin indicator can also be provided in a 2D view of <FIG> on medical images, rendered in its original or processed form, in addition to the 3D view.

In one example embodiment herein, the size and shape of the safety margin indicator <NUM> can be specified and adjusted by the user <NUM> by way of a user interface (e.g., <NUM>) (using, e.g., module <NUM> of <FIG>), and the size and shape also can be adjusted/moved by, for example, the user selecting and dragging a border of the indicator <NUM> to increase (or decrease) its size, as represented by reference numeral <NUM> in <FIG>. A center of the safety margin indicator <NUM> also can be adjusted by the user operating the user interface <NUM> to select and drag/move the indicator <NUM> object in 3D (or 2D) space. The safety margin indicator <NUM> is displayable in both 3D and 2D in the regions <NUM> and <NUM>, respectively, and the size, shape, and/or location of the indicator <NUM>, where adjusted by the user (via, e.g., the user interface <NUM>), are updated in the display region(s) substantially instantaneously by the system <NUM>. The foregoing capabilities for adjusting the safety margin indicator <NUM> can be beneficial when, for example, multiple lesions/tumors, requiring differently sized and/or shaped safety margin indicators, are to be re-sectioned in a single operative setting. Real time visual feedback provided by the system <NUM> is especially useful when the user interactively adjusts the size or the like of the safety margin indicator <NUM> as described above. In one example embodiment, this is achieved by setting a zoom factor in a rendering transformation of a 3D computer graphics library (e.g., OpenGL) corresponding to the desired, adjusted size or shape or the like of safety margin indicator <NUM>. With this approach, it is not necessary to use a more time consuming calculation of directly adjusting the size and shape of the safety margin indicator during interactive session. Only when the users specifies the suitable size or other type of adjustment from real time visual feedback, does the system <NUM> effect the adjustment, in one example.

In accordance with one example embodiment, the numerical dimensions (e.g., diameter, circumference, length, width, location, etc.) of the safety margin indicator <NUM> are shown in one or more parts (regions <NUM>, <NUM>) of the display device <NUM>. The computer program stored in the storage device <NUM> of the system <NUM> operates to continuously monitor those dimensions (for example, this can be accomplished using known dimension measuring techniques), and to cause them to be presented on the display device <NUM>. In other embodiments, they are determined and presented periodically or upon user request.

In a further exemplary embodiment herein, the system <NUM> can be operated by the user <NUM> to modify the appearance of one or more anatomical structures <NUM>, <NUM> depicted on the display device <NUM>, including those within the safety margin indicator <NUM> (or, in other embodiments, outside of the safety margin indicator). For example, the user <NUM> can operate the user interface <NUM> to specify that one or more selected anatomical structures (e.g., <NUM>, <NUM>) be masked and/or presented in a different color or appearance (as non-exclusive examples, opaque, transparent, invisible, hidden) than those of one or more other displayed structures (e.g., lesion <NUM> and/or other intra-organ anatomical structures can be masked from organ <NUM>, or vice versa), depending on the criteria specified by the user <NUM>, and the system <NUM> responds by presenting the selected structure(s) in the specified color or appearance. By virtue of this capability, the user can readily discern in an intuitive manner, which structures (e.g., <NUM>, <NUM>) are included within or otherwise affected by the safety margin indicator <NUM>. In one example, in cases where the safety margin indicator <NUM> is moved by the user (e.g., by way of the user interface <NUM>), then as the indicator <NUM> moves, the color and/or appearance of anatomical structure portions touched or overlapped by the indicator <NUM> changes, and/or the color and/or appearance of the structures change when they are disposed within the border of the safety margin indicator <NUM>.

Also, in an exemplary embodiment, one or more quantified measurements such as volumes or other physical characteristics of the parts of structures (e.g., <NUM>,<NUM>) inside the safety margin indicator <NUM> are calculated and displayed on the display device <NUM> whenever a user places the indicator <NUM> over them. In other words, the system <NUM> calculates (using, e.g., known volume calculation techniques) the volume of any portion of a structure (e.g., <NUM>, <NUM>) included within the margin indicator <NUM>, or, in another example, of any portion touched or overlapped by the indicator <NUM>. In another embodiment, the system <NUM> performs such a function in response to the user simply operating the user interface <NUM> to select the structure (e.g., <NUM>,<NUM>), whether or not any part of the structure is located within the safety margin indicator <NUM>, or the function is performed periodically or otherwise automatically. In an example embodiment, the system <NUM> displays or otherwise outputs a quantified measurement of a masked out anatomical structure, organ, and/or a remnant thereof.

In a further exemplary embodiment, the system <NUM> can calculate a distance dl between at least one object of interest (e.g., <NUM>,<NUM>) to at least one neighboring or other structure (e.g., <NUM> of <FIG>) specified by the user <NUM>, and that distance is displayed on the display device <NUM>. For example, the user <NUM> can select that object (e.g., <NUM>,<NUM>) and structure (e.g., <NUM>) by way of the user interface <NUM>, and specify that the distance between them be calculated, in which case the system <NUM> responds by calculating the distance (using, e.g., known distance calculating techniques) and presenting it on the display device <NUM>. In another example, the system <NUM> is responsive to the user <NUM> specifying that the system <NUM> display a distance between the object of interest (e.g., <NUM>, <NUM>) and a nearest part of a neighboring structure (e.g., <NUM>), by calculating the distance between the object of interest and the nearest structure, and then presenting the calculated distance on the display device <NUM>. Also, in one example, this calculation determines the distance between closest parts of the object (e.g., <NUM>, <NUM>) and structure (e.g., <NUM>). Additionally, in one example embodiment, the system <NUM> calculates and provides an indication of, a distance between two or more safety margins, and/or a distance between at least one safety margin indicator and, for example, at least one anatomical structure within an organ, the organ itself, and/or other anatomic structures. These functions can be performed automatically or in response to user instructions.

Additionally in an example embodiment, the distance between the safety margin indicator <NUM> and the structure that it surrounds, such as, e.g., structure <NUM>, can be specified by the user (e.g., via the user interface), and that safety margin indicator automatically becomes adapted so that it is located at that distance from the structure (i.e., the size of the safety margin is adapted). As an example, in <FIG> the safety margin indicator is shown adapted so that it mirrors the shape of the structure <NUM>, but each part of the indicator is separated from an adjacent surface of the structure <NUM> by a predetermined distance (e.g., one specified by a user). The shape of the safety margin indicator can be adapted automatically by the system, or can be adapted by the user. Such adjustments can be effected multiple times, depending on, for example, user selection. Additionally, in one example embodiment, more than one safety margin indicator can be displayed in each view, and the system is not limited to showing only a single safety margin indicator. Plural safety margin indicators being shown at once can be useful where, for example, multiple lesions or the like are being analyzed/resected using those indicators.

<FIG> is an exemplary flow diagram for manipulating a safety margin indicator, such as indicator <NUM>. At step <NUM>, the user <NUM> operates the user interface <NUM> (<FIG>) to input information specifying the manner in which the user desires to manipulate the safety margin indicator <NUM>. For example, this may include the user specifying information such as a desired size, dimension(s), location, and/or shape of the indicator <NUM>. At step <NUM>, the system <NUM> responds by (where needed) converting the input information using a transformation mechanism to parameters that can be used to manipulate, or fit to, the data structure type of the safety margin indicator <NUM>. Then, at step <NUM> the data structure defining the safety margin indicator <NUM> is modified, and the representation of the indicator <NUM> presented on the display device <NUM> becomes modified as well, both in accordance with the information specified at step <NUM>. The resulting, modified safety margin indicator <NUM> is presented on the display device <NUM> (step <NUM>). Also, in one example, in a case where the user specified at step <NUM> to change the appearance and/or color (e.g., mask, transparency, or the like, as described above) of displayed structures touching, overlapping, or within the borders of the indicator <NUM>, then at step <NUM> the color and/or appearance of any such structure(s) is modified accordingly (e.g., in the 3D rendering) on device <NUM>. This can be useful, for example, to mask or otherwise flag affected displayed vascular systems. In one example embodiment, masking can be performed using a standard Boolean AND operation in binary volume space, although in other embodiments, masking can be conducted using other suitable techniques.

Having described the manner in which a safety margin indicator can be formed and manipulated, another exemplary aspect herein will now be described. In accordance with this exemplary aspect, the system <NUM> enables a user to define one or more separating or resection surfaces (either of which is also referred to herein as a "cutting surface"), by using a safety margin indicator <NUM> as a separating (or resection) surface, and another user-defined separating or resection surface, as will be described below. By virtue of this exemplary aspect, the user can operate the user interface <NUM> to cause the system <NUM> to automatically generate an optimal cutting surface that enables resection to be performed in an efficient and optimal manner. For example, an optimal cutting surface defined by the user may be one that the user considers will leave a largest volume of remnant structure, while still enabling a sufficient or safe amount of lesion or tumor to be removed. As another example, an optimal cutting surface may one deemed to cause minimal potential encroachment to an underlying vascular system, and may be assessed by evaluating the spatial relationship between a user-confirmed safety margin indicator and a liver parenchyrna or the vascular system.

Reference is now made to <FIG> for a description of the manner in which a separating or resection surface can be defined, according to one exemplary embodiment herein. In this example, the user can operate the user interface <NUM> to draw curves to form one or more slices (also referred to herein as "cutting surfaces") <NUM>,<NUM>,<NUM> in a 2D view <NUM> (which, in one example, further represents the region <NUM> of <FIG>). In response, the system <NUM> (using, e.g., module <NUM>) interpolates a separating surface <NUM> in a 3D view <NUM> (which, in one example, further represents the region <NUM> of <FIG>) based on the drawn 2D curve(s). In accordance with an example aspect herein, the resulting 3D view <NUM> is rendered in real time as the 2D curves are drawn, such that there is a substantially simultaneous rendering of the interpolated 3D surface with respect to the object-of-interest and the object to be re-sectioned. In this manner, an instant 3D visual feedback is made visible to the user while the user is drawing 2D curves, and thus the user does not need to mentally envision the 3D separating surface on his/her own. In one example embodiment, curve interpolation can be done by parametrization. For example, assume there is one curve A in a slice <NUM> and one curve B in a slice <NUM>. Curve A can be parametrized from O to <NUM> so that each point on the curve has a corresponding parametric value between O to <NUM>. Similarly, the same can be done for curve B. An interpolated curve C in a slice <NUM> can be obtained by using the corresponding points with the same parametric value from curve A and curve B. The slice of the curve C can be linearly interpolated based on the slice positions of slice <NUM> and slice <NUM> together with the distances of slice <NUM> to slice <NUM> and slice <NUM>. These curves can then be used to construct a 3D surface by stitching or coupling corresponding parametric points between every adjacent pair of curves. In one example, a standard computer graphics technique called triangulation can be used for stitching these curves into a mesh to form a surface.

The system <NUM> also enables the user to navigate between 2D and 3D views, and to navigate to specific location(s) by cross-referencing the 3D and 2D views. For example, if the user considers a certain part of a resection surface to be unsatisfactory in one view, such as the 3D view <NUM>, then by selecting that part in the 3D view, he/she can be directed by the system <NUM> to the corresponding 2D axial, saggital, and/or coronal view <NUM>, and adjust the surface if deemed needed. The user also can toggle back and forth between the 2D and 3D views <NUM>,<NUM> by operating the system <NUM>. If the user selects a 2D image slice of view <NUM>, for example, the corresponding position can be visualized in the 3D space as well, in view <NUM>. The safety margin indicator (e.g., <NUM> of <FIG>), and vascular/biliary territories, can be displayed in both the 3D view <NUM> and 2D view <NUM>, for use as reference when the user is defining a resection surface.

In another exemplary embodiment, a separating or resection surface can be formed using a plane, such as plane <NUM> illustrated in <FIG>. In the example of <FIG>, the plane <NUM> is a 3D plane that passes through an object <NUM> to be re- sectioned, and this forms a cutting plane (e.g., resection or separating plane). By operating the user interface <NUM> (<FIG>), for example, the user <NUM> can adjust the position, orientation, size, thickness, and/or shape of the plane in the 3D view (e.g., view <NUM> of <FIG>). In one example, the user also can operate the user interface <NUM> to bend or otherwise deform the shape of the plane <NUM> (see, e.g., bended plane <NUM>), from its borders (or from another part of the plane <NUM>), using graphical handlers (see, e.g., the arrow represented in <FIG>) or some other mechanism. In this manner, the user <NUM> can perform direct manipulation of the separating plane <NUM> (or surface) to place it at a desired location and orientation, and also can vary its size, shape, thickness, and dimensions. The plane <NUM> can be used in conjunction with both 2D and 3D views (see, e.g., <FIG> and <FIG>), and thus the user <NUM> can interactively make local adjustments to the separation plane <NUM> in either a 2D or 3D view, in order to, for example, effect separations and the like in a manner that minimizes potential damage to nearby vital anatomies.

In another exemplary embodiment, a user may draw several 3D contours from different viewing angles to define a separation area. For example, while observing an object-of-interest with respect to an object to be re-sectioned in a 3D view, the user may draw a contour in one viewing angle that can separate the object-of-interest from the rest with extended extrusion of the drawn contour along the viewing direction. The user may then rotate the 3D view by another angle and draw another contour to exclude another part. By repeating this process, a final separated piece is the intersection (of several cylinders, in the below example) based on the drawn contours in different viewing angles.

This example can be further understood in view of <FIG>. In that Figure, and merely for ease of understanding, a cylinder <NUM> is depicted in association with each depicted instance of an anatomical structure <NUM>, lesion or tumor <NUM>, and safety margin indicator <NUM>. The cylinder <NUM> is shown merely for assisting the reader to understand the 3D inter-relationships of the three views of <FIG> (labeled as <NUM>, <NUM>, and <NUM>), which represent the mentioned components as viewed from respective, different angles, and in actual usage the cylinders <NUM> need not be shown. As shown in the upper view <NUM> of the Figure, a contour <NUM> is provided in one viewing angle <NUM> (for example, by the user operating the user interface <NUM>). View <NUM> shows the components of view <NUM>, after having been rotated (vertically) by the user by about <NUM> degrees. In the illustrated example, a second contour <NUM> drawn by the user (e.g., after such rotation) is provided. View <NUM> represents the components rotated (horizontally) by about <NUM> degrees by, e.g., the user. The structure <NUM> is now confined by the two cylinders <NUM>. By being able to manipulate and rotate the components in the respective 3D views directly in this manner, more flexibility can be provided in defining the shape of the separating surface, and the user can be more sure that the separating surface is defined correctly throughout the volume of the structure being separated, because he/she can move/manipulate/rotate the volume and view it from various perspectives. In conjunction with this approach, in one exemplary embodiment 2D Multiple-planar resection (MPR) images, taken from perspectives that are offset perpendicularly (or by some other angle) from the views <NUM>, <NUM>, and <NUM>, can also be displayed (e.g., in region <NUM> of <FIG>), so that the user also can use such view(s) as well for reference when defining a cutting contour.

As can be appreciated in view of the above description, the system <NUM> herein provides a user with great flexibility with respect to how to define separating and resections surfaces. Indeed, in a further exemplary embodiment, the system <NUM> can be operated to employ one or more predefined templates from among a plurality of available, stored templates, for use in resection planning. The manner in which the template(s) are selected can be done either by user selection, or automatically based on predefined matching criteria. The templates may form different shapes from one another and from any of the shapes described above, and may include, by example only (and without limitation), one or more spherical templates, wedge templates, ellipsoidal templates, bowl template, etc., or any combination(s) thereof.

By operating the user interface <NUM>, the user <NUM> can adjust the position, orientation, size, thickness, and/or shape of the template in the 3D view (e.g., <NUM> of <FIG>), in a similar manner as described above for the plane <NUM> of <FIG>. In one example, the user <NUM> also can operate the user interface <NUM> to bend or otherwise deform the original shape of the template, from its borders (or from another part of the template), in a similar manner as for plane <NUM> described above, and the template can define a cutting surface/area. In this manner (and in a similar manner as described above for plane <NUM>), the user <NUM> can perform direct manipulation to place the separating template at a desired location and orientation, and also can vary its size, shape, thickness, and dimensions. The user can interactively make local adjustments to the separation template in either a 2D or 3D view (e.g., <NUM>, <NUM> o <FIG>), in order to, for example, effect separations and the like in a manner that minimizes potential damage to nearby vital anatomies. As but one example, the user can operate the user interface <NUM> to drag and/or push on one or more surfaces of the template to deform the template to desired shape, and/or drag or push on contours displayed on the 2D cross- sectional display (e.g., <NUM> of <FIG>) to effect template shape adjustment. This functionality of the system <NUM> can be useful in cases where, for example, an atypical separating or resection surface is needed in order to avoid interfering with a vascular branch or the like.

Once a target object is separated (or re-sectioned) by any of the above techniques, the user (e.g., <NUM>) can operate the user interface <NUM> (<FIG>) to manipulate the separated pieces as desired in 2D and/or 3D space (e.g., via views <NUM>, <NUM> of <FIG>), in order to discern further desired information as deemed necessary to carry out the applicable operative procedure.

Having described exemplary embodiments for defining resection and separating surfaces, exemplary embodiments regarding the performance of resections and separations will now be described.

<FIG> represents how a piece can be removed from (and replaced to) a structure in 3D space, according to an exemplary embodiment herein. A user can operate the user interface of the system <NUM> to form a separating surface <NUM> in any of the above-described manners, and then to separate a portion <NUM> from an anatomical structure <NUM> (this procedure is referred to herein, for convenience, as an "opening" operation <NUM>). As shown in <FIG>, reference numeral <NUM> represents an original state in which the portion <NUM> is not yet removed from the structure <NUM>, and reference numeral <NUM> represents a state after which the portion <NUM> has been removed from the structure <NUM>, leaving an opening <NUM>. Thus, the system <NUM> (using, e.g., module <NUM>) effects those operations <NUM> and <NUM>, and enables the user <NUM> to perceive the components <NUM> and <NUM> as coupled together (<NUM>), and as separated from one another (<NUM>) in animated form. This procedure also can de-couple and re-couple sub-structures (e.g., intra-organ anatomical structures) within the structure <NUM>, and can enable the user to view removed or remnant portions with corresponding intra-organ anatomies together.

In an exemplary embodiment herein, the system <NUM> can be operated by the user <NUM> to modify the appearance of one or more of the portions <NUM>, <NUM>, <NUM>, and/or one or more portions of those components. For example, the user <NUM> can operate the user interface <NUM> to specify that one or more selected ones of those components be masked and/or presented in a different color or appearance than one or more other ones of those components, as described above, depending on the particular criteria specified by the user <NUM>, and the system <NUM> responds accordingly. Thus, as an example , when the two parts <NUM> and <NUM> are separated from one another, the user <NUM>, by virtue of such functionality of the system <NUM>, can view the surface <NUM> of the cut portion <NUM> in different appearance and/or color than those of components <NUM> and <NUM>, and can obtain a more direct visual understanding and interpretation about the cut surface area. The user also can operate the user interface <NUM> to place the portion <NUM> back into the opening <NUM> (for convenience, this procedure is referred to as a "closing" operation <NUM>) to view the two pieces <NUM> and <NUM> coupled back together again in animated form.

In one example, the "opening" and "closing" procedures can be performed (e.g., by module <NUM>) using a computer program loop with small changes of rendering parameters such as positions of the two pieces, although in other embodiments, other techniques can be employed.

In an additional exemplary embodiment herein, the anatomic structures embedded inside separated components also can be separated into two or more portions. For example, referring to <FIG>, the user <NUM> (<FIG>) can operate the user interface <NUM> (<FIG>) to specify that an anatomical structure <NUM>, such as, e.g., a liver, and one or more selected components <NUM> disposed at least partially in an interior thereof, be separated into at least two parts, and the system <NUM> responds by performing such separations on display device <NUM>. Again, any one or more of those components specified by the user, and/or one or more portions thereof, such as, e.g., without limitation, a cross section <NUM>, can be masked and/or presented in a different color and/or appearance than one or more other ones of those components, either automatically, or in response to user selection.

Each portion of the anatomic structure(s) shown in <FIG>, and/or a selected cross section <NUM> thereof (such as an area to be affected by a resection), also can be measured in a similar manner as described above, for its volume, area, and size/dimensions, or the like. In other words, the system <NUM> calculates (using, e.g., known calculation techniques) those dimensional characteristics for any user-specified portion(s) of <FIG>. In response, the system <NUM> (using, e.g., module <NUM>) responds by calculating the applicable dimensional characteristics and presenting them on the display device <NUM>. For example, in a case where the user selects a particular cross-sectional area, such as one of the cross sectional areas <NUM>, the system <NUM> analyzes the shape of the applicable cross-sectional area <NUM> and computes a diameter thereof along one or more directions, such as, e.g., along a longest axis direction of the cross section (i.e., along a widest diameter). The direction of the computed diameter can be indicated on the display device <NUM> by using, for example, two 3D arrow indicators <NUM>, and the user can maneuver those indicators <NUM> to specify the specific diameter (and its direction) to be computed/displayed. Measurement <NUM> of the approximated diameter can be presented on the display device <NUM>. Additionally, the user <NUM> can operate the user interface <NUM> to interactively adjust the direction of the diameter along the cross-sectional area <NUM> to get a more meaningful measurement by, for example, dragging or otherwise moving the 3D arrow indicators <NUM> and turning them around, and the like, in order to take the diameters along different desired directions.

During a resection planning, in addition to determining the diameter of a resection or cut surface cross section, it also may be desired to obtain diameters along other cross sections, such, as, for example, other parts of a vascular branch of interest. Reference will now be made to <FIG> which depict an exemplary workflow representing how the diameter of a tube-like structure can be determined, according to an exemplary embodiment herein. Those Figures may represent respective views presented on display device <NUM> (<FIG>), in one example. It should be noted that, although described in the context of a tube-like structure, the below procedure is not limited for use only with such a structure, and it is within the scope hereof to determine the diameter of other types of structures as well.

In <FIG>, a thin tube-like structure <NUM> (such as a vessel) is depicted, residing in 3D space <NUM>. In <FIG>, the user operates the user interface <NUM> to select point <NUM> on a surface of the structure <NUM> where the user wishes to have the diameter of the structure <NUM> determined. In response, the system <NUM> (using, e.g., module <NUM>) calculates the diameter of the structure <NUM>, at a cross section of the structure <NUM>, defined by the location where the point <NUM> was selected by the user. <FIG> indicates the diameter by reference numeral <NUM>, and the location of that diameter <NUM> is further indicated by arrow indicators <NUM>, <NUM>. The value (e.g., <NUM>) of the diameter calculated by the system <NUM> is represented by reference numeral <NUM>, and is presented to the user by way of the display device <NUM>.

The structure <NUM> of <FIG> (or just the diameter itself) is shown as being rotationally offset relative to that in structure <NUM> of <FIG> Such an offset may (or may not) be represented on the display device <NUM>, and can be provided either automatically or in response to user instruction. <FIG> represents a further rotational offset with respect to that shown in <FIG>, that can be effected automatically or upon user instruction. Arrow indicators that are shown as further rotated in that Figure (relative to in <FIG>) are identified by reference numerals <NUM>, <NUM>. By virtue of rotating the structure <NUM> and/or the diameter as shown in <FIG>and/or <FIG>, the user can perceive both ends of the diameter right away without obstruction from arrow indicators, and, in the case of automatic rotation by the system <NUM>, in one example the user does not need to take any action to effect the rotation besides specifying (e.g., in a "single click") that the diameter be determined. Even where, in some cases, the rotated diameter is not exactly the same as the diameter in an initial orientation (i.e., <FIG>), the diameter values still may be very close in value, particularly for, e.g., a tube-like thin structure.

In one exemplary embodiment, to obtain a diameter from a single click selection point, such as point <NUM>, the system <NUM> finds the proper direction for measurement using, in one example, a straight ray direction of the clicked point with respect to (i.e., substantially perpendicular to) the viewer. However, for a more precise measurement in a case where the point selected by the user is off- center in the structure <NUM>, in another exemplary embodiment a 3D surface normal to the selected point <NUM> is employed to determine the diameter, or a 3D surface normal to a smoothed surface patch may be employed for additional robustness of the measurement (e.g., in one example, the normal can be calculated based on a graphical mesh representation of the surface). The end of the diameter opposite to where the user made the selection can be obtained by the system <NUM> searching from the selected point, along an inverse direction of the surface normal to the selected point, until an empty point is encountered (i.e., until a region outside of the opposite surface of the structure <NUM> is reached). In one example, the foregoing determinations are made by the system <NUM> automatically and in real-time, although in other embodiments they need not be.

As described above, the diameter can be rotated, as represented in <FIG>. The system <NUM> can rotate the diameter measurement using any suitable technique, such as, e.g., defining a cross sectional plane that contains an imaginary line composed of the selected point <NUM> and the surface normal vector. There may be unlimited choices since there are an unlimited number of planes that pass through this line. The system <NUM> can find a plane that is orthogonal to a center line direction of the tube-like thin structure <NUM>. To even further ensure use of a reliable center line of the tube-like thin structure, despite possible shape variations or tiny noisy structures on the surface that may affect center line extraction, additional techniques can be employed, such as, for example, a brute-force search scheme that determines the rotated diameter measurement, or other suitable techniques.

In one exemplary embodiment, from the center of the original diameter measurement, the system <NUM> searches in directions that are perpendicular to the original diameter measurement direction. The directions are spread in a half circle with a few degrees of separation. In each direction, the system searches from the center to at least two directions and finds the empty points. The length of the (two) ends besides the empty spots is calculated for each instance. In one example, the direction of the minimum calculated length is used as the rotated diameter measurement. One rationale of this approach is that the plane perpendicular to the center line of the tube-like thin structure <NUM> should have a cross-sectional circular disk that has the smallest radius. By searching at least some of these kinds of cross-sectional circular disks, a smallest one can be used as a correct cross-section. Also the computation cost is low since the system <NUM> only needs to sample a limited number of angles in a half circle.

<FIG> is an example of a flow diagram of the above operation. This technique takes advantage of the 3D view <NUM> in the system <NUM>. The user <NUM> can select a point that is on a displayed 3D object (step <NUM>). For example, the user <NUM> may select a point, on 2D view <NUM>, that is on the 3D object surface in 3D view <NUM>. In step <NUM>, the system <NUM> then transforms this 2D point to an actual 3D point using 2D/3D transformation mechanism <NUM>. The system <NUM> then determines a search direction (e.g., based on either a straight ray direction or a surface normal), and examines points along the search direction, starting from the 3D point and into the interior of the 3D object (step <NUM>). Based on the 3D point and search direction (e.g., straight viewing direction or surface normal), in step <NUM>, the system <NUM> can search the other end of the measurement based on the presence of the 3D object by, for example, finding a first empty point along the search direction from the 3D point. Then, in step <NUM>, the system <NUM> displays the measurement result (or diameter) with two end points indicator and the value of the measurements. Alternatively, to be even more user friendly, the system <NUM> can find a reasonable cross sectional plane that contains the measurement (e.g., the 3D point and the first empty point (step <NUM>). Then the system <NUM> rotates the measurement by <NUM> degrees (or another predetermined angle) within the cross sectional plane so that the end point indicators do not block the view of the measurement segment from the user point of view, and, in one example, this can be done without any further scene rotation being selected by the user (step <NUM>).

<FIG> depicts a flow diagram for pre-operative planning, according to an exemplary embodiment herein. The present example is described in the context of resection planning, although it can be employed in conjunction with other types of (pre-)operative assessments as well. At step <NUM>, data and segmented objects, such as, e.g., one or more anatomical structures, are loaded and displayed on a display device (e.g., display device <NUM>) in an exemplary screen layout. At step <NUM>, a user (e.g., user <NUM>) selects at least one displayed object-of-interest and activates a resection analysis function. For an example case involving hepatic lesion resection planning, the object-of-interest may be, e.g., one or multiple hepatic lesions to be re-sectioned. A safety margin indicator (e.g., <NUM>) is shown on the display device surrounding the object-of-interest.

At step <NUM>, the user operates a user interface (e.g., <NUM>) to adjust the size, location, shape, and/or orientation, etc. of the safety margin indicator, as described above. The safety margin indicator is instantaneously updated and displayed in both 3D and 2D views on a display (e.g., <NUM>), in accordance with the adjustment. The system (e.g., <NUM>) may also update the display of (e.g., appearance and/or color) vital anatomies that are within (or which touch) the adjusted safety margin.

At step <NUM>, the user operates a user interface (e.g., <NUM>) to define at least one resection cutting surface using, for example, one or more of the separating surface definition methods previously described. At this step, the system (e.g., <NUM>) also can display in 3D and 2D views the user-confirmed safety margin (e.g., <NUM>), vascular systems, biliary systems, and vascular/biliary territories, as well as intensity information contained in original images. Such information can be helpful to enable the user to best determine a resection surface. At step <NUM>, the system (e.g., <NUM>) separates (on a display device, such as device <NUM> of <FIG>) the target object using the user defined/confirmed safety margin and/or separating surface, in response to user instruction. A volume of each part of the target object after resection is automatically calculated, in a manner as described above. For example, in a case involving a hepatic lesion resection, a volume of the liver sub-volume to be re-sectioned and a volume of the remnant are calculated, respectively. The user can visualize these elements in 3D and 2D views and evaluate the vascular/biliary territories on the remaining liver, by virtue of the presentation on the display device (e.g., <NUM>). Such information can be helpful for resection planning.

At step <NUM>, cross-sections (e.g., <NUM>) of the resection surface, as well as the cross-sections of the vital anatomies on the resection surface, are analyzed. For example, sizes, diameters, and other dimensional characteristics of the cross- sections of vital anatomies on the resection surface can be automatically or manually measured and displayed, as described above. For each vascular/biliary branch that may be re-sectioned by the resection surface, for example, the system (e.g., <NUM>) can automatically calculate the branch's diameter for the user's reference. At step <NUM>, the user can elect to confirm the resection plan and save its results if the current plan is deemed clinically and surgically acceptable. Such results can be saved in a storage device (e.g., <NUM>) of the system (e.g., <NUM>) and reported or forwarded to another external device by way of a communication interface (not shown) and network (not shown). The user instead can reject the current resection plan in step <NUM>.

By virtue of the foregoing method, pre-operative assessments can be made in an intuitive, interactive, and real-time manner, to facilitate hepatic lesion/tumor resection planning based upon quantitative assessment and evaluation of anatomic structures and the like. Owing to the system <NUM>, apparatus, procedure(s), and program(s) described herein, a working environment is provided that fuses 3D and 2D information to perform efficient resection evaluation and analysis, without requiring unwieldy, intrusive, and often difficult-to-obtain equipment, such as stereo goggles and special 3D tracking devices and the like. As such, the system <NUM> is an easier to use, stand-alone 3D system having the functionalities and performance capabilities described herein.

Again, the example aspects described herein provide an integrated 3D visual environment, tools, and interactive schemes for assessment measurements such as safety margin adjustment and visualization, vessel alternation visualization, region open and close animation, and direct vessel diameter measurement. A 3D working environment is provided for these pre-operative assessments, rendering them more easily to operate and more intuitive to understand in 3D in nature, relative to conventional practices.

In the foregoing description, example aspects of the invention are described with reference to specific example embodiments thereof. The specification and drawings are accordingly to be regarded in an illustrative rather than in a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto, in a computer program product or software, hardware, or any combination thereof, without departing from the broader scope of the present invention.

Software embodiments of example aspects described herein may be provided as a computer program product, or software, that may include an article of manufacture on a machine accessible or machine readable medium (memory) having instructions. The instructions on the machine accessible or machine readable medium may be used to program a computer system or other electronic device. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks or other types of media/machine-readable medium suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They may find applicability in any computing or processing environment. The terms "machine accessible medium", "machine readable medium", or "memory" used herein shall include any medium that is capable of storing, encoding, or transmitting a sequence of instructions for execution by the machine and that cause the machine to perform any one of the methods described herein. Furthermore, it is common in the art to speak of software, in one form or another (e.g., program, procedure, process, application, module, unit, logic, and so on) as taking an action or causing a result. Such expressions are merely a shorthand way of stating that the execution of the software by a processing system causes the processor to perform an action to produce a result. In other embodiments, functions performed by software can instead be performed by hardcoded modules, and thus the invention is not limited only for use with stored software programs.

In addition, it should be understood that the figures illustrated in the attachments, which highlight the functionality and advantages of the present invention, are presented for example purposes only. The architecture of the example aspect of the present invention is sufficiently flexible and configurable, such that it may be utilized (and navigated) in ways other than that shown in the accompanying figures.

Claim 1:
A procedure for pre-operating assessment of one or more anatomical structures generated from medical images and provided in a rendered 3D space, comprising:
receiving, at a system (<NUM>), one or more safety margin indicators on a user interface, each safety margin indicator defining a 3D closed volume shape and having a shape corresponding to that of a respective one of the anatomical structures within an organ and having a predetermined size of safety margin from the respective one of the anatomical structures;
receiving, at the system (<NUM>), an optimal cutting surface definition, the optimal cutting surface definition comprising at least one of leaving a largest volume of remnant structure or causing a minimal potential encroachment to an underlying vascular system;
calculating, via a processor (<NUM>) of the system (<NUM>) using: (<NUM>) the optimal cutting surface definition, and (<NUM>) the one or more safety margin indicators, an optimal cutting surface of the organ; and
displaying, on a display (<NUM>), the organ with the optimal cutting surface and the one or more safety margin indicators in the rendered 3D space and at the same time in a 2D space;
receiving, on the display (<NUM>), selection of a 2D point within the 2D space;
identifying a search direction based on a straight ray detection;
transforming, via a 2D/3D transformation mechanism, the 2D point into a 3D point;
measuring a distance between the 3D point and a first empty point along the search direction; and
displaying, on the display (<NUM>) the measurement result.