Patent Description:
This disclosure relates to a graphical user interface for marking tubular anatomic structures in a visualization.

Various forms of image-guided surgical navigation systems have been developed. In some examples, such systems use medical imaging technologies to create a model of the anatomic region of interest for viewing during various phases of the procedure, including pre planning and intraoperative phases. As an example, a surgeon uses special instruments, which are tracked by the navigation system. The position of the tracked instrument in relation to the patient's anatomy may be shown on images, including as the surgeon moves the instrument during a procedure. Document <CIT> discloses a method for delivering a medical device coupled with a catheter, to a selected position within a lumen of the body of a patient, the method comprising the procedures of: registering a three-dimensional coordinate system with a two-dimensional coordinate system, the three-dimensional coordinate system being associated with a medical positioning system (MPS), the two-dimensional coordinate system being associated with a two-dimensional image of the lumen, the two-dimensional image being further associated with an organ timing signal of an organ of the patient; acquiring MPS data respective of a plurality of points within the lumen, each of the points being associated with the three-dimensional coordinate system, each of the points being further associated with a respective activity state of the organ; determining a temporal three-dimensional trajectory representation for each the respective activity states from the acquired MPS data which is associated with the respective activity state; superimposing the temporal three-dimensional trajectory representations on the two-dimensional image, according to the respective activity state; receiving position data respective of the selected position, by selecting at least one of the points along the temporal three-dimensional trajectory representation; determining the coordinates of the selected position in the three-dimensional coordinate system, from the selected at least one point; determining the current position of the medical device in the three-dimensional coordinate system, according to an output of an MPS sensor attached to the catheter in the vicinity of the medical device; maneuvering the medical device through the lumen, toward the selected position, according to the current position relative to the selected position; and producing a notification output when the current position substantially matches the selected position. Document <CIT> discloses system and methods to facilitate intra-operative procedures and planning. A method includes storing tracking data in memory, the tracking data being generated by a tracking system to represent a location of an object in a tracking coordinate system of the tracking system. The method also includes storing a patient-specific implicit model in memory, the patient-specific implicit model being generated based on image data acquired for the patient to define geometry of an anatomical structure of the patient. The method also includes registering the tracking data and the patient-specific implicit model in a common three-dimensional coordinate system. The method also includes generating an output visualization representing a location of the object relative to the geometry of the anatomical structure of the patient in the common coordinate system. Document <CIT> discloses a system and method for presenting three-dimensional image volume data utilizing an orientationally-sensitive display device whereby the image volume is navigable simply by tilting, raising and lowering the display device. Doing so presents an image on the screen that relates to the angle and position of the display device such that the user gets the impression that the device itself is useable as a window into the image volume, especially when the device is placed on or near the source of the image data, such as a patient.

This disclosure relates to a graphical user interface for marking tubular anatomic structures in a visualization-.

As one example, one or more non-transitory computer readable media having instructions stored thereon, for execution by a computer that includes a graphical user interface and a user input device, graphical user interface, the instructions to perform a method comprising:
displaying a graphical representation of an anatomic structure on the graphical user interface, the graphical representation of the anatomic structure being determined from a geometry of the anatomic structure and wherein geometric knots are determined at respective positions along a centerline of the graphical representation of the anatomic structure; determining a location parameter for a graphical marker along the graphical representation of the anatomic structure in response to a first user input from the user input device, wherein the location parameter for the graphical marker corresponds to spatial coordinates for a closest geometric knot of the geometric knots, and wherein the graphical marker represents a location of an anatomic opening between the anatomical structure and another anatomical structure; displaying the graphical marker relative to on the graphical representation of the anatomic structure on the graphical user interface based on the location parameter; changing the location parameter to move the graphical marker along the graphical representation of the anatomic structure based on a user input from a user input device; selecting a fixed location along the graphical representation of the anatomic reference to fix the graphical marker relative to the graphical representation of the anatomic structure in response to a second user input from the user input device; and generating an output visualization that includes at least one view of the graphical representation of the anatomic structure and the graphical representation of the graphical marker at the fixed location.

As another example, a computer-implemented method for marking an anatomic structure, the method comprising: displaying a graphical representation of an anatomic structure on an output device, the graphical representation of the anatomic structure being determined from a geometry of the anatomic structure and wherein geometric knots are determined at respective positions along a centerline of the graphical representation of the anatomic structure; determining a location parameter for a graphical marker with respect to the graphical representation of the anatomic structure in response to a first user input from a user input device, wherein the location parameter for the graphical marker corresponds to spatial coordinates for a closest geometric knot of the geometric knots, and wherein the graphical marker represents a location of an anatomic opening between the anatomical structure and another anatomical structure; displaying the graphical marker relative to the graphical representation of the anatomic structure on the graphical user interface based on the location parameter; changing the location parameter to move the graphical marker along the graphical representation of the anatomic structure based on a user input from a user input device;
selecting a fixed location along the graphical representation of the anatomic reference structure to fix the graphical marker relative to the graphical representation of the anatomic structure in response to a second user input from the user input device; and generating an output visualization that includes at least one view of the graphical representation of the anatomic structure and the graphical representation of the final graphical marker at the fixed location.

This disclosure relates to graphical user interface for marking anatomic structures in a visualization.

The systems and methods disclosed herein may be used in a pre-operative planning stage and/or intra-operatively to place one or more graphical markers at selected locations with respect to a graphical representation of anatomy. Each graphical marker may be visualized on the display at a fixed user-selected location to provide a guidance (e.g., a waypoint or target) that remains anatomically and geometrically consistent intra-operatively over a number of different views. For example, a ring-shaped graphical marker may be placed to designate an opening between adjacent fluidly connected anatomic structures, such as an ostium between two vessels (e.g., a main vessel and a branch vessel), an opening between an organ and a vessel, an opening between two chambers of an organ.

As an example, a graphical representation of a portion of a patient's anatomy can be visualized on an interactive display. A computer (or other processor-based apparatus) provides a graphical user interface on the display that is interactive in response to a user input received from one or more user input devices (e.g., a hardware interface, such as mouse, touch screen, keyboards, gesture interface or the like). To place a marker, a user activates the marker tool in a graphical or other user interface that includes a graphical representation of a virtual anatomic reference structure (e.g., a centerline of an anatomic lumen or an axis of symmetry for another anatomic structure). For example, user input instructions (from a user input device) position a pointer along the centerline that is visualized in a graphical user interface of the marker tool. A graphical proxy marker can be displayed at a current pointer location along the centerline (or other anatomic reference). The graphical proxy marker provides a temporary graphical visualization of the marker relative to the anatomic reference to allow the user to decide whether the location is satisfactory or if an alternative location would be more suitable. When a suitable location for the proxy marker is determined, the user can employ the user input device to select and fix the location of the marker with respect to the anatomic reference to define parameters for a graphical final marker. The selected location for this and any number of markers can be stored in memory, such as linked to a respective one of a plurality of points distributed along the length of the centerline.

The computer can generate an output visualization that is provided to the display to visualize a graphical representation of the anatomic structure that includes the graphical final marker. In some examples, an output generator can be programmed to utilize a graphics pipeline to render a number of different views of the anatomic structure, each including a visualization of the marker at a consistent anatomic location. Each view of the anatomic structure may be concurrently displayed (e.g., in different windows) or displayed at different times (e.g., in the same or different windows). Because the position of the graphical final marker is linked to a given point along the centerline (or other graphical structure) and the geometry of anatomy that marked is known, the final marker can retain a consistent and accurate relative position and size with respect to a patient's anatomy in each visualization. In this way, a surgeon or other user can place the marker to provide a waypoint (e.g., a luminal waypoint for vessel or other anatomic structure) along a path to a target site or identify the target site itself over a number of different views. For example, a ring-shaped graphical marker may be placed at an ostium of a blood vessel (e.g., the aorta) to designate the opening to a branch into which an instrument (e.g., catheter and/or guidewire) may be inserted as part of a procedure. In the context of intra-operative guidance, a graphical visualization of the catheter and/or guide wire may also be rendered in the output visualization along with the patient anatomy and the graphical marker, including during movement of the instrument along the anatomic lumen of the patient's body.

The approach herein affords a user flexibility in placing the graphical marker at a desired location to provide personalized guidance. In many example embodiments disclosed herein, the anatomic structures include tubular anatomic structures, such as one or more blood vessels (e.g., the aorta and its associated branching arteries. It is to be understood that the graphical user interface and methods are equally applicable to other anatomic tubular structures as well as to non-tubular structures, such as organs.

<FIG> depicts an example of a computer system <NUM> programmed for marking anatomic structures. The computer system <NUM> includes one or more processors <NUM> and associated memory <NUM>. The memory <NUM> can include one or more non-transitory computer-readable media having instructions and data stored thereon. The system <NUM> also includes one or more input/output (I/O) devices demonstrated as a user input device <NUM> and an output device <NUM>. For example, the user input device <NUM> may include a keyboard, mouse, touch screen interface, gesture interface or the like that is configured to provide instructions and commands that can be stored in the memory <NUM> as user input data <NUM>. As an example, the user input device <NUM> can provide the input data <NUM> to specify a position (e.g., spatial coordinates) and/or movement of a pointer with respect to a displayed graphical representation in the output device <NUM>. Additionally or alternatively, the user input device <NUM> can be configured to provide the input data <NUM> to select or activate a function at a given location (e.g., an area of the display) for which one or more functions are associated. The output device <NUM> can include one or more displays, such as a monitor, heads up display or virtual reality headset or goggles. Thus, in some examples, the user input device <NUM> and the output device <NUM> may be integrated to provide both inputs and outputs for the system <NUM>.

As mentioned, the memory <NUM> includes instructions that can be executed by the processor <NUM> to perform various functions disclosed herein. In the example of <FIG>, the instructions include an output generator <NUM> programmed to generate an interactive output visualization that can be rendered as a graphical output and displayed in the output device <NUM>. The output generator <NUM> can include one or more rendering methods <NUM> for rendering various objects in the output visualization. The output generator <NUM> can also include display controls <NUM> programmed to set one or more visualizations parameters to control, for example, viewing angle, size, colors and the like in the output visualization that is provided by the rendering method <NUM>. The output generator <NUM> can also include a user interface <NUM> to enable user interaction (e.g., through user input device <NUM>) with the various functions of the output generator <NUM>.

The output generator <NUM> also includes a marker tool <NUM> that is programmed to mark anatomic structures in response user input data <NUM> provided based on the user input device <NUM>. The marker tool <NUM> includes the marker generator <NUM> that is programmed to generate one or more graphical markers that can be visualized relative to anatomic structures. For example, the marker generator <NUM> is programmed to generate marker data <NUM> that is utilized by the rendering methods <NUM> of the output generator <NUM> to display one or more graphical markers relative to an anatomic structure in response to a user input. The marker data <NUM> can include parameters to determine a position, size and other graphical attributes (e.g., color, brightness transparency) of the marker with respect to the anatomic structure where it is being placed.

By way of example, as part of executing the marker tool <NUM>, the output generator <NUM> can render a graphical representation of a virtual anatomic reference. For instance, the anatomic reference can be a centerline of a tubular anatomic structure, such as a blood vessel (e.g., the aorta or other arteries, veins) or other tubular structures, such as the gastrointestinal tract, ureter or the like. The type of anatomic reference can be based on the shape of anatomic structure and how it is stored in memory (e.g., model data <NUM>, <NUM>).

The marker generator <NUM> can include a position calculator <NUM> programmed to determine one or more location parameters for a marker in response to the user input data <NUM>. The location parameters can be stored as part of the marker data <NUM>. The rendering method of output generator <NUM> employs the location parameter to control visualizing the graphical marker with respect to the anatomic reference in the displayed graphical representation. For example, the position calculator <NUM> can determine the location parameter according to a location of the pointer, as provided by user input data <NUM>, with respect to predetermined spatial points distributed along the anatomic reference (e.g., the centerline).

As an example, the centerline is generated from a three-dimensional patient-specific implicit centerline model that is described in centerline model data <NUM> stored in the memory <NUM>. The centerline model data <NUM> may include one or more parameters to describe a spline equation corresponding to a series of geometric knots in three-dimensional space. The rendering method <NUM> thus can be programmed to calculate spline control points from the geometric knots and evaluate the spline equation using the computed control points for a given parameter value, such that the centerline is a function of a single parameter. For example, the rendering method <NUM> can compute the centerline as a function of a single parameter (u) representing geometric knots that goes from zero to one and varies along a path corresponding to the axis of the tubular structure taken by selecting a spacing of the parameter u. The value can be computed at each respective spacing and the rendering method <NUM> can plot the curve as a series of corresponding line segments drawn in the output visualization between the values of the parameter u. For example, if a spacing of <NUM> is selected, the rendering method can evaluate the curve corresponding to the centerline at u = <NUM>, u = <NUM>, u = <NUM>, etc. and the corresponding points for each value of u can be connected to provide a plot corresponding to the centerline in the graphical output visualization in the output device <NUM>.

The geometric knots distributed along the length of the centerline thus have a known position in the graphical visualization that is provided in the output device <NUM> as does the pointer based on the user input data <NUM>. The position calculator <NUM> thus can determine a relative position of the pointer and geometric knots along the centerline based on the user input data <NUM> and centerline model data <NUM>. For example, the position calculator <NUM> computes a distance between the pointer element and the geometric knots to, in turn, identify a closest geometric knot to the pointer based on the computed distance. For example, a user can hover the pointer over the length of the centerline in the output visualization and a closest geometric knot can be identified and stored as part of the marker data <NUM> as the user input data changes according to the movement of the pointer with respect to the centerline.

The marker generator <NUM> can provide the marker data <NUM> to include both proxy marker data <NUM> and final marker data <NUM>. The proxy marker data <NUM> provides parameters for a graphical proxy marker to be displayed with respect to the anatomic reference structure. The graphical proxy marker and anatomic structure are utilized by the marker tool to identify and then select a fixed location for the final marker. Thus, the proxy marker data <NUM> can change according to the pointer position. For example, the rendering method can render the graphical proxy marker along the length of a centerline, based on the calculated position, as the pointer hovers over the graphical representation of the centerline in the visualization that is provided on the output device <NUM>.

In some examples, the size, shape and orientation of the graphical proxy marker may remain fixed as the pointer of the user input device moves along the centerline (or other anatomic reference) to change the location parameter for the graphical proxy marker. As mentioned, the location parameter for the proxy marker may correspond to spatial coordinates of geometric knots determined by the rendering method based on the model data <NUM>. The size, shape and orientation of the graphical marker can be stored in the marker data <NUM> (e.g., proxy marker data <NUM> or final marker data <NUM>).

In other examples, the graphical marker can have a size, shape and/or orientation parameter that varies based on the location of the marker along the centerline. As such, the marker generator <NUM> includes a shape/orientation calculator <NUM> programmed to determine a shape, size and/or orientation of the marker that the rendering method <NUM> is to visualize on the output device <NUM>. For example, the shape/orientation calculator <NUM> can dynamically determine a size of the graphical proxy marker of the geometric knots according to the radius of the tubular structure as provided by surface model data <NUM>. For example, the surface model data <NUM> describes surface geometry of the structure tin three-dimensional space implicitly by a set of one equations and parameters derived from image data, as disclosed below. Thus, as the pointer of the user input device <NUM> moves along the centerline to change the location parameter, the size of the graphical proxy marker can reflect the size of the tubular anatomic structure.

Additionally or alternatively, the shape/orientation calculator <NUM> can dynamically determine an orientation of a graphical marker with respect to the position and direction of the centerline structure (as provided by centerline model data <NUM>) at the position along the centerline where the marker is being rendered. For example, as the position of the graphical proxy marker is moved along the centerline (e.g., by moving the pointer of the user input device <NUM> along the centerline), the shape/orientation calculator <NUM> can determine an orientation of the proxy marker as to be substantially perpendicular to the direction of the centerline. For example, a virtual plane extending through the body of the graphical proxy marker (e.g., along orthogonal axes of the marker) can be maintained perpendicular to the centerline.

In some examples, the graphical proxy marker has a toroidal (e.g., ring-shaped) geometry having a center or centroid that is positioned along the curved centerline (e.g., co-located with the geometric knots of the centerline based on model data <NUM>). The size of the toroidal shape of the graphical proxy marker may be fixed or it may vary according to the surface geometry of the tubular structure at the position where it is rendered along the centerline. Additionally or alternatively, the orientation of the toroidal shape of the graphical proxy marker may vary (e.g., be perpendicular) according the direction of the tubular structure at the position where it is rendered along the centerline.

The position, shape, size and orientation of graphical proxy marker can be stored in the proxy marker data <NUM> for use by the rendering method <NUM> in generating the visualization in the graphical output provided to the output device <NUM>. By changing the position of the graphical proxy marker as the pointer moves along the centerline, the user is afforded a dynamic visualization to decide upon placement for the marker.

Once a user has decided to set the marker at a desired location along the length of the anatomic structure, a user can employ the user input device <NUM> to select the fixed location for a final graphical marker. For example, the user may utilize the user input device <NUM> to double click on (or near) the graphical proxy marker, activate a GUI button or select another user interface element to set a location that is stored in the user input data <NUM>. The marker generator <NUM> in turn specifies the selected fixed location with respect to the anatomic structure and stores it as part of the final marker data <NUM>. As an example, the location parameter for the final marker data <NUM> can correspond to a given geometric knot along the centerline, which may be added or linked to the centerline model data to specify the location along the centerline where the final graphical marker is to be rendered in the visualization.

Either automatically following configuring the final graphical marker or in response to a user input with the user input device <NUM>, the marker tool may be closed from the GUI and the output generator <NUM> may enter a normal visualization/guidance mode. In the normal mode, the rendering method <NUM> can generate the output visualization to include one or more views of the anatomic structure, which also includes a graphical representation of the final marker based on final marker data <NUM> at the selected location. As a user input with the user input device may utilize the display controls <NUM> to change or add viewing angles, the graphical marker remains at the selected location relative to the anatomic structure based on the final marker data <NUM>.

In some examples, the output generator <NUM> is also programmed to generate the graphical representation of an instrument, such as a catheter and/or guide wire that is inserted in the tubular structure. The location of the instrument (e.g., guide wire or catheter) can be provided based on tracking data for a sensor that is located on or fixed with respect to the guide wire and/or catheter. The location and graphical description of such instrument may be provided by other data <NUM> that is stored in the memory <NUM>. For example, the other data <NUM> includes model data describing the geometry of the instrument as well as location information for the instrument based on the tracking data, which is variable and can change over time in response to the position of the instrument within the anatomic structure. The output generator <NUM> thus may generate an aggregate output visualization of the anatomic structure, such as based on the centerline model data <NUM> and the surface model data <NUM>, a graphical marker based on the marker data <NUM> as well as the graphical representation of the instrument based on the other data <NUM>. In this way, the position of the instrument can be visualized and tracked as it moves within the anatomic structure based on its relative position that changes with respect to time according to the relative position of the sensor within the anatomic structure. As disclosed herein, where the anatomic structure is a tubular structure including one or more blood vessels, the final graphical marker can specify the location and provide a visualization of an ostium.

By way of further example, <FIG> and <FIG> depict examples of a graphical user interface demonstrating a process that can be implemented for setting a location for one or more graphical markers in response to a user input. The examples of <FIG> and <FIG> are illustrated in the context of an aorta that includes a plurality of branch arteries. It is understood that the approach disclosed herein for placing a marker can be applied to other blood vessels as well as to other shapes and sizes of anatomic structures. In the examples of <FIG> and <FIG>, the graphical user interface <NUM> illustrates a patient-specific centerline for the descending aorta, demonstrated at <NUM>, and other centerlines <NUM>, <NUM>, and <NUM> for branching arteries.

In the example of <FIG>, a marker tool (e.g., tool <NUM>) has been activated to provide the resulting graphical user interface <NUM> for placement of one or more graphical markers. A pointer <NUM> can be moved within the GUI <NUM> in response to the user input with a user input device (e.g., device <NUM>) and stored as user input data (e.g., data <NUM>). The pointer <NUM> could be moved along any of the centerlines <NUM>, <NUM>, <NUM> and <NUM> for marker placement (in response to a user input). In the following example, the pointer <NUM> will be moved along branch centerline <NUM> such as in the direction indicated at <NUM>. As shown in <FIG>, as the pointer <NUM> moves along (e.g., spatially intersects) the centerline <NUM>, a graphical proxy marker <NUM> is generated (e.g., by output generator <NUM>) based on the pointer location with respect to the centerline <NUM> (e.g., provided by proxy marker data <NUM> in response to position calculator <NUM> determining the pointer position coinciding with the centerline).

In this example, the graphical proxy marker <NUM> has a toroidal shape having a center co-located with a nearest given geometric knot along the centerline. For example, the graphical proxy marker is oriented to lie in a plane that is orthogonal to a direction of the centerline at the given geometric knot. As the pointer <NUM> moves along the centerline <NUM> (e.g., along direction <NUM>), the position of the proxy marker <NUM> would change based on the determined location of the pointer with respect to the centerline. Thus, as pointer <NUM> moves along centerline <NUM>, the position of the proxy marker <NUM> will change accordingly. In some examples, the size and/or orientation of the proxy marker also can vary such as disclosed herein.

As shown in the example of <FIG>, a fixed location along centerline <NUM> is selected in response to a user input (e.g., by user input device <NUM>). In response to setting the location, a graphical attribute of the marker can change to indicate that the location is set. In the examples of <FIG> and <FIG>, this is demonstrated as changing from a dashed line to a solid line. In other examples, the color and/or brightness of the marker that is displayed in the GUI <NUM> can change to indicate that the marker location has been set. After one or more marker locations have been set with respect to the desired anatomic structures (e.g., centerline), final marker data (e.g., data <NUM>) can be stored in memory for use in rendering the graphical representation of the final graphical marker at the selected fixed location relative to the anatomic structure.

<FIG> demonstrate output visualizations <NUM> for a plurality of different views for the same portion of the descending aorta shown in <FIG>. In this example, the visualization <NUM> depicts an outer surface of the descending aorta <NUM> (e.g., generated by rendering method <NUM> based on surface model data <NUM>). Also demonstrated in the visualization <NUM> are centerlines <NUM>, <NUM>, <NUM> and <NUM> for the aorta and the branch arteries (e.g., generated by rendering method <NUM> based on centerline model data <NUM>). The visualization <NUM> also includes a graphical representation of the final graphical marker shown at <NUM>. The shape of the graphical marker <NUM> in the output visualization <NUM> is different from the proxy marker <NUM> shown in <FIG> and <FIG>.

As demonstrated in <FIG>, showing a side perspective view of the marker <NUM>, the shape and cross-sectional configuration of the marker <NUM> is defined by the shape and cross-sectional dimension of the branch artery (e.g., derived based on the surface model data <NUM>). For example, a predetermined toroidal shape for the proxy marker <NUM> may be used for setting the location and when the visualization of the anatomic structure is generated, such as shown in <FIG>, the final marker can be generated at the fixed location according to the cross-sectional shape of the surface at the centroid location for such geometric knot. For example, the final marker <NUM> can be a three-dimensional ring having a short cylindrical sidewall and a radius at each end thereof that is based on the radius of the tubular anatomic structure at the geometric knot where the marker has been placed. The axial ends of the marker may be circular or have non-circular, anatomic contours defined by the model data <NUM>, which can be lofted by interpolating between the respective contours (e.g., by lofting with Bezier curves). The final marker can be rendered in a different color from the surface of the anatomic structure (e.g., red and blue, respectively), which color can remain the same among difference views. The final marker can also be provided a radial thickness (e.g., a number of voxels thick) to at least approximate the thickness of the vessel wall, such as shown in the views of <FIG>. Thus, the shape of the marker <NUM> is configured to conform to the shape of the surface of the vessel wall.

With reference back to <FIG>, the rendering method <NUM> may be programmed to generate the surface of the tubular anatomic structure as a lofted surface. For example, at each geometric knot of the centerline a given number of unit vectors are created perpendicular to the tangent line of the control point, such as by creating an orthonormal basis. Each vector direction extends outwards until the voxel values drop below a threshold. The length of each vector may be set vary according to the radius of the vessel surface being created at each centroid location. Such as may be generated from a spline at the selected geometric knot.

In an example, the final graphical marker <NUM> may also be generated (e.g., by rendering method as a lofted surface. As discussed with respect to <FIG>, the marker <NUM> is visible with respect to the anatomic structure being visualized from multiple different views. In each of the views, the marker will be positioned at the same location as provided in the final marker data <NUM> (e.g., linked to a geometric knot along the centerline). In the examples of <FIG>, <FIG>, the reference marker <NUM> is placed at the user selected location to identify an ostium between a branch artery and the main descending aorta <NUM>.

<FIG> demonstrates an example of an output visualization <NUM> looking into the descending aorta through the ostium represented at the marker <NUM>. <FIG> illustrates a cut-away view of the descending aorta exposing the ostium viewed axially (e.g., along the branch centerline) into the branch artery and showing the location of the marker <NUM> with respect to the branch artery. The cut-away view of <FIG> also shows an interior sidewall <NUM> of the aorta. Each of the different views shown in <FIG>, <FIG>, for example, may be selected by implementing display controls <NUM> in response to user input data from the user input device. Additionally, in some examples, multiple views may be displayed concurrently in one output device (e.g., in a plurality of windows) or in a plurality of such output devices <NUM>.

<FIG> depicts an example of a visualization system <NUM> that includes an output generator <NUM> programmed to generate visualization data <NUM>, which can be provided to a display to render a corresponding graphical representation. The output generator <NUM> can generate the visualization data <NUM> that is provided to a display device <NUM> for displaying a corresponding visualization <NUM>. The visualization <NUM> can provide intra-operative guidance such as for surgical navigation, which can include one or more graphical markers as disclosed herein. The output generator <NUM> can generate the visualization data <NUM> based on tracking data <NUM>, anatomic model data <NUM> and object model data <NUM>. The anatomic model data <NUM> can be a patient-specific implicit model that describes one or more anatomic structures in three-dimensional space (e.g., corresponding to the centerline and surface model data <NUM> and <NUM> of <FIG>).

The object model data <NUM> can correspond to another three-dimensional implicit model that has been generated to describe geometry of an instrument (e.g., a catheter and/or guidewire) that is being inserted into the patient during a procedure. For example, the object model data <NUM> can correspond to an analytical or parametric representation of a surgical instrument, which may be a generally rigid surgical instrument or an articulated instrument that includes a flexible tip such as wires, catheters and the like. Accordingly, the complexity of the model data <NUM> and the corresponding implicit model that it defines can vary according to the type of instrument or other object that is being tracked within the patient's anatomy. In addition to parameterizing the geometry of the object, the object model data <NUM> can also be configured to model other properties of the object (e.g., resilience and/or ductility).

The system <NUM> can include a tracking system <NUM> that includes one or more sensors <NUM> can be affixed to or integrated into the instrument to enable tracking of its location by the tracking system. The tracking system <NUM> is configured to generate the tracking data <NUM>, which is registered in a common three-dimensional coordinate system with the anatomic model data <NUM> and the object model data <NUM> to enable real-time tracking. The common coordinate system can represent a coordinate system for image data that has been acquired preoperatively relative to the tracking data that is generated intra-operatively by the tracking system <NUM>.

For example, the tracking system <NUM> can include a transmitter (e.g., an electromagnetic field generator) that provides a non-ionizing field, demonstrated at <NUM>, which is detected by each sensor <NUM> to provide a corresponding sensor signal to the tracking system. An example tracking system <NUM> is commercially available from Northern Digital, Inc. , of Ontario, Canada. The tracking system <NUM> can provide the tracking data <NUM> at an output sample rate (e.g., sixty samples per second) sufficient to enable substantially real time determination of sensor location (e.g., to provide a vector describing sensor position and orientation). A tracking processing subsystem (not shown) thus can process each frame of tracking data <NUM> such that the registered tracking data can likewise represent real time tracking data acquired by the tracking system <NUM> that can be registered into the coordinate system of the anatomic model and rendered as a graphical representation, as disclosed herein. The tracking system <NUM> can provide the tracking data with an output sample rate to enable computation of real time positioning and generation of the visualization <NUM> to include the object to which the sensor is attached.

As a further example, the output generator <NUM> includes a registration matrix that is programmed to convert the tracking data <NUM> from a coordinate system of the tracking system <NUM> to a coordinate system that is common for the model data <NUM> and <NUM>. The common coordinate system may be a spatial coordinate system of a patient or coordinate system of a pre-operative image that is used to generate the anatomic model. The tracking system <NUM> may generate the tracking data <NUM> as a plurality of frames and the registration matrix may be applied to convert each of the plurality of frames of the tracking data into a coordinate system to enable processing of the tracking data and the model data <NUM> and <NUM> in a common coordinate system.

The output generator <NUM> includes a rendering method <NUM> programmed to produce a three-dimensional plot corresponding to the visualization data <NUM> based on the input data <NUM>, <NUM> and <NUM>. Various types of rendering software (e.g., commercially available or proprietary) can be utilized and implemented as the rendering method <NUM> and can vary according to the type of models generated for use by the system <NUM>.

The output generator <NUM> can also include display controls <NUM> that can control the output that is provided intraoperatively. The display controls <NUM> can be configured to selectively generate any number of one or more displays concurrently on one or more display devices <NUM>, each of which can include a different view of the object and the anatomic structure. The respective views can be selected automatically such as by default parameters or it can be adjusted in response to the user input just as can be provided a user interface <NUM>. This display controls <NUM> can further control a viewing angle for each of the visualizations of the anatomic model and the object that are presented to the user. Since the structures in each visualization are virtual renderings based on implicit models, the output visualization is not constrained to any particular viewing angle or type of visualization.

In some examples, the display controls <NUM> can compute and display task-specific visualizations, such as may include an optimal view for a particular task (for example, cannulating a renal artery). For example, when cannulating a vessel, it is useful to visualize the vessel and wire without distractions or obstructions. The output generator <NUM> is able to create this visualization since each graphical marker, each vessel and each device are virtual renderings generated by the rendering method <NUM>. Additionally, because each the models <NUM> and <NUM> are easily separated into its constituent parts, other items can be effectively removed from the display and only show the clinician the pertinent geometry and telemetry for the current task.

The rendering method <NUM> can correspond to the rendering method <NUM> of <FIG>. By way of example, the rendering method <NUM> includes a centerline render function programmed to plot a centerline of the anatomic structure based on the anatomic model data <NUM>, which includes parameters describing the geometry of the anatomic structure (e.g., an elongated curved tubular structure, such as a vessel or intestine). As mentioned, the anatomic model data <NUM> can include a single parameter to describe the centerlines (e.g., geometric knots) that can be evaluated to form a spline curve. As an example, the centerline render function <NUM> can compute the centerline as a function of a single parameter (u) that goes from zero to one and varies along a path corresponding to the axis of the tubular structure taken by selecting a spacing of the parameter u.

The rendering method <NUM> can also include a surface render function that can produce a plot for a surface of the anatomic structure based on the implicit surface model defined by the model data <NUM>. As an example, the rendering method <NUM> can compute the surface as a function of two variables, such as the variable u, which extends along the centerline of the tubular structure and another parameter (v) which varies as one travels tangentially around the surface. As disclosed herein, the anatomic model data <NUM> can store the surface information as a series of slices in which each slice can be represented by a series of geometric knots of an elongated tubular structure. The surface render function can create vectors perpendicular to the tangent line of the control point at each geometric knot unit vectors, which vectors can have a radius corresponding to surface points defined by radius of the anatomic structure at such point. The surface render function can visualize the surface by evaluating the parameters to generate polygons (e.g., triangles) that tessellate the surface. Such polygons can be rendered efficiently with various computer graphics, hardware and software.

The rendering method <NUM> can also include an object render function to render a graphical representation of the object based on the object model data <NUM> and the tracking data <NUM>, which is registered in a common coordinate system with the anatomic model data. The registered tracking data <NUM> represents a position and orientation of one or more sensors in three-dimensional space corresponding to the same coordinate system in which the anatomic model has been registered. The object render function can be programmed to generate a graphical representation for each object depending on the location of the object defined by the tracking data <NUM>. By way of example, the object render function can plot rigid objects (and parts of objects that are rigid) by applying a transformation matrix multiplied by the overall registration matrix. For the case of articulated objects (e.g., instruments with a flexible tip, such as wires and catheters), the object render function can be programmed to plot different parts of the structure separately. As an example, the object render function can render an elongated instrument in discrete parts.

The output generator <NUM> can also include a guidance generator <NUM> programmed to generate user perceptible guidance that can be included in the visualization data <NUM> and presented in the visualization <NUM>. Some guidance can be static whereas other guidance can be dynamic. For example, the guidance generator <NUM> can include a marker tool <NUM> programmed to place a graphical marker at a user-selected location with respect to an anatomic structure. The marker tool <NUM> can correspond to the marker tool <NUM> of <FIG>.

By way of example, the marker tool <NUM> may be activated in response to a user input of through the user interface <NUM>. The marker tool 336can be programmed to display a graphical representation of a virtual anatomic reference structure, such as a centerline. The marker tool can further be programmed to determine a location parameter for the marker along the centerline (or other reference) in response to a first user input from the user input device. The location parameter may be derived by hovering a pointer of the user interface <NUM> over the graphical representation of the centerline. The rendering method <NUM> may generate the visualization data <NUM> to display a graphical proxy marker based on the location parameter. For example, the graphical proxy marker may be a toroidal object circumscribing the graphical representation of centerline. The placement of the proxy marker can be adjusted as the pointer moves along the centerline. A fixed location along the centerline for a final graphical marker may then be selected in response to a user input from the from the user interface <NUM>, such as disclosed herein. The fixed location can be stored in memory associated with a respective point along the centerline (e.g., geometric knot of a centerline model). The marker tool may be closed and corresponding visualization data <NUM> be generated to includes a visualization of the anatomic structure and a graphical representation of the final graphical marker at the selected anatomic location of the structure. If the tracking system is operating and the sensor is within the anatomic structure, the output generator can also include a graphical representation of the object in the visualization (see, e.g., <FIG>).

In some examples, the guidance generator <NUM> also includes an object tool <NUM> programmed to provide graphical guidance with respect to an object (e.g., guidewire, catheter and/or other instrument) that is rendered in the visualization based on the tracking data and the object model data. For example, object tool <NUM> provides guidance relating to the position and/or trajectory path of the object and the spatial position (three-dimensional spatial position) of one or more anatomic features or graphical marker. As mentioned, the graphical marker may be placed at a waypoint or target site. As an example, a distance calculator can determine a spatial distance between a selected feature (e.g., graphical marker or anatomic landmark) and a point along the object (e.g., corresponding to a distal tip of an instrument or other predetermined location along the object). The guidance generator <NUM> can provide a visual indicator, an audible indicator or a combination of audible and visual indicators based on the distance. For example, an audible indicator can provide a series of beeps or tones that increase in frequency as a function of decreasing distance between the object and the location of the target feature. The guidance generator <NUM> can specify a color code to be applied to the graphical marker or other selected feature of the output visualization, such as green to indicate that the position is on target, yellow to indicate a deviation within a predetermined parameter or red to indicate that the position is outside of expected parameters.

As a further example, the guidance generator <NUM> can also include a directional indicator that can produce virtualized graphical indictor, such as to show a direction that a distal tip of the object (e.g., a catheter or wire) is oriented. The graphical indicator can be rendered as a series of short lines translated along a given sensors positive Z axis. The visual indicator thus can provide an easy way to determine whether the object is aligned with a given part of the anatomic structure (e.g., as specified by a graphical marker) to facilitate advancing the object (e.g., catheter and/or guidewire) through or into a graphical marker that has been positioned at an ostium of a target branch vessel. As mentioned above, the appearance of the graphical marker and other guidance visualization further will vary depending on the particular viewing angle that is being produced.

The guidance generator <NUM> can also provide information to graphically differentiate anatomic locations or other target sites, such as by using different color codes for different structures of interest. For example, the graphical marker can be rendered as a thick (e.g., multi-voxel thick) ring or cylindrical body residing at a selected location (e.g., ostium waypoint or target location). The guidance generator <NUM> can provide additional feedback to the user. For example, when the tip of the object gets within a predetermined distance of an ostium, which has been indicated by a graphical marker placed at such location, as disclosed herein, the guidance generator <NUM> can change colors (and/or brightness) of the graphical marker in each view as the tip is determined to be within a predetermined spatial distance of the graphical marker.

<FIG> depicts an example of an output visualization <NUM> that can be generated by the output generator <NUM> (also corresponding to output generator <NUM> of <FIG>) based on the tracking data <NUM>, the anatomic model data <NUM> and the object model data <NUM>. In the example of <FIG>, the output visualization <NUM> is a cut-away view (similar to <FIG>) that includes a graphical representation of the anatomic structure <NUM> (e.g., the descending aorta) as well as the graphical marker <NUM> and a graphical representation of an instrument <NUM>. As described above, the graphical marker <NUM> is placed at a user-selected position (e.g., using marker tool <NUM>, <NUM>) to facilitate intra-operative position of an actual physical surgical instrument that is graphically represented by <NUM> (e.g., generated by output generator <NUM>, <NUM>). That is, the position of the instrument <NUM> with respect to the anatomic structure <NUM> and graphical marker <NUM> in the visualization <NUM> varies over time based on tracking data <NUM> that is registered into the same coordinate system as the data used to produce the visualization of the anatomic structure <NUM> and graphical marker <NUM>.

In view of the foregoing structural and functional features described above, methods that can be implemented will be better appreciated with reference to <FIG>. While, for purposes of simplicity of explanation, the method of <FIG> is shown and described as executing serially, they are not to be limited by the illustrated order, as the methods could occur in different orders and/or concurrently with other aspects from that shown and described herein. Moreover, not all illustrated features may be required to implement a method. The methods or portions thereof can be implemented as instructions stored in one or more non-transitory storage media as well as be executed by a processor of a computer device or special purpose computer device (e.g., a dedicated computer or workstation) to access data sources and perform the functions disclosed herein, for example.

<FIG> depicts a flow diagram of a method <NUM> that can be utilized to place one or more reference markers with respect to anatomic structures. The method <NUM> begins at <NUM> such as in connection with activating a marker tool (e.g., marker tool <NUM>, <NUM>) of a visualization system (e.g., system <NUM>, <NUM>). The method <NUM> can implemented in conjunction with a computer system that includes a user input device (e.g., user input device <NUM> or interface <NUM>) such as disclosed herein.

At <NUM>, a position of a pointer is calculated. For example, the position may be calculated in response to user input data from an input device. The position can be spatial coordinates in a visualization that also includes a graphical representation of an anatomic reference of an anatomic structure. For example, the anatomic reference may be a centerline or an axis of symmetry, which may vary depending upon the shape and size of the anatomic structure being visualized. In an example, the pointer position may be calculated by a position calculator <NUM> or provide as part of user input data from the user interface associated with the input device.

At <NUM>, a determination is made as to whether the pointer is detected along the reference (e.g., centerline). For example, the determination may be made by position calculator <NUM> by comparing spatial coordinates of the pointer and the graphical representation of the anatomic reference (e.g., centerline) that is rendered in the visualization. If the pointer is not detected to be along the reference, the method returns to <NUM> to continue monitoring the relative position between the pointer and the visualized reference. If the pointer is detected along the centerline at <NUM>, method proceeds to <NUM>. Additionally, in response to detecting the pointer along the centerline, the location (e.g., spatial coordinates in the visualization) of the pointer on the centerline may be stored in memory, which may be linked to a nearest given point along the reference structure (e.g., a geometric knot). At <NUM>, a proxy marker is generated (e.g., by marker generator <NUM>) at the detected location. For example, the location can correspond to a nearest geometric knot of the reference structure.

At <NUM>, a determination is made as to whether a user input has set the marker location. For example, the marker location may be set in response to a user input with a user input device (e.g., clicking on the proxy marker or selecting a button). If no user input is received to set the location, the method returns to <NUM> to continue and repeat the process in which the proxy marker may be generated and visualized with respect to the reference structure of the marker tool as the pointer moves along the reference structure. In response to a user input setting the marker location at <NUM>, the method proceeds to <NUM> to generate an output visualization that includes a final marker in the output visualization, such as disclosed herein (see, e.g., <FIG> and <FIG>). For example, the output visualization can be generated (e.g., by output generator <NUM> of <FIG> or generator <NUM> of <FIG>) to represent a location of the final graphical marker at the user-selected location relative to the geometry of the anatomic structure. In some examples, the output visualization can include a graphical representation of an instrument generated in substantially real time based on tracking data from one or more sensors to facilitate positioning and guidance of the instrument. The output visualization can include any number of concurrently generated views, which can be modified (e.g., in response to a user input), such as disclosed herein. Each view can further include the graphical marker or the marker may be hidden in response to another user input.

In view of the foregoing structural and functional description, those skilled in the art will appreciate that portions of the systems and method disclosed herein may be embodied as a method, data processing system, or computer program product such as a non-transitory computer readable medium. Accordingly, these portions of the approach disclosed herein may take the form of an entirely hardware embodiment, an entirely software embodiment (e.g., in one or more non-transitory machine-readable media), or an embodiment combining software and hardware. Furthermore, portions of the systems and method disclosed herein may be a computer program product on a computer-usable storage medium having computer readable program code on the medium. Any suitable computer-readable medium may be utilized including, but not limited to, static and dynamic storage devices, hard disks, optical storage devices, and magnetic storage devices.

Certain embodiments have also been described herein with reference to block illustrations of methods, systems, and computer program products. It will be understood that blocks of the illustrations, and combinations of blocks in the illustrations, can be implemented by computer-executable instructions. These computer-executable instructions may be provided to one or more processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus (or a combination of devices and circuits) to produce a machine, such that the instructions, which execute via the processor, implement the functions specified in the block or blocks.

These computer-executable instructions may also be stored in computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture including instructions that implement the function specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

Claim 1:
One or more non-transitory computer readable media (<NUM>) having instructions stored thereon, for execution by a computer that includes a graphical user interface (<NUM>) and a user input device (<NUM>), the instructions to perform a method (<NUM>) comprising:
displaying a graphical representation of an anatomic structure on the graphical user interface (<NUM>), the graphical representation of the anatomic structure being determined from a geometry of the anatomic structure and wherein geometric knots are determined at respective positions along a centerline of the graphical representation of the anatomic structure;
determining a location parameter for a graphical marker along the graphical representation of the anatomic structure in response to a first user input from the user input device, wherein the location parameter for the graphical marker corresponds to spatial coordinates for a closest geometric knot of the geometric knots, and wherein the graphical marker represents a location of an anatomic opening between the anatomical structure and another adjacent fluidly connected anatomical structure;
displaying the graphical marker relative to the graphical representation of the anatomic structure on the graphical user interface based on the location parameter;
changing the location parameter to move the graphical marker along the graphical representation of the anatomic structure based on a user input from a user input device (<NUM>);
selecting a fixed location along the graphical representation of the anatomic reference to fix the graphical marker relative to the graphical representation of the anatomic structure in response to a second user input from the user input device (<NUM>); and
generating an output visualization that includes at least one view of the graphical representation of the anatomic structure and the graphical representation of the graphical marker at the fixed location.