Patent Description:
Given the breadth of additional information available during image guided surgery, a surgeon or other practitioner may sometimes become spatially disoriented while switching between views of the surgical site, or while switching between direct viewing of the patient anatomy and viewing of simulated images of the patient anatomy. For example, in some cases a surgeon may be viewing a CT slice of an axial view of the patient's head on a display of the image guided surgery navigation system, while also occasionally viewing the patient's head directly. In such cases, the surgeon may become disoriented and unable to determine the spatial and directional correspondence between their direct perceptions and the axial view, which may lead to erroneous movements of the surgical instrument within the patient's head.

While several systems and methods have been made and used in ENT procedures, it is believed that no one prior to the inventors has made or used the invention described in the appended claims.

<CIT> describes an intra-operative medical image viewing system can allow a surgeon to maintain a viewing perspective on the patient while calling-up visual images on-the-fly. A digital image source has at least one image file representative of an anatomical or pathological feature of a patient. A display is worn by the surgeon or positioned between the surgeon and her patient during surgery. The display is selectively transparent, and exhibits to the surgeon an image derived from the image file. An image control unit retrieves the image file from the image source and controls the display so that at least a portion of the image depiction can be exhibited and modified at will by the surgeon. A plurality of peripheral devices are each configured to receive an image control input from the surgeon and, in response, generate an image control signal. Each peripheral accepts a different user-interface modality. <CIT> describes a surgical navigation system. <CIT> describes methods and systems for display of patient data in computer-assisted surgery.

For clarity of disclosure, the terms "proximal" and "distal" are defined herein relative to a surgeon, or other operator, grasping a surgical instrument having a distal surgical end effector. The term "proximal" refers to the position of an element arranged closer to the surgeon, and the term "distal" refers to the position of an element arranged closer to the surgical end effector of the surgical instrument and further away from the surgeon. Moreover, to the extent that spatial terms such as "upper," "lower," "vertical," "horizontal," or the like are used herein with reference to the drawings, it will be appreciated that such terms are used for exemplary description purposes only and are not intended to be limiting or absolute. In that regard, it will be understood that surgical instruments such as those disclosed herein may be used in a variety of orientations and positions not limited to those shown and described herein.

As used herein, the terms "about" and "approximately" for any numerical values or ranges indicate a suitable dimensional tolerance that allows the part or collection of components to function for its intended purpose as described herein.

When performing a medical procedure within a head (H) of a patient (P), it may be desirable to have information regarding the position of an instrument within the head (H) of the patient (P), particularly when the instrument is in a location where it is difficult or impossible to obtain an endoscopic view of a working element of the instrument within the head (H) of the patient (P). <FIG> shows an exemplary IGS navigation system (<NUM>) enabling an ENT procedure to be performed using image guidance. In addition to or in lieu of having the components and operability described herein IGS navigation system (<NUM>) may be constructed and operable in accordance with at least some of the teachings of <CIT>; and <CIT>, now abandoned.

IGS navigation system (<NUM>) of the present example comprises a field generator assembly (<NUM>), which comprises set of magnetic field generators (<NUM>) that are integrated into a horseshoe-shaped frame (<NUM>). Field generators (<NUM>) are operable to generate alternating magnetic fields of different frequencies around the head (H) of the patient (P) to produce a tracked area that the IGS navigation system (<NUM>) associates a coordinate system with. A navigation guidewire (<NUM>) is inserted into the head (H) of the patient (P) in this example. Navigation guidewire (<NUM>) may be a standalone device or may be positioned on an end effector or other location of a medical instrument such as a surgical cutting instrument or dilation instrument. In the present example, frame (<NUM>) is mounted to a chair (<NUM>), with the patient (P) being seated in the chair (<NUM>) such that frame (<NUM>) is located adjacent to the head (H) of the patient (P). By way of example only, chair (<NUM>) and/or field generator assembly (<NUM>) may be configured and operable in accordance with at least some of the teachings of <CIT>.

IGS navigation system (<NUM>) of the present example further comprises a processor (<NUM>), which controls field generators (<NUM>) and other elements of IGS navigation system (<NUM>). For instance, processor (<NUM>) is operable to drive field generators (<NUM>) to generate alternating electromagnetic fields; and process signals from navigation guidewire (<NUM>) to determine the location of a sensor in navigation guidewire (<NUM>) within the head (H) of the patient (P). Processor (<NUM>) comprises a processing unit (e.g., a set of electronic circuits arranged to evaluate and execute software instructions using combinational logic circuitry or other similar circuitry) communicating with one or more memories. Processor (<NUM>) of the present example is mounted in a console (<NUM>), which comprises operating controls (<NUM>) that include a keypad and/or a pointing device such as a mouse or trackball. A physician uses operating controls (<NUM>) to interact with processor (<NUM>) while performing the surgical procedure.

Navigation guidewire (<NUM>) includes a sensor (not shown) that is responsive to positioning within the alternating magnetic fields generated by field generators (<NUM>). A coupling unit (<NUM>) is secured to the proximal end of navigation guidewire (<NUM>) and is configured to provide communication of data and other signals between console (<NUM>) and navigation guidewire (<NUM>). Coupling unit (<NUM>) may provide wired or wireless communication of data and other signals.

In the present example, the sensor of navigation guidewire (<NUM>) comprises at least one coil at the distal end of navigation guidewire (<NUM>). When such a coil is positioned within an alternating electromagnetic field generated by field generators (<NUM>), the alternating magnetic field may generate electrical current in the coil, and this electrical current may be communicated along the electrical conduit(s) in navigation guidewire (<NUM>) and further to processor (<NUM>) via coupling unit (<NUM>). This phenomenon may enable IGS navigation system (<NUM>) to determine the location of the distal end of navigation guidewire (<NUM>) or other medical instrument (e.g., dilation instrument, surgical cutting instrument, etc.) within a three-dimensional space (i.e., within the head (H) of the patient (P), etc.). To accomplish this, processor (<NUM>) executes an algorithm to calculate location coordinates of the distal end of navigation guidewire (<NUM>) from the position related signals of the coil(s) in navigation guidewire (<NUM>). While the position sensor is located in guidewire (<NUM>) in this example, such a position sensor may be integrated into various other kinds of instruments, such as dilation catheters, guide catheters, guide rails, suction instruments, pointer instruments, registration probes, curettes, patient trackers, and other instruments, including those described in greater detail below.

Processor (<NUM>) uses software stored in a memory of processor (<NUM>) to calibrate and operate IGS navigation system (<NUM>). Such operation includes driving field generators (<NUM>), processing data from navigation guidewire (<NUM>), processing data from operating controls (<NUM>), and driving display screen (<NUM>). In some implementations, operation may also include monitoring and enforcement of one or more safety features or functions of IGS navigation system (<NUM>). Processor (<NUM>) is further operable to provide video in real time via display screen (<NUM>), showing the position of the distal end of navigation guidewire (<NUM>) in relation to a video camera image of the patient's head (H), a CT scan image of the patient's head (H), and/or a computer generated three-dimensional model of the anatomy within and adjacent to the patient's nasal cavity. Display screen (<NUM>) may display such images simultaneously and/or superimposed on each other during the surgical procedure. Such displayed images may also include graphical representations of instruments that are inserted in the patient's head (H), such as navigation guidewire (<NUM>), such that the operator may view the virtual rendering of the instrument at its actual location in real time. By way of example only, display screen (<NUM>) may provide images in accordance with at least some of the teachings of <CIT>. In the event that the operator is also using an endoscope, the endoscopic image may also be provided on display screen (<NUM>).

The images provided through display screen (<NUM>) may help guide the operator in maneuvering and otherwise manipulating instruments within the patient's head (H) when such instruments incorporate navigation guidewire (<NUM>). It should also be understood that other components of a surgical instrument and other kinds of surgical instruments, as described below, may incorporate a sensor like the sensor of navigation guidewire (<NUM>).

In some implementations, the IGS navigation system (<NUM>) may include a patient tracking assembly (<NUM>) that may be placed on the head (H) of the patient, or another appropriate portion of the patient (P) as shown in <FIG>. By tracking the head (H) separately from the guidewire (<NUM>) but within the same coordinate system, the positions and orientations of the guidewire (<NUM>) and the head (H) may be determined relative to each other during a procedure. This may be advantageous where the head (H) is registered to determine its initial position within the coordinate system, but then later moves during the procedure. By tracking the head (H) independently, a rotation or other movement may be detected, and the initial registration may be updated to account for the new position of the head (H). In this manner, any image guided navigation features being used during the procedure, such as the display of CT image slices with overlaid markers indicating the position of the guidewire (<NUM>) within the head (H), may be automatically updated in response to such movements. Implementations of the patient tracking assembly (<NUM>) may be constructed and operable with the IGS navigation system (<NUM>) in accordance with any of the teachings of <CIT>.

As one example of the patient tracking assembly (<NUM>), <FIG> shows a patient tracking assembly (<NUM>) that may be readily incorporated into the IGS navigation system (<NUM>). The patient tracking assembly (<NUM>) includes a disposable portion (<NUM>) and a reusable portion (<NUM>). The disposable portion (<NUM>) is configured to attach to the patient's head (H), or another suitable portion of patient (e.g., using a flexible adhesive pad), and is configured to selectively couple with a coupling block (<NUM>) of the reusable portion (<NUM>) such that the reusable portion (<NUM>) is fixed relative to the head or another portion of patient (P) during exemplary use; while the reusable portion (<NUM>) is configured to communicate with the processor (<NUM>) in order to track the position of the head (H) of patient (P).

The reusable portion (<NUM>) includes a cable (<NUM>) extending proximally from a coupling assembly (<NUM>), and a sensor (<NUM>). The coupling assembly (<NUM>) is adapted to couple the reusable portion (<NUM>) with the disposable portion (<NUM>) during use. When properly coupled with the disposable portion (<NUM>), the sensor (<NUM>) may be utilized with the processor (<NUM>) to determine the location of the tracked anatomy, such that the processor (<NUM>) may accurately display the location of the navigation guidewire (<NUM>) (or any other suitable instrument) relative to the anatomy of patient (P) during exemplary use. The cable (<NUM>) is configured to provide a conduit for communication between the sensor (<NUM>) and the processor (<NUM>) during exemplary use. Therefore, the cable (<NUM>) may directly connect such that sensor (<NUM>) is in wired communication with the processor (<NUM>) via the cable (<NUM>). Alternatively, the cable (<NUM>) may connect the sensor (<NUM>) with a wireless communication device that is in wireless communication with the processor (<NUM>), similar to how the coupling unit (<NUM>) establishes wireless communication between the navigation guidewire (<NUM>) and the processor (<NUM>).

<FIG> shows a visualization system (<NUM>) that may be implemented with the IGS navigation system (<NUM>) in order to provide additional information and image navigation views during a surgical procedure. In particular, the visualization system (<NUM>) may provide navigation views that display information in an augmented reality view that is overlaid upon images of the patient that are captured during a procedure in near real-time. This may advantageously reduce the frequency and need for a surgeon to view the display screen (<NUM>) or another display instead of viewing the patient during a procedure, and may also aid the surgeon in spatially orienting themselves between the physical world and the image navigation views provided by the IGS navigation system (<NUM>).

The visualization system (<NUM>) may be implemented with the IGS navigation system (<NUM>) by configuring one or more of a head mounted display ("HMD") (<NUM>), a handheld display ("HHD") (<NUM>), or another similar device to communicate with the IGS navigation system (<NUM>) during an image guided surgical procedure, as will be described in more detail below. <FIG> shows a perspective view of the HMD (<NUM>), which includes a frame (<NUM>) that may be worn on the face of a surgeon or other user involved with the image guided surgical procedure. A case (<NUM>) is shown mounted to the frame (<NUM>), but may also be worn or mounted elsewhere and coupled with devices mounted on the frame (<NUM>) via a wired or wireless connection to those devices. The case (<NUM>) includes a perspective sensor (<NUM>) that is operable to provide spatial information indicating the perspective, the location, or both of the HMD (<NUM>). The perspective sensor (<NUM>) may include one or more of a gyroscope, an accelerometer, or a position sensor (e.g., such as the sensor of the guidewire (<NUM>)), for example. Information from the perspective sensor (<NUM>) may be usable to determine one or more aspects of a visual perspective of a camera (<NUM>) that is mounted to the frame (<NUM>), which may include determining the orientation of an optical axis the camera (<NUM>) (e.g., rotational coordinates for one or more axes), the location of the camera (<NUM>) relative to the coordinate system of the IGS navigation system (<NUM>) (e.g., position coordinates for one more axes), or both, as will be described in more detail below.

The case (<NUM>) also includes a communication device (<NUM>), which may be a wired or wireless transceiver capable of communicating with the processor (<NUM>) or other devices, and a processor and memory (<NUM>) configured to process and store data and execute functions related to the function of the HMD (<NUM>). A power source (<NUM>) is also included, which may be a battery or a connection to an external power source, and which is configured to provide power to the processor and memory (<NUM>), communication device (<NUM>), camera (<NUM>), display (<NUM>), and other components of the HMD (<NUM>).

When worn, the frame (<NUM>) positions the camera (<NUM>), which is mounted to the frame (<NUM>) and/or the case (<NUM>), such that its optical axis is substantially parallel to an optical axis of the wearer's eyes when the wearer looks straight ahead. When used herein, the term "neutral optical axis" may refer to the optical axis of a wearer's eye, when the wearer of the HMD (<NUM>) looks substantially straight ahead (e.g., where the pupil of the eye is substantially centered both vertically and horizontally within the orbit or socket of the eye). In this manner, the camera (<NUM>) captures images that have a similar field of view as that of the wearer of the HMD (<NUM>). As an example, the camera (<NUM>) may capture images that include some or all of the field of view of the wearer's right eye, which the camera (<NUM>) is positioned most proximately to, when the wearer is looking straight ahead. The camera (<NUM>) may be capable of capturing images, video, and audio, which may be stored by the processor and memory (<NUM>), transmitted to the processor (<NUM>) via the communication device (<NUM>), or transmitted to and displayed or presented on another device, as will be apparent to those skilled in the art in light of this disclosure. Image data captured by the camera (<NUM>) may also be used for computer vision and other analysis which may include, for example, identifying objects or other visual characteristics of a captured image. Such analysis may be performed by the processor (<NUM>), the processor (<NUM>), or both, or may also be performed using various cloud computing or edge processing techniques, as will be apparent to those skilled in the art in light of this disclosure.

A display (<NUM>) is also mounted to the frame (<NUM>) and/or the case (<NUM>) and is positioned to be within the field of view of the wearer of the HMD (<NUM>). In some implementations, the display (<NUM>) is at least partially translucent if not transparent and is operable by the processor (<NUM>) to render images that appear to be overlaid upon the field of view of the wearer. As an example, the display (<NUM>) may display an image captured by the camera (<NUM>), which may block some or all of the wearer's field of view from the proximate eye, but would otherwise appear similar to the wearer's normal field of vision for that eye. As another example, image analysis may be performed on a captured image to identify objects of interest within that image (e.g., in the context of surgical navigation, a human face or another portion of human anatomy), and the display (<NUM>) may be operated to render a visual marker that appears, to the wearer, to be overlaid upon their direct view (e.g., viewed through the transparent portion of the display (<NUM>)) of the identified object. In some implementations of the above, optical markers or other fiducial markers may be placed on objects of interest in order to provide image data having objects that are easily identifiable due to their reflectivity, shape, or other visual characteristics, such that an optical marker placed on a human face may be identified rather than the human face itself.

As yet another example, the display (<NUM>) may be operated to render visual markings that overlay the wearer's direct view of their field of view based on other inputs instead of or in addition to image analysis or machine vision. This may include, for example, rendering visual markings based on information from the perspective sensor (<NUM>), the processor (<NUM>), the patient tracker (<NUM>) (e.g., through communication with the processor (<NUM>)), and other devices. This could include rendering a visual marking providing information associated with the rotational perspective of the HMD (<NUM>) (e.g., based on a gyroscopic feature of the perspective sensor (<NUM>)). As another example, this could include rendering a visual marking that overlays a surgical instrument (e.g., the guidewire (<NUM>)), based upon tracking information associated with the surgical instrument and the HMD (<NUM>). In other words, when the processor (<NUM>) is able to track and determine the relative positions of the surgical instrument and the HMD (<NUM>), and the orientation of the HMD (<NUM>) may be determined (e.g., using the perspective sensor (<NUM>)), the position and scale of the tracked objects relative to each other may be determined and produced as rendered markings via the display (<NUM>).

As will be apparent to those skilled in the art in light of this disclosure, some or all of the above features may also be performed with other displays beyond the display (<NUM>). For example, in some implementations, a separate display (e.g., the display screen (<NUM>) or a wall mounted display that is visible to the entire room) may be configured to receive and display images captured from the camera (<NUM>) and any markings, renderings, or other overlay data that may be added. In such an implementation, the display (<NUM>) may render only overlay images, while the separate display may render a combination of an image captured by the camera (<NUM>) and any corresponding overlay images. This may allow other participants in the procedure to view the additional navigation views in addition to the wearer of the HMD (<NUM>). In some such implementations, the HMD (<NUM>) may not include the display (<NUM>), and combinations of captured images and rendered overlays may be viewable on the display screen (<NUM>) or another display positioned near the patient.

The HHD (<NUM>) may share some or all of the capabilities of the HMD (<NUM>), and may be, for example, a smartphone, a tablet, a proprietary device, or other handheld computing devices having capabilities such as processing and storing data, executing functions, sending and receiving data, capturing images, providing spatial information (e.g., orientation, location, or both). The display of the HHD (<NUM>) may commonly be a LED or LCD display, and so may not be capable of overlaying rendered markings onto a transparent surface through which objects are viewed directly, such as the display (<NUM>) might. In some implementations, the HMD (<NUM>) or HHD (<NUM>) may be modified to include additional capabilities. For example, some implementations of the HMD (<NUM>) may not include the capability to self-position relative to the coordinate system of the IGS navigation system (<NUM>). In such cases, a sensor may be mounted (e.g., externally or internally) on the HMD (<NUM>) that allows it to interact with and be tracked by the IGS navigation system (<NUM>), similar to the guidewire (<NUM>) and the patient tracking assembly (<NUM>). Thus, the capabilities of the perspective sensor (<NUM>) may include both those present in the HMD (<NUM>) by default, as well as those that may be later added, whether they are within the case (<NUM>) or externally mounted on the HMD (<NUM>).

In some implementations, information indicating the orientation and location of the HMD (<NUM>) may be available from multiple sources. As one example, the perspective sensor (<NUM>) may include a gyroscopic feature capable of determining rotational orientation, may include or incorporate a tri-axis sensor capable of being tracked by the IGS navigation system (<NUM>) to determine rotational orientation, and may be configured to identify optical fiducials present within images captured by the camera (<NUM>). In such examples, the processing components (e.g., the processor (<NUM>), the processor (<NUM>), or other processors) may be configured to determine orientation or location in various ways by balancing performance and accuracy, as will be apparent to those skilled in the art in light of this disclosure. For example, some implementations may determine orientation or location based only on gyroscopic information from the HMD (<NUM>) itself or tracking information from the IGS navigation system (<NUM>) to emphasize performance, while other implementations may use a combination of gyroscopic, magnetic tracking, and image analysis information with a goal of achieving a higher accuracy at the potential cost of some performance (e.g., the delay, if any, between the orientation or location of the HMD (<NUM>) changing and a determination of the new orientation or location being completed).

The visualization system (<NUM>) allows for additional inputs and information to be gathered and used during surgical navigation in order to provide additional navigation views and other feedback to users of the HMD (<NUM>) or the HHD (<NUM>), or viewers of other displays that are configured to display images from the camera (<NUM>) and any corresponding overlay renderings. In the absence of the visualization system (<NUM>), there is a breadth of useful information available to the IGS navigation system (<NUM>) that is largely confined to being used or displayed in the context of pre-operative images (e.g., <NUM>-D patient models produced from pre-operative imaging, CT, MRI, or ultrasound image sets).

As an example, the IGS navigation system (<NUM>) may allow a surgeon to view a set of CT images for a patient prior to a procedure and plot out a surgical plan or surgical path that the surgeon will navigate one or more surgical instruments along during the procedure. During the procedure, the surgical path may then be overlaid on the CT image set and displayed via the display screen (<NUM>), allowing the surgeon to switch between a limited number of views (e.g., axial, coronal, sagittal) and slices as may be desired. As another example, during a procedure, a surgical instrument (e.g., the guidewire (<NUM>)) may also be tracked and similarly displayed on the CT image set. When displayed together, a surgeon may find it beneficial to view CT images that show the tracked position of the surgical instrument and the planned surgical path in relation to each other.

While useful, the above features can distract the surgeon's attention from the patient, as they may need to look away from the patient to view a nearby display device. It can also be disorienting when switching between the axial, coronal, and sagittal views, as the surgeon's actual location relative to the patient has not changed. For example, a surgeon may be viewing an axial plane CT image of the patient head and, when returning their view to the patient, may be observing a sagittal plane of the patient's head. In order to make use of the information displayed on the CT image, such as the location and orientation of the surgical instrument, the surgical path, and nearby anatomical cavities and structures, the surgeon must first mentally transform or relate their spatial understanding of the axial plane to the sagittal plane in order to know which direction to navigate the surgical instrument. This process can be mentally taxing, may consume valuable time during a procedure, and may also lead to erroneous navigation of the surgical instrument.

To address this, the visualization system (<NUM>) provides a framework for relating the coordinate system and associated information to the physical world perceived by the wearer of the HMD (<NUM>). Such associated information may include, for example, CT images, configured surgical paths, real time surgical instrument tracking, real time patient tracking, configured points of interest indicating areas that should be investigated or avoided, and other similar information that may be correlated to a coordinate system for IGS navigation, which may be collectively referred to herein as correlated datasets.

Once related, these correlated datasets can then be displayed via the display (<NUM>) so that they are available when directly looking at the patient instead of only being displayed on the display screen (<NUM>) or another nearby display. In addition to reducing the need to refer to external displays, such an implementation allows the wearer of the HMD (<NUM>) to browse or navigate the correlated datasets by changing their perspective relative to the patient, instead of using a mouse or keyboard to step through image slices, switch between viewable planes, or rotate <NUM>-D models. For example, in the case of a tracked surgical instrument location and surgical path, the surgeon may be able to view a rendered overlay of the instrument location and surgical path within the patient's head from different perspectives by moving and observing from different angles, rather than being confined to stepping through CT image slices and between CT image planes using a mouse or keyboard interface.

As an exemplary implementation of the above, <FIG> shows a set of steps (<NUM>) that may be performed with the visualization system (<NUM>) to render and display correlated datasets via the HMD (<NUM>) or another viewing device, while <FIG> and <FIG> show exemplary interfaces that may be displayed or viewed with the visualization system (<NUM>). After positioning a patient for a procedure, the patient may be registered (block <NUM>) and, in implementations including the patient tracking assembly (<NUM>), tracked within the IGS navigation coordinate system. Registration of the patient may include using a registration probe or other device to provide the coordinate system with a plurality of locations that correspond to patient anatomy, may include placing, calibrating, and using the patient tracking assembly (<NUM>), or both, or may include other registration techniques. Registration (block <NUM>) may also include registering and tracking other devices and instruments, such as the guidewire (<NUM>) and other trackable surgical instruments. Registration (block <NUM>) may also include registering and tracking the HMD (<NUM>), where it is capable of being positionally tracked by the IGS navigation system (<NUM>).

The IGS navigation system (<NUM>) may also receive (block <NUM>) one or more correlated datasets that are associated with the patient and procedure, which may include pre-operative images of the patient anatomy (e.g., CT, MRI, and ultrasound image sets), pre-configured surgical plans and surgical paths, and other pre-configured or preoperatively captured or generated datasets that may be associated with the IGS navigation coordinate system. The received (block <NUM>) correlated datasets may also include data that is captured in real-time during the procedure and then associated with the coordinate system, such as position tracking data indicating the location of the guidewire (<NUM>) and other tracked surgical instruments, and position tracking data indicating the location of the HMD (<NUM>).

When the HMD (<NUM>) or another device (e.g., the HHD (<NUM>)) is in use with the visualization system (<NUM>), images may be captured (block <NUM>) by the camera (<NUM>), as has been described. In some implementations, images will be captured (block <NUM>) constantly based upon a configured framerate of the camera (<NUM>), such that each subsequent image may change slightly from the previous image based upon movements of the wearer of the HMD (<NUM>). Captured (block <NUM>) images may be stored by the processor and memory (<NUM>) and may be transmitted to the processor (<NUM>) or another device.

As images are captured (block <NUM>), the visualization system (<NUM>) may repeatedly determine (block <NUM>) the orientation of the HMD (<NUM>) relative to the anatomy and repeatedly determine (block <NUM>) the distance of the HMD (<NUM>) relative to the anatomy. These determinations (block <NUM>, block <NUM>) may be made continuously and independently, or may be made for each captured (block <NUM>) image one a one-to-one basis (e.g., where the camera (<NUM>) captures thirty images or frames per second, the visualization system (<NUM>) would determine orientation (block <NUM>) and distance (block <NUM>) thirty times per second, once for each image) or some other correspondence (e.g., the visualization system may be configured to determine orientation (block <NUM>) and distance (block <NUM>) once for every three captured (block <NUM>) images), as will be apparent to those skilled in the art in light of this disclosure.

The orientation (<NUM>) and distance (<NUM>) may be determined in varying ways, as has already been described. For example, in some implementations, each of the HMD (<NUM>) and the patient head (H) may be positionally and orientationally tracked by the IGS navigation system (<NUM>) and the distance and orientation may be determined using the IGS navigation coordinate system. In some implementations, the orientation and/or distance may be determined using the perspective sensor (<NUM>) of the HMD (<NUM>).

In some implementations, the orientation and/or distance may be determined using image analysis of a captured (block <NUM>) image to identify a particular object (e.g., an optical fiducial) or patient anatomy (e.g., an eye). For example, particularly in the case of an optical fiducial having a predictable size, shape, and other characteristics, image analysis of an image containing the optical fiducial can indicate the distance and perspective from which the optical fiducial is viewed. With reference to <FIG>, the patient tracking assembly (<NUM>) can be seen placed on the head (H) of a patient. One or more optical fiducials (<NUM>, <NUM>) may be placed on the patient tracking assembly (<NUM>) or elsewhere, as may be desirable.

The appearance of the optical fiducial (<NUM>) in an image provides an indication of the perspective from which the image was captured (e.g., the optical fiducial (<NUM>) may appear as a circle when viewed as shown in <FIG>, but may appear as an oval as a viewer moves to the left or right) as well as the distance (e.g., the optical fiducial might have a diameter of <NUM>. Where the optical fiducial (<NUM>) has an asymmetrical shape of sufficient complexity, or where the surface of the optical fiducial (<NUM>) has a pattern or other visible characteristic of sufficient complexity, a single fiducial may provide enough information to fully determine orientation and distance.

Additionally, where several optical fiducials (<NUM>, <NUM>) are used, such as is shown in <FIG>, the apparent positions of each fiducial relative to the others may be used as an indication of orientation. As such, it may be advantageous to place two or more optical fiducials on the tracking assembly (<NUM>) or the head (H). In some implementations, one or more fiducials may be integrated onto the surface of the tracking assembly (<NUM>) at the time of manufacture, which may advantageously indicate a static and known positioning and distance from each other which may be used to aid in subsequent fiducial based orientation and distance determinations.

As has been described, implementations may vary in the particular approach that is taken for determining the orientation (block <NUM>) and the distance (block <NUM>), and while some implementations may rely entirely on tracking each object of interest within the IGS navigation coordinate system, others may combine such tracking with image analysis of optical fiducials or other techniques in order to improve accuracy, performance, or other characteristics of the results. As such, it should be understood that various combinations of the disclosed methods and others exist and will provide varying advantages for determining the orientation (block <NUM>) and the distance (block <NUM>), and such combinations will be apparent to those skilled in the art based on this disclosure.

Once determined, the distance and orientation relative to the viewed anatomy can then be used to transform a correlated dataset so that it may be displayed via the display (<NUM>) as a rendered overlay of the viewed anatomy, or displayed via another device as a rendered overlay of a captured image. Correlated dataset transformations may include, for example, transforming (block <NUM>) a surgical path to match the scale and perspective of the captured (block <NUM>) image, transforming (block <NUM>) a CT image or other image type to match the scale and perspective of the captured (block <NUM>) image, transforming (block <NUM>) the tracked location of a surgical tool distal tip to match the scale and perspective of the captured (block <NUM>) image, and other transformations. While they are discussed within the context of <FIG>, it should be understood that a surgical path, CT image, and tracked surgical tool are not required to be present within the received (block <NUM>) correlated datasets.

Transformation of correlated datasets will vary depending upon the particular data represented in a correlated dataset. For example, with reference to <FIG>, where the system is configured to overlay some or all of a CT image slice on a patient's head (H), the distance (<NUM>) between a perspective point (<NUM>) (e.g., the lens of the camera (<NUM>)) and a viewed point (<NUM>) (e.g., the first point, voxel, or other object within the coordinate system intersected by the optical axis of the camera (<NUM>)) may be used to determine and apply a scaling factor to transform and control the scale at which the CT image slice is displayed via the display (<NUM>). As the wearer of the HMD (<NUM>) moves toward the head (H), the distance (<NUM>) will reduce and the scale of the displayed CT image slice will increase. Similarly, movement away from the head (H) will increase the distance (<NUM>) and reduce the scale of the displayed CT image slice. In this manner, the scaling factor can be configured to provide, based on the distance (<NUM>), an appropriately sized rendering of the CT image that corresponds to the perceived size of the head (H) at that distance.

Continuing the above example, the position of the perspective point (<NUM>) relative to the viewed point (<NUM>), within a three dimensional coordinate system (<NUM>), may be used to determine an appropriate CT image slice to render, and may be used to transform an appropriate CT image slice so that it may be overlaid on the head (H). As an example, in some implementations a CT image slice of the head (H) may be selected and rendered as an overlay depending upon the perspective from which the head (H) is viewed, such that a perspective above the head (H) might show an axial view, a perspective from in front of the head (H) might show a coronal view, and a perspective from the side of the head might show a sagittal view, and views may be switched automatically as the wearer of the HMD (<NUM>) moves between perspectives.

As another example, in some implementations a CT image slice may be transformed to create a new image having an appearance of that of the two-dimensional input image as if it were fixed in place and viewed from a different perspective (e.g., a two-dimensional image perspective transformation). In this manner, a two-dimensional CT image slice displayed on the head (H) might be perspective transformed as the wearer of the HMD (<NUM>) moves between perspectives.

As yet another transformation example, in the case of a correlated dataset containing a surgical path the coordinates of the surgical path may be rendered and overlaid on the head (H) as a set of points, a line, or a dotted line. The distance (<NUM>) may be used to transform the scale of the surgical path so that, when overlaid upon the head (H), each coordinate of the surgical path is appropriately positioned relative to the head (H). The position of the perspective point (<NUM>) relative to the viewed point (<NUM>) within the coordinate system (<NUM>) may be used to transform the surgical path as the wearer of the (HMD) moves between perspectives. Such a transformation may be performed as a perspective transformation, as described above, or may be performed using other three dimensional rotational and depth transformations, as will be apparent to those skilled in the art based on the disclosure herein.

After each correlated dataset is transformed (e.g., scaled, perspective transformed, or otherwise), they may be rendered or displayed (block <NUM>) on one or more viewing devices, which may include displaying rendered markers as overlays via the display (<NUM>), displaying rendered markers as overlays on a captured image via the HHD (<NUM>) or another display, or both. For a user wearing the HMD (<NUM>), the rendered markers may appear to overlay objects within their field of view (e.g., the patient's head or other anatomy) that are viewed through the transparent display (<NUM>). For users of the HHD (<NUM>) or viewers of a wall mounted display or other device, a captured image may be displayed with the rendered markings overlaid thereon. As has been described, the steps of capturing images, determining perspective, and transforming correlated datasets may be repeated continuously as images are captured so that users may move and look around a procedure area as normal while receiving continuous updates of overlaid information.

For example, where either a movement of the viewed anatomy occurs (block <NUM>) or a movement of the HMD (<NUM>), HHD (<NUM>), or other viewing device occurs (block <NUM>), the next captured (block <NUM>) image will be received, and the orientation (block <NUM>) and distance (block <NUM>) will be redetermined. The newly determined orientation and distance will then be used for one or more transformations, and the newly produced overlays will account for any movements or changes that have occurred since the prior image.

<FIG> show an example of an interface (<NUM>) that includes a rendered and transformed surgical path, such as may be displayed via the display (<NUM>) and overlaid upon a directly viewed patient's head (H), or such as may be overlaid upon a captured image of the head (H) and displayed. In <FIG>, the head (H) is viewed from the front (e.g., a view of the coronal plane), and rendered markers have been overlaid via the display (<NUM>) or directly onto a captured image. The rendered markers include a surgical path (<NUM>) indicating the path that a surgical tool should follow, a target marker (<NUM>) indicating a destination or portion of anatomy involved in the procedure, and a tool marker (<NUM>) indicating the current location of a tracked surgical instrument. The scale of the markers has been determined based upon the determined distance (<NUM>), such that the surgical path (<NUM>) accurately overlays the patient's head (H) and appears as it would were it to be viewed directly on a coronal plane CT image slice rather than as an overlay rendered via the display (<NUM>) or another device. The positions and orientations of the markers have also been determined based upon the determined orientation (<NUM>).

<FIG> shows the head (H) viewed from the side (e.g., a view of the sagittal plane). Each marker has been rendered in a new position and orientation based upon the change in perspective. For example, in <FIG> the surgical path (<NUM>) is seen to traverse along the coronal plane of the head (H). The coronal plane is not visible in <FIG>, and the surgical path (<NUM>) can instead be seen traversing along the sagittal plane. Similarly, the position and orientation of the target marker (<NUM>) and tool marker (<NUM>) are each changed from <FIG>, and the depth at they are positioned within the head (H) can be determined. <FIG> shows the head (H) viewed from a third perspective, such as might be viewed when a viewer moves between the view of <FIG> and the view of <FIG>. As the viewer moves between the views of <FIG> and <FIG>, or to other perspectives from which the head (H) may be viewed, each of the markers may be updated and newly rendered to provide additional information and context without being limited to only those perspectives offered by a CT image set, and without needing to look away from the head (H) to the display screen (<NUM>) or another display, which may be distracting and disorienting as has been described.

As another example of an interface that may be provided by the visualization system (<NUM>), <FIG> shows an image (<NUM>) selected from a CT image set, and <FIG> show an example of an interface (<NUM>) that includes a rendered overlay of the image (<NUM>) and the surgical path (<NUM>), such as may be displayed via the display (<NUM>) and overlaid upon a directly viewed patient's head (H), or such as may be overlaid upon a captured image of the head (H) and displayed. The image (<NUM>), the surgical path (<NUM>), and the target marker (<NUM>) may be received (<NUM>) as correlated datasets prior the procedure.

In <FIG>, the head (H) is viewed from the same perspective as that of <FIG>. The image (<NUM>) is also overlaid upon the head (H) after having been transformed so that the scale of the image (<NUM>) corresponds to the perceived size of the head (H) (e.g., as opposed to being too large such that the image (<NUM>) extends beyond the head (H)). The tool marker (<NUM>), surgical path (<NUM>), and target marker (<NUM>) may also be overlaid and visible with the CT image (<NUM>), and each may be rendered at varying degrees of transparency to either completely obfuscate direct viewing of the head (H) through the renderings, or to allow some degree of transparency. While varying implementations of the visualization system (<NUM>) may provide interfaces such as shown in <FIG> or <FIG>, some implementations may provide both, and may, for example, allow certain layers to be enabled or disabled as desirable. For example, this may include enabling or disabling the overlay of the image (<NUM>) over the head (H) as may be desired.

<FIG> each show the head (H) from different perspectives, matching those shown in <FIG>. The image (<NUM>) is simulated as a shaded area, and the other rendered markings are each transformed to account for the altered perspective. As has been described, <FIG> may show a substantially unaltered CT image of the coronal plane as the image (<NUM>), while <FIG> may show a substantially unaltered CT image of the sagittal plane as the image (<NUM>). In some implementations, <FIG> may show a transformed CT image from the coronal or sagittal plane, depending upon whether the viewer's perspective is closer to the coronal plane or the sagittal plane.

It should be understood that the interfaces of <FIG> are representative of interfaces that may be displayed when correlated datasets are rendered and overlaid on objects that are visible through the transparent portions of the display (<NUM>). For example, with reference to <FIG>, the head (H) is not rendered via the display (<NUM>) but is a visible part of the augmented reality interface because it is being viewed through the transparent portion of the display (<NUM>). The interfaces of <FIG> are also representative of interfaces that may be displayed when correlated datasets are rendered and overlaid on images captured via the camera (<NUM>), such as may be displayed via the display screen (<NUM>), or may be displayed via the HHD (<NUM>). With such interfaces, the head (H) would also be rendered on the display based on the captured image.

Claim 1:
A system (<NUM>) for ENT visualization comprising:
(a) an image guided surgery ("IGS") navigation system (<NUM>) operable to:
(i) maintain a coordinate system (<NUM>) corresponding to a tracked area,
(ii) track one or more position sensors with the coordinate system, and
(iii) register the location of a patient anatomy with the coordinate system;
(b) a patient tracking assembly (<NUM>, <NUM>) configured to be positioned on the patient anatomy, wherein:
(i) the patient tracking assembly comprises a position sensor (<NUM>) of the one or more position sensors, and
(ii) the IGS navigation system is configured to update the location of the patient anatomy with the coordinate system based on movements of the position sensor;
(c) a head mounted display ("HMD") (<NUM>) comprising a wearable frame (<NUM>) and a display (<NUM>), wherein the display is positioned on the wearable frame to be intersected by a neutral optical axis of the wearer;
(d) a sensor (<NUM>) operable to produce a set of perspective data indicating the perspective of the HMD relative to the patient anatomy; and
(e) a processor (<NUM>);
wherein the processor is configured to:
(i) receive one or more correlated datasets, wherein each of the one or more correlated datasets comprise data associated with the coordinate system,
(ii) while the patient anatomy is viewed from a present perspective, determine an orientation of the neutral optical axis relative to a viewed point (<NUM>) on the patient anatomy based on the set of perspective data,
(iii) determine a distance (<NUM>) between an origin of the neutral optical axis and the viewed point based on the set of perspective data,
(iv) transform the one or more correlated datasets based on the orientation and the distance to produce an overlay image that corresponds to the patient anatomy at the present perspective, and
(v) render the overlay image via the display.