Patent Description:
During a selected procedure, a user may acquire images of a subject that are based upon image data according to the subject. Generally the image data may be acquired using various imaging techniques or systems and the image data may be reconstructed for viewing by the user on a display device, such as a flat panel or flat screen, cathode ray tube, or the like that is positioned away from a region of operation. The region of operation may be relative to a subject, such as a human patient, for performing a selected procedure. For example, a sinus procedure may be performed and images of a subject's sinuses may be displayed on a display device that does not overlay the subject.

The procedure may further include directly viewing at least a portion of a region of interest or operation, such as with an endoscope. An endoscope may position a camera at a selected location, such as within a nasal or an accessible sinus cavity. An endoscope may have limited range of movement and/or field of view at various locations within selected anatomical areas.

The invention provides a system as defined in claim <NUM> and a method as defined in claim <NUM>. The documents <CIT>, <CIT> and <CIT> which are considered to represent the closest prior art describe display devices that are adapted to track the position of instruments/ tools with respect to a patient's site during a medical/surgical intervention.

In various embodiments, an instrument may be positioned relative to a portion of a subject for performing a procedure. A subject may include a living subject or a non-living subject. In various embodiments, a living subject may include a human and the procedure being performed may be performed relative to a nasal passage and/or sinus cavity. For example, a balloon sinus dilation procedure may occur, such as one performed with a NuVent® EM Balloon Sinus Dilation System, sold by Medtronic, Inc. having a place of business in Minnesota. It is understood that the dilation of a sinus need not be performed with an electromagnetic (EM) navigated instrument, however the dilation of sinuses with an inflatable instrument may include instruments including various inflation and expansion features such as the NuVent® sinus dilation surgery.

In various embodiments, a procedure may occur in a region of operation of a subject. The region of operation may be a specific or limited area or volume on which a procedure is being performed or relative to which a procedure is being performed. The region of operation may also be referred to as a region of interest or include a region of interest therein. In various embodiments, for example, the sinuses of a human subject may be operated on, such as for performing debridement, dilation, or other appropriate procedures. The procedure may occur within a region of operation while a region of interest may include an entire head of the patient or cranium.

The operation performed on the sinus may generally be a low invasive or non-open procedure. In the low invasive procedure various natural body cavities such as nasal passages, are used to access the sinuses. Upon access to the sinuses, therefore, the operating end of an instrument may not be visible to a user.

The instrument that is being used to perform the procedure may include a tracking device configured or operable to be tracked by a tracking system. In various embodiments, the tracking system may include a visual or optical tracking system that tracks, such as by viewing or recording, the tracking device on the instrument. A navigation or tracking system, including a processor system, may then determine the position of the operating end relative to the tracking device based upon known and/or predetermined geometric relationships between the operating end and the tracking device.

The tracking system may include one or more cameras or optical tracking devices positioned relative to a display device. The display device may include a transparent or semi-transparent viewscreen. The viewscreen may be positioned relative to a user, such as allowing the user to view the subject through the viewscreen. The viewscreen may be mounted to a structure that allows the user to wear the viewscreen relatively close to a user's eyes such that the viewscreen fills all or a substantial portion of a field of view of the user. A displayed image may then be displayed on the viewscreen to allow the user to view the image while also viewing the subject. The tracked location of the instrument, or at least a portion thereof such as the operating end, may also be displayed on the display using the viewscreen. Accordingly, cameras may be associated with the device worn by the user to allow for a determination of a location of the instrument relative to the region of operation and/or region of interest in the region of operation and superimpose on an image or augmenting the user's view of the subject by displaying the tracked location of the instrument.

The system may include a viewing portion or system and a tracked portion. The viewing portion may view a real object (e.g. a subject) and displayed images. The viewing system, therefore, may be an augmented reality viewing system. In addition thereto, or alternatively thereto, viewing systems may include a view screen or monitor separate from or spaced away from the subject. In addition, images may be captured in real time with selected imaging systems, such as an endoscope. An endoscope may be positioned relative to the subject, such as within a nasal passage and/or sinus, and the images acquired with the endoscope may be displayed simultaneously with views of the subject that are acquired prior to an operative procedure. Therefore the viewing system may be used to display and view real time and pre-acquired images. Both types of images may be registered to the subject with various techniques, such as those described further herein.

With reference to <FIG>, in an operating theatre or operating room <NUM>, a user, such as a surgeon <NUM>, can perform a procedure on a subject, such as a patient <NUM> which may lay or be supported by a patient bed or support <NUM>. The patient <NUM> may define a patient longitudinal axis <NUM>. To assist in performing the procedure, the user <NUM> can use an imaging system <NUM> to acquire image data of the patient <NUM> to allow a selected system to generate or create images to assist in performing a procedure. The imaging system <NUM> may include any appropriate imaging system such as a computer tomography (CT) imager, O-Arm® imaging system sold by Medtronic, Inc. , and/or a NewTom® VGi evo cone beam imager sold by NewTom having a place of business in Verona, Italy.

A model (such as a three-dimensional (3D) image) can be generated using the image data. The generated model may be displayed as an image <NUM> on a display device <NUM>. In addition, or alternatively to the display device <NUM>, projection images (e.g. 2D x-ray projections) as captured with the imaging system <NUM> may be displayed. Furthermore, an augmented viewscreen (AV) or display device <NUM> may be provided to or used by the user <NUM>. The AV <NUM> may be worn by the user <NUM>, as discussed further herein. Further, the AV <NUM> may also be referred to as a viewing system that is an integrated system or a portion of a system for viewing various items, as discussed herein.

Either or both of the display device <NUM> or the augmented viewscreen <NUM> can be part of and/or connected to a processor system <NUM> that includes an input device <NUM> (input devices may include a keyboard, a mouse, a microphone for verbal inputs, and inputs from cameras) and a processor <NUM> which can include one or more processors or microprocessors incorporated with the processing system <NUM> along with selected types of non-transitory and/or transitory memory <NUM>. A connection <NUM> can be provided between the processor <NUM> and the display device <NUM> or the augmented viewscreen <NUM> for data communication to allow driving the display device <NUM> to display or illustrate the image <NUM>.

The imaging system <NUM>, as discussed above, can include an O-Arm® imaging system sold by Medtronic Navigation, Inc. having a place of business in Louisville, CO, USA. The imaging system <NUM>, including the O-Arm® imaging system, or other appropriate imaging systems may be in use during a selected procedure, such as the imaging system described in <CIT>; <CIT>; and <CIT>. The imaging system <NUM> may be used to acquire image data of the patient <NUM> prior to or during use of the AV <NUM>.

The imaging system <NUM>, when, for example, including the O-Arm® imaging system, may include a mobile cart <NUM> that includes a controller and/or control system <NUM>. The control system may include a processor 33a and a memory 33b (e.g. a non-transitory memory). The memory 33b may include various instructions that are executed by the processor 33a to control the imaging system, including various portions of the imaging system <NUM>.

An imaging gantry <NUM> in which a source unit <NUM> and a detector <NUM> is positioned may be connected to the mobile cart <NUM>. The gantry <NUM> may be O-shaped or toroid shaped, wherein the gantry <NUM> is substantially annular and includes walls that form a volume in which the source unit <NUM> and detector <NUM> may move. The mobile cart <NUM> can be moved from one operating theater to another and the gantry <NUM> can move relative to the cart <NUM>, as discussed further herein. This allows the imaging system <NUM> to be mobile and moveable relative to the subject <NUM> thus allowing it to be used in multiple locations and with multiple procedures without requiring a capital expenditure or space dedicated to a fixed imaging system. The control system may include the processor 33a which may be a general purpose processor or a specific application processor and the memory system 33b (e.g. a non-transitory memory such as a spinning disk or solid state nonvolatile memory). For example, the memory system may include instructions to be executed by the processor to perform functions and determine results, as discussed herein.

The source unit <NUM> may be an x-ray emitter that can emit x-rays through the patient <NUM> to be detected by the detector <NUM>. As is understood by one skilled in the art, the x-rays emitted by the source <NUM> can be emitted in a cone and detected by the detector <NUM>. The source/detector unit <NUM>/<NUM> is generally diametrically opposed within the gantry <NUM>. The detector <NUM> can move in a <NUM>° motion around the patient <NUM> within the gantry <NUM> with the source <NUM> remaining generally <NUM>° opposed (such as with a fixed inner gantry or moving system) to the detector <NUM>. Also, the gantry <NUM> can move isometrically relative to the subject <NUM>, which can be placed on the patient support or table <NUM>, generally in the direction of arrow <NUM> as illustrated in <FIG>. The gantry <NUM> can also tilt relative to the patient <NUM> illustrated by arrows <NUM>, move longitudinally along the line <NUM> relative to the longitudinal axis <NUM> of the patient <NUM> and the cart <NUM>, can move up and down generally along the line <NUM> relative to the cart <NUM> and transversely to the patient <NUM>, to allow for positioning of the source/detector <NUM>/<NUM> relative to the patient <NUM>. The imaging device <NUM> can be precisely controlled to move the source/detector <NUM>/<NUM> relative to the patient <NUM> to generate precise image data of the patient <NUM>. The imaging device <NUM> can be connected with the processor <NUM> via connection <NUM> which can include a wired or wireless connection or physical media transfer from the imaging system <NUM> to the processor <NUM>. Thus, image data collected with the imaging system <NUM> can be transferred to the processing system <NUM> for navigation, display, reconstruction, etc..

The source <NUM> may be any appropriate x-ray source, such as a multiple power x-ray source. It is understood, however, that the imaging system <NUM> may be any appropriate imaging system, such as a magnetic resonance imaging (MRI) system, C-arm x-ray imaging system; computed tomography (CT) imaging system, etc. The image data and/or images acquired with the selected imaging system, however, may be displayed on one or more of the display devices <NUM>, <NUM>.

It is further understood that the imaging system <NUM> may be operated to acquire image data and/or images prior to performing a procedure on the patient <NUM>. For example, images may be acquired and studied to diagnose and/or plan a procedure for the patient <NUM>. Thus, the user <NUM> that performs a procedure on the patient <NUM> need not use the imaging system <NUM> in the same room as the procedure being performed.

According to various embodiments, the imaging system <NUM> can be used with a tracking system and navigation system, including various portions as discussed herein, operable to track a location of the imaging device <NUM> and/or other portions. The tracking system may include a localizer and/or digitizer, including either or both of an optical localizer <NUM> and an electromagnetic localizer <NUM> can be used to generate a field and/or receive and/or send a signal within a navigation domain relative to the patient <NUM>. A navigated space or navigational domain relative to the patient <NUM> can be registered to the image <NUM>. Correlation, as understood in the art, is to allow registration of a navigation space defined within the navigational domain and an image space defined by the image <NUM>.

In various embodiments, a patient tracker or dynamic reference frame <NUM> can be connected to the patient <NUM> to allow for a dynamic tracking and maintenance of registration of the patient <NUM> to the image <NUM>. The patient tracking device or dynamic registration device <NUM> allows for images to be registered and then used for a selected procedure. In various embodiments, the localizers <NUM>, <NUM> may track the patient tracker. Further communication lines <NUM> may be provided between various features, such as the localizers <NUM>, <NUM>, the imaging system <NUM>, and an interface system <NUM> and the processor system <NUM>, which may be a navigation processor system. In various embodiments, the communication system <NUM> may be wired, wireless, or use a physical media transfer system (e.g. read/write to a memory chip).

Further, the gantry <NUM> can include a tracking device, such as an optical tracking device <NUM> or an electromagnetic tracking device <NUM>, to be tracked, such as with one or more of the optical localizer <NUM> or electromagnetic localizer <NUM>. Accordingly, the imaging device <NUM> can be tracked relative to the patient <NUM> as can the instrument <NUM> to allow for initial registration, automatic registration, or continued registration of the patient <NUM> relative to the image <NUM>. Registration and navigated procedures are discussed in the above incorporated <CIT>.

One skilled in the art will understand that the instrument <NUM> may be any appropriate instrument, such as a ventricular or vascular stent, spinal implant, neurological stent or stimulator, ablation device, dilator, or the like. The instrument <NUM> can be an interventional instrument or can include or be an implantable device. Tracking the instrument <NUM> allows for viewing a location (including x,y,z position and orientation) of the instrument <NUM> relative to the patient <NUM> with use of the registered image <NUM> without direct viewing of the instrument <NUM> within the patient <NUM>.

With continuing reference to <FIG> and additional reference to <FIG>, the patient <NUM>, in addition to and/or alternatively to the patient tracker <NUM>, may include one or more patient markers or trackers <NUM> (herein referenced to <NUM> and a lowercase letter). The patient trackers <NUM> may include various features such as being opaque or imageable with various imaging systems, such as X-rays or MRI. The trackers <NUM>, according to various embodiments, may generally be visible or captured in the image data acquired with the imaging system <NUM>, according to various embodiments. Thus, the patient markers <NUM> may be identifiable in an image or image data of the subject <NUM>.

In various embodiments, the patient markers <NUM> may be identified substantially automatically by the processor system <NUM> and/or 33a, or any other appropriate imaging processor or processor system. The markers <NUM> may include a selected and/or unique geometry that may be identified in the image or image data. Various techniques may be used to segment and identify the markers <NUM> in the selected image or image data. It is also understood, however, that the user <NUM> may identify the markers in the image data such as by selecting portions in the image and identifying the portions as the markets <NUM> with one or more of the user inputs <NUM>.

The markers <NUM> may be positioned on the patient <NUM> in any appropriate manner. For example, the markers <NUM> may be adhered to the patient <NUM> such as with a self-adhesive backing, an appropriate glue or adhesive material added to the marker <NUM> and/or the patient <NUM>, or other appropriate mechanism. Further, or in addition thereto, the markers <NUM> (e.g. the markers 100d and 100e) may be fixed to a bony structure of the patient <NUM>. The markers <NUM> may be formed or provided as screws or have threaded portions that allow them to be threaded and fixedly positioned into a bone structure, such as a cranium of the patient <NUM>.

Regardless of the connection technique, the markers <NUM> are positioned on the patient <NUM> at a selected time. Generally the markers <NUM> are positioned on the patient <NUM> prior to imaging, such as acquiring image data or images of the patient <NUM> with the imaging system <NUM>. Therefore when the image system <NUM> is operated to acquire images of the patient <NUM>, the markers <NUM> are positioned on the patient <NUM> and will appear in any acquired image data or images.

The markers <NUM> may, therefore, be used as fiducial points. It is understood that the patient <NUM> may also include various physical and anatomical fiducial points, such as a tip of the nose, corner of an eye, earlobe, or the like. Nevertheless, the fiducial points, whether provided by the markers <NUM> and/or an anatomical fiducial point to the patient <NUM>, may be used for registration of the images acquired with the imaging system <NUM>. It is understood by one skilled in the art, images acquired to the patient <NUM> may define an image space. The image space may be of a region of operation or procedure (RO) and/or an area greater than, but at least including, the RO, such as referred to as a region of interest (ROI). It is understood that a procedure may occur in a specific area in a region of interest. Nevertheless coordinates of the image data or space may be correlated or registered to the patient <NUM> in a physical or real space. The markers <NUM> may be used to register the image space to the patient and/or navigation space as defined by the patient <NUM> and/or a region of operation within or relative to the patient <NUM>.

The instrument <NUM>, as noted above, may be any appropriate instrument. The instrument <NUM> may include a dilator instrument that includes a distal end <NUM> that may be positioned within a portion of the patient <NUM>, such as through a nasal passage <NUM> and into one or more sinuses of the patient <NUM>, such as the frontal sinus. The instrument <NUM> may further include a proximal end <NUM> to which a connector <NUM> to which an inflation system (not illustrated) may be connected, such as to a connector or a nipple. The connector <NUM> may allow for material to be passed through the instrument <NUM>, such as a handle <NUM>, into a balloon or other expandable portion <NUM> at or near the distal end <NUM> of the instrument <NUM>. Inflation of the balloon <NUM> may, as is generally understood by one skilled in the art, expand to dilate or expand a portion of the sinus.

In various embodiments, the user <NUM> may grasp the instrument <NUM> such as with a hand <NUM>. The user <NUM> may then move the instrument <NUM> relative to the patient <NUM>, such as within the patient <NUM>, such as to move the instrument distal end <NUM> into a sinus, such as a maxillary sinus <NUM>. It is understood, however, that various instruments, including the instrument <NUM>, according to various configurations may also be used to access one or more portions of other sinuses of the patient <NUM> such as a frontal sinus <NUM> and/or a sphenoid sinus <NUM>.

The instrument <NUM> may further include a tracking device <NUM>. The tracking device <NUM> may be affixed to the instrument <NUM>, such as the elongated handle portion <NUM>. Generally the tracking device <NUM> is substantially fixed relative to the handle <NUM> such that movement of the handle <NUM> by the user <NUM>, such as with the users hand <NUM>, moves the handle <NUM> and the tracking device <NUM>. According to various embodiments, the tracking device <NUM> is also rigidly fixed in space relative to the distal end <NUM> of the instrument <NUM>. Accordingly, knowing the position (e.g. location and/or orientation) of the tracking device <NUM> will allow for knowing the position of the distal end <NUM>. Further, the distal end <NUM> may extend along an axis 110a that is at an angle <NUM> relative to an axis 120a of the handle <NUM>. Accordingly, the tracking device <NUM> may be positioned and have a known or predetermined position and geometry relative to the distal end <NUM> to be tracked to determine the position of the distal tip <NUM> relative to the tracking device <NUM> that is affixed to the handle <NUM>.

In addition to the instrument <NUM> that may be used by the user <NUM>, additional instruments may also be used relative to the subject <NUM>. For example, as illustrated in <FIG>, the user <NUM> may include or operate an imaging system <NUM>. The imaging system <NUM> may include a distal image capturing portion <NUM>, such as a camera lenses or camera. The imaging instrument <NUM> may generally be understood to be an endoscope, such as a EVIS EXERA III endoscope, sold by Olympus America. The endoscope <NUM> may be positioned relative to the subject <NUM> by the user <NUM>, according to various generally known techniques and embodiments. The image gathering portion <NUM> may image various portions of the subject <NUM>, such as internal portions of the subject <NUM> including the sinus cavity <NUM>, or other appropriate portions, including the nasal passage <NUM>. As discussed further herein, the endoscope <NUM> may capture images that may be in substantially real time, such as during positioning of the instrument <NUM> within the subject <NUM>. The real time images captured with the endoscope <NUM> may be displayed on various display devices or view systems, such as the display device <NUM> and/or the AV <NUM>. Therefore, the endoscope <NUM> may be used to capture images at the imaging portion or end <NUM> and display the images according to generally known techniques. The images may be transmitted through various systems, such as wirelessly or wired transmission systems, to the processing system <NUM> for display on the selected display or viewing systems, including the display device <NUM> and/or the AV <NUM>. The signal from the endoscope <NUM> may be a digital signal and/or an analogue signal and may be transmitted directly from the endoscope and/or through the interface system <NUM>. Regardless, the images acquired at the imaging portion <NUM> of the endoscope <NUM> may be viewed by the user <NUM> and/or any other appropriate individual. Further, the images may be captured and recorded for various purposes.

The endoscope <NUM>, as discussed above, may be used to acquire images of the subject <NUM>. To assist in acquiring the images or in performing a procedure, the position of the endoscope, particularly the position of the images being acquired, may be determined by one or more tracking devices. For example, an endoscope tracking device <NUM> may be incorporated onto the endoscope <NUM> similar to the tracking device <NUM> connected to the instrument <NUM>, as discussed above. The tracking device <NUM> may include one or more viewable markers or portions 147a, similar to the markers 146a on the tracking device <NUM>. As discussed herein, therefore, the tracking device <NUM> may be viewed or imaged with the AV <NUM> to be tracked by the selected tracking system as discussed further herein. The tracking device <NUM> may be used to determine the position of the end <NUM> capturing images to assist in determining a location of the image within the patient <NUM>. The endoscope <NUM> may be positioned within the subject <NUM>, such as in the sinus cavity <NUM> that is not directly viewable by the user <NUM>. Additional and/or alternative tracking devices may include an end tracking device <NUM> that may be positioned or incorporated into the endoscope <NUM> at or near the image capture end <NUM>. The tracking device <NUM> may be similar to the tracking device <NUM>, discussed above. The tracking device <NUM> may be an optical tracking device, EM tracking device, ultrasound tracking device, or other appropriate tracking device. As discussed further herein, registration of the instrument <NUM>, such as with the tracking device <NUM>, and the patient or subject tracker <NUM> may be used to assist in registering and maintaining registration of the endoscope <NUM> relative to the subject <NUM>.

The user <NUM> may also have and/or use the alternative or augmented viewscreens or viewing system <NUM> for use during the procedure. The AV <NUM> may be an appropriate device that includes at least one viewscreen and generally two viewscreens including a first viewscreen <NUM> and a second viewscreen <NUM>. The viewscreens may be fixed to a frame member <NUM> that may have one or more temple members <NUM> to allow the AV <NUM> to be worn by the user <NUM> in a similar manner to eyeglasses. Therefore, the viewscreens <NUM>, <NUM> may be positioned generally in front of, respectively, both eyes <NUM> and <NUM> of the user <NUM>. In this manner images may be displayed on one or both of the viewscreens <NUM>, <NUM> to allow the user <NUM> to view images. The AV <NUM> may include one or more various devices and systems such as the Hololens® wearable computer peripherals sold by Microsoft Corporation, R-<NUM> Smartglasses wearable computer peripherals sold by Osterhout Design Group, having a place of business in San Francisco, California, and/or DAQRI Smart Glasses® wearable computer peripherals sold by DAQRI having a place of business at Los Angeles, CA.

In various embodiments, the viewscreens <NUM>, <NUM> may also be substantially transparent except for the portion displaying an image (e.g. an icon or rendering). Therefore, the user <NUM> may view the patient <NUM> and any image displayed by the viewscreens <NUM>, <NUM>. Moreover, due to the two viewscreens <NUM>, <NUM> displaying selected images, the display may be perceived to be substantially stereoscopic and/or three-dimensional by the user <NUM> relative to the patient <NUM>. As discussed further herein, therefore, the user <NUM> may view the patient <NUM> and an image when performing a procedure.

The AV <NUM> may also include one or more cameras, such as a first camera <NUM> and a second camera <NUM>. The two cameras <NUM>, <NUM> may be used to view the region of interest, such as a head of the patient <NUM>. As illustrated in <FIG> the user <NUM> may view substantially a head portion and a neck portion of a patient when performing a procedure in a region of operation, such as in a sinus of the patient <NUM>. Therefore the cameras <NUM>, <NUM> may also view the patient <NUM>, for various purposes, as discussed further herein. Moreover, the cameras may view other objects in the region of interest such as the tracking device <NUM> on the instrument <NUM> and/or the markers <NUM>. The tracking device <NUM> may include one or more viewable markers or portions 146a that are viewable by the cameras <NUM>, <NUM> to be used to determine a perspective or view of the tracking device <NUM> by the AV <NUM>.

While the use of two cameras <NUM>, <NUM> are disclosed and discussed herein to view and determine the location of the tracking device <NUM> and/or markers <NUM>, it is understood by one skilled in the art that only one camera, such as only one of the cameras <NUM>, <NUM> may be required for tracking, as discussed herein. Based on various features (e.g. shapes, images, etc.) on the tracking device <NUM>, tracking device <NUM>, and/or the markers <NUM> a single camera, such as the camera <NUM>, may be used to determine the location (i.e. x, y, z coordinates and orientation) relative to the camera <NUM> and/or relative to other trackable items. For example, the camera <NUM> may be used to determine the relative location of the tracking device <NUM> (and therefore the instrument <NUM>) relative to the markers <NUM>. Further, the camera <NUM> may be placed at any location relative to the user <NUM>, such as on a head of the user <NUM> separate from the AV <NUM>. The camera <NUM>, however, may still remain in communication with the processor system <NUM> for display of various images on one or more of the viewscreens <NUM>, <NUM>.

The AV <NUM> including the cameras <NUM>, <NUM>, therefore, may view the markers <NUM> on the patient <NUM> in combination with the tracking device <NUM> on the instrument <NUM>. The markers <NUM> on the patient <NUM> may be viewed by the cameras <NUM>, <NUM> and may be identified by the user <NUM> and/or substantially automatically by executing instructions on a processor system, such as by executing instructions with the processor <NUM>.

As discussed above, the processor system <NUM> may have access to instructions, such as those saved on the memory <NUM>, to assist in identifying the markers <NUM> in an image. The cameras <NUM>, <NUM> may have a field of view that includes the region of interest including the head of the patient <NUM> and also viewing the markers <NUM>. The instructions, which may be included in selected software, may identify the markers <NUM> in a viewed image, such as by segmentation of the image and identifying a selected shape, density, color, or like of the markers <NUM>.

Once the markers <NUM> are identified, images acquired with the imaging system <NUM> may be registered, such as with the processor system <NUM>, to register the images including the markers <NUM> therein in the field of view of the cameras <NUM>, <NUM> of the AV <NUM>. A registration may occur by matching the identified markers <NUM> in the image data acquired by the image device <NUM> and the markers <NUM> in the field view image acquired with the cameras <NUM>, <NUM> of the AV <NUM>. The markers <NUM> are generally maintained in the same position on the patient during acquisition of image data with the imaging system <NUM> and when in the field of view of the cameras <NUM>, <NUM> during the procedure. Registration may also occur due to the tracking device <NUM> on the endoscope <NUM> and the patient tracker or dynamic reference frame <NUM>. As discussed above, registration may occur due to various registration techniques, such as those disclosed in <CIT>. Registration may be made by tracking the tracking device <NUM> (associated with the endoscope <NUM>) and/or the patient tracker <NUM>. Further, the AV <NUM> may include a tracking device <NUM> that may allow the AV <NUM> to be tracked in a same or similar tracking system and/or frame of reference relative to the subject or dynamic reference frame <NUM> and/or the tracker <NUM> on the endoscope <NUM>. Thus, the AV <NUM> may be registered relative to the endoscope <NUM> and/or the subject <NUM>. This image is acquired with the endoscope <NUM> may be used to be displayed relative or for viewing by the user <NUM> relative to a view of the user <NUM>, as discussed further herein.

As discussed above, in reference to <FIG> and <FIG>, image data and/or images acquired with the imaging system <NUM> where the markers <NUM> are connected to the patient <NUM> will include data of the markers <NUM>. The markers <NUM> may be identified in the image data or images acquired with the imaging system <NUM> as discussed above. The markers <NUM> may also be viewed by the cameras <NUM>, <NUM> associated with the AV <NUM>. The cameras <NUM>, <NUM> may also view the patient <NUM> to identify or assist in identifying anatomical fiducial markers. Nevertheless, the markers <NUM> identified in the image data acquired with the imaging system <NUM> may also be identified in images acquired with the cameras <NUM>, <NUM>. It is understood by one skilled in the art, the images acquired with the cameras <NUM>, <NUM> may be any appropriate type of images, such as color images, infrared light images, or the like. Nevertheless, matching of the markers <NUM> identified in the image data acquired with the imaging system <NUM> may be matched to locations identified of the markers viewed with the cameras <NUM>, <NUM> to register the space or field of view viewed by the AV <NUM> to the image space of images acquired with the imaging system <NUM>. As discussed further herein, therefore, images or portions of images acquired with the imaging system <NUM> may be displayed with the viewscreens <NUM>, <NUM> as appearing to be superimposed on the patient <NUM>.

The subject <NUM> may also be registered relative to currently acquired images or real time images acquired with the endoscope <NUM> due to the subject tracker <NUM> and the tracker <NUM> on the endoscope <NUM>. Registration may be made to the subject <NUM> such as with the fiducial markers 100d and/or other fiducial features. Thus, the real time images acquired with the endoscope <NUM> may also be registered to the pre-acquired images. Thus the pre-acquired images may be registered to the instrument <NUM> such as with the cameras <NUM>, <NUM> and/or to the AV <NUM> via the AV <NUM> tracker <NUM>. Thus images acquired with the endoscope <NUM> may be registered to pre-acquired images of the subject <NUM> and for viewing by the user <NUM>, such as with the AV <NUM>.

The user <NUM> may view the subject <NUM> through the AV <NUM>, as illustrated in <FIG> and <FIG>. The user <NUM> may view or have a field of view 21f, as illustrated by dash-lines in <FIG> and <FIG>. The field of view 21f may represent a view by the user <NUM> through the AV <NUM> when viewing the subject <NUM>, or any area through the AV <NUM>. Thus, the user <NUM> may view the subject <NUM>, that is real or physical, and the view may also be augmented by graphical representations (also referred to herein as icons) that are displayed by the AV <NUM>. The icons or graphical representations may be displayed in the field of view 21f for viewing by the user <NUM> when viewing the field of view through the AV <NUM>.

In addition to the icons, as discussed further herein, additional images or image areas <NUM> may be displayed within the AV field of view 21f to be viewed by the user <NUM>. The supplemental viewing areas <NUM> may be used to display various images or information for use or viewing by the user <NUM>. For example, the real time images acquired by the endoscope <NUM> may be displayed in the auxiliary or augmented viewing area <NUM>. Thus, the user <NUM> may view the subject <NUM> in the field of view 21f to view the subject <NUM>, the various graphical representations as discussed further herein, and additional images (e.g. endoscopic images) in the auxiliary or additional viewing area <NUM>. The user <NUM> may also selectively select or choose information to be displayed in the auxiliary display area <NUM> such as pre-acquired images, the real time images with the endoscope, or other appropriate information.

With additional reference to <FIG> and <FIG>, for example, sinuses such as the maxillary sinus <NUM> may be displayed as a maxillary sinus icon <NUM>'. <FIG> is an illustration of a point of view of the user <NUM> viewing through the AV <NUM> the patient <NUM> and various portions that are displayed with the viewscreens <NUM>, <NUM>. As discussed above, therefore, the maxillary sinus icon <NUM>' may be displayed for viewing by the user <NUM> as if the user <NUM> could see into the patient <NUM> and view the maxillary sinus <NUM>. The maxillary icon <NUM>' may be graphical rendering of the image data or an artificially created icon to represent the maxillary sinus.

It may also be selected to illustrate other portions of the anatomy of the patient <NUM> such as the frontal sinus <NUM> and one or more of the sphenoid sinuses <NUM>. The user <NUM> may also view any real world object, such as the patient <NUM> and/or the markers <NUM> affixed to the patient <NUM>. The user may also view other real world portions, such as the patient support <NUM>. Therefore the user <NUM> may view both features superimposed on the patient <NUM> due to the viewscreens <NUM>, <NUM> and items in the real world by viewing through transparent portions of the viewscreens <NUM>, <NUM>.

Further, the cameras <NUM>, <NUM> may view the tracking device <NUM>. By viewing the tracking device <NUM> the cameras <NUM>, <NUM> may determine the position of the tracking device <NUM> relative to the markers <NUM>. The position of the markers <NUM> are placed on the patient <NUM> to identify locations of the patient <NUM>. The known position of the instrument tracking device <NUM> relative to one or more of the markers <NUM> allow for a determination of a portion of the instrument <NUM> relative to the tracking device <NUM> and the patient <NUM>. As discussed above, the distal end <NUM> of the instrument may be at a known and fixed position relative to the tracking device <NUM>. The known and fixed relative position (e.g. the geometry) of the distal end <NUM> relative to the tracking device <NUM> may, therefore, be stored in the memory <NUM> or other appropriate memory.

The tracked location of the tracking device <NUM>, may be determined by triangulating the location of the tracking device <NUM> based on a "view" of the tracking device with one or more of the cameras <NUM>, <NUM>. The processor system <NUM> may execute instructions, as generally understood in the art, to then determine the position of the distal end <NUM> and/or the working portion such as the inflatable member of <NUM> and an instrument icon may be illustrated to include or illustrate the various portions relative to the patient <NUM> by displaying it on the viewscreen <NUM>, <NUM>. The instructions that are executed by the processor system <NUM> may include instructions stored and recalled from the memory <NUM>. The instructions may include those that are based on an algorithm to triangulate the location of the viewed portion, such as the tracking device <NUM>, based on separate views from the two cameras <NUM>, <NUM>. The separate views may be used to generate signals from the two cameras <NUM>, <NUM> (e.g. including image data) and the signals may be transmitted to the processor system <NUM>. Triangulation of the location of the tracking device <NUM> may be based on a known distance between the two cameras <NUM>, <NUM> and each separate view captured by each of the two cameras <NUM>, <NUM>.

Accordingly, the viewscreens <NUM>, <NUM> may include a graphical representation also referred to as an icon <NUM>' of the distal end and/or an icon <NUM>' of the inflatable portion of the instrument <NUM>. The icon, such as the distal icon <NUM>' may be illustrated as an icon on one or more of the viewscreens <NUM>, <NUM> such as it appears to be superimposed or displayed relative to the patient <NUM>. It is understood that an icon of more than a portion of the instrument <NUM> may be used, therefore, an instrument icon <NUM>' may be illustrated as the entire instrument including all portions of the instrument <NUM>.

The AV <NUM> may be in communication with the processor system <NUM> and/or may include onboard processing and/or other communication features to communicate with other processor systems. Accordingly, the view of the region of interest, such as the head of the patient <NUM>, by the cameras <NUM>, <NUM> of the AV <NUM> may be transmitted to the processor system <NUM>. Due at least to the spacing apart of the cameras <NUM>, <NUM>, a triangulation may be determined for each viewed point in space, such as the markers <NUM> and/or the tracking device <NUM>, relative to the cameras <NUM>, <NUM>. A relative location of the tracking device <NUM> to one or more of the markers <NUM> may be determined such as by executing instructions with the processor system <NUM>.

The processor system <NUM> receiving images from one or both of the two cameras <NUM>, <NUM> may process and determine the distance between the various tracked, or any viewed, portions such as the markers <NUM> and the tracking device <NUM>. The processor system <NUM>, therefore, executing instructions accessed in the memory <NUM> may then provide to the viewscreens <NUM>, <NUM> the selected and/or appropriate image portions such as the instrument icon <NUM>' or portions thereof and/or other imaged features, such as icons representing the sinuses including the frontal sinus <NUM>' or other appropriate portion from the image data. The registration of the pre-acquired images, such as those acquired with the imaging system <NUM>, based upon the tracked location of the tracking device <NUM> and/or the markers <NUM> may be based upon known registration techniques such as those disclosed in the <CIT>, incorporated herein by reference. The registration may be substantially automatic and/or based upon identification of fiducial markers, such as the markers <NUM>, in the images <NUM> and/or the markers <NUM> on the patient <NUM>.

The user <NUM>, therefore, may view both the patient <NUM> and other features, such as the instrument icon <NUM>', relative to the patient <NUM> based upon the viewscreens <NUM>, <NUM>. The cameras <NUM>, <NUM> may provide all of the tracking information relative to the user <NUM> and the patient <NUM> for determining a location of various portions of the instrument <NUM>, such as the distal tip <NUM> for displaying them with the viewscreens <NUM>, <NUM>. The perception of the user <NUM> may be that the instrument <NUM> is viewable relative to the patient <NUM> even though it is within the patient <NUM>. Further, the image data acquired with the imaging system <NUM> may be displayed as features, such as icons, with the viewscreens <NUM>, <NUM> relative to the patient <NUM>. Again, the perception by the user <NUM> of the patient <NUM> may be that the various portions, such as the sinuses <NUM>, <NUM>, <NUM>, are viewable by the user <NUM> due to the AV <NUM>. Accordingly, as illustrated in <FIG>, the view of the patient <NUM> may be augmented to illustrate features that are otherwise unviewable by the user <NUM> with the users <NUM> regular vision. In other words, the user <NUM> may view the patient <NUM>, as illustrated in <FIG> in physical space, and a representation of an area within the patient <NUM>, such as with the icons or renderings discussed above. This view may also be 3D and change in perspective as the user moves relative to the patient <NUM> and/or the instrument <NUM>.

The patient <NUM> may be viewed through the view screens <NUM>, <NUM> as specifically illustrated in <FIG>. The various icons, such as the maxillary icon <NUM>' and the sphenoid icon <NUM>' may be displayed relative to the icon <NUM>' of the instrument <NUM>. The icons may have various and selected opacities and/or cutaways for viewing of the instrument icon <NUM>' relative to the anatomy icons, such as the sphenoid icon <NUM>'. Accordingly, the user <NUM> viewing the field of view including the icons, such as the sphenoid icon <NUM>' and the instrument icon <NUM>' may see both the icons simultaneously. Moreover the user <NUM> may perceive a position of the instrument <NUM> within the selected sinus, such as the sphenoid sinus <NUM>, by viewing the instrument icon <NUM>' and the sinus icon, such as the sphenoid sinus icon <NUM>', substantially simultaneously. In other words, the opacity of various icons, such as the sinus icons, may be selected to have a transparent view to be able to view the instrument icon within or as if it is within the selected anatomical portion. This allows the user <NUM> to view the patient <NUM> and the icons of the instrument and the anatomy substantially simultaneously and as if present on the patient <NUM>, as illustrated in <FIG>.

In addition to, or alternatively thereto, the various icons may be displayed at a position away from the patient <NUM>. For example, as illustrated in <FIG>, the instrument icon <NUM>' may be displayed away from the patient <NUM> although at a tracked and determined location relative to an anatomical portion icon, such as the sphenoid icon <NUM>'. It may be selected to illustrate only those anatomical portions that are interacting or having been passed through by the instrument <NUM> therefore all icons may not be necessarily to be shown. It is understood that various pathways, such as an icon pathway <NUM>' (See <FIG> and <FIG>) between various potions of the anatomy, such as through the nasal passage <NUM> even when the instrument <NUM> is within the nasal passage <NUM> and obscured from a non-augmented view of the user <NUM>. Therefore, as illustrated in <FIG>, it is understood that the displayed portions of the anatomy that are represented or based upon the image data acquired of the patient <NUM> may be displayed at a location away from the respective and relative physical location on the patient <NUM>. Accordingly the icons, such as the sinus icons may be displayed at a distance away from the patient <NUM>. This may allow the user <NUM> to have a more and/or substantially unobstructed view of the patient <NUM> while also being able to view the relative location of the instrument <NUM> relative to selected anatomical portions.

Moreover the view screens <NUM>, <NUM> may be used to display other images such as an endoscopic image that may be acquired substantially simultaneously and in real time, if selected. That is, the user <NUM> may place an endoscope in the nasal passage <NUM> as well and one or more of the viewscreens <NUM>, <NUM> may display the endoscope view. Accordingly, it is understood that the user <NUM> may position an endoscope through the nasal passage <NUM> with the instrument <NUM> to provide a real time and endoscopic point of view which also may be displayed on the view screens <NUM>, <NUM> and relative to selected icons, such as the sinus icons and/or the instrument icon <NUM>'.

Moreover, it is understood that various images may be displayed on both of the view screens <NUM> and <NUM> or only one of the view screens <NUM>, <NUM>. It will be understood that images displayed on the two view screens <NUM>, <NUM> may be substantially similar, but altered to allow for a perception of depth and/or three-dimensionality of the selected portions, such as of the sinuses and/or the instrument <NUM> either based upon the image data and/or icons, by the user <NUM>. Accordingly, the displays <NUM>, <NUM> may have identical displays, substantially different displays or only one display per view screen, or be similar to provide a perception of depth for viewing by the user <NUM>.

As discussed above, the auxiliary image <NUM> may show or illustrate the position of the instrument <NUM>, such as a distal end image <NUM>" illustrated in the auxiliary image <NUM>. The auxiliary image <NUM> may be the real time image acquired with the endoscope <NUM>, as discussed above. The distal end image <NUM>", therefore, may also be a real time image of the instrument <NUM>. The auxiliary image <NUM> may also display a surface of the anatomy, such as within the sinuses, for viewing by the user <NUM> during a selected procedure. Therefore the field of view 21f may allow the user <NUM> to view the subject <NUM>, graphical representations of instruments displayed relative to and/or superimposed on the subject <NUM>, pre-acquired images of the subject displayed relative thereto and/or superimposed on the subject <NUM>, and/or auxiliary images such as real time images of the instrument <NUM>. Thus the user <NUM> may select which images to view in the field of view 21f. It is understood that any of the images or graphical representations may also be displayed on various other display devices, such as the display device <NUM>. The display device <NUM> may also view or display both the graphical representations of the locations of the instrument <NUM>, pre-acquired images, and real time images, either alone or in combination.

With reference to <FIG>, the field of view may also be displayed with the display device <NUM>. In various embodiments, the view of the endoscope may be displayed as an auxiliary view <NUM>' on the display <NUM>. The icons <NUM>' and the portions of the anatomy, such as the sinus <NUM>', may also be display with the display device <NUM>. The graphical representations may be substantially three-dimensional (3D) when displayed on the display device <NUM>. Thus, the field of view display 21f may be substantially reproduced on the display device <NUM>, though the patient <NUM> may not be displayed, but only the acquired images, such as the sinus <NUM>', and the images that are acquired in real time, such as with the endoscope <NUM>. The display device <NUM>, it is understood, may be mobile and positioned for a best view of the user <NUM>.

Whether displayed on the display <NUM> and/or in the field of view 21f with the AV <NUM>, the display of the graphical representations (e.g. the sinus <NUM>' and the instrument <NUM>') may be from the point of view of the user <NUM>. Thus, as the user <NUM> moves relative to the subject <NUM>, the display in the field of view 21f and/or on the display device <NUM> may alter to provide a display for the user <NUM> as if the user <NUM> were looking within the subject <NUM> at the selected position. In other words, is the user <NUM> moved to a position at a head of the subject <NUM> looking inferiorly, rather than superiorly, the display of the graphical representations would be altered to match the position of the user <NUM> relative to the subject <NUM>. The determined position of the user <NUM> may be determined, in various embodiments, by the tracker <NUM> and/or the views of the imaging device <NUM>,<NUM> associated with the AV <NUM>.

With continuing reference to <FIG> and additional reference to <FIG> a method of using the AV <NUM> by the user <NUM> to assist in performing a procedure, such as a procedure relative to one or more of the sinuses including the frontal sinus <NUM>, is described in the flowchart <NUM>. Generally the process may start in the start block <NUM>. After initiating the process in start block <NUM>, acquisition of image data or images of the subject <NUM> may be performed. Acquisition of the image data or images may be performed in any appropriate manner. For example, images of the subject <NUM> may be acquired and stored on a memory system, such as the memory system <NUM> and/or the memory system 33b, at a selected time prior to performing a procedure, such as when the patient <NUM> is prepared for introduction of the instrument <NUM> into the patient <NUM>. The images may be acquired with a selected imaging system such a CT scanner and/or an MRI and saved in an appropriate format, such as raw data and/or reconstructed images. The reconstructed images may include images that have been rendered in a three-dimensional manner for viewing by the user <NUM> with varying display devices, such as the display device <NUM> and/or the AV <NUM>. Further, various portions of the image or image data may be segmented, such as segmenting the sinuses, including the frontal sinus <NUM> from the other image data. Moreover, the identification of the markers <NUM> in the image or image data may be performed such as by a processor, including the processor <NUM> and/or 33a. Segmenting the various portions of the anatomy, such as the frontal sinus <NUM>, and/or identifying the markers <NUM> may be performed using various segmentation techniques. Segmentation techniques may include those incorporated in various imaging and navigation systems such as the FUSION ™ navigation system sold by Medtronic, Inc.

Image data may also be acquired substantially during or immediately prior to a procedure such as with the imaging device <NUM> that may be used substantially intraoperatively (e.g. when the patient is prepared for the procedure). The various portions of the image or image data may be segmented, as discussed above but rather than being stored on the memory prior to the procedure for a selected period of time, the data may be transferred substantially in real time to the processor system <NUM> for use during the procedure. Nevertheless, it may be understood that the image data may be stored for a selected period of time, such as to analyze and/or process the image data or images for use during the procedure.

The acquisition of the image data may be optional, as preparing images for display by the AV <NUM> and/or use during a procedure is not required. For example, as discussed herein, the system including the AV <NUM> and the processor system <NUM> may track the markers <NUM> and the tracking device <NUM> to represent the positon of the instrument without image data of the patient <NUM>. In various embodiments, the images may be accessed by the processor system <NUM> for display with the AV <NUM>, as discussed above.

The AV <NUM> may be placed for the procedure in block <NUM>. Placement of the AV <NUM> for the procedure may include placing the AV <NUM> on the user <NUM>, as discussed further herein. Moreover, the AV <NUM> may be placed in communication with the processor system <NUM> such as for providing processing ability to track the patient <NUM>, such as with the markers <NUM>, and/or the instrument <NUM>, such as with the tracking device <NUM>. The AV <NUM> may therefore view the region of interest in block <NUM> and the user may confirm being able to view the region of interest in block <NUM>. In viewing the region of interest, the cameras <NUM>, <NUM> may be able to view at least the portion of the patient <NUM> on which a procedure is to occur, such as generally the head region. As discussed above, the region of operation may be substantially unviewable by the user <NUM> through various external tissues of the patient <NUM>. Therefore, the region of the operation may include the sinuses, such as the frontal sinus <NUM>, and the region of interest may include the entire head of the patient <NUM>. Accordingly, the cameras <NUM>, <NUM> may be positioned to view the region of interest and the user <NUM> may confirm viewing the region of interest through the viewscreens <NUM>, <NUM>. The viewscreens <NUM>, <NUM> are substantially transparent when no icons are displayed on a portion of the viewscreens <NUM>, <NUM>.

Once the AV <NUM> has the view of the region of interest, a recording of the region of interest with the AV <NUM> cameras may be performed in block <NUM>. Recording of the region of interest in block <NUM> may allow for collection of images with the cameras <NUM>, <NUM> (although it is understood that more than two or less than two cameras may be used). The recording of the region of interest may include imaging at least a portion of the patient <NUM> in an ongoing manner, such as during the entire procedure. Imaging the region of interest of the patient <NUM> may include imaging the markers <NUM> and/or other fiducial points or portions of the patient <NUM>. Accordingly, the recorded region of interest may include identifying patient markers in block <NUM>.

Identifying of patient markers may include segmenting image data recorded at the region of interest in block <NUM> to identify the patient markers <NUM> in the image. The identified patient markers <NUM> may be displayed as an icon with the viewscreens <NUM>, <NUM> such as with an icon 100a' which may include a three-dimensional cube positioned over the marker 100a on the patient <NUM> when viewed by the user <NUM>, as illustrated in <FIG>. Nevertheless, identifying the patient markers in block <NUM> may not require or provide for the display of the icon 100a' but may simply be performed to identify the marker to identify the region of the patient <NUM> by the processor system <NUM> such as for identification of a location of the instrument <NUM>, or a portion thereof, such as the distal end <NUM>, as discussed further herein.

Identifying the markers <NUM> on the patient <NUM> allows the processor system, such a portion of the navigation system, to track the patient <NUM> when the markers <NUM> are within the field of view of the cameras of the AV <NUM>. The markers <NUM> may include portions that are identifiable in the image acquired with the cameras <NUM>, <NUM> such as a color, pattern, shape, etc. Further, the markers <NUM> may include features that are identifiable for determining a position, including a pose, location and orientation of the marker relative to the AV <NUM>. Therefore, the patient <NUM> may be tracked relative to the AV <NUM> worn by the user <NUM>.

In block <NUM> the instrument tracker <NUM> may be identified. The instrument tracker <NUM> may include portions that are identifiable in the image acquired with the cameras <NUM>, <NUM> such as a color, pattern, shape, etc. Further, the instrument tracker <NUM> may include features that are identifiable for determining a position, including a pose, location and orientation of the marker relative to the AV <NUM>. For example, the tracking device <NUM> may include a pattern that is viewable by the cameras <NUM>, <NUM>. The pattern on the tracking device <NUM> may be substantially or entirely unique from different perspectives relative to the tracking device. Thus, the viewed pattern on the tracking device <NUM> may be used to determine the positon of the instrument tracker <NUM> and, therefore, the instrument <NUM>.

The instrument tracker <NUM> may be fixed to the instrument <NUM>, as discussed above. A geometric relationship between various portions of the instrument <NUM>, such as the distal tip <NUM> and/or an operating portion <NUM>, may be predetermined and entered for processing by the processor system <NUM>. In various embodiments, the geometry may be saved in the memory <NUM> and recalled due to automatic identification of the instrument <NUM> (e.g. by viewing the instrument with the cameras <NUM>, <NUM>) and/or entering the identification of the instrument by the user <NUM>. Nevertheless the AV <NUM> may be used to view the tracking device <NUM> to determine a position including a location (e.g. a three-dimensional coordinates) and an orientation in various degrees of freedom (e.g. three-degrees of freedom). The tracked position of the instrument <NUM> may be used by the processing system <NUM> for various purposes.

For example, as illustrated in <FIG>, performed in block <NUM> the instrument icon <NUM>' may be displayed with the AV <NUM>, such as being displayed on one or more of the viewscreens <NUM>, <NUM>. The viewscreens <NUM>, <NUM> may be substantially transparent save for the portions illustrating the icons. The icon <NUM>' and/or portions of the instrument such as the distal tip icon <NUM>' and/or the operating portion <NUM>' may be illustrated on the viewscreens <NUM>, <NUM> relative to the patient <NUM>. The user <NUM> may then view the patient <NUM> and the icon or icons through the viewscreens <NUM>, <NUM>. Accordingly the user <NUM> may view a position of at least a portion of the instrument <NUM> relative to the patient <NUM>, including a portion of the patient <NUM>.

A display of a subject portion icon may selectively or alternatively be displayed in block <NUM>. For example, as illustrated in <FIG> and/or <FIG>, the frontal sinus <NUM>' icon may be displayed. The frontal sinus icon <NUM>' may be displayed relative to the instrument icon <NUM>' and the patient <NUM>. Therefore, the user <NUM> may view the patient <NUM>, the instrument icon <NUM>', and the frontal sinus icon <NUM>'. Due to the tracking of the markers <NUM> on the patient <NUM> the relative position of the instrument <NUM> may be displayed on the viewscreens <NUM>, <NUM> with the instrument icon <NUM>'. Further the relative position of the subject portion, such as the frontal sinus <NUM>, may be displayed due to registration of the pre-acquired image to the patient using the markers <NUM>, as discussed above.

Again, as the AV <NUM> is able to track the patient <NUM> due to the markers <NUM> the relative positions of the instrument <NUM> and the subject portions, such as the frontal sinus <NUM>, may be updated in substantially real time and displayed on the viewscreens <NUM>, <NUM> for viewing by the user <NUM> along with the subject <NUM>. It is understood that the icons, such as the instrument icon <NUM>' and the subject portion icon <NUM>' may be generated and displayed on the viewscreens <NUM>, <NUM> while the user is able to view the patient <NUM> through the viewscreens <NUM>, <NUM> in real time and in physical space. It is further understood that the icons may be displayed on only or both of the viewscreens <NUM>, <NUM>, as selected by the user <NUM>.

As discussed above, the user <NUM> may also select to have displayed real time images, optionally, in block <NUM>. The real time images may be images acquired with the endoscope <NUM>, as discussed above and as generally understood by one skilled in the art. The real time images may include surfaces, such as internal surfaces, of the subject <NUM>. Further, or in addition thereto, the images may include displays or images of the instrument <NUM>, such as the distal end <NUM>" display of the instrument <NUM>. The user <NUM> may select to have the auxiliary image <NUM> displayed in the field of view 21f or on any appropriate display, such as the display device <NUM>. The user <NUM> may also select to have the auxiliary display <NUM> turned off or not displayed such that the user <NUM> only use the subject <NUM> and selected augmented reality portions, such as the graphical representation or icons as discussed above. It is further understood, that the user <NUM> may select to have graphical representations displayed in the auxiliary display area <NUM> and the real time images displayed superimposed or displayed relative to the subject <NUM> in the field of view 21f. As discussed above, the images acquired with the endoscope <NUM> may be registered relative to the subject <NUM> due to the selected fiducial portions and/or markers on the subject and the patient or subject tracker <NUM>. Thus, the user <NUM> may view the icons relative to the instrument <NUM>, icons relative to selected sinuses or internal portions in part or because of the pre-acquired images (e.g. MRI and/or CT image), and real time images acquired with the endoscope <NUM> or other appropriate imaging system.

In block <NUM> a procedure may be performed by viewing the subject <NUM> and selected icons, such as the instrument icon <NUM>' and/or the frontal sinus icon <NUM>' due to the AV <NUM>. It is further understood that other appropriate icons may be displayed such as the maxillary sinus icon <NUM>' and/or the sphenoid sinus icon <NUM>'. Moreover, additional instrument icons may also be displayed due to various tracking devices associated with instruments. Further different instruments maybe have different geometries that may also be entered and/or recalled prior to displaying an icon on the display device, including the AV <NUM>. The method <NUM> may then end in block <NUM> including various other procedures, such as various staunching and/or closing procedures.

Claim 1:
A system for viewing a subject when performing an operation on the subject, comprising:
a wearable viewscreen and camera system configured to be held relative to a first eye of a user, having:
a mounting structure;
a first viewscreen (<NUM>) configured to be positioned near the first eye of the user and fixed to the mounting structure; and
a first camera (<NUM>) configured to have a first field of view;
an instrument tracking device (<NUM>) fixed to an instrument viewable by the first camera;
a subject marker (146a) viewable by the first camera; and
a processor system (<NUM>) configured to execute instructions (i) to determine a relative position of the instrument tracking device relative to the subject marker based on a first signal from the first camera regarding a view of the instrument tracking device relative to the subject marker and (ii) generate an instrument icon to be displayed in the first viewscreen to be viewed by the first eye; and
further comprising:
a memory (<NUM>) having stored thereon an image of at least a portion of a region of operation of the subject;
wherein the image includes image portions that represent the subject marker;
wherein the processor system executes further instructions to (i) register the image of at least the portion of the region of the operation and (ii) display an anatomy representation with the first viewscreen;
wherein the anatomy representation is configured to appear in the user's field of vision relative to the subject.