Patent Description:
When performing an image-guided procedure, a monitor displays for the user a model of a body part and a model of a medical tool, both registered in a reference coordinate system. The tool is being tracked relative to a reference unit rigidly attached to the body or fixed relative thereto. If the reference unit accidently moves, the model of the tool is displayed at an erroneous position and orientation relative to the model of the body part. When performing visor guided procedures, a model of the relevant body part is displayed to the user on the visor at a perspective corresponding to the position and orientation of the user relative to the relevant body part, such that the model appears to the user to be at the position and orientation of the body part. Specifically, in minimally invasive visor guided procedures, the user sees a representation of an internal body part (i.e., a model of the internal body part) where other body parts are perceived by the user to be transparent, and the user may perform the procedure without exposing the internal body part. To that end, the model is registered with the relevant body part (i.e., the position and orientation associated with the model corresponds to the position and orientation of the internal body part in a reference coordinate system). However, if the model and the internal body part fall out of registration, the user may be presented with erroneous information regarding to the position and orientation of the internal body part. One prior art technique for verifying registration during image-guided procedures employs intra-operative imaging (e.g. X-ray images or live fluoroscopy using a C-arm) presented on a monitor. The surgeon verifies that the position of the tool, as captured in the intra-operative images and presented in real-time on a monitor, corresponds to the position of the tool relative to the internal body part as it appears in the monitor of the image-guiding system, preferably from more than one perspective. Another prior art technique includes placing the tracked tool at selected locations on the body part that are visible by the surgeon and that also appear in the model of the body part, and confirming that the model of the tool is displayed on the monitor of the image-guiding system at the corresponding position on the model of the body part. Both techniques may interfere with the surgical workflow, distract the surgeon and lengthen the surgery, which may lead, among others, to an increased probability of infection.

<CIT>, entitled "System and Method for Dynamic Validation, Correction or Registration for Surgical Navigation" directs in one embodiment therein, to an overlay of a visualization of a tracked instrument and patient imaging information, on a video image provided during a procedure. According to the method directed to by Sela et al, any registration errors may be recognized by a user as a simple misalignment of the instrument visualization and the actual physical instrument seen on the video image.

<CIT>, entitled "System and Method for Dynamic Validation, Correction or Registration for Surgical Navigation" discloses a system and method for dynamic validation, correction of registration for surgical navigation during medical procedures on a patient which involves confirmation of registration between one or more previously registered virtual objects, such as surgical tools etc. in a common coordinate frame of a surgical navigation system and an operating room, and intra-operatively acquired imaging during the medical procedure in the common coordinate frame. The method includes displaying intraoperatively acquired imaging of the surgical field containing the one or more real objects corresponding to the one or more previously registered virtual objects, with the real objects being tracked by a tracking system. The method overlaying a virtual image containing the previously registered virtual objects onto the intraoperatively acquired imaging, from the point of view of the intra-operatively acquired imaging, and detecting for any misalignment between any one of the one or more previously registered virtual objects contained in the virtual image and its corresponding real object contained in the intra-operatively acquired imaging.

<CIT>, entitled "Method and System for Providing Accuracy Evaluation of Image Guided Surgery" discloses methods and systems for the accuracy evaluation of an Image Guided Surgery system. One embodiment includes: identifying a position of a landmark in a three-dimensional image of an object; and overlaying a first marker on a reality view of the object according to registration data that correlates the three-dimensional image of the object with the object, to represent the position of the landmark as being identified in the three-dimensional image. In one embodiment, the reality view of the object includes a real time image of the object; a position of the landmark is determined on the object via a position determination system; and a second marker is further overlaid on the real time image of the object, to represent the position of the landmark as being determined via the position determination system.

<CIT>, entitled "Methods and Apparati for Surgical Navigation and Visualization with Microscope ("Micro Dex-Ray")" discloses a system and method for macroscopic and microscopic surgical navigation and visualization. In exemplary embodiments of the invention an integrated system can include a computer which has stored three dimensional representations of a patient's internal anatomy, a display, a probe and an operation microscope. In exemplary embodiments of the invention reference markers can be attached to the probe and the microscope, and the system can also include a tracking system which can track the 3D position and orientation of each of the probe and microscope. In exemplary embodiments of the invention a system can include means for detecting changes in the imaging parameters of the microscope, such as, for example, magnification and focus, which occur as a result of user adjustment and operation of the microscope. The microscope can have, for example, a focal point position relative to the markers attached to the microscope and can, for example, be calibrated in the full range of microscope focus. In exemplary embodiments of the invention, the position of the microscope can be obtained from the tracking data regarding the microscope and the focus can be obtained from, for example, a sensor integrated with the microscope. Additionally, a tip position of the probe can also be obtained from the tracking data of the reference markers on the probe, and means can be provided for registration of virtual representations of patient anatomical data with real images from one or more cameras on each of the probe and the microscope. In exemplary embodiments of the invention visualization and navigation can be provided by each of the microscope and the probe, and when both are active the system can intelligently display a microscopic or a macroscopic (probe based) augmented image according to defined rules.

The present invention is defined in the independent claims <NUM> and <NUM>.

It is an object of the disclosed technique to provide a novel methods and system for a method and system for verifying registration of a model of an internal body part with the internal body part in a reference coordinate system.

In accordance with the disclosed technique, there is thus provided a method for verifying registration of a model of an internal body part with the internal body part in a reference coordinate system. The internal body part is at least partially unseen directly by a user. The method includes the procedures of continuously determining a position and orientation of a head mounted display in the reference coordinate system, determining a display location of at least one virtual marker, and displaying on said head mounted display the at least one virtual marker according to the display location. The display location of the at least one virtual marker is determined according to an expected position of a respective at least one reference point relative to the head mounted display. The at least one reference point is directly visible to the user. The relative position between the at least one reference point and the internal body part is substantially constant. The position of the at least one reference point in the reference coordinate system is predetermined. When the model and the internal body part are effectively registered, the at least one reference point and corresponding at least one virtual marker appears visually in alignment.

According to another aspect of the disclosed technique, there is thus provided a system for verifying registration of a model of a body part with the internal body part in a reference coordinate system. The body part is at least partially unseen directly by a user. The system includes a head mounted display to be donned by the user, a head mounted display tracking module and a processor coupled with the head mounted display tracking module. The head mounted display tracking module is configured to continuously determine information relating to the position and orientation of the head mounted display in the reference coordinate system. The processor is configured to determine the position and orientation of the head mounted display in the reference coordinate system. The processor is further configured to determine the display location of at least one virtual marker according to the position of a respective at least one reference point relative to the head mounted display. The reference point is visible to the user. The relative position between the at least one reference point and the body part is substantially constant. The position of the at least one reference point in the reference coordinate system is predetermined. The processor is configured to render the at least one virtual marker to be displayed on the head mounted display. When the model and the body part are effectively registered, the at least one reference point and the corresponding virtual marker are displayed on the head mounted display visually in alignment.

The disclosed technique overcomes the disadvantages of the prior art by providing a method and a system for verifying registration of a model of an internal body part with the internal body part, in a reference coordinate system. The internal body part is unseen directly by a user. The term 'unseen directly' by a user or a camera or an optical detector, refers to herein that an unobstructed Line Of Sight (LOS) does not exist between an object and the user or the camera or the optical detector. Conversely, the terms 'directly seen' or 'directly visible' by a user or a camera or an optical detector refers to herein that an unobstructed LOS exists between the object and the user or the camera or the optical detector. According to the disclosed technique, at least one reference point is determined, the point being directly visible to said user. The relative position between the reference point and the internal body part is substantially constant. The position of the at least one reference point in said reference coordinate system is predetermined. Furthermore, the position and orientation, abbreviated herein 'P&O', of a Head Mounted Display (HMD) relative to a reference tracking unit which is fixed relative to the internal body part, is continuously determined. Thus, the relative positions and orientations, abbreviated herein 'P&Os', between the HMD and the reference points are also continuously determined. At least one virtual marker corresponding to the at least one reference point is displayed on the HMD. As long as the relative position between the reference tracking unit and the internal body part does not change (i.e., the model and the internal body part are registered), the at least one reference point and the corresponding displayed virtual marker appear to the user to be in visual alignment. When the model and the internal body part are effectively not registered, the at least one reference point and the corresponding displayed virtual marker appear to the user to be out of visual alignment. Thus, the user is provided with a visual indication that the model and the internal body part are not registered. Accordingly, the user may visually verify throughout the procedure (i.e., without interrupting the procedure workflow) that the model and the internal body part are indeed registered. It is noted that the terms 'effectively fall out of registration', 'effective miss-registration', 'effectively miss-registered', 'effectively not-registered' and 'effectively out of registration' are used herein interchangeably and are further defined herein below in conjunction with <FIG>. It is further noted that the disclosed technique is applicable to any kind of procedure in which the internal body part is unseen or partially unseen directly by the user, such that the user cannot verify that the displayed model is perceived as aligned correctly with the body part. As such, the technique is also applicable to open procedures. It is further noted that the term 'visor' used herein relates in general to optically see-through displays such as smart glasses, augmented reality glasses, near-to-eye displays, head mounted displays and head wearable displays. Such see-through displays may be based on technologies such as visor projection, combiners, waveguide technologies (e.g., microstructure extraction, diffractive optics or holograms, micro-mirrors beam-splitter, polarization reflection), retinal scanning, on-pupil optics or contact lenses and the like.

Reference is now made to <FIG>, <FIG>, <FIG>, and <FIG>, which are schematic illustrations of registration verification of a model <NUM> of an internal body part <NUM> with internal body part <NUM>, in a reference coordinate system <NUM> operative in accordance with an embodiment of the disclosed technique. <FIG> illustrate the physical objects (i.e., the "real world") observed by a user. These physical objects are patient <NUM> lying on treatment bed <NUM> as well reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and reference tracking unit <NUM>. <FIG> illustrates a visor <NUM> (a physical object) and the virtual objects displayed on visor <NUM>. These virtual objects are virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and a virtual perspective of model <NUM> of the internal body part. <FIG> and <FIG> illustrate the view point of a user looking at a patient through a see-through HMD display, such as visor <NUM>, which displays a virtual perspective of model <NUM> and virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. Internal body part <NUM> is unseen directly by the user (i.e., as indicated by the hatched line in <FIG>). Initially, a reference tracking unit <NUM> is fixedly placed relative to internal body part <NUM> such that the relative P&O between reference tracking unit <NUM> and internal body part <NUM> is substantially constant. Reference tracking unit <NUM> defines reference coordinate system <NUM>. Thereafter, the P&O of reference tracking unit <NUM> is determined relative to internal body part <NUM>, as further elaborated below. In other words, the P&O of internal body part <NUM>, in reference coordinate system <NUM> is determined. Furthermore, the P&O of model <NUM> in reference coordinate system <NUM> is determined also as further explained below. Thus, model <NUM> of internal body part <NUM> is registered with the respective internal body part <NUM> thereof.

During a medical procedure, a patient <NUM> lies on a treatment bed <NUM>, and the P&O of the HMD relative to reference tracking unit <NUM> is continuously determined. Consequently, the tracking system determines the P&O of the HMD relative to internal body part <NUM>. Thus, model <NUM> may be rendered in the correct perspective and displayed, for example, on visor <NUM> of the HMD.

However, during the medical procedure, model <NUM> may effectively fall out of registration with respective internal body part <NUM>, for example, when reference tracking unit <NUM> is accidently moved. Thus, the relative P&O between reference tracking unit <NUM> and internal body part <NUM> has changed. However, the registered position of model <NUM> relative to reference tracking unit <NUM> does not change with the motion thereof. As such model <NUM> is no longer registered with respective internal body part <NUM>. Consequently, the user may be displayed with erroneous information with respect to the P&O of internal body part <NUM> without any indication thereof.

According to the disclosed technique, at least one reference point directly visible to a user is determined within the field of view of the user. In the example brought forth in <FIG>, three reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, directly visible to the user, are determined within the field of view of the user. The relative position between reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and internal body part <NUM> is substantially constant. Reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may be markings marked by the user, for example, with a marker (e.g., on the skin of patient <NUM>). Reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may also be markers (e.g., adhesive markers) attached to patient <NUM> or to treatment bed <NUM>. As a further example, these reference points may be prominent features (e.g., screw heads) of a fixture with a substantially fixed spatial relation (i.e., a fixed P&O) relative to body part <NUM> (e.g. a Mayfield clamp holding the head during head surgery). Reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may further be prominent anatomical features which are directly visible to the user such as the outside corner of the eye, a tooth or the surface of a thigh. In general, the term 'reference point' relates herein to a point, surface or object that is directly visible to the user, where the position of the reference point, relative to internal body part <NUM>, is substantially constant and the position thereof, in reference coordinate system <NUM>, can be determined either manually or automatically as further elaborated below in conjunction with <FIG>.

The position of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, in reference coordinate system <NUM> (i.e. relative to reference tracking unit <NUM>) is determined (e.g., by placing the tip of a tracked wand on reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>) prior to the onset of the procedure (i.e., the position of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> is predetermined). As mentioned above, the relative positions between reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and internal body part <NUM> are substantially constant. In general, when the reference points are located on the skin of the patient, the relative position may momentarily change, for example, when the reference tracking unit is attached to a vertebra and the surgeon applies force to puncture the vertebral bone with an awl. However, once the surgeon stops applying force, the original relative positions are restored.

Since the P&O of the HMD relative to reference tracking unit <NUM> is continuously determined during the procedure, the expected positions of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> relative to the HMD are also continuously determined (i.e. the P&O of the HMD relative to the positions of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in coordinate system <NUM>, as determined at the beginning of the procedure). During the procedure, an image of model <NUM> rendered from the determined perspective of the HMD and three virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> corresponding to reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, are displayed to the user donning the HMD. The system continuously updates the rendered image of model <NUM> and determines the display location of the rendered image and the display location of each one of virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> (i.e., the display locations at which the rendered image of model <NUM> and virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> should be displayed). The display locations are determined according to the P&O of the HMD and the expected position of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> relative to the HMD as further explained below. Accordingly, virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, corresponding to reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, are displayed on the HMD, aligned with the expected location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> relative to the HMD, when observed by the user looking through visor <NUM>.

When the relative position between reference tracking unit <NUM> and reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and between reference tracking unit <NUM> and internal body part <NUM> does not change (i.e., model <NUM> and the internal body part <NUM> are registered), virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> are displayed in visual alignment (<FIG>) with the corresponding reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. In addition to visually verifying the registration, when several body parts that can move one relative to the other are involved, the visual alignment of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and the corresponding virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> provides the surgeon with an indication that the displayed model of the body part may be relied on. In such cases, applying force on one body part may move the body part relative to other body parts, and may also momentarily move the tracker reference unit that is attached to one of the body parts.

If, for example, reference tracking unit <NUM> accidently moves, the relative position between reference tracking unit <NUM> and reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and between reference tracking unit <NUM> and internal body <NUM> part changes as well (i.e., model <NUM> and the internal body <NUM> part effectively fall out of registration). However, virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> are displayed on the HMD according to the originally determined relative position between reference tracking unit <NUM> and reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. Thus, reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and the corresponding virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> are displayed out of alignment (<FIG>) and the user may visually detect (i.e., without any action and with no interruption to the workflow of the procedure) that model <NUM> and the internal body part <NUM> are not registered. Also, the distance between the reference points and the corresponding virtual markers, as perceived by the user looking through visor <NUM>, may indicate the level of miss-registration between model <NUM> and the internal body part <NUM>.

In <FIG> and <FIG>, each one of virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> corresponds to a respective one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. However, virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may be replaced with a geometrical shape (e.g., a polygon, a circle, an ellipse and the like), where reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> are located on the perimeter of the geometrical shape. For example, virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may be replaced with a triangle where each vertex corresponds to one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, as illustrated by the hatched triangle in <FIG>. According to another example, when four reference points are employed, the virtual marker may be a square were each vertex corresponds to one of the reference points. According to yet another example, the virtual marker may be a single marker (i.e., similar to one of virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM>), which is alternately associated with each of the reference points for a predetermined period of time. In general, at least one virtual marker is associated with at least one of the at least one reference point for at least a portion of the time (e.g. all virtual markers are blinking or turned on and off by the user). When one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> is, for instance, a tooth, the virtual marker may take the form of an outline of the tooth as seen from the current perspective of the user. When the reference point is, for instance, the surface of a thigh, the virtual marker may take the form of a wireframe wrapping the thigh or two lines overlaying the edges of the thigh, as seen from the current perspective of the user.

Reference is now made to <FIG>, which is a schematic illustration of a system, generally referenced <NUM>, for verifying registration of a model of an internal body part with the internal body part in a reference coordinate system, constructed and operative in accordance with another embodiment of the disclosed technique. System <NUM> includes an HMD tracking unit <NUM>, a reference tracking unit <NUM>, an HMD <NUM>, a processor <NUM> and a database <NUM>. System <NUM> optionally includes an interrogation camera <NUM>, the connectivity and functionality of which is elaborated below. HMD <NUM> includes a see through display <NUM>. HMD tracking unit <NUM> is fixedly attached to HMD <NUM>. Reference tracking unit <NUM> defines a reference coordinate system <NUM>, and is fixedly placed relative to internal body part <NUM> such that the relative P&O between reference tracking unit <NUM> and internal body part <NUM> is substantially fixed. System <NUM> optionally includes a tool tracking unit <NUM> attached to a medical tool <NUM> (e.g., a needle, a pointer, a medical power drill and the like).

Processor <NUM> is coupled with database <NUM>, with HMD tracking unit <NUM>, with HMD <NUM>, with reference tracking unit <NUM> and with tool tracking unit <NUM>. HMD <NUM> along with HMD tracking unit <NUM> is intended to be donned by a user <NUM>. As mentioned above, reference tracking unit <NUM> is attached to internal body part <NUM> of patient <NUM> or fixed relative thereto, such that the relative P&O between reference tracking unit <NUM> and the internal body part <NUM> is substantially fixed. Internal body part <NUM> is, for example, a vertebra, the cranium and brain, the mandible or a femur. Similar to as mentioned above, internal body part <NUM> is unseen directly by user <NUM> (i.e., an unobstructed LOS does not exist between user <NUM> and internal body part <NUM>). System <NUM> is associated with a reference coordinate system <NUM> which, in the system <NUM> is also the coordinate system associated with reference tracking unit <NUM>. Patient <NUM> is lying on treatment bed <NUM>.

HMD tracking unit <NUM> and reference tracking unit <NUM> define an HMD tracking module, which along with processor <NUM> define an HMD tracking system which tracks the P&O of HMD <NUM> relative to reference tracking unit <NUM> (i.e., in reference coordinate system <NUM>). Tool tracking unit <NUM> and reference tracking unit <NUM> define a tool tracking module, which along with processor <NUM> define a tool tracking system which tracks the P&O of tool <NUM> relative to reference tracking unit <NUM> (i.e., in reference coordinate system <NUM>). In the example brought forth in <FIG>, both the HMD tracking system and the tool tracking system exhibit an in-out-out-in configuration. As such, HMD tracking unit <NUM> includes an optical detector and at least two emitters <NUM><NUM> and <NUM><NUM>. Reference tracking unit <NUM> includes at least one optical detector and further includes at least one emitter <NUM> (i.e., a reference emitter). Reference tracking unit <NUM> may include two or more optical detectors and two or more emitters, thereby increasing the field of view thereof. Tool tracking unit <NUM> also includes an optical detector and at least two emitters <NUM><NUM>, <NUM><NUM>. It is noted that light emitters <NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may each be a light source or a light reflector.

When tracking HMD <NUM> in reference coordinate system <NUM>, HMD tracking unit <NUM> acquires an image or images of emitter <NUM> and provides these images to processor <NUM>. Also reference tracking unit <NUM> acquires an image or images of emitters <NUM>, and <NUM><NUM> and provides these images to processor <NUM>. These acquired images include information relating to the P&O of HMD <NUM> in reference coordinate system <NUM>. Processor <NUM> determines the P&O of HMD <NUM> relative to reference tracking unit <NUM> according to these acquired images. It is noted that since processor <NUM> determines the P&O of HMD <NUM> based the information relating to the P&O included in the acquired images, the determined P&O of HMD <NUM> may differ from actual P&O of HMD <NUM> (e.g., due to partial obstruction of light emitters or optical detector lenses). Similarly, when tracking tool <NUM> in reference coordinate system <NUM>, tool tracking unit <NUM> acquires an image or images of emitter <NUM> and provides these images to processor <NUM>. Reference tracking unit <NUM> acquires an image or images of emitters <NUM>, and <NUM><NUM> and provides these images to processor <NUM>. These images include information relating to the P&O of tool <NUM> in reference coordinate system <NUM>. Processor <NUM> determines the P&O of tool <NUM> relative to reference tracking unit <NUM> according to these acquire images. Similar to as described above , since processor <NUM> determines the P&O of tool <NUM> based the information relating to the P&O included in the acquired images, the determined P&O of tool <NUM> may differ from actual P&O of tool <NUM>.

During the procedure, HMD <NUM> displays on visor <NUM> a model of body part <NUM> at a perspective corresponding to the position and orientation of user <NUM> relative to internal body part <NUM>, such that user <NUM> perceives the model to be located at the P&O of internal body part <NUM>. The model may be one-dimension (1D), two-dimensional (2D) or three-dimensional (3D). A 1D model is, for example, a single point that was determined by the surgeon as the target destination for inserting a needle or another instrument (e.g. the point is the center of a tumor when planning a brain biopsy, or a chosen point in a brain ventricle when planning the insertion of a shunt). The surgeon may determine this point at the beginning of the procedure, for instance by employing a cursor on a screen showing an MRI image data set of the brain. The term 'image data set' relates herein to collection of images of the body part. The surgeon may also determine an entry point and a trajectory which may also be displayed. An example for a 2D model is a single slice rendered from a 3D image data set. Such a 3D image data set may be, for example, a collection of two-dimensional CT or MRI axial slices also referred to herein as 'imageries'. The single rendered slice may be axial, coronal, sagittal or oblique. The surgeon can select which slice to display and change the selected slice during the procedure, for instance by using a user-interface as further describe below. Alternatively, the user may point a tracked wand at the desired slice level (i.e., relative to body part <NUM>) and the corresponding slice is displayed (i.e., according to the position of the tip of the tracked wand). A 3D model is, for example, a segmentation of the image representation of body part <NUM>, generated before or at the beginning of the procedure from a 3D image data set. A 3D model may also be a volume rendering visualization of a 3D image data set, generated dynamically from the 3D image data set according to the P&O of HMD <NUM> in reference coordinate system <NUM>. In such a case, the registration is determined between the 3D image data set from which the volume rendering is generated and body part <NUM> and thus with reference coordinate system <NUM>.

In general, when the image data set (i.e., the slice images) is registered with reference coordinate <NUM>, all the models or visualizations that are derived therefrom will also be registered with reference coordinate <NUM>. Registration of a model of a body part with body part <NUM> relates herein also to the registration of the image data set that the model is based on, or was generated from, with body part <NUM>. The model may be generated from the raw data via an automatic process, a manual process or a mixed process (e.g. automatic segmentation, manual segmentation or automatic segmentation and manual fine-tuning of the segmentation). Also, the model may comprise the image data set without any processing. The model may be based on pre-acquired images or intra-operative images. These pre-acquired or intra-operative images may be for instance Computer Tomography (CT) images, Magnetic Resonance Imager (MRI) images, ultrasound images, Proton Emission Tomography (PET) images and the like. A 3D ultrasound image data set may be generated by a tracked 2D or 3D ultrasound transducer used intraoperatively.

To display a model of body part <NUM> at a perspective corresponding to the P&O of user <NUM> relative to internal body part <NUM> (i.e., such that user <NUM> perceives the model to be located at the P&O of internal body part <NUM>), a model of internal body part <NUM> (i.e., generated from pre-acquired or intra-operative images of internal body part <NUM>) should be registered with internal body part <NUM>. Accordingly, at the onset of a medical procedure, a model of internal body part <NUM> is registered with internal body part <NUM>. To register a model of internal body part <NUM> with internal body part <NUM>, the P&O of internal body part <NUM> is determined in reference coordinate system <NUM>.

According to one exemplary alternative for registering pre-acquired imageries of internal body part <NUM> with internal body part <NUM>, a tracked C-Arm X-ray imager acquires at least two 2D X-ray images of internal body part <NUM> at two different positions and orientations relative to internal body part <NUM> in reference coordinate system <NUM>. In other words, the C-Arm X-ray imager acquires at least two images of internal body part <NUM> from at least two different perspectives. The P&O of the C-arm imager is tracked in reference coordinate system <NUM>, hence the P&Os from which the images are acquired (i.e., also in reference coordinate system <NUM>) may be determined, as further described below. Thereafter, processor <NUM> determines a P&O of the pre-acquired imageries which corresponds with the at least two acquired 2D X-ray images (i.e. processor <NUM> determines the P&O of the pre-acquired imageries in reference coordinate system <NUM>, or equivalently the P&O of internal body part <NUM> in reference coordinate system <NUM>, which are consistent with the X-ray images of internal body part <NUM>). When internal body part <NUM> includes several body parts (e.g. bones) that can move one relative to the other, processor <NUM> may determine the P&O of each body part separately. For example, when internal body part <NUM> is the spine, processor <NUM> may determine the P&O of each vertebra separately. This is required if the pre-acquired images were acquired when the patient was lying in one position, whereas during the surgery the patient is lying in another position and therefore, each vertebra may exhibit a different relative P&O.

The P&Os from which the X-Ray images are acquired in reference coordinate system <NUM> are determined similarly to as described above with respect to HMD <NUM> and tool <NUM>. To that end, an imager tracking unit is fixedly placed on the imager at a known P&O relative to the imager coordinate system. This imager tracking unit is coupled with processor <NUM> and includes an optical detector and at least two emitters. The imager tracking unit acquires an image or images (i.e., tracking images employed for tracking) of emitter <NUM> at each of the at least two perspectives, and provides these images to processor <NUM>. Reference tracking unit <NUM> acquires an image or images (i.e., also tracking images employed for tracking) of the emitters of the imager tracking unit and provides these images to processor <NUM>. The images acquired by the imager tracking unit and reference tracking unit <NUM> include information relating to the P&O of the imager coordinate system (i.e., since the imager tracking unit is fixedly placed on the imager at a known P&O relative to the imager coordinate system) in reference coordinate system <NUM>. Processor <NUM> determines the P&O of the X-ray imager coordinate system according to the images acquired by reference tracking unit <NUM> and the imager tracking unit.

According to one exemplary alternative for registering intra-operative imageries with internal body part <NUM>, a tracked 3D imager acquires intra-operative 3D image data. An imager tracking unit is fixedly attached to the 3D imager at a known P&O relative to the imager coordinate system. Hence, the P&O of the imager coordinate system in reference coordinate system <NUM> may be determined directly. The P&O of the imager coordinate system in reference coordinate system <NUM> is determined as described above. Consequently, the P&O of the 3D image data can directly be determined in reference coordinate system <NUM>.

As mentioned above, during the medical procedure internal body part <NUM> and the model thereof may effectively fall out of registration, for example if tracking reference unit <NUM> accidently moves from its original P&O. Thus, the relative P&O between reference tracking unit <NUM> and internal body part <NUM> changes. However, the registered position of the model relative to reference tracking unit <NUM> does not change with the motion thereof. As such the model is no longer registered with respective internal body part <NUM>. As another example, HMD <NUM> moves relative to the head of user <NUM>. Consequently the relative position between visor <NUM> and the eyes of user <NUM>, which was calibrated prior to the onset of the procedure, changes. However the model is rendered according to the original relative position between visor <NUM> and the eyes of user <NUM>. Thus, the model is displayed at an erroneous perspective and display position on visor <NUM> relative to the eyes of user <NUM> HMD <NUM> may include a sensor for tracking the location of the eye, and movements of HMD <NUM> relative to the head of user <NUM> may be automatically compensated for, and consequently, in such a case, the model will not effectively fall out of registration even when the HMD moves relative to the head. As a further example, when the tracking system is an optical tracking system, emitter <NUM> may be partially obscured from HMD tracking unit <NUM> (e.g., due to splattering of blood thereon) or the aperture of the optical detector of reference tracking unit <NUM> may partially covered. Similarly, light emitters <NUM><NUM> and <NUM><NUM> may be partially obscured from reference tracking unit <NUM> or the aperture of the optical detector of HMD tracking unit <NUM> may partially covered. Thus, either the images of light emitter <NUM> acquired by the HMD tracking unit <NUM>, the images of light emitters <NUM><NUM> and <NUM><NUM> acquired by the reference tracking unit <NUM>, or both may include erroneous information regarding the relative P&O between HMD tracking unit <NUM> and reference tracking unit <NUM>. The model of internal body part <NUM> may be rendered according to this erroneous relative P&O between HMD tracking unit <NUM> and reference tracking unit <NUM>. Thus, the model is displayed at an erroneous perspective and display position on visor <NUM>. It is noted that the terms 'effectively fall out of registration', 'effective miss-registration', 'effectively not-registered', herein above and below relate to at least one of the above mentioned events of movement of reference unit <NUM> from the original P&O thereof, movement of HMD <NUM> relative to the head of user <NUM> and any other events which cause erroneous display, such as emitter <NUM> being partially obscured from HMD tracking unit <NUM> or the aperture of optical detector of reference tracking unit <NUM> being partially covered. When other tracking technologies are employed, as further elaborated below, the falling out of registration may be caused by other respective issues. For example, when the tracking system is an electromagnetic tracking system, the introduction or motion of a metallic object in the vicinity of the tracking system may result in the falling out of registration.

When internal body part <NUM> and the model thereof effectively fall out or registration, user <NUM> may be presented with erroneous information regarding the position of internal body part <NUM>. To aid user <NUM> in verifying the registration between internal body <NUM> and the model thereof and to detect when they effectively fall out of registration, according to the disclosed technique, at least one reference point, directly seen by user <NUM>, is determined within the field of view of user <NUM>. In the example brought forth in <FIG>, three reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, directly seen by user <NUM>, are determined within the field of view of user <NUM>. The relative position between reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and internal body part <NUM> is substantially constant. Reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may be markings marked by user <NUM>, for example, with a marker (e.g., on the skin of patient <NUM> or on treatment bed <NUM>). Reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may also be markers (e.g., adhesive markers) attached to patient <NUM>. As a further example, reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> may be prominent features (e.g., screw heads) of a fixture with a fixed spatial relation (i.e., a fixed P&O) relative to the body part, or anatomical features similar to as mentioned above. In <FIG>, reference points <NUM><NUM>, <NUM><NUM> are exemplified as markers attached to patient <NUM> and reference point <NUM><NUM> is exemplified as attached to a fixture <NUM> attached to bed <NUM> (e.g., a fixture to which a retractor is attached). The positions of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, relative to reference tracking unit <NUM>, and thus in reference coordinate system <NUM> are determined prior to the onset of the procedure (i.e., the positions of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> are predetermined).

To determine the positions of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> relative to reference tracking unit <NUM> (i.e., and thus relative to internal body part <NUM>), user <NUM> employs a tracked pointer (i.e., tool <NUM> is a tracked pointer) and places the tip of the tracked pointer on each one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. Processor <NUM> determines the P&O of tool <NUM> in reference coordinate system <NUM> as described above, and further determines the position of the tip of tool <NUM> according to a known relative position between tool tracking unit <NUM> and the tip of tool <NUM>. User <NUM> may instruct the system to determine the P&O of tool <NUM> and determine the position of a reference point via a user interface as described below.

According to another alternative, the user moves around reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> through at least two positions. Each position is associated with a respective viewing angle of the reference point. At each viewing angle, the user designates each one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> by aiming the head such that a cross, which is displayed in the center of the visor <NUM>, aligns with each of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, and instructing the system (via a user interface) to determine the P&O of HMD <NUM> while doing so. For each viewing angle, processor <NUM> determines position related information respective of each one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> according to the P&O of HMD <NUM>. The position related information includes respective directions toward each of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. Each direction defines a line in reference coordinate system <NUM>. The intersection of the at least two lines associated with each reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> defines the location of that reference point in reference coordinate system <NUM>. The above mentioned user interface may be, for example, one of a touchscreen operated by non-sterile staff, a sterile tablet (e.g., covered with nylon), and a menu that appears on visor <NUM> that may be controlled, for example, via a sterile wireless remote control or a pedal.

According to yet another alternative, the pre-acquired or intra-operative imageries of internal body part <NUM> may include, in addition to the representation of internal body part <NUM>, representations of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the form of fiducial markers or prominent anatomical features. For example, when the internal body part is the skull or inner parts of the head, pre-acquired MRI imageries may also include the patient's eyes, specifically the canthi (i.e. the corners of the eyes). The canthi may be employed as reference points, for instance during an Ear Nose and Throat (ENT) procedure. According to another example, when the internal body part is the femur, the imageries may also include a representation of the surface of the thigh which may be used as a reference surface. In such cases, user <NUM> may mark these reference points on the image data set, for example, with the aid of a pointer on a display. These reference points (or reference surface) may also be automatically identified using image processing. For example, when the reference points are adhesive radio-opaque fiducial markers, they can be automatically identified in CT imageries by image processing. Since the model is registered with internal body part <NUM>, each position in the imageries has a corresponding position in reference coordinate system <NUM>. Thus, Processor <NUM> determines the position of each one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in reference coordinate system <NUM> and relative to reference tracking unit <NUM>.

As mentioned, the relative position between reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and reference tracking unit <NUM> is known and expected to stay substantially constant throughout the procedure. Also as mentioned above, processor <NUM> continuously determines the P&O of HMD tracking unit <NUM> relative to reference tracking unit <NUM>. Thus, processor <NUM> continuously determines the expected positions of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> relative to HMD tracking unit <NUM>. Processor <NUM> determines the display location of each one of the virtual markers (e.g., virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> - <FIG>) corresponding to reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, according to the expected position of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> relative to HMD tracking unit <NUM>, according to the known P&O between HMD tracking unit <NUM> and visor <NUM> and the known position of the eye of user <NUM> relative to HMD <NUM>. Processor <NUM> may further determine the expected perspective of each one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. Processor <NUM> renders the virtual markers (e.g., virtual markers <NUM><NUM>, <NUM><NUM> and <NUM><NUM> - <FIG>) according to the expected perspective and display location and provides an image of the rendered virtual markers to HMD <NUM> and HMD <NUM> displays this image to user <NUM>.

The virtual markers are displayed on HMD <NUM> according to the originally determined relative position between reference tracking unit <NUM> and reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, according to the relative P&O between reference tracking unit <NUM> and HMD tracking unit <NUM>, and according to the position of the eye of user <NUM> relative to HMD <NUM>. When internal body part <NUM> and the model thereof are effectively registered, reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and the corresponding virtual markers thereof are displayed in visual alignment. However, when internal body part <NUM> and the model thereof effectively fall out of registration, for instance when reference tracking unit <NUM> is moved or the determined relative P&O between reference tracking unit <NUM> and HMD tracking unit <NUM> is erroneous, the virtual markers are displayed out of alignment with respect to reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, and user <NUM> may visually detect that internal body part <NUM> and the model thereof are effectively not registered. Also, when HMD <NUM> moves relative to the head and thus the eyes of user <NUM>, the relative P&O between visor <NUM> and the eyes of user <NUM> changes and the virtual markers are displayed out of alignment with respect to reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>.

When user <NUM> identifies that internal body part <NUM> and the model thereof effectively fall out of registration, the system may aid user <NUM> to determine the source of the effective miss-registration and rectify the problem. For example, the system may request user <NUM> to validate the relative position between the eye thereof and HMD <NUM>, by holding tool <NUM> in front of HMD <NUM> and viewing a virtual model of tool <NUM> displayed on visor <NUM>. Processor <NUM> determines the P&O of tool <NUM> relative to HMD <NUM> using tool tracking unit <NUM> and HMD tracking unit <NUM> and displays the model of tool <NUM> accordingly. When user <NUM> indicates to the system that the virtual model of tool <NUM> is not aligned with tool <NUM>, the system notifies user <NUM> that the problem may be that HMD <NUM> has moved relative to the head of user <NUM> or that at least one of emitters <NUM>, and <NUM><NUM> and the optical detector in HMD tracker unit <NUM> might be partially obscured (e.g., by blood splatter). If the problem persists after cleaning the optical detector in HMD tracker unit <NUM> or emitters <NUM>, and <NUM><NUM>, the system may request the user to calibrate the current eye position, for instance by using a special calibration jig.

According to another scenario, when reference tracking unit <NUM> includes more than one optical detector, the system may instruct user <NUM> to move around patient <NUM> to check whether the virtual marker or markers appear visually out of alignment with reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, while tracking with at least one other detector in reference tracking unit <NUM> (i.e. the system may notify the user which optical detector is currently in use, for instance via a message displayed on visor <NUM>). If the virtual marker or markers appear in visual alignment with reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>, when tracking with at least one other detector in reference tracking unit <NUM>, this may indicate that the first detector was partially obscured. If the virtual marker or markers still appear out of visual alignment with reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> when tracking with at least one other detector in reference tracking unit <NUM>, then this may indicate that the reference tracking unit <NUM> has moved. The nature of such a movement may be that reference tracking unit <NUM> can be returned to its original position (e.g., when the adaptor that holds the reference tracking unit <NUM> has several discrete states or when the adaptor has one degree of freedom which is more inclined to move). In such a case, the system may instruct user <NUM> to move reference tracking unit <NUM> while continuously checking if the problem is solved. This can potentially obviate the need for re-registering (i.e., thus saving the time required by the registration process).

According to another embodiment of the disclosed technique, processor <NUM> automatically detects effective miss-registration between internal body part <NUM> and the model thereof (i.e., without the involvement of the user). According to one option, a camera acquires an image from the point of view of the user. The image includes real-world features (e.g., reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>) and virtual features (e.g., the virtual markers) as seen by the user. The camera is located along the optical axis of the line of sight of the user. Positioning the camera along the optical axis of the line of sight of the user may be implemented with a beam splitter. The camera provides the acquired image to processor <NUM> which detects the alignment or misalignment between reference <NUM><NUM>, <NUM><NUM> and <NUM><NUM> and the virtual markers.

According another option for automatic detection, reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> are light emitters (e.g., markers which include LEDs operated by a small battery, or LEDs connected with a wire to a power supply) detectable by the optical detector in HMD tracking unit <NUM>. The position of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> is determined by the user at the beginning of the procedure as described above, or alternatively the position may be automatically determined. For example, the user moves the around patient <NUM> through at least one position. Each position is associated with a respective viewing angle reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. For each position, processor <NUM> determines the P&O of HMD <NUM> in reference coordinate system <NUM>. Substantially simultaneously therewith, and for each position, the optical detector of HMD tracking unit <NUM> acquires at least one image of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. Processor <NUM> identifies reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the acquired image and determines position related information respective of each one of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> according to the P&O of HMD <NUM>. The position related information includes respective directions toward each of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. Each direction defines a line in reference coordinate system <NUM>. The intersection of the at least two lines associated with each reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> defines the location of that reference point in reference coordinate system <NUM>.

During the procedure, HMD tracking unit <NUM> acquires an image (i.e., a tracking image employed for tracking) of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> (i.e., as well as of light emitter <NUM> of reference tracking unit <NUM>) and provides the acquired image to processor <NUM>. Also, processor <NUM> determines an expected location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the acquired image according to the P&O of HMD tracking unit <NUM> and the known position of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in reference coordinate system <NUM>. Processor <NUM> then identifies and determines the actual location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the image acquired by HMD tracking unit <NUM> (i.e. in the same method used to determine the location of light emitter <NUM> in the image). When the determined actual location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the image corresponds to the expected location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the image, processor <NUM> determines that internal body part <NUM> and the model thereof are effectively registered. When determined actual location of points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the image does not correspond to the expected location, processor <NUM> determines that internal body part <NUM> and the model thereof are effectively not-registered. Once processor <NUM> detects effective miss-registration between internal body part <NUM> and the model thereof, processor <NUM> may provide an indication (e.g., a visual indication, an audio indication or a tactile indication) to user <NUM>, alerting user <NUM> that a problem exists and may further aid the user to correct the problem as described above.

According a further embodiment of the disclosed technique, processor <NUM> detects effective miss-registration between internal body part <NUM> and the model thereof by employing interrogation camera <NUM>, fixedly attached to HMD <NUM>. Interrogation camera <NUM> is coupled with processor <NUM>. During the procedure, interrogation camera <NUM> acquires an image of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> (referred to herein as 'interrogation image') and provides the acquired interrogation image to processor <NUM>. Also, since interrogation camera <NUM> is fixedly attached to HMD <NUM> in a known P&O relative to HMD tracking unit <NUM>, and processor <NUM> continuously determines the P&O of HMD <NUM> in reference coordinate system <NUM>, processor <NUM> also determines the P&O of interrogation camera <NUM> in reference coordinate system <NUM>. Processor <NUM> determines an expected location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the interrogation image according to the P&O of interrogation camera <NUM> and the known position of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in reference coordinate system <NUM>. Processor <NUM> then employs image processing techniques to identify and determine the actual location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the interrogation image. When the determined actual location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the image are congruent with (i.e., substantially equal) the expected location of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the image, processor <NUM> determines that internal body part <NUM> and the model thereof are effectively registered. When the determined actual location of points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> in the interrogation image are not congruent with the expected location, processor <NUM> determines that internal body part <NUM> and the model thereof are effectively not-registered. Once processor <NUM> detects effective miss-registration between internal body part <NUM> and the model thereof, processor <NUM> may provide an indication to user <NUM>, similar to as describe above, alerting user <NUM> that a problem exist and may further aid the user to correct the problem also as described above. It is noted that in such a case it is not required that the reference points are marked with markers that include LEDs. It is also noted that automatic detection of effective miss-registration as described above cannot detect movement of the HMD relative to the eye of the user.

As mentioned above, the system depicted in <FIG> exhibits the in-out-out-in configuration in which reference tracking <NUM> unit includes one emitter <NUM> and HMD tracking unit <NUM> includes two emitters <NUM><NUM> and <NUM><NUM>. However, reference tracking unit <NUM> may include at least two emitters and HMD tracking unit <NUM> may include one emitter. In general, in the in-out-out-in configuration reference tracking unit <NUM> includes a reference optical detector and at least one reference emitter and HMD tracking unit <NUM> includes an HMD optical detector and at least one HMD emitter. In the in-out-out-in configuration, the total number of emitters is at least three. Furthermore, either the HMD tracking system or the tool tracking system or both may be configured to exhibit either the in-out configuration or the out-in configuration. As such, in the in-out configuration, the reference tracking unit includes at least three reference emitters (i.e., similar to reference tracking unit <NUM> - <FIG>) and HMD tracking unit <NUM> and tool tracking unit <NUM> each includes a respective optical detector. In one out-in configuration, the reference tracking unit includes a reference optical detector and HMD tracking unit <NUM> and tool tracking unit <NUM> each includes at least three emitters. According to another out-in configuration, one or more optical detectors are placed near the surgical field, such that a clear line-of-sight exists between the optical detectors and the surgical field, and HMD tracking unit <NUM>, tool tracking unit <NUM> and reference tracking unit <NUM> include at least three emitters. It is noted that in all the above mentioned configurations, the light emitters may each be a light source or a light reflector. When light reflectors are used a light source is also required, which is typically located adjacent to the optical detectors. It is further noted that if more than one optical detector is used, some of the tracking units may include less than three emitters.

In the example brought forth in <FIG>, the tracking systems are optical tracking systems based on discrete light emitters. However, other tracking technologies may be employed. For example the tracking system may be based on 3D optical sensors, which in addition to a regular 2D image, also generate a depth image (i.e. for each pixel in the 2D image the 3D sensor determines the distance to the point in the scenery represented by that pixel), such as stereoscopic cameras, a structured-light 3D scanner or a Time-Of-Flight (TOF) camera. The P&O of the tracked object is determined relative to a tracking reference (e.g., a LED or LEDs, track balls, selected features in the scene) the position and orientation of which is substantially constant relative to the body part. The tracking systems may alternatively be magnetic tracking systems or ultrasonic tracking systems. When the tracking system is a magnetic tracking system, HMD tracking unit <NUM>, tool tracking unit <NUM> and reference tracking unit <NUM> include receiving coils. A reference tracking unit, which includes a transmitting coil or coils, is placed near the surgical field in a fixed manner relative to the patient and the HMD and the tool are tracked relative to this tracking unit. When the tracking systems are ultrasonic tracking systems, HMD tracking unit <NUM> and tool tracking unit <NUM> may include receivers and reference tracking unit <NUM> may include ultrasonic transmitters the HMD and the tool are tracked relative to this tracking unit. When for example magnetic tracking is employed, the disclosed technique may further provide an indication that the magnetic field according to which tracking is performed has changed (e.g., due to motion of a metallic object). In general, regardless of the tracking technology employ, the P&O of HMD <NUM> is determined relative to a tracking reference (e.g., a tracking unit or reference feature).

Reference is now made to <FIG>, which is a schematic illustration of a method for verifying registration of a model of an internal body part with the internal body part in a reference coordinate system in accordance with a further embodiment of the disclosed technique. The internal body part is unseen directly by a user. In procedure <NUM>, a model of an internal body part is registered with a corresponding body part in a reference coordinate system. With reference to <FIG> and <FIG>, model <NUM> is registered with internal body part <NUM>.

In procedure <NUM>, the position of at least one reference point is determined in the reference coordinate system, the reference point being directly visible to the user, the relative position between the at least one reference point and the internal body part is substantially constant. The reference points may be markings marked by the user or attached to the patient or the treatment bed (e.g., stickers or LEDs). These reference points may be prominent features of a fixture with a substantially fixed spatial relation relative to the body part. The Reference points may further be prominent anatomical features which are directly visible to the user such as the outside corner of the eye, a tooth or the surface of a thigh. The position of the reference points may be determined by placing the tip of a tracked tool on each reference point. Alternatively, the position of the reference points may be determined by viewing each reference point from at least two different directions, where each direction defines a line respective of each reference points and identifying the reference points (i.e., either automatically when the reference points are LEDs or by aiming a cross, which is displayed in the center of the visor, such that the cross aligns with each of reference points). The intersection at least two lines associated with each reference point defines the location of that reference pointe in reference coordinate system. When the reference points are anatomical features such as the corner of the eye or the surface of the thigh, the position of such reference points may be determined, for example, by designating these reference points on acquired image data set. With reference to <FIG>, processor <NUM> determines the position of at least one reference point in reference coordinate system <NUM>.

In Procedure <NUM>, the P&O of the HMD is continuously determined relative to the internal body part in the reference coordinate system. With reference to <FIG>, processor <NUM> continuously determines the P&O of HMD <NUM> in reference coordinate system <NUM>.

In procedure <NUM> the display location of the at least one virtual marker is determined according to the expected position of a respective one of the at least one reference point relative to the HMD, according to the known position and orientation between the HMD and a visor thereof and the position of the eye of a user relative to the HMD. Furthermore, the perspective of the at least one virtual marker is determined. With reference to <FIG>, processor <NUM> determines the display location of at least one virtual marker corresponding to a respective at least one reference point.

In procedure <NUM>, at least one virtual marker is displayed to the user (i.e., on the visor of the HMD) according to the determined display location. When the model and the internal body part are registered, each of the at least one reference point and corresponding virtual marker appear visually in alignment. When the model and the internal body part effectively fall out of registration, the at least one reference point and corresponding virtual marker appear visually out of alignment. With reference to <FIG>, processor <NUM> renders at least one virtual marker at the expected perspective and display location, and provides the rendered virtual markers to HMD <NUM>. HMD <NUM> displays the at least one virtual marker to user <NUM>.

As mentioned above, effective miss-registration between the internal body part and the model thereof may be detected automatically (e.g., by employing either an interrogation camera or the optical detector of the HMD tracking module).

Reference is now made to <FIG>, which is a schematic illustration of a method for detecting miss-registration between the internal body part and the model thereof. The internal body part is unseen directly by a user. In procedure <NUM>, a model of an internal body part is registered with a corresponding body part in a reference coordinate system. With reference to <FIG> and <FIG>, model <NUM> is registered with internal body part <NUM>.

In procedure <NUM>, the position of at least one reference point is determined in the reference coordinate system, the reference point being directly visible by an imager, the relative position between the at least one reference point and the internal body part is substantially constant. The reference points may be markings marked by the user or attached to the patient or the treatment bed (e.g., stickers or LEDs). These reference points may be prominent features of a fixture with a substantially fixed spatial relation relative to the body part. The Reference points may further be prominent anatomical features which are directly visible to the user such as the outside corner of the eye, a tooth or the surface of a thigh. The position of the reference points may be determined by placing the tip of a tracked tool on each reference point. Alternatively, the position of the reference points may be determined by viewing each reference point from at least two different directions, where each direction defines a line respective of each reference points and identifying the reference points (i.e., either automatically when the reference points are LEDs or by aiming a cross, which is displayed in the center of the visor, such that the cross aligns with each of reference points). The intersection at least two lines associated with each reference point defines the location of that reference pointe in reference coordinate system. When the reference points are anatomical features such as the corner of the eye or the surface of the thigh, the position of such reference points may be determined, for example, by designating these reference points on acquired image data set. With reference to <FIG>, processor <NUM> determines the position of at least one reference point in reference coordinate system <NUM>.

In Procedure <NUM>, an image of the at least one reference point is acquired, and the position and orientation from which the image was acquired is determined in the reference coordinate system. As mentioned above, according to one alternative, the image may be an interrogation image acquired by a dedicated interrogation camera. According to another alternative, when the reference points are marked with light emitters, the image is acquired by the optical detector of the HMD (i.e., a tracking image). With reference to <FIG>, according to one alternative, interrogation camera <NUM> acquires an interrogation image of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>. According to another alternative, when reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM> are light emitters, the optical detector of HMD tracking unit <NUM> acquires an image of reference points <NUM><NUM>, <NUM><NUM> and <NUM><NUM>.

In procedure <NUM>, the at least one reference point is identified in the acquired image and the location of the at least one reference point in the image is determined. With reference to <FIG>, processor <NUM> identifies the at least one reference point in the acquired image and determines the location thereof in the image.

In procedure <NUM>, the expected location of the at least one reference point in the image is determined according to the position of the at least one reference point in the reference coordinate system and the determined position and orientation from which the image was acquired (i.e., also in the reference coordinate system). With reference to <FIG>, processor <NUM> determines the expected location of the at least one reference point in the image.

Claim 1:
A method for verifying registration of a model of an internal body part (<NUM>) with the internal body part in a reference coordinate system (<NUM>), said internal body part being at least partially unseen directly by a user, the method comprising the procedures of:
continuously determining (<NUM>) a position and orientation of a head mounted display (<NUM>) in said reference coordinate system;
determining (<NUM>) a display location in a display of said head mounted display of at least one virtual marker (<NUM>) according to an expected position of a respective at least one reference point (<NUM>) relative to said head mounted display, said at least one reference point being directly visible to said user, the relative position between said at least one reference point and said internal body part being substantially constant, the position of said at least one reference point in said reference coordinate system is predetermined; and
displaying (<NUM>) on said display of said head mounted display said at least one virtual marker according to said display location,
wherein, when said model and said internal body part are effectively registered, said at least one reference point and said corresponding at least one virtual marker appears visually in alignment.