Patent Publication Number: US-2018049622-A1

Title: Systems and methods for sensory augmentation in medical procedures

Description:
CLAIM OF BENEFIT OF FILING DATE 
     This application claims the benefit of U.S. Provisional Application Ser. No. 60/375,483 titled: “Systems and Methods of Sensory Augmentation in Medical Procedures” filed on Aug. 16, 2016. 
    
    
     FIELD OF INVENTION 
     The present invention relates to novel visualization and sensory augmentation devices, systems, methods and apparatus for positioning, localization, and situational awareness during medical procedures including but not limited to surgical, diagnostic, therapeutic and anesthetic procedures. 
     BACKGROUND INFORMATION 
     Current medical procedures are typically performed by a surgeon or medical professional with little or no assistance outside of the required tools to affect changes on the patient. For example, an orthopedic surgeon may have some measurement tools (e.g. rulers or similar) and cutting tools (e.g. saws or drills), but visual, audible and tactile inputs to the surgeon are not assisted. In other words, the surgeon sees nothing but what he or she is operating on, hears nothing but the normal communications from other participants in the operating room, and feels nothing outside of the normal feedback from grasping tools or other items of interest in the procedure. Alternatively, large console type navigation or robotic systems are utilized in which the display and cameras are located outside the sterile field away from the surgeon. These require the surgeon to repeatedly shift his or her gaze between the surgical site and the two-dimensional display. Also, the remote location of the cameras introduces line-of-sight issues when drapes, personnel or instruments obstruct the camera&#39;s view of the markers in the sterile field and the vantage point of the camera does not lend itself to imaging within the wound. Anatomic registrations are typically conducted using a stylus with markers to probe in such a way that the markers are visible to the cameras. 
     SUMMARY OF INVENTION 
     The present invention provides projection of feedback necessary for the procedure(s) visually into the user&#39;s field of view that does not require an unnatural motion or turning of the user&#39;s head to view an external screen. The augmented or virtual display manifests to the user as a natural extension or enhancement of the user&#39;s visual perception. Further, sensors and cameras located in the headpiece of the user have the same vantage point as the user, which minimizes line of site obscuration issues associated with external cameras. 3D mapping of anatomic surfaces and features with the present invention and matching them to models from pre-operative scans are faster and represent a more accurate way to register the anatomy during surgery than current stylus point cloud approaches. 
     The present invention comprises a novel sensory enhancement device or apparatus generally consisting of at least one augmentation for the user&#39;s visual, auditory or tactile senses that assists in the conduct of medical procedures. Visual assistance can be provided in the form of real time visual overlays on the user&#39;s field of view in the form of augmented reality or as a replacement of the visual scene in the form of virtual reality. Auditory assistance can be provided in the form of simple beeps and tones or more complex sounds like speech and instruction. Tactile assistance can be provided in the form of simple warning haptic feedback or more complex haptic generation with the goal of guiding the user. In the preferred embodiments, the visual (augmented or virtual) assistance will be supplemented by audio or tactile or both audio and tactile feedback. 
     The present invention provides a mixed reality surgical navigation system comprising: a head-worn display device (e.g., headset or the like), to be worn by a user (e.g., surgeon) during surgery, comprising a processor unit, a display generator, a sensor suite having at least one tracking camera; and at least one visual marker trackable by the camera, is fixedly attached to a surgical tool; wherein the processing unit maps three-dimensional surfaces of partially exposed surfaces of an anatomical object of interest with data received from the sensor suite; the processing unit establishes a reference frame for the anatomical object by matching the three dimensional surfaces to a three dimensional model of the anatomical object; the processing unit tracks a six-degree of freedom pose of the surgical tool with data received from the sensor suite; the processing unit communicates with the display to provide a mixed reality user interface comprising stereoscopic virtual images of desired features of the surgical tool and desired features of the anatomical object in the user&#39;s field of view. 
     The present invention further provides a method of using a mixed reality surgical navigation system for a medical procedure comprising: (a) providing a mixed reality surgical navigation system comprising (i) a head-worn display device comprising a processor unit, a display, a sensor suite having at least one tracking camera; and (ii) at least one visual marker trackable by the camera; (b) attaching the display device to a user&#39;s head; (c) providing a surgical tool having the marker; (d) scanning an anatomical object of interest with the sensor suite to obtain data of three-dimensional surfaces of desired features of the anatomical object; (e) transmitting the data of the three-dimensional surfaces to the processor unit for registration of a virtual three-dimensional model of the desired features of the anatomical object; (f) tracking the surgical tool with a six-degree of freedom pose with the sensor suite to obtain data for transmission to the processor unit; and (g) displaying a mixed reality user interface comprising stereoscopic virtual images of the features of the surgical tool and the features of the anatomical object in the user&#39;s field of view. 
     The present invention further provides a mixed reality user interface for a surgical navigation system comprising: stereoscopic virtual images of desired features of a surgical tool and desired features of an anatomical object of interest in a user&#39;s field of view provided by a mixed reality surgical navigation system comprising: (i) a head-worn display device comprising a processor unit, a display, a sensor suite having at least one tracking camera; and (ii) at least one visual marker trackable by the camera; wherein the mixed reality user interface is obtained by the following processes: (a) attaching the head-worn display device to a user&#39;s head; (b) providing a surgical tool having the marker; (c) scanning a desired anatomical object with the sensor suite to obtain data of three-dimensional surfaces of partially exposed surfaces of the anatomical object; (d) transmitting the data of the three-dimensional surfaces to the processor unit for registration of a virtual three-dimensional model of the features of the anatomical object; (e) tracking the surgical tool with a six-degree of freedom pose with the sensor suite to obtain data for transmission to the processor unit; and (f) displaying a mixed reality user interface comprising stereoscopic virtual images of the features of the surgical tool and the features of the anatomical object in the user&#39;s field of view. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Some embodiments of the present invention are illustrated as an example and are not limited by the figures of the accompanying drawings, in which like references may indicate similar elements and in which: 
         FIG. 1  is a diagrammatic depiction of an augmentation system in accordance to the principles of the present invention; 
         FIG. 2A  shows a perspective front view of a diagrammatic depiction of a display device of the system of  FIG. 1 ; 
         FIG. 2B  shows a perspective back view of the display device of  FIG. 2A ; 
         FIG. 3  is a diagrammatic depiction of another embodiment of the display device of the system of  FIG. 1 ; 
         FIG. 4  is a schematic view of the electrical hardware configuration of system of  FIG. 1 ; 
         FIG. 5  is a diagrammatic depiction of markers and cameras of the system of  FIG. 1 ; 
         FIG. 6  is a diagrammatic depiction of a mixed reality user interface image (“MXUI”) provided by system of  FIG. 1  during positioning of an acetabular shell in a hip replacement procedure showing a virtual pelvis; 
         FIG. 7  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during positioning of an acetabular shell in a hip replacement procedure showing a virtual pelvis and virtual acetabular impactor; 
         FIG. 8  is a flowchart showing the operational processes of the system of  FIG. 1  during a medical procedure; 
         FIG. 9  is a flowchart showing a method of using the system of  FIG. 1  to perform a hip replacement procedure in accordance to the principles of the present invention; 
         FIG. 10  is a flowchart showing a method of using the system of  FIG. 1  to perform a general medical procedure in accordance to the principles of the present invention; 
         FIG. 11  shows a perspective view of a diagrammatic depiction of a hip impactor assembly including an acetabular shell and an optical marker; 
         FIG. 12  shows an exploded view of the hip impactor assembly shown in  FIG. 11 ; 
         FIG. 13A  shows a perspective view of a diagrammatic depiction of an anatomy marker assembly that is optionally included in the system of  FIG. 1 ; 
         FIG. 13B  shows a perspective view of a clamp assembly of the anatomy marker shown in  FIG. 13A ; 
         FIG. 14  shows an exploded view of the anatomy marker assembly shown in  FIG. 13A ; 
         FIG. 15  shows a perspective view of a diagrammatic depiction of a calibration assembly that is optionally included in the system of  FIG. 1 ; 
         FIG. 16  shows an exploded front view of the calibration assembly shown in  FIG. 15 ; 
         FIG. 17  shows an exploded back view of the calibration assembly shown in  FIG. 16 ; 
         FIG. 18  shows a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during various calibration steps; 
         FIG. 19  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during a pelvic registration step of a hip replacement procedure; 
         FIG. 20  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during insertion of a pin into a pelvis of a hip replacement procedure; 
         FIG. 21  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during a pelvic registration step of a hip replacement procedure; 
         FIG. 22  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during a femoral registration step of a hip replacement procedure; 
         FIG. 23  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during resection of the femoral neck in a hip replacement procedure; 
         FIG. 24  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during positioning of an acetabular shell in a hip replacement procedure; 
         FIG. 25  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during positioning of an acetabular shell in a hip replacement procedure; 
         FIG. 26  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during repositioning of the femur in a hip replacement procedure; 
         FIG. 27  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  using a C-arm during a hip replacement procedure; 
         FIG. 28  is a flowchart showing how the system of  FIG. 1  can be used in conjunction with a C-arm in a surgical procedure in accordance to the principles of the present invention; 
         FIG. 29  shows a front view of a diagrammatic depiction of an equipment identification and tracking label that is optionally included in the system of  FIG. 1 ; 
         FIG. 30  is a flowchart of a method for registering, sharing and tracking medical equipment using the system of  FIG. 1  in accordance to the principles of the present invention; 
         FIG. 31  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during registration of a spine with an ultrasound probe in a spinal fusion procedure; 
         FIG. 32  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during registration of a spine with a stylus in an open spinal fusion procedure; 
         FIG. 33  is a close-up front view of the surgical exposure portion of  FIG. 32 ; 
         FIG. 34  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during drilling of a pedicle in a spinal fusion procedure; 
         FIG. 35  is a close-up view of the virtual drill and target portion of  FIG. 34 ; 
         FIG. 36A  shows a perspective front view of a diagrammatic depiction of a user wearing an AR headset of the system of  FIG. 1 ; 
         FIG. 36B  shows a perspective back view of a diagrammatic depiction of a user wearing an AR headset of the system of  FIG. 1  having a protective face shield; 
         FIG. 37A  is a perspective front view of diagrammatic depiction of a user wearing an AR headset of the system of  FIG. 1  having a surgical helmet; 
         FIG. 37B  is a perspective back view of the items shown in  FIG. 37A ; 
         FIG. 38A  is a perspective front view of diagrammatic depiction of various components of the system of  FIG. 1 ; 
         FIG. 38B  is a perspective back view of the surgical helmet shown in  FIG. 37A ; 
         FIG. 39  shows a perspective front view of the AR headset shown in  FIG. 36A ; 
         FIG. 40  is an exploded view of the surgical helmet shown in  FIG. 37A ; 
         FIG. 41A  is a perspective bottom view of the electromechanical coupling plate shown in  FIG. 40 ; 
         FIG. 41B  is a perspective top view of the electromechanical coupling plate shown in  FIG. 40 ; 
         FIG. 42  is a perspective front view of components of the system shown in  37 A used in a knee replacement procedure; 
         FIG. 43  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during registration of a distal femur in a knee replacement procedure; 
         FIG. 44  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during resection plane planning in a knee replacement procedure; 
         FIG. 45  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during placement of pins for location of cutting blocks in a knee replacement procedure; 
         FIG. 46  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during tibial resection in a knee replacement procedure; 
         FIG. 47  is a perspective front view of a diagrammatic depiction of a knee balancing device that is optionally included in the system of  FIG. 1  in use during a knee replacement procedure; 
         FIG. 48  is a diagrammatic depiction of a MXUI provided by system of  FIG. 1  during a balancing assessment in a knee replacement procedure; and 
         FIG. 49  is a perspective front view of the knee balancing device shown in  FIG. 47 . 
     
    
    
     DETAILED DESCRIPTION 
     The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well as the singular forms, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising”, when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, and/or groups thereof. 
     Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one having ordinary skill in the art to which this invention belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. 
     In describing the invention, it will be understood that a number of techniques and steps are disclosed. Each of these has individual benefit and each can also be used in conjunction with one or more, or in some cases all, of the other disclosed techniques. Accordingly, for the sake of clarity, this description will refrain from repeating every possible combination of the individual steps in an unnecessary fashion. Nevertheless, the specification and claims should be read with the understanding that such combinations are entirely within the scope of the invention and claims. 
     New sensory augmentation devices, apparatuses, and methods for providing data to assist medical procedures are discussed herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention may be practiced without the specific details. 
     I. The Sensory Augmentation System 
     Referring to  FIGS. 1, 2A -B, and  3 , a sensory augmentation system  10  of the present invention is provided for use in medical procedures. The system  10  includes one or more visual markers ( 100 ,  108 ,  110 ), a processing unit  102 , a sensor suite  210  having one or more tracking camera(s)  206 , and a display device  104  having a display generator  204  that generates a visual display on the display device  104  for viewing by the user  106 . The display device  104  is attached to a user  106  such that the display device  104  can augment his visual input. In one preferred embodiment, the display device  104  is attached to the user&#39;s  106  head. Alternatively, the display device  104  is located separately from the user  106 , while still augmenting the visual scene. In one embodiment, each of the markers ( 100 ,  108 , and  110 ) is distinct and different from each other visually so they can be individually tracked by the camera(s)  206 . 
     Referring to  FIGS. 2A-2B , another exemplary embodiment of the display device  104  includes a visor housing  200  having optics  202  that allows focusing of the display generator&#39;s  204  video display onto the user&#39;s  106  eyes. The sensor suite  210  is attached or made part of the display device  104 . The visor housing  200  includes an attachment mechanism  208  that allows attachment to the user&#39;s  106  head or face such that the alignment of the display device  104  to the user&#39;s  106  visual path is consistent and repeatable 
     Referring to  FIG. 3 , another exemplary embodiment of the display device  104  includes a clear face shield  300  that allows a projection from the display generator  302  onto the shield  300  that overlays data and imagery within the visual path of the user&#39;s  106  eyes. The sensor suite  306  is attached or made part of the display device  104 . The display device  104  further includes the attachment mechanism  304 . The sensor suite  306  and the attachment mechanism  304  serve the same functions as the sensor suite  210  and the attachment mechanism  208  described above. 
     Referring to  FIG. 4  which shows the electronic hardware configuration of the system  10 , the sensor suite ( 210 ,  306 ) not only includes one or more tracking cameras  402 ,  404 ,  406  (same as  206 ), it may optionally include an inertial measurement unit (“IMU”)  408 ; a radio  410  for communication to other sensors or control units; a microphone  416  for voice activation of different display modes, including but not limited to removal of all displayed items for a clear field of view; one or more speakers  418  for audible alerts and other purposes; and haptic feedback  420  in the form of shaker motors, piezoelectric buzzers or other embodiments. The IMU  408  provides added orientation and localization data for an object that is not visually based. The IMU  408  can be used for, but is not limited to, generation of simultaneous localization and mapping (“SLAM”) data from camera tracking and IMU&#39;s  408  data to determine non-marker specific room features that assist in localization and generation of surface maps of the objects of interest. Furthermore, the sensor suite(s) ( 400 ,  210 , and  306 ) includes external data  414  as relayed by wire, radio or stored memory. External data  414  may optionally be in the forms of fluoroscopy imagery, computerized axial tomography (“CAT or CT”) scans, positron emission tomography (“PET”) scans or magnetic resonance imaging (“MRI”) data, or the like. Such data may be combined with other data collected by the sensor suite ( 400 ,  210 , and  306 ) to create augmentation imagery. 
     During operation of the system  10 , the display generator  412  (same as  204  and  302 ) and the processing unit  401  (same as  102 ) are in electronic communication with the components described above for the sensor suite ( 210 ,  306 ). The processing unit  401  is a central processing unit (“CPU”) that controls display management and algorithm prosecution. Referring to  FIG. 4 , the system  10  may optionally include one or more remote sensor suites  422 . These remote sensor suites are physically located away from the display device  104 . Each of these remote sensor suites  422  includes some or all of the components described above for the sensor suite ( 210 ,  306 ). It may also optionally include a separate and remote processing unit. The remote sensor suites  422  contribute data to the external data  414 , which may be further processed by the processing unit  401  if desired. In another embodiment, the system  10  uses the remote suite(s)  422  to track not only the markers located in the field of regard, but also any marker(s) attached to the display unit  104  worn by the user  106 , in order to localize the objects in the field of regard with respect to the user  106 . 
     In one exemplary embodiment, the system  10  uses the sensor suite(s) ( 422 ,  210 ,  306 ) to create a three-dimensional point cloud of data representing objects in the workspace. This data can be used to create or match to already modeled objects for use in subsequent tracking, visualization or playback at a later time. 
     Furthermore, the system  10  can optionally overlay imagery and masks using art-disclosed means in order to obscure objects in the field of view, including but not limited to retractors or soft tissue around an exposure that are not the subject of the procedure to assist in highlighting the area and items of interest. In one embodiment, the external image can be projected with overlays in an augmented reality (“AR”) mode. In another embodiment, the external image may be ignored and only computer-generated graphics may be used to display data to the user  106  in a virtual reality (“VR”) mode. VR mode is supported if the display device  104  or part thereof is made opaque to block the external visual data or if some other method is used to emphasize to the user  106  that concentration should be on the imagery and not the external imagery. 
     Other alternative embodiments of the display device  104  would include, but not be limited to, holographic or pseudo holographic display projection into the field of regard for the user  106 . Furthermore, the display device may optionally provide art-disclosed means of eye tracking that allows determination of the optimal displayed imagery with respect to the user&#39;s  106  visual field of view. 
     The system  10  can optionally use algorithms to discriminate between items in the field of view to identify what constitutes objects of interest versus objects not important to the task at hand. This could include, but is not limited to, identifying bony landmarks on a hip acetabulum for use in comparison and merge with a pre-operative scan in spite of soft tissue and tools that are visible in the same field of regard. 
     Referring to  FIG. 5 , the one or more cameras  500 ,  506  of the sensor suites ( 400 ,  422 ,  210 , and  306 ) and the one or more visual markers  502 ,  504  are used to visually track a distinct object (e.g., a surgical tool, a desired location within an anatomical object, etc.) and determine attitude and position relative to the user  106 . In one embodiment, each of the one or more markers is distinct and different from each other visually. Standalone object recognition and machine vision technology can be used for marker recognition. Alternatively, the present invention also provides for assisted tracking using IMUs  408  on one or more objects of interest, including but not limited to the markers  502 ,  504 . Please note that the one or more cameras  500 ,  506  can be remotely located from the user  106  and provide additional data for tracking and localization. 
     Optimal filtering algorithms are optionally used to combine data from all available sources to provide the most accurate position and orientation data for items in the field of regard. This filter scheme will be able to accommodate events including but not limited to occlusions of the camera(s) field(s) of view, blood, tissue, or other organic temporary occlusions of the desired area of interest, head movement or other camera movement that move the camera(s) field(s) of view away from the area of interest, data drop outs, and battery/power supply depletion or other loss of equipment. 
     Referring to  FIGS. 36A-B ,  37 A-B,  38 A-B, and  39 - 41 A-B, another exemplary embodiment of the display device  104  is an AR headset  3600 . The AR headset  3600  is used in various sterile surgical procedures (e.g., spinal fusion, hip and knee arthroplasty, etc.). The AR headset  3600  is clamped on the head of a surgeon  3602  (i.e., user  106 ) by adjusting a head strap  3604  by turning a thumb wheel  3606 . A transparent protective face shield  3608  is optionally attached to the device  3600  by attachment to Velcro strips  3610 . Alternatively, attachment may be via adhesive, magnetic, hooks or other art-disclosed attachment means. A coupling feature  3612  is present for attachment of a surgical helmet  3700  both mechanically and electrically to the AR headset  3600 . The surgical helmet  3700  is optionally connected to a surgical hood (not shown) that provides full body coverage for the surgeon  3602 . Full body coverage is useful for certain surgical procedures such as hip and knee arthroplasty or the like. If the surgical helmet  3700  is to be attached to a surgical hood, then a fan draws air in through the surgical hood into air inlet  3702  and is circulated under the surgical hood and helmet to cool the surgeon  3602  and prevent fogging of the optical components. A chin piece  3704  spaces the helmet  3700  (and if applicable, the attached surgical hood) away from the surgeon&#39;s  3602  face. The location of the surgical helmet  3700  relative to the AR headset  3600  is designed to allow unobstructed view of the surgical site for the surgeon  3602  and all cameras and sensors. The surgical helmet  3700  includes the necessary features to attach to and interface with the surgical hood. A flexible cord  3706  connects the AR headset  3600  to a hip module  3708 , which can be worn on the surgeon&#39;s  3602  belt. A replaceable battery  3800  inserts into the hip module  3708 . 
     Referring to  FIG. 39 , the AR headset  3600  includes a display section  3900  having a pair of see through optical displays  3902  for visual augmentation and two tracking cameras  3904  for performing tracking and stereoscopic imaging functions including two-dimensional and three-dimensional digital zoom functions. A depth sensor  3906  and a structured-light projector  3908  are included in the display section  3900 . It is preferred that the depth sensor  3906  and the projector  3908  are located in the middle of the display section  3900 . A surgical headlight  3909  is optionally mounted to the display section  3900  and may be electrically connected the AR headset  3600  to allow its brightness to be controlled by the software of the AR headset  3600  including by voice command. This feature may be deployed, for example, to dim or switch off the surgical headlight when in mixed reality mode to allow better visualization of virtual content against a bright background. It may also be adjusted to optimize optical tracking which at times can be impaired by high contrast illumination of targets or by low ambient lighting. In another exemplary embodiment, the operating room lights may be controlled wirelessly by the software of the AR headset  3600  for the same reasons. 
     Referring to  FIGS. 39-40 , the rear section  3910  of the AR headset  3600  may optionally contain the heat-generating and other components of the circuitry such as the microprocessor and internal battery. The arch-shaped bridge section  3912  and the head strap  3604  of the AR headset  3600  mechanically connect the rear section  3910  to the display section  3900 . A portion of the bridge section  3912  is flexible to accommodate size adjustments. The bridge section  3912  may include wiring or a flexible circuit board to provide electrical connectivity between the display section  3900  and the rear section  3910 . The bridge section  3912  includes the coupling feature  3612 , which is a ferromagnetic plate with a plurality of locating holes  3914  and an aperture  3918 , which provides access to two electrical contacts  3916  for powering the fan of the surgical helmet  3700 . In alternative embodiments, the coupling feature  3612  can be other art-disclosed means such as Velcro, latches or threaded fasteners or the like. The coupling feature  3612  may optionally include a vibration isolation mount to minimize transmission of mechanical noise from the fan of the surgical helmet  3700  to the AR headset  3600 , which can be detrimental to tracking performance. The fan  4004  may be software controlled allowing it to be slowed or shut down to minimize the generation of mechanical noise. It may also be controlled by the surgeon  3602  using voice commands. A flexible cord  3706  connects the rear section  3910  to the hip module  3708 . 
     Referring to  FIG. 40 , the surgical helmet  3700  includes a hollow shell  4002  into which a fan  4004  draws air which is exhausted through various vents in the shell to provide cooling air for the surgeon. A brim vent  4006  provides airflow over the visor of the surgical hood and rear vents  4008  provide cooling air to the rear including to the rear section  3910  of the AR headset  3600 . 
     Referring to  FIGS. 41A-B , the coupling plate  3802  includes a plurality of bosses  4102  for location with the holes  3914  in the AR headset  3600 . The coupling plate  3802  also includes spring-loaded electrical contacts  4104 , which connect with the electrical contacts  3916  of the AR headset  3600  to provide power to the fan  4004 . The coupling plate  3802  further includes a magnet  4106 , which provides a mechanical retention force between the coupling plate  3802  and the coupling feature  3612 . 
     In an exemplary embodiment, the AR headset  3600  is optionally used as a system for reporting device complaints or design feature requests. The user interface can have a menu option or voice command to initiate a report at the time that it occurs. This would activate voice and video camera recording allowing the user  106  to capture and narrate the complaint in 3D while the issue is occurring. The user  106  terminates complaint with voice or selecting an option. The complaint record is compressed and transmitted to the company via the internet wirelessly providing complaint handling staff excellent data to be able to “re-live” the situation first hand for better diagnosis. Artificial intelligence can be used to parse and aggregate the complaint material to establish patterns and perform statistical analysis. The same sequence can be used to connect to live technical support during the procedure with the exception that the data stream is transmitted real-time. 
     II. Pre-Operative Procedures 
     The present invention can be used for pre-operative tasks and surgical procedures. For example, an alternate general surgical procedure that includes possible pre-operative activities is now described. First, a scan of the region of interest of the patient such as CT or MRI is obtained. If possible, the patient should be positioned in a way that approximates positioning during surgery. Second, segmentation of the scan data is performed in order to convert it into three-dimensional models of items of interest including but not limited to: teeth and bony structures, veins and arteries of interest, nerves, glands, tumors or masses, implants and skin surfaces. Models are segregated so that they can later be displayed, labeled or manipulated independently. These will be referred to as pre-operative models. Third, pre-operative planning is performed (optionally using VR for visualization and manipulation of models) using models to identify items including but not limited to: anatomic reference frames, targets for resection planes, volumes to be excised, planes and levels for resections, size and optimum positioning of implants to be used, path and trajectory for accessing the target tissue, trajectory and depth of guidewires, drills, pins, screws or instruments. Fourth, the models and pre-operative planning data are uploaded into the memory of the display device  104  prior to or at time of surgery. This uploading process would most conveniently be performed wirelessly via the radio. 
     Fifth, the patient is prepared and positioned for surgery. During surgery, the surgical site is ideally be draped in a way that maximizes the visualization of skin surfaces for subsequent registration purposes. This could be achieved by liberal use of Ioban. It would be beneficial to use a film like Ioban that fluoresced or reflected differently when targeted by a specific LED or visible light emitter in a broad illumination, point or projected pattern. This film may also have optical features, markers or patterns, which allowed for easy recognition by the optical cameras of the headpiece. 
     Sixth, after the patient has been prepped and positioned for surgery, the system  10  (e.g., via the AR headset  3600 ) scans the present skin envelope to establish its present contour and creates pre-operative 3D models available for user  106  to see on the display device  104 . The preferred method is to project a grid or checkerboard pattern in infrared (“IR”) band that allows for determination of the skin envelope from the calculated warp/skew/scale of the known image. An alternate method is to move a stylus type object with a marker attached back and forth along exposed skin, allowing the position and orientation track of the stylus and subsequent generation of the skin envelope. Optionally, the skin model is displayed to the user  106 , who then outlines the general area of exposed skin, which has been scanned. An optimum position and orientation of the pre-operative skin model is calculated to match the present skin surface. The appropriate pre-operative models are displayed via the display device  104  to the user  106  in 3D. Optionally, the user  106  may then insert an optical marker into a bone of the patient for precise tracking. Placement of this marker may be informed by his visualization of the pre-operative models. The position and orientation of pre-operative models can be further refined by alternative probing or imaging including, but not limited to ultrasound. 
     Seventh, during surgery, the user  106  using the system  10  with the display device  104 , can see the pre-operative planning information and can track instruments and implants and provide intraoperative measurements of various sorts including but not limited to depth of drill or screw relative to anatomy, angle of an instrument, angle of a bone cut, etc. 
     Referring to  FIG. 8 , an exemplary embodiment of the operational flow during a procedure using the system  10  is presented. In this embodiment, the CPU  401  boots ( 800 ) and initializes one or more cameras  402 ,  404 ,  406  ( 802 ). When in the field of view of the camera(s)  402 ,  404 ,  406 , the first marker  100  is located and identified ( 804 ), followed by subsequent markers  108 ,  110  ( 806 ). The track of these markers  100 ,  108 ,  110  provides position and orientation relative to each other as well as the main camera locations ( 808 ). Alternate sensor data from sensors such as IMUs and cameras from the remote sensor suites  422  ( 810 ) can be optionally incorporated into the data collection. Further, external assistance data ( 812 ) about the patient, target, tools, or other portions of the environment may be optionally incorporated for use in the algorithms. The algorithms used in the present invention are tailored for specific procedures and data collected. The algorithms output ( 814 ) the desired assistance data for use in the display device ( 816 ). 
     III. Hip Replacement Procedures 
     In one exemplary embodiment of the present invention and referring to  FIG. 6 , the system  10  is used for hip replacement surgery wherein a first marker  600  is attached via a fixture  602  to a pelvis  604  and a second marker  606  is attached to an impactor  608 . The user  106  can see the mixed reality user interface image (“MXUI”) shown in  FIG. 6  via the display device  104 . The MXUI provides stereoscopic virtual images of the pelvis  604  and the impactor  604  in the user&#39;s field of view during the hip replacement procedure. 
     The combination of markers ( 600 ,  606 ) on these physical objects, combined with the prior processing and specific algorithms allows calculation of measures of interest to the user  106 , including real time version and inclination angles of the impactor  608  with respect to the pelvis  604  for accurate placement of acetabular shell  612 . Further, measurements of physical parameters from pre- to post-operative states can be presented, including but not limited to change in overall leg length. Presentation of data can be in readable form  610  or in the form of imagery including, but not limited, to 3D representations of tools or other guidance forms. 
       FIG. 7  depicts an alternate view of the MXUI previously shown in  FIG. 6 , wherein a virtual target  700  and a virtual tool  702  are presented to the user  106  for easy use in achieving the desired version and inclination. In this embodiment, further combinations of virtual reality are used to optimize the natural feeling experience for the user by having a virtual target  700  with actual tool  702  fully visible or a virtual tool (not shown) with virtual target fully visible. Other combinations of real and virtual imagery can optionally be provided. Presentation of data can be in readable form  704  or in the form of imagery including but not limited to 3D representations of tools or other guidance forms. 
     Referring to  FIG. 9 , the present invention further provides a method of using the system  10  to perform a hip replacement procedure ( 900 ) in which a hip bone has the socket reamed out and a replacement cup is inserted for use with a patient&#39;s leg. In this embodiment, a first marker (e.g.,  100 ,  108 , or  110 , etc.) is installed on a fixture of known dimensions with respect to the marker and this fixture is installed on the hip bone of a patient ( 902 ). A second distinct marker (e.g.,  100 ,  108 , or  110 , etc.) is installed on a pointing device of known dimensions with respect to the first marker ( 904 ). Bony landmarks or other anatomic landmarks position and orientation relative to the hip fixture are registered using the optical markers and the position/orientation difference between the hip and the pointer ( 906 ). These points are used to determine a local coordinate system ( 908 ). The pointer is used to determine position and orientation of the femur before the femur is dislocated and the acetabulum of the hip bone is reamed to make room for the replacement shell ( 910 ). An impactor with replacement shell installed on it has a third distinct marker installed with known dimensions of the impactor ( 912 ). The impactor with shell is tracked per the previously described algorithm with respect to the hip marker ( 914 ). The relative position and orientation between the hip marker and impactor are used to guide surgical placement of the shell via AR or VR display into the socket at a desired position and angle per medical requirement for the patient ( 916 ). The change in leg length can also be calculated at this point in the procedure using the marker position and orientation of the replaced femur ( 918 ). Another embodiment augments this procedure with pre-operative CT data to determine component positioning. Another embodiment uses the display output in an AR or VR manner to determine the femoral head cut. Another embodiment uses the data to place screws in the acetabulum. 
     The coordinate reference frame of the table or support on which the patient lies is desirable in some implementations. Table alignment with respect to ground, specifically gravity, can be achieved as follows. The IMU (from each of the sensor suites such as the one located within the AR headset  3600 ) provides the pitch and roll orientation of the display device  104  with respect to gravity at any given instant. Alternatively, SLAM or similar environment tracking algorithms will provide the pitch and roll orientation of the display device  104  with respect to gravity, assuming most walls and features associated with them are constructed parallel to the gravity vector. Separate from the display device&#39;s  104  relationship between to gravity, the table orientation may be determined by using the stylus to register three (3) independent points on the table. With these three points selected in the display device  104  coordinate frame, the table roll and pitch angles with respect to gravity can then be determined as well. Alternatively, the table may be identified and recognized using machine vision algorithms to determine orientation with respect to gravity. The alignment of the patient spine relative to the display device  104 , and therefore any other target coordinate systems such as defined by the hip marker, in pitch and roll is now known. To provide a yaw reference, the stylus can be used in conjunction with the hip marker to define where the patient head is located, which provides the direction of the spine with respect to him. Alternatively, image recognition of the patients head can be used for automatic determination. Ultimately, the roll, pitch and yaw of the table and/or patient spine are now fully defined in the display device  104  and all related coordinate systems. 
     Referring to  FIGS. 11-12 , the system  10  may optionally include a hip impactor assembly  1100  for use in hip arthroplasty procedures. The assembly includes an acetabular shell  1102 , and an optical marker  1104  (same as  100 ,  108 ,  110 ,  502 ,  504 ,  600 ,  606 ,  804 ,  806 ,  904 ,  912  described above) assembled to an acetabular impactor  1106 .  FIG. 12  depicts an exploded view of the assembly  1100  illustrating how the optical marker  1104  attaches to the impactor  1106  in a reproducible way by insertion of an indexed post  1200  into an indexed hole  1202 . The acetabular shell  1102  assembles reproducibly with the impactor  1106  by screwing onto a threaded distal end  1204  of the impactor and seating on a shoulder  1206 . The marker  1104  includes a first fiduciary  1108 , a second fiduciary  1110  and a third fiduciary  1112 ; each having adjacent regions of black and white wherein their boundaries form intersecting straight lines. Algorithms in the AR headset  3600  are used to process the images from the stereoscopic cameras ( 3904 ) to calculate the point of intersection of each fiduciary ( 1108 ,  1110 ,  1112 ) and thereby determine the six-degrees of freedom pose of the marker  1104 . For the purpose of this specification, “pose” is defined as the combination of position and orientation of an object. The fiducials ( 1108 ,  1110 , and  1112 ) can be created by printing on self-adhesive sticker, by laser-etching the black regions onto the surface of white plastic material or alternative methods. The shell contains a fixation hole  1114  through which a screw is optionally used to fixate the shell  1102  to the bone of the acetabulum. 
     In another exemplary embodiment and referring to  FIGS. 13A-B  and  14 , the system  10  optionally includes an anatomy marker assembly  1300  comprised of a clamp assembly  1302  and an optical marker  1304 . The clamp assembly  1302  includes a base  1400 , a first teardrop-shaped hole  1402 , and a second teardrop-shaped hole  1404 . Fixation pins (not shown) which have been fixed to the bone can be inserted through the teardrop shaped holes ( 1402 ,  1404 ) and clamped between a clamp jaw  1406  and the body  1400  thereby fixing the clamp assembly  1302  to the pins and therefore to the bone. A clamp screw  1408  engages threads in the jaws and is used to tighten the assembly  1302  onto the pins. A hexagonal hole  1410  allows a hex driver to be used to tighten the assembly  1302 . A first retaining pin  1412  and a second retaining pin  1414  prevent disassembly of the clamp assembly  1302 . A marker body  1416  has a first locating post  1418 , as second locating post  1420  and a third locating post  1422  which provide location to the base  1400  by engaging two locating posts with a locating hole  1424  and locating slot  1426  in the base. The design provides for two possible rotational positions of the marker  1304  which allows the marker  1304  to be oriented relative to the cameras (e.g.,  3904 ) in the display device  104  (e.g., the AR headset  3600 ) for optimal tracking. The marker body  1416  encapsulates a magnet (not shown) which provides sufficient holding force to the base  1400 . 
     Referring to  FIGS. 15-17 , the system  10  may optionally include a calibration assembly  1500  comprising a plate  1502  and a marker  1504  with tongue and groove assembly features for coupling them ( 1502 ,  1504 ). The tongue and groove assembly features are especially useful for precisely assembling a metal part to a plastic part, which has a different rate of thermal expansion than the metal part. The plate  1502  has a plurality of holes  1506  having a plurality of thread types to accept various impactor types. The marker  1504  has a dimple  1508  into which the tip of a stylus may be inserted for registration. The marker  1504  has a plurality of fiducials  1510 . 
       FIG. 18  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  (e.g., the AR headset  3600 ) showing the calibration assembly  1500  being used for various calibration steps. First, the hip impactor assembly  1100  can be screwed into the appropriate hole of the plate  1502  so that the shoulder  1206  is seated squarely without play against the surface of the plate  1502 . The cameras  3904  of the AR headset  3600  can then capture images which processed by an algorithm to determine the relationship between the shoulder of the impactor on which the acetabular shell will seat and the marker  1104  of the hip impactor assembly  1100 . A stylus  1800  is shown which contains a plurality of fiducials  1802  for tracking. The tip  1804  of the stylus  1800  may be inserted into the dimple  1508  of the plate  1502  allowing the coordinate of the tip  1804  relative to the marker of the stylus  1800  to be determined. A virtual guide point  1806  is shown which is projected into the user&#39;s  106  field of view at a specific location relative to the marker  1504 . The user  106  places the tip  1804  of the actual stylus  1800  where the virtual guide point  1806  is located according to the user&#39;s  106  depth perception thereby connecting his actual view with the virtual view represented by the virtual guide point. An algorithm then applies a correction factor to account for variables such as the intraocular distance of the user  106 . This is beneficial if the user&#39;s depth perception will be relied on in a mixed reality state for precise location of tools or implants. 
       FIG. 19  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  of a patient  1900  at the beginning of a hip replacement procedure. A femur marker  1902 , having a plurality of fiducials  1904  for tracking, is attached to the skin of the patient&#39;s  1900  thigh with adhesive tape such as Ioban. Alternatively, the femur marker  1902  could be fixated directly to the bone of the femur by use of pins and a clamp assembly like that depicted in  FIG. 13B . The user  106  registers the anterior landmarks of the pelvis using the tip  1804  of the stylus  1800  to determine the location of the pelvis in the reference frame of the femur marker  1902  to establish a temporary pelvic reference frame. In another embodiment, this registration can be in the body reference frame defined by SLAM scanning of the visible surface of the patient. In another embodiment, the anterior landmarks of the pelvis can be registered by generating a surface map with SLAM and having the user  106  identify each point by positioning a virtual point  1910  on each landmark in turn by motion of his head. In another embodiment, a single fiduciary  1906  can be placed at the location to be registered. A virtual circle  1908  can be used to define a mask whose position is controlled by the gaze of the user  106 . The machine vision algorithm only looks for a single fiduciary  1906  within the virtual circle  1908 . Registration steps may be triggered with a voice command by the user  106  such as “register point”. The user  106  may also register a point representing the distal femur such as the center of the patella or the medial and lateral epicondyles. When each point is registered, a virtual marker, such as a small sphere, may be positioned and remain at the location of the tip at the time of registration and beyond to provide the user  106  a visual confirmation to the user  106  and check on the quality of the registration. 
       FIG. 20  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  of a virtual pelvis  2000  and a virtual femur  2002  during a hip replacement procedure. If patient-specific models had been uploaded into the display device  104  then virtual models of these would be displayed along with any other virtual features of interest such as neurovascular structures. If not, the virtual pelvis and virtual femur could be gender-specific models, which have been scaled to best match the spacing of the registered landmarks. A first virtual trajectory  2004  and a second virtual trajectory  2006  for each of two fixation pins are displayed. In other embodiments, these may be tube-shaped or cone shaped. A drill  2008  is shown which includes a plurality of fiducials  2010  defining markers on a plurality of surfaces, which allows its pose to be tracked from various vantage points. Insertion of each pin can be guided either by lining up an actual pin  2012  with the virtual trajectory  2004  in the case where the drill is not tracked or by lining up a virtual pin (not shown) with the virtual trajectory in the case where the drill is tracked. If the drill is tracked, the angle of the drill relative to the pelvic reference frame is displayed numerically for additional augmentation. Virtual text  2014  is located on a surface  2016  of the actual drill and moves with the drill making it intuitive to the user the object to which the angles represented by the virtual text are associated. 
       FIG. 21  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during a hip replacement procedure with the anatomy marker  1300  attached to the patient&#39;s pelvis by way of clamping onto the pins  2106  inserted into the iliac crest. At this point, the reference frame relating to tracking the pelvis is transferred from the previous reference frame to that of the anatomy marker  1300 . If desired, the pelvis may be re-registered to increase accuracy. The user  106  then makes an incision and exposes the femur using a virtual pelvis  2102 , a virtual femur  2104  and virtual neurovascular structures (not shown) as a guide for the location of the incision and dissection of the muscles and joint capsule to expose the hip joint and neck of the femur. At this point, the user  106  places the leg in a reference position having approximately neutral abduction, flexion and rotation relative to the pelvis. 
       FIG. 22  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during femoral registration of a hip replacement procedure. The tip of the stylus  1800  is placed on a reference point  2200  on the proximal femur. At this time, the baseline orientation of the femur relative to the pelvis as defined by the relationship between markers  1902  and  1300  is determined and recorded. In addition, the coordinates of the reference point  2200  in the pelvic reference frame are recorded. The reference point  2200  may be enhanced by marking with a surgical pen, drilling a small hole in the bone or inserting a small tack. To improve the precision of the registration, a magnified stereoscopic image  2202  centered on the tip of the stylus is displayed as shown in  FIG. 22 . To aid the user  106  in finding the reference point later in the procedure, a baseline image, or images of the region around the point of the stylus may be recorded at the time of registration. These may be stereoscopic images. The user  106  then registers a point on the desired location of the femoral neck cut using the tip  1804  of the stylus  1800 . This is typically the most superior/lateral point of the femoral neck. An optimum resection plane is calculated which passes through this point at the appropriate abduction and version angles. 
       FIG. 23  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during resection of the femoral neck of a hip replacement procedure with a virtual resection guide  2300 . A sagittal saw  2302  is shown having a plurality of fiducials  2304  defining a marker, allows the pose of the sagittal, saw  2302  to be tracked. Resection of the femoral neck can be guided either by lining up the actual saw blade  2306  with the virtual resection guide  2300  in the case where the drill is not tracked or by lining up a virtual saw blade (not shown) with the virtual resection guide  2300  in the case where the saw  2302  is tracked. As with the tracked drill shown in  FIG. 20 , the angles of the saw  2302  may be displayed numerically if the saw  2302  is tracked. These angles could be displayed relative to the pelvic reference frame or the femoral reference frame. 
       FIG. 24  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during positioning of the acetabular shell of a hip replacement procedure wherein a virtual target  2400  for the acetabular impactor assembly  1100  and a virtual shell  2402  are shown. Placement of the acetabular impactor assembly  1100  is guided by manipulating it to align with the virtual target  2400 . The posterior/lateral quadrant of the shell portion of the virtual target may be displayed in a different color or otherwise visually differentiated from the rest of the shell  2402  to demarcate to the user  106  a target for safe placement of screws into the acetabulum. The numerical angle of the acetabular impactor and the depth of insertion relative to the reamed or un-reamed acetabulum are displayed numerically as virtual text  2404 . A magnified stereoscopic image (not shown) similar to  2202  centered on the tip of the impactor may be displayed showing how the virtual shell interfaces with the acetabulum of the virtual pelvis  2102 . 
       FIG. 25  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during positioning of the acetabular shell of a hip replacement procedure wherein a virtual axis  2500  of the acetabular impactor and the virtual target  2400  are shown. Placement of the acetabular impactor is guided by manipulating it to align the virtual axis  2500  with the virtual target  2400 . 
       FIG. 26  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during repositioning and registration of the femur of a hip replacement procedure. A virtual femur target  2600  is shown which represents the preoperative orientation of the femur relative to the pelvis during baseline femoral registration. The superior apex of this placed near the reference point on the proximal femur. A virtual femur frame  2602  is shown which represents the current orientation of the femur. As the femur is moved, the virtual femur frame  2602  rotates about the superior apex of the virtual femur target  2600 . Re-positioning the femur to the baseline orientation is achieved by manipulating the femur to align the virtual femur frame  2602  with the virtual femur target  2600  in abduction, flexion, and rotation. With the femur re-positioned in the baseline orientation, the user then uses the tip  1804  of the stylus  1800  to re-register a reference point on the proximal femur to determine the change in leg length and lateral offset from the baseline measurement. The baseline image  2604  recorded earlier during baseline femoral registration may be displayed to assist in precisely re-registering the same reference point. 
     IV. Use of System in Conjunction with a C-Arm System 
       FIG. 27  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during imaging of a patient with a C-arm. A C-arm imaging system  2700  is shown having an X-ray source  2702 , an imaging unit  2704  and a display unit  2706 . A trackable label  2708  has been attached to the C-arm  2700 . A virtual hip alignment guide  2710  and a virtual pelvis alignment guide  2712  are shown. These are perpendicular to the anterior pelvic plane and centered over the hip joint and pubic symphysis respectively. Placement of the C-arm  2700  is guided by adjusting the surface of the imaging unit  2704  to be aligned with the appropriate virtual alignment guide. If the C-arm  2700  is trackable, then a virtual C-arm alignment guide  2714  may be displayed. In this case, placement of the C-arm  2700  is guided by adjusting the virtual C-arm alignment guide  2714  to be aligned with the appropriate virtual alignment guides  2710  or  2712 . The positional and angular misalignment relative to the target can also be displayed numerically as virtual text  2718 . 
       FIG. 28  depicts a flowchart showing how the system  10  and its display device  104  (e.g., the AR headset  3600 ) can be used in conjunction with the C-arm  2700  in a surgical procedure. The camera  3904  (e.g., a high definition camera or the like) incorporated in the AR headset  3600  can be used to capture the image displayed on the C-arm monitor ( 2800 ). The image can be adjusted to “square it up” so that it matches what would be seen if the camera  3904  had been perfectly centered on and normal to the image on the monitor ( 2802 ). The knowledge of the position of the imager and source relative to the anatomy being imaged can be used to correct images for magnification and parallax distortion due to divergence of the X-ray beam from the source ( 2804 ). The corrected image can then be displayed in the AR headset  3600  ( 2806 ). This can then be used to allow the user  106  to make measurements relevant to the procedure such as acetabular cup placement or leg length ( 2808 ). Other images can be simultaneously displayed, overlaid, mirrored, or otherwise manipulated to allow the user  106  to make comparisons ( 2810 ). 
     In another embodiment, image capture can also be achieved by wireless communication between the C-arm  2700  and the AR headset  3600  for example by transfer of file in DICOM format. Alternatively, algorithms incorporating machine vision could be employed to automatically make measurements such as the inclination and version of an acetabular shell. Edge detection can be used to trace the outline of the shell. The parameters of an ellipse, which optimally matches the outline, can be determined and used to calculate the anteversion of the shell from the ratio of the length of the minor and major axes of the optimum ellipse. The inclination can be calculated for example by placing a line tangential to the most inferior aspects of the pubic rami and calculating the angle between the major axis of the shell ellipse and this line. Similarly, the comparative leg length and lateral offset of the femur can be determined and could be corrected for changes or differences in abduction of the femur by recognizing the center of rotation from the head of the femur or the center of the spherical section of the shell and performing a virtual rotation about this point to match the abduction angles. This type of calculation could be performed almost instantaneously and save time or the need to take additional radiographic images. Furthermore, and in another embodiment, an algorithm could correct for the effect of mispositioning of the pelvis on the apparent inclination and anteversion of the shell by performing a virtual rotation to match the widths and aspect ratios of the radiolucent regions representing the obturator foramens. 
     In yet another embodiment, C-arm imaging can be used to register the position of anatomy, such as the pelvis. For this, the anatomy marker  1300  would incorporate radio-opaque features of known geometry in a known pattern. The C-arm image is captured and scaled based on known marker features and displayed in the AR headset  3600 . A virtual model of the anatomy generated from a prior CT scan is displayed to the user  106 . The user  106  can manipulate the virtual model to position it in a way that its outline matches the C-arm image. This manipulation is preferably performed by tracking position and motion of the user&#39;s  106  hand using SLAM. Alternatively, the user  106  can manipulate a physical object, which incorporates a marker with the virtual model moving with the physical object. When the virtual model is correctly aligned with the C-arm image, the relationship between the patient&#39;s anatomy and the anatomy marker  1300  can be calculated. These steps and manipulations could also be performed computationally by the software by using edge detection and matching that to a projection of the profile of the model generated from the CT. 
     V. Spinal Procedures 
       FIG. 31  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during registration of a spine with ultrasound. An anatomy marker  1300  is fixated to a vertebra adjacent to the operative site. An ultrasound probe  3104  which includes a plurality of fiducials  3106  defining a marker is provided. In one embodiment, the ultrasound probe  3104  is battery operated, cordless, and can communicate with the AR headset  3600  via radio. The software has geometric and other information necessary to be able to position and scale the 2D ultrasound image relative to the marker&#39;s  1300  position. The ultrasound probe  3104  is moved over the surface of the patient  3100  to scan the region of interest. The software combines the 2D image data with the six degree of freedom pose information of the ultrasound probe  3104  relative to the anatomy marker  1300  to generate a virtual model  3108  representing the surface of the vertebrae of interest. The ultrasound probe  3104  may be rotated relative to anatomy of interest to get a more complete 3D image. The posterior contour of the spinous process and the left and right mammillary processes can be matched to the same features of a CT generated 3D model of the vertebra to register and subsequently position the virtual model of the vertebra in a mixed reality view. Alternatively, any appropriate features which are visible on an ultrasound scan can be utilized or the position of the virtual model can be relative to the surface of the patient as determined by SLAM. The latter is appropriate for procedures in which the patient anatomy of interest is stationary for the duration of the procedure and attachment of a marker would be unnecessarily invasive or burdensome. Ultrasound can similarly be used in this way to generate models of anatomy of interest such as, but not limited to, bony structures, nerves and blood vessels. Registration of any anatomy can be achieved. For example, a pelvic reference frame can be established using ultrasound to locate the proximal apex of the left and right ASIS and the pubis. The same method can be used to track the position of tools or implants percutaneously. 
       FIG. 32  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during registration of a spine with a stylus  1800 . The anatomy marker  1300  is fixated to a vertebra adjacent to the operative site. A virtual model  3200  of the patient&#39;s vertebra generated from pre-operative imaging is displayed. This virtual model includes a first landmark  3202 , a second landmark  3204  and a third landmark  3206 .  FIG. 33  depicts a close up view of the exposed anatomy shown in  FIG. 32 . The soft tissues of the patient have been dissected sufficiently to expose a first bony process  3300 , a second bony process  3302  and a third bony process  3304  which contain the three landmarks. The user  106  registers the three landmarks by placing the stylus tip  1804  at the points on the actual vertebra that best match the location of the landmarks shown on the virtual model. The software then re-positions the virtual model  3200  in the user&#39;s view to best align these points. The user  106  visually verifies the quality of the registration by comparison of the virtual model to the actual exposed regions of the vertebra. If necessary, the user  106  may make adjustments by using the tip  1804  of the stylus  1800  to reposition the virtual model. In an alternative embodiment, the landmarks are arcs traced over the most posterior aspect of each process. In another embodiment, the contours of the exposed processes are established with SLAM and the software performs a best fit on the position of the virtual model to match these contours. 
       FIG. 34  depicts an exemplary embodiment of a MXUI shown to the user  106  via the display device  104  during a spinal fusion procedure. A virtual target  3400  for the drill bit and a virtual drill bit  3402  are shown. A virtual vertebra  3404 , rendered to be transparent relative to the virtual target  3400  and virtual drill bit  3402  are shown. The numerical angle of the drill bit and the depth of penetration or distance from the tip of the drill bit to the maximum safe depth of insertion are displayed numerically as virtual text  3406 .  FIG. 35  depicts a close up view of the virtual target  3400  and virtual drill bit  3402  shown in  FIG. 34 . The virtual target  3400  is shown in the form of a rod  3500  which has a proximal cross-hair  3502  and a distal cross-hair  3504 . To maintain the actual drill bit in a safe target trajectory the user must maintain a position in which the virtual drill bit  3402  passes through the rings of both cross-hairs of the virtual target  3400 . The ideal trajectory is achieved when the virtual drill bit  3402  passes through the center of both cross hairs. If the actual drill bit moves outside a safe target trajectory the color of the virtual target  3400  changes to alert the user and an audible warning is emitted. The distal cross-hair  3504  is positioned at the planned starting point on the surface of the bony. The axial length of the virtual target  3400  and the virtual drill bit  3402  are scaled so that their proximal ends are coincident when the drill reaches its maximum planned depth. The scaling for motions of displacement of the virtual drill bit  3402  is 1:1 when it is far from the virtual target  3400  but expands to a higher magnification for greater precision when closer allowing greater precision. 
     Although this is described in the context of drilling with a drill bit, this mixed reality view can be used for multiple steps including tapping of a pedicle or driving in a pedicle screw or use of a trackable awl to find the canal of the pedicle screw. As a quick means to re-calibrate the axial location of the tip of the drill, tap or screw as they are swapped out, the user places the tip into a dimple of a marker. Implants can be introduced less invasively by AR guidance for example an interbody cage can be positioned during a PLIF, XLIF or TLIF procedure. 
     In another embodiment, a surgical drill could be equipped to communicate wirelessly with the headset to provide two-way communication. This could facilitate various safety and usability enhancing features including the following. Automatically stopping the drill or preventing operation if the drill is not within the safe target trajectory or reaches the maximum safe depth. Providing a convenient user interface to specify appropriate torque setting parameters for a torque limiting application. For example, a maximum insertion torque for a pedicle screw of a given size or a seating torque for the set screw of a pedicle screw. Actual values used could be recorded with the patient record for documentation or research purposes for example, the torque curve during drilling, the final seating torque of a pedicle screw or set screw, the implanted position of a pedicle screw or the specific implants used. 
     In another embodiment, the AR headset  3600  could be connected wirelessly to a neuromonitoring/nerve localization system, to provide the user  106  (e.g., spine surgeon) real-time warnings and measurements within his field of view, particularly during minimally invasive procedures such as XLIF. Further, when used in conjunction with pre-operative imaging in which the patient&#39;s actual nerves have been imaged and reconstructed into 3D models, if the system detects that a particular nerve has been stimulated or is being approached by the stimulating probe, the hologram representing that nerve structure can be highlighted to the user  106  to make it easier to avoid contact with or injury to the nerve structure. 
     VI. Knee Replacement Procedures 
     In another exemplary embodiment of the present invention and referring to  FIG. 42 , the system  10  is used for knee replacement surgery. A pelvis  4202 , femur  4204  and tibia  4206  of a knee replacement patient are shown in  FIG. 42 , the surgeon  4208  (i.e., the user  106 ) is shown wearing the AR headset  3600 . A femur marker  4210  and tibia marker  4212  are fixated to the femur and tibia respectively with pins. The femur is moved through a range of motion to determine the center of rotation as a proxy for the center of the hip in the reference frame of the femur marker  4210 . The knee is then flexed through a range of motion to determine the baseline, pre-operative flexion axis of the knee. The surgeon  4208  then makes an incision to expose the knee joint. A stylus  1800  is used for registration of the center of the distal femur, based on a landmark such as the most distal point of the sulcus of the trochlea. The proximal center of the tibia is defined by registration of the footprint of the ACL with the tip of the stylus. For certain minimally-invasive procedures, bony landmarks may be registered arthroscopically by insertion of the stylus through one port into the joint capsule and visualizing it with an arthroscope  4214  inserted through a second port. Further, the arthroscopic image  4216  from the arthroscope may be communicated wirelessly to the AR headset  3600  and displayed as part of a MRUI. In an alternative embodiment, a stylus tip could be incorporated in a trackable arthroscope allowing landmark registrations to be performed through a single port. The stylus  1800  may then be used to register the medial and lateral malleoli and determine the center of the ankle in the reference frame of the tibia marker  4212  by interpolation of these points. At this point a femoral reference frame is established with its origin at the center of the distal femur, with a first axis extending toward the center of the hip, a second axis defined by the flexion axis of the knee and a third axis defined as the normal to the first and second axes. A tibial reference frame is defined with its origin at the center of the proximal tibia, with a first axis extending toward the center of the ankle, a second axis defined by the flexion axis of the knee and a third axis defined as the normal to the first and second axes. These reference frames may be presented as virtual images in a MRUI. 
       FIG. 43  shows an exemplary embodiment of a MXUI shown to the surgeon  4208  via the AR headset  3600  during a knee replacement surgery with the knee exposed. A topographical map of the femoral condyles  4302  and tibial plateau  4304  can be generated by scanning with the depth sensor  3906  in the AR headset  3600  or by use of the stereoscopic cameras  3904  and SLAM. The knee would be flexed through a range of motion and the surgeon  4208  would adjust his vantage point to allow as much visualization of the condyles as possible. A circle  4306  at the center of the field of view is used by the surgeon  4208  to “paint” the condyles during the registration process and is used as a mask for the mapping algorithm. This circle may be coincident with the projection field of a structured light projector used to enhance the speed and precision of mapping. As surfaces are mapped, a virtual 3D mesh  4308  of mapped areas may be projected onto the articular surfaces to guide the surgeon  4208  and provide a visual confirmation of the quality of the surface registration. An algorithm is then used to determine the lowest point on the articular surfaces of the distal femur and the proximal tibia to determine the depth of the distal femoral and proximal tibial resections. The ideal implant sizes can be determined from the topographical map. 
     Referring to  FIG. 44 , a virtual tibial implant  4402  and virtual femoral implant  4404  can be displayed in a MXUI shown to the surgeon  4208  via the AR headset  3600 . The surgeon  4208  may switch the sizes and adjust the position of these virtual models until satisfied. In another embodiment, the virtual tibial implant may be displayed during preparation of the tibia for broaching to provide a guide for the rotational alignment of the tibial component. 
     Referring to  FIG. 45 , virtual guides  4502  for location of pins for the tibial cutting block are displayed in a MXUI shown to the surgeon  4208  via the AR headset  3600 . Virtual guides  4504  for location of pins for the distal femoral cutting block are displayed. Virtual guides  4506  for location of pins for the 4 in 1 cutting block are displayed. Placement of the actual pins is guided by aligning them with the virtual guides  4502 ,  4504  or  4506 . The femur  4508  and tibia  4510  may then be resected by placing cutting blocks on these pins. 
       FIG. 46  depicts an alternative embodiment of the MXUI shown in  FIG. 45  wherein a virtual guide  4602  is used to display the ideal plane of resection and the surgeon  4208  may resect the bone directly by alignment of the actual saw blade with the virtual guide  4602 . Alternatively, in the case of a tracked saw  4604 , the surgeon  4208  may resect the bone by alignment of a virtual saw blade  4606  with the virtual guide  4602 . Virtual text  4608  showing the varus/valgus angle, flexion angle and depth of each resection may be displayed numerically when relevant. 
       FIGS. 47 and 49  depict a knee balancing device  4700  that may be optionally included in the system  10  having a base element  4702 , a spring  4902 , a condylar element  4904 , and a condylar plate  4906 . The base element  4702  includes a handle  4908 , a target  4714  and a tibial plate  4910 . The condylar element  4904  includes a handle  4912  and a cylindrical bearing hole  4914 . The condylar plate  4906  includes a cylindrical bearing shaft  4916 , a target  4716  and two paddles  4706  and  4707 . The condylar plate  4906  pivots about a cylindrical bearing  4916 , which allows medial/lateral tilt of the condylar plate  4906  relative to the base plate  4910 . In an alternative embodiment, the bearing  4916  may be a ball-type allowing medial/lateral and flexion/extension tilt of the condylar plate  4906 . In another embodiment, the condylar plate  4906  may be contoured to match the topography of the bearing surface of a tibial implant. In another embodiment, the design could include two fully independent condylar elements each with a rigidly integrated distraction paddle and a marker. 
     Referring to  FIG. 47 , the tibial plate  4910  is seated on the resected tibia  4704 , and the distraction paddles  4706  and  4707  maintain contact with the medial femoral condyle  4708  and the lateral femoral condyle  4712  respectively. The distraction paddles  4706  and  4707  are pushed by the spring  4902  and pivot about an anteroposterior axis to provide a nearly equal and constant distraction force between each femoral condyle and the tibia. Each element includes an optical marker  4714  which allows the software to measure the degree of distraction of each femoral condyle. 
     As the knee is flexed through a range of motion, the position of each target is tracked, as is the pose of the tibia and femur. This data is used to generate a plot of medial and lateral laxity as a function of flexion angle. This information is used to calculate the ideal location of the distal femoral cutting block location pins to achieve balance through the range of motion of the knee or to guide the user in removing osteophytes or performing soft tissue releases to balance the knee through its range of motion. This plot may be displayed in a MXUI as shown in  FIG. 48  in which a first three-dimensional arc  4802  represents the medial laxity and a second three-dimensional arc  4804  represents the lateral laxity through the range of motion of the knee. The numerical values at the current flexion angle of the actual knee can be displayed as virtual text  4806 . 
     VII. Other Medical Procedures 
     Referring to  FIG. 10 , the present invention further provides a method of using the system  10  to perform other surgical procedures (specific examples are provided below). The method includes data collection ( 1000 ) that includes, but is not limited to, tracking and recognition of visual markers and IMUs. This data is used to determine relative and/or absolute orientation and position of multiple items in the work view ( 1002 ). External data ( 1004 ) is brought into the algorithm. Algorithms are used to process the data for specific use cases ( 1006 ) and determine the required output ( 1008 ). This data is used in an augmented reality AR or virtual reality VR output display ( 1010 ) to assist the medical professional. 
     For example, the method can be used for total hip arthroplasty. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ) and the determination of position and orientation ( 1002 ) of hip and surgical tools. Algorithms ( 1006 ) are used to determine solutions including, but not limited to, component positioning, femoral head cut, acetabulum positioning, screw placement, leg length determination, and locating good bone in the acetabulum for revision setting. 
     The method can also be used for total knee arthroplasty. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ) and the determination of position and orientation ( 1002 ) of knee, tibia and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to location, angle and slope of tibial cut, placement and fine-tuning of guide, avoidance of intra-medullary guide and improvement of femoral cuts. 
     The method can be used for corrective osteotomy for malunion of distal radial fractures. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan data for the determination of position and orientation ( 1002 ) of malunion and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to location of osteotomy, angle of cut and assessment of results. 
     The method can be used for corrective osteotomy for malunion of arm bones including the humerus, distal humerus, radius and ulna with fractures that can be complicated and involve angular and rotational corrections. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan data for the determination of position and orientation ( 1002 ) of malunion and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to location of osteotomy site, angle of cut, degree of correction and assessment of results. 
     The method can be used for distal femoral and proximal tibial osteotomy to correct early osteoarthritis and malalignment. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan data or long-leg X-ray imagery for the determination of position and orientation ( 1002 ) of osteotomy location and scale and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to location of osteotomy site, angle of cut, degree of correction and assessment of results. 
     The method can be used for peri-acetabular osteotomy for acetabular dysplasia. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan data for the determination of position and orientation ( 1002 ) of osteotomy location and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to location of osteotomy site, angulation, degree of correction and assessment of results. 
     The method can be used for pediatric orthopedic osteotomies similar to the previous embodiments. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan data for the determination of position and orientation ( 1002 ) of osteotomy location and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to location of osteotomy site, angle of cut, degree of correction and assessment of results. 
     The method can be used for elbow ligament reconstructions including but not limited to radial collateral ligament reconstruction (RCL) and UCL reconstruction (Tommy-John). The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or Mill data for the determination of position and orientation ( 1002 ) of isometric points for ligament reconstruction and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of tunnel placement and assessment of results. 
     The method can be used for knee ligament reconstructions including but not limited to MCL, LCL, ACL, PCL and posterolateral corner reconstructions. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or MRI data for the determination of position and orientation ( 1002 ) of isometric points for ligament reconstruction and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of tunnel placement, tunnel depth, tunnel angle, graft placement, and assessment of results. 
     The method can be used for ankle ligament reconstructions including but not limited to reconstruction to correct instability. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or Mill data for the determination of position and orientation ( 1002 ) of isometric points for ligament reconstruction and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of tunnel placement, tunnel depth, tunnel angle, and assessment of results. 
     The method can be used for shoulder acromioclavicular (AC) joint reconstruction surgical procedures including by not limited to placement not tunnels in the clavicle. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or MRI data for the determination of position and orientation ( 1002 ) of isometric points for ligament reconstruction and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of tunnel placement, tunnel depth, tunnel angle, and assessment of results. 
     The method can be used for anatomic and reverse total shoulder replacement (TSA and RSA) surgical procedures including revision TSA/RSA. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or MM data for the determination of position and orientation ( 1002 ) of humeral head, related landmarks and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of humeral head cut and glenoid bone placement, baseplate and screws, and reaming angle and guide placement for glenoid correction, and assessment of results. 
     The method can be used for total ankle arthroplasty surgical procedures. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or MRI data for the determination of position and orientation ( 1002 ) of tibia, fibula, talus, navicular and other related landmarks and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of tibial head cut, anatomic axis determination, and assessment of results. 
     The method can be used for percutaneous screw placement for pelvic fractures, tibial plateau, acetabulum and pelvis, but not limited to these areas. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or MRI data for the determination of position and orientation ( 1002 ) of anatomic and other related landmarks and surgical tools including screws. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of bones receiving screws, surrounding anatomy and soft tissue features to be avoided, localization of screws, angle of insertion, depth of insertion, and assessment of results. 
     The method can be used for in-office injections to areas including but not limited to ankle, knee, hip, shoulder and spine. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or MM data for the determination of position and orientation ( 1002 ) of related landmarks and surgical tools. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of injection location, angulation, and depth in order to maximize effect and minimize interaction with internal organs and anatomy. 
     The method can be used for pedicle screw placement for spinal fusion procedures including the lumbar and thoracic spine, but not limited to these areas. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or MRI data for the determination of position and orientation ( 1002 ) of anatomic and other related landmarks and surgical tools including screws. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of bones receiving screws, opening of the cortex, cranial-caudal angulation or similar, medio-lateral inclination, screw insertion trajectory, depth of insertion, and assessment of results. 
     The method can be used for visualization of alternate spectrum imagery including but not limited to infrared, ultraviolet, ankle, knee, hip, shoulder and spine. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may include, but is not limited to, dual color camera(s) with alternate spectrum sensitivities and/or injection dye for highlight of the patient&#39;s features for the determination of position and orientation ( 1002 ) of related landmarks and surgical tools and position, location, and type of anatomic features more readily visible in alternate spectrums including nerves, tumors, soft tissues and arteries. Algorithms ( 1006 ) are used to determine solutions including but not limited to precise localization of nerves, tumors, soft tissues of interest, arteries and other features of interest that can be enhanced with this technique. 
     The method can be used for tumor diagnostic, staging and curative surgical procedures. The markers (e.g.,  100 ,  108 ,  110 , etc.) for anatomic landmarks and tools are used for data collection ( 1000 ), which may be combined with pre-operative CT scan or MRI data for the determination of position and orientation ( 1002 ) of tumor location and surgical tools. Alternately during diagnostic surgery, localization of the tumor with respect to anatomic landmarks can be performed. Algorithms ( 1006 ) are used to determine solutions including but not limited to location of tumor site and size extent, removal guidance and assessment of results. 
     The method can be used for projection of a visible or invisible but camera visible point of light on objects of interest in the field of regard, including but not limited to bony landmarks, nerves, tumors, and other organic and inorganic objects. The markers (e.g.,  100 ,  108 ,  110 , etc.) are used to augment or supersede external data sets for anatomic data, and can be used in place of a physical pointer or tool as has been described previously. The point of light can be displayed from the user&#39;s head display or other location. The point of light can also be manifested as a pattern or other array of lights. These light(s) highlight features on the patient for determination of position and orientation ( 1002 ) of related landmarks and surgical tools, as well as augmentation of data sets including but not limited to fluoroscopy, CT scans and MRI data. Algorithms ( 1006 ) are used to determine solutions previously described but with the alternate or added selection option. 
     The method can be used for minimally invasive positioning of implants and inserting locking screws percutaneously. A marker (e.g.,  100 ,  108 , or  110 , etc.) is mounted on the proximal end of an intramedullary nail. Another marker (e.g.,  100 ,  108 , or  110 , etc.) is mounted on the cross-screw insertion tool. A virtual model of the nail is displayed including the target trajectory for the locking cross-screw. The surgeon is able to insert the cross screw by aligning the virtual cross-screw with the target trajectory. In another embodiment, the same method can be applied to the external fixation plates. In this case virtual locking plate with a plurality of locking screw trajectories, one for each hole, would be displayed. 
     VIII. Database of Trackable Instruments and Equipment 
     The present invention optionally includes the construction of an electronic database of instruments and equipment in order to allow the AR headset  3600  to identify what instruments are present in the surgical field or in the operating room area. Referring to  FIG. 29 , a serialized tracking label  2900  is optionally included in the system to facilitate the construction of such database. The serialized tracking label  2900  includes a machine-readable serial number code  2902 , a human readable serial number  2904  and a set of optical features which facilitate six-degree of freedom optical pose tracking such as a plurality of fiducials  2906 . In one embodiment, the machine-readable number code  2902  pattern can be imaged by the camera(s)  3904  of the AR headset  3600  and used alone to determine pose and position of the medical instrument using machine vision algorithms. In another embodiment, the serial number image  2904  can be imaged by the camera(s)  3904  and used alone to determine pose and position of the medical instrument using machine vision algorithms. In yet another embodiment, the entire physical model of the tracking label  2900  can be imaged by the camera(s)  3904  and used alone to determine pose and position of the medical instrument using machine vision algorithms. In another embodiment, the tracking label  2900  may be comprised or contain a wireless RFID tag for non-optical identification of equipment in a kit that can be then verified automatically using optical recognition. 
     Referring to  FIG. 30 , a flowchart showing a system for registering item type and physical parameters of equipment and storing and sharing this data for use in surgery using an augmented reality headset is provided. In this exemplary embodiment, serialized trackable labels are pre-printed on durable self-adhesive material. The label is attached ( 3002 ) to an item of equipment ( 3000 ), which could be but is not limited to a C-arm, impactor, pointer, or any other equipment used in the procedure, in a location which will be most advantageously viewed during a surgical procedure or in the preparatory effort leading to the procedure (i.e. back table operations). The label is then registered ( 3004 ) by viewing with the camera(s)  3904 , identifying the label, and initiating a database record associated with that serial number. Geometry of interest relating to the item of equipment can also be registered ( 3006 ) and stored relative to the trackable sticker. For example, in the case of a C-arm, a registration stylus may be used to register three points around the perimeter of the face of the imager and a point representing the origin of the X-ray beam source. This provides a coordinate frame, orientation (pose) data, and position data of the X-ray beam source with respect to the AR headset  3600  coordinate frame for use by the AR headset&#39;s  3600  algorithms. In one alternate embodiment, the cameras  3904  are stereo cameras and are used to scan and recognize C-arm geometry by recognition of key features such as the cylindrical or rectangular surface of the imager. Additional relevant specifications ( 3008 ) for the item of equipment can be entered into the record and includes but is not limited to the equipment type and model, calibration due date, electronic interface parameters and wireless connectivity passwords. An image of the device is captured  3010  with the camera(s)  3904 . An image of the equipment label ( 3012 ) of the device is captured. All these items are added to the completed record ( 3014 ), which is currently local to the AR headset  3600 . The record is then time-stamped and shared with a central database ( 3016 ). This may be located on a local server within the hospital system or in any remote server including any cloud based storage via the internet. Upload of the database may be done via Wi-Fi common network protocols or other art-disclosed means. The above actions may be performed by a company representative, a technician employed by the hospital, or any other trained individuals. To prevent poorly registered equipment entering the database, administrator privileges may be required to capture a record. 
     When an item of equipment is being used in surgery, the camera(s)  3904  are utilized to recognize the label as a trackable item of equipment and read the serial number ( 3018 ). The AR headset  3600  can then connect ( 3020 ) to the database and download the equipment record ( 3022 ). The equipment can thus be used in a six-degree of freedom trackable manner during the surgery ( 3024 ). If applicable, to the equipment with the data labels, the records ( 3026 ) may also be updated with data specific to the equipment itself, for example, upload images captured by the equipment during a surgery or capture logs of equipment activity during a surgery in a log. Log entries describing the use of the equipment in the surgery can be added to the database and to the patient record showing utilization of the equipment. The database thus generated can be mined for various reasons such as retrieving usage of defective equipment. 
     The system may also be used to recognize surgical instruments and implants encountered during surgery. A database of CAD models of instruments and equipment to scale is held in memory. During a procedure, SLAM or similar machine vision algorithms can capture topography of items in the scene and compare to the database on instruments and equipment. If a match is found, system can then take actions appropriate such as tracking the position and orientation of instruments relative to the patient and other instruments being used in surgery or enter a mode relevant to use of that instrument. For example, in a hip replacement procedure, if an acetabular impactor is detected, the mode for cup placement navigation is entered. 
     Although the present invention has been illustrated and described herein with reference to preferred embodiments and specific examples thereof, it will be readily apparent to those of ordinary skill in the art that other embodiments and examples may perform similar functions and/or achieve like results. All such equivalent embodiments and examples are within the spirit and scope of the present invention, are contemplated thereby, and are intended to be covered by the following claims. 
     Unless stated otherwise, dimensions and geometries of the various structures depicted herein are not intended to be restrictive of the invention, and other dimensions or geometries are possible. Plural structural components can be provided by a single integrated structure. Alternatively, a single integrated structure might be divided into separate plural components. In addition, while a feature of the present invention may have been described in the context of only one of the illustrated embodiments, such feature may be combined with one or more other features of other embodiments, for any given application. It will also be appreciated from the above that the fabrication of the unique structures herein and the operation thereof also constitute methods in accordance with the present invention.