Patent Description:
<CIT>, cited under Article <NUM>(<NUM>) EPC, discloses devices and methods for performing a surgical step or surgical procedure with visual guidance using an optical head mounted display.

According to an aspect, there is provided an augmented reality device as set out in claim <NUM>. Optional features are set out in claims <NUM> to <NUM>.

According to another aspect, there is provided a method for using an augmented reality device as set out in claim <NUM>. Optional features are set out in claims <NUM> to <NUM>.

The embodiments, examples, or aspects according to the present description which do not fall within the scope of the appended claims are provided for illustrative purposes only and do not form part of the present invention.

Systems and methods for using an augmented reality device during a surgical procedure are described herein. The systems and methods herein describe uses for the augmented reality device, such as to display virtual components or representations of real objects overlaid on a real environment. An augmented reality (AR) device allows a user to view displayed virtual objects that appear to be projected into the real environment, which is also visible. AR devices typically include two display lenses or screens, including one for each eye of a user. Light is permitted to pass through the two display lenses such that aspects of the real environment are visible while also projecting light to make virtual elements visible to the user of the AR device.

Described herein are various techniques to use augmented reality to aide in a surgical procedure. The AR devices herein include displays which may provide soft tissue visualization (e.g., a ligament, a tendon, a muscle, etc.) to allow a surgeon to view virtual soft tissue when the actual soft tissue is not visible to the surgeon (e.g., it is blocked by other tissues, a surgical instrument, implant, trial, etc.). AR displays during a minimally invasive surgery may provide visualization of anatomy without the need to expand the incision beyond what is necessary to insert the surgical instruments. The AR displays described herein may display a virtual trial or virtual implant overlaid on a bone. The virtual trial or virtual implant may be modified within the AR display, such as by user input (e.g., a gesture), which may be captured by a camera or a sensor. The modifications may include modifying a fit, size, or location of the virtual trial or implant and providing the modifications to a surgery planning system. Once a virtual trial or implant is determined to be acceptable, a physical trial or implant may be created using the virtual trial or implant as a template. Alternatively, the surgical procedure may be conducted based solely on feedback and adjustments made using the virtual trial implants.

In an example, a surgery planning system may be used to create a surgical plan preoperative. The surgical plan may be modified using the AR displays described herein, and may be modified preoperatively or intraoperatively. For example, an implant or trial size that was selected for the surgical plan may be modified using an AR display. Modifications may be sent to the surgery planning system to output a new surgical plan including the modifications. In another example, when the modifications are made intraoperatively, a new surgical plan may be updated in real time using the surgery planning system.

The AR devices described herein may be used to display a virtual guide for pin placement on a reference location of a bone. The pin placement may include displaying a cone or funnel for alignment and orientation of a pin, which may then be driven into the bone at the reference location based on the virtual guide. The AR devices may provide a virtual representation of part of a surgical instrument, such as a saw blade or other cutting tool (e.g., burr, rasp, reamer) when a portion of the surgical instrument is blocked by a bone, a cut guide, another instrument, or the like. The virtual representation of part of the surgical instrument may appear to be a virtual extension of a visible portion of the surgical instrument. A position or orientation of the virtual representation may be tracked to accurately represent the blocked or hidden portion. The AR devices described herein may display a virtual plane, axis, angle, or rotational information for range of motion testing (e.g., intraoperative range of motion testing).

While some examples and figures described herein relate to specific examples, such as total or partial knee arthroplasty, many of the techniques described herein may be used on hip arthroplasty, shoulder replacement procedures, other orthopedic implant procedures, or the like. The AR devices described herein may display a virtual representation that may replace a physical component, such as a guide, a trial, or the like. In other examples, the AR devices described herein may display virtual representations of hidden or obstructed objects, such as soft tissue, instruments, bones, or the like. The various embodiments described herein may be combined in a unified AR display, such as displaying combinations of one or more guides, trials, hidden or obstructed objects or portions of objects, etc..

<FIG> illustrates surgical field <NUM> in accordance with some embodiments. The surgical field <NUM> is illustrated in <FIG> including a surgeon <NUM>, a patient <NUM>, and may include a camera <NUM> (or in some examples multiple cameras) or a sensor. The surgeon is wearing an augmented reality (AR) device <NUM> which may be used to display a virtual object <NUM> to the surgeon <NUM>. The virtual object <NUM> may not be visible to others within the surgical field <NUM> if they are not wearing an AR device. Even if another person is viewing the surgical field with an AR device, the person may not be able to see the virtual object <NUM> or may be able to see the virtual object <NUM> in a shared augmented reality with the surgeon <NUM>, or may be able to see a modified version of the virtual object <NUM> (e.g., according to customizations unique to the surgeon <NUM> or the person). Augmented reality is explained in more detail below.

Augmented reality is a technology for displaying virtual or "augmented" objects or visual effects overlaid on a real environment. The real environment may include a room or specific area (e.g., the surgical field <NUM>), or may be more general to include the world at large. The virtual aspects overlaid on the real environment may be represented as anchored or in a set position relative to one or more aspects of the real environment. For example, the virtual object <NUM> may be configured to appear to be resting on a table. An AR system may present virtual aspects that are fixed to a real object without regard to a perspective of a viewer or viewers of the AR system (e.g., the surgeon <NUM>). For example, the virtual object <NUM> may exist in a room, visible to a viewer of the AR system within the room and not visible to a viewer of the AR system outside the room. The virtual object <NUM> in the room may be displayed to the viewer outside the room when the viewer enters the room. In this example, the room may act as a real object that the virtual object <NUM> is fixed to in the AR system.

The AR device <NUM> may include one or more screens, such as a single screen or two screens (e.g., one per eye of a user). The screens may allow light to pass through the screens such that aspects of the real environment are visible while displaying the virtual object <NUM>. The virtual object <NUM> may be made visible to the surgeon <NUM> by projecting light. The virtual object <NUM> may appear to have a degree of transparency or may be opaque (i.e., blocking aspects of the real environment).

An AR system may be viewable to one or more viewers, and may include differences among views available for the one or more viewers while retaining some aspects as universal among the views. For example, a heads-up display may change between two views while virtual objects may be fixed to a real object or area in both views. Aspects such as a color of an object, lighting, or other changes may be made among the views without changing a fixed position of at least one virtual object.

A user may see the virtual object <NUM> presented in an AR system as opaque or as including some level of transparency. In an example, the user may interact with the virtual object <NUM>, such as by moving the virtual object <NUM> from a first position to a second position. For example, the user may move an object with his or her hand. This may be done in the AR system virtually by determining that the hand has moved into a position coincident or adjacent to the object (e.g., using one or more cameras, which may be mounted on an AR device, such as AR device camera <NUM> or separate, and which may be static or may be controlled to move, or one or more sensors), and causing the object to move in response. Virtual aspects may include virtual representations of real world objects or may include visual effects, such as lighting effects, etc. The AR system may include rules to govern the behavior of virtual objects, such as subjecting a virtual object to gravity or friction, or may include other predefined rules that defy real world physical constraints (e.g., floating objects, perpetual motion, etc.). An AR device <NUM> may include a camera <NUM> on the AR device <NUM> (not to be confused with the camera <NUM>, separate from the AR device <NUM>). The AR device camera <NUM> or the camera <NUM> may include an infrared camera, an infrared filter, a visible light filter, a plurality of cameras, a depth camera, etc. The AR device <NUM> may project virtual items over a representation of a real environment, which may be viewed by a user.

The AR device <NUM> may be used in the surgical field <NUM> during a surgical procedure, for example performed by the surgeon <NUM> on the patient <NUM>. The AR device <NUM> may project or display virtual objects, such as the virtual object <NUM> during the surgical procedure to augment the surgeon's vision. The surgeon <NUM> may control the virtual object <NUM> using the AR device <NUM>, a remote controller for the AR device <NUM>, or by interacting with the virtual object <NUM> (e.g., using a hand to "interact" with the virtual object <NUM> or a gesture recognized by the camera <NUM> of the AR device <NUM>). The virtual object <NUM> may augment a surgical tool. For example, the virtual object <NUM> may appear (to the surgeon <NUM> viewing the virtual object <NUM> through the AR device <NUM>) to be coupled with or remain a fixed distance from the surgical tool. In another example, the virtual object <NUM> may be used to guide the surgical tool, and may appear to be fixed to the patient <NUM>.

<FIG> illustrates augmented reality displays 200A-200B including a partially hidden surgical instrument in accordance with some embodiments. The first AR display 200A illustrates a surgical instrument partially hidden by a bone <NUM> that is being cut by a surgeon during a surgical procedure. The surgical instrument includes a visible portion <NUM> and a hidden portion <NUM>, that is hidden within or behind the bone <NUM>.

While the hidden portion <NUM> of the surgical instrument is not visible to the surgeon, the hidden portion <NUM> may be tracked. For example, a length of the surgical instrument may be known, and a length or orientation of the visible portion <NUM> may be identified (e.g., optically) to determine an amount hidden in the hidden portion <NUM>. In another example, an end of the visible portion <NUM> may be known (e.g., have a sensor or a tracker) and the bone <NUM> may be fixed or include a sensor or a tracker to determine an entry point of the surgical instrument. In this example, the length or orientation between the sensor or tracker of the surgical instrument and the sensor or tracker of the bone may be calculated, which results in a determination of lengths or orientations of the visible portion <NUM> and the hidden portion <NUM>. Other techniques may be used to determine lengths or orientations of the visible portion <NUM> and the hidden portion <NUM>.

An AR device may be used by the surgeon to view the hidden portion <NUM> during the surgical procedure. The AR device may allow the visible portion <NUM>, the bone <NUM>, or other aspects of a surgical field to be visible through the AR device (e.g., as if the AR device was not used). The hidden portion <NUM> may be generated and displayed using the AR device. For example, the hidden portion <NUM> may appear overlaid on the bone <NUM> when viewed through the AR device. The hidden portion <NUM> may appear similar to the visible portion <NUM>. The hidden portion <NUM> may appear connected to the visible portion <NUM>, such that the visible portion <NUM> and the hidden portion <NUM> appear to be the surgical instrument (e.g., if it was not inserted into the bone <NUM>). In an example, the hidden portion <NUM> may be displayed with an augmented virtual effect, such as a color augmentation, flashing light, text, or the like.

When the surgical instrument moves, such as to go deeper or pull out of the bone <NUM>, the amount of the visible portion <NUM> and the hidden portion <NUM> may change. The AR device may update the display to accurately reflect the amount of the surgical instrument that is hidden. For example, when the surgical instrument is inserted deeper into the bone <NUM> to cut additional bone, the hidden portion <NUM> will increase, and the virtual representation of the hidden portion <NUM> may increase accordingly. The virtual representation of the hidden portion <NUM> may thus move and appear or disappear according to the movement and disappearance or appearance of the hidden portion <NUM> in the bone <NUM>.

In an example, the AR device may optionally display a cut depth <NUM>. For example, the cut depth <NUM> may be a predetermined maximum depth to cut the bone <NUM> or a target depth to cut the bone <NUM>. The cut depth <NUM> may be displayed virtually, such as overlaid on the bone <NUM> using the AR device. The cut depth <NUM> may include a plurality of depths for different cuts using the surgical device. When the surgical instrument approaches the cut depth <NUM>, the AR device may display an alert, additional information, or the like. For example, the AR device may display an enlarged view of an area of the bone <NUM> surrounding or adjacent to the cut depth <NUM>, may include a flashing light, changing color of the cut depth <NUM> or the hidden portion <NUM> (or part of the hidden portion <NUM>), a current cut depth of the surgical instrument, etc. The cut depth <NUM> may be two-dimensional or three-dimensional.

In an example, the surgical instrument may be controlled by a robot (e.g., a robotic arm), which may guide the surgical instrument in making a cut. When controlled by the robot, the hidden portion <NUM> of the surgical device may be displayed as described above. In an example, when the surgical instrument approaches the cut depth <NUM>, the surgeon may be alerted, the robot may proceed more slowly, the robot may relinquish control of the surgical instrument to the surgeon to finish the cut, the robot may change to a power-assisted mode (e.g., to allow the surgeon to control orientation, direction, and movement of the surgical instrument while providing power to the movement), or the like. The surgeon may monitor the robot making the cut by watching the hidden portion <NUM> and the cut depth <NUM>. The surgeon may stop the robot, such as if the robot approaches the cut depth <NUM> too quickly, is at the cut depth <NUM>, or is past the cut depth <NUM>.

In an example, bone <NUM> may obstruct the surgical instrument when the bone <NUM> is a tibia and the surgical instrument is a saw blade or other cutting tool (e.g., burr, rasp, reamer). The saw blade or other cutting tool (e.g., burr, rasp, reamer) may make a cut under a tibia plateau of the bone <NUM>, and be shielded from view by the bone <NUM>. The hidden portion <NUM> may be displayed with the AR device as a "ghost," superimposed on the bone <NUM>. The surgical procedure may be, for example, a total knee arthroplasty. In another example, the surgical instrument may be performing a cut during a revision surgery, the surgical instrument may include a reamer, and the cut depth <NUM> may include a depth of a reamer cut. The depth cut <NUM> may include a confirmation of depth of a cut on a resection, such as a tibia cut (e.g., proximal).

The second AR display 200B illustrates a surgical instrument partially hidden by a cut guide <NUM> and a bone (e.g., the bone <NUM>) that is being cut by a surgeon (using the cut guide <NUM>) during a surgical procedure. The cut guide <NUM> includes a slot <NUM> for inserting the surgical instrument. The surgical instrument in the second AR display 200B includes a visible portion <NUM> and a hidden portion <NUM>, similar to that described above with respect to the first AR display 200A. The second AR display 200B may include the cut depth <NUM>, as described above.

In the second AR display 200B, the hidden portion <NUM> of the surgical instrument is hidden by both the bone <NUM> and a portion of the cut guide <NUM>. The hidden portion <NUM> may be shown as a single virtual component or may have different attributes in the sections hidden behind the bone <NUM> and the cut guide <NUM>. In an example, the cut guide <NUM> or the slot <NUM> may be used to determine a length of the visible portion <NUM> or the hidden portion <NUM>, such as using sensors, imaging, or the like.

<FIG> illustrates an augmented reality display <NUM> including soft tissue visualization in accordance with some embodiments. The augmented reality display <NUM> allows a femur bone <NUM> and a tibia bone <NUM> to be viewed. The augmented reality display <NUM> includes virtual representations of soft tissue anatomy, such as virtual representations of an anterior cruciate ligament (ACL) <NUM>, a posterior cruciate ligament (PCL) <NUM>, a lateral collateral ligament (LCL) <NUM>, and a medial collateral ligament (MCL) <NUM>. These virtual ligaments are shown as examples of what virtual representations of soft tissue may be displayed using an AR display. Other examples, such as tendons, muscles, skin, blood vessels, or the like may be displayed virtually using an AR display. The virtual soft tissue shown in the AR display <NUM> may be shown overlaid in the AR display such that when a device, such as a surgical instrument, or a bone is present, the virtual soft tissue is visible through the AR display, even if the surgical instrument, other device, or bone blocks the actual soft tissue anatomy from display. For example, the patella may block all or part of the actual MCL, so the virtual MCL <NUM> may be displayed in a manner that appears to be in front of the patella such that a surgeon may see where the actual MCL is located. In a minimally invasive procedure, the augmented reality display <NUM> may include virtual representations of the femur <NUM>, the tibia <NUM> and the fibula <NUM> in addition to the soft tissues described above. The augmented reality display <NUM> may be overlaid on the surgical field where portions of the bones and/or soft tissue come into and out of actual view, with the augmented reality aspect shifting in coordination with what is actually visible to the surgeon. In an example, a sensor may be used to detect a location of an anatomical feature (e.g., bones, etc.).

In an example, the virtual soft tissue may be visible only when the actual soft tissue is blocked from view (e.g., by detecting the actual soft tissue is blocked using a camera coupled to the AR display <NUM>). The virtual soft tissue may augment the actual soft tissue. For example, the virtual soft tissue may be color coded or include a high contrast color (e.g., ligaments in a first color and tendons in a second, or each ligament may be individually color coded). The virtual soft tissue may outline or highlight the actual soft tissue. In an example, the virtual soft tissue may include a safety zone around the actual soft tissue (e.g., visually represent an area larger than the actual soft tissue) to allow a surgeon to avoid contact with the actual soft tissue with a factor of safety.

In an example, the virtual ligaments <NUM>-<NUM> shown in the AR display <NUM> may be selected, such as by a surgeon, to activate or deactivate the virtual ligaments from view. The virtual ligaments <NUM>-<NUM> may be selected together, individually, in pairs (e.g., the virtual ACL <NUM> and the virtual PCL <NUM>, or the virtual LCL <NUM> and the virtual MCL <NUM>), etc. In another example, the virtual ligaments <NUM>-<NUM> may be displayed according to a procedure being performed by the surgeon. For example, when placing a template during a knee arthroplasty, the virtual ACL <NUM> may be actively shown to allow the surgeon to avoid contact with the actual ACL. The AR display <NUM> may be used to show the virtual ACL <NUM> when inserting an ACL preserving prosthetic, to allow for easier placement.

In an example, the actual soft tissue anatomy to be displayed virtually by the AR display may be captured using imaging, such as a contrast X-ray, an MRI or ultrasound. Once the actual soft tissue anatomy is imaged, a processor may create the virtual soft tissue for display using the AR display <NUM>. The AR display <NUM> may identify (e.g., with a camera, a sensor, or a user input) a reference point (e.g., on the femur <NUM>), and the virtual soft tissue may then be displayed based on the reference point. In another example, the AR display <NUM> may show the virtual soft tissue when a device (e.g., a surgical instrument such as a saw blade, burr, rasp, reamer, etc.) approaches or enters a danger zone surrounding the actual soft tissue. The AR display <NUM> may alert the surgeon, such as by flashing a warning or increasing intensity of light or changing color of the virtual soft tissue when the danger zone is approached or entered.

In an example, the AR display <NUM> may be used to visualize range of motion aspects of the knee. A plane of motion (e.g., a flexion-extension plane, an abduction-adduction plane <NUM>, etc.) may be virtually displayed for testing a range of motion of the knee (e.g., intraoperatively or postoperatively). The plane of motion may be based on a preoperative plan or detected plane. In another example, the AR display <NUM> may include varus angle information, valgus angle information, rotational information (e.g., about a virtually displayed axis), or the like in virtual representations. The knee or leg may be rotated or moved to test various aspects of range of motion of the knee or leg, such as along a virtual plane of motion displayed in the AR display <NUM>, along a virtually displayed axis, or according to virtually displayed rotation information. In an example, aspects of the virtually displayed ligaments <NUM>-<NUM> may be displayed to check for laxity when the leg or knee is moved. An angular change in the gap between femur and tibia, such as when a varus or valgus stress is placed on the knee joint may be captured and displayed. Another example includes displaying an anterior distance change between femur and tibia (e.g., natively or with implants in place), such as when an anteriorly directed force is applied to the joint (e.g., anterior drawer test or Lachman's test).

While soft tissue has been described in reference to <FIG>, other anatomical aspects may be displayed, highlighted, or augmented in an AR display. For example, a bone outline, for example a posterior-lateral corner of the tibia, which is difficult to visualize in standard incisions for total knee arthroplasty procedures, may be displayed. Displaying virtual bone may aid in providing information to a surgeon about when to stop a saw or in sizing of implants, particularly the tibia. Other hidden bone or hard tissue may be displayed in an AR display.

<FIG> illustrates augmented reality displays 400A-400B including a virtual planned component in accordance with some embodiments. The AR displays 400A-400B allows a patient's femur bone <NUM> to be visible while displaying virtual components. The femur <NUM> may be augmented by virtual components in the AR displays 400A-400B, such as an implant or trial <NUM> in AR display 400A or resection cut guides 406A-406D in AR display 400B.

The AR display 400A illustrates the virtual trial <NUM> projected on the bone <NUM>. Preoperative determinations may be made regarding the trial to be used in a knee replacement procedure, such as a size or location of the trial. Using the preoperative determinations, a virtual trial <NUM> may be created (e.g., by a processor) for display in the AR display 400A. The virtual trial <NUM> may be adjusted, changed, or moved within the AR display 400A based on a displayed fit of the virtual trial <NUM> with the femur <NUM>. In another example, the virtual trial <NUM> may be created intraoperatively, such as without preoperative determinations for size or location of a trial. In this example, a size or location of the virtual trial <NUM> may be determined, and a physical trial may be created using the size or location information of the virtual trial <NUM>.

The AR display 400A may include a virtual representation of a gap <NUM> between a femoral trial (represented or approximated in the AR display 400A by the virtual trial <NUM>) and a tibia <NUM>. The gap <NUM> may be displayed with a length value of the gap <NUM> (e.g., <NUM> or <NUM>). The gap <NUM> may be adjusted, such as by making changes to the virtual trial <NUM> (which may represent the actual trial). A proposed bearing thickness option may be displayed, such as based on known resections or locations of tibial or femoral trials. In an example, an anterior distance change between femur and tibia (e.g., natively or with implants in place) may be displayed. When an anteriorly directed force is applied to the joint (e.g., anterior drawer test or Lachman's test), changes in the distance may be displayed. The AR display 400A may project a lighting effect (e.g., a green or red color) based on an assessment of the gap <NUM> and known minimum implant thickness. For example, when the gap <NUM> is less than the minimum implant thickness, (e.g., insufficient bone is removed), the gap may show red and if sufficient bone is removed the gap may show green.

In an example, after the actual trial is inserted, a virtual effect may be added to the actual trial, such as by using laser markings on the trial or an augmented virtual outline of the trial. The virtual effect may allow tracking of orientation of the trial without requiring optical navigation or other tracking systems. The virtual effect may be applied to the actual trial with the AR display 400A, such as using a camera (e.g., a visual light camera) on an AR device presenting the AR display 400A.

In an example, virtual markers may be placed on an actual trial and the AR display 400A may guide a surgeon in placing the actual trial, such as according to a preoperative plan. In another example, tracking markers (e.g., non-virtual) may be placed on the actual trial. The femur <NUM> may include virtual or actual markers to indicate alignment for the actual trial. In an example, the virtual or actual markers on the actual trial may align with the virtual or actual markers on the femur <NUM> to place the trial.

The AR display 400B illustrates virtual resection cuts 406A-406D to be made on the femur <NUM> such that a trial or implant may be placed. The virtual resection cuts 406A-406D may be shown virtually as lines or planes and may be moved by a surgeon operating an AR device that displays the AR display 400B and allows the femur <NUM> to be viewed. One or more of the virtual resection cuts 406A-406D may be displayed virtually in place of a cut guide or a cut block. The surgeon may make cuts with a surgical saw along the virtual resection cuts 406A-406D to prepare the femur <NUM> to receive an implant. The virtual resection cuts 406A-406D may be created based on a preoperative plan. In an example, once a cut is performed, the corresponding virtual resection cut may be removed or may be modified, such as to suggest an additional cut to fit a trial. The virtual trial <NUM> of AR display 400A and the virtual resection cuts 406A-406D AR display 400B may be interchanged, displayed concurrently, or modified, such as based on a user input. A virtual cut guide, virtual trial, or virtual implant may be used for placing an implant on the tibia <NUM> or other bone during an implant procedure (e.g., shoulder replacement, hip replacement, etc.). The virtual resection cuts 406A-406D may be used to monitor a robot performing cuts.

<FIG> illustrates augmented reality displays 500A-500B including an example virtual guide <NUM> for pin placement in accordance with some embodiments. The virtual guide <NUM> is shown in relation to a bone <NUM> for placing a pin <NUM> at a reference location <NUM> on the bone <NUM>. The first AR display 500A illustrates a perspective view of the bone <NUM> and the second AR display 500B illustrates a side view of the bone <NUM>.

In an example, the virtual guide <NUM> may appear as a funnel or cone shape to guide a surgeon in placing the pin <NUM> at the reference location <NUM>. The virtual guide <NUM> may taper or narrow down to the reference location <NUM> on a surface of the bone <NUM>, while appearing to start off the surface of the bone <NUM>. The virtual guide <NUM> may include a visual identification of orientation of the pin <NUM>. For example, the virtual guide <NUM> may be color coded according to proximity of the pin <NUM> to a portion of the virtual guide <NUM> (e.g., if the pin <NUM> is angled, a quadrant or other portion of the virtual guide <NUM> may change color or light up. The virtual guide <NUM> may be created by an AR system based on preoperative planning. The virtual guide <NUM> may be three dimensional and allow for instrumentless pin placement using AR guidance in three dimensions. In another example, the virtual guide <NUM> may be used to guide a punch or other instrument, such as an instrument with a linear action and that may have a position in multiple degrees of freedom.

AR guides similar to those illustrated in <FIG> may also be generated to guide drilling or cutting procedures. Similar guides can be used to guide cut angles or locations for a surgical saw, or other cutting tool (e.g., burr, rasp, reamer), for example.

<FIG> illustrates a system <NUM> for using an augmented reality device during a surgical procedure in accordance with some embodiments. The system includes an augmented reality device <NUM> that may be in communication with a database <NUM>. The augmented reality device <NUM> includes a processor <NUM>, memory <NUM>, an AR display <NUM>, and a camera <NUM>. The augmented reality device <NUM> may include a sensor <NUM>, a speaker <NUM>, or a haptic controller <NUM>. The database <NUM> may include image storage <NUM> or preoperative plan storage <NUM>. In an example, the augmented reality device <NUM> may be a HoloLens manufactured by Microsoft of Redmond, Washington.

The processor <NUM> of the augmented reality device <NUM> includes an augmented reality modeler <NUM>. The augmented reality modeler <NUM> may be used by the processor <NUM> to create the augmented reality environment. For example, the augmented reality modeler <NUM> may receive dimensions of a room, such as from the camera <NUM> or sensor <NUM>, and create the augmented reality environment to fit within the physical structure of the room. In another example, physical objects may be present in the room and the augmented reality modeler <NUM> may use the physical objects to present virtual objects in the augmented reality environment. For example, the augmented reality modeler <NUM> may use or detect a table present in the room and present a virtual object as resting on the table. The AR display <NUM> may display the AR environment overlaid on a real environment. The display <NUM> may show a virtual object, using the AR device <NUM>, such as in a fixed position in the AR environment.

The augmented reality device <NUM> may include a sensor <NUM>, such as an infrared sensor. The camera <NUM> or the sensor <NUM> may be used to detect movement, such as a gesture by a surgeon or other user, that may be interpreted by the processor <NUM> as attempted or intended interaction by the user with the virtual target. The processor <NUM> may identify an object in a real environment, such as through processing information received using the camera <NUM>.

The AR display <NUM>, for example during a surgical procedure, may present, such as within a surgical field while permitting the surgical field to be viewed through the augmented reality display, a virtual feature corresponding to a physical feature hidden by an anatomical aspect of a patient. The virtual feature may have a virtual position or orientation corresponding to a first physical position or orientation of the physical feature. For example, the physical feature may include soft tissue, and the virtual feature may include virtual soft tissue, which may be superimposed on the anatomical aspect of the patient. The surgical procedure may include a knee arthroplasty (e.g., total or partial). The virtual soft tissue may include one or more ligaments, such as an ACL, PCL, MCL, or LCL, to avoid when placing a template during the surgical procedure. In an example, a relative physical position or orientation of the physical feature (e.g., a ligament) to the anatomical aspect may be determined preoperatively, such as using an x-ray, an MRI, or ultrasound.

In an example, the virtual position or orientation of the virtual feature may include an offset from the first physical position or orientation of the physical feature. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical aspect, or the like.

In an example, the physical feature is a saw blade, the virtual feature is a virtual saw blade, and the anatomical aspect is a bone of the patient. The saw blade may be hidden by the bone, such as a tibia plateau while performing a cut, and the virtual saw blade may be shown overlaid on the tibia plateau to allow a virtual view of the hidden portion of the saw blade. The AR display <NUM> may display a virtual indication of a cut completion location for the saw blade, in an example. The AR display <NUM> may display a warning that the hidden physical feature is approaching or has crossed a threshold safety zone. The physical feature may be a cutting tool (e.g., burr, rasp, reamer), and the virtual feature may be a virtual representation of the cutting tool.

The camera <NUM> may detect movement of the anatomical aspect or the physical feature. The processor <NUM> may be used to cause the virtual feature to move such that the virtual position and orientation of the virtual feature correspond to a second physical position and orientation of the physical feature. Causing the virtual feature to move may be performed in response to the detected movement of the anatomical aspect or the physical feature. The speaker <NUM> may be used to play an audible warning that the hidden physical feature is approaching or has crossed a threshold safety zone.

The AR display <NUM>, for example during a surgical procedure, may project a virtual representation of an aspect of a preoperative plan, such as at a planned location within a surgical field while permitting the surgical field to be viewed through the AR display <NUM>. In an example, the aspect of the preoperative plan may include a virtual representation of a trial or an implant. The surgical component may include the trial or the implant. The AR display <NUM> may present a virtual representation of a gap thickness, such as the extension-flexion gap thickness. The trial may include laser markings or other markings for tracking an orientation of the trial. The AR display <NUM> may project a proposed bearing thickness option, such as based on known resections or preoperatively planned resections, or a location of a trial or implant. In an example, the aspect may include a virtual guide. The virtual guide may include a funnel or cone or funneling cone, such as for pin placement. The funneling cone may narrow down to a pin point (e.g., reference location) on a bone, such as a femur.

The camera <NUM> may capture an image of a surgical component within the surgical field. The camera <NUM> may detect a gesture (or capture a series of images and send the series of images to the processor <NUM> to detect the gesture), such as from a surgeon. The gesture may change a preoperative plan, move a virtual component, play or pause a video, change to a next step of a surgical plan, add or subtract virtual components, or the like. The processor <NUM> may be used to compare a location of the surgical component within the surgical field from the captured image to another location, such as the planned location within the preoperative plan. The processor <NUM> may output results of the comparison, such as for display on the AR display <NUM>.

In an example, in response to the results of the comparison being output by the processor <NUM>, the AR display <NUM> may present a visual indication of the results. The speaker <NUM> may play an audible alert or indication of the results, such as in response to the results of the comparison being output by the processor <NUM>. The haptic controller <NUM> may cause a vibration to indicate the results, such as in response to the results of the comparison being output by the processor <NUM>. In an example, the preoperative plan includes a range of motion, the virtual representation of the aspect of the preoperative plan includes a virtual range of motion visual effect, and the surgical component includes a patient limb to be moved through the range of motion.

In an example, the aspect of the preoperative plan includes a virtual representation of an implant and the surgical component may be the implant. To compare the location of the surgical component to the planned location, the processor <NUM> may determine a degree of overlap between the surgical component and the virtual representation. The results of the comparison may include the degree of overlap. In an example, such as in response to determining the degree of overlap is above a threshold, the AR display <NUM> may project the virtual representation in a color, such as green. In another example, such as in response to determining the degree of overlap is below the threshold, the AR display may project the virtual representation in a different color, such as red.

<FIG> illustrates a flow chart showing a technique <NUM> for using an augmented reality device during a surgical procedure to present a hidden feature in accordance with some embodiments. The technique <NUM> includes an operation <NUM> to present, within a surgical field while permitting the surgical field to be viewed through an augmented reality display, a virtual feature corresponding to a physical feature hidden by an anatomical aspect of a patient. The virtual feature may be presented using an AR display of the AR device. The virtual feature may have a virtual position or orientation corresponding to a first physical position or orientation of the physical feature.

In an example, presenting the virtual feature includes presenting the virtual feature at the virtual position or orientation of the virtual feature at an offset from the first physical position or orientation of the physical feature. In another example, presenting the virtual feature includes presenting a virtual representation of soft tissue superimposed on the anatomical aspect of the patient. The virtual feature may be presented during a total or partial knee arthroplasty (or shoulder replacement or hip arthroplasty). One or more virtual ligaments may be presented in the AR display, to allow a surgeon to avoid the actual ligaments when placing a template, trial, implant during a surgical procedure, or when resecting tissue, such as removing bone. The technique <NUM> may include determining a relative physical position and orientation of the physical feature to the anatomical aspect, such as preoperatively using an x-ray or an MRI. Presenting the virtual feature may include presenting a virtual saw blade corresponding to a saw blade, and wherein the anatomical aspect is a bone of the patient. The AR display may display a virtual indication of a cut completion location for the saw blade. The technique <NUM> may include displaying, using the AR display, a warning that the hidden physical feature is approaching or has crossed a threshold safety zone. An audible warning may be played indicating that the hidden feature is approaching or has crossed a threshold safety zone. In some examples, displaying the virtual feature may include calculating an approximate location of the virtual feature in reference to portion of adjacent anatomy, such as approximating positions of soft tissue based on known positions of adjacent bones. In these examples, pre-operative imaging allows for creation of starting 3D models of anatomy surrounding a joint, such as the knee, with approximations generated based on movement of adjacent bones from known locations in the starting 3D model.

The technique <NUM> includes an operation <NUM> to detect movement of the anatomical aspect or the physical feature, such as using a sensor. The technique <NUM> includes an operation <NUM> to cause, such as with a processor, the virtual feature to move such that the virtual position or orientation of the virtual feature correspond to a second physical position or orientation of the physical feature, for example, in response to the detected movement of the anatomical aspect or the physical feature.

<FIG> illustrates a flow chart showing a technique <NUM> for using an augmented reality device during a surgical procedure to project an aspect of a preoperative plan in accordance with some embodiments. The technique <NUM> includes an operation <NUM> to project, using an AR display, a virtual representation of an aspect of a preoperative plan at a planned location within a surgical field while permitting the surgical field to be viewed. In an example, the aspect of the preoperative plan includes a virtual representation of an implant. The surgical component may be the implant. When comparing the location of the surgical component to the planned location, the technique <NUM> may include determining a degree of overlap between the surgical component and the virtual representation and wherein the results of the comparison include the degree of overlap. The technique <NUM> may include projecting, using the AR display, the virtual representation in a first color (e.g., green), for example in response to determining the degree of overlap is above a threshold. The technique <NUM> may include projecting, using the AR display, the virtual representation in a second color (e.g., red) in response to determining the degree of overlap is below the threshold.

The preoperative plan may include one or more planes, such as those defining an anatomical or a mechanical axis of a joint. The one or more planes may be determined preoperatively and displayed using the AR display intraoperatively. In an example, the one or more planes may be used for range of motion testing intraoperatively, such as to guide a range of motion test. In an example, the preoperative plan may include a virtual limb alignment, for example, mechanical or anatomic, such as based on one or more bony landmarks.

In an example, the aspect of the preoperative plan includes a virtual representation of a trial and wherein the surgical component is the trial. The technique <NUM> may include presenting, using the augmented reality display, a virtual representation of an extension or flexion gap thickness. In an example, a change in gap, such as when under a varus or valgus load, may be displayed as an angle or linear measurement. The actual trial may include laser or other markings for tracking an orientation of the trial. The technique <NUM> may include projecting, using the augmented reality display, a proposed bearing thickness option based on known resections and locations of the actual trial.

The technique <NUM> includes an operation <NUM> to capture an image of a surgical component within the surgical field, such as by using a sensor. The technique <NUM> includes an operation <NUM> to compare a location of the surgical component within the surgical field from the captured image to the planned location within the preoperative plan. The technique <NUM> includes an operation <NUM> to output results of the comparison.

The aspect may include a virtual guide. The virtual guide may include a funnel, a cone, or a funneling cone, such as for placement of a pin. The funneling cone may narrow down to a reference point on a bone, such as a femur or a tibia, for placing the pin in the bone. The technique <NUM> may include detecting a gesture from a surgeon to change the preoperative plan. In an example, the technique <NUM> includes presenting, using the augmented reality display, a visual indication of the results, such as in response to the results of the comparison being output by the processor. The technique <NUM> may include playing an audible alert or indication of the results, such as in response to the results of the comparison being output. In an example, the technique <NUM> includes causing a vibration to indicate the results, such as in response to the results of the comparison being output.

<FIG> illustrates generally an example of a block diagram of a machine <NUM> upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform in accordance with some embodiments. The machine <NUM> may be a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine.

Examples, as described herein, may include, or may operate on, logic or a number of components, modules, or like mechanisms. Such mechanisms are tangible entities (e.g., hardware) capable of performing specified operations when operating. In an example, the hardware may be specifically configured to carry out a specific operation (e.g., hardwired). In an example, the hardware may include configurable execution units (e.g., transistors, circuits, etc.) and a computer readable medium containing instructions, where the instructions configure the execution units to carry out a specific operation when in operation. The configuring may occur under the direction of the executions units or a loading mechanism. Accordingly, the execution units are communicatively coupled to the computer readable medium when the device is operating. For example, under operation, the execution units may be configured by a first set of instructions to implement a first set of features at one point in time and reconfigured by a second set of instructions to implement a second set of features.

In an example, the display unit <NUM>, alphanumeric input device <NUM> and UI navigation device <NUM> may be a touch screen display. The display unit <NUM> may include goggles, glasses, or other AR or VR display components. For example, the display unit may be worn on a head of a user and may provide a heads-up-display to the user. The alphanumeric input device <NUM> may include a virtual keyboard (e.g., a keyboard displayed virtually in a VR or AR setting.

The machine <NUM> may include an output controller <NUM>, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices.

The storage device <NUM> may include a machine readable medium <NUM> that is non-transitory on which is stored one or more sets of data structures or instructions <NUM> (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein.

While the machine readable medium <NUM> is illustrated as a single medium, the term "machine readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) configured to store the one or more instructions <NUM>.

The term "machine readable medium" may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine <NUM> and that cause the machine <NUM> to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine readable medium examples may include solid-state memories, and optical and magnetic media. Specific examples of machine readable media may include: nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions <NUM> may further be transmitted or received over a communications network <NUM> using a transmission medium via the network interface device <NUM> utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) <NUM> family of standards known as Wi-Fi®, as the personal area network family of standards known as Bluetooth® that are promulgated by the Bluetooth Special Interest Group, peer-to-peer (P2P) networks, among others. In an example, the network interface device <NUM> may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network <NUM>. In an example, the network interface device <NUM> may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine <NUM>, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

Claim 1:
An augmented reality device (<NUM>) for use during a surgical procedure, the device comprising:
an augmented reality display (<NUM>) to present, within a surgical field while permitting the surgical field (<NUM>) to be viewed through the augmented reality display (<NUM>), a virtual feature corresponding to a physical feature hidden by an anatomical aspect of a patient, the virtual feature having a virtual position and orientation corresponding to a first physical position and orientation of the physical feature, the physical feature including at least one of a soft tissue or a bone;
a sensor (<NUM>) to detect movement of the anatomical aspect or the physical feature; and
a processor (<NUM>) to, in response to the detected movement of the anatomical aspect or the physical feature, cause the virtual feature to move such that the virtual position and orientation of the virtual feature correspond to a second physical position and orientation of the physical feature.