Patent Publication Number: US-2022211507-A1

Title: Patient-matched orthopedic implant

Description:
BACKGROUND 
     Surgical joint repair procedures involve repair and/or replacement of a damaged or diseased joint. Many times, a surgical joint repair procedure, such as joint arthroplasty as an example, involves replacing the damaged joint with a prosthetic, or set of prosthetics, that is implanted into the patient&#39;s bone. Proper selection of a prosthetic that is appropriately sized and shaped and proper positioning of that prosthetic to ensure an optimal surgical outcome can be challenging. To assist with positioning, the surgical procedure often involves the use of surgical instruments to control the shaping of the surface of the damaged bone and cutting or drilling of bone to accept the prosthetic. 
     SUMMARY 
     This disclosure describes a variety of techniques for designing, manufacturing, and using patient specific implants for surgical joint repair procedures. The techniques may be used independently or in various combinations to support particular phases or settings for surgical joint repair procedures or to provide a multi-faceted ecosystem to support surgical joint repair procedures. In various examples, this disclosure describes techniques for preoperative surgical planning including implant design, implant manufacture, intra-operative surgical planning, intra-operative surgical guidance, intra-operative surgical tracking and post-operative analysis using mixed reality (MR)-based visualization. In some examples, the disclosure also describes surgical items and/or methods for performing surgical joint repair procedures. 
     The details of various examples of the disclosure are set forth in the accompanying drawings and the description below. Various features, objects, and advantages will be apparent from the description, drawings, and claims. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram of an orthopedic surgical system according to an example of this disclosure. 
         FIG. 2  is a block diagram of an orthopedic surgical system that includes a mixed reality (MR) system, according to an example of this disclosure. 
         FIG. 3  is a flowchart illustrating example phases of a surgical lifecycle. 
         FIG. 4  is a flowchart illustrating preoperative, intraoperative and postoperative workflows in support of an orthopedic surgical procedure. 
         FIG. 5  is a schematic representation of a visualization device for use in a mixed reality (MR) system, according to an example of this disclosure. 
         FIG. 6  is a block diagram illustrating example components of a visualization device for use in a mixed reality (MR) system, according to an example of this disclosure. 
         FIG. 7  is a flowchart illustrating example steps in the preoperative phase of the surgical lifecycle. 
         FIG. 8  is a flowchart illustrating example steps for tailoring a surgical plan to a patient. 
         FIG. 9  is a flowchart illustrating example steps for obtaining a model of a bone of a patient. 
         FIGS. 10A-10D  are conceptual diagrams illustrating example phases in a mask generation process. 
         FIG. 11  is a flowchart illustrating example steps for generating a patient matched implant model. 
         FIGS. 12A-12I  are conceptual diagrams illustrating example phases in a patient matched implant design process. 
         FIGS. 13A-13C  are conceptual diagrams illustrating example views of a virtual extrusion for a patient matched implant design process. 
         FIGS. 14A and 14B  are conceptual diagrams illustrating a virtual extrusion and corresponding projected points for a patient matched implant design process. 
         FIGS. 15A and 15B  are conceptual diagrams illustrating examples of patient matched implants. 
         FIGS. 16A and 16B  are conceptual diagrams illustrating examples of patient matched implants. 
         FIGS. 17A and 17B  are conceptual diagrams illustrating examples of patient matched implants. 
         FIG. 18  illustrates an example of a page of a user interface of a mixed reality (MR) system, according to an example of this disclosure. 
         FIG. 19  is an example of an install guide page of the user interface of  FIG. 18 , according to an example of this disclosure. 
         FIG. 20  is an example of an install implant page of the user interface of  FIG. 18 , according to an example of this disclosure. 
         FIG. 21  is a flowchart illustrating example stages of a shoulder joint repair surgery. 
         FIG. 22  illustrates an image perceptible to a user when in an augment surgery mode of a mixed reality (MR) system, according to an example of this disclosure. 
         FIG. 23  is a conceptual diagram illustrating an MR system providing virtual guidance to a user for installation of a guide in a glenoid of a scapula, in accordance with one or more techniques of this disclosure. 
         FIG. 24  is a conceptual diagram illustrating an example guide as installed in a glenoid in a shoulder arthroplasty procedure. 
         FIG. 25  is a conceptual diagram illustrating reaming of a glenoid in a shoulder arthroplasty procedure, in accordance with one or more techniques of this disclosure. 
         FIGS. 26 and 27  are conceptual diagrams illustrating creation of a central hole in a glenoid in a shoulder arthroplasty procedure, in accordance with one or more techniques of this disclosure. 
         FIG. 28  is a conceptual diagram illustrating a glenoid prosthesis with keel type anchorage. 
         FIGS. 29-31  are conceptual diagrams illustrating creation of keel type anchorage positions in a glenoid in a shoulder arthroplasty procedure, in accordance with one or more techniques of this disclosure. 
         FIG. 32  is a conceptual diagram illustrating a glenoid prosthesis with pegged type anchorage. 
         FIGS. 33 and 34  are conceptual diagrams illustrating creation of pegged type anchorage positions in a glenoid in a shoulder arthroplasty procedure, in accordance with one or more techniques of this disclosure. 
         FIG. 35  is a conceptual diagram illustrating attachment of an implant to a glenoid in a shoulder arthroplasty procedure, in accordance with one or more techniques of this disclosure. 
         FIGS. 36 and 37  illustrate screws and a central stem that may be used to attach a prothesis to a glenoid in a shoulder arthroplasty procedure. 
     
    
    
     DETAILED DESCRIPTION 
     In some orthopedic surgical procedures, a surgeon may implant one or more implant devices in a patient. The implant devices may be available in several different standard shapes, styles, and sizes. The surgeon may select a particular prosthetic device (e.g., a particular shape, style, and/or size) to implant based on various characteristic of the patient. The surgeon may perform various steps to prepare the patient&#39;s bone to receive the implant device. These steps may include removal of portions of the bone (e.g., via reaming) in order to create a surface of the bone that matches a surface of the implant device. Matching surfaces between the bone and the implant device may provide for better patient outcomes (e.g., as the implant device may have a better fit with the bone and be more solidly affixed to the bone). However, in some examples, it may be desirable to minimize, or eliminate, the need to remove portions of a bone to prepare the bone to receive an implant device. For instance, patients who undergo an orthopedic surgical procedure may have limited healthy bone available. 
     In accordance with one or more techniques of this disclosure, a system (e.g., a surgical planning system) may facilitate the designing of patient specific implant devices. For instance, the system may obtain a three-dimensional (3D) model of a bone of the patient (e.g., generated based on images of the bone, such as x-ray or magnetic resonance imaging (MRI) images), and a template model of an implant device (e.g., a computer-aided design (CAD) model of the implant device). The system may generate a model of a patient specific implant device based on the 3D model of the bone and the template model of the implant device. For instance, the system may generate the model of a patient specific implant device such that a surface of the patient specific implant device matches a surface of the bone. 
     The system may output the generated model for manufacturing. For instance, the system may output the model to be manufactured into a physical patient specific implant device that a surgeon may subsequently implant into the patient. In this way, the system may facilitate the design of patient specific implant devices. 
     Orthopedic surgery can involve implanting one or more prosthetic devices to repair or replace a patient&#39;s damaged or diseased joint. Virtual surgical planning tools that use image data of the diseased or damaged joint may be used to generate an accurate three-dimensional bone model that can be viewed and manipulated preoperatively by the surgeon. These tools can enhance surgical outcomes by allowing the surgeon to simulate the surgery, select or design an implant that more closely matches the contours of the patient&#39;s actual bone, and select or design surgical instruments and guide tools that are adapted specifically for repairing the bone of a particular patient. Use of these planning tools typically results in generation of a preoperative surgical plan, complete with an implant and surgical instruments that are selected or manufactured for the individual patient. Oftentimes, once in the actual operating environment, the surgeon may desire to verify the preoperative surgical plan intraoperatively relative to the patient&#39;s actual bone. 
     This verification may result in a determination that an adjustment to the preoperative surgical plan is needed, such as a different implant, a different positioning or orientation of the implant, and/or a different surgical guide for carrying out the surgical plan. In addition, a surgeon may want to view details of the preoperative surgical plan relative to the patient&#39;s real bone during the actual procedure in order to more efficiently and accurately position and orient the implant components. For example, the surgeon may want to obtain intraoperative visualization that provides guidance for positioning and orientation of implant components, guidance for preparation of bone or tissue to receive the implant components, guidance for reviewing the details of a procedure or procedural step, and/or guidance for selection of tools or implants and tracking of surgical procedure workflow. 
     Accordingly, this disclosure describes systems and methods for using a mixed reality (MR) visualization system to assist with creation, implementation, verification, and/or modification of a surgical plan before and during a surgical procedure. Because MR, or in some instances VR, may be used to interact with the surgical plan, this disclosure may also refer to the surgical plan as a “virtual” surgical plan. Visualization tools other than or in addition to mixed reality visualization systems may be used in accordance with techniques of this disclosure. A surgical plan, e.g., as generated by the BLUEPRINT™ system, available from Wright Medical Group, N.V., or another surgical planning platform, may include information defining a variety of features of a surgical procedure, such as features of particular surgical procedure steps to be performed on a patient by a surgeon according to the surgical plan including, for example, bone or tissue preparation steps and/or steps for selection, modification and/or placement of implant components. Such information may include, in various examples, dimensions, shapes, angles, surface contours, and/or orientations of implant components to be selected or modified by surgeons, dimensions, shapes, angles, surface contours and/or orientations to be defined in bone or tissue by the surgeon in bone or tissue preparation steps, and/or positions, axes, planes, angle and/or entry points defining placement of implant components by the surgeon relative to patient bone or tissue. Information such as dimensions, shapes, angles, surface contours, and/or orientations of anatomical features of the patient may be derived from imaging (e.g., x-ray, CT, MRI, ultrasound or other images), direct observation, or other techniques. 
     In this disclosure, the term “mixed reality” (MR) refers to the presentation of virtual objects such that a user sees images that include both real, physical objects and virtual objects. Virtual objects may include text, 2-dimensional surfaces, 3-dimensional models, or other user-perceptible elements that are not actually present in the physical, real-world environment in which they are presented as coexisting. In addition, virtual objects described in various examples of this disclosure may include graphics, images, animations or videos, e.g., presented as 3D virtual objects or 2D virtual objects. Virtual objects may also be referred to as virtual elements. Such elements may or may not be analogs of real-world objects. In some examples, in mixed reality, a camera may capture images of the real world and modify the images to present virtual objects in the context of the real world. In such examples, the modified images may be displayed on a screen, which may be head-mounted, handheld, or otherwise viewable by a user. This type of mixed reality is increasingly common on smartphones, such as where a user can point a smartphone&#39;s camera at a sign written in a foreign language and see in the smartphone&#39;s screen a translation in the user&#39;s own language of the sign superimposed on the sign along with the rest of the scene captured by the camera. In some examples, in mixed reality, see-through (e.g., transparent) holographic lenses, which may be referred to as waveguides, may permit the user to view real-world objects, i.e., actual objects in a real-world environment, such as real anatomy, through the holographic lenses and also concurrently view virtual objects. 
     The Microsoft HOLOLENS™ headset, available from Microsoft Corporation of Redmond, Washington, is an example of a MR device that includes see-through holographic lenses, sometimes referred to as waveguides, that permit a user to view real-world objects through the lens and concurrently view projected 3D holographic objects. The Microsoft HOLOLENS™ headset, or similar waveguide-based visualization devices, are examples of an MR visualization device that may be used in accordance with some examples of this disclosure. Some holographic lenses may present holographic objects with some degree of transparency through see-through holographic lenses so that the user views real-world objects and virtual, holographic objects. In some examples, some holographic lenses may, at times, completely prevent the user from viewing real-world objects and instead may allow the user to view entirely virtual environments. The term mixed reality may also encompass scenarios where one or more users are able to perceive one or more virtual objects generated by holographic projection. In other words, “mixed reality” may encompass the case where a holographic projector generates holograms of elements that appear to a user to be present in the user&#39;s actual physical environment. 
     In some examples, in mixed reality, the positions of some or all presented virtual objects are related to positions of physical objects in the real world. For example, a virtual object may be tethered to a table in the real world, such that the user can see the virtual object when the user looks in the direction of the table but does not see the virtual object when the table is not in the user&#39;s field of view. In some examples, in mixed reality, the positions of some or all presented virtual objects are unrelated to positions of physical objects in the real world. For instance, a virtual item may always appear in the top right of the user&#39;s field of vision, regardless of where the user is looking. 
     Augmented reality (AR) is similar to MR in the presentation of both real-world and virtual elements, but AR generally refers to presentations that are mostly real, with a few virtual additions to “augment” the real-world presentation. For purposes of this disclosure, MR is considered to include AR. For example, in AR, parts of the user&#39;s physical environment that are in shadow can be selectively brightened without brightening other areas of the user&#39;s physical environment. This example is also an instance of MR in that the selectively-brightened areas may be considered virtual objects superimposed on the parts of the user&#39;s physical environment that are in shadow. 
     Furthermore, in this disclosure, the term “virtual reality” (VR) refers to an immersive artificial environment that a user experiences through sensory stimuli (such as sights and sounds) provided by a computer. Thus, in virtual reality, the user may not see any physical objects as they exist in the real world. Video games set in imaginary worlds are a common example of VR. The term “VR” also encompasses scenarios where the user is presented with a fully artificial environment in which some virtual object&#39;s locations are based on the locations of corresponding physical objects as they relate to the user. Walk-through VR attractions are examples of this type of VR. 
     The term “extended reality” (XR) is a term that encompasses a spectrum of user experiences that includes virtual reality, mixed reality, augmented reality, and other user experiences that involve the presentation of at least some perceptible elements as existing in the user&#39;s environment that are not present in the user&#39;s real-world environment. Thus, the term “extended reality” may be considered a genus for MR and VR. XR visualizations may be presented in any of the techniques for presenting mixed reality discussed elsewhere in this disclosure or presented using techniques for presenting VR, such as VR goggles. 
     These mixed reality systems and methods can be part of an intelligent surgical planning system that includes multiple subsystems that can be used to enhance surgical outcomes. In addition to the preoperative and intraoperative applications discussed above, an intelligent surgical planning system can include postoperative tools to assist with patient recovery and which can provide information that can be used to assist with and plan future surgical revisions or surgical cases for other patients. 
     Accordingly, systems and methods are also described herein that can be incorporated into an intelligent surgical planning system, such as artificial intelligence systems to assist with planning, implants with embedded sensors (e.g., smart implants) to provide postoperative feedback for use by the healthcare provider and the artificial intelligence system, and mobile applications to monitor and provide information to the patient and the healthcare provider in real-time or near real-time. 
     Visualization tools may utilize patient image data to generate three-dimensional models of bone contours to facilitate preoperative planning for joint repairs and replacements. These tools allow surgeons to design and/or select surgical guides and implant components that closely match the patient&#39;s anatomy. These tools can improve surgical outcomes by customizing a surgical plan for each patient. An example of such a visualization tool for shoulder repairs is the BLUEPRINT™ system available from Wright Medical Group, N.V. The BLUEPRINT™ system provides the surgeon with two-dimensional planar views of the bone repair region as well as a three-dimensional virtual model of the repair region. The surgeon can use the BLUEPRINT™ system to select, design or modify appropriate implant components, determine how best to position and orient the implant components and how to shape the surface of the bone to receive the components, and design, select or modify surgical guide tool(s) or instruments to carry out the surgical plan. The information generated by the BLUEPRINT™ system is compiled in a preoperative surgical plan for the patient that is stored in a database at an appropriate location (e.g., on a server in a wide area network, a local area network, or a global network) where it can be accessed by the surgeon or other care provider, including before and during the actual surgery. 
       FIG. 1  is a block diagram of an orthopedic surgical system  100  according to an example of this disclosure. Orthopedic surgical system  100  includes a set of subsystems. In the example of  FIG. 1 , the subsystems include a virtual planning system  102 , a planning support system  104 , a manufacturing and delivery system  106 , an intraoperative guidance system  108 , a medical education system  110 , a monitoring system  112 , a predictive analytics system  114 , and a communications network  116 . In other examples, orthopedic surgical system  100  may include more, fewer, or different subsystems. For example, orthopedic surgical system  100  may omit medical education system  110 , monitoring system  112 , predictive analytics system  114 , and/or other subsystems. In some examples, orthopedic surgical system  100  may be used for surgical tracking, in which case orthopedic surgical system  100  may be referred to as a surgical tracking system. In other cases, orthopedic surgical system  100  may be generally referred to as a medical device system. 
     Users of orthopedic surgical system  100  may use virtual planning system  102  to plan orthopedic surgeries. Users of orthopedic surgical system  100  may use planning support system  104  to review surgical plans generated using orthopedic surgical system  100 . Manufacturing and delivery system  106  may assist with the manufacture and delivery of items needed to perform orthopedic surgeries. Intraoperative guidance system  108  provides guidance to assist users of orthopedic surgical system  100  in performing orthopedic surgeries. Medical education system  110  may assist with the education of users, such as healthcare professionals, patients, and other types of individuals. Pre- and postoperative monitoring system  112  may assist with monitoring patients before and after the patients undergo surgery. Predictive analytics system  114  may assist healthcare professionals with various types of predictions. For example, predictive analytics system  114  may apply artificial intelligence techniques to determine a classification of a condition of an orthopedic joint, e.g., a diagnosis, determine which type of surgery to perform on a patient and/or which type of implant to be used in the procedure, determine types of items that may be needed during the surgery, and so on. 
     The subsystems of orthopedic surgical system  100  (i.e., virtual planning system  102 , planning support system  104 , manufacturing and delivery system  106 , intraoperative guidance system  108 , medical education system  110 , pre- and postoperative monitoring system  112 , and predictive analytics system  114 ) may include various systems. The systems in the subsystems of orthopedic surgical system  100  may include various types of computing systems, computing devices, including server computers, personal computers, tablet computers, smartphones, display devices, Internet of Things (IoT) devices, visualization devices (e.g., mixed reality (MR) visualization devices, virtual reality (VR) visualization devices, holographic projectors, or other devices for presenting extended reality (XR) visualizations), surgical tools, and so on. A holographic projector, in some examples, may project a hologram for general viewing by multiple users or a single user without a headset, rather than viewing only by a user wearing a headset. For example, virtual planning system  102  may include a MR visualization device and one or more server devices, planning support system  104  may include one or more personal computers and one or more server devices, and so on. A computing system is a set of one or more computing systems configured to operate as a system. In some examples, one or more devices may be shared between two or more of the subsystems of orthopedic surgical system  100 . For instance, in the previous examples, virtual planning system  102  and planning support system  104  may include the same server devices. 
     In the example of  FIG. 1 , the devices included in the subsystems of orthopedic surgical system  100  may communicate using communications network  116 . Communications network  116  may include various types of communication networks including one or more wide-area networks, such as the Internet, local area networks, and so on. In some examples, communications network  116  may include wired and/or wireless communication links. 
     Many variations of orthopedic surgical system  100  are possible in accordance with techniques of this disclosure. Such variations may include more or fewer subsystems than the version of orthopedic surgical system  100  shown in  FIG. 1 . For example,  FIG. 2  is a block diagram of an orthopedic surgical system  200  that includes one or more mixed reality (MR) systems, according to an example of this disclosure. Orthopedic surgical system  200  may be used for creating, verifying, updating, modifying and/or implementing a surgical plan. In some examples, the surgical plan can be created preoperatively, such as by using a virtual surgical planning system (e.g., the BLUEPRINT™ system), and then verified, modified, updated, and viewed intraoperatively, e.g., using MR visualization of the surgical plan. In other examples, orthopedic surgical system  200  can be used to create the surgical plan immediately prior to surgery or intraoperatively, as needed. In some examples, orthopedic surgical system  200  may be used for surgical tracking, in which case orthopedic surgical system  200  may be referred to as a surgical tracking system. In other cases, orthopedic surgical system  200  may be generally referred to as a medical device system. 
     In the example of  FIG. 2 , orthopedic surgical system  200  includes a preoperative surgical planning system  202 , a healthcare facility  204  (e.g., a surgical center or hospital), a storage system  206 , and a network  208  that allows a user at healthcare facility  204  to access stored patient information, such as medical history, image data corresponding to the damaged joint or bone and various parameters corresponding to a surgical plan that has been created preoperatively (as examples). Preoperative surgical planning system  202  may be equivalent to virtual planning system  102  of  FIG. 1  and, in some examples, may generally correspond to a virtual planning system similar or identical to the BLUEPRINT™ system. 
     In the example of  FIG. 2 , healthcare facility  204  includes a mixed reality (MR) system  212 . In some examples of this disclosure, MR system  212  includes one or more processing device(s) (P)  210  to provide functionalities that will be described in further detail below. Processing device(s)  210  may also be referred to as processor(s). In addition, one or more users of MR system  212  (e.g., a surgeon, nurse, or other care provider) can use processing device(s) (P)  210  to generate a request for a particular surgical plan or other patient information that is transmitted to storage system  206  via network  208 . In response, storage system  206  returns the requested patient information to MR system  212 . In some examples, the users can use other processing device(s) to request and receive information, such as one or more processing devices that are part of MR system  212 , but not part of any visualization device, or one or more processing devices that are part of a visualization device (e.g., visualization device  213 ) of MR system  212 , or a combination of one or more processing devices that are part of MR system  212 , but not part of any visualization device, and one or more processing devices that are part of a visualization device (e.g., visualization device  213 ) that is part of MR system  212 . 
     In some examples, multiple users can simultaneously use MR system  212 . For example, MR system  212  can be used in a spectator mode in which multiple users each use their own visualization devices so that the users can view the same information at the same time and from the same point of view. In some examples, MR system  212  may be used in a mode in which multiple users each use their own visualization devices so that the users can view the same information from different points of view. 
     In some examples, processing device(s)  210  can provide a user interface to display data and receive input from users at healthcare facility  204 . Processing device(s)  210  may be configured to control visualization device  213  to present a user interface. Furthermore, processing device(s)  210  may be configured to control visualization device  213  to present virtual images, such as 3D virtual models, 2D images, and so on. Processing device(s)  210  can include a variety of different processing or computing devices, such as servers, desktop computers, laptop computers, tablets, mobile phones and other electronic computing devices, or processors within such devices. In some examples, one or more of processing device(s)  210  can be located remote from healthcare facility  204 . In some examples, processing device(s)  210  reside within visualization device  213 . In some examples, at least one of processing device(s)  210  is external to visualization device  213 . In some examples, one or more processing device(s)  210  reside within visualization device  213  and one or more of processing device(s)  210  are external to visualization device  213 . 
     In the example of  FIG. 2 , MR system  212  also includes one or more memory or storage device(s) (M)  215  for storing data and instructions of software that can be executed by processing device(s)  210 . The instructions of software can correspond to the functionality of MR system  212  described herein. In some examples, the functionalities of a virtual surgical planning application, such as the BLUEPRINT™ system, can also be stored and executed by processing device(s)  210  in conjunction with memory storage device(s) (M)  215 . For instance, memory or storage system  215  may be configured to store data corresponding to at least a portion of a virtual surgical plan. In some examples, storage system  206  may be configured to store data corresponding to at least a portion of a virtual surgical plan. In some examples, memory or storage device(s) (M)  215  reside within visualization device  213 . In some examples, memory or storage device(s) (M)  215  are external to visualization device  213 . In some examples, memory or storage device(s) (M)  215  include a combination of one or more memory or storage devices within visualization device  213  and one or more memory or storage devices external to the visualization device. 
     Network  208  may be equivalent to network  116 . Network  208  can include one or more wide area networks, local area networks, and/or global networks (e.g., the Internet) that connect preoperative surgical planning system  202  and MR system  212  to storage system  206 . Storage system  206  can include one or more databases that can contain patient information, medical information, patient image data, and parameters that define the surgical plans. For example, medical images of the patient&#39;s diseased or damaged bone typically are generated preoperatively in preparation for an orthopedic surgical procedure. The medical images can include images of the relevant bone(s) taken along the sagittal plane and the coronal plane of the patient&#39;s body. The medical images can include X-ray images, magnetic resonance imaging (MRI) images, computerized tomography (CT) images, ultrasound images, and/or any other type of 2D or 3D image that provides information about the relevant surgical area. Storage system  206  also can include data identifying the implant components selected for a particular patient (e.g., type, size, etc.), surgical guides selected for a particular patient, and details of the surgical procedure, such as entry points, cutting planes, drilling axes, reaming depths, etc. Storage system  206  can be a cloud-based storage system (as shown) or can be located at healthcare facility  204  or at the location of preoperative surgical planning system  202  or can be part of MR system  212  or visualization device (VD)  213 , as examples. 
     MR system  212  can be used by a surgeon before (e.g., preoperatively) or during the surgical procedure (e.g., intraoperatively) to create, review, verify, update, modify and/or implement a surgical plan. In some examples, MR system  212  may also be used after the surgical procedure (e.g., postoperatively) to review the results of the surgical procedure, assess whether revisions are required, or perform other postoperative tasks. To that end, MR system  212  may include a visualization device  213  that may be worn by the surgeon and (as will be explained in further detail below) is operable to display a variety of types of information, including a 3D virtual image of the patient&#39;s diseased, damaged, or postsurgical joint and details of the surgical plan, such as a 3D virtual image of the prosthetic implant components selected for the surgical plan, 3D virtual images of entry points for positioning the prosthetic components, alignment axes and cutting planes for aligning cutting or reaming tools to shape the bone surfaces, or drilling tools to define one or more holes in the bone surfaces, in the surgical procedure to properly orient and position the prosthetic components, surgical guides and instruments and their placement on the damaged joint, and any other information that may be useful to the surgeon to implement the surgical plan. MR system  212  can generate images of this information that are perceptible to the user of the visualization device  213  before and/or during the surgical procedure. 
     In some examples, MR system  212  includes multiple visualization devices (e.g., multiple instances of visualization device  213 ) so that multiple users can simultaneously see the same images and share the same 3D scene. In some such examples, one of the visualization devices can be designated as the master device and the other visualization devices can be designated as observers or spectators. Any observer device can be re-designated as the master device at any time, as may be desired by the users of MR system  212 . 
     In this way,  FIG. 2  illustrates a surgical planning system that includes a preoperative surgical planning system  202  to generate a virtual surgical plan customized to repair an anatomy of interest of a particular patient. For example, the virtual surgical plan may include a plan for an orthopedic joint repair surgical procedure, such as one of a standard total shoulder arthroplasty or a reverse shoulder arthroplasty. In this example, details of the virtual surgical plan may include details relating to at least one of preparation of glenoid bone or preparation of humeral bone. In some examples, the orthopedic joint repair surgical procedure is one of a stemless standard total shoulder arthroplasty, a stemmed standard total shoulder arthroplasty, a stemless reverse shoulder arthroplasty, a stemmed reverse shoulder arthroplasty, an augmented glenoid standard total shoulder arthroplasty, and an augmented glenoid reverse shoulder arthroplasty. 
     The virtual surgical plan may include a 3D virtual model corresponding to the anatomy of interest of the particular patient and a 3D model of a prosthetic component matched to the particular patient to repair the anatomy of interest or selected to repair the anatomy of interest. Furthermore, in the example of  FIG. 2 , the surgical planning system includes a storage system  206  to store data corresponding to the virtual surgical plan. The surgical planning system of  FIG. 2  also includes MR system  212 , which may comprise visualization device  213 . In some examples, visualization device  213  is wearable by a user. In some examples, visualization device  213  is held by a user, or rests on a surface in a place accessible to the user. MR system  212  may be configured to present a user interface via visualization device  213 . The user interface is visually perceptible to the user using visualization device  213 . For instance, in one example, a screen of visualization device  213  may display real-world images and the user interface on a screen. In some examples, visualization device  213  may project virtual, holographic images onto see-through holographic lenses and also permit a user to see real-world objects of a real-world environment through the lenses. In other words, visualization device  213  may comprise one or more see-through holographic lenses and one or more display devices that present imagery to the user via the holographic lenses to present the user interface to the user. 
     In some examples, visualization device  213  is configured such that the user can manipulate the user interface (which is visually perceptible to the user when the user is wearing or otherwise using visualization device  213 ) to request and view details of the virtual surgical plan for the particular patient, including a 3D virtual model of the anatomy of interest (e.g., a 3D virtual bone of the anatomy of interest) and a 3D model of the prosthetic component selected to repair an anatomy of interest. In some such examples, visualization device  213  is configured such that the user can manipulate the user interface so that the user can view the virtual surgical plan intraoperatively, including (at least in some examples) the 3D virtual model of the anatomy of interest (e.g., a 3D virtual bone of the anatomy of interest). In some examples, MR system  212  can be operated in an augmented surgery mode in which the user can manipulate the user interface intraoperatively so that the user can visually perceive details of the virtual surgical plan projected in a real environment, e.g., on a real anatomy of interest of the particular patient. In this disclosure, the terms real and real world may be used in a similar manner. For example, MR system  212  may present one or more virtual objects that provide guidance for preparation of a bone surface and placement of a prosthetic implant on the bone surface. Visualization device  213  may present one or more virtual objects in a manner in which the virtual objects appear to be overlaid on an actual, real anatomical object of the patient, within a real-world environment, e.g., by displaying the virtual object(s) with actual, real-world patient anatomy viewed by the user through holographic lenses. For example, the virtual objects may be 3D virtual objects that appear to reside within the real-world environment with the actual, real anatomical object. 
       FIG. 3  is a flowchart illustrating example phases of a surgical lifecycle  300 . In the example of  FIG. 3 , surgical lifecycle  300  begins with a preoperative phase ( 302 ). During the preoperative phase, a surgical plan is developed. The preoperative phase is followed by a manufacturing and delivery phase ( 304 ). During the manufacturing and delivery phase, patient-specific items, such as parts and equipment, needed for executing the surgical plan are manufactured and delivered to a surgical site. For instance, a patient specific implant may be manufactured based on a design generated during the preoperative phase. An intraoperative phase follows the manufacturing and delivery phase ( 306 ). The surgical plan is executed during the intraoperative phase. In other words, one or more persons perform the surgery on the patient during the intraoperative phase. The intraoperative phase is followed by the postoperative phase ( 308 ). The postoperative phase includes activities occurring after the surgical plan is complete. For example, the patient may be monitored during the postoperative phase for complications. 
     As described in this disclosure, orthopedic surgical system  100  ( FIG. 1 ) may be used in one or more of preoperative phase  302 , the manufacturing and delivery phase  304 , the intraoperative phase  306 , and the postoperative phase  308 . For example, virtual planning system  102  and planning support system  104  may be used in preoperative phase  302 . Manufacturing and delivery system  106  may be used in the manufacturing and delivery phase  304 . Intraoperative guidance system  108  may be used in intraoperative phase  306 . Some of the systems of  FIG. 1  may be used in multiple phases of  FIG. 3 . For example, medical education system  110  may be used in one or more of preoperative phase  302 , intraoperative phase  306 , and postoperative phase  308 ; pre- and postoperative monitoring system  112  may be used in preoperative phase  302  and postoperative phase  308 . Predictive analytics system  114  may be used in preoperative phase  302  and postoperative phase  308 . 
     Various workflows may exist within the surgical process of  FIG. 3 . For example, different workflows within the surgical process of  FIG. 3  may be appropriate for different types of surgeries.  FIG. 4  is a flowchart illustrating preoperative, intraoperative and postoperative workflows in support of an orthopedic surgical procedure. In the example of  FIG. 4 , the surgical process begins with a medical consultation ( 400 ). During the medical consultation ( 400 ), a healthcare professional evaluates a medical condition of a patient. For instance, the healthcare professional may consult the patient with respect to the patient&#39;s symptoms. During the medical consultation ( 400 ), the healthcare professional may also discuss various treatment options with the patient. For instance, the healthcare professional may describe one or more different surgeries to address the patient&#39;s symptoms. 
     Furthermore, the example of  FIG. 4  includes a case creation step ( 402 ). In other examples, the case creation step occurs before the medical consultation step. During the case creation step, the medical professional or other user establishes an electronic case file for the patient. The electronic case file for the patient may include information related to the patient, such as data regarding the patient&#39;s symptoms, patient range of motion observations, data regarding a surgical plan for the patient, medical images of the patients, notes regarding the patient, billing information regarding the patient, and so on. 
     The example of  FIG. 4  includes a preoperative patient monitoring phase ( 404 ). During the preoperative patient monitoring phase, the patient&#39;s symptoms may be monitored. For example, the patient may be suffering from pain associated with arthritis in the patient&#39;s shoulder. In this example, the patient&#39;s symptoms may not yet rise to the level of requiring an arthroplasty to replace the patient&#39;s shoulder. However, arthritis typically worsens over time. Accordingly, the patient&#39;s symptoms may be monitored to determine whether the time has come to perform a surgery on the patient&#39;s shoulder. Observations from the preoperative patient monitoring phase may be stored in the electronic case file for the patient. In some examples, predictive analytics system  114  may be used to predict when the patient may need surgery, to predict a course of treatment to delay or avoid surgery or make other predictions with respect to the patient&#39;s health. 
     Additionally, in the example of  FIG. 4 , a medical image acquisition step occurs during the preoperative phase ( 406 ). During the image acquisition step, medical images of the patient are generated. The medical images may be generated in a variety of ways. For instance, the images may be generated using a Computed Tomography (CT) process, a Magnetic Resonance Imaging (MRI) process, an ultrasound process, or another imaging process. The medical images generated during the image acquisition step include images of an anatomy of interest of the patient. For instance, if the patient&#39;s symptoms involve the patient&#39;s shoulder, medical images of the patient&#39;s shoulder may be generated. The medical images may be added to the patient&#39;s electronic case file. Healthcare professionals may be able to use the medical images in one or more of the preoperative, intraoperative, and postoperative phases. 
     Furthermore, in the example of  FIG. 4 , an automatic processing step may occur ( 408 ). During the automatic processing step, virtual planning system  102  ( FIG. 1 ) may automatically develop a preliminary surgical plan for the patient. In some examples of this disclosure, virtual planning system  102  may use machine learning techniques to develop the preliminary surgical plan based on information in the patient&#39;s virtual case file. 
     The example of  FIG. 4  also includes a manual correction step ( 410 ). During the manual correction step, one or more human users may check and correct the determinations made during the automatic processing step. In some examples of this disclosure, one or more users may use mixed reality or virtual reality visualization devices during the manual correction step. In some examples, changes made during the manual correction step may be used as training data to refine the machine learning techniques applied by virtual planning system  102  during the automatic processing step. 
     A virtual planning step ( 412 ) may follow the manual correction step in  FIG. 4 . During the virtual planning step, a healthcare professional may develop a surgical plan for the patient. In some examples of this disclosure, one or more users may use mixed reality or virtual reality visualization devices during development of the surgical plan for the patient. As discussed in further detail below, during the virtual planning step, virtual planning system  102  may design a patient matched implant. 
     Furthermore, in the example of  FIG. 4 , intraoperative guidance may be generated ( 414 ). The intraoperative guidance may include guidance to a surgeon on how to execute the surgical plan. In some examples of this disclosure, virtual planning system  102  may generate at least part of the intraoperative guidance. In some examples, the surgeon or other user may contribute to the intraoperative guidance. 
     Additionally, in the example of  FIG. 4 , a step of selecting and manufacturing surgical items is performed ( 416 ). During the step of selecting and manufacturing surgical items, manufacturing and delivery system  106  ( FIG. 1 ) may manufacture surgical items for use during the surgery described by the surgical plan. For example, the surgical items may include surgical implants (e.g., generic and/or patient specific), surgical tools, and other items required to perform the surgery described by the surgical plan. 
     In the example of  FIG. 4 , a surgical procedure may be performed with guidance from intraoperative system  108  ( FIG. 1 ) ( 418 ). For example, a surgeon may perform the surgery while wearing a head-mounted MR visualization device of intraoperative system  108  that presents guidance information to the surgeon. The guidance information may help guide the surgeon through the surgery, providing guidance for various steps in a surgical workflow, including sequence of steps, details of individual steps, and tool or implant selection, implant placement and position, and bone surface preparation for various steps in the surgical procedure workflow. 
     Postoperative patient monitoring may occur after completion of the surgical procedure ( 420 ). During the postoperative patient monitoring step, healthcare outcomes of the patient may be monitored. Healthcare outcomes may include relief from symptoms, ranges of motion, complications, performance of implanted surgical items, and so on. Pre- and postoperative monitoring system  112  ( FIG. 1 ) may assist in the postoperative patient monitoring step. 
     The medical consultation, case creation, preoperative patient monitoring, image acquisition, automatic processing, manual correction, and virtual planning steps of  FIG. 4  are part of preoperative phase  302  of  FIG. 3 . The surgical procedures with guidance steps of  FIG. 4  is part of intraoperative phase  306  of  FIG. 3 . The postoperative patient monitoring step of  FIG. 4  is part of postoperative phase  308  of  FIG. 3 . 
     As mentioned above, one or more of the subsystems of orthopedic surgical system  100  may include one or more mixed reality (MR) systems, such as MR system  212  ( FIG. 2 ). Each MR system may include a visualization device. For instance, in the example of  FIG. 2 , MR system  212  includes visualization device  213 . In some examples, in addition to including a visualization device, an MR system may include external computing resources that support the operations of the visualization device. For instance, the visualization device of an MR system may be communicatively coupled to a computing device (e.g., a personal computer, backpack computer, smartphone, etc.) that provides the external computing resources. Alternatively, adequate computing resources may be provided on or within visualization device  213  to perform necessary functions of the visualization device. 
       FIG. 5  is a schematic representation of visualization device  213  for use in an MR system, such as MR system  212  of  FIG. 2 , according to an example of this disclosure. As shown in the example of  FIG. 5 , visualization device  213  can include a variety of electronic components found in a computing system, including one or more processor(s)  514  (e.g., microprocessors or other types of processing units) and memory  516  that may be mounted on or within a frame  518 . Furthermore, in the example of  FIG. 5 , visualization device  213  may include a transparent screen  520  that is positioned at eye level when visualization device  213  is worn by a user. In some examples, screen  520  can include one or more liquid crystal displays (LCDs) or other types of display screens on which images are perceptible to a surgeon who is wearing or otherwise using visualization device  213  via screen  520 . Other display examples include organic light emitting diode (OLED) displays. In some examples, visualization device  213  can operate to project 3D images onto the user&#39;s retinas using techniques known in the art. 
     In some examples, screen  520  may include see-through holographic lenses. sometimes referred to as waveguides, that permit a user to see real-world objects through (e.g., beyond) the lenses and also see holographic imagery projected into the lenses and onto the user&#39;s retinas by displays, such as liquid crystal on silicon (LCoS) display devices, which are sometimes referred to as light engines or projectors, operating as an example of a holographic projection system  538  within visualization device  213 . In other words, visualization device  213  may include one or more see-through holographic lenses to present virtual images to a user. Hence, in some examples, visualization device  213  can operate to project 3D images onto the user&#39;s retinas via screen  520 , e.g., formed by holographic lenses. In this manner, visualization device  213  may be configured to present a 3D virtual image to a user within a real-world view observed through screen  520 , e.g., such that the virtual image appears to form part of the real-world environment. In some examples, visualization device  213  may be a Microsoft HOLOLENS™ headset, available from Microsoft Corporation, of Redmond, Wash., USA, or a similar device, such as, for example, a similar MR visualization device that includes waveguides. The HOLOLENS™ device can be used to present 3D virtual objects via holographic lenses, or waveguides, while permitting a user to view actual objects in a real-world scene, i.e., in a real-world environment, through the holographic lenses. 
     Although the example of  FIG. 5  illustrates visualization device  213  as a head-wearable device, visualization device  213  may have other forms and form factors. For instance, in some examples, visualization device  213  may be a handheld smartphone or tablet. 
     Visualization device  213  can also generate a user interface (UI)  522  that is visible to the user, e.g., as holographic imagery projected into see-through holographic lenses as described above. For example, UI  522  can include a variety of selectable widgets  524  that allow the user to interact with a mixed reality (MR) system, such as MR system  212  of  FIG. 2 . Imagery presented by visualization device  213  may include, for example, one or more 3D virtual objects. Details of an example of UI  522  are described elsewhere in this disclosure. Visualization device  213  also can include a speaker or other sensory devices  526  that may be positioned adjacent the user&#39;s ears. Sensory devices  526  can convey audible information or other perceptible information (e.g., vibrations) to assist the user of visualization device  213 . 
     Visualization device  213  can also include a transceiver  528  to connect visualization device  213  to a processing device  510  and/or to network  208  and/or to a computing cloud, such as via a wired communication protocol or a wireless protocol, e.g., Wi-Fi, Bluetooth, etc. Visualization device  213  also includes a variety of sensors to collect sensor data, such as one or more optical camera(s)  530  (or other optical sensors) and one or more depth camera(s)  532  (or other depth sensors), mounted to, on or within frame  518 . In some examples, the optical sensor(s)  530  are operable to scan the geometry of the physical environment in which a user of MR system  212  is located (e.g., an operating room) and collect two-dimensional (2D) optical image data (either monochrome or color). Depth sensor(s)  532  are operable to provide 3D image data, such as by employing time of flight, stereo or other known or future-developed techniques for determining depth and thereby generating image data in three dimensions. Other sensors can include motion sensors  533  (e.g., Inertial Mass Unit (IMU) sensors, accelerometers, etc.) to assist with tracking movement. 
     MR system  212  processes the sensor data so that geometric, environmental, textural, or other types of landmarks (e.g., corners, edges or other lines, walls, floors, objects) in the user&#39;s environment or “scene” can be defined and movements within the scene can be detected. As an example, the various types of sensor data can be combined or fused so that the user of visualization device  213  can perceive 3D images that can be positioned, or fixed and/or moved within the scene. When a 3D image is fixed in the scene, the user can walk around the 3D image, view the 3D image from different perspectives, and manipulate the 3D image within the scene using hand gestures, voice commands, gaze line (or direction) and/or other control inputs. As another example, the sensor data can be processed so that the user can position a 3D virtual object (e.g., a bone model) on an observed physical object in the scene (e.g., a surface, the patient&#39;s real bone, etc.) and/or orient the 3D virtual object with other virtual images displayed in the scene. In some examples, the sensor data can be processed so that the user can position and fix a virtual representation of the surgical plan (or other widget, image or information) onto a surface, such as a wall of the operating room. Yet further, in some examples, the sensor data can be used to recognize surgical instruments and the position and/or location of those instruments. 
     Visualization device  213  may include one or more processors  514  and memory  516 , e.g., within frame  518  of the visualization device. In some examples, one or more external computing resources  536  process and store information, such as sensor data, instead of or in addition to in-frame processor(s)  514  and memory  516 . In this way, data processing and storage may be performed by one or more processors  514  and memory  516  within visualization device  213  and/or some of the processing and storage requirements may be offloaded from visualization device  213 . Hence, in some examples, one or more processors that control the operation of visualization device  213  may be within visualization device  213 , e.g., as processor(s)  514 . Alternatively, in some examples, at least one of the processors that controls the operation of visualization device  213  may be external to visualization device  213 , e.g., as processor(s)  210 . Likewise, operation of visualization device  213  may, in some examples, be controlled in part by a combination one or more processors  514  within the visualization device and one or more processors  210  external to visualization device  213 . 
     For instance, in some examples, when visualization device  213  is in the context of  FIG. 2 , processing of the sensor data can be performed by processing device(s)  210  in conjunction with memory or storage device(s) (M)  215 . In some examples, processor(s)  514  and memory  516  mounted to frame  518  may provide sufficient computing resources to process the sensor data collected by cameras  530 ,  532  and motion sensors  533 . In some examples, the sensor data can be processed using a Simultaneous Localization and Mapping (SLAM) algorithm, or other known or future-developed algorithms for processing and mapping 2D and 3D image data and tracking the position of visualization device  213  in the 3D scene. In some examples, image tracking may be performed using sensor processing and tracking functionality provided by the Microsoft HOLOLENS™ system, e.g., by one or more sensors and processors  514  within a visualization device  213  substantially conforming to the Microsoft HOLOLENS™ device or a similar mixed reality (MR) visualization device. 
     In some examples, MR system  212  can also include user-operated control device(s)  534  that allow the user to operate MR system  212 , use MR system  212  in spectator mode (either as master or observer), interact with UI  522  and/or otherwise provide commands or requests to processing device(s)  210  or other systems connected to network  208 . As examples, control device(s)  534  can include a microphone, a touch pad, a control panel, a motion sensor or other types of control input devices with which the user can interact. 
       FIG. 6  is a block diagram illustrating example components of visualization device  213  for use in a MR system. In the example of  FIG. 6 , visualization device  213  includes processors  514 , a power supply  600 , display device(s)  602 , speakers  604 , microphone(s)  606 , input device(s)  608 , output device(s)  610 , storage device(s)  612 , sensor(s)  614 , and communication devices  616 . In the example of  FIG. 6 , sensor(s)  616  may include depth sensor(s)  532 , optical sensor(s)  530 , motion sensor(s)  533 , and orientation sensor(s)  618 . Optical sensor(s)  530  may include cameras, such as Red-Green-Blue (RGB) video cameras, infrared cameras, or other types of sensors that form images from light. Display device(s)  602  may display imagery to present a user interface to the user. 
     Speakers  604 , in some examples, may form part of sensory devices  526  shown in  FIG. 5 . In some examples, display devices  602  may include screen  520  shown in  FIG. 5 . For example, as discussed with reference to  FIG. 5 , display device(s)  602  may include see-through holographic lenses, in combination with projectors, that permit a user to see real-world objects, in a real-world environment, through the lenses, and also see virtual 3D holographic imagery projected into the lenses and onto the user&#39;s retinas, e.g., by a holographic projection system. In this example, virtual 3D holographic objects may appear to be placed within the real-world environment. In some examples, display devices  602  include one or more display screens, such as LCD display screens, OLED display screens, and so on. The user interface may present virtual images of details of the virtual surgical plan for a particular patient. 
     In some examples, a user may interact with and control visualization device  213  in a variety of ways. For example, microphones  606 , and associated speech recognition processing circuitry or software, may recognize voice commands spoken by the user and, in response, perform any of a variety of operations, such as selection, activation, or deactivation of various functions associated with surgical planning, intra-operative guidance, or the like. As another example, one or more cameras or other optical sensors  530  of sensors  614  may detect and interpret gestures to perform operations as described above. As a further example, sensors  614  may sense gaze direction and perform various operations as described elsewhere in this disclosure. In some examples, input devices  608  may receive manual input from a user, e.g., via a handheld controller including one or more buttons, a keypad, a touchscreen, joystick, trackball, and/or other manual input media, and perform, in response to the manual user input, various operations as described above. 
     As discussed above, surgical lifecycle  300  may include a preoperative phase  302  ( FIG. 3 ). One or more users may use orthopedic surgical system  100  in preoperative phase  302 . For instance, orthopedic surgical system  100  may include virtual planning system  102  to help the one or more users generate a virtual surgical plan that may be customized to an anatomy of interest of a particular patient. As described herein, the virtual surgical plan may include a 3-dimensional virtual model that corresponds to the anatomy of interest of the particular patient and a 3-dimensional model of one or more prosthetic components matched to the particular patient to repair the anatomy of interest or selected to repair the anatomy of interest. The virtual surgical plan also may include a 3-dimensional virtual model of guidance information to guide a surgeon in performing the surgical procedure, e.g., in preparing bone surfaces or tissue and placing implantable prosthetic hardware relative to such bone surfaces or tissue. 
       FIG. 7  is a flowchart illustrating example steps in preoperative phase  302  of surgical lifecycle  300 . In other examples, preoperative phase  302  may include more, fewer, or different steps. Moreover, in other examples, one or more of the steps of  FIG. 7  may be performed in different orders. In some examples, one or more of the steps may be performed automatically within a surgical planning system such as virtual planning system  102  ( FIG. 1 ) or  202  ( FIG. 2 ). 
     In the example of  FIG. 7 , a model of the area of interest is generated ( 700 ). For example, a scan (e.g., a CT scan, MRI scan, or other type of scan) of the area of interest may be performed. For example, if the area of interest is the patient&#39;s shoulder, a scan of the patient&#39;s shoulder may be performed. Furthermore, a pathology in the area of interest may be classified ( 702 ). In some examples, the pathology of the area of interest may be classified based on the scan of the area of interest. For example, if the area of interest is the user&#39;s shoulder, a surgeon may determine what is wrong with the patient&#39;s shoulder based on the scan of the patient&#39;s shoulder and provide a shoulder classification indicating the diagnosis, e.g., such as primary glenoid humeral osteoarthritis (PGHOA), rotator cuff tear arthropathy (RCTA) instability, massive rotator cuff tear (MRCT), rheumatoid arthritis, post-traumatic arthritis, and osteoarthritis. 
     Additionally, a surgical plan may be selected based on the pathology ( 704 ). The surgical plan is a plan to address the pathology. For instance, in the example where the area of interest is the patient&#39;s shoulder, the surgical plan may be selected from an anatomical shoulder arthroplasty, a reverse shoulder arthroplasty, a post-trauma shoulder arthroplasty, or a revision to a previous shoulder arthroplasty. The surgical plan may then be tailored and/or matched to the patient ( 706 ). For instance, tailoring the surgical plan may involve designing, selecting and/or sizing surgical items needed to perform the selected surgical plan. Additionally, the surgical plan may be tailored to the patient in order to address issues specific to the patient, such as the presence of osteophytes. As described in detail elsewhere in this disclosure, one or more users may use mixed reality systems of orthopedic surgical system  100  to tailor the surgical plan to the patient. 
     The surgical plan may then be reviewed ( 708 ). For instance, a consulting surgeon may review the surgical plan before the surgical plan is executed. As described in detail elsewhere in this disclosure, one or more users may use mixed reality (MR) systems of orthopedic surgical system  100  to review the surgical plan. In some examples, a surgeon may modify the surgical plan using an MR system by interacting with a UI and displayed elements, e.g., to select a different procedure, change the sizing, shape or positioning of implants, or change the angle, depth or amount of cutting or reaming of the bone surface to accommodate an implant. 
     Additionally, in the example of  FIG. 7 , surgical items needed to execute the surgical plan may be requested ( 710 ). For instance, one or more files representing patient matched implants may be transmitted to a manufacturing system, such as manufacturing and delivery system  106  of  FIG. 1   
     As described in the following sections of this disclosure, orthopedic surgical system  100  may assist various users in performing one or more of the preoperative steps of  FIG. 7 . 
     As discussed above, in some examples, it may be desirable for a surgeon to utilize a patient matched (e.g., patient specific, custom, etc.) implant when performing an orthopedic surgical procedure. For instance, using an implant that is custom designed and manufactured for a particular patient (i.e., a patient matched implant) may enable the surgeon to minimize, or eliminate, the need to remove portions of a bone to prepare the bone to receive an implant device. Additionally, using a patient matched implant may improve fixation of an implant to bone, which may yield better patient outcomes. 
       FIG. 8  is a flowchart illustrating example steps for tailoring a surgical plan to a patient. The steps of  FIG. 8  may be considered one example of step  706  of  FIG. 7  and/or one example of step  412  of  FIG. 4 . In other examples, the technique of  FIG. 8  may include more, fewer, or different steps. Moreover, in other examples, one or more of the steps of  FIG. 8  may be performed in different orders. In some examples, one or more of the steps may be performed automatically within a surgical planning system such as virtual planning system  102  ( FIG. 1 ) or  202  ( FIG. 2 ). 
     A surgical planning system may obtain a 3D model of a bone of a patient ( 802 ). For instance, virtual planning system  102  may obtain the 3D model of the bone generated from medical images of the bone. As discussed above, the medical images may be acquired during the pre-operative phase (e.g., during step  406  of  FIG. 4 ). Virtual planning system  102  may generate the 3D model based on various features of the bone in the image. For instance, as discussed below with reference to  FIG. 9 , where the bone is a scapula, virtual planning system  102  may generate a 3D model of a glenoid of the scapula. 
     In some examples, the surgical planning system may facilitate the design of a patient matched implant to conform to a patient&#39;s bone as it exists pre-operation. In such examples, virtual planning system  102  may use an unmodified version of the 3D model of the bone. In other examples, the surgical planning system may facilitate the design of a patient matched implant to conform to a patient&#39;s bone as it will exist after one or more work steps are performed during an operation (e.g., reaming). In such examples, virtual planning system  102  may use a modified version of the 3D model of the bone that represents a shape of the bone after the planned work steps are performed. 
     The surgical planning system may identify an implant type ( 804 ). For instance, virtual planning system  102  may determine the type of implant selected during step  704  of  FIG. 7 . The determined implant type may indicate one or more of: a style (e.g., stemmed/stemless, anatomic/reversed, etc.), a manufacturer, a model, a part number, or any other identifying characteristic of the selected implant. 
     In some examples, identifying the implant type may include identifying one or more features of the identified implant. Some example features include, but are not limited to, articular surface shape, articular surface location, peripheral shape, anchorage type, anchorage location, modified vs. unmodified bone (e.g., reamed vs. un-reamed bone), etc. The surgical planning system may automatically identify, suggest, or recommend any of the features. Similarly, the surgeon may provide user input to the surgical planning system to manually select and of the features. One of more the features may be selected from a pre-defined library. For instance, the peripheral shape and/or anchorage type may be selected from a pre-defined library. Additionally or alternatively, one of more the features may be selected from a parametric shape library. For instance, the peripheral shape and/or anchorage type may be selected from a parametric shape library. 
     The surgical planning system may obtain a template model corresponding to the identified implant type ( 806 ). The template model may be a model of an implant that is used as a starting point for the generation of a patient matched implant. For instance, virtual planning system  102  may obtain, from a storage system (e.g., storage system  206  of  FIG. 2 ), a 3D model (e.g., a CAD model) of at least a portion of the identified implant type. As one specific example, where the identified implant type is a glenoid implant, virtual planning system  102  may obtain a 3D model of a baseplate of the glenoid implant. 
     The surgical planning system may generate, based on the 3D model and the template model, a patient matched implant model ( 808 ). For instance, to determine the patient matched implant model, virtual planning system  102  may determine a 3D shape bounded on one side by a surface of the 3D model of the bone and bounded on another side by a surface of the obtained template model. As one specific example, virtual planning system  102  may virtually extrude a boss from a surface of the template model (e.g., a lower surface), and remove portions of the extruded boss that overlap with the 3D model of the glenoid (e.g., perform a Boolean intersection). The combination of the determined 3D shape and the template model may represent the patient matched implant model. In some examples, as discussed in further detail below, virtual planning system  102  may generate the patient matched implant model as including one or more porous sections and one or more solid sections. 
     The surgical planning system may output the generated patient matched implant model for manufacturing ( 810 ). For instance, virtual planning system  102  may output a file containing the generated patient matched implant model to manufacturing and delivery system  106 , which may manufacture a physical patient matched implant corresponding to the patient matched implant model. As one example, manufacturing and delivery system  106  may use additive manufacturing (e.g., 3D printing) techniques (e.g., direct metal laser sintering (DMLS)) to manufacture the physical patient matched implant. Other example additive manufacturing techniques include, but are not limited to, fused deposition modeling (FDM), fused filament fabrication (FFF), and electron beam melting (EBM). 
       FIG. 9  is a flowchart illustrating example steps for obtaining a model of a bone of a patient. The steps of  FIG. 9  may be considered one example of step  802  of  FIG. 8 . In other examples, the technique of  FIG. 9  may include more, fewer, or different steps. Moreover, in other examples, one or more of the steps of  FIG. 9  may be performed in different orders. In some examples, one or more of the steps may be performed automatically within a surgical planning system such as virtual planning system  102  ( FIG. 1 ) or  202  ( FIG. 2 ). 
     A surgical planning system may obtain a 3D model of the bone generated from medical images of the bone ( 902 ). As discussed above, the medical images may be acquired during the pre-operative phase (e.g., during step  406  of  FIG. 4 ). In the example of  FIG. 10A , virtual planning system  102  may obtain 3D model  903  of a scapula of a patient, including glenoid  905 . 
     The surgical planning system may generate a mask defining an outline of an area of interest in the 3D model. For instance, virtual planning system  102  may identify anterior, posterior, superior, and inferior points of the area of interest in the 3D model ( 904 ). Virtual planning system  102  may identify the points automatically, with manual input, or a combination of automatic and manual input. In the example of  FIG. 10A , where the area of interest is a glenoid of a scapula, virtual planning system  102  may identify anterior points  952 , posterior points  954 , superior points  956 , and inferior points  958  of glenoid  905  on 3D model  903 . 
     Virtual surgical system  102  may generate anterior, posterior, superior, and inferior masks based on the identified anterior, posterior, superior, and inferior points ( 906 ). For instance, in the example of  FIG. 10B , virtual surgical system  102  may generate anterior mask  953 , posterior mask  955 , superior mask  957 , and inferior mask  959 . Collectively, the generated masks may define the outline of the area of interest in the 3D model. For instance, in the example of  FIG. 10C , generate anterior mask  953 , posterior mask  955 , superior mask  957 , and inferior mask  959  may be combined to form glenoid mask  960  that defines an outline of glenoid  905 . 
     The surgical planning system may utilize the generated mask to identify the area of interest in the 3D model ( 908 ). For instance, in the example of  FIG. 10D , virtual planning system  102  may use glenoid mask  960  of  FIG. 10C  to “mask out” (e.g., cover up, remove, etc.) portions of 3D model  903  other than glenoid  905  (i.e., the area of interest). In this way, the techniques of this disclosure enable a system to obtain a 3D model of the area of interest. 
       FIG. 11  is a flowchart illustrating example steps for generating a patient matched implant model. The steps of  FIG. 11  may be considered one example of step  808  of  FIG. 8 . In other examples, the technique of  FIG. 11  may include more, fewer, or different steps. Moreover, in other examples, one or more of the steps of  FIG. 11  may be performed in different orders. In some examples, one or more of the steps may be performed automatically within a surgical planning system such as virtual planning system  102  ( FIG. 1 ) or  202  ( FIG. 2 ). 
     The surgical planning system may obtain a baseplate final state model ( 1102 ). For instance, virtual planning system  102  may obtain, from a storage system (e.g., storage system  206  of  FIG. 2 ), a 3D model (e.g., a CAD model) of a version of a baseplate of the identified implant type. In one specific example where the identified implant type is a glenoid implant, virtual planning system  102  may obtain baseplate final state model  1103 A of  FIG. 12A . The baseplate final state model may include a surface defined as a backside. For instance, in the example of  FIG. 12A , baseplate final state model  1103 A may include backside  809 . The backside may be considered to be a surface of an implant that faces away from an articular surface of the implant. The baseplate final state model may include various additional features. For instance, in the example of  FIG. 12A , baseplate final state model  1103 A may include holes  812 A- 812 F (collectively, “holes  812 ”) (hole  812 F is not shown in  FIG. 12A  as it is obstructed by another portion of baseplate final state model  1103 A). 
     The surgical planning system may generate a patient matched augment model based on the baseplate final state model and the 3D model of the area of interest ( 1104 ). In general, a patient matched augment model may define a volume that is matched to the patient. For instance, virtual planning system  102  may determine a shape of a backside (e.g., bottom) of the baseplate final state model, and determine a volume between the shape of the backside and a surface of a bone defined by the model of the bone. The determined shape may include an outline of the backside and/or may include various features (e.g., holes  812 ). For instance, in the example of  FIG. 12B , virtual planning system  102  may determine that shape  1103 B of backside  809  of baseplate final state model  1103 A is a circle with a particular diameter (e.g., 25 mm, 29 mm, etc.) including several holes. 
     Virtual planning system  102  may determine a virtual extrusion (e.g., a boss) of the determined shape. In other words, virtual planning system  102  may extend the 2-dimensional determined shape of the backside of baseplate final state model  1103 A into the 3 rd  dimension.  FIGS. 13A-13C  are conceptual diagrams illustrating example views of a virtual extrusion for a patient matched implant design process.  FIG. 13A  illustrates a first view,  FIG. 13B  illustrates a second view that is 90 degrees offset from the first view in a first direction, and  FIG. 13C  illustrates a third view that is 90 degrees offset from the first view in a second direction that is opposite the first direction. As shown in the example of  FIGS. 13A-13C , virtual planning system  102  may determine virtual extrusion  907  (e.g., shown as a cylinder as, in this example, the outline of shape  1103 B of the backside of baseplate final state model  1103 A is a circle, however other shapes are possible). Virtual extrusion  907  may include a first face  909  and a second face  910 . In examples where the patient matched implant is a glenoid implant, first face  909  may be referred to as a medial face and second face  910  may be referred to as a lateral face. Virtual planning system  102  may create virtual extrusion  907  based on a uniform repartition of points (e.g., an even distribution of points) on the determined shape of the backside (shown in  FIG. 14A  as points  909 ). In some examples, such as where the determined shape of the backside includes one or more voids (e.g., holes for fasteners) virtual planning system  102  may generate virtual extrusion  907  to include the holes. In the example of  FIGS. 13A-13C , even though shape  1103 B includes various holes, virtual extrusion  907  is illustrated as a cylinder for simplicity. 
     Virtual planning system  102  may determine the patient matched augment model based on the virtual extrusion and the 3D model of the area of interest. For instance, to determine the patient-matched implant model, virtual planning system  102  may modify a face of virtual extrusion  907  to conform to a surface of the area of interest. As shown in the example of  FIGS. 13A-13C , virtual planning system  102  may conform first face  909  (e.g., the medial face) of virtual extrusion  907  to a surface of the 3D model of glenoid  905  (e.g., as masked out from 3D model  903  as discussed above). As one example, virtual planning system  102  may perform a Boolean intersection of points on virtual extrusion  907  and points on the 3D model of glenoid  905 . In other words, virtual planning system  102  may identify points that are within virtual extrusion  907  that are also within the 3D model of glenoid  905 . Virtual planning system  102  may remove the portion of virtual extrusion  907  that intersects the 3D model of glenoid  905  from virtual extrusion  907 , resulting in a patient-matched augment model. 
     As another example, virtual planning system  102  may compute projections of the points of the surface of the extrusion on the 3D model of the area of interest. For instance, virtual planning system  102  may determine a projection of the points on the surface of extrusion  907  and the surface of the 3D model of glenoid  905 . As shown in the example of  FIGS. 14A and 14B , virtual planning system  102  may project points  909  of virtual extrusion  907  onto the surface of glenoid  905  to obtain projected points  911 . As discussed below, the surface defined by the obtained projected points may be used to generate the patient-matched augment model. 
       FIG. 12C  is a conceptual diagram of a patient-matched augment model  1105  that may be generated based on virtual extrusion  907 . As shown in  FIG. 12C , patient-matched augment model  1105  includes surface  980  that is matched to a corresponding surface of a bone of the patient. For example, virtual planning system  102  may utilize projected points  911  to define the shape of surface  980 . Where the implant is a glenoid implant, surface  980  may be a medial surface that conforms to a glenoid of the patient. In other words, surface  980  may be complimentary to a surface of the glenoid of the patient. As discussed above, in some examples, virtual planning system  102  may generate the virtual extrusion to include one or more holes. In such examples, the determined patient matched augment model may include the one or more holes. For instance, as shown in  FIG. 12C , patient matched augment model  1105  includes holes corresponding to the holes in shape  1103 B of  FIG. 12B . 
     As discussed above, in some examples, virtual planning system  102  may generate the patient matched implant model as including one or more porous sections and one or more solid sections. When the patient matched implant model is manufactured into a physical patient matched implant, the sections defined as porous may be manufactured to be porous and the sections defined as solid may be manufactured to be solid. Including one or more porous sections in an implant may provide one or more advantages. As one example, including one or more porous sections in an implant may facilitate bony ingrowth into the implant, which may improve implant fixation. In some examples, there may be a sharp transition between solid and porous sections. In other examples, there may be a transition region between solid and porous sections with different porosity than the porous section. For instance, pores of the transition region may be smaller than pores of the porous section. Including a transition region may provide various benefits such as reduced manufacturing complexity. 
     The surgical planning system may obtain a pre-defined porous model ( 1106 ). For instance, virtual planning system  102  may obtain, from a storage system (e.g., storage system  206  of  FIG. 2 ), a 3D model (e.g., a CAD model) of a portion of the identified implant type that is to be formed of a porous structure. As one specific example, where the identified implant type is a glenoid implant, virtual planning system model  102  may obtain pre-defined porous model  1107  of  FIG. 12D . 
     The surgical planning system may generate a porous patient matched model based on the pre-defined porous model and the patient matched augment model ( 1108 ). For instance, virtual planning system  102  may add/merge (e.g., Boolean add the volumes) the patient matched augment model (e.g., the volume determined between backside  809  and the glenoid represented in the 3D model) to the pre-defined porous model to generate the porous patient matched model. In other words, virtual planning system  102  may identify points that are within the patient matched augment model and points that are within the pre-defined porous model. Virtual planning system  102  may combine the points identified within the patient matched augment model and the points identified within pre-defined porous model, resulting in a porous patient matched model (e.g., a patient matched porous model). As one specific example, virtual planning system  102  may add patient matched augment  1105  of  FIG. 12C  to pre-defined porous model  1107  of  FIG. 12D  to obtain porous patient matched model  1109 A of  FIG. 12E . 
     The surgical planning system may populate (e.g., fill) the obtained porous patient matched model with a porous structure. For instance, virtual planning system  102  may modify one or more parameters of the porous patient matched model to indicate that the volume defined by the porous patient matched model is porous. As one specific example, virtual planning system  102  may populate porous patient matched model  1109 A with a porous structure to obtain porous patient matched model  1109 B of  FIG. 12D . In some examples, the porous structure may be predefined such that virtual planning system  102  uses the same porosity for all patients (i.e., the porous structure may be generic). In some examples, the porous structure may be patient specific. For instance, virtual planning system  102  may select a particular combination of pore size and pore density based on one or more parameters of the patient (e.g., bone density, age, etc.). 
     The surgical planning system may obtain a pre-defined solid model ( 1110 ). For instance, virtual planning system  102  may obtain, from a storage system (e.g., storage system  206  of  FIG. 2 ), a 3D model (e.g., a CAD model) of a portion of the identified implant type that is to be formed of a solid structure. The pre-defined solid model may define a generic structure that is to be included in all patient matched implants of the identified implant type. As one specific example, where the identified implant type is a glenoid implant, virtual planning system model  102  may obtain pre-defined solid model  1111  of  FIG. 12G . 
     The surgical planning system may generate a mixed patient matched implant model based on the pre-defined solid model and the porous patient matched model ( 1112 ). For instance, virtual planning system  102  may add (e.g., Boolean add the volumes) the pre-defined solid model and the porous patient matched model to generate the mixed patient matched implant model. As one specific example, virtual planning system  102  may add porous patient matched model  1109 B of  FIG. 12D  to pre-defined solid model  1111  of  FIG. 12G  to obtain mixed patient matched model  1113 A of  FIG. 12H . 
     As discussed above, in some cases, the surgical planning system may generate a patient matched implant model without any porous portions. In such examples, the surgical planning system may generate the patient matched implant model by adding the patient matched augment to a pre-defined solid model. 
     The surgical planning system may generate a file that includes the mixed patient matched implant model. For instance, virtual planning system  102  may generate a “.stl” file, a CAD file, or any other type of file capable of representing the mixed patient matched implant model. Virtual planning system  102  may output the generated file for manufacturing into a physical patient matched implant. For instance, virtual planning system  102  may output the generated file to an additive manufacturing device (e.g., a 3D printer) to fabricate physical patient matched implant model  1115  of  FIG. 12I . 
     The physical patient matched implant may be manufactured based on the patient matched mixed model ( 1114 ). For instance, manufacturing and delivery system  106  may use additive manufacturing (e.g., 3D printing) techniques (e.g., direct metal laser sintering (DMLS)) to manufacture the physical patient matched implant. In some examples, manufacturing and delivery system  106  may manufacture one or more other components in addition to the physical patient matched implant. For instance, manufacturing and delivery system  106  may manufacture one or more patient matched guides (e.g., patient-matched guide  1600  of  FIG. 19 ) and/or one or more patient matched models (e.g., models of the patient&#39;s anatomy on-which a surgeon can practice before an actual implantation procedure). Where manufacturing and delivery system  106  manufactures the other components, the other components may be packaged and shipped to the surgical center along with the physical patient matched implant. 
       FIGS. 12I and 12J  are conceptual diagrams illustrating an example patient matched implant  1115 .  FIG. 12I  illustrates a side view of patient matched implant  1115  and  FIG. 12J  illustrates a top view of patient matched implant  1115 . As shown in the example of  FIG. 12I , patient matched implant  1115  may include porous portions  972  and solid portions  974 . Additionally, as shown in the example of  FIG. 12I , surface  970  (e.g., a medial surface in the context of a glenoid implant) of patient matched implant  1115  may be contoured to match a shape of a glenoid of the patient for which implant  1115  is matched. 
     In some examples, the mixed patient matched model may include components that will be removed during the manufacturing process. For instance, as shown in  FIG. 12H , mixed patient matched model  1113  may include flange  971  which may be fabricated as part of the physical patient matched implant. As shown in  FIG. 12G , flange  971  may be included in mixed patient matched model  1113  from pre-defined solid model  1111 . However, during the manufacturing process, flange  971  may be removed. For instance, the physical patient matched implant may be turned (e.g., on a lathe) to remove flange  971 . 
     The physical patient matched implant may be processed in one or more ways during or post fabrication. As one example, the physical patient matched implant may be heat treated after 3D printing, before removal of components (e.g., before removal of flange  971 ). As another example, the physical patient matched implant may be cleaned, packaged, labeled, sterilized, etc. prior to shipment to a surgical center (e.g., at which the physical patient matched implant is to be implanted into the patient). 
     In some examples, the steps of the technique of  FIG. 11  may be performed by a single device or system. For instance, the steps of the technique of  FIG. 11  may be performed by virtual planning system  102  (e.g., running the BLUEPRINT™ system available from Wright Medical Group, N.V.). In other examples, the steps of the technique of  FIG. 11  may be performed by multiple devices or systems. For instance, a first set of the steps of the technique of  FIG. 11  (e.g., steps  1102  and  1104 ) may be performed by a first device (e.g., a computer directly used by a surgeon) and a second set of the steps of the technique of  FIG. 11  (e.g., steps  1106 - 1112 ) may be performed by one or more servers (e.g., a cloud computing system). Similarly, the manufacturing process (e.g., step  1114  of  FIG. 11 ) may be performed at a manufacturing facility. 
       FIGS. 15A and 15B  are conceptual diagrams illustrating examples of patient matched implants. As shown in the example of  FIG. 15A , patient matched implant  1115 A may be a glenoid implant for a reverse shoulder arthroplasty. Patient matched implant  1115 A may include post  982 , or other anchorage, configured to be inserted into a hole made in glenoid  905  (e.g. using the techniques discussed below with reference to  FIGS. 26-37 ), and glenoid sphere  984  configured to engage a corresponding element attached to a humerus of the patient. As shown in  FIG. 15A , surface  980  of patient matched implant  1115 A may be configured to match a surface of glenoid  905 . For instance, patient matched implant  1115 A, including surface  980 , may be designed and fabricated using the techniques discussed above with reference to  FIGS. 8-14B . 
     As shown in the example of  FIG. 15B , patient matched implant  1115 A may be another example of a glenoid implant for a reverse shoulder arthroplasty. Similar to patient matched implant  1115 A, patient matched implant  1115 B includes surface  980  configured to match a surface of glenoid  905 . Patient matched implant  1115 A and patient matched implant  1115 B may be considered examples of full augment patient matched implants in that the entire contact area between the implants and the bone is “matched” to the bone. For instance, patient matched implant  1115 A and patient matched implant  1115 B may be considered examples of full augment patient matched implants because the entire area of surface  980  is matched to the contour of glenoid  905 . 
       FIGS. 16A and 16B  are conceptual diagrams illustrating examples of patient matched implants. As shown in the examples of  FIGS. 16A and 16B , patient matched implant  1115 C and patient matched implant  1115 D may be other examples of glenoid implants for a reverse shoulder arthroplasty. Similar to patient matched implants  1115 A and  1115 B, patient matched implants  1115 C and  1115 D each include surface  980  configured to match a surface of glenoid  905 . However, in contrast to patient matched implants  1115 A and  1115 B, surface  980  of patient matched implants  1115 C and  1115 D does not span the entire contact area between the implants and the bone. In particular, patient matched implants  1115 C and  1115 D both include a portion of surface  980  (i.e., portion  981 ) where surface  980  is not matched to the contour of glenoid  905 . As such, of patient matched implants  1115 C and  1115 D may be considered to be examples of partial augment patient matched implants. 
       FIGS. 17A and 17B  are conceptual diagrams illustrating examples of patient matched implants. As shown in the examples of  FIGS. 17A and 17B , implant  1117  and patient matched implant  1115 E may be other examples of glenoid implants for a reverse shoulder arthroplasty. As discussed above, in some examples, a patient matched implant may be designed and manufactured to conform to a patient bone as it exists pre-operation. Similarly, in other examples, a patient matched implant may be designed and manufactured to conform to a patient bone as it will exist after one or more work steps are performed during an operation (e.g., reaming). In the example of  FIG. 17B , patient matched implant  1115   
       FIG. 18  illustrates an example of a page of a user interface of a mixed reality system, according to an example of this disclosure, e.g. as produced for a particular patient&#39;s surgical plan. Using visualization device  213 , a user can perceive and interact with UI  522 . In the example shown in  FIG. 18 , UI  522  includes a workflow bar  1000  with selectable buttons  1002  that represent a surgical workflow, spanning various surgical procedure steps for operations on the humerus and glenoid in a shoulder arthroplasty procedure. Selection of a button  1002  can lead to display of various selectable widgets with which the user can interact, such as by using hand gestures, voice commands, gaze direction, connected lens and/or other control inputs. Selection of widgets can launch various modes of operation of MR system  212 , display information or images generated by MR system  212 , allow the user to further control and/or manipulate the information and images, lead to further selectable menus or widgets, etc. 
     The user can also organize or customize UI  522  by manipulating, moving and orienting any of the displayed widgets according to the user&#39;s preferences, such as by visualization device  213  or other device detecting gaze direction, hand gestures and/or voice commands. Further, the location of widgets that are displayed to the user can be fixed relative to the scene. Thus, as the user&#39;s gaze (i.e., eye direction) moves to view other features of the user interface  522 , other virtual images, and/or real objects physically present in the scene (e.g., the patient, an instrument set, etc.), the widgets may remain stationary and do not interfere with the user&#39;s view of the other features and objects. As yet another example, the user can control the opacity or transparency of the widgets or any other displayed images or information. The user also can navigate in any direction between the buttons  1002  on the workflow bar  1000  and can select any button  1002  at any time during use of MR system  212 . Selection and manipulation of widgets, information, images or other displayed features can be implemented based on visualization device  213  or other device detecting user gaze direction, hand motions, voice commands or any combinations thereof. 
     In the example of  FIG. 18 , UI  522  is configured for use in shoulder repair procedures and includes, as examples, buttons  1002  on workflow bar  1000  that correspond to a “Welcome” page, a “Planning” page, a “Graft” page, a “Humerus Cut” page, an “Install Guide” page, a “Glenoid Reaming” page, and a “Glenoid Implant” page. The presentation of the “Install Guide” page may be optional as, in some examples, glenoid reaming may be accomplished using virtual guidance and without the application of a glenoid guide. 
     As shown  FIG. 18 , the “Planning” page in this example of UI  522  displays various information and images corresponding to the selected surgical plan, including an image  1006  of a surgical plan file (e.g., a pdf file or other appropriate media format) that corresponds to the selected plan (including preoperative and postoperative information); a 3D virtual bone model  1008  and a 3D virtual implant model  1010  along with a 3D image navigation bar  1012  for manipulating the 3D virtual models  1008 ,  1010  (which may be referred to as 3D images); a viewer  1014  and a viewer navigation bar  1016  for viewing a multi-planar view associated with the selected surgical plan. MR system  212  may present the “Planning” page as a virtual MR object to the user during preoperative phase  302  ( FIG. 3 ). For instance, MR system  212  may present the “Planning” page to the user to help the user classify a pathology, select a surgical plan, tailor the surgical plan to the patient, revise the surgical plan, and review the surgical plan, as described in steps  702 ,  704 ,  706 , and  708  of  FIG. 7 . 
     The surgical plan image  1006  may be a compilation of preoperative (and, optionally, postoperative) patient information and the surgical plan for the patient that are stored in a database in storage system  206 . In some examples, surgical plan image  1006  can correspond to a multi-page document through which the user can browse. For example, further images of pages can display patient information, information regarding the anatomy of interest, postoperative measurements, and various 2D images of the anatomy of interest. Yet further page images can include, as examples, planning information associated with an implant selected for the patient, such as anatomy measurements and implant size, type and dimensions; planar images of the anatomy of interest; images of a 3D model showing the positioning and orientation of a surgical guide selected for the patient to assist with execution of the surgical plan; etc. 
     It should be understood that the surgical plan image  1006  can be displayed in any suitable format and arrangement and that other implementations of the systems and techniques described herein can include different information depending upon the needs of the application in which the plan image  1006  is used. 
     Referring again  FIG. 18 , the Planning page of UI  522  also may provide images of the 3D virtual bone model  1008  and the 3D model of the implant components  1010  along with navigation bar  1012  for manipulating 3D virtual models  1008 ,  1010 . For example, selection or de-selection of the icons on navigation bar  1012  allow the user to selectively view different portions of 3D virtual bone model  1008  with or without the various implant components  1010 . For example, the scapula of virtual bone model  1008  and the glenoid implant of implant model  1010  have been de-selected, leaving only the humerus bone and the humeral implant components visible. Other icons can allow the user to zoom in or out, and the user also can rotate and re-orient 3D virtual models  1008 ,  1010 , e.g., using gaze detection, hand gestures and/or voice commands. 
     The Planning page of UI  522  also provides images of 3D virtual bone model  1008  and the 3D model of the implant components  1010  along with navigation bar  1012  for manipulating 3D virtual models  1008 ,  1010 . The Planning page presented by visualization device  213  also includes multi-planar image viewer  1014  (e.g., a DICOM viewer) and navigation bar  1016  that allow the user to view patient image data and to switch between displayed slices and orientations. For example, the user can select 2D Planes icons  1026  on navigation bar  1016  so that the user can view the 2D sagittal and coronal planes of the patient&#39;s body in multi-planar image viewer  1014 . 
     Workflow bar  1000  in  FIG. 18  includes further pages that correspond to steps in the surgical workflow for a particular orthopedic procedure (here, a shoulder repair procedure). In the example of  FIG. 18 , workflow bar  1000  includes elements labeled “Graft,” “Humerus Cut,” “Install Guide,” “Glenoid Reaming,” and “Glenoid Implant” that correspond to workflow pages for steps in the surgical workflow for a shoulder repair procedure. In general, these workflow pages include information that can be useful for a health care professional during planning of or during performance of the surgical procedure, and the information presented upon selection of these pages is selected and organized in a manner that is intended to minimize disturbances or distractions to the surgeon during a procedure. Thus, the amount of displayed information is optimized and the utility of the displayed information is maximized. These workflow pages may be used as part of intraoperative phase  306  ( FIG. 3 ) to guide a surgeon, nurse or other medical technician through the steps in a surgical procedure. In some examples, these workflow pages may be used as part of preoperative phase  302  ( FIG. 3 ) to enable a user to visualize 3-dimensional models of objects involved in various steps of a surgical workflow. 
     With reference to  FIG. 19 , the Install Guide page allows the user to visualize a physical position of a patient-specific or patient-matched guide  1600 , e.g., for guidance of a drill to place a reaming guide pin in the glenoid bone, on the patient&#39;s glenoid  1602  in order to assist with the efficient and correct placement of the guide  1600  during the actual surgical procedure. Selection of items on menu  1604  can remove features from the 3D images or add other parameters of the surgical plan, such as a reaming axis  1606 , e.g., by voice commands, gaze direction and/or hand gesture selection. Placement of guide  1600  may be unnecessary for procedures in which visualization device  213  presents a virtual reaming axis or other virtual guidance, instead of a physical guide, to guide a drill for placement of a reaming guide pin in the glenoid bone. The virtual guidance or other virtual objects presented by visualization device  213  may include, for example, one or more 3D virtual objects. In some examples, the virtual guidance may include 2D virtual objects. In some examples, the virtual guidance may include a combination of 3D and 2D virtual objects. 
     With reference to  FIG. 20 , the Glenoid Implant page allows the user to visualize the orientation and placement of a glenoid implant  1700  and bone graft  1402  on glenoid  1602 . 
     It should be understood that the workflow pages illustrated and described herein are examples and that UI  522  can include fewer, more, or different pages. For example, in applications of MR system  212  for procedures involving other patient anatomies, such as the ankle, foot, knee, hip or elbow, UI  522  can include pages corresponding to the particular steps specific to the surgical workflow for those procedures. 
     The images displayed on UI  522  of MR system  212  can be viewed outside or within the surgical operating environment and, in spectator mode, can be viewed by multiple users outside and within the operating environment at the same time. In some circumstances, such as in the operating environment, the surgeon may find it useful to use a control device  534  to direct visualization device  213  such that certain information should be locked into position on a wall or other surface of the operating room, as an example, so that the information does not impede the surgeon&#39;s view during the procedure. For example, relevant surgical steps of the surgical plan can be selectively displayed and used by the surgeon or other care providers to guide the surgical procedure. 
     In various some examples, the display of surgical steps can be automatically controlled so that only the relevant steps are displayed at the appropriate times during the surgical procedure. 
     As discussed above, surgical lifecycle  300  may include an intraoperative phase  306  during which a surgical operation is performed. One or more users may use orthopedic surgical system  100  in intraoperative phase  306 . 
     In some examples, one or more users, including at least one surgeon, may use orthopedic surgical system  100  in an intraoperative setting to perform shoulder surgery.  FIG. 21  is a flowchart illustrating example stages of a shoulder joint repair surgery. As discussed above,  FIG. 21  describes an example surgical process for a shoulder surgery. The surgeon may wear or otherwise use visualization device  213  during each step of the surgical process of  FIG. 18 . In other examples, a shoulder surgery may include more, fewer, or different steps. For example, a shoulder surgery may include step for adding a bone graft, adding cement, and/or other steps. In some examples, visualization device  213  may present virtual guidance to guide the surgeon, nurse, or other users, through the steps in the surgical workflow. 
     In the example of  FIG. 21 , a surgeon performs an incision process ( 1900 ). During the incision process, the surgeon makes a series of incisions to expose a patient&#39;s shoulder joint. In some examples, an MR system (e.g., MR system  212 , MR system  1800 A, etc.) may help the surgeon perform the incision process, e.g., by displaying virtual guidance imagery illustrating how to where to make the incision. 
     Furthermore, in the example of  FIG. 21 , the surgeon may perform a humerus cut process ( 1902 ). During the humerus cut process, the surgeon may remove a portion of the humeral head of the patient&#39;s humerus. Removing the portion of the humeral head may allow the surgeon to access the patient&#39;s glenoid. Additionally, removing the portion of the humeral head may allow the surgeon to subsequently replace the portion of the humeral head with a humeral implant compatible with a glenoid implant that the surgeon plans to implant in the patient&#39;s glenoid. 
     As discussed above, the humerus preparation process may enable the surgeon to access the patient&#39;s glenoid. In the example of  FIG. 21 , after performing the humerus preparation process, the surgeon may perform a registration process that registers a virtual glenoid object with the patient&#39;s actual glenoid bone ( 1904 ) in the field of view presented to the surgeon by visualization device  213 . 
     In general terms, registration can be viewed as determining a first local reference coordinate system with respect to the 3D virtual model and determining a second local reference coordinate system with respect to the observed real anatomy. In some examples, MR system  212  also can use the optical image data collected from optical cameras  530  and/or depth cameras  532  and/or motion sensors  533  (or any other acquisition sensor) to determine a global reference coordinate system with respect to the environment (e.g., operating room) in which the user is located. In other examples, the global reference coordinate system can be defined in other manners. In some examples, depth cameras  532  are externally coupled to visualization device  213 , which may be a mixed reality headset, such as the Microsoft HOLOLENS™ headset or a similar MR visualization device. For instance, depth cameras  532  may be removable from visualization device  213 . In some examples, depth cameras  532  are part of visualization device  213 , which again may be a mixed reality headset. For instance, depth cameras  532  may be contained within an outer housing of visualization device  213 . 
     The registration process may result in generation of a transformation matrix that then allows for translation along the x, y, and z axes of the 3D virtual bone model and rotation about the x, y and z axes in order to achieve and maintain alignment between the virtual and observed bones. In some examples, after registration is complete, MR system  212  utilize the results of the registration to perform simultaneous localization and mapping (SLAM) to maintain alignment of the virtual model to the corresponding observed object. 
     Once registration is complete the surgical plan can be executed using the Augment Surgery mode of MR system  212 . For example,  FIG. 22  illustrates an image perceptible to a user when in the augment surgery mode of a mixed reality system, according to an example of this disclosure. As shown in the example of  FIG. 22 , the surgeon can visualize a virtually planned entry point  2700  and drilling axis  2702  on observed bone structure  2200  and use those virtual images to assist with positions and alignment of surgical tools. Drilling axis  2702  may also be referred to as a reaming axis and provides a virtual guide for drilling a hole in the glenoid for placement of a guide pin that will guide a reaming process. 
     The registration process may be used in conjunction with the virtual planning processes and/or intra-operative guidance described elsewhere in this disclosure. Thus, in one example, a virtual surgical plan is generated or otherwise obtained to repair an anatomy of interest of a particular patient (e.g., the shoulder joint of the particular patient). In instances where the virtual surgical plan is obtained, another computing system may generate the virtual surgical plan and an MR system (e.g., MR system  212 ) or other computing system obtains the virtual surgical plan from a computer readable medium, such as a communication medium or a non-transitory storage medium. In this example, the virtual surgical plan may include a 3D virtual model of the anatomy of interest generated based on preoperative image data and a prosthetic component selected for the particular patient to repair the anatomy of interest. Furthermore, in this example, a user may use a MR system (e.g., MR system  212 ) to implement the virtual surgical plan. In this example, as part of using the MR system, the user may request the virtual surgical plan for the particular patient. 
     Additionally, the user may view virtual images of the surgical plan projected within a real environment. For example, MR system  212  may present 3D virtual objects such that the objects appear to reside within a real environment, e.g., with real anatomy of a patient, as described in various examples of this disclosure. In this example, the virtual images of the surgical plan may include one or more of the 3D virtual model of the anatomy of interest, a 3D model of the prosthetic component, and virtual images of a surgical workflow to repair the anatomy of interest. Furthermore, in this example, the user may register the 3D virtual model with a real anatomy of interest of the particular patient. The user may then implement the virtually generated surgical plan to repair the real anatomy of interest based on the registration. In other words, in the augmented surgery mode, the user can use the visualization device to align the 3D virtual model of the anatomy of interest with the real anatomy of interest. 
     In such examples, the MR system implements a registration process whereby the 3D virtual model is aligned (e.g., optimally aligned) with the real anatomy of interest. In this example, the user may register the 3D virtual model with the real anatomy of interest without using virtual or physical markers. In other words, the 3D virtual model may be aligned (e.g., optimally aligned) with the real anatomy of interest without the use of virtual or physical markers. The MR system may use the registration to track movement of the real anatomy of interest during implementation of the virtual surgical plan on the real anatomy of interest. In some examples, the MR system may track the movement of the real anatomy of interest without the use of tracking markers. 
     In some examples, as part of registering the 3D virtual model with the real anatomy of interest, the 3D virtual model can be aligned (e.g., by the user) with the real anatomy of interest and generate a transformation matrix between the 3D virtual model and the real anatomy of interest based on the alignment. The transformation matrix provides a coordinate system for translating the virtually generated surgical plan to the real anatomy of interest. For instance, the registration process may allow the user to view steps of the virtual surgical plan projected on the real anatomy of interest. For instance, the alignment of the 3D virtual model with the real anatomy of interest may generate a transformation matrix that may allow the user to view steps of the virtual surgical plan (e.g., identification of an entry point for positioning a prosthetic implant to repair the real anatomy of interest) projected on the real anatomy of interest. 
     In some examples, the registration process (e.g., the transformation matrix generated using the registration process) may allow the user to implement the virtual surgical plan on the real anatomy of interest without use of tracking markers. In some examples, aligning the 3D virtual model with the real anatomy of interest including positioning a point of interest on a surface of the 3D virtual model at a location of a corresponding point of interest on a surface of the real anatomy of interest and adjusting an orientation of the 3D virtual model so that a virtual surface normal at the point of interest is aligned with a real surface normal at the corresponding point of interest. In some such examples, the point of interest is a center point of a glenoid. 
     With continued reference to  FIG. 21 , after performing the registration process, the surgeon may perform a reaming axis drilling process ( 1906 ). During the reaming axis drilling process, the surgeon may drill a reaming axis guide pin hole in the patient&#39;s glenoid to receive a reaming guide pin. At a later stage of the shoulder surgery, the surgeon may insert a reaming axis pin into the reaming axis guide pin hole. In some examples, an MR system (e.g., MR system  212 , MR system  1800 A, etc.) may present a virtual reaming axis to help the surgeon perform the drilling in alignment with the reaming axis and thereby place the reaming guide pin in the correct location and with the correct orientation. 
     The surgeon may perform the reaming axis drilling process in one of various ways. For example, the surgeon may perform a guide-based process to drill the reaming axis pin hole. In the case, a physical guide is placed on the glenoid to guide drilling of the reaming axis pin hole. In other examples, the surgeon may perform a guide-free process, e.g., with presentation of a virtual reaming axis that guides the surgeon to drill the reaming axis pin hole with proper alignment. An MR system (e.g., MR system  212 , MR system  1800 A, etc.) may help the surgeon perform either of these processes to drill the reaming axis pin hole. 
     Furthermore, in the surgical process of  FIG. 21 , the surgeon may perform a reaming axis pin insertion process ( 1908 ). During the reaming axis pin insertion process, the surgeon inserts a reaming axis pin into the reaming axis pin hole drilled into the patient&#39;s scapula. In some examples, an MR system (e.g., MR system  212 , MR system  1800 A, etc.) may present virtual guidance information to help the surgeon perform the reaming axis pin insertion process. 
     After performing the reaming axis insertion process, the surgeon may perform a glenoid reaming process ( 1910 ). During the glenoid reaming process, the surgeon reams the patient&#39;s glenoid. Reaming the patient&#39;s glenoid may result in an appropriate surface for installation of a glenoid implant. In some examples, to ream the patient&#39;s glenoid, the surgeon may affix a reaming bit to a surgical drill. The reaming bit defines an axial cavity along an axis of rotation of the reaming bit. The axial cavity has an inner diameter corresponding to an outer diameter of the reaming axis pin. After affixing the reaming bit to the surgical drill, the surgeon may position the reaming bit so that the reaming axis pin is in the axial cavity of the reaming bit. Thus, during the glenoid reaming process, the reaming bit may spin around the reaming axis pin. In this way, the reaming axis pin may prevent the reaming bit from wandering during the glenoid reaming process. In some examples, multiple tools may be used to ream the patient&#39;s glenoid. An MR system (e.g., MR system  212 , MR system  1800 A, etc.) may present virtual guidance to help the surgeon or other users to perform the glenoid reaming process. For example, the MR system may help a user, such as the surgeon, select a reaming bit to use in the glenoid reaming process. In some examples, the MR system present virtual guidance to help the surgeon control the depth to which the surgeon reams the user&#39;s glenoid. In some examples, the glenoid reaming process includes a paleo reaming step and a neo reaming step to ream different parts of the patient&#39;s glenoid. 
     As discussed above, in some examples, the use of a patient-matched (e.g., patient-specific) implant may reduce or eliminate the need to perform the glenoid reaming process. For instance, by using a patient-matched implant designed in accordance with the technique discussed above with reference to  FIGS. 8-17B , the surgeon can reduce or eliminate the need to perform the glenoid reaming process. 
     Additionally, in the surgical process of  FIG. 21 , the surgeon may perform a glenoid implant installation process ( 1912 ). During the glenoid implant installation process, the surgeon installs a glenoid implant in the patient&#39;s glenoid. In some instances, when the surgeon is performing an anatomical shoulder arthroplasty, the glenoid implant has a concave surface that acts as a replacement for the user&#39;s natural glenoid. In other instances, when the surgeon is performing a reverse shoulder arthroplasty, the glenoid implant has a convex surface that acts as a replacement for the user&#39;s natural humeral head. In this reverse shoulder arthroplasty, the surgeon may install a humeral implant that has a concave surface that slides over the convex surface of the glenoid implant. As in the other steps of the shoulder surgery of  FIG. 21 , an MR system (e.g., MR system  212 , MR system  1800 A, etc.) may present virtual guidance to help the surgeon perform the glenoid installation process. 
     In some examples, the glenoid implantation process includes a process to fix the glenoid implant (e.g., a patient-matched glenoid implant) to the patient&#39;s scapula ( 1914 ). In some examples, the process to fix the glenoid implant to the patient&#39;s scapula includes drilling one or more anchor holes or one or more screw holes into the patient&#39;s scapula and positioning an anchor such as one or more pegs or a keel of the implant in the anchor hole(s) and/or inserting screws through the glenoid implant and the screw holes, possibly with the use of cement or other adhesive. An MR system (e.g., MR system  212 , MR system  1800 A, etc.) may present virtual guidance to help the surgeon with the process of fixing the glenoid implant the glenoid bone, e.g., including virtual guidance indicating anchor or screw holes to be drilled or otherwise formed in the glenoid, and the placement of anchors or screws in the holes. 
     Furthermore, in the example of  FIG. 21 , the surgeon may perform a humerus preparation process ( 1916 ). During the humerus preparation process, the surgeon prepares the humerus for the installation of a humerus implant. In instances where the surgeon is performing an anatomical shoulder arthroplasty, the humerus implant may have a convex surface that acts as a replacement for the patient&#39;s natural humeral head. The convex surface of the humerus implant slides within the concave surface of the glenoid implant. In instances where the surgeon is performing a reverse shoulder arthroplasty, the humerus implant may have a concave surface and the glenoid implant has a corresponding convex surface. As described elsewhere in this disclosure, an MR system (e.g., MR system  212 , MR system  1800 A, etc.) may present virtual guidance information to help the surgeon perform the humerus preparation process. 
     Furthermore, in the example surgical process of  FIG. 21 , the surgeon may perform a humerus implant installation process ( 1918 ). During the humerus implant installation process, the surgeon installs a humerus implant on the patient&#39;s humerus. As described elsewhere in this disclosure, an MR system (e.g., MR system  212 , MR system  1800 A, etc.) may present virtual guidance to help the surgeon perform the humerus preparation process. 
     After performing the humerus implant installation process, the surgeon may perform an implant alignment process that aligns the installed glenoid implant and the installed humerus implant ( 1920 ). For example, in instances where the surgeon is performing an anatomical shoulder arthroplasty, the surgeon may nest the convex surface of the humerus implant into the concave surface of the glenoid implant. In instances where the surgeon is performing a reverse shoulder arthroplasty, the surgeon may nest the convex surface of the glenoid implant into the concave surface of the humerus implant. Subsequently, the surgeon may perform a wound closure process ( 1922 ). During the wound closure process, the surgeon may reconnect tissues severed during the incision process in order to close the wound in the patient&#39;s shoulder. 
     As discussed above with regard to step  1904 , the surgeon may perform a registration process. For a shoulder arthroplasty application, the registration process may start by virtualization device  213  presenting the user with 3D virtual bone model  1008  of the patient&#39;s scapula and glenoid that was generated from preoperative images of the patient&#39;s anatomy, e.g., by surgical planning system  102 . The user can then manipulate 3D virtual bone model  1008  in a manner that aligns and orients 3D virtual bone model  1008  with the patient&#39;s real scapula and glenoid that the user is observing in the operating environment. As such, in some examples, the MR system may receive user input to aid in the initialization and/or registration. However, discussed above, in some examples, the MR system may perform the initialization and/or registration process automatically (e.g., without receiving user input to position the 3D bone model). For other types of arthroplasty procedures, such as for the knee, hip, foot, ankle or elbow, different relevant bone structures can be displayed as virtual 3D images and aligned and oriented in a similar manner with the patient&#39;s actual, real anatomy. 
     Regardless of the particular type of joint or anatomical structure involved, selection of the augment surgery mode initiates a procedure where 3D virtual bone model  1008  is registered with an observed bone structure. In general, the registration procedure can be considered as a classical optimization problem (e.g., either minimization or maximization). For a shoulder arthroplasty procedure, known inputs to the optimization (e.g., minimization) analysis are the 3D geometry of the observed patient&#39;s bone (derived from sensor data from the visualization device  213 , including depth data from the depth camera(s)  532 ) and the geometry of the 3D virtual bone derived during the virtual surgical planning state (such as by using the BLUEPRINT™ system). Other inputs include details of the surgical plan (also derived during the virtual surgical planning stage, such as by using the BLUEPRINT™ system), such as the position and orientation of entry points, cutting planes, reaming axes and/or drilling axes, as well as reaming or drilling depths for shaping the bone structure, the type, size and shape of the prosthetic components, and the position and orientation at which the prosthetic components will be placed or, in the case of a fracture, the manner in which the bone structure will be rebuilt. 
     Upon selection of a particular patient from the welcome page of UI  522  of MR system  212  ( FIG. 5 ), the surgical planning parameters associated with that patient are connected with the patient&#39;s 3D virtual bone model  1008 , e.g., by one or more processors of visualization device  213 . In the Augment Surgery mode, registration of 3D virtual bone model  1008  (with the connected preplanning parameters) with the observed bone by visualization device  213  allows the surgeon to visualize virtual representations of the surgical planning parameters on the patient. 
     The optimization (e.g., minimization) analysis that is implemented to achieve registration of the 3D virtual bone model  1008  with the real bone generally is performed in two stages: an initialization stage and an optimization (e.g., minimization) stage. During the initialization stage, the user approximately aligns the 3D virtual bone model  1008  with the patient&#39;s real bone, such as by using gaze direction, hand gestures and/or voice commands to position and orient, or otherwise adjust, the alignment of the virtual bone with the observed real bone. The initialization stage will be described in further detail below. During the optimization (e.g., minimization) stage, which also will be described in detail below, an optimization (e.g., minimization) algorithm is executed that uses information from the optical camera(s)  530  and/or depth camera(s)  532  and/or any other acquisition sensor (e.g., motion sensors  533 ) to further improve the alignment of the 3D model with the observed anatomy of interest. In some examples, the optimization (e.g., minimization) algorithm can be a minimization algorithm, including any known or future-developed minimization algorithm, such as an Iterative Closest Point algorithm or a genetic algorithm as examples. 
     In this way, in one example, a mixed reality surgical planning method includes generating a virtual surgical plan to repair an anatomy of interest of a particular patient. The virtual surgical plan including a 3D virtual model of the anatomy of interest is generated based on preoperative image data and a prosthetic component selected for the particular patient to repair the anatomy of interest. Furthermore, in this example, the method includes using a MR visualization system to implement the virtual surgical plan. In this example, using the MR system may comprise requesting the virtual surgical plan for the particular patient. Using the MR system also comprises viewing virtual images of the surgical plan projected within a real environment. For example, visualization device  213  may be configured to present one or more 3D virtual images of details of the surgical plan that are projected within a real environment, e.g., such that the virtual image(s) appear to form part of the real environment. The virtual images of the surgical plan may include the 3D virtual model of the anatomy of interest, a 3D model of the prosthetic component, and virtual images of a surgical workflow to repair the anatomy of interest. Using the MR system may also include registering the 3D virtual model with a real anatomy of interest of the particular patient. Additionally, in this example, using the MR system may include implementing the virtually generated surgical plan to repair the real anatomy of interest based on the registration. 
     Furthermore, in some examples, the method comprises registering the 3D virtual model with the real anatomy of interest without using virtual or physical markers. The method may also comprise using the registration to track movement of the real anatomy of interest during implementation of the virtual surgical plan on the real anatomy of interest. The movement of the real anatomy of interest may be tracked without the use of tracking markers. In some instances, registering the 3D virtual model with the real anatomy of interest may comprise aligning the 3D virtual model with the real anatomy of interest and generating a transformation matrix between the 3D virtual model and the real anatomy of interest based on the alignment. The transformation matrix provides a coordinate system for translating the virtually generated surgical plan to the real anatomy of interest. In some examples, aligning may comprise virtually positioning a point of interest on a surface of the 3D virtual model within a corresponding region of interest on a surface of the real anatomy of interest; and adjusting an orientation of the 3D virtual model so that a virtual surface shape associated with the point of interest is aligned with a real surface shape associated with the corresponding region of interest. In some examples, aligning may further comprise rotating the 3D virtual model about a gaze line of the user. The region of interest may be an anatomical landmark of the anatomy of interest. The anatomy of interest may be a shoulder joint. In some examples, the anatomical landmark is a center region of a glenoid. 
     In some examples, after a registration process is complete, a tracking process can be initiated that continuously and automatically verifies the registration between 3D virtual bone model  1008  and observed bone structure  2200  during the Augment Surgery mode. During a surgery, many events can occur (e.g., patient movement, instrument movement, loss of tracking, etc.) that may disturb the registration between the 3D anatomical model and the corresponding observed patient anatomy or that may impede the ability of MR system  212  to maintain registration between the model and the observed anatomy. Therefore, by implementing a tracking feature, MR system  212  can continuously or periodically verify the registration and adjust the registration parameters as needed. If MR system  212  detects an inappropriate registration (such as patient movement that exceeds a threshold amount), the user may be asked to re-initiate the registration process. 
     In some examples, tracking can be implemented using one or more optical markers that is fixed to a particular location on the anatomy. MR system  212  monitors the optical marker(s) in order to track the position and orientation of the relevant anatomy in 3D space. If movement of the marker is detected, MR system  212  can calculate the amount of movement and then translate the registration parameters accordingly so as to maintain the alignment between the 3D model and the observed anatomy without repeating the registration process. 
     In other examples, tracking is markerless. For example, rather than using optical markers, MR system  212  implements markerless tracking based on the geometry of the observed anatomy of interest. In some examples, the markerless tracking may rely on the location of anatomical landmarks of the bone that provide well-defined anchor points for the tracking algorithm. In situations or applications in which well-defined landmarks are not available, a tracking algorithm can be implemented that uses the geometry of the visible bone shape or other anatomy. In such situations, image data from optical camera(s)  530  and/or depth cameras(s)  532  and/or motion sensors  533  (e.g., IMU sensors) can be used to derive information about the geometry and movement of the visible anatomy. An example of a tracking algorithm that can be used for markerless tracking is described in David J. Tan, et al., “6D Object Pose Estimation with Depth Images: A Seamless Approach for Robotic Interaction and Augmented Reality,” arXiv:1709.01459v1 [cs,CV] (Sept. 5, 2017), although any suitable tracking algorithm can be used. In some examples, the markerless tracking mode of MR system  212  can include a learning stage in which the tracking algorithm learns the geometry of the visible anatomy before tracking is initiated. The learning stage can enhance the performance of tracking so that tracking can be performed in real time with limited processing power. 
     As discussed elsewhere in this disclosure, orthopedic surgical procedures may involve performing various work on a patient&#39;s anatomy. Some examples of work that may be performed include, but are not necessarily limited to, cutting, drilling, reaming, screwing, adhering, and impacting. In general, it may be desirable for a practitioner (e.g., surgeon, physician&#39;s assistant, nurse, etc.) to perform the work as accurately as possible. For instance, if a surgical plan for implanting a prosthetic in a particular patient specifies that a portion of the patient&#39;s anatomy is to be reamed at a particular diameter to a particular depth, it may desirable for the surgeon to ream the portion of the patient&#39;s anatomy to as close as possible to the particular diameter and to the particular depth (e.g., to increase the likelihood that the prosthetic will fit and function as planned and thereby promote a good health outcome for the patient). 
     A visualization system, such as MR visualization system  212 , may be configured to display virtual guidance including one or more virtual guides for performing work on a portion of a patient&#39;s anatomy. For instance, the visualization system may display a virtual cutting plane overlaid on an anatomic neck of the patient&#39;s humerus. In some examples, a user such as a surgeon may view real-world objects in a real-world scene. The real-world scene may be in a real-world environment such as a surgical operating room. In this disclosure, the terms real and real-world may be used in a similar manner. The real-world objects viewed by the user in the real-world scene may include the patient&#39;s actual, real anatomy, such as an actual glenoid or humerus, exposed during surgery. The user may view the real-world objects via a see-through (e.g., transparent) screen, such as see-through holographic lenses, of a head-mounted MR visualization device, such as visualization device  213 , and also see virtual guidance such as virtual MR objects that appear to be projected on the screen or within the real-world scene, such that the MR guidance object(s) appear to be part of the real-world scene, e.g., with the virtual objects appearing to the user to be integrated with the actual, real-world scene. For example, the virtual cutting plane/line may be projected on the screen of a MR visualization device, such as visualization device  213 , such that the cutting plane is overlaid on, and appears to be placed within, an actual, observed view of the patient&#39;s actual humerus viewed by the surgeon through the transparent screen, e.g., through see-through holographic lenses. Hence, in this example, the virtual cutting plane/line may be a virtual 3D object that appears to be part of the real-world environment, along with actual, real-world objects. 
     A screen through which the surgeon views the actual, real anatomy and also observes the virtual objects, such as virtual anatomy and/or virtual surgical guidance, may include one or more see-through holographic lenses. The holographic lenses, sometimes referred to as “waveguides,” may permit the user to view real-world objects through the lenses and display projected holographic objects for viewing by the user. As discussed above, an example of a suitable head-mounted MR device for visualization device  213  is the Microsoft HOLOLENS™ headset, available from Microsoft Corporation, of Redmond, Wash., USA. The HOLOLENS™ headset includes see-through, holographic lenses, also referred to as waveguides, in which projected images are presented to a user. The HOLOLENS™ headset also includes an internal computer, cameras and sensors, and a projection system to project the holographic content via the holographic lenses for viewing by the user. In general, the Microsoft HOLOLENS™ headset or a similar MR visualization device may include, as mentioned above, LCoS display devices that project images into holographic lenses, also referred to as waveguides, e.g., via optical components that couple light from the display devices to optical waveguides. The waveguides may permit a user to view a real-world scene through the waveguides while also viewing a 3D virtual image presented to the user via the waveguides. In some examples, the waveguides may be diffraction waveguides. 
     The presentation virtual guidance such as of a virtual cutting plane may enable a surgeon to accurately resect the humeral head without the need for a mechanical guide, e.g., by guiding a saw along the virtual cutting plane displayed via the visualization system while the surgeon views the actual humeral head. In this way, a visualization system, such as MR system  212  with visualization device  213 , may enable surgeons to perform accurate work (e.g., with the accuracy of mechanical guides but without the disadvantages of using mechanical guides). This “guideless” surgery may, in some examples, provide reduced cost and complexity. 
     The visualization system (e.g., MR system  212 /visualization device  213 ) may be configured to display different types of virtual guides. Examples of virtual guides include, but are not limited to, a virtual point, a virtual axis, a virtual angle, a virtual path, a virtual plane, and a virtual surface or contour. As discussed above, the visualization system (e.g., MR system  212 /visualization device  213 ) may enable a user to directly view the patient&#39;s anatomy via a lens by which the virtual guides are displayed, e.g., projected. 
     The visualization system may obtain parameters for the virtual guides from a virtual surgical plan, such as the virtual surgical plan described herein. Example parameters for the virtual guides include, but are not necessarily limited to: guide location, guide orientation, guide type, guide color, etc. 
     The techniques of this disclosure are described below with respect to a shoulder arthroplasty surgical procedure. Examples of shoulder arthroplasties include, but are not limited to, reversed arthroplasty, augmented reverse arthroplasty, standard total shoulder arthroplasty, augmented total shoulder arthroplasty, and hemiarthroplasty. However, the techniques are not so limited, and the visualization system may be used to provide virtual guidance information, including virtual guides in any type of surgical procedure. Other example procedures in which a visualization system, such as MR system  212 , may be used to provide virtual guides include, but are not limited to, other types of orthopedic surgeries; any type of procedure with the suffix “plasty,” “stomy,” “ectomy,” “clasia,” or “centesis,”; orthopedic surgeries for other joints, such as elbow, wrist, finger, hip, knee, ankle or toe, or any other orthopedic surgical procedure in which precision guidance is desirable. 
     A typical shoulder arthroplasty includes various work on a patient&#39;s scapula and performing various work on the patient&#39;s humerus. The work on the scapula may generally be described as preparing the scapula (e.g., the glenoid cavity of the scapula) for attachment of a prosthesis and attaching the prosthesis to the prepared scapula. Similarly, the work on the humerus may generally be described as preparing the humerus for attachment of a prosthesis and attaching the prosthesis to the prepared humerus. As described herein, the visualization system may provide guidance for any or all work performed in such an arthroplasty procedure. 
     As discussed above, a MR system (e.g., MR system  212  etc.) may receive a virtual surgical plan for attaching a prosthetic to a patient and/or preparing bones, soft tissue or other anatomy of the patient to receive the prosthetic. The virtual surgical plan may specify various work to be performed and various parameters for the work to be performed. As one example, the virtual surgical plan may specify a location on the patient&#39;s glenoid for performing reaming and a depth for the reaming. As another example, the virtual surgical plan may specify a surface for resecting the patient&#39;s humeral head. As another example, the virtual surgical plan may specify locations and/or orientations of one or more anchorage locations (e.g., screws, stems, pegs, keels, etc.). 
     Many different techniques may be used to prepare a humerus for prosthesis attachment and to perform actual prosthesis attachment. Regardless of the technique used, MR system  212  may provide virtual guidance to assist in one or both of the preparation and attachment. As such, while the following techniques are examples in which MR system  212  provides virtual guidance, MR system  212  may provide virtual guidance for other techniques. 
     In an example technique, the work steps include resection of a humeral head, creating a pilot hole, sounding, punching, compacting, surface preparation, with respect to the humerus, and attaching an implant to the humerus. Additionally, in some techniques, the work steps may include bone graft work steps, such as installation of a guide in a humeral head, reaming of the graft, drilling the graft, cutting the graft, and removing the graft, e.g., for placement with an implant for augmentation of the implant relative to a bone surface such as the glenoid. 
     A surgeon may perform one or more steps to expose a patient&#39;s humerus. For instance, the surgeon may make one or more incisions to expose the upper portion of the humerus including the humeral head. The surgeon may position one or more retractors to maintain the exposure. In some examples, MR system  212  may provide guidance to assist in the exposure of the humerus, e.g., by making incisions, and/or placement of retractors. 
     Many different techniques may be used to prepare a scapula for prosthesis attachment and to perform actual prosthesis attachment. Regardless of the technique used, MR system  212  may provide virtual guidance to assist in one or both of the preparation and attachment. As such, while the following techniques are examples in which MR system  212  provides virtual guidance, MR system  212  may provide virtual guidance for other techniques. 
     In an example technique, the surgical procedure steps include installation of a guide in a glenoid of the scapula, reaming the glenoid, creating a central hole in the glenoid, creating additional anchorage positions in the glenoid, and attaching an implant to the prepared glenoid. As a guide pin is used, the example technique may be considered a cannulated technique. However, the techniques are similarly applicable to non-cannulated techniques. 
     A surgeon may perform one or more steps to expose a patient&#39;s glenoid. For instance, with the patient&#39;s arm abducted and internally rotated, the surgeon may make one or more incisions to expose the glenoid. The surgeon may position one or more retractors to maintain the exposure. In some examples, MR system  212  may provide guidance to assist in the exposure and/or placement of retractors. 
       FIG. 23  is a conceptual diagram illustrating an MR system providing virtual guidance to a user for installation of a guide in a glenoid of a scapula, in accordance with one or more techniques of this disclosure. As shown in  FIG. 23 , MR system  212  may display virtual guidance, e.g., in the form of virtual axis  5104 , on glenoid  5102  of scapula  5100 . To display virtual axis  5104 , MR system  212  may determine a location on a virtual model of glenoid  5102  at which a guide is to be installed. MR system  212  may obtain the location from a virtual surgical plan (e.g., the virtual surgical plan described above). The location obtained by MR system  212  may specify one or both of coordinates of a point on the virtual model and a vector. The point may be the position at which the guide is to be installed and the vector may indicate the angle/slope at which the guide is to be installed. 
     As discussed above, the virtual model of glenoid  5102  may be registered with glenoid  5102  such that coordinates on the virtual model approximately correspond to coordinates on glenoid  5102 . As such, by displaying virtual axis  5104  at the determined location on the virtual model, MR system  212  may display virtual axis  5104  at the planned position on glenoid  5102 . 
     As also discussed above, the virtual model of glenoid  5102  may be selectively displayed after registration. For instance, after the virtual model of glenoid  5102  is registered with glenoid  5102 , MR system  212  may cease displaying of the virtual model. Alternatively, MR system  212  may continue to display the virtual model overlaid on glenoid  5102  after registration. The display of the virtual model may be selective in that the surgeon may activate or deactivate display of the virtual model. 
     MR system  212  may display the virtual model and/or virtual guides with varying opacity (e.g., transparency). The opacity may be adjusted automatically, manually, or both. As one example, the surgeon may provide user input to MR system  212  to manually adjust the opacity of the virtual model and/or virtual guides. As another example, MR system  212  may automatically adjust the opacity based on an amount of light in the viewing field (e.g., amount of light where the surgeon is looking). For instance, MR system  212  may adjust the opacity (e.g., increase the transparency) of the virtual model and/or virtual guides to positively correlate with the amount of light in the viewing field (e.g., brighter light results in increased opacity/decreased transparency and dimmer light results in decreased opacity/increased transparency). 
     The surgeon may attach a physical guide using the displayed virtual guidance. As one example, where the guide is a guide pin with a self-tapping threaded distal tip, the surgeon may align the guide pin with the displayed virtual axis  5104  and utilize a drill or other instrument to install the guide pin. As another example, where the guide is a guide pin without a self-tapping tip, the surgeon may align a drill bit of a drill with the displayed virtual axis  5104  and operate the drill to form a hole to receive the guide pin and then install the guide pin in the hole. In some examples, MR system  212  may display depth guidance information to enable the surgeon to install the guide pin to a planned depth. Examples of depth guidance information are discussed in further detail herein with reference to  FIG. 66 . 
       FIG. 24  is a conceptual diagram illustrating guide  5200 , i.e., a guide pin in this example, as installed in glenoid  5102 . As shown in  FIGS. 51 and 52 , by displaying virtual axis  5104 , a surgeon may drill in alignment with the virtual axis, which may be referred to as a reaming axis, and thereby form a hole for installation of guide  5200  at the planned position on glenoid  5102 . In this way, MR system  212  may enable the installation of a guide without the need for an additional mechanical guide. 
       FIG. 25  is a conceptual diagram illustrating an MR system providing virtual guidance for reaming a glenoid, in accordance with one or more techniques of this disclosure. As shown in  FIG. 25 , reaming tool  5300  may be used to ream the surface of glenoid  5102 . In this example, reaming tool  5300  may be a cannulated reaming tool configured to be positioned and/or guided by a guide pin, such as guide  5200 . For example, the shaft of cannulated reaming tool may receive guide  5200  such that the tool shaft is mounted substantially concentrically with the pin. In other examples, reaming tool  5300  may not be cannulated and may be guided without the assistance of a physical guide pin. 
     The surgeon may attach reaming tool  5300  to guide  5200  (e.g., insert proximal tip of guide  5200  into reaming tool  5300 ), and attach a drill or other instrument to rotate reaming tool  5300 . To perform the reaming, the surgeon may rotate reaming tool  5300  to advance reaming tool  5300  down guide  5200  until reaming is complete. 
     As discussed above, in some examples, the techniques of this disclosure may reduce or eliminate the need to perform reaming of the glenoid. In particular, by using a patient matched glenoid implant (i.e., an implant with a surface shaped to conform to a patient&#39;s glenoid), a surgeon may avoid (or reduce) the need to perform reaming of the glenoid. 
     MR system  212  may display virtual guidance to assist in the reaming process. As one example MR system  212  may provide depth guidance. For instance, MR system  212  may display depth guidance to enable the surgeon to ream to a target depth. As another example, MR system  212  may provide targeting guidance. For instance, MR system  212  may display an indication of whether reaming tool  5300  is aligned with a virtual reaming axis. 
     While described herein as a single reaming step, the surgery may include multiple reaming steps. The various reaming steps may use the same axis/guide pin or may use different axes/guide pins. In examples where different reaming steps use different axes, MR system  212  may provide virtual guidance for reaming using the different axes. 
       FIGS. 26 and 27  are conceptual diagrams illustrating an MR system providing virtual guidance for creating a central hole in a glenoid, in accordance with one or more techniques of this disclosure. As shown in  FIGS. 26 and 27 , drill bit  5400  may be used to drill central hole  5500  in glenoid  5102 . In this example, drill bit  5400  may be a cannulated drill bit configured to be positioned and/or guided by a guide pin, such as guide  5200 . In other examples, drill bit  5400  may not be cannulated and may be guided without the assistance of a physical guide pin. For instance, MR system  212  may provide virtual guidance to enable a surgeon to drill glenoid  5102  without the use of guide  5200 . As discussed in further detail below, central hole  5500  may facilitate the attachment of a prosthesis to glenoid  5102 , e.g., via one or more anchors. 
     MR system  212  may display virtual guidance to assist in the creation of central hole  5500 . For instance, MR system  212  may display depth guidance to enable the surgeon to drill central hole  5500  to a target depth. As another example, MR system  212  may provide targeting guidance. For instance, MR system  212  may display an indication of whether drill bit tool  5400  is on a prescribed axis selected to form the central hole  5500  at a proper position at with a proper orientation. 
     In addition to a central hole (e.g., central hole  5500 ), it may be desirable for the surgeon to create additional anchorage positions in the glenoid. This additional anchorage positions may improve the fixation between the prosthesis and the glenoid. For instance, the additional anchorage positions may provide anti-rotation support between the prosthesis and the glenoid. Several different styles of anchorage may be used, depending on the type of prosthesis to be installed. Some examples of anchorage include, but are not necessarily limited to, keel and pegged anchors. However, the virtual guidance techniques discussed herein may be applicable to any type of anchorage. Example MR guidance for keel type anchorage is discussed below with reference to  FIGS. 26-29 . Example MR guidance for pegged type anchorage is discussed below with reference to  FIGS. 30-32 . In each case, the anchorage may help in placing a glenoid implant, such as a glenoid base plate for anatomic arthroplasty or a glenoid base plate and glenosphere for reverse arthroplasty, onto the glenoid and fixing it in place. 
       FIG. 28  is a conceptual diagram illustrating a glenoid prosthesis with keel type anchorage. As shown in  FIG. 28 , glenoid prosthesis  5600  includes rear surface  5602  configured to engage a prepared surface of glenoid  5102  (e.g., a reamed surface), and a keel anchor  5604  configured to be inserted in a keel slot created in glenoid  5102  (e.g., keel slot  5902  of  FIG. 31 ). 
     In some examples, glenoid prosthesis  5600  may be a patient matched glenoid implant. For instance, at least a portion of rear surface  5602  may be contoured to match a surface of glenoid  5102  using the techniques discussed above with reference to  FIGS. 8-17B . 
       FIGS. 29-31  are conceptual diagrams illustrating an MR system providing virtual guidance for creating keel type anchorage positions in a glenoid, in accordance with one or more techniques of this disclosure. As shown in  FIG. 29 , MR system  212  may provide virtual guidance for drilling additional holes in glenoid  5102 . MR system  212  may provide the virtual guidance for drilling the additional holes in any of a variety of manners. As one example, MR system  212  may display virtual guidance such as virtual markers having specified shapes (e.g., axes, arrows, points, circles, X shapes, crosses, targets, etc.), sizes and/or colors, at the locations the additional holes are to be drilled. For instance, in the example of  FIG. 29 , MR system  212  may display virtual markers  5700 A and  5700 B at the locations the additional holes are to be drilled. As another example, MR system  212  may display virtual axes at the locations the additional holes are to be drilled to aid the surgeon in properly aligning a drill bit to make the holes in the glenoid bone. 
     MR system  212  may determine the locations of the additional holes based on the virtual surgical plan. For instance, similar to virtual axis  5104  of  FIG. 23 , MR system  212  may obtain, from the virtual surgical plan, the location(s) of the additional holes to be drilled on the virtual model of glenoid  5102 . As such, by displaying virtual markers  5700 A and  5700 B at the determined locations on the virtual model, MR system  212  may display virtual markers  5700 A and  5700 B at the planned positions on glenoid  5102 . As discussed above, the virtual surgical plan may be patient specific in that the plan may be specifically developed for a particular patient. As such, the planned positioned on glenoid  5102  at which MR system  212  displays virtual markers  5700 A and  5700 B may be considered patient-specific planned positions. Therefore, the locations of the planned positions will vary from patient to patient according to individual patient-specific surgical plans. 
     The surgeon may utilize a drill bit and a drill to create the additional hole(s) at the location(s) indicated by MR system  212 . For instance, as shown in  FIG. 30 , the surgeon may drill hole  5800 A at the location of virtual marker  5700 A and drill hole  5800 B at the location of virtual marker  5700 B. The surgeon may use the same drill bit for each hole or may use different drill bits for different holes. 
     MR system  212  may provide virtual guidance for the drilling in addition to or in place of the virtual markers, such as those described above, which indicate the locations the additional holes are to be drilled. As one example, MR system  212  may provide targeting guidance to indicate whether the drill is on a target axis. In this case, as an addition or alternative to the virtual markers, MR system  212  may display guide axes that extend outward from the locations of each of the respective holes to be drilled. As another example, MR system  212  may display a mask with holes in the mask that correspond to the locations at which the holes are to be drilled. As another example, MR system  212  may display depth guidance to enable the surgeon to drill holes  5800 A and  5800 B to target depths. 
     MR system  212  may provide virtual guidance for working the holes into a keel slot that may accept keel anchor  5604  of glenoid prosthesis  5600 . As an example, MR system  212  may display virtual outline  5802  around holes  5800 A,  5500 , and  5800 B. For instance, MR system  212  may display virtual outline  5802  as approximately corresponding to a final outline of the desired keel slot to be created. 
     The surgeon may utilize a tool to work holes  5800 A,  5500 , and  5800 B into keel slot  5902 . As shown in  FIG. 29 , the surgeon may utilize keel punch  5900  to work holes  5800 A,  5500 , and  5800 B into keel slot  5902 . For instance, the surgeon may impact keel punch  5900  into the area indicated by virtual outline  5802 . In this case, virtual outline  5802  defines a shape and dimension of the desired keel slot  5902 , permitting the surgeon to work the holes into a form that visually matches or approximates the displayed virtual outline of the keel slot. 
     MR system  212  may provide additional or alternative virtual guidance for creating keel slot  5902 . As one example, MR system  212  may display depth guidance to enable the surgeon to impact keel punch  5900  to a target depth. As another example, MR system  212  may provide targeting guidance to indicate whether keel punch  5900  is on a target axis. As another example, MR system  212  may display a mask with a cutout for virtual outline  5802 . 
       FIG. 32  is a conceptual diagram illustrating a glenoid prosthesis with pegged type anchorage. As shown in  FIG. 32 , glenoid prosthesis  6000  includes rear surface  6002  configured to engage a prepared surface of glenoid  5102  (e.g., a reamed surface), a central peg anchor  6004  configured to be inserted in a central hole created in glenoid  5102 , and one or more peg anchors  6006 A- 6006 C (collectively, “peg anchors  6006 ”) respectively configured to be inserted in additional holes created in glenoid  5102 . 
     In some examples, glenoid prosthesis  6000  may be a patient matched glenoid implant. For instance, at least a portion of rear surface  6002  may be contoured to match a surface of glenoid  5102  using the techniques discussed above with reference to  FIGS. 8-17B . 
       FIGS. 33 and 34  are conceptual diagrams illustrating an MR system providing virtual guidance for creating pegged type anchorage positions in a glenoid, in accordance with one or more techniques of this disclosure. As shown in  FIG. 33 , MR system  212  may provide virtual guidance for drilling additional holes in glenoid  5102 . MR system  212  may provide the virtual guidance for drilling the additional holes in any of a variety of manners. As one example, MR system  212  may display virtual markers (e.g., axes, points, circles, X shapes, etc.) at the locations the additional holes are to be drilled. For instance, in the example of  FIG. 33 , MR system  212  may display virtual markers  5700 A- 5700 C at the locations the additional holes are to be drilled. As another example, MR system  212  may display virtual axes extending from the locations at which the additional holes are to be drilled. As another example, MR system  212  may display a mask (effectively an inverse of the virtual markers) that indicates where the holes are to be drilled. 
     MR system  212  may determine the locations of the additional holes based on the virtual surgical plan. For instance, similar to virtual axis  5104  of  FIG. 23 , MR system  212  may obtain, from the virtual surgical plan, which may be patient-specific, the location(s) of the additional holes to be drilled on the virtual model of glenoid  5102 . As such, by displaying virtual markers  5700 A- 5700 C at the determined locations on the virtual model, MR system  212  may display virtual markers  5700 A- 5700 C at the planned positions on glenoid  5102 . 
     The surgeon may utilize a drill bit (or multiple drill bits) and a drill to create the additional hole(s) at the location(s) indicated by MR system  212 . For instance, as shown in  FIG. 34 , the surgeon may drill hole  5800 A at the location of virtual marker  5700 A, drill hole  5800 B at the location of virtual marker  5700 B, and drill hole  5800 C at the location of virtual marker  5700 C. 
     MR system  212  may provide virtual guidance for the drilling in addition to or in place of the virtual markers that indicate the locations the additional holes are to be drilled. As one example, MR system  212  may provide targeting guidance to indicate whether the drill is on a target axis. As another example, MR system  212  may display depth guidance to enable the surgeon to drill holes  5800 A- 5800 C to target depths. 
     It is noted that different implants may have different profiles, such as augmented profiles. Additionally, as discussed herein, some implants may be implanted with additional materials harvested from the patient, such as bone grafts. In some of such examples, MR system  212  may provide virtual guidance for placement of the additional materials. For instance, MR system  212  may provide virtual guidance for attaching a bone graft to an implant and guidance for attaching the graft/implant assembly to the patient. 
     In some examples, regardless of the anchorage type being used, the surgeon may utilize a trial component to determine whether glenoid  5102  has been properly prepared. The trial component may have a rear surface and anchors sized and positioned identical to the rear surface and anchors of the prosthesis to be implanted. 
       FIG. 35  is a conceptual diagram illustrating an MR system providing virtual guidance for attaching an implant to a glenoid, in accordance with one or more techniques of this disclosure. A tool may be used to attach the implant (e.g., a pegged implant, a keeled implant, or any other type of implant) to glenoid  5102 . For instance, the surgeon may utilize impactor  6302  to insert prosthesis  6300  into the prepared glenoid  5102 . In some examples, one or more adhesives (e.g., glue, cement, etc.) may be applied to prosthesis  6300  and/or glenoid  5102  prior to impaction. 
     In some examples, one or more fasteners may be used to attach a prosthesis to glenoid  5102 . For instance, as shown in  FIGS. 36 and 37 , screws  6400 A- 6400 D (collectively, “screws  6400 ”) and central stem  6402  may be used to attach prosthesis  6300  to glenoid  5102 . These fasteners may be used in addition to, or in place of, any anchorages included in the prosthesis (e.g., pegs, keels, etc.). 
     MR system  212  may provide virtual guidance to facilitate the installation of the additional fasteners. For instance, as shown in  FIG. 35 , MR system  212  may display virtual axes  6500 A- 6500 D (collectively, “virtual axes  6500 ”), which may be referred to as “virtual screw axes,” to guide the surgeon in the installation of screws  6400 . In examples where screws  6400  are not “self-tapping”, MR system  212  may display virtual guidance (e.g., virtual axes) to guide drilling of pilot holes for screws  6400 . For instance, MR system  212  may display a virtual drilling axis obtained from the virtual surgical plan that guides drilling of a pilot hole for a screw of screws  6400 . 
     To display the virtual guides for installation of the fasteners, MR system  212  may register a virtual model of the prosthesis to the actual observed prosthesis. For instance, MR system  212  may obtain a virtual model of prosthesis  6300  from the virtual surgical plan and perform the registration in a manner similar to the registration process described. 
     MR system  212  may obtain locations for each of the fasteners to be installed. For instance, MR system  212  may obtain, from the virtual surgical plan, coordinates on the virtual model of the prosthesis and vector for each of the fasteners. In some examples, MR system  212  may determine that the coordinates for each fastener are the centroid of a corresponding hole in the prosthesis. For instance, MR system  212  may determine that the coordinates for screw  6400 A are the centroid of hole  6502 . 
     The surgeon may install the fasteners using the displayed virtual guidance. For instance, the surgeon may use a screwdriver or other instrument to install screws  6400 . 
     MR system  212  may display virtual guidance to assist in the fastener attachment. As one example MR system  212  may provide depth guidance. For instance, MR system  212  may display depth guidance to enable the surgeon to install each of screws  6400  to a target depth. As another example, MR system  212  may provide targeting guidance. For instance, MR system  212  may display an indication of whether each of screws  6400  is being installed on a prescribed axis. As another example, MR system  212  may provide guidance on an order in which to tighten screws  6400 . For instance, MR system  212  may display a virtual marker on a particular screw of screws  6400  that is to be tightened. 
     As discussed above, MR system  212  may provide a wide variety of virtual guidance. Example of virtual guidance that may be provided by MR system  212  include, but are not limited to, targeting guidance and depth guidance. MR system  212  may provide targeting guidance to assist a surgeon in performing work (e.g., drilling a hole, reaming, installing a screw, etc.) along a particular axis. MR system  212  may provide depth guidance to assist a surgeon in performing work (e.g., drilling a hole, reaming, installing a screw, etc.) to a desired depth. 
     While the techniques been disclosed with respect to a limited number of examples, those skilled in the art, having the benefit of this disclosure, will appreciate numerous modifications and variations there from. For instance, it is contemplated that any reasonable combination of the described examples may be performed. It is intended that the appended claims cover such modifications and variations as fall within the true spirit and scope of the invention. 
     It is to be recognized that depending on the example, certain acts or events of any of the techniques described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially. 
     In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium. 
     By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. 
     Operations described in this disclosure may be performed by one or more processors, which may be implemented as fixed-function processing circuits, programmable circuits, or combinations thereof, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Fixed-function circuits refer to circuits that provide particular functionality and are preset on the operations that can be performed. Programmable circuits refer to circuits that can programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For instance, programmable circuits may execute instructions specified by software or firmware that cause the programmable circuits to operate in the manner defined by instructions of the software or firmware. Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function circuits perform are generally immutable. Accordingly, the terms “processor” and “processing circuity,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. 
     Various examples have been described. These and other examples are within the scope of the following claims.