Patent Description:
The present disclosure relates to computer assisted navigation of equipment and operations during surgery.

Surgical operating rooms can contain a diverse range of medical equipment, which can include computer assisted surgical navigation systems, medical imaging devices (e.g., computerized tomography (CT) scanners, fluoroscopy imaging, etc.), surgical robots, etc..

A computer assisted surgical navigation system can provide a surgeon with computerized visualization of the present pose of a surgical tool relative to medical images of a patient's anatomy. Camera tracking systems for computer assisted surgical navigation typically use a set of cameras to track pose of a reference array on a surgical tool, which is being positioned by a surgeon during surgery, relative to a patient reference array (also "dynamic reference base" (DRB)) attached to a patient. The reference arrays allow the camera tracking system to determine a pose of the surgical tool relative to anatomical structure imaged by a medical image of the patient and relative to the patient. The surgeon can thereby use real-time visual feedback of the pose to navigate the surgical tool during a surgical procedure on the patient.

Many surgical workflows using computer assisted surgical navigation systems require image scans, such as CT scans or magnetic resonance imaging scans, during the surgical procedure. Perpendicular scan slices (axial, sagittal, and coronal) may be used to enable operators to visualize the patient's anatomy alongside the relative poses of surgical instruments. The surgical workflows may be challenging for surgeons and other surgical team members to recall, interpret, and follow under the time constraints and other pressures of a surgery environment. Improved surgical workflows and computer implemented operations to reduce the workload on the surgery team and to ensure processes and best practices are followed.

<CIT> discloses a surgical implant planning computer for intra-operative CT workflow, preoperative CT imaging workflow, and fluoroscopic imaging workflow. A network interface is connectable to a CT image scanner and a robot surgical platform having a robot base coupled to a robot arm that is movable by motors. A CT image of a bone is received from the CT image scanner and displayed. A user's selection is received of a surgical screw from among a set of defined surgical screws. A graphical screw representing the selected surgical screw is displayed as an overlay on the CT image of the bone. Angular orientation and location of the displayed graphical screw relative to the bone in the CT image is controlled responsive to receipt of user inputs. An indication of the selected surgical screw and an angular orientation and a location of the displayed graphical screw are stored in a surgical plan data structure.

The present invention is defined by appended claim <NUM>. Specific embodiments are set forth in the dependent claims.

Some embodiments of the present disclosure are directed to a surgical system for computer assisted navigation during surgery. The surgical system includes at least one processor that is operative to obtain a three-dimensional (3D) radiological representation of a targeted anatomical structure of a patient and a set of fiducials of a registration fixture. The operations attempt to register locations of the set of fiducials in the 3D radiological representation to a 3D imaging space tracked by a camera tracking system. Based on determining one of the fiducials of the set has a location that was not successfully registered to the 3D imaging space, the operations display at least one view of the 3D radiological representation with a graphical overlay indicating the fiducial has not been successfully registered to the 3D imaging space, receive user-supplied location information identifying where the fiducial is located in the 3D radiological representation, and register the location of the fiducial to the 3D imaging space based on the user-supplied location information.

In some further embodiments, the operation to attempt to register locations of the set of fiducials in the 3D radiological representation to the 3D imaging space tracked by the camera tracking system, includes to obtain, from at least one camera of the camera tracking system, an optical image of a reference array fixated to the patient. The reference array includes a set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space. The operations attempt to register locations of a pattern of the set of optical markers to locations of a pattern of the set of fiducials in the 3D radiological representation, and identify any of the optical markers of the reference array that are not successfully registered to any of the fiducials of the registration fixture.

In some further embodiments, the operations include to display a virtual implant device as an overlay on a view of the 3D radiological representation of the targeted anatomical structure. The operations display a graphical indication of a trajectory of the virtual implant device representing an implantation trajectory of the virtual implant device into the targeted anatomical structure. The operations update pose of the graphical indication of the trajectory of the virtual implant device displayed in the view of the 3D radiological representation, to track steering inputs received through a user interface of the surgical system. The operations store as a planned trajectory of the virtual implant device, a user-designated one of the poses of the graphical indication of the trajectory.

In some further embodiments, prior to obtaining the 3D radiological representation of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture, the operations include to obtain, from at least one camera of the camera tracking system, optical images of a reference array fixated to the patient and of a registration fixture attached to a radiological imaging device. The reference array includes a first set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space, and the registration fixture includes a second set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space. The operations obtain a fluoroscopic image of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture. The operations determine whether a first condition is satisfied based on a defined number of the optical markers in the first set being detected by the at least one camera of the camera tracking system in the 3D imaging space. The operations determine whether a second condition is satisfied based on a defined number of the optical markers in the second set being detected by the at least one camera of the camera tracking system in the 3D imaging space. The operations determine whether a third condition is satisfied based on a defined number of the set of fiducials of the registration fixture being visible in the fluoroscopic image. When one of the first, second, and third conditions is not satisfied, the operations display an indication of the not satisfied one of the first, second, and third conditions, and inhibit capture by a radiological imaging process of the 3D radiological representation of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture. In contrast, when each of the first, second, and third conditions are satisfied, the operations enable capture by the radiological imaging process of the 3D radiological representation of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture.

The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in a constitute a part of this application, illustrate certain non-limiting embodiments of inventive concepts. In the drawings:.

The following discussion is presented to enable a person skilled in the art to make and use embodiments of the present disclosure. Various modifications to the illustrated embodiments will be readily apparent to those skilled in the art, and the principles herein can be applied to other embodiments and applications without departing from embodiments of the present disclosure. Thus, the embodiments are not intended to be limited to embodiments shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein. The following detailed description is to be read with reference to the figures, in which like elements in different figures have like reference numerals. The figures, which are not necessarily to scale, depict selected embodiments and are not intended to limit the scope of the embodiments. Skilled artisans will recognize the examples provided herein have many useful alternatives and fall within the scope of the embodiments.

The ExcelsiusHub, by Globus Medical, Inc. (hereinafter "Globus Medical" or "Globus"), enables real-time surgical navigation and visualization using radiological patient images, and guides compatible surgical instruments to a precise location and trajectory based on implant planning or provides visualization for assisting with free-hand navigation. The software reformats patient-specific CT images acquired before or during surgery, or fluoroscopic images acquired during surgery, and displays them on-screen based on the preferred perspective. Prior to operating, the surgeon may create, store, access, and simulate instrument and other trajectories relative to patient anatomy captured in the CT images. During surgery, the system recognizes the instrument in use and aids the user by way of free-hand navigation to place implants with consistent accuracy. ExcelsiusHub tracks the position of surgical instruments in or on the patient anatomy, and continuously updates the instrument position on these images. The surgery is performed by the surgeon, using various Globus Medical specialized surgical instruments.

Although various embodiments are described in the context of operational extensions to Excelsius system products and other Globus Medical products, these and other embodiments are not limited thereto and can be used with any surgical procedure navigation system.

The ExcelsiusHub is a surgical procedure navigation system that enables real-time surgical visualization using radiological patient images (e.g., preoperative CT, intraoperative CT and/or fluoroscopy), a patient dynamic reference base, and an advanced camera tracking system. The system is mapped based on the registration between the virtual patient (points on the patient images) and the physical patient (corresponding points on the patient's anatomy). Once this registration is created, the software displays the relative position of a tracked instrument on the patient images. This visualization can help guide the surgeon's planning and approach for implant placement and other surgical procedures. The patient's scan coupled with the registration provides guidance assistance to the surgeon when using the system independently for free hand navigation or can provide robotic guidance and align the end effector when used with the ExcelsiusGPS Robotic System. During surgery, the system uses the camera tracking system to track the position of compatible instruments, including the end effector on the robotic arm, in or on the patient anatomy and continuously updates the instrument position on patient images utilizing optical tracking. System software may be responsible for navigation functions, data storage, network connectivity, user management, case management, and safety functions. The ExcelsiusHub surgical instruments are typically non-sterile, re-usable instruments that can be operated manually.

The ExcelsiusHub freehand instrumentation includes registration instruments, patient reference instruments, and implant-specific surgical instruments. The system can also be used with the ExcelsiusGPS robotic system, including actively tracked end-effectors. Registration instruments have incorporated arrays of reflective markers, which are used to track patient anatomy and surgical instruments and implants. Components include the verification probe, surveillance marker, surgical instrument arrays, intra-op CT registration fixture, fluoroscopy registration fixture, and dynamic reference base (DRB). Patient reference instruments are either clamped or driven into any appropriate rigid anatomy that is considered safe and provides a point of rigid fixation for the DRB. Surgical instruments are used to prepare the implant site or implant the device, and include awls, drills, drivers, taps, and probes.

The ExcelsiusHub can be used as an aid for precisely locating anatomical structures in open or percutaneous procedures and for precisely positioning compatible surgical instruments or implants during surgery. The ExcelsiusHub is indicated for any medical condition in which the use of stereotactic surgery may be appropriate, and where reference to a rigid anatomical structure, such as the spine or pelvis, can be identified relative to a CT, X-ray, and/or MRI based model of the anatomy. The ExcelsiusHub supports pre-operative CT, intra-operative CT, and/or intra-operative Fluoroscopic procedures.

The ExcelsiusHub Visualization System provides standalone navigation and guidance for previously cleared posterior fixation and interbody implant placement. The ExcelsiusHub can operate with the ExcelsiusGPS Robotic System to provide camera and tracking system functionality. The ExcelsiusHub can also provide a universal viewing station and registration operations for intraoperative mobile CT systems. Excelsius enabling technologies can be supported and communicate through the ExcelsiusHub, which can provide desired integral components for operating rooms.

A camera tracking system <NUM> The ExcelsiusHub components can be divided into three subassemblies as shown in <FIG>: a camera arm assembly <NUM>, a display assembly <NUM>, and a housing base assembly <NUM>, according to some embodiments. The camera arm assembly <NUM> includes spaced apart stereo cameras <NUM> and an articulating arm <NUM>. The housing base assembly <NUM> includes a processing platform with at least one processor and memory, an input and/or output user interface, and communication circuitry which is configured to communicate through wired (e.g., connector panel <NUM>) and/or wireless connections with other system components. The cameras <NUM> operating with the processing platform are configured to detect pose of markers, e.g., reflective markers, on instruments, the patient (e.g., Dynamic Reference Base (DRB)), a surgical robot, etc..

The ExcelsiusHub works with all preexisting Globus navigated, arrayed instrumentation. This includes the drills, awls, probes, taps, and drivers for placement of Globus Screws and the dilators, disc preparation instruments (curettes, Cobb elevators, rasps, scrapers, etc.), trials, and inserters for navigated placement of Globus Interbodies. Each instrument is identified by a unique array pattern that is recognized by the camera.

The ExcelsiusHub may be used with existing patient fixation instruments and the current DRB.

The ExcelsiusHub may be used with existing intra-op registration fixtures and the fluoroscopy registration.

The software operating with the ExcelsiusHub may include existing Spine software available on the ExcelsiusGPS Robotic System. The system software may be responsible for all navigation functions, data storage, network connectivity, user management, case management, and safety functions.

Spine surgical procedures are supported by the ExcelsiusHub System. The CONFIGURE tab displays procedure types. The spine procedural steps are the same as seen on ExcelsiusGPS Robotic System as the software is the same on both pieces of hardware.

Various Spinal surgical procedures Spinal surgical procedures supported by the ExcelsiusHub System are listed in the table below.

Globus spinal implant systems that are compatible with the ExcelsiusHub System include those listed in the table below.

After selecting a case, the CONFIGURE tab is displayed on the monitor. An example user interface is shown in <FIG> according to some embodiments. Using the CONFIGURE tab select the surgeon from among a list of registered surgeons, the imaging modality and the procedure type. The imaging modality can include intra-operative CT, fluoroscopy, and pre-operative CT. Click the right arrows to advance to the next tab.

Using the WORKFLOW tab, select the desired stage of the procedure (e.g., interbody, or screws) in the desired order of operation (e.g., interbody first). For each stage, select the imaging modality, interbody implant system, and desired interbody level on the anatomical model, which may include cervical, thoracic, lumbar, and sacroiliac. Add stages to the workflow by clicking the "Add Stage" button. Click "Verify Instruments" to proceed to advance to the next tab. An example user interface is shown in <FIG> according to some embodiments.

The VERIFY tab displays navigation details including visibility, location and verification status of the instruments selected on the WORKFLOW tab. Verification is used to indicate whether one or more instruments may be damaged, e.g., during handling. All instruments with arrays should be verified prior to use, either with a verification adapter, instrument, implant, or dilator, as appropriate.

An example user interface is shown in <FIG> according to some embodiments. The VERIFY tab shows CAMERA VIEW and INSTRUMENT STATUS listing a set of identified instruments.

CAMERA VIEW can be operationally updated in real-time from the perspective of the camera with, e.g., color circles being displayed indicating instrument location. A solid colored circle can displayed to indicate that the corresponding instrument is visible to the cameras <NUM>, while a hollow circle indicates that it is not visible to the cameras <NUM>. Size of the colored circle can be dynamically updated to change size to indicate distance from the physical cameras <NUM>. In one embodiment, the circle size is adapted to grow larger as the instrument is moved closer to the cameras <NUM> and smaller as the instrument is moved away from the cameras <NUM>. An ideal distance from the cameras <NUM> may be approximately <NUM> meters or <NUM> feet, in accordance with some embodiments.

INSTRUMENT STATUS lists each instrument and its verification status, with corresponding color circles to identify each instrument. Verification status symbols are shown in <FIG>, in accordance with some embodiments.

With reference to <FIG>, each instrument can be verified by placing the tip of the instrument into a verification dot provided at a known location on another piece of equipment, e.g., on a registered instruments array or on a registered robot art of a surgical robot system, or by placing a verification adapter to be verified into a verification divot provided at a known location on the instrument, according to some embodiments. Divots can be formed at defined locations on navigated instrument arrays for use in verifying the instruments. If using a ExcelsiusGPS Robotic System, a divot can also be located on the top surface of the End Effector or other known location which can be tracked by the cameras <NUM>. Next, ensure both instruments are visible to the cameras <NUM> and held steady. A pop-up screen then is displayed on the VERIFY tab to indicate operations of the verification progress.

As shown in <FIG>, once verification is complete, verification status is indicated on the screen with the tip error displayed in mm, according to some embodiments. If verification has failed a corresponding indication <NUM> is displayed, e.g., a red crossed circle is displayed and verification operations can be repeated until verification is completed successfully. In contrast, successful verification is indicated by another displayed indication <NUM>, e.g., a green circle. When all instruments are successfully verified, the user can click the right arrow to advance to the next tab.

Patient attachment instruments are secured to rigid bony anatomy neighboring the surgical site. A user selects the desired instrument. Patient attachment instruments should be placed no more than <NUM> from the center of the surgical site to maintain accuracy, in accordance with some embodiments. Bone clamps are clamped onto anatomical structures such as the spinous process, iliac crest, long bone, or any rigid bony structure that can be safely clamped. Quattro spikes are inserted into the iliac crest or a long bone. Rod attachments are secured to an existing spinal rod, e.g., <NUM> to <NUM> in diameter. Example recommended anatomic locations for the various patient attachment instruments are described in the table below according to some embodiments.

A user positions a compression clamp on the Dynamic Reference Base (DRB) over the patient attachment instrument and tightens the knob, as shown in <FIG> according to some embodiments. If needed, a clamp driver can be used to further tighten the DRB knob, as shown in <FIG> according to some embodiments. The user positions the reflective markers on the DRB in the direction of the cameras <NUM>. Care should be taken with initial placement of the patient reference instrument as to not interfere with the surgical procedure. Following navigation, the patient attachment instrument is removed.

A surveillance marker is inserted into rigid bony anatomy to enable the camera tracking system to use the cameras <NUM> to track the relative distance to the DRB, e.g., to identify unwanted shifts in the DRB during the procedure. <FIG> illustrates a placement of a DRB <NUM> spaced apart from a surveillance marker <NUM> according to some embodiments.

Surveillance markers may be inserted into the iliac crest or long bone, or may be attached to the spinous process using a bone clamp. The user verifies that the clamp is rigidly secured. The surveillance marker should be placed no more than <NUM> from the Dynamic Reference Base in some embodiments. Example recommended anatomic locations for the various surveillance markers are described in the table below, according to some embodiments.

The user attaches a disposable reflective marker to the marker post of the surveillance marker <NUM>. [<NUM>] The user attaches the impaction cap, designed to fit over the reflective marker sphere <NUM>, onto the surveillance marker <NUM>. The user inserts the surveillance marker <NUM> into rigid bony anatomy near the surgical site, and may gently impact with a mallet. The user removes the impaction cap. The user removes the reflective marker <NUM> prior to using the removal tool. To use a bone clamp with the marker, the user attaches a disposable marker onto the tip of the bone clamp. The user may use the clamp driver to secure the bone clamp. The user then verifies that the clamp is rigidly secured.

The quattro spikes and surveillance marker are removed from bony anatomy manually or using the removal tool. The bone clamp is removed by loosening the clamp with the clamp driver, attaching the removal tool and lifting up the bone clamp, as shown in <FIG> according to some embodiments.

A pivoting arm starburst <NUM> is placed over a starburst post of the registration fixture post <NUM> of a registration fixture <NUM> and rotated <NUM>° to secure, as shown in <FIG> according to some embodiments. The registration fixture <NUM> is positioned on a patient attachment instrument post <NUM> and a compression clamp <NUM> knob is tightened, as shown in <FIG> according to some embodiments. If needed, a clamp driver can be used to further tighten the knob on the compression clamp <NUM>. The completed assembly is shown in <FIG> according to some embodiments. To release the pivoting arm, a release button <NUM> on the fixture is pushed, the pivoting arm <NUM>° is rotated and pull up. The intra-op CT registration fixture has six degrees of freedom and can be moved by adjusting one of the three joints so that it is stable and hovering over the surgical site. Only the metal fiducials embedded in the fixture need to be in the 3D scan (not the reflective markers). It may be operationally helpful or necessary for the intra-op CT registration fixture to not move between image acquisition and performing an anatomical landmark check.

Selection of the IMAGE tab shown in <FIG> displays the steps needed to load a CT scan image according to some embodiments. The image can be loaded from, e.g., a USB drive, hard drive, and/or a networked device. If the image is transferred via the Ethernet, it may automatically appear on the hard drive when the transfer is complete.

To view images on a USB drive, the USB drive is inserted into a USB port on the connector panel <NUM>. To trigger loading of an image, the hard drive or USB drive icon can be selected followed by selection of the desired patient image. The right arrows can be selected to load the patient images and advance to the next tab.

Automatic registration can be performed when loading images. However, if automatic registrations operations fail or otherwise not utilized, then a manual registration screen can be displayed to allow manual registration as shown in <FIG> according to some embodiments. The image on the left panel of the registration screen is a full scan with a depiction of the intra-operative CT.

A registration fixture and seven fiducials, in some embodiments of the fixture, should be visible below the image. Fiducials that are not registered need to be adjusted by the operator. On the screen, a fiducial is selected which is not yet registered; causing that image to then appear on the right. A colored circle <NUM> is moved on the screen by a user until the user determines that it surrounds a displayed fiducial marker. The three small boxes <NUM> shown in <FIG> at the bottom of the right panel show the x, y and z direction of the fiducial and all should be adjusted until the blue circle is centered. In the example embodiment of <FIG>, a registered fiducial <NUM> is displayed with a solid color filled-in center, another fiducial <NUM> which is not visible to the system is displayed with a non-filled-in center, and yet another fiducial <NUM> which is partially visible to the system is displayed with a partially filled-in center.

Ensure that all seven fiducials are properly identified by viewing the 3D model of the intra-op registration fixture. A fiducial may be deleted by selecting the delete icon on the right panel. Click the right arrows to confirm that the fiducials have been properly identified before proceeding to the next step.

Corresponding operations that may be performed are now described in the context of <FIG> illustrates operations that may be performed by a surgical system in accordance with some embodiments. The surgical system includes at least one processor that may reside in a component of the computer platform <NUM>, such as the camera tracking system <NUM> and/or the navigation system <NUM>. Referring to <FIG>, the operations include to obtain <NUM> a three-dimensional (3D) radiological representation, e.g., CT scan(s) and/or fluoroscopy scan(s), of a targeted anatomical structure of a patient and a set of fiducials of a registration fixture, e.g., <NUM> in <FIG> and/or <NUM> in <FIG>. The operations attempt <NUM> to register locations of the set of fiducials in the 3D radiological representation to a 3D imaging space tracked by a camera tracking system, e.g., system <NUM> in <FIG>. The registration may include attempting to correlate the pose (e.g., location and orientation) set of fiducials in the 3D radiological representation to the pose of optical markers detected in the 3D imaging space. Based on determining <NUM> one of the fiducials of the set has a location that was not successfully registered to the 3D imaging space, the operations display <NUM> at least one view of the 3D radiological representation with a graphical overlay indicating the fiducial has not been successfully registered to the 3D imaging space, receive <NUM> user-supplied location information identifying where the fiducial is located in the 3D radiological representation, and register <NUM> the location of the fiducial to the 3D imaging space based on the user-supplied location information.

The operation to receive <NUM> user-supplied location information identifying where the fiducial is located in the 3D radiological representation, includes to display three orthogonal views of the fiducial in the 3D radiological representation , and display a graphical object overlaid on an initial location in the three orthogonal views. The operation moves location of where the graphical object is displayed in the three orthogonal views responsive to input from the user through a user interface, and determine location of the fiducial in the 3D radiological representation and/or the 3D imaging space based on the location of where the graphical object is displayed in the three orthogonal views. The operation registers the location of the fiducial to the 3D imaging space is based on the determined location of the fiducial in the 3D radiological representation.

The operation to receive <NUM> may display the graphical object overlaid on the initial location in the three orthogonal views, includes to determine the initial location to correspond to a predicted location of the fiducial based on relative locations of fiducials defined by a registration fixture template.

The operation to receive <NUM> may further include to move the location where the graphical object is displayed in the three orthogonal views to track directional inputs received through the user interface of the surgical system.

The operation to attempt <NUM> to register locations of the set of fiducials in the 3D radiological representation to the 3D imaging space tracked by the camera tracking system, may include to obtain, from at least one camera of the camera tracking system, an optical image of a reference array fixated to the patient. The reference array including a set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space. The reference array may also be connected to the registration fixture. The operation may then attempt to register locations of a pattern of the set of optical markers to locations of a pattern of the set of fiducials in the 3D radiological representation, and identify any of the optical markers of the reference array that are not successfully registered to any of the fiducials of the registration fixture.

After registration has been completed, a landmark check can be performed to ensure that the registration was calculated successfully. Using the verification probe, an anatomical landmark or a fiducial is touched on the registration fixture to trigger verification that the corresponding location is shown on the system monitor. This process can be repeated using other, e.g., <NUM> to <NUM> other landmarks.

Corresponding operations that may be performed by the surgical system may include to track locations of a tool captured in video from a camera of the camera tracking system while the tool is being moved by a user toward one of the fiducials that was successfully registered to the 3D image space, and display updated representations of the tool according to the tracked locations in the 3D imaging space. The operations confirm registration accuracy of the one of the fiducials that was successfully registered to the 3D image space based on comparison of a designated one of the tracked locations of the tool to the location of the one of the fiducials registered to the 3D image space.

The Intra-op CT Registration Fixture can then be removed while ensuring the patient attachment instrument does not move.

The PLAN tab allows the user to plan all screw insertion trajectories (e.g., <NUM>) and interbody placement (e.g., <NUM>) on the patient image, such as shown in the example user interface of <FIG> according to some embodiments. Implants are preloaded (e.g., with defined characteristics) in the system and displayed on the right-hand side of the screen, based on selections made in the PREPLAN tab.

To add an implant onto the planning page, a user can drag and drop the appropriate implant label on the image at the desired slice. The active plan is shown in a defined color. Details of the active screw plan are shown on the lower right of the screen, including screw family, diameter, and length. The right arrow can be selected to advance to the next tab once plans are complete for all screws.

Once the implant is dropped on the image, the implant planning features are used to adjust implant location by, e.g., dragging the implant image on the touch screen. The user selects or otherwise defines the specific implant size (width, length, height, lordosis) on the right panel of the screen.

Corresponding operations that may be performed are now described in the context of <FIG> illustrates operations that may be performed by the surgical system in accordance with some embodiments. Referring to <FIG>, the operations include to display <NUM> a virtual implant device as an overlay on a view of the 3D radiological representation of the targeted anatomical structure, and display <NUM> a graphical indication of a trajectory of the virtual implant device representing an implantation trajectory of the virtual implant device into the targeted anatomical structure. The operations update <NUM> pose of the graphical indication of the trajectory of the virtual implant device displayed in the view of the 3D radiological representation, to track steering inputs received through a user interface of the surgical system. The operations store <NUM> as a planned trajectory of the virtual implant device, a user-designated one of the poses of the graphical indication of the trajectory.

The operations may further include to display a set of implant devices which are selectable by a user for implant planning, and generate a graphical representation of the virtual implant device based on a template of one of the set of user-selectable implant devices which is selected by a user through the user interface.

The NAVIGATE tab allows the user to visualize the navigated instrument trajectory and the planned trajectory with respect to patient anatomy. An example user interface is shown in <FIG> according to some embodiments. The desired implant label can be selected on the right of the screen.

The real-time instrument/implant trajectory 1700a,1700b (actual plan) is displayed on the patient images, e.g., in orthogonal image slice views, along with the planned screw, allowing the user to confirm the desired trajectory. If the real-time trajectory is not acceptable, the user can return to the PLAN tab to select another trajectory. If the real-time trajectory is acceptable, the user inserts the screw according to the instrument's current trajectory to the desired depth.

Navigated instruments are displayed as they are advanced to the planned position. While navigating the instruments, the user repetitively observes the monitor and surgical site to ensure consistency between tactile and navigation feedback.

Corresponding operations by the surgical system may include to display the planned trajectory of virtual implant device as an overlay on the view of the 3D radiological representation of the targeted anatomical structure. The operations obtain, from at least one camera of the camera tracking system, optical images of a reference array fixated to a real-implant device corresponding to the virtual implant device, the reference array including a set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space. The operations track pose of the real-implant device in the 3D imaging space based on pose of the reference array in the optical images while the real implant device is being positioned by a user relative to the targeted anatomical structure of the patient. The operations display updated graphical representations of the real-implant device relative to the planned trajectory of the virtual implant device according to the tracked pose in the 3D imaging space.

The IMAGE tab shows the steps needed to load a CT scan image. An example user interface is shown in <FIG> according to some embodiments. The image can be loaded from, e.g., a USB drive, hard drive, or networked device. If the image is transferred through the Ethernet, it may automatically appear on the hard drive when the transfer is complete.

To view images on a USB drive, the USB drive is inserted into the USB port on the connector panel. To load an image, the user selects the hard drive or USB drive icon and selects the desired patient image. The right arrows can be selected to load the patient images and advance to the next tab.

The PLAN tab allows the user to plan all screw trajectories and interbody placement on the patient image. An example user interface is shown in <FIG> according to some embodiments. Implants are preloaded (e.g., characteristics predefined in the system) on the right side of the screen, based on selections made in the PREPLAN tab.

To add an implant onto the planning page, a user may drag and drop the appropriate implant label on the image at the desired slice. The active plan is shown in a defined color. Details of the active screw plan are shown on the lower right of the screen, including screw family, diameter, and length. The right arrows can be selected to advance to the next tab once plans are complete for all screws.

Once the implant is dropped on the image, the implant planning features a performed to adjust implant location by, e.g., dragging the implant image on the touch screen. The specific implant size (width, length, height, lordosis) is selected or defined on the right panel of the screen.

The NAVIGATE tab allows the user to visualize the navigated instruments and trajectory alignment with respect to patient anatomy, according to the implant plan.

Another display screen, e.g., as shown in <FIG>, highlights the three steps to complete before the fluoroscopy images can be taken to register the pre-operative CT image, according to some embodiments. The steps may be to insert the DRB, position the C-ARM of an C-arm imaging device, and register a surveillance marker with the camera navigation system. Animation may be used to visually depict the steps.

A Fluoroscopy Registration Fixture <NUM> is attached to an image intensifier <NUM> on the C-arm, as shown in <FIG> according to some embodiments, by turning the clamp clockwise until tight. New optical markers are installed on the fixture <NUM> prior to orienting the fixture <NUM> such that the optical markers are facing the cameras <NUM>. A video capture cable is connected to the C-arm viewing station. A video capture USB cable is inserted into one of the USB ports on ExcelsiusHub connector panel <NUM>.

The user may ensure that the Dynamic Reference Base is visible to the cameras <NUM> after the C-Arm is in place.

The surveillance marker is registered with the camera tracking system by, e.g., placing an instrument close to the reflective sphere <NUM> on the surveillance marker <NUM> but not touching. The box is then displayed in a defined color when it is activated.

The right arrows can be selected to advance to the next tab.

Operations acquire the intra-operative fluoroscopic images, one anteroposterior (AP) and one lateral for each level planned. The same image may be used for multiple levels.

In some embodiments, the operations verify that the following three conditions are met prior to enable acquisition of the images: <NUM>) the DRB is visible by the cameras <NUM>; <NUM>) the Fluoroscopy Registration Fixture is visible by the cameras <NUM>; and <NUM>) a valid fluoroscopic image was taken.

An example user interface is shown in <FIG> according to some embodiments. Each of the three images on the left of the screen turn to a defined color when ready for image capture. When all three conditions are met, the intra-operative fluoroscopic image is acquired and then the CAPTURE button is selected to transfer the image to the system. Once both images are successfully captured, the spinal level on the right side of the screen displays a check mark. The right arrows can be selected to advance to the next tab.

An example user interface for selecting a desired level is shown in <FIG> according to some embodiments. The planned screw may be dragged-and-dropped onto the fluoroscopic images. A displayed graphical object, e.g., circle 2300a and/or 2300b, can be controlled, via user input through a user interface, to roughly position the screw within the vertebral body. The user input may correspond to a user touch-selecting or clicking on a desired location on the display and/or providing steering commands through a keyboard. The screw shank is confirmed to be positioned correctly, with the head and tail of the screws in the desired direction, and with the left/right correctly oriented. The register button can be selected when the confirmation is complete to allow registration.

A check mark is shown next to the active level when registration is successful. An example user interface is shown in <FIG> according to some embodiments. The right arrows are selected when registration is completed.

Corresponding operations that may be performed are now described in the context of <FIG> illustrates operations that may be performed by the surgical system in accordance with some embodiments. Referring to <FIG>, prior to obtaining the 3D radiological representation of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture, the operations include to obtain <NUM>, from at least one camera of the camera tracking system, optical images of a reference array (e.g., DRB <NUM> in <FIG>) fixated to the patient and of a registration fixture (e.g., fixture <NUM> in <FIG>) attached to a radiological imaging device, the reference array including a first set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space, and the registration fixture including a second set of optical markers detectable by the at least one camera of the camera tracking system in the 3D imaging space. The operations obtain <NUM> a fluoroscopic image of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture. The operations determine <NUM> whether a first condition is satisfied based on a defined number of the optical markers in the first set being detected by the at least one camera of the camera tracking system in the 3D imaging space. The operations determine <NUM> whether a second condition is satisfied based on a defined number of the optical markers in the second set being detected by the at least one camera of the camera tracking system in the 3D imaging space. The operations determine <NUM> whether a third condition is satisfied based on a defined number of the set of fiducials of the registration fixture being visible in the fluoroscopic image. A determination <NUM> is made whether any of the three conditions is not satisfied. When one of the first, second, and third conditions is not satisfied, the operations display <NUM> an indication of the not satisfied one of the first, second, and third conditions, and inhibit capture by a radiological imaging process of the 3D radiological representation of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture. In contrast, when each of the first, second, and third conditions are satisfied, the operations enable <NUM> capture by the radiological imaging process of the 3D radiological representation of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture.

The operations may trigger capture of anteroposterior and lateral fluoroscopic images at a plurality of defined locations of the targeted anatomical structure of the patient based on determining each of the first, second, and third conditions are satisfied. The operations may then compute the 3D radiological representation of the targeted anatomical structure of the patient and the set of fiducials of the registration fixture based on the captured anteroposterior and lateral fluoroscopic images at the plurality of defined locations of the targeted anatomical structure of the patient.

After registration has been completed, a landmark check, or verification, can be performed to operationally ensure that the registration was calculated successfully. Using the verification probe, touch an anatomical landmark and verify that the corresponding location is shown on the system monitor. This process may be repeated using, e.g., <NUM> to <NUM> landmarks.

With reference to <FIG>, the user selects the desired implant label on the right of the screen. The real-time instrument/implant trajectory (actual plan) is updated to be displayed on the patient images along with the planned screw, allowing the user to confirm the desired trajectory. If the real-time trajectory is not acceptable, the user can return to the PLAN tab to select another trajectory. If the real-time trajectory is acceptable, the user inserts the screw according to the instrument's current trajectory to the desired depth.

Navigated instruments are displayed as they are advanced to the planned position. While navigating the instruments, the user can repetitively observe the monitor and surgical site to ensure consistency between tactile and navigation feedback.

Carefully remove the Fluoroscopic Registration Fixture. Ensure the patient attachment instrument does not move.

One screen shown in <FIG>, which may substantially correspond to the screen of <FIG>, highlights the three steps to complete before fluoroscopic images can be taken to register the patient, such as described above for <FIG>.

The Fluoroscopy Registration Fixture can be attached to the image intensifier on the C-arm, as shown in <FIG> according to some embodiments, by turning the clamp clockwise until tight. Install new optical markers on the fixture prior to orienting the fixture such that the optical markers are facing the camera.

The user ensures that the Dynamic Reference Base is still visible to the cameras <NUM> after the C-Arm is in place. The surveillance marker may be registered by placing an instrument close to the reflective sphere on the surveillance marker but not touching. The box turns to a defined color when it is activated. The right arrows can be selected to advance to the next tab.

Intra-operative fluoroscopic images are acquired, such as one AP and one lateral.

An example user interface is shown in <FIG> according to some embodiments. Each of the three images on the left of the screen turn to a defined color when ready for image capture. When all three conditions are met, the intra-operative fluoroscopic image is acquired and then the CAPTURE button is selected to transfer the image to the system. Once both images are successfully captured, the level on the right side of the screen displays a check mark. Once the appropriate images have been loaded and selected, the right arrows can be selected to proceed.

After registration has been completed, a landmark check, or verification, can be performed to operationally ensure that the registration was calculated successfully. Using the navigated verification probe, touch an anatomical landmark and verify that the corresponding location is shown on the system monitor. This process may be repeated using, e.g., <NUM> to <NUM> landmarks.

The Fluoroscopic Registration Fixture is removed while ensuring the patient attachment instrument does not move.

The PLAN tab allows the user to plan all screw trajectories and interbody placement on the patient image. An example user interface is shown in <FIG> according to some embodiments. Implants are preloaded on the right-hand side of the screen, based on selections made in the PREPLAN tab.

To add an implant onto the planning page, the user may drag and drop the appropriate implant label on the image at the desired slice. The active plan is shown in a defined color. Details of the active screw plan are shown on the lower right of the screen, including screw family, diameter, and length. A user may select (e.g., click) on the right arrows to advance to the next tab once plans are complete for all screws.

Once the implant is dropped on the image, the implant planning features are used to adjust implant location by, e.g., dragging the implant image on the touch screen. The selects the specific implant size (width, length, height, lordosis) on the right panel of the screen. Alternatively or additionally, software of the planning system may perform automatic adjustment of the implant location so that the dropped implant satisfies one or more defined rules with respect to anatomy in the image.

The NAVIGATE tab allows the user to visualize the navigated instrument trajectory and the planned trajectory with respect to patient anatomy. An example user interface is shown in <FIG> according to some embodiments.

The user selects the desired implant label on the right of the screen. The real-time instrument/implant trajectory (actual plan) is updated to be displayed on the patient images along with the planned screw, e.g., as graphical objects 2900a and 2900b, allowing the user to confirm the desired trajectory. If the real-time trajectory is not acceptable, the user can return to the PLAN tab to select another trajectory. If the real-time trajectory is acceptable, the user inserts the screw according to the instrument's current trajectory to the desired depth.

Navigated instruments are displayed as they are advanced to the planned position. While navigating the instruments, the repetitively observes the monitor and surgical site to ensure consistency between tactile and navigation feedback.

<FIG> is an overhead view of personnel optionally wearing extended reality (XR) headsets 3150a-3150b during a surgical procedure in a surgical room that includes a camera tracking system <NUM> for navigated surgery and optionally includes a surgical robot system for robotic assistance, and each of which is configured in accordance with some embodiments.

Referring to <FIG>, the robot system <NUM> may include, for example, a surgical robot <NUM>, one or more robotic arms <NUM>, an end-effector <NUM>, for example, configured to attach to a joint manipulation arm, and an end-effector reference array which can include one or more tracking markers. The robot system <NUM> may further include one or more displays. The DRB <NUM> includes a plurality of tracking markers and is adapted to be secured directly to a patient <NUM> (e.g., to a bone of the patient <NUM>). Another reference array <NUM> is attached or formed on an instrument, etc. The camera tracking system <NUM> can have any suitable configuration to move, orient, and support the tracking cameras <NUM> in a desired position, and may contain a computer operable to track pose of reference arrays.

The tracking cameras <NUM> may include any suitable camera or cameras, such as one or more infrared cameras (e.g., bifocal or stereophotogrammetric cameras), able to identify, for example, active and passive tracking markers for various reference arrays attached as the patient <NUM> (e.g., DRB <NUM>), end-effector <NUM> (end-effector reference array), instrument(s) (e.g., instrument array <NUM>), extended reality (XR) headset(s) 3150a-3150b worn by a surgeon <NUM> and/or a surgical assistant <NUM>, etc. in a given measurement volume viewable from the perspective of the tracking cameras <NUM>. The tracking cameras <NUM> may track markers attached or formed on the robot arm <NUM> manipulated by a user (surgeon) and/or the robot system <NUM>. The tracking cameras <NUM> may scan the given measurement volume and detect light that is emitted or reflected from the reference arrays in order to identify and determine poses of the reference arrays in three-dimensions. For example, active reference arrays may include infrared-emitting markers that are activated by an electrical signal (e.g., infrared light emitting diodes (LEDs)), and passive reference arrays may include retro-reflective markers that reflect infrared light (e.g., they reflect incoming IR radiation into the direction of the incoming light), for example, emitted by illuminators on the tracking cameras <NUM> or other suitable device.

The XR headsets 3150a and 3150b (also referred to as an XR headset <NUM>) may each include tracking cameras that can track poses of reference arrays within their camera field-of-views (FOVs) <NUM> and <NUM>, respectively. Accordingly, as illustrated in <FIG>, the poses of reference arrays attached to various objects can be tracked while in the FOVs <NUM> and <NUM> of the XR headsets 3150a and 3150b and/or a FOV <NUM> of the tracking cameras <NUM>.

An XR headset may be configured to augment a real-world scene with computer generated XR images. The XR headset may be configured to provide an augmented reality (AR) viewing environment by displaying the computer generated XR images on a see-through display screen that allows light from the real-world scene to pass therethrough for combined viewing by the user. Alternatively, the XR headset may be configured to provide a virtual reality (VR) viewing environment by preventing or substantially preventing light from the real-world scene from being directly viewed by the user while the user is viewing the computer generated AR images on a display screen. An XR headset can be configured to provide both AR and VR viewing environments. Thus, the term XR headset can referred to as an AR headset or a VR headset.

<FIG> illustrates a potential configuration for the placement of the camera tracking system <NUM> and the surgical robot system <NUM> (when present) in an operating room environment. Computer-aided navigated surgery can be provided by the camera tracking system <NUM> controlling the XR headsets 3150a and 3150b, the display <NUM>, and/or the display <NUM> to display surgical procedure navigation information. Including the surgical robot system <NUM> in an operating room is optional for computer-aided navigated surgery.

The camera tracking system <NUM> may use tracking information and other information from the camera tracking system <NUM> along with other tracking information and information from one or more XR headsets 3150a and 3150b, e.g., inertial tracking information and optical tracking information as well as (optional) microphone information. The XR headsets 3150a and 3150b operate to display visual information and may play-out audio information to the wearer. This information can be from local sources (e.g., the surgical robot <NUM> and/or other operating room equipment), remote sources (e.g., patient medical image server), and/or other electronic equipment. The XR headsets 3150a and 3150b may be used to track poses of instruments, patient references, and/or a robot end-effector in <NUM> degrees-of-freedom (6DOF), and may track the hands of the wearer. The XR headsets 3150a and 3150b may also operate to track hand poses and gestures to enable gesture-based interactions with "virtual" buttons and interfaces displayed through the XR headsets 3150a and 3150b and may interpret hand or finger pointing or gesturing as triggering operation of various defined commands. Additionally, the XR headsets 3150a and 3150b may have a <NUM>-10x magnification digital color camera sensor called a digital loupe.

An "outside-in" machine vision navigation bar (tracking cameras <NUM>) may track pose of the joint manipulation arm using monochrome and/or color camera(s). The machine vision navigation bar generally has a more stable view of the environment because it does not move as often or as quickly as the XR headsets 3150a and 3150b tend to move while positioned on wearers' heads. The patient reference array <NUM> is generally rigidly attached to the patient with stable pitch and roll relative to gravity. This local rigid patient reference <NUM> can serve as a common reference for reference frames relative to other tracked arrays, such as a reference array on the end-effector <NUM>, instrument reference array <NUM>, and reference arrays on the XR headsets 3150a and 3150b.

In some embodiments, one or more of the XR headsets 3150a and 3150b are minimalistic XR headsets that display local or remote information but include fewer sensors and are therefore more lightweight.

The robot system <NUM> may be positioned near or next to patient <NUM>. The tracking camera <NUM> may be separated from the robot system <NUM> and positioned at the foot of patient <NUM>. This location allows the tracking camera <NUM> to have a direct visual line of sight to the surgical field <NUM>. It is contemplated that the robot system <NUM> and the tracking camera <NUM> will be located at any suitable position. In the configuration shown, the surgeon <NUM> may be positioned across from the robot <NUM>, but is still able to manipulate the end-effector <NUM> (and joint manipulation arm) and the display <NUM>. A surgical assistant <NUM> may be positioned across from the surgeon <NUM> again with access to both the end-effector <NUM> and the display <NUM>. If desired, the locations of the surgeon <NUM> and the assistant <NUM> may be reversed. The traditional areas for the anesthesiologist <NUM> and the nurse or scrub tech <NUM> remain unimpeded by the locations of the robot <NUM> and camera <NUM>. The anesthesiologist <NUM> can operate anesthesia equipment which can include a display <NUM>.

The end-effector <NUM> may be releasably coupled to the robotic arm <NUM> and movement of the end-effector <NUM> can be controlled by at least one motor based on input from the camera tracking system <NUM>. In some embodiments, the end-effector <NUM> can be connectable to a joint manipulation arm <NUM> and may include a guide tube <NUM> configured to receive and orient a surgical instrument, tool, or implant used to perform a surgical procedure on the patient <NUM>.

As used herein, the term "end-effector" is used interchangeably with the terms "end-effectuator" and "effectuator element. " The term "instrument" is used in a non-limiting manner and can be used interchangeably with "tool" and "implant" to generally refer to any type of device that can be used during a surgical procedure in accordance with embodiments disclosed herein. Example instruments, tools, and implants include, without limitation, joint manipulation arms, drills, screwdrivers, saws, dilators, retractors, probes, implant inserters, and implant devices such as screws, spacers, interbody fusion devices, plates, rods, etc. In some embodiments, the end-effector <NUM> can comprise any structure for effecting the movement of a surgical instrument in a desired manner.

The surgical robot <NUM> is operable to control the translation and orientation of the end-effector <NUM>. The robot <NUM> may be operable to move end-effector <NUM> under computer control along x-, y-, and z-axes, for example. The end-effector <NUM> can be configured for selective rotation under computer control about one or more of the x-, y-, and z-axis , and a Z Frame axis (such that one or more of the Euler Angles (e.g., roll, pitch, and/or yaw) associated with end-effector <NUM> can be selectively computer controlled). In some embodiments, selective control of the translation and orientation of end-effector <NUM> can permit performance of medical procedures with significantly improved accuracy compared to conventional robots that utilize, for example, a six degree of freedom robotic arm comprising only rotational axes. For example, the surgical robot system <NUM> may be used to operate on patient <NUM>, and robotic arm <NUM> can be positioned above the body of patient <NUM>, with end-effector <NUM> selectively angled relative to the z-axis toward the body of patient <NUM>.

In some example embodiments, the XR headsets 3150a and 3150b can be controlled to dynamically display an updated graphical indication of the pose of the surgical instrument so that the user can be aware of the pose of the surgical instrument at all times during the procedure.

As used herein, the term "pose" refers to the position and/or the rotational angle of one object (e.g., dynamic reference array, end-effector, surgical instrument, anatomical structure, etc.) relative to another object and/or to a defined coordinate system. A pose may therefore be defined based on only the multidimensional position of one object relative to another object and/or relative to a defined coordinate system, based on only the multidimensional rotational angles of the object relative to another object and/or to a defined coordinate system, or based on a combination of the multidimensional position and the multidimensional rotational angles. The term "pose" therefore is used to refer to position, rotational angle, or combination thereof.

In some further embodiments, the surgical robot <NUM> can be configured to correct the path of the joint manipulation arm being moved by the surgeon with guidance by the robotic arm <NUM>. In some example embodiments, surgical robot <NUM> can be configured to perform stoppage, modification, and/or manual control of the movement of end-effector <NUM>. Thus, in use, in example embodiments, a surgeon or other user can operate the system <NUM>, and has the option to stop, modify, or manually control the autonomous movement of end-effector <NUM>.

Reference arrays can be formed on or connected to the robotic arm <NUM>, the end-effector <NUM>, patient <NUM>, and/or the surgical instrument. The camera tracking system <NUM> can track poses of the reference arrays in, e.g., <NUM> degree-of-freedom (e.g., position along <NUM> orthogonal axes and rotation about the axes). In some embodiments, a reference array including a plurality of tracking markers can be provided thereon (e.g., formed-on or connected-to) to an outer surface of the robot <NUM>, such as on the robot arm <NUM> and/or on the end-effector <NUM>. A patient reference array <NUM> including one or more tracking markers can further be provided on the patient <NUM> (e.g., formed-on or connected-to). An instrument reference array <NUM> including one or more tracking markers can be provided on surgical instruments (e.g., a screwdriver, dilator, implant inserter, or the like). The reference arrays enable each of the marked objects (e.g., the end-effector <NUM>, the patient <NUM>, and the surgical instruments) to be tracked by the camera tracking system <NUM>, and the tracked poses can be used to provide navigation guidance to a user for performance of a surgical procedure and/or can be used to control movement of the surgical robot <NUM> for guiding the end-effector <NUM>.

<FIG> illustrates a block diagram of surgical system which includes a camera tracking system <NUM> and navigation system <NUM>, and further optionally includes a surgical robot <NUM>, imaging device(s) <NUM>, and an XR headset <NUM>, which are each operative in accordance with some embodiments.

The imaging devices <NUM> may include a C-arm imaging device, an O-arm imaging device, and/or a patient image database. A computer platform <NUM> includes at least one processor, at least one memory storing program instructions executable by the at least one processor to perform operations. The computer platform <NUM> may perform operations of the camera tracking system <NUM> and/or the navigation system <NUM>. The XR headset <NUM> provides an improved human interface for performing navigated surgical procedures. The XR headset <NUM> can be configured to provide functionalities, e.g., via the computer platform <NUM>, that include without limitation any one or more of: display camera tracking information and surgical procedure navigation information, identify hand gesture-based commands, etc. A display device <NUM> may include a video projector, flat panel display, etc. The user can view XR graphical objects as an overlay anchored to particular real-world objects viewed through a see-through display screen. The XR headset <NUM> may additionally or alternatively be configured to display on the display device <NUM> video streams from cameras mounted to one or more XR headsets <NUM> and other cameras, and/or medical images obtained from the imaging device(s) <NUM>.

Electrical components of the XR headset <NUM> can include a plurality of cameras <NUM>, a microphone <NUM>, a gesture sensor <NUM>, a pose sensor (e.g., inertial measurement unit (IMU)) <NUM>, the display device <NUM>, and a wireless/wired communication interface <NUM>. The cameras <NUM> of the XR headset <NUM> may be visible light capturing cameras, near infrared capturing cameras, or a combination of both.

The cameras <NUM> may be configured to operate as the gesture sensor <NUM> by tracking user hand gestures performed within the field of view of the camera(s) <NUM>. Alternatively, the gesture sensor <NUM> may be a proximity sensor and/or a touch sensor that senses hand gestures performed proximately to the gesture sensor <NUM> and/or senses physical contact, e.g. tapping on the sensor <NUM> or its enclosure. The pose sensor <NUM>, e.g., IMU, may include a multi-axis accelerometer, a tilt sensor, and/or another sensor that can sense rotation and/or acceleration of the XR headset <NUM> along one or more defined coordinate axes. Some or all of these electrical components may be contained in a head-worn component enclosure or may be contained in another enclosure configured to be worn elsewhere, such as on the hip or shoulder.

As explained above, a surgical system includes a camera tracking system <NUM> which may be part of the computer platform <NUM> which may also provide functionality of the navigation system <NUM> and/or of the XR headset controller <NUM>. The surgical system may include the imaging devices and/or the surgical robot <NUM>. The navigation system <NUM> can be configured to provide visual navigation guidance to an operator for moving and positioning an instrument relative and/or an end effector relative to patient anatomy (e.g., relative to the DRB <NUM>) based on a surgical plan, e.g., from a surgical planning function, defining where a surgical procedure is to be performed using the instrument on the anatomy and based on a pose of the anatomy determined by the camera tracking system <NUM>. The navigation system <NUM> may be further configured to generate navigation information based on a target pose for the instrument, a present pose of the patient anatomy, and a present pose of the instrument and/or an end-effector of the surgical robot <NUM>, where the steering information is used to display information through the XR headset <NUM> and/or another display to indicate where the instrument and/or the end-effector of the surgical robot <NUM> should be moved to perform the surgical plan.

The electrical components of the XR headset <NUM> can be operatively connected to the electrical components of the computer platform <NUM> through a wired/wireless interface <NUM>. The electrical components of the XR headset <NUM> may be operatively connected, e.g., through the computer platform <NUM> or directly connected, to various imaging devices <NUM>, e.g., the C-arm imaging device, the I/O-arm imaging device, the patient image database, and/or to other medical equipment through the wired/wireless interface <NUM>.

The surgical system further includes at least one XR headset controller <NUM> that may reside in the XR headset <NUM>, the computer platform <NUM>, and/or in another system component connected via wired cables and/or wireless communication links. Various functionality is provided by software executed by the XR headset controller <NUM>. The XR headset controller <NUM> is configured to receive information from the camera tracking system <NUM> and the navigation controller <NUM>, and to generate an XR objects based on the information for display on the display device <NUM>.

The XR headset controller <NUM> can be configured to operationally process signaling from the cameras <NUM>, the microphone <NUM>, and/or the pose sensor <NUM>, and be connected to display XR images on the display device <NUM> for user viewing. Thus, the XR headset controller <NUM> illustrated as a circuit block within the XR headset <NUM> is to be understood as being operationally connected to other illustrated components of the XR headset <NUM> but not necessarily residing within a common housing or being otherwise transportable by the user. For example, the XR headset controller <NUM> may reside within the computer platform <NUM> which, in turn, may reside within a housing of the surgical robot <NUM>, the camera tracking system <NUM>, etc..

In the above-description of various embodiments of the present disclosure, aspects of the present disclosure may be illustrated and described herein in any of a number of patentable classes or contexts including any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Accordingly, aspects of the present disclosure may be implemented in entirely hardware, entirely software (including firmware, resident software, micro-code, etc.) or combining software and hardware implementation that may all generally be referred to herein as a "circuit," "module," "component," or "system. " Furthermore, aspects of the present disclosure may take the form of a computer program product comprising one or more computer readable media having computer readable program code embodied thereon.

Any combination of one or more computer readable media may be used. The computer readable media may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an appropriate optical fiber with a repeater, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Program code embodied on a computer readable signal medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that when executed can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions when stored in the computer readable medium produce an article of manufacture including instructions which when executed, cause a computer to implement the function/act specified in the flowchart and/or block diagram block or blocks. The computer program instructions may also be loaded onto a computer, other programmable instruction execution apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatuses or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the present disclosure.

Like reference numbers signify like elements throughout the description of the figures.

Claim 1:
A surgical system for computer assisted navigation during surgery, the surgical system comprising at least one processor operative to:
obtain (<NUM>) a three-dimensional (3D) radiological representation of a targeted anatomical structure of a patient and a set of fiducials of a registration fixture (<NUM>);
attempt (<NUM>) to register locations of the set of fiducials (<NUM>) in the 3D radiological representation to a 3D imaging space tracked by a camera tracking system (<NUM>); and
based on determining one of the fiducials of the set has a location that was not successfully registered to the 3D imaging space,
display (<NUM>) at least one view of the 3D radiological representation with a graphical overlay indicating the fiducial has not been successfully registered to the 3D imaging space,
receive (<NUM>) user-supplied location information identifying where the fiducial is located in the 3D radiological representation, and
register the location of the fiducial to the 3D imaging space based on the user-supplied location information,
wherein:
the operation to receive user-supplied location information identifying where the fiducial is located in the 3D radiological representation, comprises to:
display three orthogonal views of the fiducial in the 3D radiological representation;
display a graphical object overlaid on an initial location in the three orthogonal views;
move location of where the graphical object is displayed in the three orthogonal views responsive to input from the user through a user interface;
determine location of the fiducial in the 3D radiological representation based on the location of where the graphical object is displayed in the three orthogonal views; and
the operation to register the location of the fiducial to the 3D imaging space is based on the determined location of the fiducial in the 3D radiological representation,
characterised in that the operation to display the graphical object overlaid on the initial location in the three orthogonal views, comprises:
determine the initial location to correspond to a predicted location of the fiducial based on relative locations of fiducials defined by a registration fixture template.