Patent ID: 12201379

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The present disclosure can be understood more readily by reference to the following detailed description taken in connection with the accompanying figures and examples, which form a part of this disclosure. It is to be understood that this disclosure is not limited to the specific devices, methods, applications, conditions or parameters described and/or shown herein, and that the terminology used herein is for the purpose of describing particular embodiments by way of example only and is not intended to be limiting of the scope of the present disclosure. Also, as used in the specification including the appended claims, the singular forms “a,” “an,” and “the” include the plural, and reference to a particular numerical value includes at least that particular value, unless the context clearly dictates otherwise.

The term “plurality”, as used herein, means more than one. When a range of values is expressed, another embodiment includes from the one particular value and/or to the other particular value. Similarly, when values are expressed as approximations, by use of the antecedent “about,” it will be understood that the particular value forms another embodiment. All ranges are inclusive and combinable.

A medical professional can use a medical imaging device, for instance a C-arm device, to perform various medical procedures on a patient. For example, medical professionals can use imaging devices to assess bone fractures, guide surgical procedures, or verify results of surgical repairs. C-arm devices, for example, provide spot imaging and fluoroscopic imaging, which allows the generation of continuous real-time moving images. Such images may be provided to a display of the C-arm device.

It is recognized herein that, in some cases, the display of the C-arm system is not positioned in a manner that adequately assists a medical professional during surgery. In various embodiments described herein, images provided by imaging devices may be transmitted in real-time prior to the start of surgery to a display that can be mounted to a surgical instrument, such that fluoroscopic imaging provided by the imaging device can be processed in combination with image sensor data so that alignment of the surgical instrument is based on such image sensor data, as discussed further below.

The display can receive images in real-time, such that the images are displayed by the display in real time (e.g., at the same time that the images are generated by the imaging device). Displaying in real time includes updating the displayed image at least every 0.2 seconds. For example, the display may be updated with the current image every 0.001 to 0.2 seconds. In some embodiments, displaying the orientation in real time includes updating the displayed image at least every 1 second, 0.1 seconds, 0.01 seconds, 0.001, and/or 0.0001 seconds.

In one example, the display is mounted to a surgical drill, such that a representation of fluoroscopic images provided by the imaging device can be viewed before, during, and/or after an intramedullary (IM) nailing procedure. In some embodiments, a representation of the alignment of the surgical instrument can be displayed the display mounted to the surgical instrument, so as to guide the medical professional during the IM nailing procedure, when the display is displaying a representation of image sensor data instead of X-ray image data.

The display can be interactive and can aid in various aspects of an IM nailing procedure. For example, the display can aid in determining and enabling the proper entry point trajectory of a given IM nail, as well as determining and enabling the proper location and orientation for distal locking screws for the IM nail.

As an initial matter, because fluoroscopy is a type of medical imaging that shows a continuous X-ray image on a monitor, the terms fluoroscopic data, fluoroscopic image, video data, and X-ray image may be used interchangeably herein, without limitation, unless otherwise specified. Thus, an X-ray image may refer to an image generated during a fluoroscopic procedure in which an X-ray beam is passed through the anatomy of a patient. Further, it will be understood that fluoroscopic data can include an X-ray image, video data, or computer-generated visual representations. Thus, fluoroscopic data can include still images or moving images.

Referring toFIG.1, a medical imaging system102can include a medical imaging device104and a surgical instrument assembly202in electrical communication with the imaging device104. The medical imaging device104, which can be a C-arm device, can include an X-ray generator or transmitter106configured to transmit X-rays through a body (e.g., bone) and an X-ray detector or receiver108configured to receive the X-rays from the X-ray transmitter106. Thus, the medical imaging device104can define a direction of X-ray travel128from the X-ray transmitter106to the X-ray receiver108. The X-ray transmitter106can define a flat surface106athat faces the X-ray receiver108. The medical imaging device104can further include an arm110that physically connects the X-ray transmitter106with the X-ray receiver108. The medical imaging device104can further be communication with a medical imaging device display112that is configured to display X-ray images from the X-ray detector108. In some cases, the medical imaging device display112can be hard-wired with the X-ray detector108, such that the display112can be in a fixed position relative to the arm110.

The medical imaging device104is presented as a C-arm device to facilitate description of the disclosed subject matter, and is not intended to limit the scope of this disclosure. Further, the imaging system102and the imaging device104are presented as a medical imaging system and a medical imaging device, respectively, to facilitate description of the disclosed subject matter, and are not intended to limit the scope of this disclosure. Thus, it will be appreciated that other devices, systems, and configurations may be used to implement the embodiments disclosed herein in addition to, or instead of, a system such as the system102, and all such embodiments are contemplated as within the scope of the present disclosure. It is recognized herein that the position of the display112can create problems for a medical professional. For example, in some cases, the medical professional may need to view images or data rendered by the display112while viewing a patient positioned between the X-ray generator106and the X-ray detector108.

In an example, a medical professional may face challenges placing distal locking screws during an IM nailing procedure due to insufficient assistive instruments or guidance systems, such as an aiming arm used in placement of proximal screws. Distal screws are commonly inserted in a freehand technique under fluoroscopic guidance. The freehand technique is commonly referred to as the perfect circle technique. For example, once a perfect circle is established during an IM nailing procedure, it may be difficult to properly align a drill bit to the axis of the distal locking hole due to lack of visibility while using radiographic images. Improper alignment can lead to breaching or cracking of an implant during the drilling of a pilot hole, which can result in implant breakage, poor reduction/fixation, delay of surgery, or the like. It is further recognized herein that an orientation of an X-ray image rendered by the display112might not match the orientation of the patient's anatomy, thereby creating further challenges for a medical professional.

As another example of a technical problem addressed by embodiments described herein, before the distal locking screws are placed, a medical professional may face challenges placing the IM nail due to insufficient assistive instruments or guidance systems. IM nails are commonly inserted in a freehand technique under fluoroscopic guidance. Improper placement, however, may result in pain to the patient. For example, different bones and different IM nails require the IM nails to be inserted into the bone at different points of entry and different trajectories, so as to minimize pain. Further, current approaches to determining the appropriate point of entry and trajectory for a specific bone, for instance by consulting a technique guide, can result in errors or delays. In various examples described herein, a surgical instrument assembly can be configured so as guide and help a medical professional during various operations, such as an IM nailing procedure.

The surgical instrument assembly202may include a surgical instrument203and a sensing unit207that is attached to a front portion of the surgical instrument203. The sending unit207may include an image sensor209attached to a top of a measuring device211, which may be attached to the surgical instrument. The image sensor209may be configured to face forward, away from the display212. For example, the image sensor209may face forward at a downward angle toward a longitudinal axis A (e.g., about 30° relative to a plane defined by aa lateral axis L and a transverse axis T, each of which are perpendicular to one another and to the longitudinal axis A).

The surgical instrument assembly202may include a display212that faces away from the image sensor209and the measuring device211. For example, the display212may face rearward.

In some embodiments, the sensing unit includes a second image sensor (e.g., a second camera). The second image sensor may be configured to provide for stereovision, redundant monocular-vision, or a combination of the two to achieve greater accuracy or confidence in the image data generated by the image sensors.

In the illustrated embodiment, the image sensor is in electronic communication with the processor via a wired connection. In an embodiment, the image sensor is in electronic communication with the processor via a wireless connection.

The image sensor209may be small and light enough to add to the drill without interfering with the mobility of the user. The image sensor209may be configured to generate image sensor data with sufficient image quality to be able to detect fiducial markers (e.g., ArUco markers213illustrated inFIG.4D) accurately. For example, the image sensor209may be configured to detect visible light, and examples of characteristics of the image sensor209are provided in the table below. The image sensor209may have any one or combination of the characteristics identified in the table below. The characteristics of the image sensor209may be balanced with size, cost, and other considerations when selecting for the surgical instrument assembly202.

Example Image SensorCharacteristicsFirst OptionPreferred OptionRangeResolution in Megapixels5165-16(MP)Frame Rate Per Second (fps)7.5307.5-30Field of View (degrees)609060-110Focusing Min Range in100150100-200millimeters (mm)Power Supply in Volts (V)555Working Current in milliamps180200180-220(mA)

In an embodiment, the image sensor209may be combined with a lens to achieve one or more of the characteristics identified in the table above.

Referring now toFIG.3, in one embodiment, data (e.g., video or still images) (an example of X-ray image data) provided by the medical imaging device104can be received by an instrument application, for instance an alignment application, which can be a program, such as a software or hardware or combination of both, that can be run on any suitable computing device. As discussed further below, a user can use the instrument application to view representations of an alignment of the surgical instrument203and/or representations of images generated by the medical imaging device104. The instrument application can receive, process, and/or display representations of the alignment and representations of the fluoroscopic images at various locations, for instance at a location that is aligned with a desired trajectory for drilling.

Referring toFIGS.2and3, any suitable computing device204can be configured to host the instrument application. It will be understood that the computing device204can include any appropriate device, examples of which include a portable computing device, such as a laptop, tablet, or smart phone. In another example, the computing device204can be internal to the surgical instrument203.

In an example configuration, the computing device204may include a processing portion or unit206(an example of a processor), a power supply208, the image sensor209, an input portion210, the display212, a memory portion214, a user interface portion216, a wireless transceiver217, and/or an accelerometer215. It is emphasized that the block diagram depiction of computing device204is an example and not intended to imply a specific implementation and/or configuration. The processing portion206, the image sensor209, the input portion210, the display212, memory214, user interface216, the wireless transceiver217, and/or the accelerometer215may be coupled together to allow communications therebetween. The accelerometer215can be configured to generate accelerometer information that corresponds to an orientation of the computing device204. As should be appreciated, any of the above components may be distributed across one or more separate devices and/or locations.

In various embodiments, the input portion210includes a receiver of the computing device204, a transmitter of the computing device204, or a combination thereof. The input portion210is capable of receiving information, for instance fluoroscopic data in real-time, from the medical imaging device104. As should be appreciated, transmit and receive functionality may also be provided by one or more devices external to the computing device204, and thus the surgical instrument assembly202. The input portion210may receive and send information via the wireless transceiver217to another component, for example, to and from the image sensor209.

Depending upon the exact configuration and type of processor, the memory portion214can be volatile (such as some types of RAM), non-volatile (such as ROM, flash memory, etc.), or a combination thereof. The computing device204can include additional storage (e.g., removable storage and/or non-removable storage) including, but not limited to, tape, flash memory, smart cards, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, universal serial bus (USB) compatible memory, or any other medium which can be used to store information and which can be accessed by the computing device204.

The computing device204also can contain the user interface portion216allowing a user to communicate with the computing device204. The user interface216can include inputs that provide the ability to control the computing device204, via, for example, buttons, soft keys, a mouse, voice actuated controls, a touch screen, movement of the computing device204, visual cues (e.g., moving a hand in front of a camera on the computing device204), or the like. The user interface portion216can provide outputs, including visual information (e.g., via a display), audio information (e.g., via speaker), mechanically (e.g., via a vibrating mechanism), or a combination thereof. In various configurations, the user interface portion216can include a display, a touch screen, a keyboard, a mouse, an accelerometer, a motion detector, a speaker, a microphone, a camera, a tilt sensor, or any combination thereof. The user interface portion216can further include any suitable device for inputting biometric information, such as, for example, fingerprint information, retinal information, voice information, and/or facial characteristic information. Thus, a computer system such as the computing device204can include a processor, a display coupled to the processor, and a memory in communication with the processor. The memory can have stored therein instructions that, upon execution by the processor, cause the computer system to perform operations, such as the operations described herein. The display212can be configured to display visual information, such as described with reference toFIGS.4A-4O,FIGS.5A-G, andFIGS.10A to20.

Referring toFIGS.1and3, a transmitter unit114can be electrically coupled to, or can be part of, the medical imaging device104. The transmitter unit114can be any suitable computing device configured to receive and send images, for instance video signals including fluoroscopic images. It will be understood that the transmitter unit114can include any appropriate device, examples of which include a portable computing device, such as a laptop, tablet, or smart phone.

Referring in particular toFIG.3, in an example configuration, the transmitter unit114can include a processing portion or unit116, a power supply118, an input portion120, and an output portion122. It is emphasized that the block diagram depiction of transmitter unit114is an example and not intended to imply a specific implementation and/or configuration. The processing portion116, input portion120, and output portion122can be coupled together to allow communications therebetween. As should be appreciated, any of the above components may be distributed across one or more separate devices and/or locations.

In various embodiments, the input portion120includes a receiver of the transmitter unit114, and the output portion122includes a transmitter of the transmitter unit114. The input portion120is capable of receiving information, for instance fluoroscopic images or video data, from the medical imaging device104, in particular an output interface105of the medical imaging device104. The output interface105can include a coaxial output, a usb output, a component output, a wireless output, or the like. As should be appreciated, transmit and receive functionality may also be provided by the medical imaging device104. In an example, the transmitter unit114is electrically coupled to the output interface105of the medical imaging device104, so as to establish a wired or wireless electrical connection between the transmitter unit114and the display112. The output interface105can include or more video output connectors using the matching input module. In an example, the processing portion116, which can include or more processors running on an embedded operating system, can detect the presence of a signal, for instance a video signal including fluoroscopic images, from the medical imaging device104. The processing portion116can process the signal as necessary for transmitting to the surgical instrument assembly202. For example, the processing portion116can compress the signal so as to reduce the bandwidth that is used for transmitting the signal.

After the processing portion116performs processing on the video signal, as necessary, the video signal that can include fluoroscopic images can be sent by the output portion122of the transmitter unit114to the input portion210of the computing device204. The output portion122of the transmitter unit114can be configured to transmit fluoroscopic images in accordance with any communication protocol as desired. For example, the output portion122can include a ZigBee module connected to the processing portion206via a universal serial bus (USB), such that the output portion122can send data wirelessly (via a wireless communications channel) in accordance with any ZigBee protocol. The output portion122can send video signals, for instance fluoroscopic images, over Wi-Fi, Bluetooth, broadcast, or any other wireless communication channels as desired. The output portion122can send a single X-ray image (also referred to as a “snapshot”) at a time.

Accordingly, the input portion210of the device204can receive data or video signals in real-time, for instance fluoroscopic images, which are sent via a wireless communication channel from the medical imaging device104. The input portion210can be configured to receive ZigBee messages, Wi-Fi messages, Bluetooth messages, broadcast messages, or messages formatted in accordance with any wireless protocol as desired. In an example, when the input portion210of the device204receives the fluoroscopic images from the medical imaging device104, the images can be retrieved and verified by the processing portion206of the computing device204. For example, the processing portion206can verify that the received images are from the appropriate medical imaging device. The images can be forwarded to the display212, for example, when the images are verified. The processing portion206can also ensure that valid data is displayed. For example, if there is an interruption to the wireless communication channel or connection between the computing device204and the medical imaging device104, the processing portion206can identify the interruption, and send a message to the display212so that the interruption is conveyed to a medical professional who views the display212. In some cases, the processor206can cause the surgical instrument assembly202to display an indication of error on the display212when a quality of the communication link between the imaging device104and the surgical instrument assembly202is below a predetermined threshold. Thus, a wireless point-to-point communication channel or connection between the transmitter unit114and the computing device204can be established, and the wireless point-to-point connection can be managed by the input portion210and the output portion122on the physical layer, and the processing portions116and206at the application layer.

Referring again toFIG.1, the image sensor209may provide image sensor data to a processing unit206. The image sensor209may be connected to the processing unit206via a cable. For example, the image sensor209may be directly connected to the processing unit206. In an embodiment, the image sensor is housed together with the processing unit206.

In another embodiment, the image sensor is wirelessly connected to the processing unit206(e.g., via a wifi or Bluetooth module). For example, the image sensor data generated by the image sensor may be sent via wifi to a private URL which may be accessible to the processing unit206to obtain the image sensor data.

The processing unit206may receive the image sensor data from the image sensor and generate navigation graphics on the display212based on the image sensor data.

The display unit212may be reusable. For example, the display unit212may be sterilized before surgery. The sterilized display unit212may be sterilely removably attached to the surgical instrument203after such sterilization. For example, the display unit212may be removable attached via gravity, a push-lock, or a magnet that ensure the display unit212. The display unit212and the surgical instrument203may include corresponding connectors that align with one another to ensure the display212is correctly attached to the surgical instrument. In an embodiment, the display and the image sensor may be sterilized and packaged together. For example, the image sensor and the display may be removed from the respective packaging and each may be separately attached to the corresponding location of the surgical instrument.

In an embodiment, the display may be able to be removed from the surgical instrument and placed elsewhere in the room for viewing. For example, the display may include a clamp, a stand, or another component configured to hold or place the display on a table or other fixture. Removing the display may provide the surgeon with flexibility by reducing drill weight and/or reducing view obstruction.

With reference toFIGS.2A-Dand13, the display212may be configured to be fixed relative to the surgical device203(e.g., along a longitudinal axis A of the drill bit).

The surgical instrument203can define a proximal end203band a working end203aopposite the proximal end203b. The working end203acan be configured to operate on, for instance cut, drill, or otherwise target, a structure, for instance the anatomical structure124, of a medical patient. The display212can face the proximal end203b. The display212can be positioned so as to provide a line of sight to both the working end203aand the display212from a location proximate of the surgical instrument203. Thus, in some cases, for example, a medical professional can, while operating the surgical instrument203, view both the display212and the working end203aof the surgical instrument203.

In an example, the surgical instrument203includes a cutting instrument226that includes a proximal end226badjacent to the body205of the surgical instrument203, and a cutting tip226aopposite the proximal end226bof the cutting instrument226. The cutting tip226acan define a terminal end of the cutting instrument that is opposite to the proximal end226bof the cutting instrument226. The cutting instrument226can have the cutting tip226athat can be configured to remove anatomical material from an anatomical structure, for instance the anatomical structure124. In the illustrated example, the cutting instrument226is a drill bit, and the cutting tip226ais a tip of the drill bit, though it be appreciated that other instruments and configurations may be used to implement the embodiments disclosed herein in addition to, or instead of, an instrument such as the cutting instrument226, and all such embodiments are contemplated as within the scope of the present disclosure.

The image sensor209may be positioned in front of the display212so as to have an unobstructed view of the cutting tip226a, the fiducial markers213, and/or fiducial markers discussed below with reference toFIGS.4D-4Z. The image sensor209may be rotatable relative to the surgical instrument203and the display212. The image sensor may be removable from the surgical instrument203, which may provide for the image sensor209to be sterilized and reused.

In an embodiment, the image sensor may be configured for a single use. For example, the image sensor may be permanently attached to the surgical instrument.

The image sensor209may be calibrated. For example, the image sensor209may be calibrated by using a standard chessboard calibration method or any other well accepted method of obtaining the intrinsic parameters and distortion coefficients. In an embodiment, the image sensor calibration is done with calibration techniques using a chessboard or other planar pattern. For example, the technique proposed by Zhengyou Zhang wherein at least two orientations of a planar pattern are observed by the pattern to calibrate the camera in a closed-form manner. Zhang, Zhengyou. (2000). A Flexible New Technique for Camera Calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on. 22. 1330-1334. 10.1109/34.888718. In an embodiment, another suitable camera calibration method is used to calibrate the image sensor.

In an embodiment, the image sensor is calibrated to obtain the intrinsic parameters and the distortion coefficients during device manufacturing and the calibration is stored by the image sensor or another device in communication with the processing unit206(FIG.3).

Turning now toFIG.8, an example flow chart of steps followed by an X-ray imaging application, for instance an orientation application, which can be a program, such as a software or hardware or combination of both, that can be run on the computing device204or another computing device. At step302a geometry of the implant may be input or detected. For example, the user may input a specific IM nail and choose a target hole in the IM nail via the user interface216. The specific nail geometry and the target hole location may be input or detected.

In an embodiment, the user inputs the model of the C-arm that will generate an X-ray image(s) of the IM nail and adjacent anatomical structure. The X-ray imaging application may be configured to access calibration data specific to the C-arm. In an embodiment, the C-arm includes an inertial motion unit (e.g., including an accelerometer) to determined the orientation of the C-arm. The orientation may be provided by the inertial motion unit to the computing device of the surgical instrument assembly, for example.

C-arm calibration data that is model-specific for the C-arm may be stored by the computing device204and/or the medical imaging device104. The calibration data may include a full volume of C-arm imagable space. Calibration of the C-arm can be done once per model and stored on the computing device204and/or the medical imaging device104for later use.

Calibration of the C-arm may include using a volumetric fixture with identifiable radiopaque markers placed throughout the fixture at predetermined locations. Fluoroscopic images may be taken of the markers and the 2D image locations of each marker mapped to the exact spatial position, which may be known from the fixture. Thereby the entire volume of imagable space between the C-arm transmitter106and the receiver108can be mapped out and output as the calibration data. X-ray images generated by the C-arm may be processed using the mapping to determine the 3D coordinates of features on the 2D image X-ray image.

The calibration may be done in one orientation and the negligible changes to the image geometry due to c-arm angle may be disregarded. In an embodiment, the calibration may be done at different angles and an angle measuring device may be integrated into the c-arm for use in selecting or interpolating to the calibration data that is most appropriate. Some modern c-arms may already have this functionality built-in. An example of the volumetric calibration is given in: Hosseinian S, Arefi H, Navab N. Toward an End-to-End Calibration for Mobile C-Arm in Combination with a Depth Sensor for Surgical Augmented Reality Applications. Sensors (Basel). 2019; 20(1):36. Published 2019 Dec. 19. doi:10.3390/s20010036.

The user may input the model of the medical imaging device102(e.g., the C-arm) to the user interface216, for example, when inputting the specific IM nail. The display212may display a dropdown list of stored C-arm models to be selected from. In an embodiment, the C-arm has an electronic or scannable identification label (such as a QR code) that can provide the system any necessary model-specific information.

In an embodiment, a single fluoroscopic image may be used to approximate the C-arm imaging geometry as that of a pinhole camera. Intrinsic matrix and distortion coefficients may be estimated using the pattern on rigid markers (for example, four outer corners of an ArUco marker) using Zhang's method, for example. In an embodiment, multiple fiducial markers may be detected in the single fluoroscopic image to estimate the matrix and distortion coefficients using Zhang's method. In some embodiments, the geometry of the IM nail, including holes, may be used in addition to the fiducial markers to give more calibration points for using Zhang's method.

In the case of ArUco markers, the markers may detected using OpenCV's detectMarkers function. This may provide image points for each of the four corners of each ArUco marker. Object points for each ArUco marker may be predetermined, as discussed below. Zhang's method may be used the ArUco markers using OpenCV's calibrateCamera function with the image points and the object points. For example, the ArUco markers may be detected using OpenCV's functions, such as cv::aruco::detectMarkers, cv::aruco::estimatePoseBoard, and/or cv::aruco::estimatePoseSingleMarkers.

Registering the fiducial markers to a desired trajectory (e.g., a central axis of a target hole in the IM nail) may include determining a world coordinate system at step330. For example, a rotation matrix or a translation matrix for mapping the C-arm coordinate system (e.g., when the C-arm is modeled as a pinhole camera) to a world coordinate system centered at the desired trajectory is calculated. For the world coordinate system, a Z-axis may be aligned with the desired trajectory (e.g., the central axis of the targeted through-hole) and the X-axis may be aligned with the longitudinal axis of the implant (e.g., the IM nail).

The rotation and translation matrices may be determined by using a PnP method with the detected edges of the targeted through-hole (e.g., when depicted as a perfect circle in the fluoroscopic image). Edge detection may be done by any suitable detection process. For example, OpenCV's Canny Edge Detector may perform the edge detection. In an embodiment, OpenCV's Blob Detector may be used to find a circular shape that meets certain shape pre-requisites such as circularity and minimum/maximum area. Multiple points along the edge may be used as a 2D set of points to match with the known 3D points of the through-hole's perimeter (e.g., using a predetermined radius of the through-hole) at optional step324.

If the fluoroscopic image does not show a perfect circle, then a transformation application may be used to determine the angles of rotation needed to align with the perfect circle and those angles can be used to create the rotation matrix. The translation matrix may also be found by identifying the center point of the detected through-hole (e.g., a circle, or circular arc in the case of the non-perfect circle image) rather than using edge points.

The 3D coordinates of each individual marker key points would be stored in the computing device204based on the known geometry of the fiducial markers. For example, the four corners of an ArUco marker of side length L may be saved as the following array with the origin set as the upper left hand corner [[0,0,0],[L,0,0],[L,−L,0],[0,−L,0]]. All fiducial marker information may be combined into an array of arrays, which may be referred to as an array of object points for the board of fiducial markers.

Key points of each fiducial marker may be detected using PnP. The relative pose of the c-arm (e.g., in the form of rotation and translation vectors) may be stored in the computing device204. For example, ArUco markers may be detected thereby providing the image points of the four corners. After, the image point data may be used with the known object points to determine the rotation and translation vectors. This determination may be done at the same time as the C-arm calibration is being done using the single C-arm image. The rotation and translation vectors may be saved in an array and correlated with the specific identification of the corresponding marker.

The object points of the marker corners may then be put through a change of basis from their local coordinate system (e.g., defined in the plane of the marker with the origin at the upper left hand corner) to the C-arm coordinate system. Next the object point may be put through another change of basis from the C-arm coordinate system to the world coordinate system centered on the desired trajectory (e.g., the central axis of the target through-hole).

The object points may then be used to create an ArUco board object at step332(e.g., an array of arrays of each marker's world coordinates which correspond to another array of marker identifiers). The ArUco board object may be used in later marker board detection using the image sensor209.

The resulting image sensor209pose may be provided relative to the world coordinate system centered at the image sensor209. In an embodiment, the drill bit is targeted at the world origin via the navigation graphics.

If the fiducial markers are observed in more than one fluoroscopic image (e.g., in both the first x-ray with 30% of two holes shown at step304and in the second perfect circle x-ray at step316), the fiducial markers from both images may be used to more accurately calibrate the C-arm with the calibration application at step326. In an embodiment, the C-arm image is cropped down to a central region (e.g., as exemplified inFIGS.4G and4L), which typically may have less distortion than the peripheral radial edges of the fluoroscopic image. The intrinsic matrix and distortion coefficients may be determined based on the cropped image of the central region, rather than the entire image or any markers or data outside of the central region.

Re-projection error may be checked for the set of points and compared against a pre-determined threshold to determine if the quality of calibration is sufficient to enable accurate navigation. The calibration may alternatively or in addition employ the same methods as used in the MaxFrame project as described in U.S. Patent Publication No. US 2021/0077194 entitled “Othopedic Fixation Control and Manipulation.”

At step304the fluoroscopic image may be taken. At step306it may be determined whether a sufficient portion of the target is imaged. For example, a sufficient portion of the targeted area of the IM nail may be at least 30% of the two distal holes of the IM nail being shown in the fluoroscopic image. If so, an angular adjustment for the C-arm may be determined based on the orientation of the target (e.g., the target through-hole) in the image. For example, angular adjustments to the C-arm that will result in a perfect circle image at step312may be determined at step310. The adjustment information may be conveyed to the appropriate application or user (e.g., C-arm tech or surgeon) for adjustment at step312.

In an embodiment, the orientation of the desired trajectory (e.g., the central axis of the target through-hole) is determined without achieving the perfect circle image. For example, the orientation of the central axis of the target through-hole may be determined based on the predetermined characteristics of the IM nail and/or the target through-hole discussed above.

If a sufficient portion of the target is not imaged, the C-arm may be adjusted to a different orientation at step308and step304may be repeated with the C-arm at a different orientation relative to the implant.

In an embodiment, the fiducial marker locations in the fluoroscopic image are detected and compared to the partial holes that can be seen on the image (e.g., as shown inFIGS.4E-4H). The spatial locations of each fiducial marker may be calculated relative to the IM nail geometry provided at step328. Determining the relationship between the fiducial markers and the IM nail may provide for rendering steps310-316optional. Thus, determining the relationship between the fiducial markers and the IM nail may provide for full navigation to any distal locking hole of the IM nail, without the additional perfect circle fluoroscopic image. In this case, the entirety of the distal locking may be performed based solely on a single fluoroscopic image.

A further fluoroscopic image may be taken at step314after adjusting the orientation of the C-arm at step312to achieve the perfect circle fluoroscopic image (e.g., as shown inFIGS.4J-4M.

If one or more fiducial markers is not detected or if the perfect circle fluoroscopic image is not achieved, the orientation of the C-arm may be adjusted at step318so that at least one fiducial marker is in view of the C-arm image and so that the perfect circle is achieved. In an embodiment, the orientation and/or position of the fiducial markers may be adjusted.

The orientation of the fiducial markers may be detected in the image at step320and the relative spatial locations compared to the desired trajectory may be determined at step322. Depth information (e.g., in the direction parallel to the central axis of the targeted through-hole) may be determined by the processing unit based on the image sensor data. In an embodiment, the depth information is not determined (e.g., if the user is able to drill along the axis of the target hole, they will feel when they are drilling bone and do not need Z-axis visual feedback). Fiducial marker detection in the fluoroscopic image, the image sensor image, or both may be less accurate for depth (Z-axis) than for X-coordinates and Y-coordinates.

In an embodiment, C-arm intrinsic and extrinsic parameters may be determined using only the single fluoroscopic image. The C-arm calibration may be performed, as discussed above, based on the single fluoroscopic image. The origin of the world coordinate system may be placed with the X-axis parallel to the longitudinal axis of the IM nail and the Z-axis may be parallel to the desired trajectory (e.g., the central axis of the targeted distal locking through-hole).

If a fiducial marker falls off, is outside the field of view of the C-arm, or otherwise becomes obscured from view or ineffective, there may still be sufficient quantity of other fiducial markers in the array of fiducial markers to perform the steps laid out above. If there are insufficient markers remaining, one or more additional fiducial markers may be placed in the field of view to supplement the existing fiducial markers in the field of view. After placing the additional fiducial marker(s) in the field of view, one X-ray image may be taken to register the fiducial markers to the IM nail.

As shown inFIGS.2A-2D, the measuring device211may be located in front of the display212. The measuring device211may be releasably attached or fixed relative to the surgical instrument203. For example, an attachment member218may releasably attach the measuring device211to the surgical instrument203. In an embodiment, the measuring device is configured and used in the manner described in U.S. Pat. No. 10,736,644 issued on Aug. 11, 2020 and entitled “Surgical power drill including a measuring unit suitable for bone screw length determination.”

A sleeve219may be provided to interact with the measuring device211. The sleeve may be configured to slide onto the drill bit226such that the sleeve219is moveably along the longitudinal axis A. The sleeve219may include fiducial markers213, as noted above. For example, a rearwardly facing surface of the sleeve219may include the fiducial markers213.

The measuring device211may be configured to detect the fiducial markers213on the rearward facing surface of the sleeve219. The distance between the measuring device211and the sleeve may be determined based on the detected fiducial markers213of the sleeve219. For example, the measuring device211may determine the distance between the sleeve219and the measuring device211based on an orientation and/or size of the detected fiducial markers213.

The sleeve219may be positioned at a predetermined location on the drill bit (e.g., spaced a predetermined distance along the longitudinal axis A from the drill tip226aor flush with the drill tip226a). Thus, the orientation and length of the drill tip226may be determined based on the orientation of the fiducial markers213detected by the measuring device211.

In an embodiment, the measuring device outputs to the computing device204data representing the orientation and/or position of the sleeve and/or the drill tip. In an alternative embodiment, the sleeve may be displaceable an angle relative to the longitudinal axis A and the measuring device may be configured to detect such angular displacement.

In an embodiment, the sleeve is includes a reflector plate to register the drill bit (e.g., the end of drill bit tip). For example, the measuring device may include a laser device for detecting linear displacement assessment between the measuring device and the reflector plate of the sleeve. This laser device may be configured to emit light and detect the emitted light that is reflected back by the reflector plate.

The measuring device may be configured to determine linear displacement between the sleeve and the measuring device based on the detected reflected light. For example, the measuring device may perform laser triangulation for linear displacement assessment. In an embodiment, the linear displacement assessment can be performed by using ultra sound position sensors to detect ultra sound waves emitted at the reflector plate.

The orientation, size, and/or position of the drill bit226relative to the image sensor209may be determined based on the orientation, size, and/or position of the drill bit in image data, representing the drill bit226, detected by the image sensor209. For example, based on the orientation, size, and/or position detected by the measuring device211. The measuring device211may communicate (e.g., via a wired or wireless electronic connection) the orientation, size, and/or position of the drill bit226to the computing device204.

The position/location of the drill bit relative to the image sensor209may be determined so that it can be referenced for the navigation output to the display212. Registering the drill bit226may be based on the output of the measuring device211The registration may be based on the orientation, size, and/or position detected by the measuring device211and determining the orientation and position of the image sensor relative to the drill bit226based on a predetermined relationship between the image sensor209and the measuring device211.

In an embodiment, the image sensor is configured to determine the distance between the sleeve and the image sensor based on an orientation and/or size of the fiducial markers of the sleeve detected by the image sensor. For example, the image sensor may be configured to perform the functions of the measuring device211discussed above. In some embodiments, the sleeve includes a unique geometry that is detected by the image sensor instead of fiducial markers or in addition to the fiducial markers.

In an embodiment, the image sensor is configured to determine the orientation, size, and/or position of the drill bit relative to the image sensor. The image sensor may be configured to detect the drill bit and use image recognition to automatically determine the orientation, size, and/or position of the drill bit relative to the image sensor. For example, the image sensor and/or the computing device may be trained (e.g., by using neural networks) to distinguish drill bits from other objects in the environment.

The diameter of the drill bit may be determined based on the target IM nail and target through-hole input by the user (e.g., a 5.2 millimeter (mm) diameter distal locking hole may mean the user is using a 5.0 mm drill bit). The diameter may then used to fit the expected points on the drill bit outer diameter to drill bit edges detected by the image sensor. Thus, the image sensor and/or the computing device may determine the 3D coordinates of the drill bit using, for example, the Points-N-Perspective (PnP) method discussed below.

In an embodiment, the spatial relationship between the image sensor, the drill bit, and the fiducial markers may be determined using any variation of the PnP method. The PnP method may use a given set of rigidly arranged 3D points and their corresponding 2D image projections with a calibrated image sensor209to determine the image sensor pose. In some embodiments, one or more of the following functions or methods may be used: OpenCV's SOLVEPNP_ITERATIVE, Efficient PnP (EPnP) as proposed by Vincent Lepetit, Perspective-Three-Point (P3P) as proposed by Xiao-Shan Ga, Algebraic P3P (AP3P) as proposed by Tong Ke, or Consistently Fast and Globally OptimalSolution PnP (SQPnP) as proposed by George Terzakis.

In some embodiments, the image sensor provides sufficient information of the pose of the drill relative to the image sensor and the fiducial markers such that neither the measuring device nor the accelerometer are utilized. For example, in an embodiment, the accelerometer and measuring device may be omitted from the surgical instrument assembly.

In some embodiments, the image sensor is the only component mounted to the surgical instrument. For example, the image sensor may be configured to communicate with the computing device, which may be separate from the surgical instrument. Such communication may be wireless. In an embodiment, the image sensor and the display are the only components mounted to the surgical instrument. The image sensor and the computing device may be configured to communicate with one another and/or the display.

The surgical instrument assembly may be assembled by the user (e.g., the surgeon) or another person (e.g., a scrub technician).

Fiducial markers213a, which may be substantially the same as those discussed above, may be fixed relative to a target area for surgery. The fiducial markers213amay be placed at any time prior to taking the X-ray image at step304. The fiducial markers213amay be placed after step304and prior to step314, if steps310-316are being performed. The fiducial markers213amay be fixed relative to the patient's anatomy being operated on prior to the surgical procedure (e.g., while patient prep is being done). The fiducial markers213amay not interfere with other steps of the IM nailing procedure.

For example, an array of fiducial markers213may be stuck to the patient's skin (e.g., as shown inFIGS.4U and4W-4Z). In an embodiment, the fiducial markers are adhered to a layer of material that is in contact with the patient's skin. Adhesion to the patient skin or to a layer adjacent to the patient skin (eg. drape, Ioban). May be incorporated into the drape or adhered onto the drape. May be taped onto the patient using Ioban or other tape- or strap-like devices. May use staples, tacks, or burr-like mechanisms to latch onto the skin or layer. May include protrusion(s) that can be hammered or drilled into the patient bone for securement (ie. bone tack). May attach to the proximal end of the nail or the nail handle with an extended, adjustable means to place the markers in the correct area near the distal locking portion of the nail.

In an embodiment exemplified inFIG.4T, the array of fiducial markers213ais fixed relative to the anatomical structure124and the implant125with a stake340. In an embodiment exemplified inFIG.4V, the array fiducial markers213ais fixed relative to the anatomical structure124and the implant125via a jig342that is attached to the implant125and extends externally of the anatomical structure124.

In the illustrated embodiment inFIGS.4D,4P, and4Q, for example, fiducial markers are ArUco markers. In some embodiments, other fiducial markers are provided instead of the ArUco markers to determine pose and distance. For example, an array of small rounded markers213band large rounded markers213cmay be provided in a rectangular pattern, as exemplified inFIG.4R. In another embodiment, one or more spherical fiducials213d, as exemplified inFIG.4S, may be fixed relative to the patient. The spherical shape may provide for improved detection of the fiducial markers, for example, when arranged on a curved surface and/or the fiducial markers213dare detected at different angles of projection.

Regardless of the fiducial marker used, each individual fiducial marker and/or the array of fiducial markers may be visually recognizable by the image sensor209to determine pose and distance relative to the image sensor209. In an embodiment, the circular fiducial markers may be colored and/or have different diameters to distinguish features of the pattern to determined pose and distance. Different coloring of the markers may provide for quick identification of an area of interest, which can be focused on to decrease the amount of image data that must be processed.

The fiducial markers213may be individually recognizable or as an array recognizable when placed onto a curved surface. For example, when placed on the patient's skin as exemplified inFIGS.4U and4W-4Z. The distortion caused to the shapes and relative distances of the fiducial markers213amay be detected by the image sensor209. A curvature of the fiducial markers213amay be determined based on detected distortion. The pose of the fiducial markers may be determined in part based on the curvature. For example, the pose of the fiducial markers213amay be determined at least in part based on the detected curvature from one or more viewpoints exemplified in each ofFIGS.4W-4Z. In an embodiment, the X-ray image data may include the curvature and/or relative distance of the fiducial markers from more than one viewpoint (e.g., of the viewpoints exemplified inFIGS.4W-4Z.

Referring now toFIGS.4W-4Z, the image sensor209may generate image data representing multiple viewpoints and including fiducial markers213athat are not represented in the X-ray image data. For example, one or more of the fiducial markers213amay be radiopaque and represented in a single X-ray image generated by the medical imaging device104(shown inFIG.1). The orientation and/or position of each radiopaque marker213arelative to the desired trajectory (e.g., a central axis of a hole of the implant and/or a hole of an anatomical structure) may be determined in the same manner discussed above with reference toFIG.8. The viewpoints may be perpendicular to one another such that navigation guidance may be provided for two perpendicular planes based on a single 2D X-ray image (i.e., a snapshot).

Multiple images representing different viewpoints from the image sensor209may be daisy-chained to determine the orientation and/or position of fiducial markers213a, relative to the desired trajectory, that are not represented in the X-ray image data. Fiducial markers213athat are not detected by the medical imaging device104(e.g., fiducial markers213athat are not radiopaque or otherwise not detectable by the medical imaging device104from its viewpoint when the single X-ray image is generated) may be detected by the image sensor209. For example, the lowest and highest fiducial markers213awhen viewingFIG.4Wand the highest fiducial markers213awhen viewingFIGS.4X-4Z, respectively, may not be radiopaque. The image sensor209may detect each of the fiducial markers213athat are not radiopaque by generating image data from multiple different viewpoints relative to the fiducial markers213a(e.g., from the viewpoints shown inFIGS.4W-4Z).

With respect toFIG.4W, the pose of the non-radiopaque fiducial markers213a, that are represented along with the radiopaque fiducial marker213a(e.g., the fiducial marker213abetween the lowest and highest fiducial markers213ainFIG.4W), may be determined based on the image sensor data representing the fiducial markers213ainFIG.4Wand the single X-ray image generated by the medical imaging device104. For example, the pose of the non-radiopaque fiducial markers213a, relative to the desired trajectory, may be determined based on the detected pose of the radiopaque fiducial marker213a. Turning toFIG.4X, the pose of the uppermost non-radiopaque fiducial maker213arelative to the desired trajectory may be determined based on the determined pose of the uppermost non-radiopaque fiducial marker213ainFIG.4W. Turning toFIG.4Y, the pose of the uppermost non-radiopaque fiducial maker213arelative to the desired trajectory may be determined based on the determined pose of the uppermost non-radiopaque fiducial marker213ainFIG.4X. Turning toFIG.4Z, the pose of the uppermost non-radiopaque fiducial maker213arelative to the desired trajectory may be determined based on the determined pose of the uppermost non-radiopaque fiducial marker213ainFIG.4Y.

In some embodiments, image sensor generates images from the different viewpoints in a different order. For example, the image sensor may generate image data representing the viewpoint inFIG.4Zbefore generating image data representing the viewpoint inFIG.4Xor the viewpoint inFIG.4Y. For example, the pose of the of the non-radiopaque fiducial markers213ashown inFIG.4Zrelative to one another may be determined before determining the relative pose of the non-radiopaque fiducial markers shown inFIG.4X or4Y.

The fiducial markers213may be made by cutting or machining metal, printing the pattern with radiopaque ink, and/or filling in a cavity with curable radiopaque material. The radiopaque component of the fiducial markers213may be relatively thin, less than 0.3 mm.

The fiducial markers213may each have an area of anywhere from 25 mm to 2,500 mm. For example, the fiducial markers213may have a square outer periphery with the sides of each fiducial marker213having a length of anywhere from 5 mm to 50 mm. In some embodiments, the fiducial markers may each have an area of 400 mm to 625 mm (e.g., with lengths of anywhere from 20 mm to 25 mm). In an embodiment, the fiducial markers each have a square outer periphery and have sides having a length of about 23 mm (i.e., an area of about 529 mm).

Fiducial markers with relatively large sizes may impact the ability of the C-arm and/or the image sensor209to accurately detect the pattern of the fiducial markers. For example, the size may be limited by the anatomical space around the surgical site (e.g., locking site for the IM nail). Larger sizes may limit the quantities of markers that can be placed within the field of view of the C-arm and/or the image sensor209, may obstruct the surgeon's view or range of movement, and/or may be more difficult to adhere securely to the patient's skin.

In an embodiment, a single fiducial marker is provided for detection by the C-arm and the image sensor209. One fiducial marker may be sufficient to determine the pose of the fiducial marker in the X-ray image generated by the C-arm and in the image sensor image generated by the image sensor209. In some embodiments, more than one fiducial marker is provided. For example, at least three fiducial markers may be provided for detection by the C-arm and the image sensor209. Detecting three fiducial markers may result in higher accuracy than when fewer fiducial markers are detected. In some embodiments, at least four fiducial marks are provided. Detecting four fiducial markers may result in higher accuracy than when fewer fiducial markers are detected.

Referring toFIG.7, a user may select one or more operations by inputting an option on an example a user interface2100, which can be displayed by the display212. For example, the user can select an IM trajectory option2104to perform IM drilling operations. The user can select a plating option2103to perform operations associated with securing a plate to a bone. The user can select a nailing option2102to perform operations associated with securing a nail with a distal locking screw. It will be understood that alternative or additional options may be rendered by the user interface2100as desired. Further, it will be understood that the inputting of the options may result in further displays being rendered, so as to guide the user through a particular operation.

Referring generally toFIGS.2A-Dand13, the surgical instrument assembly202that can include the computing device204mounted to a surgical instrument203. The surgical instrument203can be configured to operate on the anatomical structure124(shown inFIG.4A).

The surgical instrument203can define a body205, and the computing device204can be attached anywhere to the body205as desired. In an example, referring toFIGS.2A-D, the computing device204, and the display212, can be supported by a mount228. The mount228may be attachable to the attachment member218. For example, the attachment member may include a rotatable base230that the mount228is attachable to (e.g., via a bolted connection). In some embodiments, the computing device204permanently attached to the surgical instrument.

The rotatable base230may be configured to remain in a fixed position regardless of the orientation of the surgical instrument203, unless rotated by a user. In an embodiment, the rotatable base may be configured to be locked in a desired position.

While the surgical instrument203is depicted as a surgical drill for purposes of example, it will be appreciated that the computing device204can be removably attached to or permanently attached to other suitable equipment or instruments. For example, the surgical instrument assembly202may include an instrument or equipment configured to target an area of bone or other part of the anatomy, remove a medical implant, perform an osteotomy, or any other procedure, for instance any other procedure (e.g., using a combination of fluoroscopy and image sensor images), as desired. Thus, although the anatomical structure124is presented as a bone, it will be understood that structures on which the surgical instrument assembly can be configured to operate are not limited to bones.

The computing device204can include the display212that can be attached to the surgical instrument203. The display212can be configured to display representations of the anatomical structure124(e.g., based on fluoroscopic data generated by the imaging device104and/or image sensor data generated by the image sensor209). In an example configuration, the display212can display representations of the anatomical structure124in real-time, such that the representations of the anatomical structure124are displayed by the display212at the same time that corresponding image sensor data is generated by the image sensor209and/or that images are generated by the imaging device104. In some embodiments, the display can include a plurality of displays, for instance a first display and a second display that has a different orientation as compared to an orientation of the first display.

Referring also toFIGS.4A-4D, a representation of the anatomical structure124can include one or more target locations126. The target locations126may represent locations on the anatomical structure124that the surgical instrument203can drill, cut, or otherwise target. In accordance with the illustrated example, the target locations126can be defined by an implant125, for instance an IM nail or rod, in a bone. It will be understood that an example operation performed by the surgical instrument assembly is presented as an IM nailing operation to facilitate description of the disclosed subject matter, and the example IM operation is not intended to limit the scope of this disclosure. Thus, it will be appreciated that the surgical instrument assembly202may be used to perform other operations in addition to, or instead of, an operation such as the example IM nailing operation.

With reference toFIG.4D, an array of fiducial markers213amay be arranged in rectangular pattern about one or more of the target locations126. As exemplified inFIG.4D, the target locations126may be depicted as extending directly into the page when viewingFIG.4D. The view thatFIG.4Drepresents may be referred to as being aligned with perfect circle. When the target locations126are cylindrical through holes, for example, the X-ray image aligned with perfect circle represents the target locations126as perfect circles.

The computing device104may receive the such perfect circle X-ray image and determine a world coordinate system220. For example, the computing device104may determine the orientation of each fiducial marker213aof the array of fiducial markers213aand from such determine the world coordinate system220based on the X-ray image. The world coordinate system220may include a X1-Axis, a Y1-axis, and a Z1-axis that are each perpendicular to one another and fixed relative to the array of fiducial markers213a.

The world coordinate system220may be defined by a corner of the array of fiducial markers213a. The X1-Axis may extend along one side (e.g., a bottom side) of the rectangular array of fiducial markers213a, the anatomical structure124, and the implant125. The Y1-axis may extend along another side (e.g., a left side) of the rectangular array of fiducial markers213a. The Z1-axis may extend away from a corner of the rectangular array of fiducial markers213a.

The computing device104may determine an image sensor coordinate system222based on the image sensor data. For example, the image sensor coordinate system222may include a X2-Axis, a Y2-axis, and a Z2-axis that are each perpendicular to one another and determined based on the image sensor data. The image sensor coordinate system222may be fixed relative to array of fiducial markers213a, the anatomical structure124, and the implant125. For example, Z2-axis may be centered at one of the target locations126an X2-Axis may extend along a length of the implant125. The Z2-axis may extend radially away from the center of the target location126. A desired trajectory may extend along the Z2axis to the center of the opening of the target location126.

In real time, the computing device104may determine a transformation matrix to determine orientations and/or positions of objects detected in the X-ray image in relation to objects detected in the image sensor data. For example, the computing device104may determine a real time pose of the surgical instrument203(e.g., the drill bit226of the surgical instrument204) in the camera coordinate system222based on the real time image sensor data. The computing device104may use the transformation matrix in real time to determine the real time pose of the surgical instrument203, including the drill bit226, in the world coordinate system220. The computing system104may determine the pose of the surgical instrument203relative to the target location126, for example, based on the determined world coordinates of the surgical instrument203.

The pose drill bit226and the drill tip226arelative to the target location126may be determined by the computing system104in substantially the same manner described above.

In an embodiment, image sensor coordinate system may be fixed relative to the image sensor.

Turning toFIGS.4E-4I, a non-perfect circle X-ray image of the implant125is illustrated and shown at various steps of processing to determine coordinates of the target location126.FIG.4Eexemplifies an initial X-ray image of the implant125along with the array of fiducial markers213a. The X-ray image may be adjusted, for example, the brightness, contract, and/or noise may be adjusted to more clearly identify the implant125and/or the fiducial markers213a, for example.

The adjustment may result in an adjusted X-ray image exemplified inFIG.4F. A portion of the X-ray image illustrated inFIG.4Fmay be cropped so that the computing device204may focus on the area near the target location126, for example, as exemplified inFIG.4G. As shown inFIG.4H, the computing device204may determine the world coordinate system220based on the cropped X-ray image inFIG.4G. As discussed further below, the computing device104may determine the pose of the target location126based on the shape of the target location126depicted in theFIG.4G or4Hin relation to the world coordinate system220. The X1-Axis, the Y1-axis, and the Z1-axis coordinates (e.g., true 3D coordinates) of the target location126(e.g., a center of the opening of the target location126) in relation to the world coordinate system220may be determined by the computing system104.

Turning toFIGS.4J-4O, a perfect circle X-ray image of the implant125is illustrated and shown at various steps of processing to determine coordinates of the target location126.FIG.4Jexemplifies an initial X-ray image of the implant125along with the array of fiducial markers213a. The X-ray image may be adjusted, for example, the brightness, contract, and/or noise may be adjusted to more clearly identify the implant125and/or the fiducial markers213a, for example.

The adjustment may result in an adjusted X-ray image exemplified inFIG.4K. A portion of the X-ray image illustrated inFIG.4Kmay be cropped so that the computing device204may focus on the area near the target location126, for example, as exemplified inFIG.4L. As shown inFIG.4M, the computing device204may determine the world coordinate system220based on the cropped X-ray image inFIG.4L. As discussed further below, the computing device104may determine the location of the target location126relative to the fiducial markers213adepicted inFIG.4M, for example.

As exemplified inFIG.4N, a cylinder method may be used to determine a position of a center, for example, of an opening of the target location126to determine the X1-Axis and Y1-axis coordinates of the target location in relation to the world coordinate system. The desired trajectory for drilling may extend along a central axis of the cylinder.

As discussed above in relation toFIG.4Iand as exemplified in FIG. O, the computing system204may determine a depth of the target location126based on a detected contour of the target location126. Thus, the X1-Axis, the Y1-axis, and the Z1-axis coordinates (e.g., true 3D coordinates) of the target location126in relation to the world coordinate system220may be determined by the computing system104.

The display212can display representations of fluoroscopic images associated with IM nailing operations, among others. Further, the display212can display images or data associated with a depth of the drill bit226. Further still, the display212can display images or data associated with the depth at the same time that the display212renders representations of fluoroscopic images of the anatomical structure124.

The display212can be configured to display, for example, representation images400a-400cof the anatomical structure124, generated by, for example, the computing device104based on X-ray image data received from the medical imaging device104and image sensor data received from the image sensor209. Referring in particular toFIG.4A, the display212can display the representation image400aof the implant125in the anatomical structure124. The implant125can define one or more target locations126at which material can be removed from the anatomical structure124. In an example IM nailing operation, by viewing the display212that displays representation images based on the X-ray image data and the image sensor data, a medical professional can maneuver the patient or the surgical instrument203while viewing the patient and display212simultaneously, until the drill bit tip226ais located at a desired entry point of the paint (e.g., as shown inFIG.4C). In the IM nailing example, when the drill bit tip226ais centered over the respective target location126, the drill bit tip226amay be at the proper entry point for locking screws.

Referring now toFIG.4B, the display212can display the representation image400bof the implant125and the anatomical structure124. Thus, the display212can be configured to display a representation of a position of the cutting tip226aof the cutting instrument226relative to the target location126. The representation image400bcan depict, for example, the position of the cutting tip226athat is shown inFIG.6B.

The cutting tip226acan be configured to remove anatomical material from the one or more target locations126of the anatomical structure124. Further, as shown inFIG.4C, the tip226aof the cutting instrument226(e.g., drill bit) can be positioned on the anatomical structure124, for instance at the center of the target location126. The display212can be positioned so as to provide a line of sight to both the tip226aand the display212from a location proximate of the surgical instrument203, such that a medical professional can view both the representation images400band400c, the tip226a, and the anatomical structure124, so as to center the tip226aat the target location126.

In some embodiments, for instance based on a user selection via the user interface216, the surgical instrument assembly202can rotate the displayed representation images400a-400con the display212to a rotated orientation such that a vertical or horizontal direction on the display212corresponds with a vertical or horizontal direction, respectively, of movement of the surgical instrument203relative to the anatomical structure124. Thus, such representation images may be displayed as rotated relative to the actual position of the drill bit, the implant, and/or the anatomical structure.

Referring now toFIGS.5A-5C, the display212can also be configured to provide a visual indication, for instance an orientation image129, of an alignment of the cutting tip226awith respect to the desired trajectory (e.g., the central axis of the target hole126) based on the X-ray image data and the image sensor data. In an example, the display212is configured to display the representation images400a-400cin real time based on the image sensor data that is generated in real time by the image sensor209, and is configured to simultaneously display the orientation screens (e.g., orientation screens500a-500c) that include a visual indication of an orientation of the cutting instrument226. In an embodiment, the display includes more than one display, each of which displays a different one of a representation image or an orientation screen.

In an embodiment, the user can select an option via the user interface216to select which of the representation images, orientation screens, or depth information are displayed by the display212. In some embodiments, the display212can be separated, for instance split in half or split in thirds, such that any combination of the representation images, orientation screens, and depth information can be displayed by the display212at the same time. It will be understood that the examples described herein of images (e.g.,FIGS.4A-4C,5A-5C, and10A-20) that can be displayed by the display212are not exhaustive. The display212can provide a user with various information via a variety of arrangements or alternative visual depictions.

The visual indication of alignment, for instance the orientation image129, can be based on real time image sensor data of an orientation of the drill bit226relative to the fiducial markers213a(e.g., as shown inFIG.4D). The visual indication of alignment may be further based on the desired trajectory relative to the fiducial markers213a(e.g., the central axis of the target126).

For example, referring toFIGS.5A-5C, the orientation screens500a-500ccan include the orientation image129that can include a static region130and a movable indicator132. The movable indicator132can be representative of the orientation of the cutting instrument226. In an example, the cutting instrument226is oriented with the desired trajectory when the movable indicator132has a predetermined spatial relationship to the static region130. In an example, a hole is drilled in the anatomical structure124while the tip226aof the cutting instrument226(e.g., drill bit) is aligned with the target location126, and the movable indicator132has the predetermined spatial relationship to the static region130. It will be understood that the predetermined spatial relationship can vary as desired.

In some cases, for example, the cutting instrument226is oriented with the desired trajectory when the movable indicator132overlies the static region130. As shown inFIG.5C, for example, the cutting instrument226is oriented with the desired trajectory when the movable indicator132is within a boundary defined by the static region130.

Turning toFIGS.5D-5G, alternative alignment graphics are illustrated. For example,FIG.5Dillustrates circle240and arrow242visual indication of the alignment of the cutting instrument relative to the desired trajectory (and the target location126). A bulls-eye244may represent an aligned orientation and the arrow242may guide a user to the aligned orientation.FIG.5Eillustrates a bubble level246visual representation, where a static center248represents an aligned orientation.FIG.5Fillustrates a pair of circles250,252that may each represent a different portion of the drill bit226relative to the target location126. When each circle250,252is centered on the target location126, the drill bit226may be aligned.FIG.5Gillustrates a movable dot254that moves, with the orientation of the drill bit226, relative to a static dot256. The movable dot254being centered on the static dot256may indicate the drill bit226is aligned with the target location.

In an embodiment, when the drill bit tip is located within a predetermined distance from the center of the target126, the center of the static region target may change color. For example, the center of the static region may turn green. When both the drill bit tip and the alignment of the drill bit axis are within a predetermined range, the outline of the display may change color. For example, the outline of the display may turn green.

As described above with reference toFIGS.4A-4C, the display212can display representation images400a-400cand user interfaces associated with placing locking screws to secure an IM nail. Referring now toFIGS.13to20, the display212can additionally, or alternatively, display representation images and user interfaces associated with placing the implant125, for instance an IM nail. The display212can be configured to display representation images based on the image sensor data in combination with the orientation of a desired trajectory relative to the fiducial markers213a, which are fixed relative to the anatomical structure124in a manner discussed above. For example, representation image602(FIGS.13,14A,14B), representation image604(FIG.15), representation image606(FIG.16), representation image608(FIG.17), representation image610(FIG.18), and representation images630aand630b(FIG.20).

As used herein, unless otherwise specified, X-ray image data and X-ray image can be used interchangeably, without limitation. Referring in particular toFIG.14A, the display212can display the representation image602of the anatomical structure124(e.g., a representation of X-ray image data of the anatomical structure). In accordance with the illustrated example, the representation image602may include a representation of the cutting instrument226in real time relative to the representation of the anatomical structure124(e.g., based on the real time image sensor data that is based on the orientation of the fiducial markers213arelative to the image sensor209). The cutting instrument226as represented inFIG.14Amay be positioned to drill a hole in the anatomical structure224for the implant125.

In an example, a hole can be drilled so as to meet the IM canal of the anatomical structure or bone124. Thus, the hole can define a point of entry into the bone and a trajectory between the point of entry and the IM canal, and the implant125, for instance an IM nail or rod, can be inserted into the hole that is sized so as to receive the implant125. It is recognized herein that the desired trajectory (also referred to herein as an “appropriate trajectory”) and point of entry (e.g., to minimize pain) of the drilling operation can vary depending on the type of bone and/or the implant that is to be inserted. It is further recognized herein that the appropriate trajectory and point of entry might not be readily accessible in a given operating room, so that a given medical professional might rely on personal knowledge to estimate the appropriate trajectory and point of entry. Further still, even if the appropriate trajectory and point of entry are known, the drilling operation is commonly performed freehand, such that the actual trajectory and point of entry can vary from the appropriate trajectory and point of entry.

In an example embodiment, referring toFIGS.14B and17, the processor of the surgical instrument assembly202can identify or determine a boundary614, for instance a first or anteroposterior (AP) boundary615(FIGS.14B and15), or a second or lateral boundary617(FIGS.16and17), of the anatomical structure124. The boundary614can define a first outermost edge614aof the anatomical structure124and a second outermost edge614bof the anatomical structure124opposite the first outermost edge614a. In some examples, the processor can determine the boundary614by performing an edge detection process that is described in U.S. Patent Application Publication No. 2007/0274584 published Nov. 29, 2007 and entitled “Method and System for Detection of Bone Fractures,” the disclosure of which is incorporated by reference as if set forth in its entirety herein.

It will be understood that other edge detection algorithms may be performed as desired, and the edge detection processes mentioned above are presented for purposes of example. In some cases, the processor can identify the boundary614based on a user selection via the user interface216. For example, the display212can display an option, such as a manual alignment option646. The user, for instance a medical professional, can actuate the manual alignment option646, for instance by touch or the like. When the manual alignment option646is actuated, the user can manually overlay one or more images (e.g., a longitudinal axis618of the anatomical structure124) on the representation image602, such that the display212displays the one or more images on the representation image.

An example of an image that the user can manually overlay is the boundary614. By way of example, users can use a stylus, finger, or the like to manually overlay images on the X-ray data. In an example, the user can actuate the manual alignment option646to adjust the boundary614that is determined by the processing unit206of the surgical instrument assembly202. For example, the processing unit206can perform an edge detection process to determine the boundary614, but in some cases, the edge detection process can result in portions of the boundary614that are offset from the actual outermost edge of the anatomical structure124. For instance, the edge detection process might incorrectly identify a fracture in the anatomical structure124as a portion of the boundary614. In the example, the user can, via the user interface216, adjust the portion of the boundary614that is incorrectly identified as representing an outermost edge of the anatomical structure124. Thus, the surgical instrument assembly202can adjust at least a portion, for instance all, of the boundary614in response to the user actuating at least one of the options of the user interface216.

As shown inFIGS.14B,15,16, and17, the display212can overlay the boundary614on the representation images of the anatomical structure124, so as to display the boundaries614of the anatomical structure124. Referring toFIGS.18and20, the processing unit206of the surgical instrument assembly202can determine an axis616of the anatomical structure124. The processing unit206of the surgical instrument assembly202can determine a representation of a trajectory618that defines a point of entry620into the anatomical structure. Referring toFIGS.14B-18and20, the display212can overlay the representation of the trajectory618on the representation images of the anatomical structure124, so as to display the representation of the trajectory618relative to the anatomical structure124. The representation of the trajectory618can define a line along which a hole can be drilled so as to meet the IM canal of the anatomical structure124.

The representation of the trajectory618can be determined based X-ray image data of the anatomical structure124. For example, processing unit206may determine the trajectory618relative to the anatomical structure and the fiducial markers213a.

Further, referring toFIGS.18and20, the display212can overlay the axis616on the representation images610,630a,630bof the anatomical structure124, so as to display the axis616of the anatomical structure124.

In some embodiments, the axis616may define a centerline along a length of the anatomical structure. Referring toFIGS.14B-17, the trajectory can be coincident with the axis616, such that the representation of the trajectory618and the axis616can overlap each other. For example, the first outermost edge614acan be spaced from the second outermost edge614bso as to define a width of the anatomical structure that is substantially perpendicular to the length of the anatomical structure. Thus, the axis616can be equidistant from the first outermost edge614aand the second outermost edge614balong the length of the anatomical structure124. In some cases, the processing unit206can identify the axis616based on a user selection via the user interface216. For example, the user, for instance a medical professional, can actuate the manual alignment option646, for instance by touch or the like. When the manual alignment option646is actuated, the user can manually overlay one or more images on the X-ray data, such that the display212displays the one or more images on the X-ray data. An example of an image that the user can manually overlay is the axis616. As shown, the axis616is represented as a dashed line, though it will be understood that that the axis616can be alternatively represented as desired, for instance by a solid line. By way of example, users can use a stylus, finger, or the like to manually overlay images on the X-ray data.

In an embodiment, the user can actuate the manual alignment option646to adjust the axis616that is determined by the processing unit206of the surgical instrument assembly202based on the boundary614, in particular the first and second outermost edges614aand614b. Thus, the surgical instrument assembly202can adjust or determine at least a portion, for instance all, of the axis616in response to the user actuating at least one of the options of the user interface216. Further, the surgical instrument assembly202can determine the axis616of the anatomical structure124based on the boundary614of the anatomical structure124such that, if the boundary614of the anatomical structure changes, the axis616of the anatomical structure124changes in accordance with the changes to the boundary614. For example, the second outermost edge614bis adjusted away from first outermost edge614a, the surgical instrument assembly202can move the axis616toward the second outermost edge614b, such that the axis616can be displayed farther away from the first outermost edge614aas compared to where the axis616is displayed before the boundary614is adjusted.

The present disclosure provides embodiments that can lessen the number of X-ray images taken in an operating room, thereby decreasing the time it takes to perform a given operation. In some embodiments described above, only a single X-ray image may be taken before the surgeon begins surgery and is guided based on the real time image sensor data. In other embodiments, only two X-ray images may be taken before such surgery begins.

In an example, with reference toFIGS.14A-15and representation image630ainFIG.20, the display212can display the representation of the anatomical structure124from a first or an anteroposterior (AP) view. The processing unit206can determine the representation of the trajectory618that defines the point of entry620into the anatomical structure124. The display212can overlay the representation of the trajectory618on the representation image of the anatomical structure124, so as to display the representation of the trajectory618.

During use, the surgeon may indent or otherwise mark the patient's skin with the cutting instrument225at the point of entry (620) using the navigation based on the real time image sensor data. The surgeon may use another instrument to make an incision in the patient's skin to access the bone.

In an embodiment, the surgical instrument assembly may include a blade attached to the surgical instrument for making an incision to access the anatomical structure, while continually using the displayed navigation based on the real time image sensor data. In some embodiments, an incision tool includes a separate computer vision module that includes an incision image sensor that is configured to detect the fiducial markers and a blade of the incision tool in substantially the same manner as the image sensor described above with reference toFIG.2A-2D. In an embodiment, the image sensor of theFIGS.2A-2Dmay be configured to attach to the incision tool to detect the blade in substantially the same manner as the image sensor detects the cutting instrument. In some embodiments, the user may use the fiducial markers as landmarks to help visually inform their decision about where the incision should be made (e.g., when the X-ray image that the target hole is halfway between marker A and marker B the user may make an incision halfway between marker A and marker B).

When navigation is based on the image sensor data, the image sensor209may detect at least one fiducial marker213a. The processing unit206, for example, may determine the pose of each fiducial marker213awithin the world coordinate system.

For example, processor unit206may compare the detected pose of each fiducial marker213aand compare such to the pose determined from the X-ray image data. If the location of any marker(s)213ais outside a range of acceptable deviation (e.g., +/−0.5 mm), such marker(s) may not used for further processing or navigation steps. If sufficient markers are within the range of acceptable deviation, navigation processing may proceed or continue as usual, but without input based on the marker(s) that is outside the range of the acceptable deviation. If insufficient markers remain, the processing unit206may generate an alert that is provided to the user and may indicate mitigation options (e.g., by taking a new X-ray image, adjusting an orientation of the image sensor209, and/or finishing the procedure freehand).

In an embodiment, the processing unit generates a visual alert, an audible alert, or some other feedback method to indicate to the user that the fiducial markers detected by the image sensor are no longer sufficient to provide accurate navigation. As an example, movement of any one of the fiducial markers detected by the image sensor relative to another may result in the processing unit generating the alert.

The processing unit206may perform a best fit algorithm to determine how the fiducial markers213aas detected by the image sensor209match with a pre-determined object board. The processing unit206may execute a least squares algorithm to minimize re-projection error. If one or more fiducial markers213aincrease the error by more than a threshold amount, such fiducial markers213amay be determined to have moved and as a result not included in further calculations based on the fiducial markers213a. When the quantity of remaining (i.e., usable) fiducial markers is too low to provide accurate navigation, the processing unit206may generate an alert to indicate to the user to take mitigation steps. For example, the processing unit206may generate a visual or audible alert indicating that the user should replace the fiducial markers213aand take a new X-ray image to generate new X-ray image data based on the newly placed fiducial makers. In some embodiments, one or more markers may be placed on a static surface, such as the OR table, to provide an immobile reference point for the markers attached to the patient.

The pose of the cutting instrument226may be determined in the world coordinate system based on the pose of the fiducial markers213adetected by and relative to the image sensor209. The processing unit206may determine, for example, the pose of the cutting instrument226in the world coordinate system based on the determined pose of the fiducial markers213ain the world coordinate system and based on the pose of the cutting instrument226relative to the fiducial markers213athat is detected by the image sensor209. The processing unit206may determine in real time a distance from the cutting instrument226to the desired trajectory (e.g., an axis of the anatomical structure124or the implant125) based on the real time image sensor data.

The navigational graphics discussed above may guide the user in real time to reduce the distance from the cutting instrument226and the desired trajectory. In an embodiment, the processing unit determines in real time a location of the drill tip and an axis of the cutting instrument in the world coordinate system based on the image sensor data. Thus, the processing unit may determine an orientation or position change to the cutting instrument that would align the cutting instrument with the desired trajectory. The processing unit may generate a display indicating to the user how to orient and/or position the cutting instrument to reach alignment, for example, as discussed above.

In an embodiment, only the drill tip of the cutting instrument is determined by the processing unit. The axis of the cutting instrument relative to the image sensor may be predetermined and/or the axis may be determined based on feedback data from the accelerometer of the surgical instrument assembly. In some embodiments, the orientation of the cutting instrument may be determined based on the image sensor data as a redundancy to double check the feedback data from the accelerometer, or visa versa.

During a drilling operation, the user may place the drill tip226aon the bone and confirm based on the navigation graphics that the cutting instrument226is aligned with desired trajectory (e.g., represented as over the center of the target hole126and represented as at a desired angle relative to target hole126as shown inFIG.18). If the cutting instrument226is oriented outside of a predetermined tolerance and/or positioned outside of a predetermined position (e.g., as exemplified inFIGS.15and16), the navigational graphics may indicate to the user how to adjust the cutting instrument226to achieve alignment.

In some embodiments, the processing unit206can determine the representation of the trajectory618in response to a user's selection via the user interface216. For example, the display212can display an option, such as an automated alignment option622. The user, for instance a medical professional, can actuate the automated alignment option622, for instance by touch or the like. When the automated alignment option622is input, the processing unit206of the surgical instrument assembly202can determine the representation of the trajectory618that defines the point of entry620into the anatomical structure124. The surgical instrument assembly can also determine the axis616or the boundary614, or both the axis616and the boundary614, responsive to the automated alignment option622being input. Further, in response to the automated alignment option622being actuated, the display212can overlay at least one of, for instance only one of, for instance any combination of, the representation of the trajectory618, the axis616, and the boundary614, on the representation images of the anatomical structure124, so as to display the representation of the trajectory618, the axis616, and/or the boundary614.

In some examples, the surgical instrument assembly202can determine the representation of the trajectory618based on technique information, for instance technique information stored in the memory214. Such technique information can include appropriate trajectories for drilling a hole in various bones for placing an IM nail. Based on the technique information, the surgical instrument assembly202can determine the representation of the trajectory. By way of example, the technique information may stipulate that the trajectory for a given bone viewed from the AP perspective is 5 degrees lateral of an axis that is measured from a point just below the lesser trochanter. Continuing with the example, the technique information may stipulate that the trajectory for the given bone from the lateral perspective is centered in the greater trochanter and in line with the medullary canal. In an example, the type of bone and nail can be input into the processor via the user interface216, and the view (e.g., lateral or AP) that corresponds to a representation of an anatomical structure can be input into the processor via the user interface216. In response, the processor can retrieve technique information that corresponds to the view of the anatomical structure, the type of bone, and the nail. Based on the technique information that is retrieved, the trajectory can be determined. In some cases, the processor first determines the boundary614, and then determines the axis616based on the boundary. The representation of the trajectory618can be determined based on the axis616and the technique information. For example, the technique information may indicate that that the trajectory is coincident with the axis616in a first view, and angularly offset from the axis by a specific angle in a second view that is substantially perpendicular to the first view (seeFIG.19).

Referring toFIG.19, a given user can retrieve technique information from the surgical instrument assembly actuating a user selection via the user interface216. For example, the user selection can cause the display212to display technique information650aand650b. The technique information650acan include a graphical depiction of an appropriate trajectory652afrom an AP view. The technique information650bcan include a graphical depiction of an appropriate trajectory652bfrom a lateral view. The technique information that can be displayed can include instructions654in text for placing an IM nail, among other operations. In an example, responsive to a user selection, the user interface216can render audible instructions associated with IM nailing operations, among others.

In some cases, a given user, for instance a medical profession, can utilize the technique information rendered by the surgical instrument assembly202to manually overlay the representation of the trajectory618on a representation image. For example, the user can actuate the manual alignment option646, for instance by touch or the like. When the manual alignment option646is actuated, the user can manually overlay the representation of the trajectory618, such that the display212displays the trajectory618on the representation image. The representation of the trajectory618can define a solid line, a dashed line, or the like. In an example, the user can actuate the manual alignment option646to adjust the axis616that is determined by the processor of the surgical instrument assembly202after the automated alignment option622is selected. The surgical instrument assembly202can adjust or determine at least a portion, for instance all, of the representation of the trajectory in response to the user actuating at least one of the options of the user interface216. Thus, the processor of the surgical instrument assembly202can adjust the representation of the trajectory so as to define a new representation of the trajectory, and the display212can overlay the new representation of the new trajectory on the representation image of the anatomical structure, so as to display the new representation of the new trajectory. In an example, the processor can adjust the representation of the trajectory in response to the user actuating at least one of the options of the user interface216.

In some embodiments, the processing unit determines the world coordinates of the representation of the trajectory, as discussed above with respect to the desired trajectory. The processing unit may also determine a pose of the cutting instrument relative to the representation of the trajectory based on real time image sensor data, as discussed above with respect to the desired trajectory.

Referring toFIG.14B, by viewing the representation of the trajectory618and the cutting instrument226that is viewable on the X-ray image602, a user can move the cutting instrument226to align with the representation of the trajectory, as shown in the representation image604ofFIG.15. In an embodiment, when the cutting instrument226is aligned with the representation of the trajectory618, the processing unit206may instruct the display212to display different representation images606or608, which may represent a viewpoint of the anatomical structure124different from that of the representation images602and604. For example, the representation images606or608may represent a viewpoint that is perpendicular to the first or AP view shown inFIGS.14B and15.

Referring toFIGS.14B,15, and20, the representation of the trajectory618can be referred to as a first representation618aof the trajectory from a first perspective, for instance from an AP perspective. In an example, referring toFIGS.16,17and20, the surgical instrument assembly202can determine a second representation618bof the trajectory that defines the point of entry620into the anatomical structure124. The second representation618bcan be from a second perspective. By way of example, the second perspective can be approximately periocular to the first perspective, such that that first perspective can define an AP view, and the second perspective can define a lateral view. The second representation618bof the trajectory can be determined and displayed in accordance with any of the embodiments described herein for determining and displaying the representation of the trajectory618.

Referring toFIGS.14B-18, the display212can display a position of the cutting tip226arelative to the point of entry620of the anatomical structure. By viewing the second representation618bof the trajectory and the cutting instrument226that is viewable on the representation images606and608, a user can move the cutting instrument226, and thus the cutting tip226a, to align with the second representation618of the trajectory. Alternatively, in an automated scenario, the cutting instrument226can be moved automatically so as to align with the second representation618bof the trajectory.

In some cases, when the cutting instrument226, and thus the cutting tip226a, is aligned with the first representation of the trajectory618aand the second representation618bof the trajectory, the drilling operation can begin, as the cutting instrument226is aligned with the appropriate point of entry and trajectory, which can be determined from the technique information described herein. The display212can be positioned so as to provide a line of sight to both the tip226aand the display212from a location proximate of the surgical instrument203, such that a medical professional can view both the representation images, and thus the tip226a, and the anatomical structure124, so as to center the tip226aat the point of entry620.

Referring now toFIG.18, the display212can also be configured to provide a visual indication, for instance the orientation image129, of an alignment of the cutting instrument226with respect to the first representation618aof the trajectory and the second representation618bof the trajectory. The visual indication of alignment, for instance the orientation image129, can be based on the image sensor data representing a relative orientation of the cutting instrument226relative to the trajectory.

For example, referring toFIG.18, the orientation image129can include the static region130and the movable indicator132. The movable indicator132can be representative of the orientation of the cutting instrument226. In an example, the cutting instrument226is oriented with the first and second representations of the trajectory618aand618bwhen the movable indicator132has a predetermined spatial relationship to the static region130. In an example, a hole is drilled in the anatomical structure124while the cutting instrument226(e.g., drill bit) is aligned with first and second representations of the trajectory, and the movable indicator132has the predetermined spatial relationship to the static region130. It will be understood that the predetermined spatial relationship can vary as desired. In some cases, for example, the cutting instrument226is oriented with the first and second representations of the trajectory when the movable indicator132overlies the static region130. In some cases, the cutting instrument226is oriented with the first and second representations of the trajectory when the movable indicator132is within a boundary defined by the static region130.

Referring now toFIGS.10A-12, the display212can also be configured to provide a visual indication, for instance a depth gauge image262, of the depth of the cutting tip226awith respect to one or more portions of the anatomical structure124. The processing unit206may determine the depth of the cutting tip226ain real time based on the real time image sensor data. For example, the processor may determine the pose of the fiducial markers213arelative to the one or more portions of the anatomical structure124in a similar manner as discussed above, based upon the X-ray image data. The processor may determine the real time position of the cutting tip226arelative to the fiducial markers213abased upon the real time image sensor data, and thereby determine the position of the cutting tip226arelative to the one or more portions of the anatomical structure124.

In an example, referring toFIG.9, the anatomical structure124may define a first or near cortex123and a second or far cortex127opposite the first cortex123along a first direction D1, which can be in the direction of drilling (and the desired trajectory). The first cortex123can define a first or near surface123aand a second or far surface123bopposite the first surface123aalong the first direction D1. Similarly, the second cortex127can define a first or near surface127aand a second or far surface127bopposite the first surface127aalong the first direction D1, which can also be along direction of drilling and the desired trajectory. The anatomical structure124can define a hollow portion131. For example, the hollow portion131can be defined between the second surface123bof the first cortex123and the first surface127bof the second cortex127.

The implant125may be disposed in the hollow portion131. The target hole126of the implant125may be aligned with the cutting instrument226and the desired trajectory. In an embodiment, the processor determines and instructs the display to display a representation of the depth of the cutting tip with respect to one or more portions of the implant, based on the real time image sensor data.

The visual indication of depth, for instance the depth gauge image262, can change as the cutting instrument226, in particular the cutting tip226a, travels into the anatomical structure124and/or the implant125. In particular, the depth gauge image262can include data that can change when the cutting instrument tip226acontacts the respective first and second surfaces of the first cortex123and the second cortex127.

In an embodiment, the depth measurement may be done using a reflector plate system with the distance sensor, as discussed above. In some embodiments, the processing unit may track fiducial markers on the reflector plate to determine an acceleration that takes place as the cutting instrument drills through the two cortices. The processing unit may determine a length of bone drilled through based on the acceleration. In some embodiments, the distance sensor may be used in conjunction with the processor determining the acceleration to give redundant measurements. In an embodiment, the distance sensor may not be provided, and instead the processor determining the acceleration may be used for depth measurement.

In an example operation, referring first toFIGS.10A and12, which depict an example depth gauge screen1000aand an example split screen1200, respectively, the depth gauge image262may be configured to measure a first distance of a reference location relative to a portion of the anatomical structure124, and the display212may be configured to indicate a second distance of the cutting tip226arelative to the portion of the anatomical structure124. The processor may determine the depth based on the first distance as the surgical instrument203drills a hole. The display212can be configured to indicate the second distance as the surgical instrument drills a hole, so as to indicate the second distance in real-time. The first cortex123can define the portion of the anatomical structure124. In an example, the first cortex123, in particular the first surface123aof the first cortex123, may define the reference location from which the distance from the reference location is determined (e.g., with a distance sensor and/or based on X-ray image data and/or the image sensor data). In an example, the cutting tip defines the reference location, such that the first distance is equal to the second distance.

The display212can display the depth gauge screen1000aand the example split screen1000. In the illustrated examples, the total drill depth indication264indicates zero (0) when the cutting instrument tip226aabuts the first surface123aof the first cortex123. In an embodiment, the processing unit may be configured such that the total drill depth indication264indicates zero (0) when a drill sleeve abuts the first surface123aof the first cortex123.

The surgical instrument203can be configured to drill a hole in the first direction D1from the first cortex123to toward the second cortex127. Thus, the total drill depth indication264can indicate zero (0) before a drilling operation, whereby the cutting instrument tip226aenters the anatomical structure124during the drilling operation. Referring also toFIGS.10B and11, which depict an example depth gauge screen1000band an example split screen1100, respectively, as the drilling operation proceeds and the cutting instrument tip226atravels through the first cortex123, the total drill depth indication264can increase so as to indicate the real-time distance that the cutting instrument tip226ahas traveled with respect to the first surface123aof the first cortex123. As shown, the indications of the depth gauge image262are rendered in millimeters, though it will be understood that the indications may be rendered in any alternative units.

The depth gauge image262can further include a recent cortex exit point indication266that indicates the distance from the cutting instrument tip226ato the far surface of the cortex that was most recently drilled. Thus, the display212can be configured to indicate a third distance when the cutting tip226aexits the first cortex123, wherein the third distance can represent a width of the first cortex123along the first direction D1. As an example, when the cutting instrument tip226atravels along the first direction D1, which can be the desired trajectory, so as to exit the second surface123bof the first cortex123, the recent cortex exit point indication266indicates the distance from the first surface123aof the first cortex123to the second surface123bof the first cortex123. Thus, in an example, at the moment that the cutting instrument tip226atravels through the second surface123bof the first cortex123, the recent cortex exit point indication266can indicate the same value as the total drill depth indication264.

As drilling is performed, the processor may smooth the image sensor data to minimize errors in the navigational guidance caused by vibration, noise, or drill bit deflection. For example, the processor may determine the cutting instrument226apose as a rolling average. In an embodiment, the display may minimize the alignment information shown during drilling. The processor may alert the user when the cutting instrument is out of tolerance, even when the alignment information is minimized. In some embodiments, the tolerance is determined based on the typical vibration, noise, and/or drill bit deflection seen during drilling.

In some embodiments, the display only displays alignment guidance prior to drilling. For example, the display may not display alignment or other navigation guidance feedback while drilling occurs.

When the cutting instrument tip226atravels along the first direction D1so as to exit the second surface127bof the second cortex127, the recent cortex exit point indication266may display the distance from the first surface123aof the first cortex123to the second surface127bof the second cortex127. Thus, the display212may be configured to indicate a fourth distance when the cutting tip226aexits the second cortex127, and the fourth distance can represent a bone width of the bone along the first direction D1. The display212can be configured to indicate the second distance, the third distance, and the fourth distance at the same time. Further, at the moment that the cutting instrument tip226atravels through the second surface127bof the second cortex127, the recent cortex exit point indication266can indicate the same value as the total drill depth indication264. The depth gauge image262can further include a previous cortex exit point indication268that displays an indication or value associated with the previous, but not most recent, cortex exit point.

Thus, continuing with the example, when the cutting instrument tip226aexits the second surface127bof the second cortex127, the previous cortex exit point268may display the distance from the first surface123aof the first cortex123to the second surface123bof the first cortex123. Thus, the value displayed in the recent cortex exit point indication266may be moved to the previous cortex exit point indication268. As the cutting instrument tip226atravels away from the second surface127bof the second cortex127, the total drill depth indication264can increase so as to indicate the real-time distance that the cutting instrument tip226ahas traveled with respect to the first surface123aof the first cortex123, as exemplified byFIGS.10B and11.

The user can view the depth gauge image262while the surgical instrument203operates, either under user control or autonomously, so as to better perform a drilling operation. For example, the user can view the total drill depth indication264while performing a drilling operation, so as to control the surgical instrument based on the total drill depth indication264. The surgical instrument203can be controlled based on the information in the depth gauge image262so that the cutting instrument203does not enter unwanted portions of the anatomy, such as soft tissue or a far cortex that is not intended to be drilled, either wholly or in part. In some cases, a user can view the depth gauge image262, in particular the total drill depth indication264or the recent cortex exit point indication266, to match the length of a screw with respective holes that are drilled, instead of having to measure the holes after the drilling operation is performed. In an example, the computing device204stores an inventory of available screws, such that a screw is automatically matched to a hole that is drilled, based on the depth of the hole in the anatomical structure124. In an example, a user can actuate a select screw option on the user interface216, so that a screw is selected that corresponds to one of the indications on the depth gauge image262, for instance the recent cortex exit point indication266or the total drill depth indication262.

Thus, in operation, the display212can receive and display a plurality of representation images of the anatomical structure in real-time, based on the real time image sensor data. The display212can display the orientation image129and the depth gauge image262, in particular the total drill depth indication262, as the surgical instrument203is operated. For example, the depth gauge image262can be representative of distances that the cutting instrument203as moved. In an embodiment, the representation images, the orientation images, and the depth gauge images are displayed by the display at the same time. As the cutting instrument203moves along a drilling direction, the distance displayed by the display may change, so as to update the distance in real-time (e.g., based on the real time image sensor data).

In an example, referring toFIG.6A, the surgical instrument203can be operated along the first direction D1that is parallel to the desired trajectory, so as to drill a hole along the first direction D1. During drilling, for example, as the orientation of the cutting instrument226moves away from the zero value, the movable indicator132can move away from the static region130. The movable indicator132can move relative to the static region130at the same time that the orientation of the cutting instrument226moves relative to the zero value, such that the movable indicator132provides a real-time representation of the orientation of the cutting instrument226. For example, as the proximal end226bof the cutting instrument226moves along a second direction D2relative to the cutting tip226aof the cutting instrument226, the movable indicator132can move along the second direction D2(e.g., seeFIG.5A). The second direction D2can be perpendicular to the first direction D1. Similarly, as the proximal end226bof the cutting instrument226moves along a third direction D3relative to the cutting tip226aof the cutting instrument226, the movable indicator132can move along the third direction D3(e.g., seeFIG.5B). The third direction D3can be perpendicular to both the first and second directions D1and D2, respectively. Further, it will be understood that as the proximal end226bof the cutting instrument226moves along both the second and third directions relative to the cutting tip226aof the cutting instrument226, the movable indicator132can move along both the second and third directions D3. Further, the orientation screens500a-500ccan include a numerical representation136of the orientation of the cutting instrument226along the second and third directions D2and D3.

Referring in particular toFIG.5C, when the cutting instrument226is oriented in accordance with the zero value, the movable indicator132can be positioned within a boundary defined by the static region130. Further, in some cases, when the cutting instrument226is precisely aligned with the desired trajectory, the numerical representation136may indicate that zero values associated with both the second and third directions. By way of an IM nailing example, a medical professional can maintain the orientation image129illustrated inFIG.5Cwhile drilling, so as to drill holes having the appropriate orientation at the target locations126.

The above processes may be repeated to select or determine a new desired trajectory to drill a new hole. For example, the C-arm may generate a new X-ray image including a new target portion of an implant or anatomy and the fiducials213afixed relative to the new target portion. A new desired trajectory may be determined relative to the fiducials213aand oriented in a world coordinate system based on the new X-ray image data of the new X-ray image. The pose of the cutting instrument226relative to the new desired trajectory may be determined based on the real time image sensor data. In an embodiment, the original X-ray image data shows a sufficient amount of the new target portion, and thus the pose of the cutting instrument226relative to the new desired trajectory may be determined without the new X-ray image. For example, the user may select the new target portion on the display and set the new desired trajectory without the new X-ray image data.

After drilling is completed, a final X-ray image may be taken of the target portion (or portions) to confirm that the distal locking, for example, completed as desired. In an embodiment, fluoroscopic video of the target portion is taken to confirm completion as desired by the user.

Referring now toFIGS.21A-21D, a second embodiment of the surgical instrument assembly is shown. It is to be appreciated that the second embodiment can be similar to the first embodiment of the surgical instrument assembly shown inFIGS.1-2D. Accordingly, the same reference numbers used above with reference to the first embodiment can be also used with a “prime” notation in reference to a second embodiment. It is also to be appreciated that, unless otherwise set forth below, the components (and features thereof) of the surgical instrument assembly202′ of the second embodiment can be similar to those of the first embodiment.

The surgical instrument assembly202′ may include the surgical instrument203, a computing device204′, an image sensor209′, and an attachment member218′ configured to attach the image sensor209′ to the surgical instrument203. The computing device204′ may not include a measuring device (e.g., the measuring device211described above).

The image sensor209′ may be attached to a top of the attachment member218′, may be attached to the surgical instrument. The image sensor209′ may be configured to face forward, away from the display212′. For example, the image sensor209′ may face forward along the longitudinal axis A. The image sensor209′ may be rotatably attached to the attachment member218′, such that the image sensor209′ may face forward at a downward angle toward a longitudinal axis A or an upward angle away from the longitudinal axis A. In an embodiment, the image sensor209′ is fixed relative to the attachment member218′ such that the image sensor209′ is not rotatable relative to the attachment member218′.

Any of the above processing steps may be performed by the processing unit206and/or stored as instructions by the memory portion214. For example, each detecting, determining, generating, and/or outputting step discussed above may be performed by the processing unit206and/or stored as instructions by the memory portion214. The memory portion214may be a non-transitory memory portion.

While example embodiments of devices for executing the disclosed techniques are described herein, the underlying concepts can be applied to any computing device, processor, or system capable of communicating and presenting information as described herein. The various techniques described herein can be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatuses described herein can be implemented, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embodied in tangible non-transitory storage media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium (computer-readable storage medium), wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for performing the techniques described herein. In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device, for instance a display. The display can be configured to display visual information. For instance, the displayed visual information can include fluoroscopic data such as X-ray images, fluoroscopic images, orientation screens, or computer-generated visual representations.

The program(s) can be implemented in assembly or machine language, if desired. The language can be a compiled or interpreted language, and combined with hardware implementations.

The techniques described herein also can be practiced via communications embodied in the form of program code that is transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via any other form of transmission. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique apparatus that operates to invoke the functionality described herein. Additionally, any storage techniques used in connection with the techniques described herein can invariably be a combination of hardware and software.

Certain terminology is used in the following description for convenience only and is not limiting. The words “right”, “left”, “lower” and “upper” designate directions in the drawings to which reference is made. The words “inner”, “internal”, and “interior” refer to directions towards the geometric center of the anatomical structure and/or the implant, while the words “outer”, “external”, and “exterior” refer to directions away from the geometric center of the implant or the anatomical structure. The words, “anterior”, “posterior”, “superior,” “inferior,” “medial,” “lateral,” and related words and/or phrases are used to designate various positions and orientations in the human body to which reference is made. When these words are used in relation to the implant or the anatomical structure, they are to be understood as referring to the relative positions of the respective anatomical structure or the implant as implanted in the body as shown inFIG.9. The terminology includes the above-listed words, derivatives thereof and words of similar import.