Surgical Imaging And Display System, And Related Methods

A medical imaging system includes a robotic arm carrying a fluoroscopic imager for generating fluoroscopic image data of anatomy along a beam axis. The arm can adjust a relative position between the fluoroscopic imager and the anatomy. A video imager generates video image data of the anatomy along a sightline axis. A marker is positionable relative to the anatomy and defines a feature for capture in the fluoroscopic and video image data. A processor is configured to execute instructions upon the fluoroscopic and video image data and: (a) register a reference position of the feature relative to the anatomy in the fluoroscopic and video image data; and (b) generate an augmented image stream showing the fluoroscopic or video image data overlaid onto the other such that the reference positions are co-registered. The system includes a display configured to present the augmented image stream of the anatomy substantially in real time.

TECHNICAL FIELD

The present invention relates to systems that can be used in conjunction with medical imaging.

BACKGROUND

A C-arm, or a mobile intensifier device, is one example of a medical imaging device that is based on X-ray technology. The name C-arm is derived from the C-shaped arm used to connect an X-ray source and an X-ray detector with one another. Various medical imaging devices, such as a C-arm device, can perform fluoroscopy, which is a type of medical imaging that shows a continuous X-ray image on a monitor. During a fluoroscopy procedure, the X-ray source or transmitter emits X-rays that penetrate a patient's body. The X-ray detector or image intensifier converts the X-rays that pass through the body into a visible image that is displayed on a monitor of the medical imaging device. Medical professionals can use such imaging devices, for example, to assess bone fractures, guide surgical procedures, or verify results of surgical repairs. Because medical imaging devices such as a C-arm device can display high-resolution X-ray images in real time, a physician can monitor progress at any time during an operation, and thus can take appropriate actions based on the displayed images.

In various embodiments described herein, images provided by imaging devices are transmitted in real-time to a display that can be mounted to an apparatus of the surgical system, such as a workstation near the operating table or directly on a surgical instrument, such that fluoroscopic imaging provided by the imaging device can be viewed by a medical professional as the medical professional operates and views a working end of the surgical instrument. The display can receive the images in real-time, such that the images are displayed by the display at the same time that the images are generated by the imaging device.

Monitoring the images, however, is often challenging during certain procedures, for instance during procedures in which attention must be paid to the patient's anatomy as well as the display of the medical imaging device. For example, aligning a drill bit to a distal locking hole can be difficult if a medical professional is required to maneuver the drill while viewing the display of the medical imaging device that is outside of the field of view of the medical procedure.

SUMMARY

According to an embodiment of the present disclosure, a medical imaging system includes a robotic arm carrying a fluoroscopic imaging device having an x-ray transmitter, wherein the fluoroscopic imaging device is configured to generate fluoroscopic image data of an anatomical structure along a beam axis. The robotic arm is manipulatable for adjusting a relative position between the fluoroscopic imaging device and the anatomical structure. The system also includes a video imaging device configured to generate video image data of the anatomical structure along a camera sightline axis, and a marker that is can be positioned with respect to the anatomical structure. The marker defines at least one reference feature configured to be captured in the fluoroscopic image data and the video image data. A processor is in communication with the fluoroscopic imaging device and the video imaging device and also with a memory having instructions stored therein. The processor is configured to execute the instructions upon the fluoroscopic image data and the video image data and responsively: (a) register a reference position of the at least one reference feature relative to the anatomical structure in the fluoroscopic image data and the video image data; and (b) generate an augmented image stream that shows one of the fluoroscopic image data and the video image data overlaid onto the other of the fluoroscopic image data and the video image data such that the reference positions are co-registered. The system also includes a display in communication with the processor, wherein the display is configured to present the augmented image stream of the anatomical structure substantially in real time.

According to another embodiment of the present disclosure, a method includes steps of generating a fluoroscopic stream of images of an anatomical structure, generating a video stream of images of the anatomical structure, co-registering the fluoroscopic stream of images with the video stream of images, and depicting, on a display, an augmented image stream that includes the co-registered fluoroscopic stream of images overlaid over the co-registered video stream of images.

According to yet another embodiment of the present disclosure, a surgical system includes a robotic arm carrying a fluoroscopic imaging device having an x-ray transmitter, wherein the fluoroscopic imaging device is configured to generate a first stream of fluoroscopic images of an anatomical structure along a first beam axis at a first orientation relative to the anatomical structure. The fluoroscopic imaging device is also configured to generate a second stream of fluoroscopic images of the anatomical structure along a second beam axis at a second orientation relative to the anatomical structure, wherein the second beam axis intersects the first beam axis and is substantially perpendicular to the first beam axis. The robotic arm is manipulatable for adjusting a relative position between the fluoroscopic imaging device and the anatomical structure. The system includes a processor in communication with the fluoroscopic imaging device and the robotic arm. The processor is further in communication with a memory having instructions stored therein, such that the processor is configured to execute the instructions upon the first and second streams of fluoroscopic images and responsively: (a) identify at least one anchor hole in an implant that resides within the anatomical structure; (b) reposition the fluoroscopic imaging device so that the first beam axis extends orthogonal to the at least one anchor hole; and (c) plot, in the second stream of fluoroscopic images, a reference axis that extends centrally through the at least one hole. The system also includes a display in communication with the processor, wherein the display is configured to depict an augmented version of the second stream of fluoroscopic images that shows the reference axis overlaying the anatomical structure.

According to an additional embodiment of the present disclosure, a method includes steps of generating a first fluoroscopic stream of images along a first beam axis, such that the first fluoroscopic stream shows an implant residing in an anatomical structure. A second fluoroscopic stream of images of the anatomical structure is generated along a second beam axis that intersects the first beam axis at an angle. The method includes processing the first and second fluoroscopic streams of images with a processor in communication with memory. This processing step comprises (a) identifying a reference feature of the implant, (b) calculating a pixel ratio of the reference feature in pixels per unit length, (c) adjusting an orientation of the first beam axis so that it extends orthogonal to the reference feature, (d) generating a reference axis extending centrally through the reference feature such that the reference axis is parallel with the first beam axis, and (e) depicting the second image stream on a display, such that the reference axis is depicted in the second image stream overlaying the anatomical structure.

The foregoing summarizes only a few aspects of the present disclosure and is not intended to be reflective of the full scope of the present disclosure. Additional features and advantages of the disclosure are set forth in the following description, may be apparent from the description, or may be learned by practicing the invention. Moreover, both the foregoing summary and following detailed description are exemplary and explanatory and are intended to provide further explanation of the disclosure.

DETAILED DESCRIPTION

In various embodiments described herein, images provided by imaging devices are transmitted in real-time to a display that can be mounted to an apparatus of the surgical system, such as a workstation near the operating table or directly on a surgical instrument, such that fluoroscopic imaging provided by the imaging device can be viewed by a medical professional as the medical professional operates and views a working end of the surgical instrument. The display can receive the images in real-time, such that the images are displayed by the display at the same time that the images are generated by the imaging device.

However, fluoroscopic images alone can omit critical information about patient anatomy and/or surgical components at a surgical treatment site, such as the location and orientation of target features of an implant with respect to the surgeon, according to one non-limiting example, and/or the precise spatial relationships between various portions of the anatomy, according to another non-limiting example, and/or a combination of the foregoing examples of critical information. Accordingly, an enhanced surgical imaging system that can generate and display augmented fluoroscopic images containing critical supplemental information would provide numerous benefits to the patient, for example, by allowing surgeons to complete surgical procedures with greater accuracy and more efficiently, thereby reducing the amount of X-ray exposure imposed on the patient (and also on the surgeon and staff).

The following disclosure describes various embodiments of surgical imaging systems that employ a fluoroscopic imaging device with an additional imaging device and uses the image data from both imaging devices to generate and display augmented fluoroscopic images that presents information obtained from both imaging devices. These augmented fluoroscopic images provide the surgeon with critical supplemental information necessary to complete various surgical procedures with greater accuracy and efficiency. By way of non-limiting examples, the various embodiments described below are expected to reduce the time necessary to complete an intramedullary (IM) nailing procedure, particularly by providing faster and more accurate techniques for determining necessary anchor length for distal locking, and also by providing simpler techniques for targeting distal locking holes of the IM nail.

In one example, the display presents an augmented image stream that includes fluoroscopic images of the treatment site paired with and superimposed onto video images of the treatment site in a continuous “augmented reality” stream, allowing the surgeon to more rapidly identify the location of distal locking holes relative to a tip of an associated surgical drill. In this example, the video camera can be mounted to the C-arm or the instrument (e.g., a surgical drill). In another example, the display presents an augmented image stream generated from two separate but intersecting fluoroscopic image streams, which allows a control system to identify target features of an implant residing in an anatomical structure and also to calculate the required orientation and length of anchors in three-dimensional (3D) space for insertion through anchor holes of the implant for anchorage to the anatomical structure. It should be appreciated that the foregoing examples are provided as non-limiting examples of the surgical imaging systems of the present disclosure.

As an initial matter, because fluoroscopy is a type of medical imaging that shows a continuous X-ray image on a monitor, the terms fluoroscopic data, fluoroscopic image, video data, and X-ray image may be used interchangeably herein, without limitation, unless otherwise specified. Thus, an X-ray image may refer to an image generated during a fluoroscopic procedure in which an X-ray beam is passed through the anatomy of a patient. Further, it will be understood that fluoroscopic data can include an X-ray image, video data, or computer-generated visual representations. Thus, fluoroscopic data can include still images or moving images.

Referring toFIG.1, an example surgical imaging system102is shown for generating and displaying augmented imagery, such as a stream of augmented images, showing an anatomical structure4during a medical imaging procedure, such as a surgical imaging procedure. In particular, the surgical imaging system102is configured such that the augmented imagery includes a first stream of images, such as a stream of fluoroscopic images, of a surgical treatment site matched with and overlapped with a second stream of images, such as a stream of video images, of the treatment site. Accordingly, the surgical imaging system102of the present embodiment can be referred to as an “augmented reality” (AR) surgical imaging system102, and the augmented imagery can be referred to as augmented reality (AR) imagery. The AR surgical imaging system102can include an imaging station103that includes a positioning mechanism, such as a robotic arm110, that carries a first imaging device104, such as a fluoroscopic imaging device104. The fluoroscopic imaging device104is configured to generate fluoroscopic image data, such as X-ray images, including a continuous stream of X-ray images, of the anatomical structure4.

The robotic arm110can be a C-arm or similar type device, by way of non-limiting example. The fluoroscopic imaging device104can include an X-ray generator or transmitter106configured to transmit X-rays through a body (e.g., bone) along a central beam axis115(also referred to herein as the “beam axis”115). The fluoroscopic imaging device104can also include an X-ray detector or receiver108configured to receive the X-rays from the X-ray transmitter106. Thus, the fluoroscopic imaging device104can define a direction of X-ray travel128from the X-ray transmitter106to the X-ray receiver108. The direction of X-ray travel128is parallel and/or colinear with the beam axis115. The X-ray transmitter106can define a flat surface106athat faces the X-ray receiver108. The area between the X-ray transmitter106and detector108can be referred to as the “imaging zone”6of the fluoroscopic imaging device104. The robotic arm110can physically connect the X-ray transmitter106with the X-ray receiver108.

The fluoroscopic imaging device104is configured to be in communication with an AR display112that is configured to display the AR imagery, which is generated in part from the fluoroscopic image data, as described in more detail below.

The AR surgical imaging system102can include a support apparatus140, such as a table140, for supporting a patient during the medical imaging procedure so that the anatomical region of interest (ROI) (e.g., the anatomical structure4at the surgical treatment site) is positioned between the X-ray transmitter106and the X-ray detector108and is thereby intersected by the X-rays.

The robotic arm110is preferably manipulatable with respect to one or more axes of movement for adjusting a relative position between the fluoroscopic imaging device104and the anatomical structure4. For example, the imaging station103can include a base150that supports the robotic arm110. The robotic arm110can include an actuation mechanism152that adjusts the position of the robotic arm110with respect to the base150, such as along one or more axes of movement. For example, the actuation mechanism152can be configured to pivot the robotic arm110about a central pivot axis154, which can extend centrally between the X-ray transmitter and detector106,108along a lateral direction Y and intersect the beam axis115perpendicularly at a central reference point155. Additionally or alternatively, the actuation mechanism152can translate the robotic arm110forward and rearward along a longitudinal axis156oriented along a longitudinal direction X. The actuation mechanism152can additionally or alternatively raise and lower the robotic arm110along a vertical axis158oriented along a vertical direction Z. The longitudinal, lateral, and vertical directions X, Y, Z can be substantially perpendicular to each other. The actuation mechanism152can optionally further pivot the robotic arm110about one or both of the longitudinal and vertical axes156,158. In manner described above, the robotic arm110can be provided with multi-axis adjustability for obtaining images of the anatomical structure4at precise locations and orientations. By way of a non-limiting example, the table140(and the anatomical structure4thereon) can be brought into the imaging zone6, and the actuation mechanism152can be employed to manipulate the relative position between the robotic arm110and the anatomical structure4such that the central reference point155is centered at a location of interest of the anatomical structure4. From this centered position, the robotic arm110can be rotated as needed, such as about axis154, to obtain fluoroscopic image data at multiple angles and orientations with the location of interest (i.e., at the central reference point155) centered in the images.

The AR surgical imaging system102includes a second imaging device105, which in the present embodiment is preferably a video camera105. The first and second imaging devices104,105can define an imaging array. As shown, the camera105can be mounted to the robotic arm110in a manner to capture video images of a field of view of the fluoroscopic imaging device104. In other embodiments (seeFIG.4A), the camera105can be remote from the robotic arm110, as will be described in more detail below. The camera105is configured to generate second image data (also referred to herein as “camera image data”), such as images, including a continuous stream of images (i.e., a video stream), along a camera sightline axis107. The camera image data is used in combination with the fluoroscopic image data to generate the AR imagery for displaying on the AR display112. The camera105can be oriented such that camera sightline axis107is substantially parallel with the beam axis115. In other embodiments, the camera sightline axis107can be angularly offset from the beam axis115.

The AR surgical imaging system102can include one or more surgical instruments203for guided use with the AR display112. In the present embodiment, the one or more surgical instruments203include a power drill203for targeting locking holes of an implant12(seeFIG.4B). It should be appreciated that other types of surgical instruments can be employed with the AR surgical imaging system102. In the illustrated embodiment, the AR display112is mountable to the surgical instrument203. In other embodiments, the AR display112can be mountable to the imaging station103, to the table140, or at another location within the AR surgical imaging system102. In further embodiments, the AR display112can be a mobile type of AR display112, such as a tablet, smart phone, headset visor, or the like, that can be carried and/or worn by a physician.

The AR surgical imaging system102includes an electronic control unit (ECU)204(also referred to herein as a “control unit”) that is configured to generate the AR imagery, such as a continuous stream of AR images that includes the fluoroscopic image data overlapped with the second image data (i.e., the camera image data). In particular, the control unit204is configured to overlap the fluoroscopic and camera image data in an anatomically matching configuration. It should be appreciated that the control unit204can include, or be incorporated within, any suitable computing device configurable to generate the AR imagery. Non-limiting examples of such computing devices include a station-type computer, such as a desktop computer, a computer tower, or the like, or a portable computing device, such as a laptop, tablet, smart phone, or the like. In the illustrated embodiment, the control unit204is incorporated into computer station211that is integrated into or with the fluoroscopic imaging device104. In other embodiments, the control unit204can be incorporated into a computer station211that can be mobile with respect to the fluoroscopic imaging device104with a wired or wireless electronic communication therewith. In further embodiments, the control unit204can be coupled to or internal to the surgical instrument203, as described in more detail below.

The AR surgical imaging system102can also include a transmitter unit114, which can be configured to communicate image data between the imaging station103and the AR display112. In the illustrated embodiment, the transmitter unit114is electronically coupled (e.g., wired) to the control unit204, which receives the fluoroscopy image data from the fluoroscopic imaging device104and also receives the camera image data from the video camera105and overlaps the fluoroscopic and camera image data to generate the AR imagery. The transmitter unit114then wirelessly transmits the AR imagery to a receiver unit113that is integrated with or connectable to the AR display112. In such embodiments, the AR imagery is generated at the computer station211and subsequently transmitted, via the transmitter and receiver units114,113, to the AR display112, which then displays the transmitted AR imagery to a physician. The transmitter unit114can be integrated with the control unit204or can be a separate unit electrically coupled thereto. The transmitter unit114can be any suitable computing device configured to receive and send images, such as the AR imagery. Non-limiting examples of such computing devices include those found in a portable computing device, such as in a laptop, tablet, smart phone, or the like.

Referring now toFIG.2A, in one example embodiment, the control unit204includes a main processing unit or “processor”206, a power supply208, an input portion210, and a memory portion214(also referred to herein as “memory”214). In particular, the main processor206is configured to receive the fluoroscopic and camera image data from the input portion210and execute machine-readable instructions (e.g., image processing instructions and augmentation instructions) to overlap the fluoroscopic image data and the camera image data and thereby generate the AR imagery. The machine-readable instructions can also include other instructions, such as for operating the imaging station103, such as for positioning the robotic arm110to locate the ROI within the imaging zone6.

The control unit204can include a station display212and a user interface216having controls219for receiving user inputs for controlling one or more operations of the control unit204. It should be understood that the station display212is separate from the AR display112described above. The main processor206, input portion210, station display212, memory214, and user interface216are preferably in communication with each other or at least connectable to provide communication therebetween. It should be appreciated that any of the above components may be distributed across one or more separate devices and/or locations. The station display212can be mounted at the computer station211and can be configured to display the fluoroscopic image data from the fluoroscopic imaging device104and/or the camera image data from the video camera105. In this manner, the station display212can be employed to ensure that the ROI is positioned within the imaging zone6. In some embodiments, the station display212can provide split-screen functionality to separately display both the fluoroscopic image data and the camera image data in real time. In various embodiments, the input portion210of the control unit204can include one or more receivers. The input portion210is capable of receiving information in real time, such as the fluoroscopic image data and the camera image data, and delivering the information to the main processor206. It should be appreciated that receiver functionality of the input portion210may also be provided by one or more devices external to the control unit204.

The memory214can store instructions therein that, upon execution by the main processor206, cause the control unit204to perform operations, such as the augmentation operations described herein. Depending upon the exact configuration and type of processor, the memory214can be volatile (such as some types of RAM), non-volatile (such as ROM, flash memory, etc.), or a combination thereof. The control unit204can include additional storage (e.g., removable storage and/or non-removable storage) including, but not limited to, tape, flash memory, smart cards, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, universal serial bus (USB) compatible memory, or any other medium which can be used to store information and which can be accessed by the control unit204.

The user interface216is configured to allow a user to communicate with and affect operation of the control unit204. The user interface216can include inputs or controls219that provide the ability to control the control unit204, via, for example, buttons, soft keys, a mouse, voice actuated controls, a touch screen, a stylus, movement of the control unit204, visual cues (e.g., moving a hand in front of a camera), or the like. The user interface216can provide outputs, including visual information (e.g., via the station display212), audio information (e.g., via speaker), mechanically (e.g., via a vibrating mechanism), or a combination thereof. In various configurations, the user interface216can include the station display212, a touch screen, a keyboard, a mouse, an accelerometer, a motion detector, a speaker, a microphone, a camera, a tilt sensor, or any combination thereof.

The transmitter unit114can include an independent power supply118and can also include an independent, secondary processing unit116for adjusting the wireless transmission signal (e.g., amplitude, frequency, phase) as needed before or during wireless transmission to the receiver unit113.

The receiver unit113can include any suitable computing device configured to receive wireless transmission of images, particularly the AR imagery. Non-limiting examples of such computing devices include those found in portable computing devices, such as a laptop, tablet, smart phone, and the like. It should be appreciated that the receiver unit113can also include an independent power supply and can also include an independent, secondary processing unit for adjusting the AR imagery (e.g., brightness, contrast, scale) as needed to enhance the visual perception displayed on the AR display112. The AR display112also includes a user interface119in communication with controls for receiving user inputs for controlling one or more operations of the AR display112, such as ON/OFF functionality and operations to be executed by the secondary processing unit, such as image adjustment (e.g., brightness, contrast, scale) and the like. The user interface119of the AR display112can include a graphical user interface (GUI) and/or other types of user interfaces. The user interface119can be operated by various types of controls and/or inputs, such as touch-screen controls, buttons, dials, toggle switches, or combinations thereof.

Referring now toFIG.2B, in another example embodiment, the control unit204is integrated with or coupled to the AR display112, whereby the AR imagery is both generated and displayed at the AR display112. In this example, the transmitter unit114receives the fluoroscopic and camera image data from the fluoroscopic imaging device104and the video camera105, respectively, and wirelessly transmits the fluoroscopic and camera image data to the control unit204. As shown, the transmitter unit114can also communicate the fluoroscopic and camera image data to a station display212, which can be configured similar to the station display212described above. In this embodiment, the control unit204includes a receiver unit113, which can be configured similarly to the receiver unit113described above. In the present embodiment, the receiver unit113is configured to receive the wireless fluoroscopic and camera image data from the transmitter unit114and convey the data to the main processor206for image processing and generating the AR imagery, which is displayed on the AR display. In the present embodiment, the memory214is integrated with the AR display112. It should be appreciated that, in the present embodiment, the main processor206can also communicate with the user interface119of the AR display112for controlling other operations of the AR display112(e.g., ON/OFF functionality, image adjustment, and the like). Because the receiver unit113and the main processor206in the present embodiment are both part of the control unit204, the receiver unit113optionally need not have an independent, secondary processor.

It should be appreciated that the block diagram depictions of the transmitter units114and the control units204shown inFIGS.2A-2Bare provided as examples and are not intended to limit the AR surgical imaging system102of the present disclosure to specific implementations and/or configurations. It should also be appreciated that the transmitter unit114and/or the control unit204can operate and/or can be configured as more fully described in U.S. Pat. No. 11,166,766, issued Nov. 9, 2021, and entitled “Surgical Instrument Mounted Display System” (hereinafter “the '766 Reference”) the entire disclosure of which is incorporated herein by this reference.

Various techniques can be employed to achieve the anatomical matching configuration of the AR images. Non-limiting examples of such techniques will now be described with reference toFIGS.3A-7.

Referring now toFIGS.3A-3D, an object, such as a reference marker or “marker”8having at least one reference feature10, can be positioned with respect to the anatomical structure4so that the reference feature(s)10is within a field of view of one or both of the fluoroscopic imaging device104and the camera105. In this manner, the reference feature(s)10can be captured in at least one of the fluoroscopic image data and the camera image data. In the present embodiment, the reference feature(s)10is preferably positioned within the ROI, which is positioned within the imaging zone6of the fluoroscopic imaging device104so that it can be captured in both of the fluoroscopic image data and the camera image data. Accordingly, in this embodiment the marker8is preferably radiopaque and is positioned at an ex vivo location adjacent the anatomical structure4within the ROI.

As shown inFIGS.3B and3C, the reference feature(s)10can be defined by one or more holes, preferably through-holes, extending orthogonally between opposed planar surfaces of the marker8. In this manner, each reference feature10defines a specific shape (“reference shape”) in a reference plane. The marker8preferably has opposite ends that are shaped differently from each other so that the orientation of the marker8in the fluoroscopic image data and camera image data is more readily discernable. With the marker8in place and the ROI positioned within the imaging zone6, the fluoroscopic imaging device104obtains fluoroscopic image data in which the marker8is discernible and the camera105obtains camera image data in which the marker8is also discernible. The control unit processes the fluoroscopic and camera image data to identify the reference feature(s)10(e.g., hole(s)) of the marker8therein and uses the reference feature(s)10to generate the AR images having the fluoroscopic and camera images overlapped in the anatomically matching configuration, as shown inFIG.3D. It should be appreciated that the marker8shown inFIGS.3B-3Drepresents a non-limiting example of the shape and type of marker that can be employed with the AR surgical imaging system102. Various other marker shapes, types, and reference feature geometries, are within the scope of the embodiments herein. It should be appreciated that such other marker shapes, types, and reference feature geometries are preferably radiopaque so as to be visible in X-ray imagery.

Referring now toFIGS.4A-4D, in other embodiments, the video camera105can be located on a surgical instrument203configured to operate on the anatomical structure4. As shown inFIG.4E, in such embodiments, the control unit204is preferably integrated with or coupled to the AR display112. For example, the control unit204can be internally located in the surgical tool203and can have a wired or wireless connection with the AR display112. The transmitter unit114transmits the fluoroscopic image data to the receiver unit213of the AR display112. The control unit204can include an input210, which is configured similar to that described above, and that receives the camera image data from the video camera105. The receiver unit113and the input210deliver the fluoroscopic and camera image data, respectively, to the main processor206for image processing and generating the AR imagery.

As described above, the marker8is positioned within the ROI, which is positioned within the imaging zone6of the fluoroscopic imaging device104so that the marker8is captured in the fluoroscopic image data. In the present embodiment, when the surgical instrument203is directed toward the ROI, the marker8can also be captured in the camera image data. The fluoroscopic imaging device104obtains fluoroscopic image data in which the marker8is discernible (FIG.4B) and the camera105obtains camera image data in which the marker8is also discernible (FIG.4C). The transmitter unit114processes the fluoroscopic data, identifies and employs the reference feature(s)10, and transmits the resulting fluoroscopic image data to the receiver unit113(FIG.4E). The receiver unit113delivers the transmitted fluoroscopic image data to the main processor206, wherein the fluoroscopic image data and the camera image data will be use to generate the AR imagery, as shown inFIG.4D. The control unit204can optionally include an accelerometer215(FIG.4E), which can be configured to generate accelerometer information that can allow the control unit204to calculate an orientation of the surgical instrument203with respect to the fluoroscopic imaging device104, as more fully described in the '766 Reference.

Referring now toFIG.5, an example augmentation algorithm or process500for generating the AR imagery can include steps502,504,600,506,700,508,510, and512. It should be appreciated that, although the example augmentation process500described below utilizes the processed video images as “base” or “reference” images and the processed X-ray images as the “source” or “slave” images during the overlapping or superimposition process, alternative processes can utilize the X-ray images as the reference images and the video images as the source images. Step502includes obtaining the camera image data using the camera105and transmitting the camera image data to the control unit204. Step504includes obtaining the fluoroscopic image data using the fluoroscopic imaging device104and transmitting the fluoroscopic image data to the control unit204, such as in DICOM format, by way of a non-limiting example. It should be appreciated that steps502,504can be performed in the manners described above with reference toFIGS.3A-3C and4A-4C. It should also be appreciated that at least steps600,506,700,508, and510are performed by the control unit204, particularly by the main processor206executing program instructions stored in the memory214. As described above, the control unit204can be located at the computer station211or at the AR display112, depending on the particular embodiment. Step600includes processing the camera image data showing the marker8to generate an image deformation matrix506(also referred to as a “transformation matrix”). Step700includes processing the fluoroscopic image data showing the marker8for comparison with the image deformation matrix506. Step508includes co-registration of the processed fluoroscopic image data with the processed camera image data. Step510includes overlapping (superimposing) the co-registered fluoroscopic image data and camera image data into an AR image stream. Step512includes displaying the AR image stream on the AR display112.

Referring now toFIG.6, further details of step600(processing the camera image data) will now be described. Step600can include sub-steps602,604,606,608,610, and612, which can each be referred to as a step. Step602, which is optional, includes mapping each image of the video stream, such as by converting each image to a 16-bit grayscale pixel map, which can have various pixel matrix configurations. By way of non-limiting examples, the pixel matrix configuration of the pixel map can be from 8×32 to 16×32 and preferably from 8×64 to 16×64, depending on the C-arm employed. As another non-limiting example, the pixel map can be an 8-bit pixel map. Step604includes adjusting each image for subsequent processing, such as by adjusting the contrast of each image as needed. Step606includes performing image segmentation on each image, which can include a sub-step607aof performing edge detection on each image, and can include another sub-step607bof performing object recognition within each image, such as by comparing image data in each image to a library of reference images stored in the computer memory. Step608includes object filtering on the segmented images. The object filtering in step608can be performed according to one or more various quality parameters. Step610includes fitting a shape with respect to each reference feature10(e.g., hole) of the marker8. For example, in the illustrated embodiments, this step610includes fitting reference circles with the holes of the marker8. For example, the shape(s) can be fitted using least squares approximation (LSQ) techniques, such as an LSQ minimum error approximation. Step612includes further object filtering the images processed according to step610(e.g., having a shape fitted to each reference feature10). The object filtering in step612can be performed according to two (2) or more quality parameters, which can include shape residuals and noise removal, by way of non-limiting examples. Step612can also be performed according to an iterative approach, in which the outcome of one or more of the filtering parameters is reapplied to each image, such as until a predetermined filtering standard or threshold is achieved for each image. At step506, the video images processed in step612, particularly the marker8and its reference feature(s)10therein, are each transform-processed to calculate an image deformation matrix for each image, which are then streamed in sequence in a processed real-time video stream.

Referring now toFIG.7, further details of the fluoroscopic image data processing700step will now be described. Step700can include sub-steps702,704,706,708,710, and712, which can each be referred to as a step. Step702, which is optional, includes mapping each image of the X-ray stream, such as by converting each X-ray image to a 16-bit grayscale pixel map. Step704includes adjusting each X-ray image for subsequent processing, such as by adjusting the contrast of each X-ray image as needed. Step704can also include upscaling each image, such as via bicubic interpolation and further linearization, by way of non-limiting examples. Step706includes performing image segmentation on each X-ray image, which can include a sub-step707aof performing edge detection on each X-ray image, and can include another sub-step707bof performing object recognition within each X-ray image, such as by comparing image data in each X-ray image to a library of reference X-ray images stored in the computer memory. Step708includes object filtering the segmented X-ray images, which can be performed according to one or more various quality parameters. Step710includes fitting a shape, such as a circle, with respect to each reference feature10(e.g., hole) of the marker8, which fitting can be performed according to LSQ techniques, such as an LSQ minimum error approximation. Step712includes further object filtering the X-ray images, which can be performed according to two (2) or more quality parameters. For example, one or more of the quality parameters in step712can involve calculations based on shape residuals. Step712can optionally be performed according to an iterative approach, in which the outcome of one or more of the filtering parameters is reapplied to each X-ray image, such as until a predetermined filtering standard or threshold is achieved for each X-ray image. The X-ray images processed according to step712, particularly the marker8and its processed reference feature(s)10therein, are ready for co-registration (step508) with the transform-processed images of the video stream.

Step508(co-registration of the video stream images and X-ray stream images) can include sub-steps802,804,806,808, and810, which can each be referred to as a step. Step802includes performing nearest-neighbor interpolation on each set of paired video and X-ray images (hereinafter referred to as “image pairs”). Step804includes performing linear interpolation on each image pair. Step806includes performing B-spline interpolation on each image pair. Step808includes iteration of one or more of steps802,804, and806, such as until a predetermined co-registration standard or threshold is achieved for each image pair. Step810includes performing object filtering on the image pairs, such as according to three (3) quality parameters. At step510, the co-registered, filtered image pairs from step508are then combined by superimposing the X-ray images onto the paired video images in anatomically matching configurations to generate a continuous stream of AR images, which is then transmitted to the display at step512.

The AR surgical imaging system102provides significant advantages over prior art surgical imaging systems, particularly with respect to surgical procedures that require precise targeting of implanted components with surgical instrumentation. Intramedullary (IM) nailing procedures are one non-limiting example of such procedures. In particular, even with fluoroscopy, aligning a drill bit to a distal locking hole of an IM nail can be difficult, especially if a surgeon is required to maneuver the drill while viewing the display of a fluoroscopic imaging device. The AR imagery of the embodiments described above allow the surgeon the ability to view, substantially in real-time, the relative position between the drill tip205and the distal locking holes of the IM nail. This can significantly reduce the amount of X-ray exposure to both the patient and the surgeon during an IM nailing procedure.

Referring now toFIG.8, another example surgical imaging system170is shown for generating and displaying augmented imagery, such as a stream of augmented images for calculating the necessary orientation and length of anchors in three-dimensional (3D) space for anchoring an implant12to an anatomical structure4. The imaging system170can include an imaging station103that includes a positioning mechanism, such as a robotic arm110, that carries an imaging array that includes an imaging device104, which is preferably a fluoroscopic imaging device104. The surgical imaging system170of the present embodiment can be similar to the AR surgical imaging station102described above. For brevity and conciseness, the following description will focus on differences of system170with respect to system102. Similar reference numbers will be used to denote similar components and subsystems of the systems102,170.

In the present embodiment, the fluoroscopic imaging device104is configured to obtain a first fluoroscopic image stream, taken at a first orientation at which beam axis115intersects pivot axis154, and a second fluoroscopic image stream, taken at a second orientation (indicated by dashed lines) at which beam axis115intersects pivot axis154. The first and second orientations are angularly offset from one another at a beam offset angle Al about the pivot axis154. The beam offset angle Al can be in a range from about 10 degrees to about 170 degrees, and is preferably from about 60 degrees to about 120 degrees, and is more preferably from about 80 degrees to 100 degrees, and even more preferably is about 90 degrees. The fluoroscopic imaging device104is configured to transmit the first and second fluoroscopic image streams to the control unit204for image processing and augmentation.

Referring now toFIG.9, an example augmentation algorithm or process900will be described for calculating the necessary orientation and length of anchors in 3D space for anchoring an implant12to an anatomical structure4. In relation to process900,FIGS.10A and10Bdepict a first fluoroscopic image stream (generated at the first orientation) at various states of process900, whileFIGS.10C and10Ddepict the second fluoroscopic image stream (generated at the second orientation) at various states of process900. Most, and optionally all, of the steps of process900are performed by the control unit204(e.g., by the processor206executing program instructions stored in the memory214) on the first and second fluoroscopic image streams. The fluoroscopic images of the first and second fluoroscopic image streams can be delivered to the control unit204in DICOM format, by way of a non-limiting example. In the present embodiment, the control unit204and the AR display112are preferably incorporated into the computer station211.

Process900can include steps902,904,906,908,910,912,914,916,918,920,922,924,926,928,930, and932. These steps can be categorized according to the following sub-routines of process900: image resolution (steps908,910,912, and914); implant processing (steps908,910,912,914,916, and918); and anatomy processing (steps920,922,924, and926). It should be appreciated that some of the foregoing steps, can be applicable to multiple sub-routines. For example, process900utilizes target structures of the implant12for the image resolution sub-routine; thus steps908,910,912, and914are utilized in both the image resolution and implant processing sub-routines. In other embodiments and processes, a separate reference marker8can be employed for the image resolution sub-routine, similar to the manner described above with reference toFIGS.3B and7(see steps700-712). Such as example process will be described in more detail below.

Step902includes positioning the anatomical structure4in the imaging zone6of the imaging array, particularly such that a ROI of the anatomical structure4can be intersected by the beam axis115at the first and second orientations. In the present example, the ROI includes an implant12residing within the anatomical structure4. In the example illustrated embodiment, the implant12is an IM nail residing within the medullary canal of a tibia, and the ROI encompasses a distal portion of the IM nail, which includes a distal tip and anchoring structures, such as distal locking holes extending transversely through the IM nail at various orientations. It should be appreciated that the augmentation process900can be employed with other implant and anchor types.

Step904includes obtaining, by the control unit204, a first fluoroscopic image stream of the ROI from the fluoroscopic imaging device104at a first orientation. Step906includes obtaining, by the control unit204, a second fluoroscopic image stream of the ROI from the fluoroscopic imaging device104at a second orientation that is angularly offset from the first orientation at the offset angle A1. The first and second fluoroscopic image streams preferably show the implant12, including one or more implant structures of interest (“ISOI”), which can also be referred to as implant “targets”14. With particular reference to the IM nail shown in the illustrated embodiments, non-limiting examples of such targets thereof can include the distal tip, the distal locking holes14, and an outer surface of the nail.

Step908includes processing the first and second fluoroscopic image streams, by the processor206executing instructions stored in the memory, to identify one or more targets14in one or both of the image streams (seeFIG.10A). For example, the processor206can convert each image of the streams into a pixel map, such as a 16-bit grayscale pixel map, and can additionally execute one or more cleaning algorithms, such as by adjusting the image contrast and/or applying one or more Gabor filters, such as from a circular Gabor filter bank. The processor206can also perform image segmentation and edge detection algorithms, such as Canny edge detector, by way of a non-limiting example, to identify edge patterns in one or both of the image streams and compare the edge patterns to a library of edge patterns associated with specific target14shapes stored in the memory. Step908can include executing an error calculation on the identified edge patterns and can identify or “register” the presence of a target14when the outcome of such error calculation falls within a threshold range.

Step910includes determining whether the target14possesses or presents its “true shape” in at least one of the image streams. As used herein, the term “true shape” means the shape of the target14when it directly faces the X-ray transmitter106, or, stated differently, when viewed along a beam axis115that orthogonally intersects the reference plane of the target14. To determine whether the target14presents its true shape in the image stream, the control unit204can process the target14to calculate a deviation between its true shape, as logged in the library in the memory, and the shape presented in the respective image stream. For example, with reference to the illustrated embodiment shown inFIG.10A, in which the target14is a locking hole, the processor206can identify whether the viewed shape is elliptical and thus deviates from a circle true shape. In step912, if the deviation result exceeds a minimum threshold, the control unit204instructs the robotic arm110to rotate the imaging array about axis154until one of the fluoroscopic imaging devices104,109obtains a view of the true shape of the target14(i.e., when the associated X-ray transmitter106directly faces the target14), as shown inFIG.10B. Accordingly, this fluoroscopic imaging stream can be referred to hereafter as the “facing stream”, and the other fluoroscopic imaging stream can be referred to as the orthogonally offset stream. It should be appreciated that steps910and912can optionally be repeated as needed to provide a confirmation mechanism for step910.

After the true shape of the target14has been confirmed, step914can be performed, which includes using a known size of the target14(as retrieved from the library), such as a width thereof, to calculate an image resolution of the facing stream, which can also approximate with a high degree of certainty the image resolution of the orthogonally offset stream. For example, when the target14has a true shape that is a circle, such as when the target14is a hole, such as a locking hole, the known size can be the diameter of the circle. The processor206can calculate the image resolution by counting the number of pixels along a line extending along the diameter of the circle, and can output the image resolution as a pixel ratio, particularly the quantity of pixels per unit distance, such as pixels per mm, for example. It should be appreciated that the calculated pixel ratio, in combination with image processing of the implant12, can also assist with determining various parameters of the implant, such as a longitudinal axis of the implant12, the implant's rotational orientation about the longitudinal axis, and the location of the distal-most end or “tip” of the implant12, by way of non-limiting examples.

Step916includes using the calculated image resolution to identify the center of the target14in the associated image stream. This step includes plotting a central axis20of the target14in the facing stream, as shown inFIG.10B. For example, plotting the central axis20in the facing image stream can include registering a center point of the target14, such as by registering a coordinate for the center point, such as coordinates (x1, z1) along respective x- and z-axes in a cartesian coordinate system. This step can also include superimposing visual indicia, such as highlighted pixilation, in the facing stream at the center coordinates (x1, z1) in a manner providing a visual reference of the location of the central axis20. Step918includes plotting the central axis20in the orthogonally offset image stream (FIG.10C), which can include registering a straight line along the x1coordinate in the orthogonally offset stream. The orthogonally offset stream can be defined with respect to x- and y-axes in the cartesian coordinate system.

Step920includes identifying one or more anatomical structures of interest (“ASOI”) in one or both of the image streams. This step can be performed by the processor206executing edge detection and/or cleaning algorithms (such as those described above at step908) to thereby identify edge geometries and patterns indicative of specific portions of the anatomical structure4, such as the outer cortical surfaces of a bone, such as a longbone, such as a tibia in which the IM nail is implanted, by way of a non-limiting example. Step922includes augmenting one or both of the facing and orthogonally offset image streams, such as by superimposing visual indicia onto the image streams. For example, as shown inFIG.10D, reference lines and/or contours24(e.g., via highlighted pixilation) can be superimposed onto edges of the outer cortical surface of one or more bones in the image stream. This can be performed according to a Circular Hough transform, by way of a non-limiting example. Such visual indicia can provide assistance during step924, which includes identifying intersection points at which the central axis20intersects ASOI in the orthogonally offset image stream. In the illustrated example, the ASOI is the outer cortical surface of the longbone in which the IM nail resides. The intersection points (y1, y2) can be plotted with respect to the y-axis of the cartesian coordinate system, for example. The intersection points (y1, y2) can be plotted by a user, such as a surgeon or medical technician, with the assistance of the user interface216and controls219, such as via mouse click or contacting a stylus against the AR display112surface (if touch-screen capable), by way of non-limiting examples. In other embodiments, the intersection points (y1, y2) can be plotted autonomously by the processor206. In such embodiments, the processor206can generate visual indicia at the autonomously selected intersection points (y1, y2), and can additionally provide the user with an option of accepting these intersection points (y1, y2) or inputting alternative intersection points directly via the controls219.

Step926includes using the image resolution (e.g., pixel ratio) to calculate a distance D1along the central axis between the intersection points (y1, y2). With reference to the illustrated example, this axial distance D1can be used to select a locking screw having sufficient length to extend through the target14locking hole and purchase within the near and far cortex of the bone. Step928includes superimposing the distance D1measurement alongside the associated axis on an X-ray image, such as an image of the orthogonally offset stream. Step930includes repeating various steps of process900for the remaining targets14of the implant12, such as steps908,910,912,914,916,918,920,922,924,926,928, for example. Process900can optionally provide a bypass feature, such as step932, which can bypass step914for subsequent targets14. The output of process900can be a reference X-ray image that identifies each target14, and depicts the superimposed axis20thereof and the associated distance D1measurement for each target14.

Referring now toFIG.11, an example of an image resolution sub-routine1100that employs a separate, ex vivo reference marker8will be described. Sub-routine1100, which can also be referred to as a process or method, can include steps1102,1104,1106,1108,1110, and1112, which are performed by the control unit204(e.g., by the processor206executing program instructions stored in the memory214) on the X-ray images received from the first and second fluoroscopic imaging devices104,109. These X-ray images can be transmitted to the control unit204in DICOM format, by way of a non-limiting example. Step1102, which is optional, includes mapping each image of at least one of the first and second X-ray streams, such as by converting each X-ray image thereof to a 16-bit grayscale pixel map, which can have various pixel matrix configurations. By way of non-limiting examples, the pixel matrix configuration of the pixel map can be from 8×32 to 16×32 and preferably from 8×64 to 16×64, depending on the C-arm employed. As another non-limiting example, the pixel map can be an 8-bit pixel map. Step1104includes adjusting each X-ray image for subsequent processing, such as by adjusting the contrast of each X-ray image as needed. Step1104can also include upscaling each image, such as via bicubic interpolation and further linearization, by way of non-limiting examples.

Step1106includes performing image segmentation on each X-ray image, which can include a sub-step of performing edge detection on each X-ray image, and can include another sub-step of performing object recognition within each X-ray image, such as by comparing image data in each X-ray image to a library of reference X-ray images stored in the computer memory214. Step1108includes object filtering the segmented X-ray images, which can be performed according to one or more various quality parameters. Step1110includes fitting a shape, such as a circle, with respect to each reference feature (e.g., hole) of the marker8, which fitting can be performed according to LSQ techniques, such as an LSQ minimum error approximation. Step1112includes further object filtering the X-ray images, which can be performed according to two (2) or more quality parameters, which can be different quality parameters than those of step1108. For example, one or more of the quality parameters in step1112can involve calculations based on shape residuals. Step1112can optionally be performed according to an iterative approach, in which the outcome of one or more of the filtering parameters is reapplied to each X-ray image, such as until a predetermined filtering standard or threshold is achieved for each X-ray image.

Steps1114and1116can be included in process1100when each X-ray image depicts multiple markers8, each having a plurality of reference features, or when each X-ray image depicts multiple groupings or “clusters” of reference features. Step1114includes clustering the reference features. Step1116includes sorting the reference features according their associated marker8or associated region of the X-ray image, which sorting can be performed according to at least one (1) quality parameter, which can involve a sort residual calculation. Step1118includes calculating a pixel ratio for each X-ray image (e.g., pixels per mm), which can be performed according to a linear scaling function.

It should be appreciated that the processes, steps, and techniques described above with reference toFIGS.8-11can be adapted as needed to calculate the required orientation and length of anchors for affixing other types of implants to adjacent anatomy. Such implants can include bone plates, bone screws, intervertebral cages, and the like.

While example embodiments of devices for executing the disclosed techniques are described herein, the underlying concepts can be applied to any control unit, computing device, processor, or system capable of communicating and presenting information as described herein. The various techniques described herein can be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatuses described herein can be implemented, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embodied in tangible non-transitory storage media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium (computer-readable storage medium), wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for performing the techniques described herein. In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device, for instance a display. The display can be configured to display visual information. For instance, the displayed visual information can include fluoroscopic data such as X-ray images, fluoroscopic images, orientation screens, or computer-generated visual representations.

The program(s) can be implemented in assembly or machine language, if desired. The language can be a compiled or interpreted language, and combined with hardware implementations.

The techniques described herein also can be practiced via communications embodied in the form of program code that is transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via any other form of transmission. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique apparatus that operates to invoke the functionality described herein. Additionally, any storage techniques used in connection with the techniques described herein can invariably be a combination of hardware and software.

While the techniques described herein can be implemented and have been described in connection with the various embodiments of the various figures, it is to be understood that other similar embodiments can be used or modifications and additions can be made to the described embodiments without deviating therefrom. For example, it should be appreciated that the steps disclosed above can be performed in the order set forth above, or in any other order as desired. Further, one skilled in the art will recognize that the techniques described in the present application may apply to any environment, whether wired or wireless, and may be applied to any number of such devices connected via a communications network and interacting across the network. Therefore, the techniques described herein should not be limited to any single embodiment, but rather should be construed in breadth and scope in accordance with the appended claims.