Patent Description:
Many microsurgical procedures require precision cutting and/or removal of body tissues. For example, certain ophthalmic surgical procedures require the cutting and/or removal of the vitreous humor, a transparent jelly-like material that fills the posterior segment of the eye. The vitreous humor, or vitreous, is composed of microscopic fibrils that are often attached to the retina. Cutting and removal of the vitreous must be done with great care to avoid traction on the retina, the separation of the retina from the choroid, a retinal tear, or, in the worst case, cutting and removal of the retina itself. Delicate operations such as mobile tissue management (e.g., cutting and removal of vitreous near a detached portion of the retina or a retinal tear), vitreous base dissection, and cutting and removal of membranes are particularly difficult.

Microsurgical cutting probes used in posterior segment ophthalmic surgery are typically inserted via an incision in the sclera near the pars plana. The surgeon may also insert other microsurgical instruments such as a fiber optic illuminator, an infusion cannula, an imaging probe (e.g., an OCT probe), or an aspiration probe during the posterior segment surgery.

To aid the surgeon with these types and other types of surgical procedures, surgeons may use an imaging system that presents a microscope view of the tissue to be treated, such as tissue of the patient's eye. Accordingly, the user of such an imaging system may be provided with a close-up view of the surgical instruments, such as forceps or other tools, as well as the region of the eye that is of interest. Such systems may also provide additional information that may be useful to the surgeon, such as an Optical Coherence Tomography (OCT) image of the region of the eye that is of interest. OCT imaging generally utilizes near-infrared light and is able to obtain or generate images of tissue beneath the surface.

Such an example of imaging and visualization systems for navigating instruments may be seen in the Patent Document <CIT>.

Despite advances in imaging systems, performing ocular surgical procedures remains challenging. Among other things, it may be difficult for a surgeon viewing a stereo microscope image to discern with precision the depth of a surgical tool inserted in the eye and its proximity to particular tissues, such as the retina. Surgeons typically rely on experience and judgment developed over time for guidance during delicate procedures, and improved techniques for visualization are needed to improve patient safety and surgical outcomes.

In certain embodiments, an ophthalmic surgical system includes an imaging unit configured to generate a fundus image of an eye and a depth imaging system configured to generate a depth-resolved image of the eye. The system further includes a tracking system communicatively coupled to the imaging unit and depth imaging system. The tracking system includes a processor and memory configured to analyze the fundus image generated by the imaging unit to determine a location of a distal tip of a surgical instrument in the fundus image, analyze the depth-resolved image generated by the depth imaging system to determine a distance between the distal tip of the surgical instrument and a retina of the eye, generate a visual indicator to overlay a portion of the fundus image, the visual indicator indicating the determined distance between the distal tip and the retina, modify the visual indicator to track a change in the location of the distal tip within the fundus image in real-time, and modify the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time.

The processor and memory of the tracking system may be further configured to determine a distance between the distal tip of the surgical instrument and a retina of the eye based on an analysis of image pixels in the depth-resolved image.

The depth-imaging system may be configured to generate a depth-resolved image of the eye based on signals received by an imaging probe integrated with the surgical instrument.

According to the present invention, defined in the appended claims, the processor and memory of the tracking system are configured to generate the visual indicator to overlay the distal tip of the surgical instrument within the fundus image.

The processor and memory of the tracking system may be configured to modify the visual indicator to indicate the change in the distance between the distal tip of the surgical instrument and the retina by increasing or decreasing the size of the visual indicator in proportion to the change in distance between the distal tip of the surgical instrument and the retina.

In certain embodiments, the processor and memory of the tracking system are configured to modify the visual indicator to indicate the change in the distance between the distal tip of the surgical instrument and the retina by modifying a color of the visual indicator.

According to certain embodiments, the depth imaging system is an Optical Coherence Tomography (OCT) system configured to generate an OCT image of the eye. The OCT system may include an OCT light source operable to generate an OCT imaging beam and a beam scanner operable to direct the OCT imaging beam. The tracking system may be configured to analyze the OCT image to determine the distance between the distal tip of the surgical instrument and the retina of the eye. The processor and memory of the tracking system may be configured to cause the beam scanner to direct the OCT imaging beam to a particular region of the eye that includes the distal tip of the surgical instrument, based on the determined location of the distal tip of the surgical instrument within the fundus image.

In certain embodiments, the surgical instrument includes a first optical fiber configured to transmit the OCT imaging beam, and a second optical fiber configured to transmit light reflected by the eye.

According to certain embodiments, the processor and memory of the tracking system are configured to determine the location of the distal tip of the surgical instrument in the fundus image by generating an enhanced image of the fundus image, estimating a marker image within the enhanced image, extracting the marker image from the enhanced image, and determining a location of the marker from the image of the marker.

In certain embodiments, the visual indicator and fundus image are displayed in an eyepiece or on a heads-up screen. The visual indicator may also be configurable by a user.

In certain embodiments, the imaging unit comprises at least one of a surgical microscope, a <NUM>-dimensional camera, a line-scan camera, and a single detector as used in a confocal scanning ophthalmoscope.

Certain embodiments include a method which is not part of the claimed invention that includes generating a fundus image of an eye, generating a depth-resolved image of the eye, analyzing the fundus image to determine a location of a distal tip of a surgical instrument in the fundus image, analyzing the depth-resolved image to determine a distance between the distal tip of the surgical instrument and a retina of the eye, generating a visual indicator to overlay the distal tip of the surgical instrument within the fundus image, the visual indicator indicating the determined distance between the distal tip and the retina, modifying the visual indicator to track a change in the location of the distal tip within the fundus image in real-time, and modifying the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time.

According to certain embodiments, modifying the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time includes increasing or decreasing the size of the visual indicator in proportion to the change in distance between the distal tip of the surgical instrument and the retina.

In certain embodiments, modifying the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in real-time includes modifying a color of the visual indicator.

Certain embodiments further include directing an imaging beam of an imaging system to a particular region of the eye that includes the distal tip of the surgical instrument, based on the determined location of the distal tip of the surgical instrument within the fundus image.

In certain embodiments, analyzing the fundus image to determine a location of a distal tip of a surgical instrument in the fundus image includes generating an enhanced image of the fundus image, estimating a marker image within the enhanced image, extracting the marker image from the enhanced image, and determining a location of the marker from the image of the marker.

In certain embodiments, the method includes displaying the visual indicator in an eyepiece or on a heads-up screen. The method may also include receiving user input related to a type of visual indicator.

Certain embodiments of the present disclosure may provide one or more technical advantages. For example, certain embodiments provide a visual indicator that may allow a surgeon to perform a vitrectomy with increased precision, and reduce the risk of damaging the retina during a vitrectomy. In particular, certain embodiments provide a visual indicator displayed as an overly at the position of a distal tip of a surgical instrument. This aspect may assist a surgeon by providing accurate, real-time information about the actual and/or relative location of, and distance between, a surgical tool and sensitive tissue without obstructing the surgeon's view of surrounding tissue. Further, by providing an indicator to alert a surgeon of the precise location of a surgical instrument, including its proximity to sensitive tissue such as a retina, certain embodiments increase the surgeon's precision, awareness, and confidence, improving patient safety and surgical outcomes. Moreover, the indicator may be provided as an image overlay within an eyepiece or heads-up display so that a surgeon may easily monitor the indicator without diverting attention from the surgical field. Additionally, features or aspects of the indicator may be modified in proportion to changes in the proximity of the tool and eye tissue, thereby providing intuitive and immediate feedback to the surgeon regarding tool position.

For a more complete understanding of the present disclosure and the advantages thereof, reference is now made to the following description taken in conjunction with the accompanying drawings in which like reference numerals indicate like features and wherein:.

One skilled in the art will understand that the drawings, described below, are for illustration purposes only, and are not intended to limit the scope of applicant's disclosure.

It will nevertheless be understood that no limitation of the scope of the disclosure is intended. Alterations and further modifications to the described systems, devices, and methods, and any further application of the principles of the present disclosure are contemplated as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is contemplated that the systems, devices, and/or methods described with respect to one embodiment may be combined with the features, components, and/or steps described with respect to other embodiments of the present disclosure. For simplicity, in some instances the same reference numbers are used throughout the drawings to refer to the same or like parts. References to location, distance, or proximity herein may refer to actual and/or relative location(s), distance(s), or proximity(ies).

In general, the present disclosure relates to an ophthalmic surgical visualization system capable of providing one or more visual indicators that conveys the proximity of a surgical tool to particular tissue, such as the retina. Certain embodiments provide a user with a microscope image of eye tissue that includes a computer-generated visual indicator (e.g., a pointer, shape, icon, or other graphic element) which indicates an distance between particular eye tissue (e.g., a retina) and a surgical tool inserted in the eye (e.g., the tip of a vitrectomy probe). One or more characteristics of the visual indicator (e.g., its color, size, shape) may be modified in real-time to reflect the distance between the surgical tool and particular eye tissue. In certain embodiments, a characteristic of the visual indicator (e.g., size, color) is modified incrementally, and proportional to the change in distance, to intuitively convey the movement of the tool. The distance between the surgical tool and particular eye tissue may be determined based on data obtained by an imaging system capable of resolving depth in real time, such as an OCT imaging system, ultrasound imaging system, a multispectral imaging system, a computerized axial tomography (CAT) scan system, a magnetic resonance imaging (MRI) system, or a positron emission tomography (PET) imaging system. Certain embodiments also track movements of the surgical tool within a microscope image in real time, and may display the visual indicator as a dynamic overlay in the microscope image presented in an eyepiece or on a heads-up display. For example, a visual indicator may be displayed as a graphic overlay superimposed on a distal end of the surgical tool as it moves within in a microscope image of a retina, and the size and/or color of the overlay indicator may be updated continuously according to the distance between the distal end of the surgical tool and the retina.

<FIG> illustrates an example of an ophthalmic surgical visualization system according to particular embodiments of the present disclosure. Surgical microscope <NUM> includes integrated OCT and display systems. Surgical microscope <NUM> may facilitate magnified viewing of a patient's eye <NUM> during a surgical procedure and may generally include eyepieces <NUM>, a relay lens <NUM>, magnifying/focusing optics <NUM>, an objective lens <NUM>, and surgical viewing optics <NUM>. Each of eyepieces <NUM>, relay lens <NUM>, magnifying/focusing optics <NUM>, objective lens <NUM>, and surgical viewing optics <NUM> may include any suitable optical components as understood by persons of ordinary skill in the art.

Surgical microscope <NUM> may additionally include an integrated OCT system <NUM> operable to generate OCT images of the patient's eye <NUM> and a real-time data projection unit <NUM> operable to display those OCT images to a surgeon via one or both eyepieces <NUM>. The location at which OCT system <NUM> is integrated into surgical microscope <NUM> (as discussed in further detail below) may advantageously allow the OCT scan range to be automatically adjusted as a surgeon manipulates the microscope field of view via the magnifying/focusing optics <NUM>. Moreover, real-time data projection unit <NUM> may advantageously allow a surgeon to view the OCT images generated by OCT system <NUM> without the need to look at a separate display monitor.

OCT system <NUM> may include a light source/analyzing unit <NUM> and a beam scanner <NUM>. In general, light source/analyzing unit <NUM> may generate an OCT imaging beam <NUM> and beam scanner <NUM> (in conjunction with other optical components of the surgical microscope) may direct the generated OCT imaging beam <NUM> to a particular region within the patient's eye <NUM>. Reflections of the OCT imaging beam <NUM> from the particular region within the patient's eye <NUM> (reflected OCT imaging beam <NUM>) may return to light source/analyzing unit <NUM> along the same optical path as OCT imaging beam <NUM>, and light source/analyzing unit <NUM> may generate OCT images of the particular region by determining interference between the reflections <NUM> and a reference arm of the OCT imaging beam <NUM>. The present disclosure contemplates that OCT system <NUM> may include any suitable additional optical components for manipulating OCT imaging beam <NUM> as would be understood by those of skill in the art, and those additional components are not depicted/described for the sake of simplicity.

In certain embodiments, the OCT imaging beam <NUM> may comprise an infrared or near infrared light beam covering a relatively narrow band of wavelengths (e.g., <NUM> - <NUM>, 790mn - <NUM>, <NUM>-<NUM>). However, an OCT imaging beam <NUM> having any suitable spectral range may be used.

In certain embodiments, the OCT imaging beam <NUM> may pass through beam scanner <NUM> (described in further detail below) along with any other suitable optical components of OCT system <NUM> (not depicted, as described above). OCT imaging beam <NUM> may then be directed to the patient's eye <NUM> via one or more of the above-described optical components of surgical microscope <NUM> (as described in further detail below).

Beam scanner <NUM> may comprise any suitable optical component or combination of optical components facilitating focusing of the OCT imaging beam <NUM> in the X-Y plane. For example, beam scanner <NUM> may include one or more of a pair of scanning mirrors, a micro-mirror device, a MEMS based device, a deformable platform, a galvanometer-based scanner, a polygon scanner, and/or a resonant PZT scanner. In certain embodiments, the position of the optical components of beam scanner <NUM> may be manipulated in an automated manner. As just one example, beam scanner <NUM> may comprise a pair of scanning mirrors each coupled to a motor drive, the motor drives operable to rotate the mirrors about perpendicular axes. As a result, by controlling the position of the coupled motors (e.g., according to a pre-determined or selected scan pattern), the X-Y positioning of OCT imaging beam <NUM> within the patient's eye <NUM> can be controlled. Additionally, the depth of focus of the OCT imaging beam <NUM> may be controlled by one or more other components of OCT system <NUM> as is understood in the art in order to facilitate <NUM>-D OCT imaging.

As described above, reflected OCT beam <NUM> may return to OCT system <NUM> along substantially the same optical path as traveled by OCT imaging beam <NUM>. Once reflected OCT beam <NUM> reaches light source/analyzing unit <NUM>, light source/analyzing unit <NUM> may construct an OCT image (A-scan) based on interference between the reflected OCT beam <NUM> and a reference arm of OCT imaging beam <NUM> (as is known in the art). Moreover, by moving the imaging beam in the X-Y plane via beam scanner <NUM> and/or changing the depth of focus of the imaging beam <NUM>, a plurality of OCT images (A-scans) may be generated and combined into an OCT cross sectional image (B-scan), and a plurality of those cross sectional images (B-scans) may be combined to generate a <NUM>-D OCT image.

In certain embodiments, OCT system <NUM> may be integrated into surgical microscope <NUM> via a beam coupler <NUM> located in the optical path of the surgical microscope <NUM>. Beam coupler <NUM> may include an optical element configured to reflect wavelengths in the spectral range of the OCT imaging beam <NUM> (e.g., infrared wavelengths) while allowing passage of light in the visible spectrum passing through surgical microscope <NUM>. As one example, beam coupler <NUM> may comprise one of a dichroic hot mirror, a polarizing beamsplitter, and a notch filter.

In certain embodiments, beam coupler <NUM> may be located along the optical path between the surgical viewing optics <NUM> and an eyepiece <NUM>. Surgical viewing optics <NUM> may include a drop-on macular lens, contact-based wide-angle lens, noncontact-based viewing system such as (binocular indirect ophthalmomicroscope) BIOM, or any other suitable viewing optics. More particularly, beam coupler <NUM> may be located along the optical path between magnifying/focusing optics <NUM> and an eyepiece <NUM>. As a result, OCT imaging beam <NUM> will pass through magnifying/focusing optics <NUM>, allowing the OCT scan range to be automatically adjusted as a surgeon manipulates the microscope field of view via the magnifying/focusing optics <NUM>. The present disclosure contemplates that, although not depicted, OCT system <NUM> may additionally include any suitable optical components facilitating appropriate focus of OCT imaging beam <NUM> within the patient's eye <NUM> in light of the fact that the OCT imaging beam <NUM> passes through magnifying/focusing optics <NUM> and objective lens <NUM>.

In certain embodiments, OCT system <NUM> may generate a visible aiming beam (not depicted) in addition to OCT imaging beam <NUM>. This visible aiming beam may be visible to the surgeon via eyepieces <NUM> and may assist the surgeon in directing OCT imaging. In such embodiments, beam coupler <NUM> may be configured to reflect both the spectral range of the OCT imaging beam <NUM> (e.g., infrared wavelengths) and a narrow band of visible light (the aiming beam falling within that narrow band) while allowing passage of visible light passing through surgical microscope <NUM> that falls outside the narrow band of the aiming beam.

The OCT image(s) generated by OCT system <NUM> (identified in <FIG> by reference numeral <NUM>), which may include an A-scan, a B-scan, or a <NUM>-D OCT image constructed by combining a plurality of B-scans as described above, may be communicated to real-time data projection unit <NUM> for display to a surgeon via one or both eyepieces <NUM>.

The present disclosure contemplates that, although not depicted, certain embodiments may include one or more additional or alternative depth-imaging systems, such as an ultrasound imaging system, a multispectral imaging system, a computerized axial tomography (CAT) scan system, a magnetic resonance imaging (MRI) system, or a positron emission tomography (PET) imaging system. Such imaging systems may be configured analogously to the OCT imaging systems described herein (e.g., integrated with microscope <NUM>, probe-based, and/or integrated with surgical instrument <NUM>) to generate depth-resolved images that may be analyzed by tracking unit <NUM>.

Real-time data projection unit <NUM> may include any suitable device for projecting an image and may include any suitable optics (not depicted) for focusing that image. For example, real-time data projection unit <NUM> may comprise one of a heads-up-display, a one-dimensional display array, a two-dimensional display array, a screen, a projector device, or a holographic display.

Real-time data projection unit <NUM> may be integrated into surgical microscope <NUM> via a beam splitter <NUM> located in the optical path of the surgical microscope <NUM>. Beam splitter <NUM> may include an optical element configured to reflect the projected image generated by real-time data projection unit <NUM> toward eyepiece(s) <NUM> without substantially interfering with visible light reflected from the patient's eye <NUM>.

In certain embodiments, surgical microscope <NUM> may additionally or alternatively include a probe-based OCT system <NUM>. Probe-based OCT system <NUM> may generate OCT images <NUM> is substantially the same manner as described above with regard to OCT system <NUM> except that the OCT imaging beam generated by probe-based OCT system <NUM> may be directed within the patient's eye <NUM> using a probe <NUM> that may be inserted into the patient's eye <NUM>. In embodiments including both an OCT system <NUM> and a probe-based OCT system <NUM>, surgical microscope <NUM> may additionally include a source selection unit <NUM>. Source selection unit <NUM> may include any suitable switch allowing selection either OCT images <NUM> (generated by OCT system <NUM>) or OCT images <NUM> (generated by probe-based OCT system <NUM>) for communication to real-time data projection unit <NUM> or display <NUM>. As a result, a surgeon may select which OCT imaging system to use for imaging during surgery.

In certain embodiments, surgical instrument <NUM> may additionally or alternatively be integrated with OCT imaging probe <NUM> or include an additional or alternative OCT imaging probe. For example, surgical instrument <NUM> may be communicatively coupled with probe-based OCT system <NUM>. Surgical instrument <NUM> may include one or more optical fibers and extending down the length of the instrument toward its distal tip to transmit and/or receive an OCT imaging beam or reflected light from eye <NUM>. The fibers may terminate at or near the distal tip to transmit an imaging beam into eye <NUM>. Such fibers and other components of surgical instrument <NUM> may be configured to transmit an OCT imaging beam to eye <NUM> and return reflections to light source/analyzing unit <NUM>. In this manner, the OCT imaging beam may be directed within the patient's eye <NUM> using surgical instrument <NUM> rather than a separate OCT probe or beam scanner. In such embodiments, the distance between the distal tip of surgical instrument <NUM> and eye <NUM> may be determined without adjusting an OCT beam toward the instrument; because the imaging beam is projected from the tip of the surgical instrument, it is not necessary to adjust an external OCT imaging beam to encompass both the eye tissue and the surgical instrument within imaging field. In certain embodiments, surgical instrument <NUM> may be integrated with or include a depth imaging probe other than an OCT imaging probe.

OCT images projected by real-time data projection unit <NUM> (e.g., OCT images <NUM> and/or OCT images <NUM>) may be displayed as a semitransparent overlay aligned with the visible structures viewed by the surgeon via eyepieces <NUM>. In such embodiments, alignment between the OCT images and the actual structures of the eye may be achieved, for example, based on retinal tracking (described further below), instrument tracking (described further below), an aiming beam, or any combination thereof.

In certain other embodiments, the OCT images projected by real-time data projection unit <NUM> may be displayed in a corner of the field of view of the surgeon or any other suitable location in which they do not substantially impair the surgeon's ability to view the eye <NUM> through eyepieces <NUM>.

Although real-time data projection unit <NUM> is described above as projecting OCT images <NUM> and/or OCT images <NUM> into the optical path of the surgical microscope <NUM> such that they are viewable through eyepiece(s) <NUM>, the present disclosure contemplates that real-time data projection unit <NUM> may, additionally or alternatively, project any other suitable information (e.g., extracted and/or highlighted information from OCT data, fundus images, surgical parameters, surgical patterns, surgical indicators, etc.) into the optical path of the surgical microscope <NUM>, according to particular needs.

Surgical microscope <NUM> may additionally include an imaging unit <NUM> and a tracking unit <NUM>. Tracking unit <NUM> may be communicatively coupled (via wired or wireless communication) to OCT system <NUM>, real-time data projection unit <NUM>, and display <NUM> to provide images, indicators, and other data for display to a system operator. As described in further detail below, OCT system <NUM>, imaging unit <NUM>, and tracking unit <NUM> may collectively facilitate tracking the location, depth, proximity, and movement of a surgical instrument <NUM> within the patient's eye <NUM>.

Imaging unit <NUM> may include any suitable device for generating a fundus image <NUM> of a patient's eye <NUM> and may include suitable magnification and focusing optics (not depicted) for performing that function. As a simplified example, visible or near infrared light <NUM> reflected by the patient's eye <NUM> along the optical path of surgical microscope <NUM> may be directed toward imaging unit <NUM> via a mirror <NUM> placed along the optical path and operable to partially reflect such light. In certain embodiment, fundus images <NUM> may be discrete still photographs of the patient's eye <NUM>. In other embodiment, the fundus image <NUM> may comprise a continuous video stream of the patient's eye <NUM>. Fundus image <NUM> may comprise multiple image frames that can be processed and modified by other components of system <NUM>. Example imaging units may include digital video cameras, line scan ophthalmoscopes or confocal-scanning ophthalmoscopes.

In the depicted embodiment, because the visible or near infrared light <NUM> is sampled from the optical path before OCT images are introduced into the optical path via real-time data projection unit <NUM>, the generated fundus images <NUM> will not include the projected OCT images (which may be beneficial for the instrument tracking described below). Although imaging unit <NUM> is depicted and described as being located at particular position relative to the optical components of the surgical microscope <NUM> and OCT system <NUM>, the present disclosure contemplates that imaging unit <NUM> may be placed at any suitable location relative to those components, according to particular needs.

Tracking unit <NUM> of surgical microscope <NUM> is operable to determine location/position, depth, proximity, and motion of surgical instrument <NUM> within the patient's eye <NUM> based at least in part on fundus images <NUM> generated by imaging unit <NUM> and depth-resolved or three-dimensional images generated by a depth imaging system, such as OCT system <NUM>, an ultrasound imaging system, a multispectral imaging system, a computerized axial tomography (CAT) scan system, a magnetic resonance imaging (MRI) system, or a positron emission tomography (PET) imaging system.

Tracking unit <NUM> may include any suitable combination of hardware, firmware, and software. In certain embodiments, tracking unit <NUM> may include processor <NUM> and memory <NUM>. Processor <NUM> may include one or more microprocessors, field-programmable gate arrays (FPGAs), controllers, or any other suitable computing devices or resources. Processor <NUM> may work, either alone or with other components depicted in <FIG>, to provide the functionality described herein. Memory <NUM> may take the form of volatile or non-volatile memory including, without limitation, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), removable media, or any other suitable memory component. Memory <NUM> may store instructions for programs and algorithms that, when executed by processor <NUM>, implement the functionality of tracking unit <NUM>.

Tracking unit <NUM> may be programmed to (or may store software in memory <NUM> that, when executed by processor <NUM>; is operable to) analyze fundus images <NUM> to determine and track the location of surgical instrument <NUM>. For example, processor <NUM> may receive and process or analyze images <NUM> acquired by imaging unit <NUM>, and may generate indicators and images for display by real-time data projection unit <NUM> or display <NUM> based on the processed images. Processor <NUM> may process or analyze multiple images to track changes in the location of surgical instrument <NUM>, and modify the indicators and images to reflect such changes. Memory <NUM> of tracking unit <NUM> may store the pre-processed and/or post-processed image data. Processor <NUM> may detect and calculate the location and/or orientation (or the change of the location and orientation) of surgical instrument <NUM> in the surgical field based on the fundus images <NUM>.

Additionally, tracking unit <NUM> may be programmed to (or may store software in memory <NUM> that, when executed by processor <NUM>, is operable to) determine the depth of distal tip <NUM> and its proximity to particular tissue of eye <NUM>. For example, processor <NUM> may receive three-dimensional or depth-resolved imaging data acquired by OCT system <NUM> (or an alternative depth imaging system) and may analyze the data to determine a distance between and/or proximity of distal tip <NUM> and the retina of eye <NUM>. Based on the determined distance, tracking unit <NUM> may generate an indicator for display by real-time data projection unit <NUM> or display <NUM> to alert a system operator about the proximity of distal tip <NUM> to particular eye tissue, such as the retina. Processor <NUM> may continuously or repeatedly determine or calculate location, orientation, and distance/proximity data related to distal tip <NUM> to track distal tip <NUM> and update the indicator to provide a real-time indication of the position of distal tip <NUM> in the microscope image and the distance between distal tip <NUM> and the retina of eye <NUM>. Processor <NUM> may also continuously or repeatedly modify a characteristic of the visual indicator (e.g., size, color) incrementally and proportionally to the change in distance between distal tip <NUM> and the retina of eye <NUM> to intuitively convey the movement of distal tip <NUM>. Memory <NUM> of tracking unit <NUM> may store the pre-processed and/or post-processed depth imaging data.

Further, tracking unit <NUM> may be programmed to (or may store software in memory <NUM> that, when executed by processor <NUM>, is operable to) generate signals <NUM> to be communicated to OCT system <NUM> to cause beam scanner <NUM> of OCT system <NUM> to direct the location of the OCT imaging beam <NUM> within the patient's eye <NUM>. For example, signals <NUM> may be generated based on the determined location of the surgical instrument <NUM> within the patient's eye <NUM>, and beam scanner <NUM> of OCT system <NUM> may direct OCT imaging beam <NUM> to a location in the vicinity of the distal tip <NUM> of surgical instrument <NUM>. As a result, the OCT images <NUM> may be generated in an area of most interest to the surgeon, and tracking unit <NUM> may calculate the distance between distal tip <NUM> and the retina throughout a procedure using data generated by OCT system <NUM>. Moreover, in embodiments in which the OCT images <NUM> are displayed as a semi-transparent overlay in the field of view of the microscope, the tracking of the surgical instrument <NUM> may additionally facilitate proper positioning of that overlay.

As another example, the signals <NUM> may be generated based on a determined location of the retina of the patient's eye <NUM> (determined by tracking unit <NUM> by processing fundus images <NUM> in a manner similar to that discussed herein with regard to tracking surgical instrument <NUM>), and beam scanner <NUM> of OCT system <NUM> may direct OCT imaging beam <NUM> to constant location relative to the retina. Moreover, in embodiments in which the OCT images <NUM> are displayed as a semi-transparent overlay in the field of view of the microscope, the tracking of the retina may additionally facilitate proper positioning of that overlay.

Although surgical microscope <NUM> is depicted and described as including OCT images displayed through a fixed, single channel (i.e., real-time data projection unit <NUM> is coupled to the optical path of one of the two eyepieces <NUM>), other embodiments are contemplated by the present disclosure (as described with regard to <FIG>, below).

The functionality and operation of tracking unit <NUM> will now be discussed in additional detail, in accordance with certain embodiments.

Tracking unit <NUM> may use various techniques to determine and track the location of surgical instrument <NUM> within a microscope image (e.g., the X-Y position of distal tip <NUM> within a microscope image). In certain embodiments, surgical instrument <NUM> may have attached or embedded sensing devices. For example, surgical instrument <NUM> may have one or more gyroscopes, accelerometers, gravity sensors, linear acceleration sensors, rotation vector sensors, geomagnetic field sensors, or other types of sensors, to sense changes in position, location, or movement. Data generated from such sensors may be provided to tracking unit <NUM>, which may analyze the data to determine the location, position, and/or movement of surgical instrument <NUM>.

In certain embodiments, tracking unit <NUM> may use image-based processing techniques to determine and track the location of surgical instrument <NUM>. For example, tracking unit <NUM> may employ machine vision or computer vision algorithms to images acquired from imaging unit <NUM> to determine and track the location of surgical instrument <NUM>. Tracking unit <NUM> may apply feature recognition or extraction techniques and/or motion-based object tracking and image processing algorithms (e.g., edge detection, corner detection, blob detection, blob extraction, ridge detection, scale-invariant feature transform, motion detection, background subtraction, frame difference, optical flow, thresholding, template matching, Hough transform, etc.). Additionally or alternatively, tracking unit <NUM> may use region-based object tracking techniques.

For example, tracking unit <NUM> may apply feature recognition or extraction techniques (e.g., edge detection, corner detection, blob detection, blob extraction, ridge detection, scale-invariant feature transform, motion detection, optical flow, thresholding, template matching, Hough transform, etc.) to depth-resolved images to detect or isolate distal tip <NUM> and the retina of eye <NUM> within image data obtained by a depth imaging system. Tracking unit <NUM> may obtain and store a reference image of eye <NUM>, and may compare images obtained during a surgical procedure with the reference image to determine the location and movement of, and distance between, a distal tip of surgical instrument <NUM> and the retina in eye <NUM>.

According to certain embodiments, tracking unit <NUM> may be programmed to determine and track the location of surgical instrument <NUM> using feature-based object tracking to extract and search for unique features of surgical instrument <NUM> (e.g., a contour, edge, shape, color, corner/interest point, etc.) within an image received from imaging unit <NUM>. In such embodiments, tracking unit <NUM> may use marker <NUM> to assist in determining and tracking the location of surgical instrument <NUM>. As shown in <FIG>, surgical instrument <NUM> may include a marker <NUM> positioned at or near a distal portion <NUM> which has a high contrast feature in the visible light or infrared spectrum, or other spectral ranges detectable by imaging unit <NUM>. High contrast may be obtained by using a color or pattern distinguishable from colors or patterns in the fundus of eye <NUM>. A light source <NUM> such as endo-illuminator or a fiber illuminator may emit an imaging light to illuminate a fundus of eye <NUM>. Marker <NUM> is discussed in additional detail below with respect to <FIG>.

Tracking unit <NUM> may also determine and track the distance and proximity between surgical instrument <NUM> and tissues in eye <NUM>. In certain embodiments, tracking unit <NUM> receives depth-resolved image data from OCT imaging system <NUM> or an alternative depth imaging system capable of determining tissue and instrument depth and position. Tracking unit <NUM> may apply image-based processing techniques to such image data in order to determine and/or extract location and position data related to surgical instrument <NUM> and tissues in eye <NUM>. Based on this analysis, tracking unit <NUM> may calculate a distance between parts of surgical instrument <NUM> (e.g., distal tip <NUM>) and tissue in eye <NUM> (e.g., the retina). Tracking unit <NUM> may track changes in this distance by processing a stream of image data obtained by the depth imaging system in real time. Tracking unit <NUM> may also store image analysis data in order to calculate and track changes in the location and movement of, and distance between, surgical instrument <NUM> and tissues in eye <NUM>.

In certain embodiments, tracking unit <NUM> receives image data from a depth resolved imaging system, such as OCT system <NUM>. Tracking unit <NUM> may be configured to analyze depth-resolved images to identify features depicted in the image, such as a retina and/or surgical tools. Tracking unit <NUM> may register depth image data and identify the coordinates of identified features in the depth-resolved image. Such coordinates may be digitized using computer vision or machine vision algorithms, such as edge or blob detection, and may be used to calculate the distance between features within the image.

In certain embodiments, a calibration sample material may be used to form a <NUM>-D array of reference marks at location within known position coordinates. A depth-resolved image (e.g., an OCT image) may be obtained to establish a mapping relationship between known position coordinates of the reference marks and the depth-resolved images of the reference marks in the obtained depth-resolved image. This mapping relationship may be stored as digital calibration data and may be used to calculate the distance between features in the depth-resolved image (e.g., a retina and a cutting tip of a surgical tool) and for controlling the imaging beam of the depth-resolved imaging system.

In embodiments in which the depth-resolved imaging probe is separate from surgical instrument <NUM>, the depth-resolved image may depict features that include a surgical instrument <NUM> and the retina of eye <NUM>. For example, tracking unit <NUM> may receive depth-resolve images (e.g., A-scans or B-scans) that depict the retina of eye <NUM> and the distal tip surgical instrument <NUM>. Tracking unit <NUM> may determine the distance or proximity between the retina of eye <NUM> and the distal tip of surgical instrument <NUM> based on characteristics of the received depth image. For example, in such images, the retina of eye <NUM> and the distal tip of surgical instrument <NUM> may appear separated by a space within the image (provided they are not in contact). Tracking unit <NUM> may determine the distance or proximity between the retina of eye <NUM> and the distal tip of surgical instrument <NUM> based on the degree of separation between them in the image. For example, digitized coordinates may be used as discussed above. As another example, tracking unit <NUM> may determine the distance or proximity based on the number of pixels separating retina of eye <NUM> and the distal tip surgical instrument <NUM> in a depth-resolved image, which may have a fixed z-depth resolution. In a depth-resolved image having a fixed z-depth (such as OCT images), distance and/or proximity between features in the image (e.g., a tool top and the retina) may be calculated based on pixels, which individually correspond to a fixed distance. Tracking unit <NUM> may identify and process pixel counts in the depth-resolved image to determine the distance between imaged objects. To facilitate this approach, tracking unit <NUM> may advantageously cause the depth imaging system to direct the depth imaging beam to a location in the vicinity of surgical instrument <NUM>, as described above.

In certain embodiments, tracking unit <NUM> may receive image data from a depth resolved imaging probe that is at least partially integrated within surgical instrument <NUM>. In one example, surgical instrument <NUM> may include an integrated depth-imaging probe. Surgical instrument <NUM> may include one or more optical fibers used by a depth-imaging system (e.g., OCT system <NUM>) to transmit an imaging beam, transmit reflected light from the eye, and generate a depth-resolved image (e.g., A-scans or B-scans) of eye <NUM>. Such depth-resolved images may depict the retina of eye <NUM>, without depicting surgical instrument <NUM> (because such images are obtained from the vantage point of the tip of instrument <NUM>). Tracking unit <NUM> may determine the distance or proximity between the retina of eye <NUM> and the distal tip of surgical instrument <NUM> based on characteristics of the received depth image. For example, as discussed above, digitized coordinates may be used, or tracking unit <NUM> may identify and process pixel counts in the depth-resolved image to determine the distance between imaged objects. In certain embodiments, tracking unit <NUM> may determine the distance or proximity based on the pixels between the edge of the image (or other feature corresponding to the distal tip of surgical instrument <NUM>) and a retina of eye <NUM> depicted in a depth-resolved image, which may have a fixed z-depth resolution. As noted above, in a depth-resolved image having a fixed z-depth (such as OCT images), distance and/or proximity between features in the image (e.g., a tool and a retina) may be calculated based on pixels, which correspond to a fixed distance. In such embodiments, the proximity of the surgical tool to eye tissue can be continuously determined without actively directing an imaging beam toward the surgical instrument <NUM>.

Tracking unit <NUM> may additionally or alternatively use various image-based processing techniques (e.g., machine vision or computer vision algorithms, motion-based object tracking algorithms, region-based object tracking techniques, and/or feature-based object tracking) to analyze depth-resolved images (e.g., OCT images) and determine and track the distance between surgical instrument <NUM> and tissues in eye <NUM>. For example, tracking unit <NUM> may apply feature recognition or extraction techniques (e.g., edge detection, corner detection, blob detection, blob extraction, ridge detection, scale-invariant feature transform, motion detection, optical flow, thresholding, template matching, Hough transform, etc.) to depth-resolved images to detect or isolate distal tip <NUM> and the retina of eye <NUM> within image data obtained by a depth imaging system. Tracking unit <NUM> may obtain and store a reference image of eye <NUM>, and may compare images obtained during a surgical procedure with the reference image to determine the location and movement of, and distance between, a distal tip of surgical instrument <NUM> and the retina in eye <NUM>.

Tracking unit <NUM> may generate an indicator (e.g., a number, shape, color, figure, or symbol, or other graphic element) for display by real-time data projection unit <NUM> or display <NUM> to identify the location, orientation, and depth of distal tip <NUM>, and its proximity to the retina. For example, tracking unit <NUM> may generate an indicator (e.g., a dot or arrow) and overlay the indicator into a microscope image at the position of distal tip <NUM>, thereby highlighting its location without interfering with a surgeon's view of surrounding tissue. In certain embodiments, tracking unit <NUM> applies the indicator as an overlay at the determined location of distal tip <NUM> in image <NUM>. In other embodiments, the indicator may be an overlay located elsewhere in the microscope image. Processor <NUM> may track the location, orientation, and depth/proximity of distal tip <NUM> in order to provide a dynamic indicated updated in real time. Accordingly, the indicator may assist a surgeon by providing an accurate, real-time indication of the distance between distal tip <NUM> and the retina, which may be difficult to precisely discern from a stereo microscope image.

Tracking unit <NUM> may generate and communicate a signal specifying the position of the indicator in an image to cause real-time data projection unit <NUM> or display <NUM> to project or display the indicator as an overlay on a microscope image. Tracking unit <NUM> may alternatively generate a modified fundus image that includes the indicator overlay, and communicate the modified image to real-time data projection unit <NUM> or display <NUM> for presentation to a user.

In some examples, an aspect of the indicator may directly indicate the distance between distal tip <NUM> and the retina. For example, the indicator may be a numerical value specifying the distance (e.g., "<NUM>" for <NUM>, "<NUM>" for <NUM>, and "<NUM>" for <NUM>)). In some examples, tracking unit <NUM> may generate an indicator having a particular characteristic (e.g., size, shape, color, flash rate, brightness, transparency, quantity, etc.) that indirectly indicates the distance between distal tip <NUM> and the retina of eye <NUM>. Additionally, the indicator may be modified or adjusted as the distance between distal tip <NUM> and the retina of eye <NUM> changes. In certain embodiments, a characteristic of the visual indicator (e.g., size, color) is modified incrementally, and proportional to the change in distance, to intuitively convey the movement of distal tip <NUM>.

In certain embodiments, tracking unit <NUM> may associate particular colors with particular distances between distal tip <NUM> and the retina. For example, tracking unit <NUM> may associate a green indicator with a distance of <NUM> or more, a yellow indicator with a distance of <NUM>, and red indicator with distances of less than <NUM>. The color scheme may be gradated so that the indicator transitions from green to yellow to red in intermediate shades as the distance decreases from <NUM> to <NUM>. The incremental change in color may be proportional to the incremental change in distance.

In certain embodiments, tracking unit <NUM> may associate a particular indicator size with particular distances, such that the generated indicator will become gradually larger or smaller as the distance between distal tip <NUM> and the retina of eye <NUM> decreases. For example, tracking unit <NUM> may generate a visual indicator as a triangular overlay on distal tip <NUM> that becomes larger as it moves away from the retina and smaller as it approaches the retina. Tracking unit <NUM> may associate smaller sizes with greater depths of distal tip <NUM>, to provide the impression that the indicator is moving away from the system operator as it approaches the retina. The incremental change in size may be proportional to the incremental change in distance. Tracking unit <NUM> may also set upper and lower limits on the size of the indicator so as to avoid obstructing the surgeon's view of surrounding tissue or scenarios where the indicator may become too small to be seen clearly.

Any number of indicator characteristics may be modified to indicate distance. In some examples, as the distance between distal tip <NUM> and the retina of eye <NUM> decreases, tracking unit <NUM> modifies the indicator to become increasingly brighter, transition from transparent to opaque, begin flashing, flash at an increasing rate, and/or change shape or form. For example, tracking unit <NUM> may cause an indicator to flash when the distance between distal tip <NUM> and retina is less than a preset threshold, such as <NUM>.

Some examples may utilize a combination of indicator characteristics to indicate distance. For example, tracking unit <NUM> may associate a minimum indicator size with a distance threshold (to avoid decreasing the size of the indicator further), and may modify the indicator to become a different color, grow brighter, and/or flash as the distance exceeds the threshold.

Incrementally-adjustable or continuously variable characteristics such as size, color, flash rate, brightness, etc. advantageously allow the characteristic to be modified in proportion to the incremental change in distance. Correlating indicator changes to distance in this manner advantageously provides a surgeon with an intuitive indication of distance, and changes in distance, that can be easily monitored during a procedure.

Certain embodiments may modify the position or location of an indicator based on distance. For example, certain embodiments may provide a visual indicator that appears only when a distance between distal tip <NUM> and the retina drops below a threshold (e.g., <NUM>). As another example, certain embodiments may display a visual indicator in a first location when tissue of eye <NUM> and surgical instrument <NUM> are separated by at least a threshold distance (e.g., near the outer edge of the microscope image), and relocate the visual indicator to a second location (e.g., near distal tip <NUM>) when the threshold is met.

In certain embodiments, the indicator may additionally or alternatively indicate an orientation, e.g., a pointing angle, of surgical instrument <NUM>. For example, an arrow may be used as the indicator to indicate the pointing direction of surgical instrument <NUM>. The indicator may also include an image, such as an OCT image of a region of a retina of eye <NUM>, or a surgical setting parameter, such as a cutting speed of a vitrectomy probe.

Various embodiments of tracking unit <NUM> may allow a user to configure the appearance, characteristics, and behavior of the indicator. For example, a user may configure a particular size and shape for an indicator, and may configure how the indicator is modified to indicate distance. Certain embodiments of tracking unit <NUM> include a user interface to receive user input regarding customized settings defining when, where, and how the visual indicator will be displayed and modified. In certain embodiments, a user may control whether a visual indicator is displayed via a pedal, switch, soft key, console button, voice command, or other input mechanism. The visual indicator may also be configured to appear based on a timer, particular movements of surgical instrument <NUM>, or the position or location of surgical instrument <NUM> and/or distal tip <NUM>.

<FIG> and <FIG> illustrate a visual indicator according to certain embodiments. <FIG> and <FIG> illustrate, on the left, eye <NUM> which includes retina <NUM>. Surgical instrument <NUM> (here, a vitrectomy probe) and light source <NUM> are inserted into a posterior region of eye <NUM>. <FIG> and <FIG> further illustrate, on the right, corresponding microscope images displayed to a system operator via eyepieces <NUM> (with input from real-time data projection unit <NUM>) and/or display <NUM>. The microscope images show fundus <NUM>, surgical instrument <NUM> (which may include marker <NUM>, not shown), and indicator <NUM> generated by tracking unit <NUM>. As shown in the embodiments of <FIG> and <FIG>, indicator <NUM> appears as an overlay superimposed at distal tip <NUM> of surgical instrument <NUM>. Indicator <NUM> may appear as an overlay superimposed entirely on distal tip <NUM> (such that no part of the indicator is outside a boundary formed by the edged of distal tip <NUM>), so as not to block a surgeon's view of fundus <NUM>. According to certain embodiments, indicator <NUM> may alternatively appear as an indicator partially covering distal tip <NUM>, or near distal tip <NUM> without covering it. As surgical instrument <NUM> moves within eye <NUM>, tracking unit <NUM> tracks distal tip <NUM> and maintains the indicator <NUM> as an overlay on or near distal tip <NUM>. Additionally, a characteristic of indicator <NUM> is shown to be modified incrementally, and (although not to scale) proportional to the change in distance between distal tip <NUM> and retina <NUM>.

In the embodiments of <FIG>, indicator <NUM> becomes as larger distal tip <NUM> approaches retina <NUM>. In <FIG>, the distal tip <NUM> of surgical instrument <NUM> is separated by a relatively large distance from retina <NUM>. Accordingly, tracking unit <NUM> generates for display a relatively large indicator <NUM> (here, a triangle) that overlays distal tip <NUM> in the microscope image shown on the right. In <FIG>, the distance between distal tip <NUM> and retina <NUM> has decreased, and in <FIG> it has decreased even further. As distal tip <NUM> approaches retina <NUM>, tracking unit <NUM> decreases the size of indicator <NUM> to convey that distal tip <NUM> is getting closer to retina <NUM> (and/or further from the imaging unit or system operator). In this manner, the size of visual indicator <NUM> may be increased or decreased in proportion to the change in distance between distal tip <NUM> and retina <NUM>. Because tracking unit <NUM> continuously (or frequently) adjusts the size of visual indicator <NUM> in small increments, it may provide a surgeon with an intuitive, real-time indication of the depth of distal tip <NUM> and its distance from retina <NUM>. Thus, in this example, a characteristic of the visual indicator (in this example, size) is modified in response to a change in distance between distal tip <NUM> and retina <NUM>. This example further illustrates how indicator <NUM> may be modified to track the movement of distal tip <NUM> within the microscope images. That is, indicator <NUM> is maintained as an overlay even as distal tip <NUM> moves within the microscope image.

<FIG> illustrates how additional characteristics and indicators may be used in certain embodiments. As with <FIG>, <FIG> shows that indicator <NUM> (here, a circle) becomes relatively smaller as distal tip <NUM> approaches retina <NUM> in <FIG> and <FIG>, and indicator <NUM> is maintained as an overlay even as distal tip <NUM> moves within the microscope images on the right side of the figure.

In addition, indicator <NUM> of <FIG> changes transparency according to the distance calculated by tracking unit <NUM>. In <FIG>, where distal tip <NUM> of surgical instrument <NUM> is a relatively large distance from retina <NUM>, indicator <NUM> appears as nearly transparent. In <FIG>, as the distance decreases, indicator <NUM> is semi-transparent. And in <FIG>, indicator <NUM> becomes opaque and begins flashing when the distance drops below a predetermined threshold distance (e.g., <NUM>). Indicator <NUM> may additionally become brighter (or more intense) as the distance decreases, such that it transitions from a low brightness in <FIG> to medium brightness in <FIG> to a maximum brightness in <FIG>. Accordingly, various characteristics of visual indicator <NUM> may be altered in proportion to the change in distance between distal tip <NUM> and retina <NUM> to provide an intuitive indication of the depth of distal tip <NUM>.

The microscope images of <FIG> also depict a second indicator <NUM>, which is a numerical value of the determined distance between distal tip <NUM> and retina <NUM> (here, in millimeters). Secondary indicator <NUM> is shown here as an overlay positioned near distal tip <NUM>, but it may be positioned elsewhere within the microscope view in other embodiments.

<FIG> additionally depicts a third indicator <NUM>, which comprises a colored ring about the outer edge of the microscope view. In this example, indicator <NUM> changes color according to the determined distance, transitioning from green to yellow and red in <FIG>, <FIG>, and <FIG>, respectively. Indicator <NUM> may change characteristics in the same manner as indicator <NUM> to indicate proximity and/or distance (e.g., change brightness, transparency, flash rate, etc.).

It should be understood that the examples discussed above are nonlimiting, and the present disclosure contemplates that indicator <NUM> may take any suitable form and have any suitable characteristics to indicate the position, orientation, and depth of distal tip <NUM>, or its distance from or proximity to the retina.

<FIG> is a flow chart illustrating a method <NUM> for determining, tracking and indicating the depth and location of surgical instrument <NUM> in accordance with certain embodiments. Certain examples of tracking unit <NUM> include a processor configured to (or may store software in memory that, when executed by a processor, is operable to) perform the steps of method <NUM>.

At step <NUM>, tracking unit <NUM> receives an image of the fundus. For example, tracking unit <NUM> may receive one or more photographs or video frames captured by imaging unit <NUM>.

At step <NUM>, tracking unit <NUM> may perform contrast and feature enhancement processing on received image. For example, tracking unit <NUM> may receive an image in Red-Green-Blue (RGB) format. Tracking unit <NUM> may convert the RGB format image into a Hue-Saturation-Value (HSV) space.

At step <NUM>, tracking unit <NUM> may determine a first-order estimation mask of a marker (e.g., marker <NUM>) in the image. For example, based on a predetermined color of marker <NUM>, tracking unit <NUM> may apply criteria to the hue and saturation channels of the HSV image to separate marker <NUM> from a background, in order to bring out and estimate the image of marker <NUM>.

At step <NUM>, tracking unit <NUM> may extract the image of marker <NUM> from a fundus image and identify the position of the marker. For example, tracking unit <NUM> may utilize a blob detection process to detect a boundary of marker <NUM> in image <NUM> by searching for regions of approximately constant properties in the image frame. Thus, tracking unit <NUM> may find the boundary of marker <NUM> and extract it from the image frame to determine its position in an image.

At step <NUM>, tracking unit <NUM> may analyze the shape and orientation of marker <NUM> extracted from an image frame and may determine the orientation of the marker based on a predetermined pattern or color (e.g., a location and direction of stripes). For example, if marker <NUM> has stripes, tracking unit <NUM> may determine the orientation of marker <NUM> based on the orientation and direction of the stripes.

At step <NUM>, tracking unit <NUM> may determine the position and orientation of distal tip <NUM> of surgical instrument <NUM> within an image frame. In particular embodiments, tracking unit <NUM> may determine the location and orientation of distal tip <NUM> in an image based on the location and orientation determinations for marker <NUM> (described in preceding steps). To facilitate such determinations, marker <NUM> may be positioned at a predetermined distance from distal tip <NUM> and may have a pattern that indicates a pointing direction of surgical instrument <NUM> (e.g., a strip, stripes, or an arrow). Thus, based on the position and pattern of marker <NUM>, tracking unit <NUM> may determine the position of distal tip <NUM> and the pointing direction or orientation of surgical instrument <NUM>.

At step <NUM>, tracking unit <NUM> receives image data from an imaging system that generates a depth-resolved image of eye <NUM>. Such images may include surgical instrument <NUM> (if obtained by an imaging probe external to the instrument), or may be obtained using surgical instrument <NUM> itself (e.g., via one or more optical fibers extending to the tip of instrument <NUM>). In certain embodiments, tracking unit <NUM> receives image data from OCT system <NUM>. In other embodiments, tracking unit <NUM> may receive image data from an alternative system that provides depth-resolved or three-dimensional image data, such as an ultrasound imaging system, a multispectral imaging system, a computerized axial tomography (CAT) scan system, a magnetic resonance imaging (MRI) system, a positron emission tomography (PET) imaging system, or other imaging system. Tracking unit <NUM> may analyze the received image data to identify the depth of, and distance or proximity between, distal tip <NUM> and the retina of eye <NUM>.

At step <NUM>, tracking unit <NUM> analyzes the received image data to determine the depth and proximity of distal tip <NUM> and the retina of eye <NUM>. Tracking unit <NUM> may process an image received at step <NUM> (or data related to or extracted from the image) to calculate a distance between a part of surgical instrument <NUM> (e.g., distal tip <NUM>) and tissue in eye <NUM> (e.g., the retina). For example, tracking unit <NUM> may register depth image data and identify the coordinates of identified features in the depth-resolved image. Such coordinates may be digitized using computer vision or machine vision algorithms, such as edge or blob detection, and may be used to calculate the distance between features within the image.

In certain embodiments, tracking unit <NUM> may determine the distance or proximity between the retina of eye <NUM> and the distal tip <NUM> based on characteristics of the received depth image. Tracking unit <NUM> may determine the distance or proximity between the retina of eye <NUM> and the distal tip <NUM> based on the degree of separation between them in the image. In certain embodiments, tracking unit <NUM> may determine distance or proximity based on the number of pixels separating retina of eye <NUM> and the distal tip <NUM> in an OCT image, which may have a fixed z-depth resolution. In certain embodiments, tracking unit <NUM> may determine the distance or proximity based on the pixels between the edge of the image (or other feature corresponding to the distal tip <NUM>) and a retina of eye <NUM> depicted in an OCT image.

At step <NUM>, tracking unit <NUM> generates an indicator for display as an overlay on distal tip <NUM> within a microscope image. Certain embodiments of tracking unit <NUM> may generate one or more visual indicators based on the determined location, depth, and orientation of surgical instrument <NUM>, and overlay the indicator into a microscope image for surgical guidance. For example, based on the determinations of steps <NUM> and <NUM>, tracking unit <NUM> may generate an indicator for display by real-time data projection unit <NUM> or display <NUM> to alert a system operator of the proximity of distal tip <NUM> to particular eye tissue, such as the retina. As described above, a characteristic of the indicator may indicate the proximity of or distance between distal tip <NUM> and the retina of eye <NUM>. Tracking unit <NUM> may apply the indicator as an overlay at the determined location of distal tip <NUM> in image <NUM>. In this manner, the indicator can alert the system operator as distal tip <NUM> approaches the retina of eye <NUM>.

Tracking unit <NUM> may perform method <NUM> for multiple fundus images <NUM> received from imaging unit <NUM> to track the position and orientation of distal tip <NUM>, and the distance between distal tip <NUM> and the retina of eye <NUM>, in real time. Thus, real-time data projection unit <NUM> and display <NUM> may project or display the generated indicator(s) as an overlay in a real-time video of the fundus to track the position and movement of distal tip <NUM>, as well as its proximity to the retina of eye <NUM>.

<FIG> illustrates various examples of marker <NUM>. Marker <NUM> may have a ring, ribbon shape configured to wrap around the distal portion <NUM> of surgical instrument <NUM>. Marker <NUM> may have an inner surface <NUM> and an outer surface <NUM>. Inner surface <NUM> may have adhesives and may be configured to adhere or bond to an exterior surface of surgical instrument <NUM>. The exterior surface of distal portion <NUM> may have a circumferential groove configured to accommodate the ring, ribbon shape marker <NUM>. Thus, marker <NUM> may fit securely in the circumferential groove. Outer surface <NUM> may have colors or patterns configured to distinguished marker <NUM> from other elements in a fundus image.

One or more markers <NUM> may be used for surgical instrument <NUM>. Marker <NUM> may be formed of bio-compatible and/or synthetic materials, such as sterile plastic. In some embodiments, marker <NUM> may be a layer of paint inscribed on an exterior surface of the distal portion <NUM> of surgical instrument <NUM>. Markers <NUM> may overlap one another or be separate. Markers <NUM> may have one or more high-contrast colors, such as green, which does not appear in a typical fundus image. Thus, a green marker <NUM> may be distinguished from other elements in the fundus image.

Markers <NUM> may have various color, texture, or special contrast characteristics. Markers <NUM> may include patterns that may identify an orientation and angle of instrument <NUM>. For example, as shown in <FIG>, marker 147a may have a solid high-contrast color. When the ring, ribbon shape marker 147a is cut open, the marker 147a may be a ribbon in solid color. In another example, marker 147b may have a texture pattern that may distinguish the marker 147b from the background fundus image. Exemplary marker 147c may include an infrared color configured to reflect or emit infrared light. Markers <NUM> with various spectral absorption/emission also may be used.

Markers <NUM> may include letters, numbers, bar codes, pattern, symbols, or pictures. Exemplary marker 147d may include letters. As shown in <FIG>, assuming that marker 147d wraps <NUM> degrees around the distal portion <NUM> of the instrument <NUM>, a letter "A" may be positioned near the zero degree position and the letter "E" may be positioned near the <NUM> degree position. Letters <NUM> "B," "C," and "D" may be positioned in between "A" and "E" at respective positions. Thus, based on the orientation of the letters, the rotational position of marker 147d and indirectly the rotational position of surgical instrument <NUM> may be determined. Exemplary marker 147e may include numbers "<NUM>" to "<NUM>. " Similarly, the numbers may indicate a rotational position of surgical instrument <NUM>. Further, the orientation of the letters or number also may indicate a tilting angle of the surgical instrument <NUM>. For example, the numbers or letters may be orientated relatively to the distal tip <NUM> such that the bottoms of the numbers or letter face toward the distal tip <NUM>. Thus, based on the orientation of the numbers or letters, the tilting angle of the distal tip <NUM> may be determined.

Exemplary marker 147f may include barcodes or stripes. The direction of the stripes may indicate a tilting angle of the surgical instrument <NUM>. Further, the number of stripes may vary to indicate a rotational position of marker 147f and indirectly, the rotational position of the surgical instrument <NUM>. Marker <NUM> has various dot patterns. The number of dots may indicate the rotational position of marker 147f and the alignment of the dots may indicate a tilting angle of the marker 147f. Other symbols also may be used on markers <NUM>. For example, various symbols, such as shapes or non-character symbols may be used at different rotational positions of markers <NUM> and 114i to indicate rotational positions. In addition, a picture may be used to indicate rotational and tilt positions of marker 114j. Other patterns or symbols that may indicate an orientation and position of the surgical instrument <NUM> also may be used on the markers <NUM>.

<FIG> illustrate embodiments of ophthalmic surgical microscope <NUM> having switchable single channel data injection, according to certain embodiments of the present disclosure. Although <FIG> do not depict certain components of ophthalmic surgical microscope <NUM> as depicted in <FIG> for the sake of simplicity, the present disclosure contemplates that those components be included and that they function in substantially the same manner as described above with regard to <FIG>.

In the embodiment depicted in <FIG>, ophthalmic surgical microscope <NUM> includes a real-time data projection unit <NUM> capable of single channel data injection (i.e., the images injected by real-time data projection unit <NUM> are viewable through only one of the two eyepieces <NUM>, as in <FIG>). However, unlike the embodiment depicted in <FIG>, the embodiment depicted in <FIG> provides the ability to change which channel (i.e., eyepiece <NUM>) onto which the data is injected. More particularly, <FIG> depicts an embodiment in which one or both of real-time data projection unit <NUM> and beam splitter <NUM> can translate side to side in order to change the channel onto which data is injected while <FIG> depicts an embodiment in which the assembly of real-time data projection unit <NUM> and beam splitter <NUM> rotatable about a midpoint of surgical microscope <NUM> in order to change the channel onto which data is injected. As a result, a surgeon may be provided the flexibility to select which eye is used to view the injected data.

<FIG> illustrates an embodiment of ophthalmic surgical microscope <NUM> having two-channel data injection, according to certain embodiments of the present disclosure. Although <FIG> does not depict certain components of ophthalmic surgical microscope <NUM> as depicted in <FIG> for the sake of simplicity, the present disclosure contemplates that those components be included and that they function in substantially the same manner as described above with regard to <FIG>.

In the embodiment depicted in <FIG>, surgical microscope <NUM> include a single real-time data projection unit <NUM> and two beam splitters <NUM> (130a and 130b) each associated with a corresponding channel of the microscope. Beam splitters 130a and 130b may be configured such that the data projected by real-time data projection unit <NUM> is duplicated and viewable via both of the eyepieces <NUM>. Reflectivities of the beam splitters 130a and 130b may be selected such that the brightness of the image viewable through each eyepiece <NUM> is the same. Moreover, beam splitters may be movable in order to change the shifted within the surgeon's field of view. Alternatively, movement within the surgeon's field of view may be achieved by placing a beam deflection device (e.g., an acoustical optical deflector) in the optical path of the image projected by real-time data projection unit <NUM>.

<FIG> illustrates an alternative embodiment of ophthalmic surgical microscope <NUM> having two-channel data injection, according to certain embodiments of the present disclosure. Although <FIG> does not depict certain components of ophthalmic surgical microscope <NUM> as depicted in <FIG> for the sake of simplicity, the present disclosure contemplates that those components be included and that they function in substantially the same manner as described above with regard to <FIG>.

In the embodiment of <FIG>, two real-time data projection units <NUM> are included (116a and 116b). Each real-time data projection unit projects an image, which is coupled into the optical path of the surgical microscope by a corresponding beam splitter <NUM>. Because each real-time data projection unit can inject a unique image, the embodiment of <FIG> may facilitate <NUM>-D perception. More particularly, each real-time data projection unit <NUM> may project the same image but with slightly different perspectives so as to provide <NUM>-D perception when viewed through eyepieces <NUM>.

<FIG> illustrate embodiments of ophthalmic surgical microscope <NUM> having two-channel data injection with <NUM>-D perception, according to certain embodiments of the present disclosure. Although <FIG> do not depict certain components of ophthalmic surgical microscope <NUM> as depicted in <FIG> for the sake of simplicity, the present disclosure contemplates that those components be included and that they function in substantially the same manner as described above with regard to <FIG>.

In the embodiments depicted in <FIG>, <NUM>-D perception is facilitated using one real-time data projection unit <NUM> rather than two (as in the embodiment described above with regard to <FIG>). In the embodiment depicted in <FIG>, a single real-time data projection unit <NUM> projects side-by-side images, which may be slightly different to provide <NUM>-D perception (as described above). The projected side-by-side images may be split by a beam splitter <NUM> and projected into each eyepiece <NUM> by beam splitter 130a and 130b. In certain embodiments, filters 502a and 502b may also be placed in the optical path of the projected images to further facilitate <NUM>-D perception.

In the embodiment depicted in <FIG>, real-time data projection unit <NUM> may project a color-coded image (such as a red and cyan coded image in anaglyph), and that color coded image may pass through beam splitters 504a and 504b to be directed toward the two channels of surgical microscope <NUM>. Filters 506a and 506b may be placed in the optical path of the image for each channel to separate the color-coded information. For example, filter 506a (such as a red filter) may be inserted into the left channel and filter 506b (such as a cyan filter) may be added to the right channel to separate the red/cyan information in the projected image. By properly calibrating the projected image, <NUM>-D perception may be provided without the need for the surgeon to wear extra glasses or optical devices.

In the embodiment depicted in <FIG>, real-time data display unit <NUM> may be a polarized display/projector (such as a polarization modulated projector) and may project a polarization encoded image. The projected polarization encoded image may pass through polarizing beam splitters 508a and 508b to be divided between the two channels. For example, a p polarized image may be split into one eye (designated as 510a) while an s polarized image will be split into the other eye (designated as 510b). Additionally or alternatively, by inserting wave plates 512a and 512b into the two channels, a left hand circular polarized image may be split into one eye while a right hand circular polarized image may be split into the other eye. By properly calibrating the projected image, <NUM>-D perception may be provided without the need for the surgeon to wear extra glasses or optical devices.

Claim 1:
An ophthalmic surgical system, comprising:
an imaging unit (<NUM>) configured to generate a fundus (<NUM>) image of an eye (<NUM>);
a depth imaging system (<NUM>) configured to generate a depth-resolved image of the eye (<NUM>);
a tracking system (<NUM>) for tracking a location, depth, proximity and movement of a surgical instrument (<NUM>), the tracking system (<NUM>) communicatively coupled to the imaging unit (<NUM>) and the depth imaging system (<NUM>), the tracking system (<NUM>) comprising a processor (<NUM>) and memory (<NUM>) configured to:
analyze the fundus (<NUM>) image generated by the imaging unit (<NUM>) to determine a location of a distal tip (<NUM>) of a surgical instrument (<NUM>) in the fundus (<NUM>) image; analyze the depth-resolved image generated by the depth imaging (<NUM>) system to determine a distance between the distal tip (<NUM>) of the surgical instrument (<NUM>) and a retina (<NUM>) of the eye (<NUM>) in the depth direction;
generate a visual indicator (<NUM>) to overlay a portion of the fundus (<NUM>) image, the visual indicator (<NUM>) indicating the determined distance between the distal tip (<NUM>) and the retina (<NUM>) in the depth direction;
modify the visual indicator (<NUM>) to track a change in the location of the distal tip (<NUM>) within the fundus (<NUM>) image in real-time; and
modify the visual indicator (<NUM>) to indicate a change in the distance between the distal tip (<NUM>) of the surgical instrument (<NUM>) and the retina (<NUM>) in the depth direction in real-time,
characterized in that the processor (<NUM>) and memory (<NUM>) of the tracking system (<NUM>) are further configured to generate the visual indicator (<NUM>) indicating the determined distance between the distal tip (<NUM>) and the retina (<NUM>) in the depth direction to overlay the distal tip (<NUM>) of the surgical instrument (<NUM>) within the fundus (<NUM>) image at the determined location of the distal tip (<NUM>) of the surgical instrument (<NUM>) in the fundus image and further configured to modify a brightness or transparency of the visual indicator to indicate a change in the distance between the distal tip of the surgical instrument and the retina in the depth direction in real-time; wherein the brightness increases or the transparency decreases transitioning from transparent to opaque, as the distance between the distal tip of the instrument and the retina decreases.