Stereoscopic visualization camera and platform

A stereoscopic visualization camera and platform are disclosed. An example stereoscopic visualization camera includes a first plurality of lenses positioned along a first optical path and a first image sensor to record a first image stream of a target site from light in the first optical path. The stereoscopic visualization camera also includes a second plurality of lenses positioned along a second optical path, parallel to the first optical path, and a second image sensor to record a second image stream of the target site from light in the second optical path. The stereoscopic visualization camera also includes a processor configured to reduce spurious parallax between the first and second images streams by selecting pixel sets of pixel grids of the first and second image sensors such that zoom repeat points (“ZRP”) are located at a center of the respective pixel grids.

BACKGROUND

Surgery is art. Accomplished artists create works of art that far exceed the capabilities of a normal person. Artists use a brush to turn canisters of paint into vivid images that provoke strong and unique emotions from viewers. Artists take ordinary words written on paper and turn them into dramatic and awe-inspiring performances. Artists grasp instruments causing them to emit beautiful music. Similarly, surgeons take seemingly ordinary scalpels, tweezers, and probes and produce life-altering biological miracles.

Like artists, surgeons have their own methods and preferences. Aspiring artists are taught the fundamentals of their craft. Beginners often follow prescribed methods. As they gain experience, confidence, and knowledge, they develop their own unique artistry reflective of themselves and their personal environment. Similarly, medical students are taught the fundamentals of surgical procedures. They are rigorously tested on these methods. As the students progress through residency and professional practice, they develop derivations of the fundamentals (still within medical standards) based on how they believe the surgery should best be completed. For instance, consider the same medical procedure performed by different renowned surgeons. The order of events, pacing, placement of staff, placement of tools, and use of imaging equipment varies between each of the surgeons based on their preferences. Even incision sizes and shapes can be unique to the surgeon.

The artistic-like uniqueness and accomplishment of surgeons make them weary of surgical tools that change or alter their methods. The tool should be an extension of the surgeon, operating simultaneously and/or in harmonious synchronization. Surgical tools that dictate the flow of a procedure or change the rhythm of a surgeon are often discarded or modified to conform.

In an example, consider microsurgery visualization where certain surgical procedures involve patient structures that are too small for a human to visualize easily with the naked eye. For these microsurgery procedures, magnification is required to adequately view the micro-structures. Surgeons generally want visualization tools that are natural extensions of their eyes. Indeed, early efforts at microsurgery visualization comprised attaching magnifying lens to head-mounted optical eyepieces (called surgical loupes). The first pair was developed in 1876. Vastly improved versions of surgical loupes (some including optical zooms and integrated light sources) are still being used by surgeons today.FIG. 1shows a diagram of a pair of surgical loupes100with a light source102and magnification lenses104. The 150-year staying power of surgical loupes can be attributed to the fact that they are literally an extension of a surgeon's eyes.

Despite their longevity, surgical loupes are not perfect. Loupes with magnifying lenses and light sources, such as the loupes100ofFIG. 1, have much greater weight. Placing even a minor amount of weight on the front of a surgeon's face can increase discomfort and fatigue, especially during prolonged surgeries. The surgical loupes100also include a cable106that is connected to a remote power supply. The cable effectively acts as a chain, thereby limiting the mobility of the surgeon during their surgical performance.

Another microsurgery visualization tool is the surgical microscope, also referred to as the operating microscope. Widespread commercial development of surgical microscopes began in the 1950s with the intention of replacing surgical loupes. Surgical microscopes include optical paths, lenses, and focusing elements that provide greater magnification compared to surgical loupes. The large array of optical elements (and resulting weight) meant that surgical microscopes had to be detached from the surgeon. While this detachment gave the surgeon more room to maneuver, the bulkiness of the surgical microscope caused it to consume considerable operating space above a patient, thereby reducing the size of the surgical stage.

FIG. 2shows a diagram of a prior art surgical microscope200. As one can imagine, the size and presence of the surgical microscope in the operating area made it prone to bumping. To provide stability and rigidity at the scope head201, the microscope is connected to relatively large boom arms202and204or other similar support structure. The large boom arms202and204consume additional surgical space and reduce the maneuverability of the surgeon and staff. In total, the surgical microscope200shown inFIG. 2could weigh as much as 350 kilograms (“kg”).

To view a target surgical site using the surgical microscope200, a surgeon looks directly though oculars206. To reduce stress on a surgeon's back, the oculars206are generally positioned along a surgeon's natural line of sight using the arm202to adjust height. However, surgeons do not perform by only looking at a target surgical site. The oculars206have to be positioned such that the surgeon is within arm's length of a working distance to the patient. Such precise positioning is critical to ensure the surgical microscope200becomes an extension rather than a hindrance to the surgeon, especially when being used for extended periods of time.

Like any complex instrument, it takes surgeons tens to hundreds of hours to feel comfortable using a surgical microscope. As shown inFIG. 2, the design of the surgical microscope200requires a substantially 90° angle optical path from the surgeon to the target surgical site. For instance, a perfectly vertical optical path is required from the target surgical site to the scope head201. This means that the scope head201has to be positioned directly above the patient for every microsurgical procedure. In addition, the surgeon has to look almost horizontally (or some slight angle downward) into the oculars206. A surgeon's natural inclination is to direct his vison to his hands at the surgical site. Some surgeons even want to move their heads closer to the surgical site to have more precise control of their hand movements. Unfortunately, the surgical microscopes200do not give surgeons this flexibility. Instead, surgical microscopes200ruthlessly dictate that the surgeon is to place their eyes on the oculars206and hold their head at arm's length during their surgical performance, all while consuming valuable surgical space above the patient. A surgeon cannot even simply look down at a patient because the scope head201blocks the surgeon's view.

To make matters worse, some surgical microscopes200include a second pair of oculars208for co-performers (e.g., assistant surgeons, nurses, or other clinical staff). The second pair of oculars208is usually positioned at a right angle from the first oculars206. The closeness between the oculars206and208dictates that the assistant must stand (or sit) in close proximity to the surgeon, further restricting movement. This can be annoying to some surgeons who like to perform with some space. Despite their magnification benefits surgical microscopes200are not natural extensions of a surgeon. Instead, they are overbearing directors in the surgical room.

SUMMARY

The present disclosure is directed to stereoscopic visualization camera and platform that is configured to effectively operate as an extension of a surgeon's eyes while giving the surgeon the freedom to conduct a microsurgery procedure generally without restrictions. The example stereoscopic visualization camera disclosed herein comprises a digital stereoscopic visualization platform with full-range, operator-independent orientation for microsurgical applications. The example stereoscopic visualization camera and platform decouples the micro-surgery visualization system from a surgeon's head and eyes to provide for a wide variety of multi-axis orientations of the surgical visualization system relative to the surgeon and to the target surgical field. As a result, the surgeon is provided with an enhanced magnified view of the surgical site without having to work around a bulky microscope positioned over the patient and in front of the surgeon's face. The example stereoscopic visualization camera accordingly enables a surgeon to complete life-altering microsurgeries comfortably in whatever position suits the surgeon. Moreover, the surgical visualization camera of the present disclosure can be positioned along and about any number of orientations relative to the surgical field that best suit the needs of the surgeon or patient, rather than the physical and mechanical limitations of the visualization apparatus.

The example stereoscopic visualization camera and corresponding platform has many distinct advantages over known monoscopic and stereoscopic cameras. Current monoscopic and stereoscopic cameras are connected to an optical path of a surgical microscope. While being connected to the optical path, the cameras have no control over focus, zooming, and/or setting a working distance. Instead, these controls are located at the scope head of the surgical microscope. In addition, optical elements in a surgical microscope provide generally acceptable image quality for oculars. However, defects in the image quality or slightly misaligned right and left views become more apparent when acquired by a camera and displayed on a video monitor.

The example stereoscopic visualization camera overcomes the above-mentioned issues of known monoscopic and stereoscopic cameras by being configured as a self-contained device that does not rely on external microscope optical elements. The example stereoscopic visualization camera instead internalizes the optical elements that are common on a surgical microscope. The optical elements may be provided on tracks and/or flexures within the camera to allow for manual and/or automatic adjustment. Accordingly, adjustment of the optical elements can be provided through camera controls and/or user input devices connected to the camera, which enables adjustment to be made specifically for the camera. In addition, the optical elements of the stereoscopic visualization camera may be automatically and/or manually adjusted to align focus points of left and right images and reduce visual defects and/or spurious parallax. The end result is a relatively lightweight maneuverable stereoscopic visualization camera that provides a virtually flawless three-dimensional stereoscopic display that allows surgeons to practice their art without visual encumbrances.

In an example embodiment, a stereoscopic imaging apparatus is configured to reduce spurious parallax between first and second images streams acquired or recorded in parallel of a target site. The apparatus includes first optical elements positioned along a first optical path. The first optical elements comprise a first plurality of lenses including a first zoom lens configured to be moveable along the first optical path in a z-direction and a first image sensor to acquire the first image stream of the target site from light in the first optical path. The apparatus also includes second optical elements positioned along a second optical path parallel to the first optical path. The second optical elements comprise a second plurality of lenses including a second zoom lens configured to be moveable along the second optical path in a z-direction and a second image sensor to acquire the second image stream of the target site from light in the second optical path. The apparatus further includes a processor configured to locate a position of a first zoom repeat point (“ZRP”) by causing the first zoom lens to move along the z-direction during a recording of the first image stream, locating a first portion of area that does not move in an x-direction or a y-direction within the images of the first image stream, and determining a first distance between an origin point within at least one of the images of the first image stream and the first portion of the area as the position of the first ZRP. The example processor is also configured to determine a first pixel set of a first pixel grid of the first image sensor using the first distance such that the first ZRP is located at a center of the first pixel set and determine a second pixel set of a second pixel grid of the second image sensor that includes an image that is aligned with an image from the first pixel set of the first image sensor. The example processor is further configured to locate a position of a second ZRP by causing the second lens to move along the z-direction during a recording of the second image stream, locating a second portion of area that does not move in the x-direction or the y-direction within the images of the second image stream, and determining a second distance between a center of the second pixel set and the second portion of the area as the position of the second ZRP. Moreover, the example processor is configured to adjust one of the second plurality of lenses or the second image sensor in at least one of the x-direction, the y-direction, and a tilt-direction to cause the second ZRP to be aligned with the center of the second pixel set based on the determined second distance.

The example processor reduces or eliminates spurious parallax by determining a first pixel set of a first pixel grid of the first image sensor using the first distance such that the first ZRP is located at a center of the first pixel set. In addition, the processor determines a second pixel set of a second pixel grid of the second image sensor that includes an image that is aligned with an image from the first pixel set of the first image sensor. Further, the example processor adjusts one of the second plurality of lenses in at least one of the x-direction and the y-direction and a tilt direction to cause the second ZRP to be aligned with a center of the second pixel set based on the determined second distance. In an alternative embodiment, the example processor may digitally change an optical property of the one of the second plurality of lenses to have the same effect as moving the one of the second plurality of lenses. The processor stores the location of the first and second pixel sets in relation to a magnification level of the first and second zoom lenses as a calibration point. The processor may use the calibration point and select the stored locations of the pixel sets when the stereoscopic imaging apparatus subsequently returns to the same or a similar magnification level.

In another embodiment, a stereoscopic imaging apparatus is configured to reduce spurious parallax between first and second image streams recorded in parallel of a target site. The example apparatus includes first optical elements positioned along a first optical path and including a first plurality of lenses including a first zoom lens configured to be moveable along the first optical path in a z-direction, and a first image sensor to record the first image stream of the target site from light in the first optical path. The example apparatus also includes second optical elements positioned along a second optical path that is parallel to the first optical path, the second optical elements including a second plurality of lenses including a second zoom lens configured to be moveable along the second optical path in the z-direction, and a second image sensor to record the second image stream of the target site from light in the second optical path. The example apparatus further includes a processor configured to locate a position of a first zoom repeat point (“ZRP”) in the first image stream, determine a first pixel set of a first pixel grid of the first image sensor such that the first ZRP is located at a center of the first pixel set, and determine a second pixel set of a second pixel grid of the second image sensor such that an image from the second pixel set is visually aligned with an image from the first pixel set.

The advantages discussed herein may be found in one, or some, and perhaps not all of the embodiments disclosed herein. Additional features and advantages are described herein, and will be apparent from the following Detailed Description and the figures.

DETAILED DESCRIPTION

The present disclosure relates in general to a stereoscopic visualization camera and platform. The stereoscopic visualization camera may be referred to as a digital stereoscopic microscope (“DSM”). The example camera and platform are configured to integrate microscope optical elements and video sensors into a self-contained head unit that is significantly smaller, lighter, and more maneuverable than prior art microscopes (such as the surgical loupes100ofFIG. 1and the surgical microscope200ofFIG. 2). The example camera is configured to transmit a stereoscopic video signal to one or more television monitors, projectors, holographic devices, smartglasses, virtual reality devices, or other visual display devices within a surgical environment.

The monitors or other visual display devices may be positioned within the surgical environment to be easily within a surgeon's line of sight while performing surgery on a patient. This flexibility enables the surgeon to place display monitors based on personal preferences or habits. In addition, the flexibility and slim profile of the stereoscopic visualization camera disclosed herein reduces area consumed over a patient. Altogether, the stereoscopic visualization camera and monitors (e.g., the stereoscopic visualization platform) enables a surgeon and surgical team to perform complex microsurgical surgical procedures on a patient without being dictated or restricted in movement compared to the surgical microscope200discussed above. The example stereoscopic visualization platform accordingly operates as an extension of the surgeon's eyes, enabling the surgeon to perform masterpiece microsurgeries without dealing with the stress, restrictions, and limitations induced by previous known visualization systems.

The disclosure herein generally refers to microsurgery. The example stereoscopic visualization camera may be used in virtually any microsurgical procedure including, for example, cranial surgery, brain surgery, neurosurgery, spinal surgery, ophthalmologic surgery, corneal transplants, orthopedic surgery, ear, nose and throat surgery, dental surgery, plastics and reconstructive surgery, or general surgery.

The disclosure also refers herein to target site, scene, or field-of-view. As used herein, target site or field-of-view includes an object (or portion of an object) that is being recorded or otherwise imaged by the example stereoscopic visualization camera. Generally the target site, scene, or field-of-view is a working distance away from a main objective assembly of the example stereoscopic visualization camera and is aligned with the example stereoscopic visualization camera. The target site may include a patient's biological tissue, bone, muscle, skin or combinations thereof. In these instances, the target site may be three dimensional by having a depth component corresponding to a progression of a patient's anatomy. The target site may also include one or more templates used for calibration or verification of the example stereoscopic visualization camera. The templates may be two-dimensional, such as a graphic design on paper (or plastic sheet) or three dimensional, such as to approximate a patient's anatomy in a certain region.

Reference is also made throughout to an x-direction, a y-direction, a z-direction, and a tilt-direction. The z-direction is along an axis from the example stereoscopic visualization camera to the target site and generally refers to depth. The x-direction and y-direction are in a plane incident to the z-direction and comprise a plane of the target site. The x-direction is along an axis that is 90° from an axis of the y-direction. Movement along the x-direction and/or the y-direction refer to in-plane movement and may refer to movement of the example stereoscopic visualization camera, movement of optical elements within the example stereoscopic visualization camera, and/or movement of the target site.

The tilt-direction corresponds movement along Euler angles (e.g., a yaw axis, a pitch axis, and a roll axis) with respect to the x-direction, the y-direction, and/or the z-direction. For example, a perfectly aligned lens has substantially a 0° tilt with respect to the x-direction, the y-direction, and/or the z-direction. In other words, a face of the lens is 90° or perpendicular to light along the z-direction. In addition, edges of the lens (if the lens has a rectangular shape) are parallel along the x-direction and the y-direction. Lens and/or optical image sensors can be titled through yaw movement, pitch movement, and/or roll movement. For example, a lens and/or optical image sensor may be titled along a pitch axis, with respect to the z-direction, to face upwards or downwards. Light along the z-direction contacts a face of a lens (that is pitched upwards or downwards) at non-perpendicular angle. Tilting of a lens and/or optical image sensor along a yaw axis, pitch axis, or roll axis enables, for example, a focal point or ZRP to be adjusted.

I. Example Stereoscopic Visualization Camera

FIGS. 3 and 4show diagrams of perspective views of a stereoscopic visualization camera300, according to an example embodiment of the present disclosure. The example camera300includes a housing302configured to enclose optical elements, lens motors (e.g., actuators), and signal processing circuity. The camera300has a width (along an x-axis) between 15 to 28 centimeters (cm), preferably around 22 cm. In addition, the camera300has a length (along a y-axis) between 15 to 32 cm, preferably around 25 cm. Further, the camera300has a height (along a z-axis) between 10 to 20 cm, preferably around 15 cm. The weight of the camera300is between 3 to 7 kg, preferably around 3.5 kg.

The camera300also includes control arms304aand304b(e.g., operating handles), which are configured to control magnification level, focus, and other microscope features. The control arms304aand304bmay include respective controls305aand305bfor activating or selecting certain features. For example, the control arms304aand304bmay include controls305aand305bfor selecting a fluorescence mode, adjusting an amount/type of light projected onto a target site, and controlling a display output signal (e.g., selection between 1080p or 4K and/or stereoscopic). In addition, the controls305aand/or305bmay be used to initiate and/or perform a calibration procedure and/or move a robotic arm connected to the stereoscopic visualization camera300. In some instances, the controls305aand305bmay include the same buttons and/or features. In other instances the controls305aand305bmay include different features. Further, the control arms304aand304bmay also be configured as grips to enable an operator to position the stereoscopic visualization camera300.

Each control arm304is connected to the housing302via a rotatable post306, as shown inFIG. 3. This connection enables the control arms304to be rotated with respect to the housing302. This rotation provides flexibility to a surgeon to arrange the control arms304as desired, further enhancing the adaptability of the stereoscopic visualization camera300to be in synchronization with a surgical performance.

While the example camera300shown inFIGS. 3 and 4includes two control arms304aand304b, it should be appreciated that the camera300may only include one control arm or zero control arms. In instances where the stereoscopic visualization camera300does not include a control arm, controls may be integrated with the housing302and/or provided via a remote control.

FIG. 4shows a bottom-up perspective view of a rear-side of the stereoscopic visualization camera300, according to an example embodiment of the present disclosure. The stereoscopic visualization camera300includes a mounting bracket402configured to connect to a support. As described in more detail inFIGS. 5 and 6, the support may include an arm with one or more joints to provide significant maneuverability. The arm may be connected to a moveable cart or secured to a wall or ceiling.

The stereoscopic visualization camera300also includes a power port404configured to receive a power adapter. Power may be received from an AC outlet and/or a battery on a cart. In some instances, the stereoscopic visualization camera300may include an internal battery to facilitate operation without cords. In these instances, the power port404may be used to charge the battery. In alternative embodiments, the power port404may be integrated with the mounting bracket402such that the stereoscopic visualization camera300receives power via wires (or other conductive routing materials) within the support.

FIG. 4also shows that the stereoscopic visualization camera300may include a data port406. The example data port406may include any type of port including, for example, an Ethernet interface, a high-definition multimedia interface (“HDMI”) interface, a universal serial bus (“USB”) interface, a Serial Digital Interface (“SDI”), a digital optical interface, an RS-232 serial communication interface etc. The data port406is configured to provide a communicative connection between the stereoscopic visualization camera300and cords routed to one or more computing devices, servers, recording devices, and/or display devices. The communicative connection may transmit stereoscopic video signals or two-dimensional video signals for further processing, storage, and/or display. The data port406may also enable control signals to be sent to the stereoscopic visualization camera300. For instance, an operator at a connected computer (e.g., a laptop computer, desktop computer, and/or tablet computer) may transmit control signals to the stereoscopic visualization camera300to direct operation, perform calibration, or change an output display setting.

In some embodiments, the data port406may be replaced (and/or supplemented) with a wireless interface. For example, the stereoscopic visualization camera300may transmit stereoscopic display signals via Wi-Fi to one or more display devices. A use of a wireless interface, combined with an internal battery, enables the stereoscopic visualization camera300to be wire-free, thereby further improving maneuverability within a surgical environment.

The stereoscopic visualization camera300shown inFIG. 4also includes a front working distance main objective lens408of a main objective assembly. The example lens408is the start of the optical path within the stereoscopic visualization camera300. Light from a light source internal to the stereoscopic visualization camera300is transmitted through the lens408to a target site. Additionally, light reflected from the target site is received in the lens408and passed to downstream optical elements.

II. Exemplary Maneuverability of the Stereoscopic Visualization Camera

FIGS. 5 and 6show diagrams of the stereoscopic visualization camera300used within a microsurgical environment500, according to example embodiments of the present disclosure. As illustrated, the small footprint and maneuverability of the stereoscopic visualization camera300(especially when used in conjunction with a multiple-degree of freedom arm) enables flexible positioning with respect to a patient502. A portion of the patient502in view of the stereoscopic visualization camera300includes a target site503. A surgeon504can position the stereoscopic visualization camera300in virtually any orientation while leaving more than sufficient surgical space above the patient502(lying in the supine position). The stereoscopic visualization camera300accordingly is minimally intrusive (or not intrusive) to enable the surgeon504to perform a life-altering microsurgical procedure without distraction or hindrance.

InFIG. 5, the stereoscopic visualization camera300is connected to a mechanical arm506via mounting bracket402. The arm506may include one or more rotational or extendable joints with electromechanical brakes to facilitate easy repositioning of the stereoscopic visualization camera300. To move the stereoscopic visualization camera300, the surgeon504, or the assistant508, actuates brake releases on one or more joints of the arm506. After the stereoscopic visualization camera300is moved into a desired position, the brakes may be engaged to lock the joints of the arm506in place.

A significant feature of the stereoscopic visualization camera300is that it does not include oculars. This means that the stereoscopic visualization camera300does not have to be aligned with the eyes of the surgeon504. This freedom enables the stereoscopic visualization camera300to be positioned and orientated in desirable positions that were not practical or possible with prior known surgical microscopes. In other words, the surgeon504can perform microsurgery with the most optimal view for conducting the procedure rather than being restricted to merely adequate view dictated by oculars of a surgical microscope.

Returning toFIG. 5, the stereoscopic visualization camera300, via the mechanical arm506, is connected to a cart510with display monitors512and514(collectively a stereoscopic visualization platform516). In the illustrated configuration, the stereoscopic visualization platform516is self-contained and may be moved to any desired location in the microsurgical environment500including between surgical rooms. The integrated platform516enables the stereoscopic visualization camera300to be moved and used on-demand without time needed to configure the system by connecting the display monitors512and514.

The display monitors512and514may include any type of display including a high-definition television, an ultra-high definition television, smart-eyewear, projectors, one or more computer screens, laptop computers, tablet computers, and/or smartphones. The display monitors512and514may be connected to mechanical arms to enable flexible positioning similar to the stereoscopic visualization camera300. In some instances, the display monitors512and514may include a touchscreen to enable an operator to send commands to the stereoscopic visualization camera300and/or adjust a setting of a display.

In some embodiments, the cart516may include a computer520. In these embodiments, the computer520may control a robotic mechanical arm connected to the stereoscopic visualization camera300. Additionally or alternatively, the computer520may process video (or stereoscopic video) signals (e.g., an image or frame stream) from the stereoscopic visualization camera300for display on the display monitors512and514. For example, the computer520may combine or interleave left and right video signals from the stereoscopic visualization camera300to create a stereoscopic signal for displaying a stereoscopic image of a target site. The computer520may also be used to store video and/or stereoscopic video signals into a video file (stored to a memory) so the surgical performance can be documented and played back. Further, the computer520may also send control signals to the stereoscopic visualization camera300to select settings and/or perform calibration.

In some embodiments, the microsurgical environment500ofFIG. 5includes an ophthalmic surgery procedure. In this embodiment, the mechanical arm506may be programmed to perform an orbiting sweep of a patient's eye. Such a sweep enables the surgeon to examine a peripheral retina during vitreo-retinal procedures. In contrast, with conventional optical microscopes, the only way a surgeon can view the peripheral retina is to push the side of the eye into the field of view using a technique known as scleral depression.

FIG. 6shows a diagram of the microsurgical environment500with the patient502in a sitting position for a posterior-approach skull base neurosurgery. In the illustrated embodiment, the stereoscopic visualization camera300is placed into a horizontal position to face the back of the head of the patient502. The mechanical arm506includes joints that enable the stereoscopic visualization camera300to be positioned as shown. In addition, the cart510includes the monitor512, which may be aligned with the surgeon's natural view direction.

The absence of oculars enables the stereoscopic visualization camera300to be positioned horizontally and lower than the eye-level view of the surgeon504. Further, the relatively low weight and flexibility enables the stereoscopic visualization camera300to be positioned in ways unimaginable for other known surgical microscopes. The stereoscopic visualization camera300thereby provides a microsurgical view for any desired position and/or orientation of the patient502and/or the surgeon504.

WhileFIGS. 5 and 6show two example embodiments for positioning the stereoscopic visualization camera300, it should be appreciated that the stereoscopic visualization camera300may be positioned in any number of positions depending on the number of degrees of freedom of the mechanical arm506. It is entirely possible in some embodiments to position the stereoscopic visualization camera300to face upwards (e.g., upside down).

III. Comparison of the Example Stereoscopic Visualization Platform to Known Surgical Microscopes

In comparing the stereoscopic visualization camera300ofFIGS. 3 to 6to the surgical microscope200ofFIG. 2, the differences are readily apparent. The inclusion of oculars206with the surgical microscope requires that the surgeon constantly orient his/her eyes to eyepieces, which are in a fixed location relative to the scope head201and patient. Further, the bulkiness and weight of the surgical microscope restricts it to being positioned only in a generally vertical orientation with respect to a patient. In contrast, the example stereoscopic visualization camera300does not include oculars and may be positioned in any orientation or position with respect to a patient, thereby freeing the surgeon to move during surgery.

To enable other clinician staff to view a microsurgical target site, the surgical microscope200requires the addition of second oculars208. Generally, most known surgical microscopes200do not allow adding third oculars. In contrast, the example stereoscopic visualization camera300may be communicatively coupled to an unlimited number of display monitors. WhileFIGS. 5 and 6above showed display monitors512and514connected to cart510, a surgical room may be surrounded in display monitors that all show the microsurgical view recorded by the stereoscopic visualization camera300. Thus, instead of limiting a view to one or two people (or requiring sharing an ocular), an entire surgical team can view a magnified view of a target surgical site. Moreover, people in other rooms, such as training and observation rooms, can be presented with the same magnified view displayed to the surgeon.

Compared to the stereoscopic visualization camera300, the two-ocular surgical microscope200is more prone to being bumped or inadvertently moved. Since surgeons place their heads on oculars206and208during surgery to look through eyepieces, the scope head201receives constant force and periodic bumps. Adding the second oculars208doubles the force from a second angle. Altogether, the constant force and periodic bumping by the surgeons may cause the scope head201to move, thereby requiring the scope head201to be repositioned. This repositioning delays the surgical procedure and annoys the surgeon.

The example stereoscopic visualization camera300does not include oculars and is not intended to receive contact from a surgeon once it is locked into place. This corresponds to a significantly lower chance of the stereoscopic visualization camera300being accidently moved or bumped during the surgeon's performance.

To facilitate the second oculars208, the surgical microscope200has to be outfitted with a beamsplitter210, which may include glass lenses and mirrors housed in precision metallic tubes. The use of a beamsplitter210reduces light received at the first oculars because some of the light is reflected to the second oculars208. Further, addition of the second oculars208and the beamsplitter210increases the weight and bulkiness of the scope head201.

In contrast to the surgical microscope200, the stereoscopic visualization camera300only contains optical paths for sensors, thereby reducing weight and bulkiness. In addition, the optical sensors receive the full incident light since beamsplitters are not needed to redirect a portion of the light. This means the image received by optical sensors of the example stereoscopic visualization camera300is as bright and clear as possible.

Some models of surgical microscopes may enable a video camera to be attached. For instance, the surgical microscope200ofFIG. 2incudes a monoscopic video camera212connected to an optical path via beamsplitter214. The video camera212may be monoscopic or stereoscopic, such as the Leica® TrueVision® 3D Visualization System Ophthalmology camera. The video camera212records an image received from the beamsplitter214for display on a display monitor. The addition of the video camera212and beamsplitter214further add to the weight of the scope head201. In addition, the beamsplitter214consumes additional light destined for the oculars206and/or208.

Each beamsplitter210and214divides the incident light fractionally into three paths, removing light from the surgeon's view. The surgeon's eye has limited low-light sensitivity such that light from the operative site presented to him/her must be sufficient to allow the surgeon to perform the procedure. However, a surgeon cannot always increase the intensity of light applied to a target site on a patient, especially in ophthalmological procedures. A patient's eye has limited high-light sensitivity before it develops light toxicity. Hence, there is a limitation to the number and fraction of beamsplitters and to the amount of light which can be split off from the first oculars206to enable the use of ancillary devices208and212.

The example stereoscopic visualization camera300ofFIGS. 3 to 6does not include beamsplitters such that optical imaging sensors receive the full amount of light from a main objective assembly. This enables the use of sensors with low-light sensitivity or even optical sensors with sensitivity outside the wavelengths of visible light to be used since post-processing can make the images sufficiently bright and visible (and adjustable) for display on the monitors.

Further, since the optical elements that define the optical paths are self-contained within the stereoscopic visualization camera300, the optical elements may be controlled through the camera. This control allows placement and adjustment of the optical elements to be optimized for a three-dimensional stereoscopic display rather than for microscope oculars. This configuration of the camera permits control to be provided electronically from camera controls or from a remote computer. In addition, the control may be provided automatically through one or more programs onboard the camera300configured to adjust optical elements for retaining focus while zooming or to adjust for optical defects and/or spurious parallax. In contrast, optical elements of the surgical microscope200are external to the video camera212and controlled only via operator input, which is generally optimized for viewing a target site through the oculars206.

In a final comparison, the surgical microscope200includes an X-Y panning device220for moving a field-of-view or target scene. The X-Y panning device220is typically a large, heavy, and expensive electromechanical module since it must rigidly support and move the surgical scope head201. In addition, moving the scope head201changes the positioning of the surgeon to the new location of the oculars206.

In contrast, the example stereoscopic visualization camera300includes a memory including instructions, which when executed, cause a processor to select pixel data of optical sensors to enable X-Y panning across a wide pixel grid. In addition, the example stereoscopic visualization camera300may include a small motor or actuator that controls a main objective optical element to change a working distance to a target site without moving the camera300.

IV. Example Optical Elements of the Stereoscopic Visualization Camera

FIGS. 7 and 8show diagrams illustrative of optical elements within the example stereoscopic visualization camera300ofFIGS. 3 to 6, according to an example embodiment of the present disclosure. It may seem relatively simple to acquire left and right views of a target site to construct a stereoscopic image. However, without careful design and compensation, many stereoscopic images have alignment issues between the left and right views. When viewed for a prolonged period of time, alignment issues can create confusion in an observer's brain as a result of differences between the left and right views. This confusion can lead to headaches, fatigue, vertigo, and even nausea.

The example stereoscopic visualization camera300reduces (or eliminates) alignment issues by having a right optical path and left optical path with independent control and/or adjustment of some optical elements while other left and right optical elements are fixed in a common carrier. In an example embodiment, some left and right zoom lenses may be fixed to a common carrier to ensure left and right magnification is substantially the same. However, front or rear lenses may be independently adjustable radially, rotationally, axially, and/or tilted to compensate for small differences in zoom magnification, visual defects, and/or spurious parallax such as movement of a zoom repeat point. Compensation provided by adjustable lenses results in almost perfectly aligned optical paths throughout a complete zoom magnification range.

Additionally or alternatively, alignment issues may be reduced (or eliminated) using pixel readout and/or rendering techniques. For example, a right image (recorded by a right optical sensor) may be adjusted upwards or downwards with respect to a left image (recorded by a left optical sensor) to correct vertical misalignment between the images. Similarly, a right image may be adjusted left or right with respect to a left image to correct horizontal misalignment between the images.

FIGS. 7 and 8below show an example arrangement and positioning of optical elements that provide for almost artifact, spurious parallax, and distortion-free aligned optical paths. As discussed later, certain of the optical elements may be moved during calibration and/or use to further align the optical paths and remove any remaining distortions, spurious parallax, and/or defects. In the illustrated embodiment, the optical elements are positioned in two parallel paths to generate a left view and a right view. Alternative embodiments may include optical paths that are folded, deflected or otherwise not parallel.

The illustrated paths correspond to a human's visual system such that the left view and right view, as displayed on a stereoscopic display, appear to be separated by a distance that creates a convergence angle of roughly 6 degrees, which is comparable to the convergence angle for an adult human's eyes viewing an object at approximately 4 feet away, thereby resulting in stereopsis. In some embodiments, image data generated from the left view and right view are combined together on the display monitor(s)512and514to generate a stereoscopic image of a target site or scene. Alternative embodiments comprise other stereoscopic displays where the left view is presented to only the left eye of a viewer and the corresponding right view is presented to only the right eye. In exemplary embodiments used to adjust and verify proper alignment and calibration, both views are displayed overlaid to both eyes.

A stereoscopic view is superior to a monoscopic view because it mimics the human visual system much more closely. A stereoscopic view provides depth perception, distance perception, and relative size perception to provide a realistic view of a target surgical site to a surgeon. For procedures such as retinal surgery, stereoscopic views are vital because surgical movements and forces are so small that the surgeon cannot feel them. Providing a stereoscopic view helps a surgeon's brain magnify tactile feel when the brain senses even minor movements while perceiving depth.

FIG. 7shows a side view of the example stereoscopic visualization camera300with the housing302being transparent to expose the optical elements.FIG. 8shows a diagram illustrative of an optical path provided by the optical elements shown inFIG. 7. As shown inFIG. 8, the optical path includes a right optical path and a left optical path. The optical paths inFIG. 8are shown from a perspective of facing a forward direction and looking down at the stereoscopic visualization camera300. From this view, the left optical path appear on the right side ofFIG. 8while the right optical path is shown on the left side.

The optical elements shown inFIG. 7are part of the left optical path. It should be appreciated that the right optical path inFIG. 7is generally identical to the left optical path regarding relation location and arrangement of optical elements. As mentioned above, the interpupillary distance between a center of the optical paths is between 58 to 70 mm, which may be scaled to 10 to 25 mm. Each of the optical elements comprise lenses having certain diameters (e.g., between 2 mm and 29 mm). Accordingly, a distance between the optical elements themselves is between 1 to 23 mm, preferably around 10 mm.

The example stereoscopic visualization camera300is configured to acquire images of a target site700(also referred to as a scene or field-of-view). The target site700includes an anatomical location on a patient. The target site700may also include laboratory biological samples, calibration slides/templates, etc. Images from the target site700are received at the stereoscopic visualization camera300via a main objective assembly702, which includes the front working distance lens408(shown inFIG. 4) and a rear working distance lens704.

A. Example Main Objective Assembly

The example main objective assembly702may include any type of refractive assembly or reflective assembly.FIG. 7shows the objective assembly702as an achromatic refractive assembly with the front working distance lens408being stationary and the rear working distance lens704being movable along the z-axis. The front working distance lens408may comprise a plano convex (“PCX”) lens and/or a meniscus lens. The rear working distance lens704may comprise an achromatic lens. In examples where the main objective assembly702includes an achromatic refractive assembly, the front working distance lens408may include a hemispherical lens and/or a meniscus lens. In addition, the rear working distance lens704may include an achromatic doublet lens, an achromatic doublet group of lenses, and/or an achromatic triplet lens.

The magnification of the main objective assembly702is between 6× to 20×. In some instances, the magnification of the main objective assembly702may vary slightly based on a working distance. For example, the main objective assembly702may have a magnification of 8.9× for a 200 mm working distance and a magnification of 8.75× for a 450 mm working distance.

The example rear working distance lens704is configured to be moveable with respect to the front working distance lens408to change a spacing therebetween. The spacing between the lenses408and704determines the overall front focal length of the main objective assembly702, and accordingly the location of a focal plane. In some embodiments, the focal length is the distance between the lenses408and704plus one-half the thickness of the front working distance lens408.

Together, the front working distance lens408and the rear working distance lens704are configured to provide an infinite conjugate image for providing an optimal focus for downstream optical image sensors. In other words, an object located exactly at the focal plane of the target site700will have its image projected at a distance of infinity, thereby being infinity-coupled at a provided working distance. Generally, the object appears in focus for a certain distance along the optical path from the focal plane. However, past the certain threshold distance, the object begins to appear fuzzy or out of focus.

FIG. 7shows working distance706, which is the distance between an outer surface of the front working distance lens408and to the focal plane of the target site700. The working distance706may correspond to an angular field-of-view, where a longer working distance results in a wider field-of-view or larger viewable area. The working distance706accordingly sets a plane of the target site or scene that is in focus. In the illustrated example, the working distance706is adjustable from 200 to 450 mm by moving the rear working distance lens704. In an example, the field-of-view can be adjusted between 20 mm×14 mm to 200 mm×140 mm using upstream zooming lenses when the working distance is 450 mm.

The main objective assembly702shown inFIGS. 7 and 8provides an image of the target site700for both the left and right optical paths. This means that the width of the lenses408and704should be at least as wide as the left and right optical paths. In alternative embodiments, the main objective assembly702may include separate left and right front working distance lenses408and separate left and right rear working distance lens704. The width of each pair of the separate working distance lenses may be between ¼ to ½ of the width of the lenses408and704shown inFIGS. 7 and 8. Further, each of the rear working distance lenses704may be independently adjustable.

In some embodiments, the main objective assembly702may be replaceable. For example, different main objective assemblies may be added to change a working distance range, a magnification, a numerical aperture, and/or refraction/reflection type. In these embodiments, the stereoscopic visualization camera300may change positioning of downstream optical elements, properties of optical image sensors, and/or parameters of image processing based on which main objective assembly is installed. An operator may specify which main objective assembly is installed in the stereoscopic visualization camera300using one of the controls305ofFIG. 3and/or a user input device.

B. Example Lighting Sources

To illuminate the target site700, the example stereoscopic visualization camera300includes one or more lighting sources.FIGS. 7 and 8show three lighting sources including a visible light source708a, a near-infrared (“NIR”) light source708b, and a near-ultraviolet (“NUV”) light source708c. In other examples, the stereoscopic visualization camera300may include additional or fewer (or no) light sources. For instance, the NIR and NUV light sources may be omitted. The example light sources708are configured to generate light, which is projected to the target scene700. The generated light interacts and reflects off the target scene, with some of the light being reflected to the main objective assembly702. Other examples may include external light sources or ambient light from the environment.

The example visible light source708ais configured to output light in the human-visible part of the light spectrum in addition to some light with wavelengths outside the visible region. The NIR light source708bis configured to output light that is primarily at wavelengths slightly past the red part of the visible spectrum, which is also referred to as “near-infrared.” The NUV light source708cis configured to output light that is primarily at wavelengths in the blue part of the visible spectrum, which is referred to as “near-ultraviolet.” The light spectra output by the light sources708is controlled by respective controllers, described below. A brightness of light emitted by the light sources708may be controlled by a switching rate and/or applied voltage waveform.

FIGS. 7 and 8illustrate that the visible light source708aand the NIR light source708bare provided directly through the main objective assembly702to the target site700. As shown inFIG. 8, visible light from the visible light source708apropagates along visible path710a. Additionally, NIR light from the NIR light source708bpropagates along NIR path710b. While the light sources708aand708bare shown as being behind the main objective assembly702(with respect to the target site700), in other examples the light sources708aand708bmay be provided before the main objective assembly702. In one embodiment, the light sources708aand708bmay be provided on an outside of the housing302and face toward the target site700. In yet other embodiments, the light sources708may be provided separate from the stereoscopic visualization camera300using, for example, a Koeher illumination setup and/or a darkfield illumination setup.

In contrast to the light sources708aand708b, NUV light from the NUV light source708cis reflected by a deflecting element712(e.g., a beamsplitter) to the main objective assembly702using an epi-illumination setup. The deflecting element712may be coated or otherwise configured to reflect only light beyond the NUV wavelength range, thereby filtering NUV light. NUV light from the NUV light source708cpropagates along NUV path710c.

In some embodiments, the NIR and NUV light sources708band708cmay be used with excitation filters to further filter light that may not be blocked by filters (e.g., filter740). The filters may be placed in front of the light sources708band708cbefore the main objective assembly702and/or after the main objective assembly. The light from the NUV and NIR light sources708band708c, after being filtered, comprises wavelengths that excite fluorescence in fluorescent sites914(shown inFIG. 9) of an anatomical object. Further, the light from the NUV and NIR light sources708band708c, after being filtered, may comprise wavelengths that are not in the same range as those being emitted by the fluorescent sites914.

The projection of the light from light sources708through the main objective assembly provides the benefit of changing the lighted field-of-view based on the working distance706and/or focal plane. Since the light passes through the main objective assembly702, the angle at which light is projected changes based on the working distance706and corresponds to the angular field-of-view. This configuration accordingly ensures the field-of-view is properly illuminated by the light sources708, regardless of working distance or magnification.

C. Example Deflecting Element

The example deflecting element712illustrated inFIGS. 7 and 8is configured to transmit a certain wavelength of light from the NUV light source708cto the target site700through the main objective assembly702. The deflecting element712is also configured to reflect light received from the target site700to downstream optical elements, including a front lens set714for zooming and recording. In some embodiments, the deflecting element712may filter light received from the target site700through the main objective assembly702so that light of certain wavelengths reaches the front lens set714.

The deflecting element712may include any type of mirror or lens to reflect light in a specified direction. In an example, the deflecting element712includes a dichroic mirror or filter, which has different reflection and transmission characteristics at different wavelengths. The stereoscopic visualization camera300ofFIGS. 7 and 8includes a single deflecting element712, which provides light for both the right and left optical paths. In other examples, the camera300may include separate deflecting elements for each of the right and left optical paths. Further, a separate deflecting element may be provided for the NUV light source708c.

FIG. 9shows a diagram of the deflecting element712ofFIGS. 7 and 8, according to an example embodiment of the present disclosure. For brevity, the main objective assembly702is not shown. In this example, the deflecting element712includes two parallel faces902and904for transmitting and reflecting light of certain wavelengths. The parallel faces902and904are set at a 45° angle with respect to the left and right optical paths (represented as path906). The 45° angle is selected since this angle causes reflected light to propagate at a 90° angle from the transmitted light, thereby providing optimal separation without causing the separated light to be detected in the downstream front lens set714. In other embodiments, the angle of the deflecting element712could be between 10 degrees and 80 degrees without unintentionally propagating light of unwanted wavelengths.

The example NUV light source708cis located behind the deflecting element712(with respect to the target site700). Light from the light source708cpropagates along path908and contacts the deflecting element712. NUV light around the primary wavelength range of the NUV light source708cis transmitted through the deflecting element712along path910to the target site700. Light from the NUV light source708cthat has a wavelength above (and below) the primary wavelength range of the NUV light source708cis reflected along path912to a light sink or unused region of the housing302.

When the NUV light reaches the target site700, it is absorbed by one or more fluorescent sites914of an anatomical object. In some instances, the anatomical object may have been injected with a contrast agent configured to absorb NUV light and emit light with a different primary wavelength. In other instances, the anatomical object may naturally absorb NUV light and emit light with a different primary wavelength. At least some of the light reflected or emitted by the fluorescent site914propagates along path916until it contacts the deflecting element712. Most of the light reflects off the surface904along path906to the front lens set714. A portion of the light, including NUV light around the primary wavelength range of the NUV light source708cis transmitted through the deflecting element712along path918to a light sink or unused region of the housing302. The deflecting element712shown inFIG. 9accordingly enables optical stimulation of a fluorescent agent at the target site700with one region of the spectrum while blocking much of the stimulating light from travelling to the downstream front lens set714.

It should be appreciated that the reflectivity and transmissivity characteristics of the deflecting element712can be changed to meet other light spectrum requirements. In some instances, the housing302may include a slot that enables the deflecting element712and/or the NUV light source708cto be replaced based on the desired light reflectivity and transmissivity characteristics. It should also be appreciated that a first path internal to the deflecting element712between path908and path910and a second path internal to the deflecting element712between path916and path918are each angled to represent schematically the refraction of the light as it travels between air and the interior of the deflecting element712. The angles shown are not meant to represent actual reflection angles.

D. Example Zoom Lenses

The example stereoscopic visualization camera300ofFIGS. 7 and 8includes one or more zoom lens to change a focal length and angle of view of the target site700to provide zoom magnification. In the illustrated example, the zoom lens includes the front lens set714, a zoom lens assembly716, and a lens barrel set718. It should be appreciated that in other embodiments, the front lens set714and/or the lens barrel set718may be omitted. Alternatively, the zoom lens may include additional lens to provide further magnification and/or image resolution.

The front lens set714includes a right front lens720for the right optical path and a left front lens722for the left optical path. The lenses720and722may each include a positive converging lens to direct light from the deflecting element712to respective lenses in the zoom lens assembly716. A lateral position of the lenses720and722accordingly defines a beam from the main objective assembly702and the deflecting element712that is propagated to the zoom lens assembly716.

One or both of the lenses720and722may be adjustable radially to match optical axes of the left and right optical paths. In other words, one or both of the lenses720and722may be moved left-right and/or up-down in a plane incident to the optical path. In some embodiments, one or more of the lenses720and722may be rotated or tilted to reduce or eliminate image optical defects and/or spurious parallax. Moving either or both of the lenses720and722during zooming may cause the zoom repeat point (“ZRP”) for each optical path to appear to remain stationary to a user. In addition to radial movement, one or both of the front lenses720and722may be moved axially (along the respective optical path) to match magnifications of the optical paths.

The example zoom lens assembly716forms an afocal zoom system for changing the size of a field-of-view (e.g., a linear field-of-view) by changing a size of the light beam propagated to the lens barrel set718. The zoom lens assembly716includes a front zoom lens set724with a right front zoom lens726and a left front zoom lens728. The zoom lens assembly716also includes a rear zoom lens set730with a right rear zoom lens732and a left rear zoom lens734. The front zoom lenses726and728may be positive converging lenses while the rear zoom lenses732and734include negative diverging lenses.

The size of an image beam for each of the left and right optical paths is determined based on a distance between the front zoom lenses726and728, the rear zoom lenses732and734and the lens barrel set718. Generally, the size of the optical paths reduces as the rear zoom lenses732and734move toward the lens barrel set718(along the respective optical paths), thereby decreasing magnification. In addition, the front zoom lenses726and728may also move toward (or away from) the lens barrel set718(such as in a parabolic arc), as the rear zoom lenses732and734move toward the lens barrel set718, to maintain the location of the focal plane on the target site700, thereby maintaining focus.

The front zoom lenses726and728may be included within a first carrier (e.g., the front zoom set724) while the rear zoom lenses732and724are included within a second carrier (e.g., the rear zoom set730). Each of the carriers724and730may be moved on tracks (or rails) along the optical paths such that left and right magnification changes concurrently. In this embodiment, any slight differences in magnification between the left and right optical paths may be corrected by moving the right front lens720and/or the left front lens722. Additionally or alternatively, a right lens barrel736and/or a left lens barrel738of the lens barrel set718may be moved axially.

In alternative embodiments, the right front zoom lens726may be moved axially separately from the left front zoom lens728. In addition, the right rear zoom lens732may be moved axially separately from the left rear zoom lens734. Separate movement may enable small magnification differences to be corrected by the zoom lens assembly716, especially when the front lens set714and the lens barrel set718are stationary along the optical paths. Further, in some embodiments, the right front zoom lens726and/or the left front zoom lens728may be radially and/or rotationally adjustable (and/or tilted) to maintain an apparent location of a ZRP in the optical path. Additionally or alternatively, the right rear zoom lens732and/or the left rear zoom lens734may be radially and/or rotationally adjustable (and/or tilted) to maintain an apparent location of a ZRP in the optical path.

The example lens barrel set718includes the right lens barrel736and the left lens barrel738, which are part of the afocal zoom system in addition with the zoom lens assembly716. The lenses736and738may include positive converging lenses configured to straighten or focus a light beam from the zoom lens assembly716. In other words, the lenses736and738focus the infinity-coupled output of the zoom lens assembly716.

In some examples, the lens barrel set718is fixed radially and axially within the housing302. In other examples, the lens barrel set718may be moveable axially along the optical path to provide increased magnification. Additionally or alternatively, each of the lenses736and738may be radially and/or rotationally adjustable (and/or tilted) to, for example, correct for differences in optical properties (from manufacturing or natural glass deviations) between the left and right lenses of the front lens set714, the front zoom lens set724, and/or the rear zoom lens set730.

Altogether, the example front lens set714, the zoom lens assembly716, and the lens barrel set718are configured to achieve an optical zoom between 5× to about 20×, preferably at a zoom level that has diffraction-limited resolution. In some embodiments, the front lens set714, the zoom lens assembly716, and the lens barrel set718may provide higher zoom ranges (e.g., 25× to 100×) if image quality can be compromised. In these embodiments, the stereoscopic visualization camera300may output a message to an operator indicative that a selected optical range is outside of an optical range and subject to a reduction in image quality.

In some embodiments, the lenses of the front lens set714, the zoom lens assembly716, the lens barrel set718, and/or the main objective assembly702may each be constructed as a doublet from multiple optical sub-elements using materials that balance each other's optical distortion parameters. The doublet construction reduces chromatic aberrations and optical aberrations. For example, the front working distance lens408and the rear working distance lens702may each be constructed as a doublet. In another example, the front lenses720and722, the front zoom lenses726and728, the rear zoom lenses732and734, and the lens barrels736and738may each comprise a doublet lens.

In yet further embodiments, the lenses of the front lens set714, the zoom lens assembly716, the lens barrel set718, and/or the main objective assembly702may be tuned differently and/or have different properties to provide two parallel optical paths with different capabilities. For example, right lenses in zoom lens assembly716may be selected to provide 5× to 10× optical zoom for the right optical path while left lenses in the zoom lens assembly716are selected to provide 15× to 20× optical zoom for the left optical path. Such a configuration may enable two different magnifications to be shown at the same time and/or on the same screen, though in a monoscopic view.

E. Example Filter

The example stereoscopic visualization camera300ofFIGS. 7 and 8includes one or more optical filters740(or filter assemblies) to selectively transmit desired wavelengths of light.FIG. 8shows that a single filter740may be applied to the right and left optical paths. In other examples, each of the optical paths may have a separate filter. The inclusion of separate filters enables, for example, different wavelengths of light to be filtered from the left and right optical paths at the same time, which enables, for example, fluorescent images to be displayed in conjunction with visible light images.

FIG. 7shows that the filter740includes a wheel that is rotated about its axis of rotation. In the illustrated embodiment, the filter740can accommodate three different optical filter pairs. However, in other embodiments, the filter740may include additional or fewer filter pairs. Generally, light received at the filter740from the target site700includes a broad spectrum of wavelengths. The lenses of the main objective assembly702, the front lens set714, the zoom lens assembly716, and the lens barrel set718are configured to pass a relatively wide bandwidth of light including wavelengths of interest to an operator and undesirable wavelengths. In addition, downstream optical image sensors are sensitive to certain wavelengths. The example filter740accordingly passes and blocks certain portions of the light spectrum to achieve different desirable features.

As a wheel, the filter740comprises a mechanical device capable of changing positions at about four times per second. In other embodiments, the filter740may include a digital micro-mirror, which can change a light path's direction at video frame rates such as 60 times per second. In these other embodiments, each of the left and right optical paths would include a micro-mirror. The left and right micro-mirror may have synchronized or simultaneous switching.

In some embodiments, the filter740may be synchronized to the light sources708to realize “time-interleaved” multispectral imaging. For example, the filter740may include an infrared cut filter, near-infrared bandpass filter, and near-ultraviolet cut filter. The different filter types are selected to work with different spectra of the light sources708and the reflectivity and transmissivity characteristics of the deflecting element712to pass certain desired wavelengths of light at predetermined times.

In one mode, the filter740and the light sources708are configured to provide a visible light mode. In this mode, the visible light source708atransmits light from the visible region onto the target site700, some of which is reflected to the main objective assembly702. The reflected light may include some light beyond the visible spectrum, which may affect optical image sensors. The visible light is reflected by the deflecting element712and passes through the front lens set714, the zoom lens assembly716, and the lens barrel set718. In this example, the filter740is configured to apply the infrared-cut filter or the near-ultraviolet cut filter to the optical paths to remove light outside the visible spectrum such that light only in the visible spectrum passes through to a final optical set742and an optical image sensor744.

In another mode, filter740and the light sources708are configured to provide fluorescence light of a narrow wavelength to the optical sensor744. In this mode, the NUV light source708ctransmits light from the deep-blue region of the spectrum to the target site700. The deflecting element712allows the desired light of the deep-blue region to pass through while reflecting undesired light. The deep-blue light interacts with the target site700such that fluorescence light is emitted. In some examples, δ-Aminolaevulinic acid (“5ala”) and/or Protoporphyrin IX is applied to the target site700to cause fluorescence light to be emitted when deep-blue light is received. The main objective assembly702receives the fluorescence light in addition to reflected deep-blue light and some visible light. The deep-blue light passes through the deflecting element712out of the right and left optical paths. Thus, only the visible light and fluorescence light pass through the front lens set714, the zoom lens assembly716, and the lens barrel set718. In this example, the filter740is configured to apply the near-ultraviolet cut filter to the optical paths to remove light outside the desired fluorescence spectrum including visible light and any remaining NUV deep-blue light. Accordingly, only fluorescence light of a narrow wavelength reaches the optical image sensor744, which enables the fluorescence light to be more easily detected and distinguished based on relative intensity.

In yet another mode, the filter740and the light sources708are configured to provide indocyanine green (“ICG”) fluorescence light to the optical sensor744. In this mode, the NIV light source708btransmits light in the far-red region (which is also considered near-infrared) of the visible spectrum to the target site700. In addition, the visible light source708atransmits visible light to the target scene700. The visible light and far-red light are absorbed by material with ICG at the target site, which then emits a highly stimulated fluorescence light in the further-red region. The main objective assembly702receives the fluorescence light in addition to reflected NIR light and visible light. The light is reflected by the deflecting element712to the front lens set714, the zoom lens assembly716, and the lens barrel set718. In this example, the filter740is configured to apply the near-infrared bandpass filter to the optical paths to remove light outside the desired fluorescence spectrum including visible light and at least some of the NIR light. Accordingly, only fluorescence light in the further-red region reaches the optical image sensor744, which enables the fluorescence light to be more easily detected and distinguished based on relatively intensity.

Table 1 above shows a summary of the different possible combinations of lights sources and filters for causing light of a certain desired wavelength to reach the optical light sensor744. It should be appreciated that other types of filters and/or light sources may be used to further increase the different types of light received at the image sensor744. For instance, bandpass filters configured to pass light of a narrow wavelength may be used to correspond to certain biological stains or contrasts applied to the target site700. In some examples, the filter740may include a cascade or more than one filter to enable light from two different ranges to be filtered. For example, a first filter740may apply an infrared cut filter and a near-ultraviolet cut filter such that only visible light of a desired wavelength range passes to the optical sensor744.

In other embodiments, separate filters740may be used for the left and right optical paths. For example, a right filter may include an infrared cut filter while a left filter includes a near-infrared pass filter. Such a configuration enables viewing of the target site700in visible wavelengths simultaneously with IGC green fluorescence wavelengths. In another example, a right filter may include an infrared cut filter while a left filter includes a near-ultraviolet cut filter. In this configuration, the target site700may be shown in visible light simultaneously with 5ALA fluorescence light. In these other embodiments, the right and left image streams may still be combined into a stereoscopic view that provides a fluorescence view of certain anatomical structures combined with a view of the target site700in visible light.

F. Example Final Optical Element Set

The example stereoscopic visualization camera300ofFIGS. 7 and 8includes the final optical element set742to focus light received from the filter740onto the optical image sensor744. The final optical element set742includes a right final optical element745and a left final optical element747, which may each comprise a positive converging lens. In addition to focusing light, the optical elements745and747may be configured to correct minor aberrations in the right and left optical paths prior to the light reaching the optical image sensor744. In some examples, the lenses745and747may be moveable radially and/or axially to correct magnification and/or focusing aberrations caused by the front lens set714, the zoom lens assembly716, and the lens barrel set718. In an example, the left final optical element747may be moved radially while the right final optical element745is fixed to remove ZRP movement during magnification changes.

G. Example Image Sensors

The example stereoscopic visualization camera300ofFIGS. 7 and 8includes the image sensor744to acquire and/or record incident light that is received from the final optical element set742. The images sensor744includes a right optical image sensor746to acquire and/or record light propagating along the right optical path and a left optical image sensor748to acquire and/or record light propagating along the left optical path. Each of the left and right optical image sensors746and748include, for example, complementary metal-oxide-semiconductor (“CMOS”) sensing elements, N-type metal-oxide-semiconductor (“NMOS”), and/or semiconductor charge-coupled device (“CCD”) sensing elements. In some embodiments, the left and right optical sensors746and748are identical and/or have the same properties. In other embodiments, the left and right optical sensors746and748include different sensing elements and/or properties to provide varying capability. For example, the right optical image sensor746(using a first color filter array) may be configured to be more sensitive to blue fluorescence light while the left optical image sensor748(using a second color filter array) is configured to be more sensitive to visible light.

FIG. 10shows an example of the right optical image sensor746and the left optical image sensor748of the image sensor744, according to an example embodiment of the present disclosure. The right optical image sensor746includes a first two-dimensional grid or matrix1002of light-sensing elements (e.g., pixels). In addition, the left optical image sensor748includes a second two-dimensional pixel grid1004of light-sensing elements. Each of the pixels includes a filter that enables only light of a certain wavelength to pass, thereby contacting an underlying light detector. Filters for different colors are spread across the sensors746and748to provide light detection for all wavelengths across grids. The light detector may be sensitive to visible light, as well as additional ranges that are above and below the visible spectrum.

The light-sensing elements of the grids1002and1004are configured to record a range of wavelengths of light as a representation of the target site700that is in the field-of-view. Light incident on a light-sensing element causes an electrical change to accumulate. The electrical charge is read to determine an amount of light being received at the sensing element. In addition, since the filter characteristics of the sensing element are known to within manufacturing tolerances, the range of wavelengths of the received light is known. The representation of the target site700is directed onto the light-sensing elements such that the grids1002and1004for the respective optical image sensors746and748sample the target site700spatially. The resolution of the spatial sampling is a parameter that affects image quality and parity.

The number of pixels shown in the pixel grids1002and1004inFIG. 10is not representative of the number of actual pixels in the optical image sensors746and748. Instead, the sensors typically have a resolution between 1280×720 pixels and 8500×4500 pixels, preferably around 2048×1560 pixels. However, not all pixels of the grids1002and1004are selected for image transmission. Instead, a subset or pixel set of the grids1002and1004are selected for transmission. For example, inFIG. 10, pixel set1006is selected from the pixel grid1002for transmission as a right image and pixel set1008is selected from pixel grid1004for transmission as a left image. As illustrated, the pixel set1006does not need to be located in the same location as the pixel set1008in relation to respective pixel grids1002and1004. The separate control of the pixel sets1006and1008enables left and right images to be aligned and/or corrected for image defects and/or spurious parallax such as moving ZRPs.

Selection of a pixel set from a pixel grid enables a portion of the pixel grid to be selected to compensate for image defects/spurious parallax and/or to more align the right and left optical images. In other words, the pixel set may be moved or adjusted (in real-time) with respect to the pixel grid to improve image quality by reducing or eliminating spurious parallax. Alternatively, either or both of the left and right views of the stereoscopic image can be moved virtually in the image processing pipeline (for example during rendering of the views for display) to accomplish the same effect. Rotational misalignment of the sensors can also be corrected virtually. A pixel set may also be moved across a pixel grid during use to provide an appearance of panning the field-of-view. In an example, a pixel set or window of 1920×1080 pixels may be selected from a pixel grid having 2048×1560 pixels. The location of the pixel window or set may be controlled by software/firmware and be moved during setup and/or use. The resolution of the optical image sensors746and748is accordingly specified based on a number of pixels in the length and width directions of the pixel set or window.

1. Color Sensing with the Example Image Sensors

As mentioned above, the optical sensing elements746and748include pixels with different filters to detect certain colors of light. For instance, some pixels are covered with filters that pass predominantly red light, some are covered with filters that pass predominantly green light, and some are covered with filters that pass predominantly blue light. In some embodiments, a Bayer pattern is applied to the pixel grids1002and1004. However, it should be appreciated that in other embodiments, a different color pattern may be used that is optimized for certain wavelengths of light. For example, a green filter in each sensing region may be replaced with a broadband filter or a near-infrared filter, thereby extending the sensing spectrum.

The Bayer pattern is implemented by grouping two rows by two columns of pixels and covering one with a red filter, one with a blue filter, and two with a green filter, each in a checkerboard pattern. Thus the resolution of red and blue are each one quarter of the whole sensing region of interest while green resolution is half that of the whole sensing region of interest.

Green may be assigned to half the sensing region to cause the optical image sensors746and748to operate as a luminance sensor and mimic the human visual system. In addition, red and blue mimic chrominance sensors of the human visual system, but are not as critical as green sensing. Once an amount of red, green, and blue are determined for a certain region, other colors in the visible spectrum are determined by averaging the red, green, and blue values, as discussed in conjunction with de-Bayer program1580aofFIG. 16discussed below.

In some embodiments, the optical image sensors746and748may use stacked components to sense color rather than filters. For example, sensing elements may include red, green and blue sensing components stacked vertically inside a pixel's area. In another example, prisms split incident light into components using specially coated beamsplitters one or more times (typically at least two times resulting in three component colors, known as “3-chip”) with sensing elements placed in each of the split beams' paths. Other sensor types use a different pattern such as replacing one of the green filters with a broadband filter or a near-infrared filter, thereby extending the sensing possibilities of the digital surgical microscope.

2. Sensing Light Outside the Visible Range with the Example Image Sensors

The example sensing element filters of the optical image sensors746and748are configured to also pass near-infrared light in a range that the sensing element can detect. This enables the optical image sensors746and748to detect at least some light outside of the visible range. Such sensitivity may decrease image quality in the visible part of the spectrum because it “washes out” the image, reducing contrast in many types of scenes and negatively affecting the color quality. As a result, the filter740may use the infrared cut filter to block near infrared wavelengths while passing the visible wavelengths to the optical image sensors746and748.

However, such near-infrared sensitivity may be desirable. For example, a fluorescent agent, such ICG, can be introduced to the target site700. ICG becomes excited or activated with visible or other wavelengths or light and emits fluorescence light in the near infrared range. As mentioned above, the NIR light source708bprovides NIR light and the visible light source708aprovides visible light to excite agents with ICG. Emitted light is further along the red spectrum, which may be passed through the filter740using a near-infrared bandpass or high-pass filter. The light from the red spectrum then is detected by the optical image sensors746and748. By matching the spectral characteristics of the filter740to the expected behaviors of the light source708and the fluorescent agent, the agent and the biological structures, such as blood that contain the agent, can be differentiated at the target site700from other structures that do not contain the agent.

Note that in this example, the NIR light source708bhas a different primary wavelength from the near-infrared filter in the filter740. Specifically, the NIR light source708bhas a primary wavelength around 780 nanometers (“nm”) (around which the majority of the light's output spectrum exists). In contrast, the near-infrared filter of the filter740transmits light at wavelengths in a range of approximately 810 nm to 910 nm. The light from the NIR light source708band light passed through the filter740are both “near-infrared” wavelengths. However, the light wavelengths are separated so that the example stereoscopic visualization camera300can stimulate with the light source708and detect with the optical image sensor744while filtering the stimulation light. This configuration accordingly enables the use of fluorescent agents.

In another embodiment, agents can be excited in the blue, violet, and near-ultraviolet region and fluoresce light in the red region. An example of such an agent includes porphyrin accumulation in malignant gliomas caused by the introduction of 5ALA. In this example, it is necessary to filter out the blue light while passing the remainder of the spectrum. A near-ultraviolet cut filter is used for this situation. As in the case with “near-infrared” discussed above, the NUV light source708chas a different primary wavelength from the near-ultraviolet cut filter in the filter740.

H. Example Lens Carrier

Section IV(D) above mentions that at least some of the lenses of the front lens set714, the zoom lens assembly716, and/or the lens barrel set718may move in one or more carriers along rails. For example, the front zoom lens set724may comprise a carrier that moves front zoom lens726and728together axially.

FIGS. 11 and 12show diagrams of example carriers, according to example embodiments of the present disclosure. InFIG. 11, carrier724includes the right front zoom lens726and the left front zoom lens728within a support structure1102. The carrier724includes a rail holder1104configured to moveably connect to rail1106. A force ‘F’ is applied to an actuation section1108to cause the carrier724to move along the rail1106. The force ‘F’ may be applied by a leadscrew or other linear actuation device. As illustrated inFIG. 11, the force ‘F’ is applied at an offset of the carrier724. Friction between the rail1106and the carrier724generates a moment Mythat causes the support structure1102to move slightly around the Y-axis shown inFIG. 11. This slight movement may cause the right front zoom lens726and the left front zoom lens728to shift slightly in opposite directions causing spurious parallax, which is an error in a parallax between views of a stereoscopic image.

FIG. 12shows another example of the carrier724. In this example, force ‘F’ is applied symmetrically at center structure1202, which is connected to the rail holder1104and the support structure1102. The force ‘F’ generates a moment Mxthat causes the carrier724to rotate or move slightly around the X-axis shown inFIG. 12. The rotational movement causes the right front zoom lens726and the left front zoom lens728to shift in the same direction by the same degree of movement, thereby reducing (or eliminating) the onset of spurious parallax.

WhileFIGS. 11 and 12show lenses726and728within one carrier, in other embodiments the lenses726and728may each be within a carrier. In these examples, each lens would be on a separate track or rail. Separate leadscrews may be provided for each of the lenses to provide independent axial movement along the respective optical path.

Section IV(D) above mentions that at least some of the lenses of the front lens set714, the zoom lens assembly716, and/or the lens barrel set718may be moved radially, rotated, and/or tilted. Additionally or alternatively, the optical image sensors746and748may be moved axially and/or tilted with respect to their respective incident optical path. The axial and/or tilt movement may be provided by one or more flexures. In some examples, the flexures may be cascaded such that a first flexure provides motion in a first direction and separate flexure provides independent motion in a second direction. In another example, a first flexure provides tilt along a pitch axis and separate flexure provides tilt along a yaw axis.

FIG. 13shows a diagram of an example dual flexure1300, according to an example embodiment of the present disclosure. The flexure1300illustrated inFIG. 13is for the optical image sensor744and is configured to independently move the right optical image sensor746and the left optical image sensor748along their respective optical axis for purposes of final focusing. The flexure1300includes a support beam1301for connection to the housing302of the example stereoscopic visualization camera300and to provide a rigid base for actuation. The flexure1300also includes a beam1302for each channel (e.g., sensor746and748) that is rigid in all directions except for the direction of motion1310. The beam1302is connected to flexing hinges1303that enable the beam1302to move in a direction of motion1310, a parallelogram translation in this example.

An actuator device1304flexes the beam1302in the desired direction for a desired distance. The actuator device1304includes a push-screw1306and a pull screw1308, for each channel, which apply opposite forces to the beam1302causing the flexing hinges1303to move. The beam1302may be moved inward, for example, by turning the push-screw1306to push on the beam1302. The flexure1300illustrated inFIG. 13is configured to independently move the right optical image sensor746and the left optical image sensor748axially along their optical axis.

After the beam1302is flexed into a desired position, a locking mechanism is engaged to prevent further movement, thereby creating a rigid column. The locking mechanism includes the push-screw1306and its respective concentric pull screw1308, that when tightened, create large opposing forces that result in the rigid column of the beam1302.

While the optical image sensors746and748are shown as being connected to the same flexure1300, in other examples, the sensors may be connected to separate flexures. For example, returning toFIG. 8, the right optical image sensor746is connected to flexure750and the left optical image sensor748is connected to flexure752. The use of the separate flexures750and752enables the optical image sensors746and748to be separately adjusted to, for example, align the left and right optical views and/or reduce or eliminate spurious parallax.

In addition, whileFIG. 13shows image sensors746and748connected to the flexure1300, in other examples, the lenses of the front lens set714, the zoom lens assembly716, the lens barrel set718, and/or the final optical element set742may be connected to alternative or additional flexures instead. In some instances, each of the right and left lenses of the front lens set714, the zoom lens assembly716, the lens barrel set718, and/or the final optical element set742may be connected to a separate flexure1300to provide independent radial, rotational, and/or tilt adjustment.

The flexure1300may provide motion resolution of less than a micron. As a result of the very fine motion adjustment, images from the right and left optical paths may have an alignment accuracy of several or even one pixel for a 4K display monitor. Such accuracy is viewed on each display512,514by overlaying the left and right views and observing both views with both eyes, rather than stereoscopically.

In some embodiments, the flexure1300can include the flexure disclosed in U.S. Pat. No. 5,359,474, titled “SYSTEM FOR THE SUB-MICRON POSITIONING OF A READ/WRITE TRANSDUCER,” the entirety of which is incorporated herein by reference. In yet other embodiments, the lenses of the front lens set714, the zoom lens assembly716, the lens barrel set718, and/or the final optical element set742may be stationary in a radial direction. Instead, a deflecting element (e.g., a mirror) with an adjustable deflection direction in an optical path may be used to steer the right and/or left optical paths to adjust alignment and/or spurious parallax. Additionally or alternatively, a tilt/shift lens may be provided in the optical path. For instance, a tilt of an optical axis may be controlled with an adjustable wedge lens. In further embodiments, lenses of the front lens set714, the zoom lens assembly716, the lens barrel set718, and/or the final optical element set742may include dynamic lenses with parameters that can be changed electronically. For example, the lenses may include Varioptic liquid lenses produced by Invenios France SAS.

V. Example Processors of the Stereoscopic Visualization Camera

The example stereoscopic visualization camera300is configured to record image data from the right and left optical paths and output the image data to the monitor(s)512and/or514for display as a stereoscopic image.FIG. 14shows a diagram of modules of the example stereoscopic visualization camera300for acquiring and processing image data, according to an example embodiment of the present disclosure. It should be appreciated that the modules are illustrative of operations, methods, algorithms, routines, and/or steps performed by certain hardware, controllers, processors, drivers, and/or interfaces. In other embodiments, the modules may be combined, further partitioned, and/or removed. Further, one or more of the modules (or portions of a module) may be provided external to the stereoscopic visualization camera300such as in a remote server, computer, and/or distributed computing environment.

In the illustrated embodiment ofFIG. 14, the components408,702to750, and1300inFIGS. 7 to 13are collectively referred to as optical elements1402. The optical elements1402(specifically the optical image sensors746and748) are communicatively coupled to an image capture module1404and a motor and lighting module1406. The image capture module1404is communicatively coupled to an information processor module1408, which may be communicatively coupled to an externally located user input device1410and one or more display monitors512and/or514.

The example image capture module1404is configured to receive image data from the optical image sensors746and748. In addition, the image capture module1404may define the pixel sets1006and1008within the respective pixel grids1002and1004. The image capture module1404may also specify image recording properties, such as frame rate and exposure time.

The example motor and lighting module1406is configured to control one or more motors (or actuators) to change a radial, axial, and/or tilt position of one or more of the optical elements1402. For instance, a motor or actuator may turn a drive screw to move the carrier724along the track1106, as shown inFIGS. 11 and 12. A motor or actuator may also turn the push-screw1306and/or the pull screw1308of the flexure1300ofFIG. 13to adjust a radial, axial, or tilt position of a lens and/or optical image sensor. The motor and lighting module1406may also include drivers for controlling the light sources708.

The example information processor module1408is configured to process image data for display. For instance, the information processor module1408may provide color correction to image data, filter defects from the image data, and/or render image data for stereoscopic display. The information processor module1408may also perform one or more calibration routines to calibrate the stereoscopic visualization camera300by providing instructions to the image capture module1404and/or the motor and lighting module1406to perform specified adjustments to the optical elements. The information processor module1408may further determine and provide in real-time instructions to the image capture module1404and/or the motor and lighting module1406to improve image alignment and/or reduce spurious parallax.

The example user input device1410may include a computer to provide instructions for changing operation of the stereoscopic visualization camera300. The user input device1410may also include controls for selecting parameters and/or features of the stereoscopic visualization camera300. In an embodiment, the user input device1410includes the control arms304ofFIG. 3. The user input device1410may be hardwired to the information processor module1408. Additionally or alternatively, the user input device1410is wirelessly or optically communicatively coupled to the information processor module1408.

The example display monitors512and514include, for example, televisions and/or computer monitors configured to provide a three-dimensional viewing experience. For example, the display monitors may include the LG® 55LW5600 television. Alternatively, the display monitors512and514may include a laptop screen, tablet screen, a smartphone screen, smart-eyewear, a projector, a holographic display, etc.

The sections that follow describe the image capture module1404, the motor and lighting module1406, and the information processor module1408in more detail.

A. Example Image Capture Module

FIG. 15shows a diagram of the image capture module1404, according to an example embodiment of the present disclosure. The example image capture module1404includes an image sensor controller1502, which includes a processor1504, a memory1506, and a communications interface1508. The processor1504, the memory1506, and the communications interface1508may be communicatively coupled together via an image sensor controller bus1512.

The processor1504is programmable with one or more programs1510that are persistently stored within the memory1506. The programs1510include machine readable instructions, which when executed, cause the processor1504to perform one or more steps, routines, algorithms, etc. In some embodiments, the programs1510may be transmitted to the memory1506from the information processor module1408and/or from the user input device1410. In other examples, the programs1510may be transmitted to the processor1504directly from the information processor module1408and/or from the user input device1410.

The example image sensor controller1502is communicatively coupled to the right optical image sensor746and the left optical image sensor748of the optical elements1402. The image sensor controller1502is configured to provide power to the optical image sensors746and748in addition to sending timing control data and/or programming data. In addition, the image sensor controller1502is configured to receive image and/or diagnostic data from the optical image sensors746and748.

Each of the optical image sensors746and748contains programmable registers to control certain parameters and/or characteristics. One or more of the registers may specify a location of the pixel sets1006and1008within the respective pixel grids1002and1004ofFIG. 10. The registers may store a value of a starting location with respect to an origin point or edge point of the pixel grids1002and1004. The registers may also specify a width and height of the pixel sets1006and1008to define a rectangular region of interest. The image sensor controller1502is configured to read pixel data for pixels that are within the specified pixel sets1006and1008. In some embodiments, the registers of the optical image sensors746and748may facilitate the designation of pixel sets of other shapes, such as circles, ovals, triangles, etc. Additionally or alternatively, the registers of the optical image sensors746and748may enable multiple pixel sets to be specified simultaneously for each of the pixel grids1002and1004.

A light-sensing portion of the pixels of the pixel grids1002and1004is controlled by embedded circuitry, which specifies different modes of light-sensing. The modes include a reset mode, an integration mode, and a readout mode. During the reset mode, a charge storage component of a pixel is reset to a known voltage level. During the integration mode, the pixel is switched to an “on” state. Light that reaches a sensing area or element of the pixel causes a charge to accumulate in a charge storage component (e.g., a capacitor). The amount of stored electrical charge corresponds to the amount of light incident on the sensing element during the integration mode. During the readout mode, the amount of electrical charge is converted into a digital value and read out of the optical image sensors746and748via the embedded circuitry and transmitted to the image sensor controller1502. To read every pixel, the charge storage component of each pixel in a given region is connected sequentially by switched internal circuitry to a readout circuit, which performs the conversion of the electrical charge from an analog value to digital data. In some embodiments, the pixel analog data is converted to 12-bit digital data. However, it should be appreciated that the resolution may be less or greater based on allowances for noise, settling time, frame rate, and data transmission speed. The digital pixel data of each pixel may be stored to a register.

The example processor1504of the image sensor controller1502ofFIG. 15is configured to receive pixel data (e.g., digital data indicative of an electrical charge stored in the pixel corresponding to an amount of incident light on an element of the pixel) from each of the pixels within the pixel sets1006and1008. The processor1504forms a right image from the pixel data received from the right optical image sensor746. In addition, the processor1504forms a left image from the pixel data received from the left optical image sensor748. Alternatively, the processor1504forms only a portion (for example, one row or several rows) of each the left and right images before transmitting the data downstream. In some embodiments, the processor1504uses a register location to determine a location of each pixel within an image.

After the right and left images are created, the processor1504synchronizes the right and left images. The processor1504then transmits both of the right and left images to the communications interface1508, which processes the images into a format for transmission to the information processor module1408via a communications channel1514. In some embodiments, the communications channel1514conforms to the USB 2.0 or 3.0 standard and may comprise a copper or fiber optical cable. The communications channel1514may enable up to approximately 60 pairs (or more) of left and right images (having a stereoscopic resolution of 1920×1080 and a data conversion resolution of 12-bits) per second to be transmitted per second. The use of a copper USB cable enables power to be provided from the information processor module1408to the image capture module1404.

The sections below further describe features provided by the processor1504of the image sensor controller1502executing certain programs1510to acquire and/or process image data from the optical image sensors746and748.

1. Exposure Example

The example processor1504may control or program an amount of time the optical image sensors746and748are in the integration mode, discussed above. The integration mode occurs for a time period referred to as an exposure time. The processor1504may set the exposure time by writing a value to an exposure register of the optical image sensors746and748. Additionally or alternatively, the processor1504may transmit instructions to the optical image sensors746and748signaling the start and end of the exposure time. The exposure time may be programmable between a few milliseconds (“ms”) to a few seconds. Preferably the exposure time is approximately the inverse of the frame rate.

In some embodiments, the processor1504may apply a rolling shutter method to the optical image sensors746and748to read pixel data. Under this method, the exposure time for a given row of pixels of the pixel sets1006and1008begins just after the pixels in that row have been read out and then reset. A short time later, the next row (which is typically physically most proximate to the row just set) is read, and accordingly reset with its exposure time restarted. The sequential reading of each pixel row continues until the last or bottom row of the pixel sets1006and1008have been read and reset. The processor1504then returns to the top row of the pixel sets1006and1008to read pixel data for the next image.

In another embodiment, the processor1504applies a global shutter method. Under this method, the processor1504implements readout and reset in a manner similar to the rolling shutter method. However, in this method integration occurs simultaneously for all pixels in the pixel sets1006and1008. The global shutter method has the advantage of reducing defects in an image compared to the rolling shutter method since all of the pixels are exposed at the same time. In comparison, in the rolling shutter method, there is a small time delay between exposing the lines of the pixel set. Small defects can develop during the times between line exposures, especially between top lines and bottom lines where small changes at the target site700between reads can occur.

2. Dynamic Range Example

The example processor1504may execute one or more programs1510to detect light that is outside of a dynamic range of the optical image sensors746and748. Generally, extremely bright light completely fills a charge storage region of a pixel, thereby resulting in lost image information regarding the exact brightness level. Similarly, extremely low light or lack of light fails to impart a meaningful charge in a pixel, which also results in lost image information. Images created from this pixel data accordingly do not accurately reflect the light intensity at target site700.

To detect light that is outside the dynamic range, the processor1504may execute one of several high dynamic range (“HDR”) programs1510including, for example, a multiple-exposure program, a multi-slope pixel integration program, and a multi-sensor image fusion program. In an example, the multiple-exposure program may utilize HDR features integrated or embedded with the optical image sensors746and748. Under this method, the pixel sets1006and1008are placed into the integration mode for a normal expose time. The lines of the pixel sets1006and1008are read and stored in a memory at the optical image sensors746and748and/or the memory1506of the image sensor controller1502. After the read is performed by the processor1504, each line in the pixel sets1006and1008is turned on again for a second exposure time that is less than the normal exposure time. The processor1504reads each of the lines of pixels after the second exposure time and combines this pixel data with the pixel data from the normal exposure time for the same lines. The processor1504may apply tone-mapping to choose between (or combine) the pixel data from the normal-length and short-length exposure times and map the resulting pixel data to a range that is compatible with downstream processing and display. Using the multiple-exposure program, the processor1504is able to expand the dynamic range of the optical image sensors746and748and compress the resulting range of pixel data for display.

The processor1510may operate a similar program for relatively dark light. However, instead of the second exposure time being less than the normal time, the second exposure time is greater than the normal time, thereby providing the pixels more time to accumulate a charge. The processor1510may use tone-mapping to adjust the read pixel data to compensate for the longer exposure time.

3. Frame Rate Example

The example processor1510may control or specify a frame rate for the optical image sensors746and748. In some embodiments, the optical image sensors746and748include on-board timing circuitry and programmable control registers to specify the number of times per second each of the pixels within the pixel sets1006and1008are to be cycled through the imaging modes discussed above. A frame or image is formed each time the pixel set progresses through the three modes. A frame rate is the number of times per second the pixels in the pixel sets1006and1008are integrated, read, and reset.

The processor1510may be synchronized with the optical image sensors746and748such that reads are conducted at the appropriate time. In other examples, the processor1510is asynchronous with the optical image sensors746and748. In these other examples, the optical image sensors746and748may store pixel data after a local read to a temporary memory or queue. The pixel data may then be read periodically by the processor1510for right and left image synchronization.

The processing of frames or images in a time-sequential manner (e.g., creation of an image stream) provides an illusion of motion conveyed as a video. The example processor1510is configured to program a frame rate that provides the appearance of a smooth video to an observer. A frame rate that is too low makes any motion appear choppy or uneven. Movie quality above a maximum threshold frame rate is not discernable to an observer. The example processor1510is configured to generate approximately 20 to 70 frames per second, preferably between 50 and 60 frames per second for typical surgical visualization.

4. Sensor Synchronization Example

The example processor1504ofFIG. 15is configured to control the synchronization of the optical image sensors746and748. The processor1504may, for instance, provide power simultaneously to the optical image sensors746and748. The processor1504may then provide a clock signal to both of the optical image sensors746and748. The clock signal enables the optical image sensors746and748to operate independently in a free-run mode but in a synchronized and/or simultaneous manner. Accordingly, the optical image sensors746and748record pixel data at nearly the same time. The example processor1504receives the pixel data from the optical image sensors746and748, constructs at least a fraction of the images and/or frames and synchronizes the images and/or frames (or fraction thereof) to account for any slight timing mismatches. Typically, the lag between the optical image sensors746and748is less than 200 microseconds. In other embodiments, the processor1504may use a synchronization pin to simultaneously activate the optical image sensors746and748after, for example, each reset mode.

B. Example Motor and Lighting Module

The example stereoscopic visualization camera300ofFIG. 15includes the motor and lighting module1406to control one or more motors or actuators for moving lenses of the optical elements1402and/or controlling lighting output from the light sources708. The example motor and lighting module1406includes a motor and lighting controller1520that contains a processor1522, a memory1524, and a communications interface1526that are communicatively coupled together via communication bus1528. The memory1524stores one or more programs1530that are executable on the processor1522to perform control, adjustment, and/or calibration of the lenses of the optical elements1402and/or the light sources708. In some embodiments, the programs1530may be transmitted to the memory1524from the information processor module1408and/or the user input device1410.

The communications interface1526is communicatively coupled to the communications interface1508of the image capture module1404and a communications interface1532of the information processor module1408. The communications interface1526is configured to receive command messages, timing signals, status messages, etc. from the image capture module1404and the information processor module1408. For example, the processor1504of the image capture module1404may send timing signals to the processor1522to synchronize timing between lighting control and exposure time of the optical image sensors746and748. In another example, the information processing module1408may send command messages instructing certain light sources708to be activated and/or certain lenses of the optical elements1402to be moved. The commands may be in response to input received from an operator via, for example, the user input device1410. Additionally or alternatively, the commands may be in response to a calibration routine and/or real-time adjustment to reduce or eliminate image misalignment and/or defects such as spurious parallax.

The example motor and lighting module1406includes drivers that provide power to control motors for adjusting an axial and/or radial position of the lenses of the optical elements1402and/or the light output from the light sources708. Specifically, the motor and lighting module1406includes a NUV light driver1534to transmit a NUV signal to the NUV light source708c, a NIR light driver1536to transmit a NIR signal to the NIR light source708b, and a visible light driver1538to transmit a visible light signal to the visible light source708a.

In addition, the motor and lighting module1406includes a filter motor driver1540to transmit a filter motor signal to a filter motor1542, which controls the filter740ofFIGS. 7 and 8. The motor and lighting module1406includes a rear zoom lens motor driver1544to transmit a rear zoom lens motor signal to a rear zoom lens motor1546, a front zoom lens motor driver1548to transmit a front zoom lens motor signal to a front zoom lens motor1550, and a rear working distance lens motor driver1552to transmit a working distance lens motor signal to a working distance lens motor1554. The motor and lighting module1406may also include a motor and/or actuator to move and/or tilt the deflecting element712.

The rear zoom lens motor1546is configured to rotate a drive screw that causes carrier730to move axially along a track or rail. The front zoom lens motor1550is configured to rotate a drive screw that causes carrier724to move axially along the track1106shown inFIGS. 11 and 12. The working distance lens motor1554is configured to rotate a drive screw that causes the rear working distance lens702to move axially along a track or rail.

The drivers1536,1538, and1540may include any type of lighting driver, transformer, and/or ballast. The drivers1536,1538, and1540are configured to output a pulse width modulation (“PWM”) signal to control an intensity of light output by the light sources708. In some embodiments, the processor1522may control the timing of the drivers1536,1538, and1540to correspond to a timing for applying a certain filter using the filter motor driver1540.

The example drivers1540,1544,1548, and1552may include, for example stepper motor drivers and/or DC motor drivers. Likewise, the motors1542,1546,1550, and/or1554may include a stepper motor, a DC motor, or other electrical, magnetic, thermal, hydraulic, or pneumatic actuator. The motors1542,1546,1550, and/or1554may include, for example, a rotary encoder, a slotted optical switch (e.g., a photointerrupter), and/or a linear encoder to report an angular position of a shaft and/or axle for feedback reporting and control. Alternative embodiments may include voice-coil motors, piezoelectric motors, linear motors, with suitable drivers, and equivalents thereof.

To control the drivers1534,1536,1538,1540,1544,1548, and1552, the processor1522is configured to use a program1530for converting a command message into a digital and/or analog signal. The processor1522transmits the digital and/or analog signal to the appropriate driver, which outputs an analog power signal, such as a PWM signal corresponding to the received signal. The analog power signal provides power to an appropriate motor or actuator causing it to rotate (or otherwise move) by a desired amount.

The processor1522may receive feedback from the drivers1534,1536,1538,1540,1544,1548, and1552, the motors1542,1546,1550, and/or1554, and/or the light sources708. The feedback corresponds to, for example, a lighting level or lighting output. Regarding the motors, the feedback corresponds to a position of a motor (or other actuator) and/or an amount of movement. The processor1522uses a program1530to translate the received signal into digital feedback to determine, for example, a radial, tilt, and/or axial position of a lens based on an angular position of the corresponding motor or actuator shaft. The processor1522may then transmit a message with the position information to the information processor module1408for display to a user and/or to track a position of the lenses of the optical elements1402for calibration.

In some embodiments, the motor and lighting module1406may include additional drivers to change an axial, tilt, and/or radial position of individual lenses within the optical elements1402. For example, the motor and lighting module1406may include drivers that control motors for actuating flexures750and752for the optical image sensors746and748for tilting and/or radial/axial adjustment. Further, the motor and lighting module1406may include drivers that control motors (or actuators) for individually tilting and/or adjusting front lenses720and722, the front zoom lenses726and728, the rear zoom lenses732and734, the lens barrels736and738, and/or final optical elements745and747radially along an x-axis or y-axis and/or axially. Independent adjustment of the lenses and/or sensors enables, for example, the motor and lighting controller1520to remove image defects and/or align the left and right images.

The following sections describe how the processor1552executes one or more programs1530to change a working distance, zoom, filter position, lens position, and/or light output.

1. Working Distance Example

The example processor1522of the motor and lighting module1406ofFIG. 15is configured to adjust a working distance of the stereoscopic visualization camera300. The working distance is set by adjusting a distance between the rear working distance lens704and the front working distance lens408. The processor1522adjusts the distance by causing the rear working distance lens704to move relative to the front working distance lens408. Specifically, the processor1522sends a signal to the rear working distance lens motor driver1552, which activates the working distance lens motor1554for a predetermined time proportional to an amount the rear working distance lens704is to be moved. The working distance lens motor1554drives a leadscrew through threads attached to a sliding track that holds the rear working distance lens704. The working distance lens motor1554causes the lens704to move a desired distance, thereby adjusting the working distance. The working distance lens motor1554may provide a feedback signal to the processor1522, which determines if the rear working distance lens704was moved the desired amount. If the movement is less or more than desired, the processor1522may send instructions further refining the position of the rear working distance lens704. In some embodiments, the information processor module1408may determine feedback control for the rear working distance lens704.

To determine a position of the rear working distance lens704, the processor1522may operate one or more calibration programs1530. For example, upon activation, the processor1522may instruct the working distance lens motor1554to drive a leadscrew to move the rear working distance lens704along a track or rail until triggering a limit switch at one end of the motion range. The processor1522may designate this stop position as a zero-point for the encoder of the motor1554. Having knowledge of the current position of the rear working distance lens704and the corresponding encoder value, the processor1522becomes capable of determining a number of shaft rotations to cause the rear working distance lens704to move to a desired position. The number of shaft rotations is transmitted in an analog signal to the working distance lens motor1554(via the driver1552) to accordingly move the lens704to a specified position.

2. Zoom Example

The example processor1522ofFIG. 15is configured to execute one or more programs1530to change a zoom level of the stereoscopic visualization camera300. As discussed above, zoom (e.g., magnification change) is achieved by changing positions of the front zoom set724and the rear zoom set730relative to each other and relative to the front lens set714and the lens barrel set718. Similar to the calibration procedure described above for the rear working distance lens704, the processor1522may calibrate positions of the sets724and730along tracks or rails. Specially, the processor1522sends instructions causing the rear zoom lens motor1546and the front zoom lens motor1550to move the sets724and730(e.g., carriers) along a rail (or rails) to a stop position at a limit switch. The processor1522receives encoder feedback from the motors1546and1550to determine an encoder value associated with the stop position for the sets724and730. The processor1522may then zero-out the encoder value or use the known encoder value at the stop position to determine how much the motors1546and1550are to be activated to achieve a desired position for the sets724and730along the rail.

In addition to calibration for stop position, the processor1522may execute programs1530that define locations for sets724and730to achieve a desired zoom level. For example, a known pattern of distance settings versus a set of desired zoom values may be stored as a program1530(or a look-up table) during a calibration procedure. The calibration procedure may include placing a template within the target site700and instructing the processor522to move the sets724and730until a certain designated marker or character is a certain size in right and left images or frames. For example, a calibration routine may determine positions of the set724and730on a rail corresponding to when character “E” on a template at the target site700is displayed in right and left images as having a height of 10 pixels.

In some embodiments, the information processor module1408may perform the visual analysis and send instructions to the processor1522regarding desired movement for the sets724and730to zoom in or zoom out. In addition, the information processor1408may send instructions for moving the focal plane such that the target site700at the desired zoom level is in focus. The instructions may include, for example, instructions to move the rear working distance lens704and/or moving the sets724and730together and/or individually. In some alternative embodiments, the processor1522may receive calibration parameters for the rail position of the front zoom set724and the rear zoom set730at certain zoom levels from the user input device1410or another computer.

The example processor1522and/or the information processor module1408may send instructions such that an image remains in focus while magnification changes. The processor1522, for example, may use a program1530and/or a look-up-table to determine how certain lenses are to be moved along an optical axis to retain focus on the target site700. The programs1530and/or look-up-table may specify magnification levels and/or set points on a rail and corresponding lens adjustments needed to keep the focal plane from moving.

Table 2 below shows an example program1530or look-up-table that may be used by the processor1522to retain focus while changing magnification. The position of the front zoom lens set724and the rear zoom lens set730is normalized based on a length of a rail to stop positions for the respective sets724and730. To decrease magnification, the rear zoom lens set is moved toward the lens barrel set718, thereby increasing a position along a rail. The front zoom lens set724is also moved. However, its movement does not necessarily equal the movement of the rear zoom lens set730. Instead, the movement of the front zoom lens set724accounts for changing a distance between the sets724and730to retain the position of the focal plane to maintain focus while changing magnifications. For example, to decrease a magnification level from 10× to 9×, the processor1522instructs the rear zoom lens set730to move from position10to position11along a rail. In addition, the processor1522instructs the front zoom lens set724to move from position5to position4along a rail (or same rail as the set730). Not only have the sets724and730moved to change magnification, the sets724and730have moved relative to each other to retain focus.

It should be appreciated that Table 2 provides an example of how the sets724and730may be moved. In other examples, Table 2 may include additional rows to account for more precise magnifications and/or positions of the sets724and730. Additionally or alternatively, Table 2 may include a column for the rear working distance lens704. For example, the rear working distance lens704may be moved instead of or in conjunction with the front zoom lens set724to retain focus. Further, Table 2 may include rows specifying positions for the sets724and730and the rear working distance lens704to retain focus during changes in working distance.

The values in Table 2 may be determined through calibration and/or received from a remote computer or the user input device1410. During calibration, the information processor module1408may operate a calibration program1560that progresses through different magnifications and/or working distances. A processor1562at the information processor module1408may perform image processing of the images themselves or received pixel data to determine when a desired magnification is achieved using, for example, a template with predetermined shapes and/or characters. The processor1562determines if the received images are in-focus. Responsive to determining images are out of focus, the processor1562sends instructions to the processor1522to adjust the front zoom lens set724and/or the rear working distance lens set704. The adjustment may include iterative movements in forward and reverse directions along an optical path until the processor1562determines images are in focus. To determine an image is in focus, the processor1562may perform, for example, image analysis searching for images where light fuzziness is minimal and/or analyzing pixel data for differences in light values between adjacent pixel regions (where greater differences correspond to more in focus images). After determining an image is in focus at a desired working distance and magnification, the processor1562and/or the processor1522may then record positions of the sets724and730and/or the rear working distance lens704and corresponding magnification level.

3. Filter Position Example

The example processor1522of the motor and lighting module1406ofFIG. 15is configured to move the filter740into the right and left optical paths based on received instructions. In some examples, the filter740may include a mirror array. In these examples, the processor1522sends instructions to the filter motor driver1540to actuate one or more motors1542to change positions of the mirrors. In some instances, the driver1540may send an electrical charge along one or more paths to the filter740, causing certain mirror elements to switch to an on or off position. In these examples, the filter type selection is generally binary based on which mirrors to actuate.

In other examples, the filter740may include a wheel with different types of filters such as an infrared cut filter, near-infrared bandpass filter, and near-ultraviolet cut filter. In these examples, the wheel is rotated by the filter motor1542. The processor1522determines stop positions of the wheel corresponding to partitions between the different filters. The processor1522also determines rotary encoder value corresponding to each of the stop positions.

The processor1522may operate a calibration program1530and/or the processor1562may operate a calibration program1560to determine the stop positions. For example, the processor1522may rotate the filter wheel740slowly, with the processor1562determining when light received at the pixels changes (using either image analysis or reading pixel data from the image capture module1404). A change in a light value at the pixels is indicative of a change in the filter type being applied to the optical paths). In some instances, the processor1522may change which light sources708are activated to create further distinction at the pixels when a different filter type is applied.

4. Light Control and Filter Example

As disclosed above, the processor1522may control the light sources708in conjunction with the filter740to cause light of a desired wavelength to reach the optical image sensors746and748. In some examples, the processor1522may control or synchronize timing between activation of one or more of the light sources708and one or more of the filters740. To synchronize timing, a program1530may specify a delay time for activating a certain filter. The processor1522uses this program1530to determine when, for example a signal to activate the filter740is to be transmitted relative to sending a signal to turn on a light source708. The scheduled timing ensures the appropriate filter740is applied when the specified light source708is activated. Such a configuration enables features highlighted by one light source708(such as fluorescence) to be shown on top of or in conjunction with features displayed under a second light source708, such as white or ambient light.

In some instances, the light sources708may be switched as fast as the light filters740may be changed, thereby enabling images recorded in different lights to be shown in conjunction on top of each other. For example, veins or other anatomical structures that emit fluorescence (due to an administered dye or contrast agent) may be shown on top of an image under ambient lighting. In this example, the veins would be highlighted relative to the background anatomical features shown in visible light. In this instance, the processor1562and/or a graphics processing unit1564(e.g., a video card or graphics card) of the information processor module1408combines or overlays one or more images recorded during application of one filter with images recorded during application of a subsequent filter.

In some embodiments, the processor1522may activate multiple light sources708at the same time. The light sources708can be activated simultaneously or sequentially to “interleave” light of different wavelengths to enable different information to be extracted using appropriate pixels at the optical image sensors746and748. Activating the light sources simultaneously may help illuminate dark fields. For example, some applications use UV light to stimulate fluorescence at a target site700. However, UV light is perceived by an operator as being very dark. Accordingly, the processor1522may activate the visible light source1538periodically to add some visible light to the viewing field so that the surgeon can observe the field-of-view without overwhelming pixels that are sensitive to UV light but can also detect some visible light. In another example, alternating between light sources708avoids, in some instances, washing out pixels of the optical image sensors746and748that have overlapping sensitivity at the edges of their ranges.

5. Light Intensity Control

The example processor1522ofFIG. 15is configured to execute one or more programs1530to change an intensity of or a level of illumination provided by the light sources708. It should be appreciated that the depth of field is dependent on the level of illumination at the target site700. Generally, higher illumination provides a greater depth of field. The processor1522is configured to ensure an appropriate amount of illumination is provided for a desired depth of field without washing out or overheating the field-of-view.

The visible light source708ais driven by the visible light driver1538and outputs light in the human-visible part of the spectrum as well as some light outside that region. The NIR light source708bis driven by the NIR light driver1536and outputs light primarily at a wavelength that referred to as near-infrared. The NUV light source708cis driven by the NUV light driver1534and outputs light primarily at a wavelength that is deep in the blue part of the visible spectrum, which is referred to as near-ultraviolet. The respective light drivers1534,1536, and1538are controlled by commands provided by the processor1522. Control of the respective output spectra of the light sources708is achieved by PWM signal, where a control voltage or current is switched between a minimum (e.g., off) and maximum (e.g., on) value. The brightness of the light that is output from the light sources708is controlled by varying the switching rate as well as the percentage of time the voltage or current is at the maximum level per cycle in the PWM signal.

In some examples, the processor1522controls an output of the light sources708based on a size of the field-of-view or zoom level. The processor1522may execute a program1530that specifies for certain light sensitive settings that light intensity becomes a function of zoom. The program1530may include, for example a look-up-table that correlates a zoom level to a light intensity value. The processor1522uses the program1530to select the PWM signal for the light source708based on the selected magnification level. In some examples, the processor1522may reduce light intensity as the magnification increases to maintain the amount of light provided to the field-of-view per unit of area.

C. Example Information Processor Module

The example information processor module1408within the stereoscopic visualization camera300ofFIG. 15is configured to analyze and process images/frames received from the image capture module1404for display. In addition, the information processor module1408is configured to interface with different devices and translate control instructions into messages for the image capture module1404and/or the motor and lighting module1406. The information processor module1408may also provide an interface for manual calibration and/or manage automatic calibration of the optical elements1402.

As shown inFIG. 15, the information processor module1408is communicatively and/or electrically coupled to the image capture module1404and the motor and lighting module1406. For example, the communications channel1514in addition to communications channels1566and1568may include USB 2.0 or USB 3.0 connections. As such, the information processor module1408regulates and provides power to the modules1404and1406. In some embodiments, the information processor module1408converts 110-volt alternating current (“AC”) power from a wall outlet into a 5, 10, 12, and/or 24 volt direct current (“DC”) supply for the modules1404and1406. Additionally or alternatively, the information processor module1408receives electrical power from a battery internal to the housing302of the stereoscopic visualization camera300and/or a battery at the cart510.

The example information processor module1408includes the communications interface1532to communicate bidirectionally with the image capture module1404and the motor and lighting module1406. The information processor module1408also includes the processor1562configured to execute one or more programs1560to process images/frames received from the image capture module1404. The programs1560may be stored in a memory1570. In addition the processor1562may perform calibration of the optical elements1402and/or adjust the optical elements1402to align right and left images and/or remove visual defects.

To process images and/or frames into a rendered three-dimensional stereoscopic display, the example information processor module1408includes the graphics processing unit1564.FIG. 16shows a diagram of the graphics processing unit1564, according to an example embodiment of the present disclosure. During operation, the processor1562receives images and/or frames from the image capture module1404. An unpack routine1602converts or otherwise changes the images/frames from a format conducive for transmission across the communications channel1514into a format conducive for image processing. For instance, the images and/or frames may be transmitted across the communications channel1514in multiple messages. The example unpack routine1602combines the data from the multiple messages to reassemble the frames/images. In some embodiments, the unpack routine1602may queue frames and/or images until requested by the graphics processing unit1564. In other examples, the processor1562may transmit each right and left image/frame pair after being completely received and unpacked.

The example graphics processing unit1564uses one or more programs1580(shown inFIG. 15) to prepare images for rendering. Examples of the programs1580are shown inFIGS. 15 and 16. The programs1580may be executed by a processor of the graphics processing unit1564. Alternatively, each of the programs1580shown inFIG. 16may be executed by a separate graphics processor, microcontroller, and/or application specific integrated circuit (“ASIC”). For example, a de-Bayer program1580ais configured to smooth or average pixel values across neighboring pixels to compensate for a Bayer pattern applied to the pixel grids1002and1004of the right and left optical image sensors746and748ofFIGS. 7 and 8. The graphics processing unit1564may also include programs1580b,1580c, and1580dfor color correction and/or white balance adjustment. The graphics processing unit1564also includes a renderer program1580efor preparing color corrected images/frames for display on the display monitors512and514. The graphics processing unit1564may further interact and/or include a peripheral input unit interface1574, which is configured to combine, fuse, or otherwise include other images and/or graphics for presentation with the stereoscopic display of the target site700. Further details of the programs1580and the information processor module1408more generally are discussed below.

The example information processor module1408may execute one or more programs1562to check for and improve latency of the stereoscopic visualization camera300. Latency refers to the amount of time taken for an event to occur at the target site700and for that same event to be shown by the display monitors512and514. Low latency provides a feeling that the stereoscopic visualization camera300is an extension of a surgeon's eyes while high latency tends to distract from the microsurgical procedure. The example processor1562may track how much time elapses between images being read from the optical image sensors746and748until the combined stereoscopic image based on the read images is transmitted for display. Detections of high latency may cause the processor1562to reduce queue times, increase the frame rate, and/or skip some color correction steps.

1. User Input Example

The example processor1562of the information processor module1408ofFIG. 15is configured to convert user input instructions into messages for the motor and lighting module1406and/or the image capture module1402. User input instructions may include requests to change optical aspects of the stereoscopic visualization camera300including a magnification level, a working distance, a height of a focal plane (e.g., focus), a lighting source708, and/or a filter type of the filter740. The user input instructions may also include requests to perform calibration, including indications of an image being in focus and/or indications of image alignment, and/or indications of aligned ZRPs between left and right images. The user input instructions may further include adjustments to parameters of the stereoscopic visualization camera300, such as frame rate, exposure time, color correction, image resolution, etc.

The user input instructions may be received from a user input device1410, which may include the controls305of the control arm304ofFIG. 3and/or a remote control. The user input device1410may also include a computer, tablet computer, etc. In some embodiments, the instructions are received via a network interface1572and/or a peripheral input unit interface1574. In other embodiments, the instructions may be received from a wired connection and/or a RF interface.

The example processor1562includes programs1560for determining an instruction type and determining how the user input is to be processed. In an example, a user may press a button of the control305to change a magnification level. The button may continue to be pressed until the operator has caused the stereoscopic visualization camera300to reach a desired magnification level. In these examples, the user input instructions include information indicative that a magnification level is to be, for example, increased. For each instruction received (or each time period in which a signal indicative of the instruction is received), the processor1562sends a control instruction to the motor and lighting processor1406indicative of the change in magnification. The processor1522determines from a program1530how much the zoom lens sets724and730are to be moved using, for example, Table 2. The processor1522accordingly transmits a signal or message to the rear zoom lens motor driver1544and/or the front zoom lens motor driver1548causing the rear zoom lens motor1546and/or the front zoom lens motor1550to move the rear zoom lens set730and/or the front zoom lens set724by an amount specified by the processor1562to achieve the desired magnification level.

It should be appreciated that in the above example, the stereoscopic visualization camera300provides a change based on user input but also makes automatic adjustments to maintain focus and/or a high image quality. For instance, instead of simply changing the magnification level, the processor1522determines how the zoom lens sets724and730are to be moved to also retain focus, thereby saving an operator from having to perform this task manually. In addition, the processor1562may, in real-time, adjust and/or align ZRPs within the right and left images as a magnification level changes. This may be done, for example, by selecting or changing locations of the pixel sets1006and1008with respect to pixel grids1002and1004ofFIG. 10.

In another example, the processor1562may receive an instruction from the user input device1410to change a frame rate. The processor1562transmits a message to the processor1504of the image capture module1404. In turn, the processor1504writes to registers of the right and left image sensors746and748indicative of the new frame rate. The processor1504may also update internal registers with the new frame rate to change a pace at which the pixels are read.

In yet another example, the processor1562may receive an instruction from the user input device1410to begin a calibration routine for ZRP. In response, the processor1562may execute a program1560that specifies how the calibration is to be operated. The program1560may include, for example, a progression or iteration of magnification levels and/or working distances in addition to a routine for verifying image quality. The routine may specify that for each magnification level, focus is to be verified in addition to ZRP. The routine may also specify how the zoom lens sets724and730and/or the rear working distance lens704are to be adjusted to achieve an in focus image. The routine may further specify how ZRP of the right and left images are to be centered for the magnification level. The program1560may store (to a look-up-table) locations of zoom lens sets724and/or the730and/or the rear working distance lens704in addition to locations of pixel sets1006and1008and the corresponding magnification level once image quality has been verified. Thus, when the same magnification level is requested at a subsequent time, the processor1562uses the look-up-table to specify positions for the zoom lens sets724and/or the730and/or the rear working distance lens704to the motor and lighting module1406and positions for the pixel sets1006and1008to the image capture module1404. It should be appreciated that in some calibration routines, at least some of the lenses of the optical elements1402may be adjusted radially/rotationally and/or tilted to center ZRPs and/or align right and left images.

2. Interface Example

To facilitate communications between the stereoscopic visualization camera300and external devices, the example information processor module1408includes the network interface1572and the peripheral input unit interface1574. The example network interface1572is configured to enable remote devices to communicatively couple to the information processor module1408to, for example, store recorded video, control a working distance, zoom level, focus, calibration, or other features of the stereoscopic visualization camera300. In some embodiments, the remote devices may provide values or parameters for calibration look-up-tables or more generally, programs1530with calibrated parameters. The network interface1572may include an Ethernet interface, a local area network interface, and/or a Wi-Fi interface.

The example peripheral input unit interface1574is configured to communicatively couple to one or more peripheral devices1576and facilitate the integration of stereoscopic image data with peripheral data, such as patient physiological data. The peripheral input unit interface1574may include a Bluetooth® interface, a USB interface, an HDMI interface, SDI, etc. In some embodiments, the peripheral input unit interface1574may be combined with the network interface1572.

The peripheral devices1576may include, for example, data or video storage units, patient physiological sensors, medical imaging devices, infusion pumps, dialysis machines, and/or tablet computers, etc. The peripheral data may include image data from a dedicated two-dimensional infrared-specialized camera, diagnostic images from a user's laptop computer, and/or images or patient diagnostic text from an ophthalmic device such as the Alcon Constellation® system and the WaveTec Optiwave Refractive Analysis (ORA™) system.

The example peripheral input unit interface1574is configured to convert and/or format data from the peripheral devices1576into an appropriate digital form for use with stereoscopic images. Once in digital form, the graphics processing unit1564integrates the peripheral data with other system data and/or the stereoscopic images/frames. The data is rendered with the stereoscopic images for display on the display monitors512and/or514.

To configure the inclusion of peripheral data with the stereoscopic images, the processor1562may control an integration setup. In an example, the processor1562may cause the graphics processing unit1564to display a configuration panel on the display monitors512and/or514. The configuration panel may enable an operator to connect a peripheral device1576to the interface1574and the processor1562to subsequently establish communications with the device1576. The processor1564may then read which data is available or enable the operator to use the configuration panel to select a data directory location. Peripheral data in the directory location is displayed in the configuration panel. The configuration panel may also provide the operator an option to overlay the peripheral data with stereoscopic image data or display as a separate picture.

Selection of peripheral data (and overlay format) causes the processor1562to read and transmit the data to the graphics processing unit1564. The graphics processing unit1564applies the peripheral data to the stereoscopic image data for presentation as an overlay graphic (such as fusing a preoperative image or graphic with a real-time stereoscopic image), a “picture-in-picture,” and/or a sub-window to the side or on top of the main stereoscopic image window.

The example de-Bayer program1580aofFIG. 16is configured to produce images and/or frames with values for red, green, and blue color at every pixel value. As discussed above, the pixels of the right and left optical image sensors746and748have a filter that passes light in the red wavelength range, the blue wavelength range, or the green wavelength range. Thus, each pixel only contains a portion of the light data. Accordingly, each image and/or frame received in the information processor module1408from the image capture module1404has pixels that contain either red, blue, or green pixel data.

The example de-Bayer program1580ais configured to average the red, blue, and green pixel data of adjacent and/or neighboring pixels to determine more complete color data for each pixel. In an example, a pixel with red data and a pixel with blue data are located between two pixels with green data. The green pixel data for the two pixels is averaged and assigned to the pixel with red data and the pixel with blue data. In some instances, the averaged green data may be weighted based on a distance of the pixel with red data and the pixel with blue data from the respective green pixels. After the calculation, the pixels with originally only red or blue data now include green data. Thus, after the de-Bayer program1580ais executed by the graphics processing unit1564, each pixel contains pixel data for an amount of red, blue, and green light. The pixel data for the different colors is blended to determine a resulting color on the color spectrum, which may be used by the renderer program1580efor display and/or the display monitors512and514. In some examples, the de-Bayer program1580amay determine the resulting color and store data or an identifier indicative of the color.

4. Color Correction Example

The example color correction programs1580b,1580c, and1580dare configured to adjust pixel color data. The sensor color correction program1580bis configured to account or adjust for variability in color sensing of the optical image sensors746and748. The user color correction program1580cis configured to adjust pixel color data based on perceptions and feedback of an operator. Further, the display color correction program1580dis configured to adjust pixel color data based on a display monitor type.

To correct color for sensor variability, the example color correction program1580bspecifies a calibration routine that is executable by the graphics processing unit1564and/or the processor1562. The sensor calibration includes placing a calibrated color chart, such as the ColorChecker® Digital SG by X-Rite, Inc. at the target site700. The processor1562and/or the graphics processing unit1564executes the program1580b, which includes sending instructions to the image capture module1404to record right and left images of the color chart. Pixel data from the right and left images (after being processed by the de-Bayer program1580a) may be compared to pixel data associated with the color chart, which may be stored to the memory1570from a peripheral unit1576and/or a remote computer via the network interface1572. The processor1562and/or the graphics processing unit1564determines differences between the pixel data. The differences are stored to the memory1570as calibration data or parameters. The sensor color correction program1580bapplies the calibration parameters to subsequent right and left images.

In some examples, the differences may be averaged over regions of pixels such that the program1580bfinds a best-fit of color correction data that can be applied globally to all of the pixels of the optical images sensors746and748to produce colors as close to the color chart as possible. Additionally or alternatively, the program1580bmay process user input instructions received from the user unit device1410to correct colors. The instructions may include regional and/or global changes to red, blue, and green pixel data based on operator preferences.

The example sensor color correction program1580bis also configured to correct for white balance. Generally, white light should result in red, green, and blue pixels having equal values. However, differences between pixels can result from color temperature of light used during imaging, inherent aspects of the filter and sensing element of each of the pixels, and spectral filtering parameters of, for example, the deflecting element712ofFIGS. 7 and 8. The example sensor color correction program1580bis configured to specify a calibration routine to correct for the light imbalances.

To perform white balance, the processor1562(per instructions from the program1580b) may display an instruction on the display monitor512and/or514for an operator to place a neutral card at the target site700. The processor1562may then instruct the image capture module1404to record one or more images of the neutral card. After processing by the unpack routine1602and the de-Bayer program1580a, the program1580bdetermines regional and/or global white balance calibration weight values for each of the red, blue, and green data such that each of the pixels have substantially equal values of red, blue, and green data. The white balance calibration weight values are stored to the memory1570. During operation, the graphics processing unit1564uses the program1580bto apply the white balance calibration parameters to provide white balance.

In some examples, the program1580bdetermines white balance calibration parameters individually for the right and left optical image sensors746and748. Of these examples, the program1580bmay store separate calibration parameters for the left and right images. In other instances, the sensor color correction program1580bdetermines a weighting between the right and left views such that color pixel data is nearly identical for the right and left optical image sensors746and748. The determined weight may be applied to the white balance calibration parameters for subsequent use during operation of the stereoscopic visualization camera300.

In some embodiments, the sensor color correction program1580bofFIG. 16specifies that the white balance calibration parameters are to be applied as a digital gain on the pixels of the right and left optical image sensors746and748. For example, the processor1504of the image capture module1404applies the digital gain to pixel data read from each of the pixels. In other embodiments, the white balance calibration parameters are to be applied as an analog gain for each pixel's color sensing element.

The example sensor color correction program1580bmay perform white balancing and/or color correction when the different light sources708and/or filter types of the filter740are activated. As a result, the memory1570may store different calibration parameters based on which light source708is selected. Further, the sensor color correction program1580bmay perform white balancing and/or color correction for different types of external light. An operator may use the user input device1410to specify characteristics and/or a type of the external light source. This calibration enables the stereoscopic visualization camera300to provide color correction and/or white balance for different lighting environments.

The example program1580bis configured to perform calibration on each of the optical image sensors746and748separately. Accordingly, the program1580bapplies different calibration parameters to the right and left images during operation. However, in some examples, calibration may only be performed on one sensor746or748with the calibration parameters being used for the other sensor.

The example user color correction program1580cis configured to request operator-provided feedback regarding image quality parameters such as brightness, contrast, gamma, hue, and/or saturation. The feedback may be received as instructions from the user input device1410. Adjustments made by the user are stored as user calibration parameters in the memory1570. These parameters are subsequently applied by the user color correction program1580cto right and left optical images after color correction for the optical image sensors746and748.

The example display color correction program1580dofFIG. 16is configured to correct image color for a display monitor using, for example, the Datacolor™ Spyder color checker. The program1580d, similar to the program1580b, instructs the image capture module1404to record an image of a display color template at the target scene700. The display color correction program1580doperates a routine to adjust pixel data to match an expected display output stored in a look-up-table in the memory1570. The adjusted pixel data may be stored as display calibration parameters to the memory1570. In some examples, a camera or other imaging sensor may be connected to the peripheral input unit interface1574, which provides images or other feedback regarding color recorded from the display monitors512and514, which is used to adjust the pixel data.

5. Stereoscopic Image Display Example

The example renderer program1580eof the graphics processing unit1564ofFIG. 16is configured to prepare right and left images and/or frames for three-dimensional stereoscopic display. After the pixel data of the right and left images is color corrected by the programs1580b,1580c, and1580d, the renderer program1580eis configured to draw left-eye and right-eye data into a format suitable for stereoscopic display and place the final rendered version into an output buffer for transmission to one of the display monitors512or514.

Generally, the renderer program1580ereceives a right image and/or frame and a left image and/or frame. The renderer program1580ecombines the right and left images and/or frames into a single frame. In some embodiments, the program1580eoperates a top-bottom mode and condenses the left image data in height by half. The program1580ethen places the condensed left image data in a top half of the combined frame. Similarly, the program1580econdenses the right image data in height by half and places the condensed right image data in a bottom half of the combined frame.

In other embodiments, the renderer program1580eoperates a side-by-side mode where each of the left and right images are condensed in width by half and combined in a single image such that the left image data is provided on a left half of the image while right image data is provided on a right half of the image. In yet an alternative embodiment, the renderer program1580eoperates a row-interleaved mode where every other line in the left and right frames is discarded. The left and right frames are combined together to form a complete stereoscopic image.

The example renderer program1580eis configured to render combined left and right images separately for each connected display monitor. For instance, if both the display monitors512and514are connected, the renderer program1580erenders a first combined stereoscopic image for the display monitor512and a second combined stereoscopic image for the display monitor514. The renderer program1580eformats the first and second combined stereoscopic images such that they are compatible with the type and/or screen size of the display monitors and/or screen.

In some embodiments, the renderer program1580eselects the image processing mode based on how the display monitor is to display stereoscopic data. Proper interpretation of stereoscopic image data by the brain of an operator requires that the left eye data of the stereoscopic image be conveyed to the operator's left eye and the right eye data of the stereoscopic image be conveyed to the operator's right eye. Generally, display monitors provide a first polarization for left eye data and a second opposing polarization for the right eye data. Thus, the combined stereoscopic image must match the polarization of the display monitor.

FIG. 17shows an example of the display monitor512, according to an example embodiment of the present disclosure. The display monitor512may be, for example, the LG® 55LW5600 three-dimensional television with a screen1702. The example display monitor512uses a polarization film on the screen1702such that all odd rows1704have a first polarization and all even rows1706have an opposing polarization. For compatibility with the display monitor512shown inFIG. 17, the renderer program1580ewould have to select the row-interleaved mode such that the left and right image data are on alternating lines. In some instances, the renderer program1580emay request (or otherwise receive) display characteristics of the display monitor512prior to preparing the stereoscopic image.

To view the stereoscopic image displayed on the screen1702, the surgeon504(remember him fromFIG. 5) wears glasses1712that include a left lens1714that comprises a first polarization that matches the first polarization of the rows1704. In addition, the glasses1712include a right lens1716that comprises a second polarization that matches the second polarization of the rows1706. Thus, the left lens1714only permits a majority of the light from the left image data from the left rows1704to pass through while blocking a majority of the light from the right image data. In addition, the right lens1716permits a majority of the light from the right image data from the right rows1706to pass through while blocking a majority of the light from the left image data. The amount of light from the “wrong” view that reaches each respective eye is known as “crosstalk” and is generally held to a value low enough to permit comfortable viewing. Accordingly, the surgeon504views left image data recorded by the left optical image sensor748in a left eye while viewing right image data recorded by the right optical image sensor746in a right eye. The surgeon's brain fuses the two views together to create a perception of three-dimensional distance and/or depth. Further, the use of such a display monitor is advantageous for observing the accuracy of the stereoscopic visualization camera300. If the surgeon or operator does not wear glasses, then both left and right views are observable with both eyes. If a planar target is placed at the focal plane, the two images will be theoretically aligned. If misalignment is detected, a re-calibration procedure can be initiated by the processor1562.

The example renderer program1580eis configured to render the left and right views for circular polarization. However, in other embodiments, the renderer program1580emay provide a stereoscopic image compatible with linear polarization. Regardless of which type of polarization is used, the example processor1562may execute a program1560to verify or check a polarity of the stereoscopic images being output by the renderer program1580e. To check polarity, the processor1562and/or the peripheral input unit interface1574inserts diagnostic data into the left and/or right images. For example, the processor1562and/or the peripheral input unit interface1574may overlay “left” text onto the left image and “right” text onto the right image. The processor1562and/or the peripheral input unit interface1574may display a prompt instructing an operator to close one eye at a time while wearing the glasses1712to confirm the left view is being received at the left eye and the right view is being received at the right eye. The operator may provide confirmation via the user input device1410indicating whether the polarization is correct. If the polarization is not correct, the example renderer program1580eis configured to reverse locations where the left and right images are inserted into the combined stereoscopic image.

In yet other embodiments, the example renderer program1580eis configured to provide for frame sequential projection instead of creating a combined stereoscopic image. Here, the renderer program1580erenders the left images and or frames time-sequentially interleaved with the right images and/or frames. Accordingly the left and right images are alternately presented to the surgeon504. In these other embodiments, the screen1702is not polarized. Instead, the left and right lenses of the glasses1712may be electronically or optically synchronized to their respective portion of a frame sequence, which provides corresponding left and right views to a user to discern depth.

In some examples, the renderer program1580emay provide certain of the right and left images for display on separate display monitors or separate windows on one display monitor. Such a configuration may be especially beneficial when lenses of right and left optical paths of the optical elements1402are independently adjustable. In an example, a right optical path may be set a first magnification level while a left optical path is set at a second magnification level. The example renderer program1580emay accordingly display a stream of images from the left view on the display monitor512and a stream of images from the right view on the display monitor514. In some instances, the left view may be displayed in a first window on the display monitor512while the right view is displayed in a second window (e.g., a picture-in-picture) of the same display monitor512. Thus, while not stereoscopic, the concurrent display of the left and right images provides useful information to a surgeon.

In another example, the light sources708and the filter740may be switched quickly to generate alternating images with visible light and fluorescent light. The example renderer program1580emay combine the left and right views to provide a stereoscopic display under different lighting sources to highlight, for example, a vein with a dye agent while showing the background in visible light.

In yet another example, a digital zoom may be applied to the right and/or left optical image sensor746or748. Digital zoom generally affects the perceived resolution of the image and is dependent on factors such as the display resolution and the preference of the viewer. For example, the processor1504of the image capture module1404may apply digital zooming by creating interpolated pixels synthesized and interspersed between the digitally-zoomed pixels. The processor1504may operate a program1510that coordinates the selection and interpolation pixels for the optical image sensors746and748. The processor1504transmits the right and left images with digital zoom applied to the information processor module1408for subsequent rendering and display.

In some embodiments, the processor1504receives instructions from the processor1562that a digital zoom image is to be recorded between images without digital zoom to provide a picture-in-picture (or separate window) display of a digital zoom of a region of interest of the target site700. The processor1504accordingly applies digital zooming to every other read from the pixel grids1002and1004. This enables the renderer program1580eto display simultaneously a stereoscopic full resolution image in addition to a digitally-zoomed stereoscopic image. Alternatively, the image to be zoomed digitally is copied from the current image, scaled, and placed during the render phase in the proper position overlaid atop the current image. This alternatively configuration avoids the “alternating” recording requirement.

6. Calibration Example

The example information processor module1408ofFIGS. 14 to 16may be configured to execute one or more calibration programs1560to calibrate, for example, a working distance and/or magnification. For example, the processor1562may send instructions to the motor and lighting module1406to perform a calibration step for mapping a working distance (measured in millimeters) from the main objective assembly702to the target site700to a known motor position of the working distance lens motor1554. The processor1562performs the calibration by sequentially moving an object plane in discrete steps along the optical axis and re-focusing the left and right images, while recording encoder counts and the working distance. In some examples, the working distance may be measured by an external device, which transmits the measured working distance values to the processor1562via the peripheral input unit interface1574and/or an interface to the user input device1410. The processor1562may store the position of the rear working distance lens704(based on position of the working distance lens motor1554) and the corresponding working distance.

The example processor1562may also execute a program1560to perform magnification calibration. The processor1562may set the optical elements1402, using the motor and lighting module1406to select magnification levels. The processor1562may record positions of the optical elements1402, or corresponding motor positions with respect to each magnification level. The magnification level may be determined by measuring a height in an image of an object of a known size. For example, the processor1562may measure an object as having a height of 10 pixels and use a look-up-table to determine that a 10 pixel height corresponds to a 5× magnification.

To match the stereoscopic perspectives of two different imaging modalities it is often desirable to model them both as if they are simple pinhole cameras. The perspective of a 3D computer model, such as a MRI brain tumor, can be viewed from user-adjustable directions and distances (e.g. as if the images are recorded by a synthesized stereoscopic camera). The adjustability can be used to match the perspective of the live surgical image, which must therefore be known. The example processor1562may calibrate one or more of these pinhole camera parameters such as, for example, a center of projection (“COP”) of the right and left optical image sensors746and748. To determine center of projection, the processor1562determines a focus distance from the center of projection to an object plane. First, the processor1562sets the optical elements1402at a magnification level. The processor1562then records measurements of a height of an image at three different distances along the optical axis including at the object plane, a distance d less than the object plane distance, and a distance d greater than the object plane distance. The processor1562uses an algebraic formula for similar triangles at the two most extreme positions to determine the focus distance to the center of projection. The processor1562may determine focus distances at other magnifications using the same method or by determining a ratio between the magnifications used for calibration. The processor may use a center of projection to match the perspective of an image of a desired fusion object, such as an MRI tumor model, to a live stereoscopic surgical image. Additionally or alternatively, existing camera calibration procedures such as OpenCV calibrateCamera may be used to find the above-described parameters as well as additional camera information such as a distortion model for the optical elements1402.

The example processor1562may further calibrate the left and right optical axes. The processor1562determines an interpupillary distance between the left and right optical axes for calibration. To determine the interpupillary distance, the example processor1562records left and right images where pixel sets1006and1008are centered at the pixel grids1002and1004. The processor1562determines locations of ZRPs (and/or distances to a displaced object) for the left and right images, which are indicative of image misalignment and degree of parallax. In addition, the processor1562scales the parallax and/or the distance based on the magnification level. The processor1562then determines the interpupillary distance using a triangulation calculation taking into account the degree of parallax and/or the scaled distance to the object in the display. The processor1562next associates the interpupillary distance with the optical axis at the specified magnification level as a calibration point.

VI. Image Alignment and Spurious Parallax Adjustment Embodiment

Similar to human vision, stereoscopic images comprise right views and left views that converge at a point of interest. The right and left views are recorded at slightly different angles from the point of interest, which results in parallax between the two views. Items in the scene in front of or behind the point of interest exhibit parallax such that distance or depth of the items from the viewer can be deduced. The accuracy of the perceived distance is dependent on, for example, the clarity of the viewer's eyesight. Most humans exhibit some level of imperfection in their eyesight, resulting in some inaccuracies between the right and left views. However, they are still able to achieve stereopsis, with the brain fusing the views with some level of accuracy.

When left and right images are recorded by a camera instead of being viewed by a human, the parallax between the combined images on a display screen produces stereopsis, which provides an appearance of a three-dimensional stereoscopic image on a two-dimensional display. Errors in the parallax can affect the quality of the three-dimensional stereoscopic image. The inaccuracy of the observed parallax in comparison to a theoretically perfect parallax is known as spurious parallax. Unlike humans, cameras do not have brains that automatically compensate for the inaccuracies.

If spurious parallax becomes significant, the three-dimensional stereoscopic image may be unviewable to the point of inducing vertigo, headaches, and nausea. There are many factors that can affect the parallax in a microscope and/or camera. For instance, optical channels of the right and left views may not be exactly equal. The optical channels may have unmatched focus, magnification, and/or misalignment of points of interest. These issues may have varying severity at different magnifications and/or working distances, thereby reducing efforts to correct through calibration.

Known surgical microscopes, such as the surgical microscope200ofFIG. 2are configured to provide an adequate view through the oculars206. Often, the image quality of optical elements of known surgical microscopes is not sufficient for stereoscopic cameras. The reason for this is because manufacturers of surgical microscopes assume the primary viewing is through oculars. Any camera attachment (such as the camera212) is either monoscopic and not subject to spurious parallax or stereoscopic with low image resolution where spurious parallax is not as apparent.

International standards, such as ISO 10936-1:2000, Optics and optical instruments—Operation microscopes—Part 1: Requirements and test methods, have been developed to provide specification limits for image quality of surgical microscopes. The specification limits are generally set for viewing through the oculars of a surgical microscope and do not consider three-dimensional stereoscopic display. For example, regarding spurious parallax, ISO 10936-1:2000 specifies that the difference in vertical axes between the left and right views should be less than 15 arc-minutes. Small angular deviations of axes are often quantified in arc-minutes, which corresponds to 1/60thof a degree, or arc-seconds, which corresponds to 1/60thof an arc-minute. The 15 arc-minute specification limit corresponds to a 3% difference between left and right views for a typical surgical microscope with a working distance of 250 mm and a field-of-view of 35 mm (which has an angular field-of-view of 8°).

The 3% difference is acceptable for ocular viewing where a surgeon's brain is able to overcome the small degree of error. However, this 3% difference produces noticeable differences between left and right views when viewed stereoscopically on a display monitor. For example, when the left and right views are shown together, a 3% difference results in an image that appears disjointed and difficult to view for extended periods of time.

Another issue is that known surgical microscopes may satisfy the 15 arc-minute specification limit at only one or a few magnification levels and/or only individual optical elements may satisfy a certain specification limit. For example, individual lenses are manufactured to meet certain criteria. However, when the individual optical elements are combined in an optical path, small deviations from the standard may be amplified rather than cancelled. This can be especially pronounced when five or more optical elements are used in an optical path including a common main objective lens. In addition, it is very difficult to perfectly match optical elements on parallel channels. At most, during manufacture, the optical elements of a surgical microscope are calibrated only at one or a few certain magnification levels to meet the 15 arc-minute specification limit. Accordingly, the error may be greater between the calibration points despite the surgical microscope allegedly meeting the ISO 10936-1:2000 specifications.

In addition, the ISO 10936-1:2000 specification permits larger tolerances when additional components are added. For example, adding second oculars (e.g., the oculars208) increases the spurious parallax by 2 arc-minutes. Again, while this error may be acceptable for viewing through oculars206and208, image misalignment becomes more pronounced when viewed stereoscopically through the camera.

In comparison to known surgical microscopes, the example stereoscopic visualization camera300disclosed herein is configured to automatically adjust at least some of the optical elements1402to reduce or eliminate spurious parallax. Embedding the optical elements within the stereoscopic visualization camera300enables fine adjustments to be made automatically (sometimes in real-time) for three-dimensional stereoscopic display. In some embodiments, the example stereoscopic visualization camera300may provide an accuracy of 20 to 40 arc-seconds, which is close to a 97% reduction in optical error compared to the 15 arc-minute accuracy of known surgical microscopes.

The improvement in accuracy enables the example stereoscopic visualization camera300to provide features that are not capable of being performed with known stereoscopic microscopes. For example, many new microsurgical procedures rely on accurate measurements in a live surgical site for optimal sizing, positioning, matching, directing, and diagnosing. This includes determining a size of a vessel, an angle of placement of a toric Intra Ocular Lens (“IOL”), a matching of vasculature from a pre-operative image to a live view, a depth of a tumor below an artery, etc. The example stereoscopic visualization camera300accordingly enables precise measurements to be made using, for example, graphical overlays or image analysis to determine sizes of anatomical structures.

Known surgical microscopes require that a surgeon place an object of a known size (such as a micro-ruler) into the field-of-view. The surgeon compares the size of the object to surrounding anatomical structure to determine an approximate size. However, this procedure is relatively slow since the surgeon has to place the object in the proper location, and then remove it after the measurement is performed. In addition, the measurement only provides an approximation since the size is based on the surgeon's subjective comparison and measurement. Some known stereoscopic cameras provide graphical overlays to determine size. However, the accuracy of these overlays is reduced if spurious parallax exists between the left and right views.

A. ZRP as a Source of Spurious Parallax

ZRP inaccuracy provides a significant source of error between left and right images resulting in spurious parallax. ZRP, or zoom repeat point, refers to a point in a field-of-view that remains in a same location as a magnification level is changed.FIGS. 18 and 19show examples of ZRP in a left and right field-of-view for different magnification levels. Specifically,FIG. 18shows a left field-of-view1800for a low magnification level and a left field-of-view1850for a high magnification level In addition,FIG. 19shows a right field-of-view1900for a low magnification level and a right field-of-view1950for a high magnification level.

It should be noted thatFIGS. 18 and 19show crosshairs1802and1902to provide an exemplary point of reference for this disclosure. The crosshairs1802include a first crosshair1802apositioned along a y-direction or y-axis and a second crosshair1802bpositioned along an x-direction or x-axis. Additionally, crosshairs1902include a first crosshair1902apositioned along a y-direction or y-axis and a second crosshair1902bpositioned along an x-direction or x-axis In actual implementation, the example stereoscopic visualization camera300by default typically does not include or add crosshairs to the optical path unless requested by an operator.

Ideally, the ZRP should be positioned at a central location or origin point. For example, the ZRP should be centered in the crosshairs1802and1902. However, inaccuracies in the optical elements1402and/or slight misalignments between the optical elements1402cause the ZRP to be located away from the center of the crosshairs1802and1902. The degree of spurious parallax corresponds to how far each of the ZRPs of the left and right views is located away from the respective centers in addition to ZRPs being misaligned between the left and right views. Moreover, inaccuracies in the optical elements1402may cause the ZRP to drift slightly as magnification changes, thereby further causing a greater degree of spurious parallax.

FIG. 18shows three crescent-shaped objects1804,1806, and1808in the field-of-views1800and1850of the target site700ofFIG. 7. It should be appreciated that the field-of-views1800and1850are linear field-of-views with respect to the optical image sensors746and748. The objects1804,1806, and1808were placed in the field-of-view1800to illustrate how spurious parallax is generated from left and right image misalignment. The object1804is positioned above crosshair1802balong crosshair1802a. The object1806is positioned along crosshair1802band to the left of the crosshair1802a. The object1808is positioned slightly below the crosshair1802band to the right of the crosshair1802a. A ZRP1810for the left field-of-view1800is positioned in a notch of the object1808.

The left field-of-view1800is changed to the left field-of-view1850by increasing the magnification level (e.g., zooming) using the zoom lens assembly716of the example stereoscopic visualization camera300. Increasing the magnification causes the objects1804,1806, and1808to appear to expand or grow, as shown in the field-of-view1850. In the illustrated example, the field-of-view1850is approximately 3× the magnification level of the field-of-view1800.

Compared to the low magnification field-of-view1800, the objects1804,1806, and1808in high magnification field-of-view1850have increased in size by about 3× while also moving apart from each other by 3× with respect to the ZRP1810. In addition, the positions of the objects1804,1806, and1808have moved relative to the crosshairs1802. The object1804is now shifted to the left of the crosshair1802aand shifted slightly further from the crosshair1802b. In addition, the object1806is now shifted further to the left of crosshair1802aand slightly above the crosshair1802b. Generally, the object1808is located in the same (or nearly the same) position with respect to the crosshairs1802, with the ZRP1810being located in the exact same (or nearly the same) position with respect to the crosshairs1802and the object1806. In other words, as magnification increases, the objects1804,1806, and1808(and anything else in the field-of-view1850) appear to move away and outward from the ZRP1810.

The same objects1804,1806, and1808are shown in the right field-of-views1900and1950illustrated inFIG. 19. However, the location of the ZRP is different. Specifically, ZRP1910is located above crosshair1902band to the left of crosshair1902ain the right field-of-views1900and1950. Thus, the ZRP1910is located at a different location than the ZRP1810in the left field-of-views1800and1850. In the illustrated example, it is assumed that the left and right optical paths are perfectly aligned at the first magnification level. Accordingly, the objects1804,1806, and1808shown in the right field-of-view1900in the same location as the same objects1804,1806, and1808in the left field-of-view1800. Since the left and right views are aligned, no spurious parallax exists.

However, in the high magnification field-of-view1950, the objects1804,1806, and1808expand and move away from the ZRP1910. Given the location of the ZRP1910, the object1804moves or shifts to the right and the object1806moves or shifts downward. In addition, the object1808moves downward and to the right compared to its location in the field-of-view1900.

FIG. 20shows a pixel diagram comparing the high magnification left field-of-view1850to the high magnification right field-of-view. A grid2000may represent locations of the objects1804(L),1806(L), and1808(L) on the pixel grid1004of the left optical image sensor748overlaid with locations of the objects1804(R),1806(R), and1808(R) on the pixel grid1002of the left optical image sensor746.FIG. 20clearly shows that the objects1804,1806, and1808are in different positions for the left and right field-of-views1850and1950. For example, the object1804(R) is located to the right of crosshair1902aand above crosshair1902bwhile the same object1804(L) is located to the left of cross hair1802aand further above cross hair1802b.

The difference in positions of the objects1804,1806, and1808corresponds to spurious parallax, which is created by deficiencies in the optical alignment of the optical elements1402that produce ZRPs1810and1910in different locations. Assuming no distortion or other imaging errors, the spurious parallax shown inFIG. 20is generally the same for all points within the image. When viewed through oculars of a surgical microscope (such as microscope200ofFIG. 2), the difference in location of the objects1804,1806, and1808may not be noticeable. However, when viewed on the display monitors512and514in a stereoscopic image, the differences become readily apparent and can result in headaches, nausea, and/or vertigo.

FIG. 21shows a diagram illustrative of spurious parallax with respect to left and right ZRPs. The diagram includes a pixel grid2100that includes overlays of the right and left pixel grids1002and1004ofFIG. 10. In this illustrated example, a left ZRP2102for the left optical path is located at +4 along the x-axis and 0 along the y-axis. In addition, a right ZRP2104for the right optical path is located at −1 along the x-axis and 0 along the y-axis. An origin2106is shown at the intersection of the x-axis and the y-axis.

In this example, object2108is aligned with respect to the left and right images at a first low magnification. As magnification is increased by 3×, the object2108increased in size and moved away from the ZRPs2102and2104. Outlines object2110shows a theoretical location of the object2108at the second higher magnification based on the ZRPs2102and2104being aligned with the origin2106. Specifically, a notch of the object2108at the first magnification level is at location +2 along the x-axis. With 3× magnification, the notch moves 3× along the x-axis such that the notch is located at +6 along the x-axis at the higher magnification level. In addition, since the ZRPs2102and2104would be theoretically aligned at the origin2106, the object2110would be aligned between the left and right views (shown inFIG. 21as a single object given the overlay).

However, in this example, misalignment of the left and right ZRPs2102and2104causes the object2110to be misaligned between the left and right views at higher magnification. Regarding the right optical path, the right ZRP2104is located at −1 along the x-axis such that it is 3 pixels away from the notch of the object2108at low magnification. When magnified 3×, this difference becomes 9 pixels, which is shown as object2110(R). Similarly, the left ZRP2102is located at +4 pixels along the x-axis. At 3× magnification, the object2108moves from being 2 pixels away to 6 pixels away, which is shown as object2110(L) at −2 along the x-axis.

The difference in positions of the object2110(L) and the object2110(R) corresponds to the spurious parallax between the left and right views at the higher magnification. If the right and left views were combined into a stereoscopic image for display, the location of the object2110would be misaligned at each row if the renderer program1850euses a row-interleaved mode. The misalignment would be detrimental to generating stereopsis and may produce an image that appears blurred or confusing to an operator.

B. Other Sources of Spurious Parallax

While ZRP misalignment between left and right optical paths is a significant source of spurious parallax, other sources of error also exist. For example, spurious parallax may result from non-equal magnification changes between the right and left optical paths. Differences in magnification between parallel optical paths may result from slight variances in the optical properties or characteristics of the lenses of the optical elements1402. Further, slight differences may result from positioning if each of the left and right front zoom lenses726and728and each of the left and right rear zoom lenses736and738ofFIGS. 7 and 8are independently controlled.

Referring back toFIGS. 18 and 19, differences in magnification change produce differently sized objects and different spacing between the objects for the left and right optical paths. If, for example, the left optical path has a higher magnification change, then the objects1804,1806, and1808will appear larger and move a greater distance from the ZRP1810compared to the objects1804,1806, and1808in the right field-of-view1950inFIG. 19. The difference in the location of the objects1804,1806, and1808, even if the ZRPs1810and1910are aligned, results in spurious parallax.

Another source of spurious parallax results from unequal focusing of the left and right optical paths. Generally, any difference in focus between left and right views may cause a perceived diminishment in image quality and potential confusion over whether the left or right view should predominate. If the focus difference is noticeable, it can result in an Out-Of-Focus (“OOF”) condition. OOF conditions are especially noticeable in stereoscopic images where left and right views are shown in the same image. In addition, OOF conditions are not easily correctable since re-focusing an out-of-focus optical path usually results in the other optical path becoming unfocused. Generally, a point needs to be determined where both optical paths are in focus, which may include changing positions of left and right lenses along an optical path and/or adjusting a working distance from the target site700.

FIG. 22shows a diagram illustrative of how an OOF condition develops. The diagram relates perceived resolution (e.g., focus) to a lens position relative to an optimal resolution section2202. In this example the left rear zoom lens734is at position L1while the right rear zoom lens732is at position R1. At position L1and R1, the rear zoom lenses732and734are in a range of optimal resolution2202such that the left and right optical paths have matched focus levels. However, there is a difference in the positions of L1and R1, corresponding to distance ΔP. At a later time, the working distance706is changed such that a point is out-of-focus. In this example, both rear zoom lenses732and734move the same distance to locations L2and R2such that distance ΔP does not change. However, the position change results in a significant change in resolution ΔR such that the left rear zoom lens734has a higher resolution (e.g., better focus) that the right rear zoom lens732. The resolution ΔR corresponds to the OOF condition, which results in spurious parallax from misalignment of focus between the right and left optical paths.

Yet another source of spurious parallax can result from imaging objects that are moving at the target site700. The spurious parallax results from small synchronization errors between exposures of the right and left optical image sensors746and748. If the left and right views are not recorded simultaneously, then the object appears to be displaced or misaligned between the two views. The combined stereoscopic image shows the same object at two different locations for the left and right views.

Moreover, another source of spurious parallax involves a moving ZRP point during magnification. The examples discussed above in Section IV(A) assume that the ZRPs of the left and right views do not move in the x-direction or the y-direction. However, the ZRPs may shift during magnification if the zoom lenses726,728,732, and/or734do not move exactly parallel with the optical path or axis (e.g., in the z-direction). As discussed above in reference toFIG. 11, the carrier724may shift or rotate slightly when a force is applied to the actuation section1108. This rotation may cause the left and right ZRPs to move slightly when a magnification level is changed.

In an example, during a magnification change, the carrier730moves in a single direction while the carrier724moves in the same direction for a portion of the magnification change and in an opposite direction for a remaining portion of the magnification change for focus adjustment. If the axis of motion of the carrier724is tilted or rotated slightly with respect to the optical axis, the ZRP of the left and/or right optical paths will shift in one direction for the first portion followed by a shift in a reverse direction for the second portion of the magnification change. In addition, since the force is applied unequally, the right and left front zoom lenses726and728may experience varying degrees of ZRP shift between the left and right optical paths. Altogether, the change in position of the ZRP results in misaligned optical paths, thereby producing spurious parallax.

C. Reduction in Spurious Parallax Facilitates Incorporating Digital Graphics and Images with a Stereoscopic View

As surgical microscopes become more digitalized, designers are adding features that overlay graphics, images, and/or other digital effects to the live-view image. For example, guidance overlays, fusion of stereoscopic Magnetic Resonance Imaging (“MRI”) images, and/or external data may be combined with images recorded by a camera, or even displayed within oculars themselves. Spurious parallax reduces the accuracy of the overlay with the underlying stereoscopic image. Surgeons generally require, for example, that a tumor visualized via MRI be placed as accurately as possible, often in three dimensions, within a fused live surgical stereoscopic view. Otherwise, the preoperative tumor image provides little information to the surgeon, thereby detracting from the performance.

For example, a surgical guide may be aligned with a right view image while misaligned with the left view. The misaligned surgical guide between the two views is readily apparent to the operator. In another example, a surgical guide may be aligned separately with left and right views in the information processor module1408prior to the graphics processing unit1564creating the combined stereoscopic image. However, misalignment between the left and right views creates misalignment between the guides, thereby reducing the effectiveness of the guides and creating confusion and delay during the microsurgical procedure.

U.S. Pat. No. 9,552,660, titled “IMAGING SYSTEM AND METHODS DISPLAYING A FUSED MULTIDIMENSIONAL RECONSTRUCTED IMAGE,” (incorporated herein by reference) discloses how preoperative images and/or graphics are visually fused with a stereoscopic image.FIGS. 23 and 24show diagrams that illustrate how spurious parallax causes digital graphics and/or images to lose accuracy when fused to a stereoscopic image.FIG. 24shows a front view of a patient's eye2402andFIG. 23shows a cross-sectional view of the eye along plane A-A ofFIG. 24. InFIG. 23, the information processor module1408is instructed to determine a caudal distance d from a focus plane2302to, for example, an object of interest2304on a posterior capsule of the eye2402. The information processor module1408operates a program1560that specifies, for example, that the distance d is determined by a triangulation calculation of image data from the left and right views of the eye2402. A view2306is shown from a perspective of the left optical image sensor748and a view2308is shown from a perspective of the right optical image sensor746. The left and right views2306and2308are assumed to be coincident with an anterior center2310of the eye2402. In addition, the left and right views2306and2308are two-dimensional views of the object2304projected onto a focal plane2302as theoretical right projection2312and theoretical left projection2314. In this example, processor1562determines the distance d to the object of interest2304by calculating an intersection of an extrapolation of the theoretical right projection2312and an extrapolation of the theoretical left projection2314using a triangulation routine.

However, in this example spurious parallax exists, which causes an actual left projection2316to be located to the left of the theoretical left projection2314by a distance P, as shown inFIGS. 23 and 24. The processor1562uses the actual left projection2316and the right projection2312to determine a distance to an intersection2320of an extrapolation of the right projection2312and an extrapolation of the actual left projection2316using the triangulation routine. The distance of the intersection point2320is equal to the distance d plus an error distance e. The spurious parallax accordingly results in an erroneous distance calculation using data taken from a stereoscopic image. As shown inFIGS. 23 and 24, even a small degree of spurious parallax may create a significant error. In the context of a fused image, the erroneous distance may result in an inaccurate placement of a tumor three-dimensional visualization for fusion with a stereoscopic image. The inaccurate placement may delay the surgery, hinder the performance of the surgeon, or cause the entire visualization system to be disregarded. Worse yet, a surgeon may rely on the inaccurate placement of the tumor image and make a mistake during the microsurgery procedure.

D. The Example Stereoscopic Visualization Camera Reduces or Eliminates Spurious Parallax

The example stereoscopic visualization camera300ofFIGS. 3 to 16is configured to reduce or eliminate visual defects, spurious parallax, and/or misaligned optical paths that typically result in spurious parallax. In some examples, the stereoscopic visualization camera300reduces or eliminates spurious parallax by aligning ZRPs of the left and right optical paths to the respective centers of pixel sets1006and1008of the right and left optical image sensors746and748. Additionally or alternatively, the stereoscopic visualization camera300may align the optical paths of the left and right images. It should be appreciated that the stereoscopic visualization camera300may perform actions to reduce spurious parallax during calibration. Additionally, the stereoscopic visualization camera300may reduce detected spurious parallax in real-time during use.

FIGS. 25 and 26illustrate a flow diagram showing an example procedure2500to reduce or eliminate spurious parallax, according to an example embodiment of the present disclosure. Although the procedure2500is described with reference to the flow diagram illustrated inFIGS. 25 and 26, it should be appreciated that many other methods of performing the steps associated with the procedure2500may be used. For example, the order of many of the blocks may be changed, certain blocks may be combined with other blocks, and many of the blocks described are optional. Further, the actions described in procedure2500may be performed among multiple devices including, for example the optical elements1402, the image capture module1404, the motor and lighting module1406, and/or the information processor module1408of the example stereoscopic visualization camera300. For example, the procedure2500may be performed by one of the programs1560of the information processor module1408.

The example procedure2500begins when the stereoscopic visualization camera300receives an instruction to align right and left optical paths (block2502). The instructions may be received from the user input device1410in response to an operator requesting that the stereoscopic visualization camera300perform a calibration routine. In other examples, the instructions may be received from the information processor module1408after determining right and left images are misaligned. The information processor module1408may determine images are not aligned by executing a program1560that overlays right and left images and determines differences in pixel values, where greater differences over large areas of pixels are indicative of misaligned images. In some examples, the program1560may compare the pixel data of the left and right images without performing an overlay function, where, for example, left pixel data is subtracted from right pixel data to determine a severity of misalignment.

After receiving instructions to reduce spurious parallax, the example stereoscopic visualization camera300locates a ZRP of one of the left or right optical path. For illustrative purposes, procedure2500includes the ZRP of the left optical path being determined first. However, in other embodiments, the procedure2500may determine the ZRP of the right optical path first. To determine the left ZRP, the stereoscopic visualization camera300moves at least one zoom lens (e.g., the left front zoom lens728and/or the left rear zoom lens734) to a first magnification level along a z-direction of the left optical path (block2504). In instances where the front zoom lenses726and728are connected to the same carrier724and the rear zoom lenses732and734are connected to the same carrier730, the movement of the left lenses causes the right lenses to also move. However, only movement of the left lenses is considered during this section of the procedure2500.

At the first magnification level, the stereoscopic visualization camera300causes the left zoom lens to move along the z-direction (block2506). The movement may include, for example, back-and-forth movement around the first magnification level. For example, if the first magnification level is 5×, the movement may be between 4× and 6×. The movement may also include movement in one direction, such as from 5× to 4×. During this movement, the stereoscopic visualization camera300may adjust one or more other lenses to maintain focus of the target site700. At block2508, during the movement of the left zoom lens, the stereoscopic visualization camera300records a stream or a sequence of images and/or frames2509of the target site700using, for example, the left optical image sensor748. The images2509are recorded using an oversized pixel set1008configured to encompass an origin of the pixel grid1004and potential locations of the left ZRP.

The example processor1562of the information processor module1408analyzes the image stream to locate a portion of area that does not move in an x-direction or a y-direction between the images (block2510). The portion of the area may include one or a few pixels and corresponds to the left ZRP. As discussed above, during a magnification change, objects move away from the ZRP or move towards the ZRP. Only objects at the ZRP remain constant in position with respect to the field-of-view as magnification changes. The processor1562may calculate deltas between the stream of images for each pixel using pixel data. An area with the smallest delta across the image stream corresponds to the left ZRP.

The example processor1562of the information processor module1408next determines coordinates of a portion of the area that does not move between the image stream (e.g., determines a location of the left ZRP) with respect to the pixel grid1004(block2512). In other examples, the processor1562of the information processor module1408determines a distance between the origin and the portion of the area corresponding to the left ZRP. The distance is used to determine a position of the left ZRP on the pixel grid1004. Once the location of the left ZRP is determined, the processor1562of the information processor module1408determines a pixel set (e.g., the pixel set1008) for the left optical image sensor748such that the left ZRP is located at a center (within one pixel) of the pixel set (block2514). At this point, the left ZRP is centered within the left optical path.

In some examples, blocks2504to2514may be performed iteratively by re-selecting the pixel set until the left ZRP is within a pixel of the origin and spurious parallax is minimized. After the pixel grid is determined, the processor1562of the information processor module1408stores at least one of coordinates of the pixel set and/or coordinates of the left ZRP to the memory1570as a calibration point (block2516). The processor1562of the information processor module1408may associate the first magnification level with the calibration point such that the same pixel set is selected when the stereoscopic visualization camera300returns to the first magnification level.

FIG. 27shows a diagram illustrative of how the left ZRP is adjusted with respect to the pixel grid of the left optical image sensor748. Initially, an initial (e.g., oversized) pixel set2702is selected, which is centered on origin2704. The pixel set2702is large enough to record potential ZRPs in the image stream. In this illustrated example, a left ZRP2706is located above and to the right of the origin2704. The processor1562of the information processor module1408determines pixel set2708based on a location of the left ZRP2706such that the left ZRP2706is located or positioned at a center of the pixel set2708.

After the left ZRP is determined and aligned with an origin of a pixel set inFIG. 25, the example procedure2500aligns the left and right images inFIG. 26. To align the images, the example processor1562compares pixel data from left and right images recorded after the left ZRP is aligned with the origin. In some embodiments, the processor1562overlays the left and right images to determine differences using, for example, a subtraction and/or template method. The processor1562selects or determines a pixel set for the right optical path such that the resulting right images align or coincide with the left images (block2519).

The example processor1562, in the illustrated embodiment, determines the right ZRP. The steps are similar to steps discussed in blocks2504to2512for the left ZRP. For example, at block2518the stereoscopic visualization camera300moves a right zoom lens to the first magnification level. In some embodiments, the magnification level for the right lens is different than the magnification level used for determining the left ZRP. The example processor1562of the information processor module1408then moves the right zoom lens around the magnification level and receives a stream of images2521from the right optical image sensor746during the movement (blocks2520and2522). The example processor1562of the information processor module1408determines the right ZRP from the right stream of images by locating a portion of an area that does not move between the images (block2524). The processor1562next determines coordinates of the right ZRP and/or a distance between a center of an aligned pixel set1006to the right ZRP (block2526).

The processor1562then instructs the motor and lighting module1406to move at least one lens in the right optical path in at least one of an x-direction, a y-direction, and/or a tilt-direction to align the right ZRP with the center of the aligned pixel set1006using, for example, the distance or coordinates of the right ZRP (block2528). In other words, the right ZRP is moved to coincide with the center of the aligned pixel set1006. In some examples, the right front lens720, the right lens barrel736, the right final optical element745, and/or the right image sensor746is moved (using for example a flexure) in the x-direction, the y-direction and/or a tilt-direction with respect to the z-direction of the right optical path. The degree of movement is proportional to the distance of the right ZRP from the center of the pixel set1006. In some embodiments, the processor1562digitally changes properties of the right front lens720, the right lens barrel736, and/or the right final optical element745to have the same effect as moving the lenses. The processor1562may repeat steps2520to2528and/or use subsequent right images to confirm the right ZRP is aligned with the center of the pixel set1006and/or to iteratively determine further lens movements needed to align the right ZRP with the center of the pixel set.

The example processor1562stores coordinates of the right pixel set and/or the right ZRP to the memory1570as a calibration point (block2530). The processor1562may also store to the calibration point a position of the right lens that was moved to align the right ZRP. In some examples, the calibration point for the right optical path is stored with the calibration point for the left optical path in conjunction with the first magnification level. Thus, the processor1562applies the data within the calibration point to the optical image sensors746and748and/or radial positioning of one or more optical elements1402when the stereoscopic visualization camera300is subsequently set to the first magnification level.

In some examples, the procedure2500may be repeated for different magnification levels and/or working distances. Accordingly, the processor1562determines if ZRP calibration is needed for another magnification level or working distance (block2532). If another magnification level is to be selected, the procedure2500returns to block2504inFIG. 25. However, if another magnification level is not needed, the example procedure ends.

Each of the calibration points may be stored in a look-up-table. Each row in the table may correspond to a different magnification level and/or working distance. Columns in the look-up-table may provide coordinates for the left ZRP, the right ZRP, the left pixel set, and/or the right pixel set. In addition, one or more columns may specify relevant positions (e.g., radial, rotational, tilt, and/or axial positions) of the lenses of the optical elements1402to achieve focus at the magnification level in addition to aligned right and left images.

The procedure2500accordingly results in the right ZRP and the left ZRP in addition to views of the target site to be aligned to pixel grids of the respective optical image sensors746and748as well as to each other in a three-dimensional stereoscopic image. In some instances, the left and right images and the corresponding ZRPs have an accuracy and alignment to within one pixel. Such accuracy may be observable on the display514or514by overlaying left and right views (e.g., images from the left and right optical paths) and observing both views with both eyes, rather than stereoscopically.

It should be appreciated that in some examples, a right pixel set is first selected such that the right ZRP is aligned with or coincident with an origin of the pixel set. Then, the right and left optical images may be aligned by moving one or more right and/or left lenses of the optical elements1402. This alternative procedure still provides right and left ZRPs that are centered and aligned between each other and with respect to the optical image sensors746and748.

The procedure2500ultimately reduces or eliminates spurious parallax in the stereoscopic visualization camera300throughout a full optical magnification range by ensuring left and right ZRPs remain aligned and the right and left images remain aligned. In other words, the dual optics of the right and left optical images sensors746and748are aligned such that parallax at a center of an image between the left and right optical paths is approximately zero at the focal plane. Additionally, the example stereoscopic visualization camera300is par focal across the magnification range, and par central across magnification and working distance ranges since the ZRP of each optical path has been aligned to a center of the respective pixel set. Accordingly, changing only the magnification will maintain a focus of the target site700in both optical image sensors746and748while being trained on the same center point.

The above procedure2500may be performed at calibration before a surgical procedure is performed and/or upon request by an operator. The example procedure2500may also be performed prior to image registration with a pre-operative microsurgical image and/or surgical guidance graphics. Further, the example procedure2500may be performed in real-time automatically during operation of the stereoscopic visualization camera300.

1. Template Matching Example

In some embodiments, the example processor1562of the information processor module1408is configured to use a program1560in conjunction with one or more templates to determine a position of the right ZRP and/or the left ZRP.FIG. 28shows a diagram illustrative of how the processor1562uses a target template2802to determine a location of a left ZRP. In this example,FIG. 28shows a first left image including the template2802aligned with an origin2804or center of the left pixel grid1004of the left optical image sensor748. The template2802may be aligned by moving the stereoscopic visualization camera300to the appropriate location. Alternatively, the template2802may be moved at the target site700until aligned. In other examples, the template2802may include another pattern that does not need alignment with a center of the pixel grid1004. For example, the template may include a graphical wave pattern, a graphical spirograph pattern, a view of a surgical site of a patient and/or a grid having visually distinguishable features with some degree of non-periodicity in both the x and y-directions. The template is configured to prevent a subset of a periodic image from being perfectly aligned onto the larger image in a plurality of locations, which makes such templates unsuitable for matching. A template image that is suitable for template matching is known as a “template match-able” template image.

The template2802shown inFIG. 28is imaged at a first magnification level. A left ZRP2806is shown with respect to the template2802. The ZRP2806has coordinates of Lx, Lywith respect to the origin2804. However, at this point in time, the processor1562has not yet identified the left ZRP2806.

To locate the ZRP2806, the processor1562causes a left zoom lens (e.g., the left front zoom lens728and/or the left rear zoom lens734) to change magnification from the first magnification level to a second magnification level, specifically in this example, from 1× to 2×.FIG. 29shows a diagram of a second left image including the target2802on the pixel grid1004with the magnification level doubled. From the first magnification level to the second magnification level, portions of the target2802increase in size and expand uniformly away from the left ZRP2806, which remains stationary with respect to the first and second images. In addition, a distance between the origin2804of the pixel grid1004and the left ZRP2806remains the same.

The example processor1562synthesizes a digital template image3000from the second image shown inFIG. 29. To create the digital template image, the processor1562copies the second image shown inFIG. 29and scales the copied image by the reciprocal of the magnification change from the first to the second magnification. For example, if the magnification change from the first image to the second image was by a factor of 2, then the second image is scaled by ½.FIG. 30shows a diagram of the digital template image3000, which includes the template2802. The template2802in the digital template image3000ofFIG. 30is scaled to be the same size as the template2802in the first left image shown inFIG. 28.

The example processor1562uses the digital template image3000to locate the left ZRP2806.FIG. 31shows a diagram that shows the digital template image3000superimposed on top of the first left image (or a subsequent left image recorded at the first magnification level) recorded in the pixel grid1004. The combination of the digital template image3000with the first left image produces a resultant view, as illustrated inFIG. 31. Initially the digital template image3000is centered at the origin2804of the pixel grid1004.

The example processor1562compares the digital template image3000to the underlying template2802to determine if they are aligned or matched. The example processor1562then moves the digital template image3000one or more pixels either horizontally or vertically and performs another comparison. The processor1562iteratively moves the digital template image3000compiling a matrix of metrics for each location regarding how close the digital template image3000matches the underlying template2802. The processor1562selects the location in the matrix corresponding to the best matching metric. In some examples, the processor1562uses the OpenCV™ Template Match function.

FIG. 32shows a diagram with the digital template image3000aligned with the template2802. The distance that the digital template image3000was moved to achieve optimal matching is shown as Δx and Δy. Knowing the digital template image3000was synthesized at a scale of M1/M2 (the first magnification level divided by the second magnification level), the processor1562determines the coordinates (Lx, Ly) of the left ZRP2806using Equations (1) and (2) below.
Lx=Δx/(M1/M2)  Equation (1)
Ly=Δy/(M1/M2)  Equation (2)

After the coordinates (Lx, Ly) of the left ZRP2806are determined, the example processor1562selects or determines a pixel subset with an origin that is aligned or coincides with the left ZRP2806, as discussed above in conjunction with procedure2500ofFIGS. 25 and 26. In some embodiments, the processor1562may use template matching iteratively to converge on a highly accurate ZRP position and/or pixel subset. Further, while the above example discussed locating the left ZRP, the same template matching procedure can be used to locate the right ZRP.

In some embodiments, the above-described template matching program1560may be used to align the left and right images. In these embodiments, left and right images are recorded at a magnification level. Both the images may include, for example, the target template2802ofFIG. 28. A portion of the right image is selected and overlaid with the left image. The portion of the right image is then shifted around the left image by one or more pixels horizontally and/or vertically. The example processor1562performs a comparison at each location of the portion of the right image to determine how close a match exists with the left image. Once an optimal location is determined, a pixel set1006of the right pixel grid1002is determined such that the right image is generally coincident with the left image. The location of the pixel set1006may be determined based on how much the portion of the right image was moved to coincide with the left image. Specifically, the processor1562uses an amount of movement in the x-direction, the y-direction, and/or the tilt-direction to determine corresponding coordinates for the right pixel set1006.

2. Right and Left Image Alignment Example

In some embodiments, the example processor1562of the information processor module1408ofFIGS. 14 to 16displays an overlay of right and left images on the display monitor512and/or514. The processor1562is configured to receive user feedback for aligning the right and left images. In this example each pixel data for the right and left images is precisely mapped to a respective pixel of the display monitor512using, for example, the graphics processing unit1564. The display of overlaid left and right images makes any spurious parallax readily apparent to an operator. Generally, with no spurious parallax, the left and right images should almost exactly align.

If an operator detects spurious parallax, the operator may actuate controls305or the user input device1410to move either the right or left image for alignment with the other of the right and left image. Instructions from the controls305may cause the processor1562to accordingly adjust the location of the left or right pixel set in real-time, such that subsequent images are displayed on the display monitor512reflective of the operator input. In other examples, the instructions may cause the processor1562to change a position of one or more of the optical elements1402via radial adjustment, rotational adjustment, axial adjustment, or tilting. The operator continues to provide input via controls305and/or the user input device1410until the left and right images are aligned. Upon receiving a confirmation instruction, the processor1562stores a calibration point to a look-up-table reflective of the image alignment at the set magnification level.

Additionally or alternatively, the template match method described above may be used to perform image alignment while focused on a planar target that is approximately orthogonal to a stereo optical axis of the stereoscopic visualization camera300. Moreover, the template match method may be used to align the left and right views in real-time whenever a “template match-able” scene is in view of both the left and right optical paths. In an example, a template image is copied from a subset of, for instance, the left view, centered upon or near the center of the view. Sampling from the center for an in-focus image ensures that a similar view of the target site700will be present in the other view (in this example the right view). For out-of-focus images, this is not the case such that in the current embodiment this alignment method is performed only after a successful auto-focus operation. The selected template is then matched in the current view (or a copy thereof) of the other view (in this example the right view) and only a y-value is taken from the result. When the views are aligned vertically, the y-value of the template match is at or near zero pixels. A non-zero y-value indicates vertical misalignment between the two views and a correction using the same value of y is applied either to select the pixel readout set of the first view or a correction using the negated value of y is applied to the pixel readout set of the other view. Alternatively, the correction can be applied in other portions of the visualization pipeline, or split between pixel readout set(s) and said pipeline.

In some examples, the operator may also manually align a right ZRP with an origin of the pixel grid1002. For instance, after determining a location of the right ZRP, the processor1562(and/or the peripheral input unit interface1574or graphics processing unit1564) causes the right ZRP to be highlighted graphically on a right image displayed by the display monitor512. The processor1562may also display a graphic indicative of the origin of the pixel grid1002. The operator uses controls305and/or the user input device1410to steer the right ZRP to the origin. The processor1562uses instructions from the controls305and/or the user input device1410to accordingly move one or more of the optical elements1402. The processor1562may provide a stream of right images in real-time in addition to graphically displaying the current location of the right ZRP and origin to provide the operator updated feedback regarding positioning. The operator continues to provide input via controls305and/or the user input device1410until the right ZRP is aligned. Upon receiving a confirmation instruction, the processor1562stores a calibration point to a look-up-table reflective of positions of the optical elements1402at the set magnification level.

3. Comparison of Alignment Error

The example stereoscopic visualization camera300produces less alignment error between right and left images compared to known digital surgical microscopes with stereoscopic cameras. The analysis discussed below compares spurious parallax generated by ZRP misalignment for a known digital surgical microscope with camera and the example stereoscopic visualization camera300. Initially, both cameras are set at a first magnification level with a focal plane positioned on a first position of a patient's eye. Equation (3) below is used to determine working distance (“WD”) from each camera to the eye.
WD=(IPD/2)/tan(α)  Equation (3)

In this equation, IPD corresponds to the interpupillary distance, which is approximately 23 mm. In addition, a is one-half of an angle between, for example, the right optical image sensor746and the left optical image sensor748, which is 2.50° in this example. The convergence angle is two times this angle, which is 5°, in this example. The resulting working distance is 263.39 mm.

The cameras are zoomed in to a second magnification level and triangulated on a second position of the patient's eye. In this example the second position is at the same physical distance from the camera as the first position, but presented at the second magnification level. The change in magnification generates spurious horizontal parallax due to misalignment of one or both of the ZRPs with respect to a center of a sensor pixel grid. For the known camera system, the spurious parallax is determined to be, for example, 3 arc-minutes, which corresponds to 0.05°. In Equation (3) above, the 0.05° value is added to α, which produces a working distance of 258.22 mm. The difference in working distance is 5.17 mm (263.39 mm-258.22 mm), which corresponds to the error of the known digital surgical microscope with camera attachment.

In contrast, the example stereoscopic visualization camera300is capable of automatically aligning ZRPs to be within one pixel of a center of a pixel set or grid. If the angular field-of-view is 5° and recorded with a 4k image sensor used in conjunction with a 4k display monitor, the one pixel accuracy corresponds to 0.00125° (5°/4000) or 4.5 arc-seconds. Using Equation (3) above, the 0.00125° value is added to α, which produces a working distance of 263.25 mm. The difference in working distance for the stereoscopic visualization camera300is 0.14 mm (263.39 mm-263.25 mm). When compared to the 5.17 mm error of the known digital surgical microscope, the example stereoscopic visualization camera300reduces alignment error by 97.5%.

In some embodiments, the stereoscopic visualization camera300may be more accurate at higher resolutions. In the example above, the resolution is about 4.5 arc-seconds for a 5° field-of-view. For an 8K ultra-high definition system (with 8000 pixels in each of 4000 rows) with a field-of-view of 2°, the resolution of the stereoscopic visualization camera300is approximately 1 arc-second. This means that ZRP of the left and right views may be aligned to one pixel or 1 arc-second. This is significantly more precise than known digital microscope systems that have spurious parallax on the order of arc-minutes.

4. Reduction of Other Sources of Spurious Parallax

The above-examples discuss how the example stereoscopic visualization camera300reduces spurious parallax as a result of misaligned ZRPs and/or left and right images themselves. The stereoscopic visualization camera300may also be configured to reduce other sources of spurious parallax. For example, the stereoscopic visualization camera300may reduce spurious parallax due to motion by simultaneously clocking the right and left optical image sensors746and748to record images at substantially the same instant.

The example stereoscopic visualization camera300may also reduce spurious parallax due to dissimilar magnification between the left and right optical paths. For example, the stereoscopic visualization camera300may set the magnification level based on the left optical path. The stereoscopic visualization camera300may then make automatic adjustments so that the magnification of the right image matches the left. The processor1562, for example, may use image data to calculate control parameters, for example by measuring a number of pixels between certain features common in the left and right images. The processor1562may then equalize the magnification levels of the left and right images by digital scaling, inserting interpolative pixels, and/or deleting extraneous pixels. The example processor1562and/or the graphics processing unit1564may re-render the right image such that the magnification is matched to the left image. Additionally or alternatively, the stereoscopic visualization camera300may include independent adjustment of the left and right optical elements1402. The processor1562may separately control the left and right optical elements1402to achieve the same magnification. In some examples, the processor1562may first set, for example, the left magnification level then separately adjust the right optical elements1402to achieve the same magnification level.

The example stereoscopic visualization camera300may further reduce spurious parallax due to dissimilar focus. In an example, the processor1562may execute a program1560that determines a best focus for each optical path for a given magnification and/or working distance. The processor1562first performs a focusing of the optical elements1402at a point of best resolution. The processor1562may then check the OOF condition at a suitable non-object-plane location and match the focus for the left and right images. The processor1562next re-checks the focus at best resolution and adjusts the focus iteratively until both left and right optical elements1402focus equally well both on and away from an object plane.

The example processor1562may measure and verify optimal focus by monitoring a signal relating to the focus of one or both of the right and left images. For example, a “sharpness” signal is generated by the graphics processing unit1564for the left and right images simultaneously and/or in synchronization. The signal changes as focus changes and may be determined from an image-analysis program, an edge detection analysis program, a bandwidth of Fourier transforms of pattern intensity program, and/or a modulation transfer function (“MTF”) measurement program. The processor1562adjusts a focus of the optical elements1402while monitoring for a maximum signal indicative of a sharp image.

To optimize the OOF condition, the processor1562may monitor sharpness signals for both the left and right images. If the focus is moved off of the object plane and the signal related to, for example, the left image increases but the signal related to the right image decreases, the processor1562is configured to determine the optical elements1402are moving out of focus. However, if the signals related to both the right and left images are relatively high and approximately equal, the processor1562is configured to determine the optical elements1402are properly positioned for focusing.

5. Benefits of Low Spurious Parallax

The example stereoscopic visualization camera300has a number of advantages over known digital surgical microscopes as a result of the low spurious parallax between right and left images. For example, almost perfectly aligned left and right images produce an almost perfect stereoscopic display for a surgeon, thereby reducing eye fatigue. This allows the stereoscopic visualization camera300to be used as an extension of a surgeon's eyes rather than a cumbersome tool.

In another example, precisely aligned left and right images allow accurate measurements of the surgical site to be digitally taken. For instance, a size of a patient's ocular lens capsule may be measured such that a properly-sized IOL can be determined and accurately implanted. In another instance, a motion of a moving blood vessel may be measured such that an infrared fluorescein overlay can be accurately placed in a fused image. Here, the actual motion velocity is generally not of interest to the surgeon but critical for the placement and real-time adjustment of the overlaid image. Properly matched scale, registration, and perspective of the overlaid images are all important to provide an accurately-fused combined live stereoscopic image and an alternate-mode image.

In some examples, the processor1562may enable an operator to draw measurement parameters on the display monitor512. The processor1562receives the drawn coordinates on a screen and accordingly translates the coordinates to the stereoscopic image. The processor1562may determine measurement values by scaling the drawn ruler on the display monitor512to a magnification level shown in the stereoscopic images. The measurements made by the processor1562include point-to-point measurements of two or three locations displayed in the stereoscopic display, point-to-surface measurements, surface characterization measurements, volume determination measurements, velocity verification measurements, coordinate transformations, instrument and/or tissue tracking, etc.

Conclusion

It will be appreciated that each of the systems, structures, methods and procedures described herein may be implemented using one or more computer programs or components. These programs and components may be provided as a series of computer instructions on any conventional computer-readable medium, including random access memory (“RAM”), read only memory (“ROM”), flash memory, magnetic or optical disks, optical memory, or other storage media, and combinations and derivatives thereof. The instructions may be configured to be executed by a processor, which when executing the series of computer instructions performs or facilitates the performance of all or part of the disclosed methods and procedures.

It should be understood that various changes and modifications to the example embodiments described herein will be apparent to those skilled in the art. Such changes and modifications can be made without departing from the spirit and scope of the present subject matter and without diminishing its intended advantages. It is therefore intended that such changes and modifications be covered by the appended claims. Moreover, consistent with current U.S. law, it should be appreciated that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, paragraph 6 is not intended to be invoked unless the terms “means” or “step” are explicitly recited in the claims. Accordingly, the claims are not meant to be limited to the corresponding structure, material, or actions described in the specification or equivalents thereof.