Patent ID: 12243195

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now toFIG.1, a surgical suite 10 or the like may provide for multiple area illuminators12aand12b, for example, positioned to illuminate an operating room table15holding a patient16for surgery. In addition, the surgical suite 10 may include multiple display lights14and other sources of light including, for example, display lights14providing for visual signals, for example, an illuminated sign display light14′ (e.g., an exit sign) or a computer monitor display light14″ (e.g., an LCD backlight or LED array), presenting data to an attending physician.

The surgical suite 10 may further hold a desktop fluorescence microscope18for use contemporaneously with surgery to analyze ex vitro tissue from the patient16or a surgical fluorescence surgical imaging system20, for example, suspended for direct viewing of tissue of the patient in vivo, or at the tip of an endoscope which may provide for microscopic or macroscopic imaging as will be described.

Each of these sources of ambient light (12and14) may intercommunicate as indicated by logical communication channel22with a controller19to switch rapidly between an on-state24in which light is output and an off-state26in which no light is output indicated schematically by an ambient illumination signal27. The logical communication channel22will be discussed below and may take a variety of forms not limited to, for example, a wired network.

The ambient illumination signal27has a frequency, intensity, and on-state duration so that the output light flashes at a rate substantially above a flicker fusion rate at which the human eye perceives a flashing. The flicker fusion rate is dependent on illumination brightness and other factors but in the present invention will typically be in excess of 24 Hz and preferably above 300 Hz. Generally the intensity of light during the on-state24will be such that an average intensity, that is, the intensity of the on-state24times the duty cycle of the on-state24, provides a desired perceived level of illumination comparable to standard illumination levels. Duty cycle refers to the on-state24duration divided by the time between successive on-states24.

Each of the sources of ambient light (12and14) may employ a light source that provides substantially white light and which may be rapidly switched between full and no illumination with minimal warm-up time or afterglow to have a rise and fall time constant that is preferably more than five times faster than the frequency of the illumination signal27. Standard light emitting diodes (LEDs) may be used for this purpose, which employ an ultraviolet LED emitter exciting a phosphor or similar material if the phosphor has a short fluorescence lifetime on the order of tens of microseconds. Alternatively, the light emitting diodes may employ a combination of red, green, and blue (and optionally orange) light emitting diodes and no phosphor to simulate white light with no phosphor afterglow.

Referring still toFIG.1, the surgical imaging system20may provide for an exciting light source25, for example, a laser having a frequency appropriate to excite fluorescence in fluorescent marker compounds28in tissue of the patient16. In one nonlimiting example, the exciting light source25may provide near infrared light suitable for stimulating indocyanine dyes. The exciting light source25may also communicate as indicated by logical communication channel22with the controller19to switch rapidly between an on-state24in which light is output and an off-state26in which no light is output indicated schematically by an exciting illumination signal29. Importantly, the on-state24of the exciting illumination signal29is coordinated to align with the off-state26of the ambient illumination signal27so as to allow fluorescent imaging with reduced interference from the ambient lighting while reducing exhaustion of the fluorescent material.

Referring now toFIG.2, the surgical imaging system20may provide a first low-light camera30and second high-light camera32co-registered with the low-light camera30to image tissue of the patient16along a common imaging axis34. This co-registration may be obtained, for example, by means of a beam splitter36dividing the light from the tissue of the patient16between the low-light camera30and high-light camera32, which may share a common lens system38. Alternatively, the cameras may have roughly aligned but independent optical paths and the registration may be done digitally by an electronic computer. In some contemplated embodiments, a single camera can capture both the low-light and high-light images with either a mechanically switched filter or a bayer pattern filter array over the pixels.

The low-light camera30will be used to acquire fluorescence imaging data and in that respect may have a blocking filter39providing a filter passing light in the desired wavelength range of the florescent signal from the tissue of the patient16. In one embodiment, the filter may be adapted to pass near-infrared light, for example, from a from fluorescent agent such as indocyanine green (ICG).

The low-light camera30is desirably a single photon type camera such as a single photon avalanche diode array (SPAD) or Quanta Image Sensor (QIS) providing a high time resolution (less than 100 ps) and sensitivity down to individual photons. The low-light camera30may have a low resolution, for example, less than 1000 pixels, or no more than 32×32 pixels, although the inventors contemplate that higher resolutions may be useful as such systems become available, including 1024×500 pixel arrays.

The high-light32camera30may be a standard CMOS camera providing color imaging and a spatial resolution of greater than the low-light camera30, for example, having a resolution higher than the low-light camera30, for example, in excess of 1 million pixels, for example, providing at least 1920×1080 pixels.

Each of cameras30and32will produce a respective set of low-light frames40and high-light frames42, for example, at a frame rate dictated by a fraction of the frequency of the signals27and29and typically at a rate above the flicker fusion rate of about 40-60 frames per second.

The high-light frames42are provided to the controller19which implements a motion extractor determining motion of the tissue of the patient16being imaged from successive frames to produce an optical flow signal46. This optical flow signal46provides a set of vectors for each pixel of the frames42indicating the relative motion direction and distance for that pixel with respect to the previous frame42. In a nonlimiting example, the motion extraction can be performed using the Gunnar-Farneback optical flow algorithm described in Farneback G. (2003) “Two-Frame Motion Estimation Based on Polynomial Expansion,” in: Bigun J., Gustavsson T. (eds) Image Analysis, SCIA 2003, Lecture Notes in Computer Science, vol 2749, Springer, Berlin, Heidelberg.

The optical flow signal46, the high-light frames42, and the low-light frames40are then provided to an integrator50which uses the optical flow signal to align successive low-light frames40for an integration process that sums the images on a pixel-by-pixel basis to improve the signal-to-noise ratio as will be discussed below. The result is a set of optical flow denoised frames52.

The denoised frames52and the high-light frames42are then provided to a trained neural network54to remove artifacts caused by the warping of the images by the integrator50used to correct for optical flow. The neural network54per its training (which will be described in more detail below) may also perform denoising of the images and may augment the information of the low-light frames40(via the denoised frames52) with the information contained in the high-light frames42.

The output of the neural network54provides reduced noise low-late image frames which may be output to display14, for example, for use during surgery or may be used in any subsequent process requiring information from fluorescent imaging or the like.

Referring now toFIG.3, the integrator50will receive successive high-light frames42, for example, at times t-1 and t, and use the optical flow signal46to warp the image from t-1 as indicated by warping block56according to the optical flow signal46obtained between times t-1 and t. This warped high-light frame of t-1 is then compared to the high-light frame42at t to compute a pixel-by-pixel difference between these frames at process block58. These difference values are applied to a thresholder59comparing each difference value to a predetermined threshold defining a point at which the pixel difference likely indicates a motion detection error. The outputs of the thresholder59provide a binary mask value60for each pixel of a frame42to create an optical flow failure map62for time t. Successive optical flow failure maps62are generated for each successive frame42.

It will be appreciated that if the warping process of warping block56perfectly corrects for motion between times t-1 and t of the high-light frames42then the optical flow failure map62will have values of zero for all pixels. On the other hand, differences between successive frames42after warping of the earlier frame for motion, for example, because one image may be occluded by a surgical instrument or the like, will produce values of one in the optical flow failure map62for the pixels in that region of occlusion. More generally the optical flow failure map62will reflect any significant difference between the warped and current image not limited to occlusion.

The optical flow failure map62is used to reset a set of averaging counters64that provide a running total of the number of successive frames in which a given pixel has not been subject to an optical flow failure. Use of the averaging counters64will be described later. Like the optical flow failure map62, the averaging counters64provide a count value for each pixel of a frame42, and snapshots of the averaging counters64may be stored for each frame time.

The optical flow signal46is also used to warp a current denoised frame70which represents a running integration of motion-corrected low-light frames40as will now be described. In this process, a current denoised frame for time t-1 is received by warping block72also receiving the optical flow signal46to warp the current denoised frame for time t-1 to the current time t. This warped frame74is then multiplied by the optical flow failure map62at multiplier76so that the warped frame74only includes valid pixels (with invalid pixels zeroed). The resulting masked signal80is then summed with the current low-light frame40at summing block82, and this used to provide the next denoised frame70for time t.

Each denoised frame70as it is computed is then normalized by divider84on a pixel-by-pixel basis by dividing the value of each pixel by the averaging counters64for that pixel. This division process compensates for the fact that the pixel values will represent integrations over different durations according to the last occurrence of an optical flow failure.

The output of the divider84then provides the denoised frames52which are input to the neural network shown inFIG.2. In one embodiment, the neural network54receives five consecutive denoised frames52together with the corresponding values of the averaging counters64and high-light frames42.

Referring now toFIG.4, the neural network54may in one embodiment provide an architecture following the teachings of FastDVNet as described in Matias Tassano, Julie Delon, and Thomas Veit: DVDNet: A fast network for deep video denoising, in2019IEEE International Conference on Image Processing (ICIP), pages 1805-1809, 2019.

Training of the neural network54is performed with a set of noisy fluorescent frames100in pairs with corresponding ground truth fluorescent frames102. Both frames of each pair may be derived from tissue samples injected with indocyanine green, for example, into the femoral artery of a chicken thigh manipulated over many frames to simulate vascular surgery. Imaging of this vascularized tissue and injected dye provide high visual contrast, low-noise images that can be used as the ground truth fluorescent frames102. The noisy fluorescent frames100are then prepared by reducing the signal strength and introducing noise104of a type expected for the particular detector (random additive Poisson noise for a SPAD detector) and other types of noise such as spatial distortion expected from the warping process of the present invention, blurring from a combination of successive frames and quantization noise. The training set may also include a high-resolution image104obtained contemporaneously with a camera similar to high-light camera32and registered with the frame102. Finally, the values of the averaging counters64for each pixel may be provided.

The training process cycles through these training set values to train the weights of the neural network54and may use a mean square error loss function in the training process and optimization using the ADAM Optimizer described in Diederik P. Kingma and Jimmy Ba: Adam: A method for stochastic optimization, in Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.

While the invention has been described in a medical context for imaging tissue, it will be appreciated that the same principles can be applied to nonmedical applications including for example LiDAR systems, thermal imaging, polarimetry, hyperspectral imaging, images of material scattering, non-line of sight imaging, and others, where there are different received illumination signals with substantially different flux, so that the stronger signal can allow motion tracking to permit integration of the weaker signal to improve its signal-to-noise ratio.

Certain terminology is used herein for purposes of reference only, and thus is not intended to be limiting. For example, terms such as “upper”, “lower”, “above”, and “below” refer to directions in the drawings to which reference is made. Terms such as “front”, “back”, “rear”, “bottom” and “side”, describe the orientation of portions of the component within a consistent but arbitrary frame of reference, which is made clear by reference to the text and the associated drawings describing the component under discussion. Such terminology may include the words specifically mentioned above, derivatives thereof, and words of similar import. Similarly, the terms “first”, “second” and other such numerical terms referring to structures do not imply a sequence or order unless clearly indicated by the context.

The term “frame” as used herein is intended to describe an array of at least two dimensions of pixels taken at a given time interval and includes frames where each pixel is a single intensity value or a histogram of fluorescence lifetimes.

When introducing elements or features of the present disclosure and the exemplary embodiments, the articles “a”, “an”, “the” and “said” are intended to mean that there are one or more of such elements or features. The terms “comprising”, “including” and “having” are intended to be inclusive and mean that there may be additional elements or features other than those specifically noted. It is further to be understood that the method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order discussed or illustrated, unless specifically identified as an order of performance. It is also to be understood that additional or alternative steps may be employed.

References to “a controller”, “a processor”, or “a computer” can be understood to include one or more circuits that can communicate in a stand-alone and/or a distributed environment(s), and can thus be configured to communicate via wired or wireless communications with other circuits. Generally, such a device may be dedicated circuitry such as constructed from discrete components, an FPGA or ASIC or the like, or may provide a standard computer architecture including one or more processors such as a CPU, GPU, and/or one or more purpose-built accelerators and computer memory holding a data and a stored program. Such devices may be associated with or include standard input and output devices including a graphic display terminal, a keyboard, a voice interface, a touchscreen, a trackball, or mouse or the like and may provide for input/output connections through standard electronic interfaces, level shifting circuits, and analog-to-digital and digital-to-analog converters and/or digital interfaces employing standard protocols for electrical communication. In particular, the present invention may provide for software and circuitry to interface with the above devices and other devices including for example other medical systems according to protocols required for DICOM®, as well as to remote devices using the Internet, various wireless and wired communications including IEEE 802.11, as well as various video and audio interfaces of types well known in the art.

The memory may store one or more types of instructions and/or data including those to implement the invention as described above, and to permit operation of the interfaces described above, and may include volatile and/or non-volatile non-transitory computer readable media, for example, RAM (Random Access Memory), flash memory, ROM (Read Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), registers, disks, drives, or any other suitable storage medium, or any combination thereof. The memory can be a component of a processor, can be operatively connected to a processor for use thereby, or a combination of both.

It is specifically intended that the present invention not be limited to the embodiments and illustrations contained herein and the claims should be understood to include modified forms of those embodiments including portions of the embodiments and combinations of elements of different embodiments as come within the scope of the following claims. All of the publications described herein, including patents and non-patent publications are hereby incorporated herein by reference in their entireties.