Patent Description:
Cataract surgery generally involves replacing a natural lens of a patient's eye with an artificial intraocular lens (IOL). During the cataract surgery, imaging devices may be used to generate an image stream of the patient's eye to help perform the surgery. For example, the image stream may provide the medical practitioner with real-time, magnified views of the patient's eye for use during the surgery. However, unprocessed individual frames of the image stream may not include enough detail to be useful to the medical practitioner and processed frames may not be processed quickly enough to enable the real-time use by the medical practitioner, which can lead to sub-optimal surgical outcomes.

Therefore, there is a need for improved systems and methods for processing frames in images streams for display to the medical practitioner. Reference is made to the documents <NPL>, and <NPL>, which have been cited as exemplary of the background state of the art.

Certain embodiments provide an ophthalmic imaging device for enhancing captured image data. The ophthalmic imaging device comprises an image capture component and an image processor. The image capture component is configured to generate an image stream. The image stream comprises a first frame capturing a branch of veins in an eye of a patient and a second frame capturing the branch of veins, where the second frame follows the first frame in the image stream. The image processor is configured to calculate first order statistics for individual blocks of a plurality of blocks for the first frame. Each block of the plurality of blocks for the first frame comprises a plurality of first frame pixels. The image processor is also configured to interpolate first order statistics for the first frame based on the calculated first order statistics for the individual blocks of the plurality of blocks for the first frame and generate a tone mapping function for pixels of the second frame based on the interpolated first order statistics for the first frame. The first order statistics for the first frame are based, at least in part, on the branch of veins. The image processor is further configured to calculate tone mapping values for individual pixels of the second frame based on the tone mapping function. The image processor is additionally configured to generate an enhanced frame based on application of the calculated tone mapping values for individual pixels of the second frame to the pixels of the second frame.

Another embodiment provides a method of processing and enhancing images. The method comprises generating an image stream. The image stream comprises a first frame capturing a branch of veins in an eye of a patient and a second frame capturing the branch of veins in the eye, where the second frame follows the first frame in the image stream. The method further comprises calculating first order statistics for individual blocks of a plurality of blocks for the first frame, the individual blocks of the plurality of blocks for the first frame comprising a plurality of first frame pixels, and interpolating first order statistics for the first frame based on the calculated first order statistics for the individual blocks of the plurality of blocks for the first frame. The first order statistics for the first frame are based, at least in part, on the branch of veins. The method also comprises generating a tone mapping function for pixels of the second frame based on the interpolated first order statistics for the first frame and calculating tone mapping values for individual pixels of the second frame based on the tone mapping function. The method additionally comprises generating an enhanced frame based on application of the calculated tone mapping values for individual pixels of the second frame to the pixels of the second frame.

An example comprises an ophthalmic imaging device for capturing and enhancing images. The imaging device comprises an image capture component and an image processor. The image capture component is configured to generate an image stream of at least a portion of an eye of a patient, where the image stream comprises a first frame capturing a branch of veins in the eye and a second frame capturing the branch of veins in the eye that follows the first frame in the image stream. The image processor is configured to divide the first frame into a plurality of blocks, where each block comprises a plurality of pixels. The image processor is further configured identify image enhancement parameters for the first frame based on the plurality of blocks and generate an image enhancement function for the second frame based on the identified image enhancement parameters of the first frame. The first order statistics for the first frame are based, at least in part, on the branch of veins. The image processor is additionally configured to apply the image enhancement function to the second frame with the identified image enhancement parameters for the first frame and generate an enhanced frame based on application of the image enhancement function to the second frame.

Other embodiments provide processing systems configured to perform the aforementioned methods as well as those described herein; non-transitory, computer-readable media comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the aforementioned methods as well as those described herein; a computer program product embodied on a computer readable storage medium comprising code for performing the aforementioned methods as well as those further described herein; and a processing system comprising means for performing the aforementioned methods as well as those further described herein.

Medical imagery is frequently used to support medical practitioners during various types of medical procedures. For example, during cataract surgery, a medical practitioner may use an imaging device, such as a retinal camera or other ocular imaging device focused on a patient's eye, to capture imagery of the patient's eye during the surgery. The captured imagery may assist the medical practitioner with, for example, monitoring or detecting tool (or other object) placement and eye condition during the surgery.

In some cases, it is difficult to enhance the imagery in real-time for display to the medical practitioner because capturing, processing, and enhancing a current frame of the imagery introduces lag. In other words, the processing and enhancement of the current frame cannot be completed quickly enough to display the enhanced current frame in real-time, or near real-time, to the medical practitioner. This lag could create safety risks for tool placement and other sensitive procedures.

The systems and methods described herein overcome problems with conventional methods by processing a preceding frame to identify enhancement parameters to apply to the current frame, which eliminates the problematic lag while enabling improved imaging characteristics. Using a preceding frame to determine image enhancement characteristics for a following frame is effective in cases where consecutive frames of imagery, for example of a patient's eye during a surgery, include portions that are relatively static between the consecutive frames; thus the enhancement characteristics for the preceding frame work well for enhancing the following frame that shares portions with the preceding frame.

A static portion of an image (referred to herein interchangeably as frame) may generally include content in a portion or area of the frame that is the same or substantially the same as frame content in corresponding portions across or between the consecutive frames. For example, structures of the patient's eye captured in a first portion of the preceding frame are likely located in a corresponding first portion of the current frame because the patient's eye and the imaging device likely will not move drastically between the consecutive frames. Thus, the enhancement parameters for the preceding frame, for at least the static portions, will generally apply also to the current frame.

By employing the enhancement parameters from the preceding frame to the current frame, enhancement of the current frame can be completed more quickly, which enables the real-time (or near real-time) enhancement and display of the current frame to the medical practitioner. The enhancement of the imagery can improve contrast of the imagery, enabling the medical practitioner to better see structures in the eye, tools, and the like, which improves on the conventional compromise between image latency and image quality.

In some embodiments, processing of one or more preceding frames can be performed in parallel with enhancing the current frame, where the preceding frame is processed to generate the enhancement parameters used to enhance the current frame while the current frame is captured. In some embodiments, such processing of one or more of the previous frames can be performed in advance of a current or subsequent frame being generated, such that when the current or subsequent frame is generated, the enhancement parameters to enhance the current or subsequent frame are already available.

Accordingly, certain aspects of the present disclosure provide image processing systems and devices for generating, enhancing, and displaying image data that may be used in medical imaging equipment as well as other types of imaging equipment. By using the enhancement parameters for preceding frames to enhance subsequent frames in the image stream, the image processing systems and devices can present enhanced image frames in real-time, or near real-time, which was not possible with conventional methods.

Note that the systems, methods, and techniques described herein are described in the context of surgical procedures as one example; however, these systems, methods, and techniques are widely applicable to various imaging systems, including pre- and/or post-operative procedures, as another example. Furthermore, the systems, methods, and techniques described herein can be applied in various other medical embodiments (for example, capturing frames of one or more aspects of an anatomy). Similarly, the systems, methods, and techniques described herein can be applied in various non-medical embodiments where contents depicted between successive frames in the imagery are relatively static such that a current frame can be enhanced based on enhancement parameters generated from a preceding frame.

<FIG> illustrates a block diagram of an image processing system <NUM> that processes and enhances imagery of, in this example, a patient's eye <NUM>, according to example embodiments described herein. The system <NUM> includes a controller <NUM> that is communicatively coupled with an imaging device <NUM>. The imaging device <NUM> is representative of one or more imaging devices used to capture the image stream of frames of the patient's eye <NUM>, in this example. The controller <NUM> may be configured to analyze the frames of the image stream received from the imaging device <NUM>, for example, a current frame <NUM> that corresponds to a most recently captured frame in the image stream. Such analysis may comprise identifying various aspects of the frame <NUM>, such as image enhancement parameters, and enhancing the frame <NUM> based on the enhancement parameters.

In some embodiments, the controller <NUM> may be remote from the imaging device <NUM> and may be communicatively coupled by a data connection, such as a network connection. For example, controller <NUM> may be configured to support multiple imaging devices at a single location (e.g., at one or more ophthalmic practices, which may be located remote from each other). Note that an ophthalmic practice herein may refer to (<NUM>) a clinic at which pre-operative and/or post-operative imaging is performed for patients' eyes in preparation for a surgical procedure and/or (<NUM>) an ophthalmic surgical practice at which intra-operative images are generated for patients' eyes during a surgical procedure.

The controller <NUM> is also coupled to a data store or memory <NUM> that stores frames captured by the imaging device <NUM> and corresponding data (such as enhancement parameters) obtained from analysis of the captured frames. In certain embodiments, the memory <NUM> is a cache memory that is co-located with the controller <NUM> and stores one or more previous frames and/or corresponding enhancement parameters for the one or more previous frames. In some embodiments, the memory <NUM> comprises a central and/or cloud-based database or repository for storing original and enhanced frames (and corresponding data) of the patient's eye <NUM>. In some embodiments, the original and enhanced frames of the patient's eye <NUM> and the corresponding enhancement parameters may be associated with a profile for the patient whose eye <NUM> is imaged with the imaging device <NUM>. In certain embodiments, the memory <NUM> may be representative of a device memory, an on-premises or cloud-based database or repository dedicated for use at the ophthalmic practice, and so forth.

In some embodiments, the controller <NUM> is a local processor or computing system accessible by the imaging devices <NUM>. For example, the controller <NUM> may refer to a computing system that is dedicated and/or local to the imaging device <NUM> and/or the ophthalmic practice. In certain embodiments, the controller <NUM> may correspond to computing resources (e.g., including one or more processors and/or computing systems) provided through a "cloud" service. In certain embodiments, the network that connects one or more components of the system <NUM> may include one or more switching devices, routers, local area networks (e.g., an Ethernet), wide area networks (e.g., the Internet), and/or the like.

The imaging device <NUM>, as shown in <FIG>, comprises any imaging device, such as an ocular imaging device in this particular example, configured to generate an image stream that comprises one or more frames (for example, the current frame <NUM>) of the patient's eye <NUM> captured over a period. In some embodiments, the frames generated by the imaging device <NUM> may be used to analyze and/or view one or more of aspects of optical components and features of the patient's eye <NUM>, relationships of components of the patient's eye <NUM>, tools and corresponding devices being used during a surgical procedure, and the like. In some embodiments, the imaging device <NUM> comprises one or more of an optical coherence tomography (OCT) device, a scanning laser ophthalmoscope (SLO) device, a scanning laser polarimetry device, <NUM>-dimensional imaging and visualization devices, a high dynamic range (HDR) camera, a retina view system, a retina viewing lens, a surgical microscope, and the like.

In some embodiments, the imaging device <NUM> includes a dedicated processor that executes instructions provided by a dedicated memory to capture frames of the patient's eye <NUM>, analyze the captured frames, enhance the captured frame for display to the medical practitioner, and allow an user to operate the imaging device <NUM> through, for example, a dedicated user interface, etc. of the imaging device <NUM>. The user interface of the imaging device <NUM> may enable a user, such as the medical practitioner, to interact with and control the imaging device <NUM>.

The imaging device <NUM> includes device features for imaging the patient's eye <NUM>. Non-limiting examples of such device features include at least one of optical features, emission features, imaging features, and control features. The optical features comprise one or more lenses or other optical components for focusing and directing light projected to and reflected by a target object of the patient's eye <NUM>. The optical features enable the imaging device <NUM> to view and image the patient's eye <NUM>, focus optical beams into the eye, etc., to generate and capture frames of the patient's eye <NUM>.

The emission features comprise a light or other signal source configured to emit a signal (e.g., the optical beams, ultrasonic sound waves, etc.) into the patient's eye <NUM>. The emission features may be adjustable with regard to positioning, focusing, power level, or otherwise directing the signal as needed by the medical practitioner or in an automated manner. The imaging features include features that generate, receive, process, and/or digitize signals that that reflect or echo back from the patient's eye <NUM> or otherwise represents the patient's eye <NUM>. The imaging features are responsible for generating multi-dimensional images based on the received signals. The imaging features may acquire, store, and/or process image data based on the received signals. Examples of imaging features in the imaging device <NUM> may include image processing components and the like.

The control features enable the medical practitioner to activate, deactivate, and adjust the device features of the imaging device <NUM>. For example, the control features include components that enable adjustment of the emission features, such as controls to turn on/off the emission features, captures frames using the imaging features, and so forth. Similarly, the control features include components that enable adjustment of the optical features, such as to enable automatic or manual focusing of the optical features or movement of the optical features to view different targets or portions of the target. In some embodiments, the user interface of the imaging device <NUM> includes the control features.

In some embodiments, the enhanced frames can be used pre-operatively to prepare a surgical plan in preparation for a surgical procedure (e.g., cataract surgery). In certain embodiments, the ophthalmic practice may use the imaging device <NUM> in connection with an operating room to obtain, process, and display an image stream of the patient's eye <NUM> during the surgical procedure.

As introduced above, the imaging device <NUM> may communicate frames of the patient's eye <NUM> to the controller <NUM> for processing (for example, enhancement), storage, and display. As part of the processing, storage, and display of the frames, the controller <NUM> may receive the frames from the image device <NUM>, process and enhance the frames, and store and/or display the enhanced frames for use by the medical practitioner. Enhancement of the image stream may comprise, for example, contrast enhancement, which increases perceived qualities for the frames in the image stream. Contrast enhancement can be applied to a current frame <NUM> (or any frame of the image stream) such that aspects of the frame <NUM> stand out more as compared to unenhanced, or raw, frames. Contrast enhancement may make optimal use of colors available on a display or output device on which the enhanced frames are displayed. Further details of the processing and enhancement of the frames will be provided below.

As described above, the memory <NUM> stores one or more frames (raw and/or enhanced) and corresponding data (for example, the enhancement parameters) associated with the stored frames. In some embodiments, the memory <NUM> stores the one or more frames after they are processed and enhanced by the controller <NUM>. In some embodiments, the one or more frames and corresponding data are associated with patient profiles. In certain embodiments, the memory <NUM> stores the frames for later retrieval, for example, in response to a request from the controller <NUM> or the like. In some embodiments, the memory <NUM> operates as a cache memory that stores frames and corresponding data and that enables the controller <NUM> to quickly and efficiently access particular processed frames and corresponding data, for example, during processing and enhancement of subsequent frames.

In some embodiments, the memory <NUM> further stores enhancement parameters, which may comprise tone mapping parameters or other similar image enhancement characteristics, parameters, or statistics generated from the processing of one or more frames. For example, where the controller <NUM> processes one of the frames generated by the imaging device <NUM> to identify one or more tone mapping parameters, such as one or more image statistics, including a mean pixel grayscale or luminance or a standard deviation of pixel grayscale or luminosity for individual blocks or an entirety of the frame, the memory <NUM> may store the processed frame and the corresponding tone mapping parameters. In some embodiments, the corresponding tone mapping parameters are associated with the frame from which the corresponding tone mapping parameters were generated. Thus, the corresponding tone function parameters may be used in conjunction with subsequent frames generated by the imaging device <NUM>.

<FIG> is an example data flow diagram (data flow) <NUM> illustrating how image data (e.g., for frame <NUM> in <FIG>) is processed to generate an enhanced frame <NUM> using an adaptive equalization algorithm or other algorithm that spreads intensities of all or a subset of image pixels out to occupy all the available intensity dynamic range, according to aspects described herein.

The data flow <NUM> shows the frame <NUM> as generated by the imaging device <NUM>, described above with reference to <FIG>. The frame <NUM> is processed, for example, by the controller <NUM> to logically divide the frame <NUM> into a plurality of blocks <NUM> or tiles, as shown by blocked frame <NUM> with <NUM> blocks 224a-224i. In certain embodiments, the controller <NUM> generates the blocked frame <NUM> based on logically dividing the frame <NUM> into a number of blocks <NUM> having the same relative size, varying in size or shape, or having consistent or even spacing, for example, at different parts of the frame <NUM>. For example, the controller <NUM> may identify the number of blocks <NUM> without actually dividing the frame <NUM> itself to create the blocked frame <NUM>. The controller <NUM> may process the frame <NUM> according to the logical blocks <NUM> of the blocked frame <NUM>.

In some embodiments, dividing the frame <NUM> into the plurality of blocks <NUM> comprises the controller <NUM> analyzing the frame <NUM> to identify one or more portions of the frame <NUM> that do not need to be processed further, such as portions of the frame <NUM> that border but are not part of the patient's eye <NUM>. For example, the frame <NUM> includes a white border portion that substantially surrounds a portion of the frame <NUM> corresponding to the patient's eye <NUM>. The white border portion is otherwise irrelevant to the analysis and enhancement of the frame <NUM>. The controller <NUM> may identify the white border portion and mark that portion as being irrelevant and not needing further processing because the white border portion does not include any aspects of the patient's eye <NUM>. By not processing such the white border portion (and similar irrelevant, or black space, portions) of the frame <NUM>, the controller <NUM> avoids processing portions of the frame <NUM> that are not of interest and could otherwise adversely impact image statistics and parameters generated by the controller <NUM> based on the frame <NUM>. This may improve efficiencies and enhancement times of frames in the image stream.

In some embodiments, the controller <NUM> identifies the white border portion (and other irrelevant portions) of the frame <NUM> based on image processing using machine learning model or other image processing algorithms. For example, the controller <NUM> may apply one or more image segmentation algorithms to the blocks <NUM> of the blocked frame <NUM>. In some embodiments, the controller <NUM> may use a trained machine learning (ML) model to identify relevant or irrelevant portions of the frame <NUM>. A collection of previously processed frames may provide a dataset (referred to as the "training dataset") for use in training the ML model that can identify relevant and irrelevant portions of blocks <NUM> of the frame.

In some instances, the ML model is trained using one or more ML algorithms, such as the image segmentation algorithms, in conjunction with the training dataset. The ML algorithms may include a supervised learning algorithm, an unsupervised learning algorithm (such as K-means and Gaussian Mixture Model), and/or a semi-supervised learning algorithm. Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labeled responses. Supervised learning is the ML task of learning a function that, for example, maps an input to an output based on example input-output pairs. Supervised learning algorithms, generally, include regression algorithms, classification algorithms, decision trees, neural networks, etc..

In certain embodiments, a trained ML model refers to a function, for example, with weights and parameters that is used to identify the relevant and irrelevant portions of the blocks <NUM> in the frame <NUM>. Once trained and deployed, the ML models are able to identify the relevant or irrelevant portions of blocks <NUM>, as output.

The controller <NUM> processes the blocked frame <NUM> to calculate one or more tone mapping parameters (or enhancement parameters) <NUM>, which may correspond to the enhancement parameters used to generate the enhanced frame <NUM> corresponding to the frame <NUM>. Calculating the one or more tone mapping parameters <NUM> may comprise the controller <NUM> calculating the tone mapping parameters for each block <NUM> of the blocked frame <NUM> or for the blocked frame <NUM> as a whole. In some embodiments, the controller <NUM> may calculate the tone mapping parameters <NUM> at a pixel level of the blocks <NUM> of the blocked frame <NUM>.

In some embodiments, where the controller <NUM> calculates the tone mapping parameters <NUM> at the block level, the controller <NUM> calculates an order statistic, such as a first order statistic, for an aspect (for example, of a block <NUM>) of the blocked frame <NUM>. A first order statistic for a portion of a frame can be calculated based on a function that determines a probability that a particular pixel having a particular value of interest occurs in the portion of the frame. The first order statistics may be based on individual pixel values independent of neighboring pixels. In some embodiments, the first order statistics may be generated based on a first-order histogram of pixel values that relates numbers of pixels with particular pixel values, such as grayscale values. Examples of the first-order statistics include mean, standard deviation, skew, and so forth. Thus, the tone mapping parameters <NUM> can comprise one or more of a mean pixel grayscale <NUM> for each block <NUM> of the blocked frame <NUM> or a standard deviation of pixel grayscale <NUM> of each block <NUM> of the blocked frame <NUM>, and so forth. In some embodiments, the mean pixel grayscale <NUM> (mean, µ) and the standard deviation of pixel grayscale <NUM> (STD, σ) are calculated according to the equations below: <MAT> <MAT> <MAT>.

The first order statistic can be a smallest sample value (i.e., a minimum sample value) of a set of sample values arranged in ascending order. For example, in the sample set <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> the first order statistic is <NUM>. Thus, as applied to the mean pixel grayscale <NUM> and the standard deviation of pixel grayscale <NUM> of each block <NUM> of the blocked frame <NUM>, the first order statistic of the mean pixel grayscale <NUM> may correspond to the smallest mean pixel grayscale for a corresponding block <NUM>. Similarly, the first order statistic of the standard deviation of pixel grayscale <NUM> may correspond to the smallest standard deviation of pixel grayscale for the corresponding block <NUM>. While the examples described herein relate to pixel grayscale (or luminosity or intensity), in some embodiments, the tone mapping parameters <NUM> comprise statistics generated from other values of the frame or image stream, such as chrominance values, color values, and the like. Furthermore, the tone mapping parameters <NUM> calculated at the pixel level or the frame level can have corresponding first order statistics calculated by the controller <NUM>. In some embodiments, the first order statistics described herein apply to a grayscale channel obtained from an analysis or processing of green pixels of the preceding frame <NUM>. For example, the grayscale described here can correspond to the green channel grayscale for a frame, such as the preceding frame <NUM>. In some embodiments, the tone mapping function may be applied to a green channel grayscale of the current frame <NUM>.

In some embodiments, based on one or more of the calculated first order statistics for the blocks <NUM> of the blocked frame <NUM> or for the pixels of the blocked frame <NUM>, the controller <NUM> interpolates a mean pixel grayscale for the entire blocked frame <NUM>, generated as an interpolated mean pixel grayscale <NUM>. Similarly, the controller <NUM> interpolates a standard deviation of pixel grayscale for the entire blocked frame <NUM> based on the first order statistics of the blocks <NUM> (or pixels) of the blocked frame <NUM>, generated as an interpolated standard deviation of pixel grayscale <NUM>. Interpolation may correspond to a process of using known data values to predict or estimate values for unknown data values. In some embodiments, interpolation can be used to combine various aspects of the blocked frame <NUM>. Interpolation can also be used to generate smooth representations of first order statistics across blocks <NUM>, thereby smoothing edges between the blocks <NUM> of the blocked frame <NUM>.

For example, the controller <NUM> interpolates (adjacent) neighboring blocks <NUM> to smooth transitions between or among pixels of the neighboring blocks <NUM>. The blocks <NUM> that have four neighbor blocks <NUM> in the frame, such as a block 224e, will be interpolated using the four neighbor blocks 224b, 224d, 224f, and <NUM> (corresponding to the blocks <NUM> to the top, bottom, left, and right) of the block 224e. For a block that is on an edge of the frame, such as the block 224f, the block 224f is missing a neighbor along the right edge of the block 224f. However, to enable interpolation similar to that performed for the block 224e, the controller <NUM> can duplicate one of (<NUM>) the block 224f to represent the missing neighbor block along the right edge of the block 224f or (<NUM>) the block 224e neighboring the block 224f on a side opposite a location of the missing neighbor block right edge and use that duplicate for the interpolation using four neighbor blocks <NUM>. In some embodiments, duplicating blocks during interpolation removes a "specialness" of the duplicated blocks, so calculation for the edge blocks can be done in the same way as non-edge blocks. This use of the duplicate block(s) may limit introduction of variance into the interpolation process that may otherwise result from performing interpolation with less than four neighbor blocks <NUM>. In some embodiments, the missing neighbor block is filled with a block from an opposite edge of the missing neighbor block.

The controller <NUM> may employ one or more adaptive or non-adaptive interpolation methods. An adaptive interpolation method comprises an interpolation that changes based on pixels being interpolated, while non-adaptive interpolation methods are applied consistently to all pixels. Examples of non-adaptive interpolation algorithms include nearest neighbor, bilinear, bicubic, spline, and the like. In certain embodiments, the interpolation performed by the controller <NUM> uses one or more non-adaptive interpolation methods.

The controller <NUM> can perform frame interpolation of the blocked frame <NUM> to identify or achieve a best approximation values (for example, the enhancement parameters introduced above) for the blocked frame <NUM>. For example, the controller <NUM> may generate a best approximation of the mean pixel grayscale and a best approximation of the standard deviation of pixel grayscale for the entire blocked frame <NUM> based on the mean pixel grayscale and the standard deviation of pixel grayscale, respectively, for the blocks of the blocked frame <NUM>.

Based on the interpolated or best approximation values generated by the controller <NUM>, the controller <NUM> generates and applies the interpolated mean pixel grayscale values and the standard deviation of pixel grayscale as inputs to a tone mapping function <NUM> (for example, a frame enhancement algorithm). The tone mapping function <NUM> may map a first set of colors (or intensities or grayscales) in a frame to a second set of colors (or intensities or grayscales, respectively). Accordingly, the controller <NUM> applies the tone mapping function <NUM> to each pixel of the blocked frame <NUM> to generate an enhanced frame <NUM>, which is an enhanced version of the frame <NUM> generated by the imaging device <NUM>. The enhanced frame <NUM> may comprise a contrast enhanced frame, as compared to the frame <NUM>, or may be the frame <NUM> enhanced in one or more other ways. In some embodiments, the enhanced frame <NUM> enables the medical practitioner to view various details of the patient's eye <NUM> that may otherwise be lost in the raw or unenhanced frame <NUM>. In some embodiments, the tone mapping function <NUM> apply to a luminance channel of the frame <NUM> when the first order statistics are generated based on a luminance channel of the preceding frame <NUM>. In some embodiments, the tone mapping function <NUM> can be applied to a luminance channel of the current frame <NUM>.

As introduced above, performing such processing of the frame <NUM> to divide the frame <NUM> into blocks and process the blocks to generate the tone mapping parameters for use in the tone mapping function <NUM> before generating the enhanced frame <NUM> involves an amount of time that is not readily available or feasible during, for example, the surgical procedure. Thus, the processing of the <FIG> may not be able to provide enhanced frames <NUM> for the frame stream generated by the imaging device <NUM> during the surgical procedure.

In certain embodiments, as introduced above, the controller <NUM> processes or analyzes a preceding frame in the frame stream while a current frame is being generated by the imaging device <NUM>. Such parallel frame processing enables the controller <NUM> to use enhancement parameters for the preceding frame to enhance the current frame and, in aggregate, provide an enhanced image stream quickly for viewing by the medical practitioner.

In some embodiments, the enhancement parameters of the preceding frame can be applied to the current frame because subsequent frames in the image stream often have continuous features that are in generally the same portions or areas in the subsequent frames, as introduced above. For example, a first portion of a preceding frame may depict a branch of veins in the patient's eye <NUM>. The current frame may include the same branch of veins in the corresponding first portion of the preceding frame. Thus, the enhancement parameters for the first portion from the preceding frame can be applied to the corresponding first portion of the current frame. The consecutive frames in the image stream may share a large amount of content, meaning that corresponding pixels in consecutive frames of the image stream generally the same or similar, or substantially the same, content. Such continuous features enable the use of the enhancement parameters for the preceding frame to the current frame because the aspects that enhance the continuous features in the preceding frame will more often than not work for the corresponding continuous features in the current frame in the similar location. Thus, processing to generate the enhanced frame <NUM> based on the frame <NUM> from the image stream, as described with reference to <FIG>, may be applied to generate an enhanced frame for a subsequent frame in the image stream based on the enhancement parameters for one or more preceding frames.

This ability to process a current frame (such as the frame <NUM>) based on enhancement parameters from a preceding frame to generate the enhanced frame <NUM> of the current frame enables faster handling and processing of frames of the image stream during the surgical procedure. Instead of having to process the frame <NUM> to identify the tone mapping parameters for the frame <NUM> before generating the enhanced frame <NUM>, the controller <NUM> can use the enhancement parameters from an frame preceding the frame <NUM> and only generate the enhanced frame <NUM> based on the frame <NUM> using the enhancement parameters. This saves time and enables display of enhanced frames to the medical practitioner in real time.

In some embodiments, because the target of the image stream generally should not move much relative to the different frames in the image stream, the enhancement parameters generated based on the frame <NUM> can be applied to a number of subsequent frames in the image stream. An example of using the enhancement parameters from a preceding frame for a subsequently capture, current frame <NUM> to generate an enhanced frame <NUM> of the current frame <NUM> is described with reference to <FIG>.

<FIG> is an example data flow diagram <NUM> illustrating employing the parallel processing to generate the enhanced frame <NUM> of the current frame <NUM> in parallel with processing the preceding frame. The data flow diagram <NUM> includes many aspects that are similar to aspects shown in the data flow <NUM> of <FIG>. Corresponding aspects between the data flows <NUM> and <NUM> may have corresponding functionality and operations, and so forth. Thus, for one or more aspects of the data flow <NUM> that correspond to the aspects in the data flow <NUM>, corresponding description will not be duplicated for brevity.

The data flow <NUM> begins with the frame <NUM> as generated by the imaging device <NUM>. The frame <NUM> may represent a non-initial frame of the image stream generated by the imaging device <NUM> of the patient's eye <NUM>. The initial frame of the image stream does not have a preceding frame. Accordingly, the initial frame cannot be enhanced based on the enhancement parameters of the preceding frame.

The data flow <NUM> includes a block <NUM> corresponding to processing, by the controller <NUM>, of the preceding frame <NUM> in the image stream that precedes the frame <NUM>. For example, if the frame <NUM> is a second frame of the image stream, the block <NUM> corresponds to processing of the preceding frame <NUM>, which is a first frame of the image stream, by the controller <NUM>. In some embodiments, the processing in the block <NUM> may be performed by the controller <NUM> on multiple preceding frames. For example, when the frame <NUM> is the <NUM>th frame in the image stream, the controller <NUM> may perform the processing of the block <NUM> based on any combination of preceding frames <NUM>-<NUM> in the image stream.

In some embodiments, the processing of the one or more preceding frames results in or generates enhancement parameters <NUM> for the one or more preceding frames. For example, the enhancement parameters <NUM> can be expressed as a weighted summation of enhancement parameters from multiple preceding frames. The enhancement parameters <NUM> can be applied using a tone mapping function <NUM> (for example, a tone mapping function <NUM>) to enhance the frame <NUM>. In some embodiments where the frame <NUM> is enhanced based on multiple preceding frames, the enhancement parameters <NUM> for the different preceding frames are weighted according to how close or proximate the preceding frame is to the frame <NUM> in the image stream. Where the enhancement parameters from the <NUM>th-<NUM>th frames are used to enhance the <NUM>th frame, the enhancement parameters from each of the <NUM>th <NUM>th, and <NUM>th frames may be weighted differently when used to enhance the <NUM>th frame. For example, the <NUM>th frame may be weighted at. <NUM>, the <NUM>th frame may be weighted at. <NUM>, and the <NUM>th frame may be weighted at <NUM>. For example, older frames in the image stream (relative to the frame <NUM>) may have smaller weights than newer frames in the image stream. In certain embodiments, the enhancement parameters <NUM> can increase or decrease across the <NUM>th -<NUM>th frames depending on the predicted trends. In some embodiments, the weights for all preceding frames sum to <NUM> or all preceding frames are weighted the same. In some embodiments, the processing of the block <NUM> occurs for each of the one or more preceding frames with equal weights, based on which the enhanced frame <NUM> is generated for the frame <NUM>.

As part of the processing of the block <NUM>, the controller <NUM> accesses the preceding frame <NUM>. In some embodiments, accessing the preceding frame <NUM> comprises the controller <NUM> requesting and receiving the preceding frame <NUM> from the memory <NUM>. In some embodiments, the preceding frame <NUM> is stored in a cache, such as a cache associated with the controller <NUM>, which may enable quicker access to the preceding frame <NUM> and corresponding enhancement parameters than the memory <NUM>.

<FIG> is an example data flow diagram <NUM> that combines the parallel processing of <FIG> with the processing algorithm of <FIG> to enable analyzing preceding frames <NUM> to generate the enhancement parameters to enhance the current frame <NUM>. The data flow diagram <NUM> includes many aspects that are similar to aspects shown in the data flow <NUM> of <FIG> and the data flow diagram <NUM> of <FIG>. Corresponding aspects between the data flows <NUM>, <NUM>, and <NUM> may have corresponding functionality and operations, and so forth. Thus, for one or more aspects of the data flow <NUM> that correspond to the aspects in the data flow <NUM> or the data flow <NUM>, corresponding description will not be duplicated for brevity.

The data flow <NUM> begins with the frame <NUM> as generated by the imaging device <NUM>. The frame <NUM> may represent a non-initial frame of the image stream generated by the imaging device <NUM> of the patient's eye <NUM>.

The data flow <NUM> includes the block <NUM> corresponding to processing, by the controller <NUM>, of the preceding frame <NUM> in the image stream that precedes the frame <NUM>.

As part of the processing of the block <NUM>, the controller <NUM> accesses the preceding frame <NUM> and divides the accessed preceding frame <NUM> into a number of blocks of a blocked frame <NUM>, as described above with respect to <FIG>.

The controller <NUM> then processes the blocks to calculate the first order statistics of the preceding frame, such as the mean pixel grayscale for each block and the standard deviation of pixel grayscale for each block. The controller <NUM> then interpolates the mean pixel grayscale and the standard deviation of pixel grayscale for the individual and neighboring blocks to generate corresponding interpolated values for the entirety of the preceding frame <NUM>. These interpolated mean grayscale and interpolated standard deviation values may correspond to the enhancement parameters for the preceding frame <NUM>.

In some embodiments, the processing of the blocks of the blocked frame <NUM> comprises calculating tone mapping parameters from subsets of pixels in one or more blocks of the blocked frame <NUM>. In some embodiments, the processing of the blocks comprises identifying one or more other parameters of each block of the blocked frame <NUM> that may be identified based on processing one or more aspects of the blocked frame <NUM>.

In some embodiments, when enhancing the image stream for display on the user interface <NUM>, each frame <NUM> of the image stream is processed to generate the enhancement, or tone mapping, parameters for a subsequent frame. Once a frame is processed, the enhancement parameters for that frame can be stored in the memory <NUM> or a similar data store. In some embodiments, the enhancement parameters for a frame can be stored instead of the storing or saving the corresponding frame, which may save system resources, processing time, and the like. Thus, when the frame <NUM> is enhanced based on one or multiple preceding frames of the image stream, the controller <NUM> may retrieve the enhancement parameters corresponding to the one or multiple preceding frames <NUM> from the memory <NUM> or similar data store instead of or in addition to the one or multiple preceding frame <NUM>.

In some embodiments, the processing in the block <NUM> to identify the tone mapping parameters for the preceding frame <NUM> in the image stream may be performed in parallel with the generation of the current frame <NUM> by the imaging device <NUM>. Thus, when the frame <NUM> is received by the controller <NUM> for processing, the controller <NUM> has processed the preceding frame <NUM> to identify the corresponding enhancement parameters. By performing the processing of a preceding frame <NUM> in parallel with the generation of the current frame <NUM>, the image processing system <NUM> is able to enhance frames in the image stream in real time for prompt and timely display to the medical practitioner. The controller <NUM> does not need to spend time processing the frame <NUM>, but only apply the tone mapping, or similar, function to the frame <NUM> based on the enhancement parameters from the preceding frame <NUM>.

In some embodiments, the processing of the preceding frame <NUM> comprises performing downsampling. Downsampling may correspond to processing the preceding frame <NUM> to reduce a size or amount of data associated with the preceding frame <NUM>. The frames of the image stream are generally spatially smooth (meaning they include much fewer sharp edges as compared to frames that are not part of the image stream). As such, it may not be necessary to use all the pixels within a block of a frame of the image stream to calculate the first order statistics for the block. Thus, a subset of pixels may be sufficient for the calculation of the first order statistics, which can be done with less computation. Thus, the downsampling may improve speed of the processing of the preceding frames and enhancement of the current frame.

<FIG> is a sequence diagram <NUM> illustrating communications exchanged between or processing performed by components of the system <NUM> of <FIG> to obtain and process frames of an image stream and generate enhanced frames based thereon, according to example embodiments described herein. While the sequence diagram <NUM> and corresponding description include reference to components of the system <NUM> of <FIG>, the steps of the sequence diagram <NUM> are not limited to that example embodiment and may apply to various other combinations of components. Furthermore, the sequence diagram <NUM> is not required to perform each of or only the shown steps and is not limited to performing the indicated steps in any particular order.

The sequence diagram <NUM> depicts interactions between the imaging device <NUM>, the controller <NUM>, the memory <NUM>, and the user interface <NUM>. The sequence diagram <NUM> begins at communication step <NUM> with the imaging device <NUM> capturing a frame, such as the frame <NUM> of <FIG>, of the patient's eye <NUM>. The captured frame may be a non-initial frame of an image stream, as introduced above.

At communication <NUM>, the imaging device <NUM> provides the captured frame <NUM> to the controller <NUM> for processing and enhancement.

In some embodiments, simultaneous to one or more of the imaging device <NUM> capturing the frame <NUM> or providing the captured frame to the controller <NUM>, the controller <NUM> may process a preceding frame, such as the preceding frame <NUM> of <FIG>, to calculate the corresponding enhancement parameters for the preceding frame. In certain embodiments, though not shown in the sequence diagram <NUM>, calculating the corresponding enhancement parameters comprises requesting and receiving the preceding frame from the memory <NUM> before calculating the enhancement parameters for the preceding frame, as described above. As part of or instead of this request for the preceding frame, the controller <NUM> may request the enhancement parameters for the preceding frame from the memory <NUM>.

In some embodiments, the controller <NUM> requests or calculates the enhancement parameters for more than one preceding frame at processing <NUM>. In some embodiments, instead of sending a request for the preceding frames or corresponding enhancement parameters and received the requested frames or parameters, the controller <NUM> may access the requested preceding frames or corresponding enhancement parameters from a cache memory.

At processing <NUM>, the controller <NUM> may generate the tone mapping or similar function based on the obtained preceding frames or corresponding enhancement parameters and the frame received from the imaging device <NUM> and for which the controller <NUM> is generating the enhanced frame, such as the enhanced frame <NUM>.

At processing <NUM>, the controller <NUM> may apply the generated tone mapping function to the pixels of the frame received from the imaging device <NUM> to generate the enhanced frame.

At communication <NUM>, the controller <NUM> provides the enhanced frame to the user interface <NUM> for display to the medical practitioner.

At communication <NUM>, the controller <NUM> stores the enhanced frame or the enhancement parameters corresponding to the enhanced frame in the memory <NUM> or a corresponding cache memory.

At communications <NUM> and <NUM>, the imaging device <NUM> generates or captures a next frame of the patient's eye <NUM> and provides the next frame to the controller <NUM>, similar to communications <NUM> and <NUM> above. At processing <NUM>, the controller <NUM> begins calculating the frame enhancement parameters for the frame captured at communication <NUM> to be used for enhancement of the next frame from communication <NUM>. The processing <NUM> may correspond to the processing <NUM>.

The communication and processing steps <NUM>-<NUM> can be repeated for individual frames of the image stream of the patient's eye.

<FIG> provides details regarding a tone mapping function (for example, the tone mapping function <NUM> of <FIG>) described herein as an example function with which the enhanced frame is generated based on the enhancement parameters from a preceding frame and the frame, according to example embodiments described herein. In some embodiments, the frame that is enhanced (for example, the frame <NUM>) is captured as a color (RGB or similar) frame and is separated into a luminance portion (for example, grayscale from sRGB, or intensity) and a color portion (for example, an RGB ratio, and so forth). The luminance portion is the portion of the frame <NUM> that is enhanced, for example, using the tone mapping function <NUM>, which can improve the contrast in the enhanced frame <NUM>.

The graph <NUM> depicts a graphical representation of input frame features corresponding to intensities of blocks of the frame (along the x-axis) that are output (along the y-axis) as output frame features to a display device, such as the user interface <NUM> of <FIG>) with a uniform linear relationship <NUM> depicted between the input and output frame features. However, this uniform linear relationship is not representative of pixel intensity within blocks of actual frames generated in the image stream. The frame features may correspond to a frame generated by an imaging device (for example, the imaging device <NUM> of <FIG>).

The graph <NUM> depicts a graphical representation of the input and output frame features, where the blocks do not have the uniform pixel intensities across the input features. Therefore, the relationship <NUM> between the input and output features varies (for example, has a different slope) for different ranges of values of the function. The relationship <NUM> represents a <NUM>-segment function (corresponding to the tone mapping function <NUM>) that utilizes the first order statistics of the preceding frame <NUM> to emphasize the majority of pixels (i.e., those pixels having intensities that fall within the +/- <NUM> standard deviations of the mean pixel grayscale). The calculated first order statistics of the mean pixel grayscale and the standard deviation of pixel grayscale indicate that a majority of pixels in the block (or corresponding frame portion) are contained within a range of +/- <NUM> standard deviations from the mean pixel grayscale. Thus, to make the current frame more clear, the pixels that fall within the +/- <NUM> standard deviations of the mean pixel grayscale are enhanced to occupy more dynamic range in the output. Specifically, the three segments of the relationship <NUM> can be established by the mean and standard deviation pixel grayscale values described above. Furthermore, the <NUM>-segment function comprises two bounding lines that limit the slopes of the segments <NUM>-<NUM> in the relationship <NUM>, the bounding lines having slopes of contrast maximum (Cmax) and contrast minimum (Cmin) values, corresponding to the contrast limit.

As shown, the relationship <NUM> between the input frame features and output frame features has three segments with different slopes. By calculating the mean and standard deviation of the intensities (or grayscale) of the input frame features, as described herein, the controller <NUM> can identify a range of intensities in which a majority of the input frame features fall. In order to make the frame that includes the input features clearer and improve contrast, etc., in the frame, the pixels that fall within the majority range are made to occupy more dynamic range, corresponding to segment <NUM> (which will occupy more dynamic range) in the relationship <NUM>. This is reflected by the segment <NUM> having a greater slope as compared to the slopes of segments <NUM> and <NUM>. A contrast limit having maximum and minimum values may limit maximum and minimum slopes for the relationship <NUM>. The contrast limit may limit, for example, the tone mapping function or other frame enhancement algorithms.

Segments <NUM> and <NUM> represent regions where dynamic range will be sacrificed to trade for gain in segment <NUM>. The maximum slope of the segment <NUM> is limited by the Cmax slope, and the minimum slopes of the segments <NUM> and <NUM> are limited by the Cmin slope. Thus, the slopes of segments <NUM>-<NUM> cannot exceed the Cmax and Cmin slopes, such that the slope of segment <NUM> cannot be steeper than the slope of the Cmax slope and the slopes of segments <NUM> and <NUM> cannot be shallower than the slope of the Cmin slope. The enhancement parameters generated here (such as the enhancement parameters <NUM>) will be attenuated to meet the Cmax and Cmin limits.

<FIG> depicts a method <NUM> for generating, enhancing, and displaying frames of an image stream of a patient's eye for use during a surgical procedure, according to embodiments of the present disclosure. For example, the method <NUM> may be performed by one or more components of the system <NUM> of <FIG>, such as the imaging device <NUM>, the controller <NUM>, and so forth.

Block <NUM> of the method <NUM> comprises generating an image stream comprising a first frame capturing a branch of veins in an eye of a patient and a second frame capturing the branch of veins in the eye, the second frame following the first frame in the image stream. In some embodiments, the first and second frames are sequential frames in the image stream, though the first and second frames may also be separated by one or more other frames in the image stream.

Block <NUM> of the method <NUM> comprises calculating first order statistics for each block of a plurality of blocks for the first frame, each block of the plurality of blocks for the first frame comprising a plurality of first frame pixels. In some embodiments, the first order statistics correspond to the mean pixel grayscale or intensity and the standard deviation of pixel grayscale or intensity described above, though they first order statistics could correspond to other enhancement parameters. In some embodiments, calculating the first order statistics for blocks of the first frame comprises dividing the first frame into a number blocks and processing the blocks to generate the enhancement parameters and corresponding first order statistics.

Block <NUM> comprises interpolating first order statistics for the first frame based on the calculated first order statistics for each block of the plurality of blocks for the first frame. The first order statistics for the first frame may be based, at least in part, on the branch of veins. This may comprise the controller <NUM> generating the enhancement parameters for the first frame based on the enhancement parameters for the blocks making up the first frame.

Block <NUM> comprises generating a tone mapping function (such as the tone mapping function <NUM>), or similar enhancement function, for pixels of the second frame based on the interpolated first order statistics for the first frame. The tone mapping function, as described herein, may allow the controller <NUM> to apply the enhancement parameters of the first frame to pixels of the second frame. As introduced above, the tone mapping function may enable the controller <NUM> to map a first set of colors or values, such as the first order statistics introduced above, to different colors or values. Thus, where the first order statistics of the mean pixel grayscale and the standard deviation grayscale are for the first frame, the tone mapping function may enable mapping of those grayscale values to corresponding values for the second frame.

Block <NUM> comprises calculating tone mapping values for individual pixels of the second frame based on the tone mapping function and the corresponding enhancement parameters of the first frame.

Block <NUM> comprises generating an enhanced frame based on application of the calculated tone mapping values for individual pixels of the second frame to the pixels of the second frame.

In some embodiments, the method <NUM> further comprises calculating first order statistics for each block of a plurality of blocks for a third image, each block of the plurality of blocks for the third frame comprising a plurality of third frame pixels. In some embodiments, the method <NUM> also comprises interpolating first order statistics for the third frame based on the calculated first order statistics for each block of the plurality of blocks for the third frame. The method <NUM> also generates first weighted statistics for the first frame based on application of a first weight to the interpolated first order statistics for the first frame and generates second weighted statistics for the third frame based on application of a second weight to the interpolated first order statistics for the third frame, the second weight being less than the first weight. In certain embodiments, generating a tone mapping function for pixels of the second frame based on the interpolated first order statistics for the first frame comprises the image processor configured to generate the tone mapping function for the pixels of the second frame based on the first weighted statistics and the second weighted statistics.

In some embodiments, the method <NUM> comprises identifying a static object based on detection of the static object at a first location in the first frame and detection of the static object at a corresponding location in the third frame. In some embodiments, the method further comprises identifying a non-static object based on detection of the non-static object in only one of the first frame and the third frame or based on detection of the non-static object at a first location in the first frame and a second location different from the first location in the third frame. In some embodiments, the first order statistics for the first frame comprise a mean pixel intensity for each block of the plurality of blocks for the first frame and an intensity standard deviation for each block of the plurality of blocks for the first frame.

In some embodiments, the method <NUM> also comprises dividing the first frame into the plurality of blocks. In some embodiments, each block of the plurality of blocks has a same size and shape and is evenly spaced through the first frame relative to other blocks of the plurality of blocks. In some embodiments, the method further comprises identifying a portion of the first frame that comprises space void of relevant aspects of the object and preventing any blocks of the plurality of blocks from being placed in the portion that comprises a white (or black or empty) space. In the method <NUM>, the first order statistics may correspond to a luminance channel of the first frame and the tone mapping function and the tone mapping values apply to a luminance channel of the second frame.

In some embodiments, the first order statistics correspond to a grayscale channel obtained from green pixels of the first frame and the tone mapping function and the tone mapping values apply to a grayscale channel from green pixels of the second frame. In some embodiments, the image capture component comprises a high dynamic range (HDR) camera. In some embodiments, the imaging device comprises an ophthalmic imaging instrument and the object comprises a patient's eye, or a portion thereof. In some embodiments, the tone mapping function is limited based on a contrast limit determined based on the first frame and the contrast limit comprises a maximum contrast value and a minimum contrast value.

The method <NUM> may further comprise calculating the first order statistics for each block of the plurality of blocks for the first frame based on a mean pixel intensity for each adjacent block adjacent to the block and an intensity standard deviation for each adjacent block adjacent to the block. In some embodiments, each non-edge block of the plurality of blocks comprises four adjacent blocks, each edge block of the plurality of blocks comprises at least one duplicated block along an edge of the frame to create four adjacent neighbor blocks, and the mean pixel intensity and the intensity standard deviation for each duplicated block of the at least one duplicated block are duplicated from an adjacent block along an opposite edge of the edge block from the duplicated block.

<FIG> is a diagram of an embodiment of a computing system <NUM> that may be representative of one or more of the imaging device <NUM>, the controller <NUM>, and the like that performs or embodies certain aspects described herein. Specifically, the computing system <NUM> may be configured to perform processing or methods illustrated with respect to one or more of the system <NUM>, the data flows <NUM>, <NUM>, and <NUM>, the sequence diagram <NUM>, and the method <NUM>.

<FIG> illustrates a computing system <NUM> where the components of the system <NUM> are in electronic communication with each other, for example, via a system bus <NUM>. The bus <NUM> couples a processor <NUM> to various memory components, such as a read only memory (ROM) <NUM>, a random access memory (RAM) <NUM>, and the like (e.g., PROM, EPROM, FLASH-EPROM, and/or any other memory chip or cartridge). The system <NUM> may further include a cache <NUM> of high-speed memory connected to, in close proximity to, or integrated with the processor <NUM>. In some embodiments, the system <NUM> may access data stored in the ROM <NUM>, the RAM <NUM>, and/or one or more storage devices <NUM> through the cache <NUM> for high-speed access by the processor <NUM>. The preceding frames in the image stream that (or the corresponding enhancement parameters) in the image stream that are used to enhance a subsequent frame may be stored in the cache <NUM>.

In some embodiments, the one or more storage devices <NUM> store software modules, such as software modules <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and the like. When executed by the processor, the software modules <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> cause the processor <NUM> to perform various operations or methods, such as the processes described herein. In some embodiments, one or more of the software modules <NUM>, <NUM>, <NUM>, <NUM>, or <NUM> includes the ML models or other algorithms described herein.

The software module <NUM> comprises instructions (for example, in the form of computer-readable code) that program the processor <NUM> to identifying preceding frame(s) with respect to a current or received frame, such as the frame <NUM>. The software module <NUM> comprises instructions that program the processor <NUM> to identify relevant blocks in the preceding frame being processed to generate enhancement parameters, where the relevant blocks include portions of a target subject of interest to, for example, the medical practitioner. The software module <NUM> comprises instructions that program the processor <NUM> to estimate enhancement parameters for the preceding frame, as described above. In some embodiments, the software module <NUM> comprises instructions that program the processor <NUM> to obtain the enhancement parameters from a memory, such as the cache <NUM> or another memory.

The software module <NUM> comprises instructions that program the processor <NUM> to generate an enhancement algorithm, such as the tone mapping algorithm described above. The software module <NUM> comprises instructions that program the processor <NUM> to generate an enhanced frame of the frame received initially based on application of the tone mapping algorithm to the current frame received from the imaging device.

Although the system <NUM> is shown with only one processor <NUM>, the processor <NUM> may be representative of one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), tensor processing units (TPUs), and the like. In some examples, the system <NUM> may be implemented as a stand-alone subsystem, as a board added to a computing device, as a virtual machine, or as a cloud-based processing machine.

To enable user interaction with the system <NUM> or communications between systems, the system <NUM> includes a communication interface <NUM> and input/output (I/O) devices <NUM>. In some examples, the communication interfaces <NUM> includes one or more network interfaces, network interface cards, and the like to provide communication according to one or more network or communication bus standards. In some examples, the communication interface <NUM> includes an interface for communicating with the system <NUM> via a network. In some examples, the I/O devices <NUM> may include on or more user interface devices (e.g., graphical user interfaces (e.g., user interface <NUM>), keyboards, pointing/selection devices (e.g., mice, touch pads, scroll wheels, track balls, touch screens, and/or the like), audio devices (e.g., microphones and/or speakers), sensors, actuators, display devices, and the like).

Each of the one or more storage devices <NUM> may include non-transitory and nonvolatile storage such as that provided by a hard disk, an optical medium, a solid-state drive, and the like. In some examples, each of the one or more storage devices <NUM> is co-located with the system <NUM> (for example, a local storage device) or remote from the system <NUM> (for example, a cloud storage device).

The preceding description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein are not limiting of the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments. For example, changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. In addition, features described with respect to some examples may be combined in some other examples. In addition, the scope of the disclosure is intended to cover such an apparatus or method that is practiced using other structure, functionality, or structure and functionality in addition to, or other than, the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein might be embodied by one or more elements of a claim.

In addition, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like.

Further, the various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions.

Claim 1:
An ophthalmic imaging device (<NUM>) comprising:
an image capture component configured to generate an image stream comprising a first frame (<NUM>) capturing a branch of veins in an eye of a patient and a second frame capturing the branch of veins, the second frame following the first frame in the image stream; and
an image processor (<NUM>) configured to:
calculate (<NUM>) first order statistics for individual blocks (224a-224i) of a plurality of blocks (<NUM>) for the first frame (<NUM>), each block of the plurality of blocks for the first frame comprising a plurality of first frame pixels;
interpolate (<NUM>) first order statistics (<NUM>, <NUM>) for the first frame (<NUM>) based on the calculated first order statistics (<NUM>, <NUM>) for the individual blocks of the plurality of blocks for the first frame, wherein the first order statistics for the first frame are based, at least in part, on the branch of veins;
generate (<NUM>) a tone mapping function for each pixel of the second frame based on the interpolated first order statistics for the first frame;
calculate (<NUM>) tone mapping values for individual pixels of the second frame based on the tone mapping function; and
generate (<NUM>) an enhanced frame (<NUM>) based on application (<NUM>) of the calculated tone mapping values for individual pixels of the second frame to the pixels of the second frame.