Patent Description:
High-resolution imaging techniques such as magnetic resonance imaging or fluorescence microscopy allow capturing time lapses of objects such as biological specimen or in-vivo biological tissue. Capturing high-resolution time lapses of biological structures is challenging because high frame rates are required to capture relatively fast movements, while also protecting biological structures from damage incurred by the microscopy method.

For example, time lapse fluorescence microscopy of biological specimens on the one hand requires high frame rates to allow for capturing fast movements of the cells and on the other hand requires minimizing the excitation laser intensity to reduce cell damage. In addition, photo bleaching leads to loss of fluorescence imaging. Such opposing requirements often lead to very small signal-to-noise ratios of the acquired images. Other microscopy methods suffer from similar problems. In confocal microscopy the low signal-to-noise ratio is even more prominent because light excluded by the pin hole does not contribute to the images. STED microscopy also suffers from the described problems because it has an intrinsically low photon budget due to depletion of large parts of the excitation at the focal point.

To improve signal-to-noise of image frame sequences, time-dependent combinations of the image frames may be applied. For example, rolling average with exponential weighting may be applied, such as according to <MAT> where I is the image frame sequence, σ is a width, w is a window size, Θ is the Heaviside function, and * denotes a convolution operation. However, because rolling averaging involves applying a blurring kernel, fast movements of objects captured in the image frame sequence are smeared. Rolling averaging therefore cannot be applied to improve signal-to-noise of image frame sequences that capture fast moving objects.

Another approach for improving signal-to-noise in image frames is applying recent developments in deep learning methods that now readily offer solutions for de-noising of images. In such approaches, a single image is inputted without considering movement of captured objects in the images. However, it is well known that a significant problem of deep learning algorithms is the creation of inadequate details, so-called hallucinations, from random patterns like noise. Such artifacts become particularly visible in noisy time lapse images.

<FIG> illustrates the problem of hallucinations in a de-noising approach of the state of the art. Column <NUM> of <FIG> reproduces two successive fluorescence microscopy image frames captured from a live biological specimen. Applying a de-noising algorithm of the state of the art, Nikon's denoise. ai, to the image frames of column <NUM> yields the image frames in column <NUM>. As is evident, the de-noising algorithm infers shapes of objects that seem realistic from the very noisy images in column <NUM>. However, a comparison of the de-noised images in column <NUM> for the successive image frames shows that shapes and positions of the predicted objects change substantially from the upper frame to the lower frame, casting strong doubt on the veracity of the predicted object shapes. Hence, applying de-nosing strongly depend on the temporal realization of noise, which implies that de-noising cannot be applied reliably to noisy time lapses.

<FIG> illustrates the problem of blurring when applying weighted rolling average according to the state of the art. Panel <NUM> illustrates an image frame of a simulated image frame sequence capturing a fast-moving object. As indicated by the white arrow, the object moves with a velocity v = (<NUM>, -<NUM>) pixel per frame. Panel <NUM> reproduces results of applying a weighted rolling average according to Equation (<NUM>) with σ = <NUM> and w = <NUM>. As shown in panel <NUM>, the object is blurred due to contribution of neighboring image frames, heavily distorting the object's true shape.

<NPL>, derives and analyzes a variational model for joint estimation of motion and a reconstruction of image sequences. The existence of a minimizer is proved and numerical solutions of the model based on primal-dual algorithms are discussed.

<CIT> discloses an interpolation frame producing method including a frame rate converting operation. The method comprises a motion vector detecting step, a frame rate calculating step based upon the detected motion vector, and an interpolation frame producing step for producing an interpolation frame based on the calculated frame rate.

The object of the present invention is to improve image quality of microscopy frames.

This object is solved by the subject-matter of the independent claims.

Embodiments of the present invention are defined by the dependent claims.

According to an embodiment, a method according to claim <NUM> is provided.

According to another embodiment, estimating the representative velocity of the optical flow comprises calculating a histogram of the optical flow between subsequent images frames in the image frame sequence and analyzing the histogram to determine the representative velocity. According to an aspect, analyzing the histogram to determine the representative velocity comprises employing the histogram to determine the representative velocity as a quantile for a predetermined threshold value. Calculating the histogram may be based on estimating a pixel-wise optical flow, such as by a method based on Farnebäck's algorithm.

The method for improving signal-to-noise in image frame sequences is applied for microscopy image frame sequences.

According to an aspect, the trained artificial neural network employed in the method for improving signal-to-noise in image frame sequences involves a feature reshaping operation with channel attention. In embodiments, the feature reshaping operation is a pixel shuffle operation.

In embodiments, the interpolation factor is a power of two and the trained artificial neural network is configured for recursively generating and adding interpolating image frames to the image frame sequence, wherein the number of recursions corresponds to the power.

According to yet another aspect, computing a time-dependent combination of image frames comprises applying a rolling average to the expanded image frame sequence. The rolling average may be a weighted rolling average and wherein applying the weighted rolling average comprises determining parameters for the weighted rolling average from the representative velocity. According to an aspect, the parameters for the weighted rolling average are a window size determining a group of image frames from the expanded image frame sequence that contribute to the weighted rolling average, and a width determining a weight by which each image frame from the group contributes to the weighted rolling average.

According to an aspect, the method for improving signal-to-noise in image frame sequences comprises applying a de-noising algorithm to the output image frame sequence.

According to an embodiment, the artificial neural network has been trained by pre-training the artificial neural network with image frame sequences that are not domain specific and training the artificial neural network with domain-specific image frames.

According to an yet another embodiment an image processing device for improving the signal-to-noise of image frames as in claim <NUM> is disclosed.

<FIG> illustrates steps of a method <NUM> for improving a signal-to-noise in a sequence of image frames. The image frames of the sequence correspond to two-dimensional, three-dimensional, or even higher-dimensional image data that may comprise several color channels. In embodiments, the image frames relate to a microscopy time lapse. For example, the microscopy time lapse may capture living biological specimens and their motion. In a particular embodiment, the image frames may be captured by a dedicated camera for fluorescence imaging integrated in a surgical microscope.

Method <NUM> comprises estimating <NUM> a representative velocity vrepr from the image frames sequence. A motion of objects captured in the image frames implies a shift of the position of the objects from one image frame to the next image frame. In the present context, the camera is held at fixed position and the velocity is due to intrinsic motion of the captured objects, e.g. of biological motion. The estimated representative velocity may be a velocity near to a maximum velocity of motion of objects captured in the image frames.

In embodiments, step <NUM> of estimating a representative velocity vrepr may comprise calculating a histogram of an optical flow between image frames in the image frame sequence. Calculating the histogram may involve determining values of a dense optical flow vopt between each pair of subsequent image frames in the image frame sequence. In an embodiment, the dense optical flow may be determined pixel-wise by employing Farnebäck's algorithm as described in <NPL>. The histogram may correspond to a histogram in the absolute values |vopt| of the optical flow.

<FIG> shows an exemplary histogram of the optical flow in an image frame sequence. Typically the histogram is peaked at zero because the image background does not move from frame to frame. In embodiments, the representative velocity vrepr is determined as a quantile for a predetermined threshold value <MAT>.

Employing the quantile is sufficient for the purpose of the present invention because this approach minimizes the effect of outliers. Typically the threshold value of p=<NUM> is employed so that <NUM>% of all optical flow values |vopt| are smaller than the representative velocity. In the example of <FIG>, the estimated representative velocity is <NUM> pixels per frame.

Again referring to <FIG>, method <NUM> further comprises step <NUM>, in which the determined representative velocity is employed to determine an interpolation factor αinterp. The interpolation factor corresponds to a desired improvement in the frame rate such as two or four.

In embodiments, the interpolating factor αinterp may be determined such that vrepr/αinterp ≲ <NUM> pixel/frame. In embodiments, the interpolating factor is determined as a power of <NUM> according to <MAT>.

Method <NUM> further includes generating <NUM> interpolating microscopic frames employing a trained artificial neural network. Generating <NUM> interpolating microscopic frames involves generating a predetermined number of interpolating image frames, wherein each interpolating image frame interpolates between subsequent image frames of the image frame sequence. Generating <NUM> interpolating microscopic frames may yield an expanded image frame sequence with a frame rate corresponding to the interpolation factor determined in step <NUM>.

As a result of a requirement of vrepr/αinterp ≲ <NUM>, the remaining optical flow between successive image frames in the expanded image frame sequence is lower than one pixel per frame and rolling average may be employed to improve signal-to-noise in the expanded image frame sequence without creating the spurious effects discussed above.

In particular, the approach according to steps <NUM> to <NUM> of method <NUM> involves determining an optical flow to determine an interpolation factor. However, in embodiments described, determining interpolating image frames corresponding to the interpolation factor does not rely on determining an optical flow. In embodiments, generating <NUM> interpolating microscopic frames may employ an artificial neural network that does not rely on estimating an optical flow, as described below with reference to <FIG>. Not relying on estimating an optical flow for generating the interpolating image frames allows reducing latency of the step of generating the interpolating image frames.

In embodiments, generating <NUM> interpolating microscopic frames involves recursive generation of interpolating image frames. In these embodiments, the interpolation factor is determined as a power of <NUM>. The input image frame sequence may be processed a first time to create one interpolating image frame between each successive image frames of the input image frame sequence to yield a first extended microscopic frame sequence with a double frame rate. The first extended image frame sequence may again be processed by the trained artificial neural network to again generate an interpolating image frame between each successive microscopic image frames of the first extended image frame sequence to yield a second extended image frame sequence with a frame rate four times the frame rate of the input image frame sequence. Hence, the frame rate of the extended image frame sequences may be doubled until the desired multiplicity of the interpolation factor of αinterp is reached.

Method <NUM> further comprises processing <NUM> the expanded image frame sequence by applying a time-dependent combination of the image frames in the expanded image frame sequence, such that the resulting enhanced image frame sequence had improved signal-to-noise. In embodiments, applying the time-dependent combination of the image frames may be based on a weighted rolling average. According to other embodiments, applying the time-dependent combination of the image frames may be based on applying a Kálmán-filter.

In a further embodiment, processing <NUM> the expanded image frame sequence may include applying a weighted rolling according to Equation (<NUM>). To avoid smearing of objects under application of Equation (<NUM>), the velocity vobj of moving objects in the image frame sequence should be vobj ≲ <NUM> pixel per frame. Moreover, σ, w ≲ <NUM>/vobj should be satisfied.

Specifically, the parameters for weighted rolling average in Equation (<NUM>) may be determined from vrepr determined in step <NUM> according to <MAT>.

Expanding the image frame sequence with an appropriate number of interpolating image frames and then applying a weighted rolling average as explained above therefore corresponds to applying a motion-aware rolling average.

In embodiments, method <NUM> may further comprise down-sampling <NUM> the enhanced image frame sequence yielded from step, such as, down-sampling the frame rate of the image frame sequence to correspond to the frame rate of the original image frame sequence. Down-sampling the enhanced image frame sequence may hence involve selecting every αinterp-th image frame of the enhanced image frame sequence. For example, when the original image frame sequence is composed of [f<NUM>, f<NUM>,. , fn], and αinterp = <NUM> is determined in method step <NUM>, performing step <NUM> yields expanded image frame sequence [f<NUM>, f<NUM>, f<NUM>, f<NUM>,. , fn], and performing step <NUM> yields enhanced image frame sequence [r<NUM>, r<NUM>, r<NUM>, r<NUM>,. , rn], from which [r<NUM>, r<NUM>,. rn] may be selected as an output image frame sequence with improved signal-to-noise ratio.

In embodiments, method <NUM> may further comprise applying <NUM> a de-noising algorithm on the enhanced and optionally down-sampled image frame sequence to further boost signal-to-noise. De-noising algorithms of the state of the art only consider single image frames without considering motion of objects depicted. In embodiments, applying the de-noising algorithm may include applying a convolutional neural network. Because de-noising is applied subsequent to forming a time-dependent combination of the image frames, the problem of hallucinations as discussed above with reference to <FIG> is reduced. In particular, applying de-noising algorithms to the enhanced image frame sequence synergistically combine de-noising approaches and interpolation approaches.

<FIG> illustrates the architecture of the chained artificial neural network <NUM> configured for performing step <NUM> of generating interpolating microscopic frames corresponding to αinterp.

Artificial neural network <NUM> may be selected as a lean artificial neural network. In particular, artificial neural network <NUM> may lack sub-modules dedicated to estimating an optical flow. Thereby, the artificial neural network <NUM> may provide low latency in performing step <NUM> of generating the interpolating image frames. Embodiments of the present disclosure may hence involve real-time processing of image frame sequences for signal-to-noise improvement.

Because optical flow determination is very sensitive to noise and because microscopy images, in particular, are affected by higher noise than usual photography images, usual frame interpolation methods by optical flow estimation fail. In addition, artificial neural networks for frame interpolation based on optical flow rely on being trained with simulated videos, as described in<NPL>. However, in the context of the present application, large-scale annotated data, e.g. for biological time lapses, are unavailable.

Furthermore, artificial neural network <NUM> may be selected to be able to cope with large movements that imply temporal jittering and motion blurriness. In contrast, well-known methods such as described in <NPL> or the phase-space based algorithm of <NPL>, require that the shift of objects between two frames is small.

In embodiments, artificial neural network <NUM> involves a feature reshaping operation with channel attention that replaces optical flow computation modules of other approaches. The feature reshaping operation may correspond to a pixel shuffle operation. Artificial neural network may be configured for distributing information in a feature map of the image frames into multiple channels and extract motion information by attending the channels for pixel-level synthesis of an interpolating frame.

Specifically, artificial neural network <NUM> may be configured to receive input image frames 402a and 402b that are separately processed by down-shuffle operation <NUM>. Down shuffle <NUM> is an operation to reorganize an image frame by pooling the image frame with several switch variables to generate down-sampled image frames, while Up Shuffle <NUM> corresponds to performing the inverse procedure. Down shuffle <NUM> reduces the spatial dimensions of an image frame F ∈ RH×W × C by a factor of s to obtain an image frame F' ∈ RH/s×W/s × s<NUM>C. After applying down shuffle <NUM>, down-shuffled image frames 406a, 406b are concatenated in the channel direction in block <NUM>. The channel dimension is reduced by performing a convolution operation <NUM> followed by ResGroups operation <NUM> which consist of <NUM> residual channel attention blocks. After another convolution <NUM>, intermediate image frame <NUM> is yielded. Up shuffle <NUM> is applied to intermediate image frame <NUM> to yield interpolating microscopy image frame <NUM> that interpolates between image frames 402a and 402b. The artificial neural network may be configured as explained in<NPL>, which is herewith incorporated in its entirety.

<FIG> relates to a method of training an artificial neural network, for example, an artificial neural network configured as described above, for frame interpolation of image frames. In method step <NUM> the artificial neural network is trained on general, not domain-specific image frames. For example, benchmark data sets like Vimeo-<NUM> (<NPL>) or SNU-FILM (<NPL>) may be employed. Step <NUM> involves that the first few layers of the artificial neural network are trained on simple image features like edges, corners of blobs, which are universal, and not specific to particular images domains. Moreover, the step of pre-training in method step <NUM> with non-microscopy videos with less noise than image frames makes it easier for a network to learn these low-level features.

Method <NUM> further includes step <NUM> of fine-tuning the pre-trained artificial neural network by training with domain-specific images, such as microscopy images or MRT images. Transfer learning as described allows to fine tune the artificial neural network for the specific application so that the pre-trained artificial neural network is trained on complex features of biological objects.

In embodiments, training data may comprise image frames of time lapses acquired across diverse types of microscopes, such as bright field microscopes, wide-field fluorescence microscopes, confocal microscopes, STED microscopes and lightsheet microscopes. To train an artificial neural network <NUM> as described above for the domain of microscopy time lapses, for example <NUM>'<NUM> image frames of time lapses were employed as training data.

<FIG> relates to an image processing device for improving signal-to-noise in an image frame sequence. Image processing device <NUM> comprises memory <NUM> for saving an image frame sequence at least temporarily. The image frame sequence may be received in memory <NUM> via a data communication link from a microscope.

Image processing device <NUM> further comprises processing circuitry <NUM> configured for estimating a representative velocity of an optical flow in an image frame sequence, as has been explained in further detail above. Processing circuitry <NUM> may be configured for determining an interpolation factor from the representative velocity of the optical flow as explained above.

The components of image processing device <NUM> may further be configured to deliver the image frame sequence and the interpolation factor to trained artificial neural network <NUM> for generating interpolating image frames in accordance with the interpolation factor. Trained artificial neural network <NUM> may output an expanded image frame sequence. Processing circuitry <NUM> may further process the expanded image frame sequence by applying a time-dependent combination of image frames to improve signal-to-noise. Processing circuitry <NUM> may optionally be configured to apply a de-noising algorithm to the enhanced image frame sequence. The enhanced image frame sequence may be outputted to a monitor in real-time.

In a particular embodiment, processing device <NUM> is provided in a surgical microscope that is equipped with a fluorescence microscope for capturing emissions from a contrast agent in biological tissue. In this embodiment, processing device <NUM> may continuously operate to provide image frame sequences with improved signal-to-noise which may be outputted as overlays in a field-of-view of the surgical microscope.

<FIG> illustrates exemplary image frames that illustrate improvements of the present disclosure over the prior art in a simulated setting. <FIG> illustrates results for the simulated image frames of <FIG>. Panel <NUM> reproduce a result of applying motion aware rolling average according to method <NUM> to the image frame of panel <NUM>. Parameters for weighted rolling average, applied as part of motion aware rolling averaging, are the same as for panel <NUM> of <FIG>, with σ = <NUM> and w = <NUM>.

<FIG> illustrates applying a motion-aware rolling average to successive fluorescence microscopy image frames of <FIG>. The image frames of column <NUM> are part of a larger image frame sequence, on which the disclosed motion-aware rolling average may be applied. Column <NUM> reproduces image frames resulting from applying the disclosed motion-aware rolling average. As is evident from the illustration of <FIG> and <FIG>, motion aware rolling average allows increasing signal-to-noise without distorting the content of the image frames.

Methods and systems described hence allow for significantly improving signal-to-noise in noisy image frame sequences, while avoiding creation of artefacts from inherent motion of captured objects. Embodiments described provide for a low-latency solution that allows applying the signal-to-noise improvement in real time.

The implementation can be performed using a non-transitory storage medium such as a digital storage medium, for example a floppy disc, an HDD, an SSD, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed.

Some embodiments which are not according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments which are not according to the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.

Other embodiments which are not according to the claimed invention comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment which is not according to the claimed invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment not according to the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor.

A further embodiment not according to the claimed invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.

A further embodiment not according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver.

Claim 1:
A method for improving a signal-to-noise ratio of image frames, the method comprising:
estimating (<NUM>) a representative velocity of an optical flow in an image frame sequence containing the image frames, wherein the image frames are microscopy image frames
determining (<NUM>) an interpolation factor from the representative velocity of the optical flow;
employing (<NUM>) a trained artificial neural network for generating an expanded image frame sequence, wherein the expanded image frame sequence includes a number of interpolating image frames, wherein each interpolating image frame interpolates between subsequent image frames of the image frame sequence, wherein the number of interpolating image frames corresponds to the interpolation factor; and
computing (<NUM>) a time-dependent combination of image frames from the expanded image frame sequence to generate an output image frame sequence.