Patent Description:
Image denoising is used to process images contaminated by additive noise to achieve both noise reduction while preserving the features of the image. The inventor has identified that additive noise may be introduced when users try to minimize photo bleaching and photo toxicity by minimizing the energy input, which may be achieved by reducing the exposure time or gain at the cost of increasing the amount of noise. In a typical workflow, images are acquired at low exposure time or gain, resulting in low quality images, which may contain noise from electronic sources (e.g. Gaussian noise, salt and pepper noise, and shot noise) or sample dependent high frequency noise due to quantization, sensitivity or heat from camera sensors.

<NPL> disclose a convolutional neural network (CNN) based method to jointly estimate and remove high ISO noise. First, a CNN-based noise estimation method is disclosed to estimate a pixel-wise noise level. Second, the noise variance map is correlated with the image content.

<CIT> discloses an image processing method comprising the steps of dividing each training image of a dataset of training images into a plurality of training images patches; and of applying an image processing algorithm, over a range of possible tuning parameters, to each of the plurality of training image patches to generate a plurality of perforniance curves; and of constructing a loss function based upon the plurality of performance curves.

<NPL> disclose to apply basic statistical reasoning to signal re-construction by machine learning thereby it is intended to map corrupted observations to clean signals and to reduce noise in images. <NPL> discloses regularizing a training of a convolutional neural network. It is disclosed to separate a data set into patches. In each patch a transformation is applied such that pixels within the patch are shuffled.

A problem solved by the present invention is how to denoise images and obtain a good signal-to-noise (SNR).

This object is solved by the disclosed embodiments. The invention is delimited by the appended independent claims. The dependent claims provide information for further embodiments.

In an embodiment of the invention there is provided a computer implemented method for training a convolutional neural network, the method comprising: receiving a captured image; generating a statistical noise model based on the captured image; and training a convolutional neural network based on the captured image and the statistical model. Thus, the SNR can be increased when training the convolutional neural network using one, or more, acquired images.

In a possible implementation of the embodiment generating the statistical model comprises: determining variance or standard deviation values for pixels in the captured image according to a ground truth or simulated ground truth; and generating the statistical model based on the variance or standard deviation values. Hence, the convolutional neural network can be trained more efficiently and based on real acquired images, and may not utilize so-called training images.

In the embodiment training the convolutional neural network comprises: dividing the captured image in to random patches; swapping pixel values in each random patch with neighbour pixel values; and training the convolutional neural network based on the random patches of the captured image and the random patches of the captured image comprising swapped pixel values.

In a possible implementation of the embodiment training the convolutional neural network comprises: determining a probability of pixel values of the random patches comprising swapped pixel values based on the statistical model; and training the convolutional neural network by maximizing the probability of each pixel value of the random patches comprising swapped pixel values. Accordingly, the convolutional neural network can be trained and utilized using the first, and subsequently, acquired images without necessarily relying on training data.

In a possible implementation of the embodiment maximizing the probability of each pixel value comprises minimizing the negative log of an average of all pixel probabilities.

In a possible implementation of the embodiment the probability for each pixel value is determined based on a Gaussian distribution with a mean value and a variance or standard deviation value, wherein the mean value is a mean value of each pixel of the random patches of the captured image and the variance or standard deviation value is obtained from the statistical model. Alternative techniques for determining the variance values may include, determining variances of noise based on the counts of the camera (i.e. the gain) in an image, or modelling the fluorescence to determine the variance of any noise between an expected object and an obtained object in a specific frame with exposure time t. Generally, techniques can be used which create a model that spits the variance values of each pixel based either on camera gain or modelling the fluorescence of an imaged object.

In a possible implementation of the embodiment the method comprises applying the trained convolutional neural network to the captured image.

In a possible implementation of the embodiment the method comprising correcting each pixel probability of an image generated by the trained convolutional neural network using the statistical model, and averaging each generated pixel probability with each corrected pixel probability.

In the embodiment the method comprising: receiving a further captured image; updating the statistical model based on the captured image and the further captured image; determining if there is a reduction in a variance or standard deviation value of the updated statistical model compared with the variance model; and retraining the convolutional neural network based on the further captured image and the updated statistical model if it is determined that there is a reduction in a variance or standard deviation value of the updated statistical model compared with the statistical model. Hence, the performance of the convolutional neural network can be improved every time a new image is acquired.

In a possible implementation of the embodiment the convolutional neural network is an encoder-decoder neural network.

In an embodiment of the invention there is provided a system comprising one or more processors and one or more storage devices, wherein the system is configured to perform the methods described above.

In a possible implementation of the embodiment the system comprises an imaging device coupled to the processor for acquiring microscopy images.

In an embodiment of the invention there is provided a trained convolutional neural network trained by: receiving a captured image; generating a statistical noise model based on the captured image; and adjusting the convolutional neural network based on the captured image and the statistical model.

In an embodiment of the invention there is provided a computer program with a program code for performing the methods described above.

The present disclosure can be understood with reference to the description of the embodiments set out below, in conjunction with the appended drawings in which:.

The inventor has identified a way of understanding the problem of denoising is to consider ideal pictures (high signal to noise ratio, SNR) together with a mathematical distribution that adds random values to each pixel of the obtained image. This function can be modelled, for example, in the case of fluorescence, using a Poisson-Gaussian distribution where Poisson noise (also called shot noise) is the dominant source.

If a major part of the information is remaining in the image, the values of the noisy pixels can be estimated statistically from the neighbourhood of each pixel. If we consider a Gaussian distribution for each pixel, we have a mean value and the variance (or standard deviation). The bigger the variance (or standard deviation), the bigger the noise and the lower the SNR (i.e. poor image quality).

<FIG> illustrates a method for training a convolutional neural network according to an embodiment of the invention. The convolutional neural network forms part of an on-the-fly training system, which may be pretrained by a manufacturer, but is trained during its application at a customer. The pretraining performed by the manufacturer may include the application of the method of embodiments of the invention. A convolutional neural network is a form of machine learning algorithm or model.

In <FIG>, there are three overall processing steps. These steps are training <NUM>, processing <NUM>, and repeated training <NUM>.

In step <NUM>, an image <NUM> is acquired from a microscope, for example (i.e. a microscopy image). A simulated ground truth image <NUM> is obtained by applying Gaussian smoothing <NUM>, for example, to the acquired image <NUM>. Alternatives to Gaussian smoothing include average filtering (e.g. weighting by average pixels), bilateral filtering, non-local means (NLM) smoothing, and block-matching and 3D filtering (BM3D). If a ground truth image exists (i.e. the same image with a very high peak signal to noise ratio, PSNR) it may not be necessary to obtain a simulated ground truth. The acquired image <NUM> is compared to the simulated ground truth image <NUM>, and the variances of each pixel value are extracted to produce an average variance value for each value in a greyscale of <NUM>-<NUM> or a colour scale of <NUM>-<NUM>. The variance values may be determined using each value in a 2D histogram based on the acquired image <NUM> and the simulated ground truth image. A model <NUM> is subsequently fitted to the average variance values. The model might be a polynomial, an exponential or other suitable statistical model, which is derived by plotting the variance values. Alternative techniques for determining the variance values may include, determining variances of noise based on the counts of the camera (i.e. the gain) in an image, or modelling the fluorescence to determine the variance of any noise between an expected object and an obtained object in a specific frame with exposure time t. Generally, techniques can be used which create a model that spits the variance values of each pixel based either on camera gain or modelling the fluorescence of an imaged object.

<FIG> illustrates a graph of average variance for each value in a grey scale for one image. In the graph the "<NUM> images variance (noise vs ground truth)" thick solid line represents an ideal statistical model, the "<NUM> image respect variance noisy versus fake ground truth" thick solid line represents the values acquired by comparing the acquired image <NUM> against the simulated ground truth image <NUM>, and the "Fitted of <NUM> image" dashed line represents the statistical model fitted to the "One image fake GT".

Returning to <FIG>, the captured image <NUM> is cropped, or divided into random patches <NUM>. These patches <NUM> are typically square and represent sections of the acquired image <NUM> and each patch may overlap with neighbouring patches. The size of each patch is determined based on the size of the image and the processing power of the image processing device used, and might be up to 256x256 pixels, for example. A typical patch might be 64x64 pixels such that a 512x512 pixel image would be divided into <NUM> patches. Some pixels are selected from each random patch <NUM> using a uniform random distribution and the pixel values of the selected pixels are swapped or replaced by values from neighbouring pixels to produce random patches of the captured image comprising swapped pixel values <NUM>. Generally, between <NUM>% and <NUM>% of the pixels may be swapped or replaced within the patches Image <NUM> illustrates a magnified section of one of the random patches <NUM> and image <NUM> illustrates a magnified section of one of the random patches of the captured image comprising swapped pixel values <NUM>.

The random patches of the captured image comprising swapped pixel values <NUM> are subsequently used to train a convolutional neural network (CNN) <NUM>. In this example, a U-net is trained, although it will be appreciated that other contracting-expanding, or encoder-decoder CNNs might be used such as a Resnet (e.g. resnet-<NUM>). In particular, a dynamic U-net (encoder resnet-<NUM>) with a sub-pixel convolution (Pixel shuffle) in the merge layer may be used. The CNN <NUM> is trained based on the random patches of the captured image comprising swapped pixel values <NUM> and variance values obtained using the model <NUM>. The CNN <NUM> maximizes the probability that each pixel in the acquired image belongs in the acquired image (i.e. the CNN <NUM> maximizes the probability that each pixel in the acquired image is not a result of noise). This is achieved by minimizing the negative log of the average of all pixel probabilities based on the following expression <NUM>: <MAT> where n is the number of random patches, M is the number of pixels masked (i.e. swapped or replaced pixels), xm is the m-th pixel value in the image with replaced values <NUM> represented by <NUM> in <FIG>, and sm is the m-th pixel value in the random patches <NUM> of the acquired image <NUM> represented by <NUM> in <FIG>. Z-scores and probabilities are derived from a Gaussian distribution with mean values obtained using the pixel values of the random patches of the captured image and variance values obtained from the variance model.

In step <NUM>, the trained CNN <NUM> is applied to denoise an acquired image <NUM>, which may correspond to the acquired image <NUM>, or similar images thereto, to obtain a denoised image <NUM>.

In step <NUM>, a new, further image <NUM> is acquired, and a new statistical model <NUM> is generated based on the variance values of the acquired image <NUM> and the acquired further image <NUM>. The new statistical model <NUM> is compared to the previous statistical model <NUM> generated in step <NUM>, and a decision <NUM> is made as to whether or not there is a reduction in any variance value in the statically model. If there is a reduction in a variance value in the new statistical model <NUM> the CNN <NUM> is retained <NUM> using the further acquired image <NUM> and the new statistical model <NUM>, and any previously acquired images, such as acquired image <NUM>. The retrained CNN is subsequently applied to the acquired further image <NUM>.

If there is no reduction in a variance value in the new statistical model <NUM> the further acquired image <NUM> is denoised <NUM> based on the CNN <NUM> without further training to obtain denoise image <NUM>.

Additionally, each pixel probability of any denoised images <NUM>, <NUM> may be estimated and corrected using a current statistical model and being averaged with itself.

It may also be desirable to update <NUM> any previously denoised images based on the latest CNN and statistical model, and to retrain the CNN <NUM> based on previously acquired images and any new statistical model.

Step <NUM> is repeated until there is no detectable reduction in a new statistical model when compared to a previous statistical model. In this regard, Step <NUM> is repeated until the differences between the previous and the new models are <NUM> +/- a tolerance (e.g. a tolerance of +/- <NUM> to <NUM>%, +/- <NUM> to <NUM>%, +/- <NUM> to <NUM>%, +/- <NUM> to <NUM>%, or +/- <NUM> to <NUM>%).

<FIG> illustrates a graph of variance against each value in a grey scale for <NUM> images. In the graph the "<NUM> images variance (noise vs ground truth)" solid line represents the variance data obtained from <NUM> images, and the "<NUM> images curve fitting" dashed line represents the statistical model fitted to the "<NUM> images variance (noise vs ground truth)" line. It is apparent that after <NUM> images, for example, there is no longer a reduction in the variance, and the model no longer needs to be updated. However, it will be appreciated that fewer images might be used, such as <NUM> images.

If a system performing the above mentioned method acquires new successive images or its parameters are updated (laser power, acquisition speed, exposure time change, etc.), the new "noisy" images may be evaluated. If the variances have reduced with respect to a previous statistical model, then the model can be updated using the methods described herein. Thus, the quality of the images can be improved iteratively along the course of an experiment, for example.

Reference has been made herein to a variance value, but it will be understood that a standard deviation value might also be used.

<FIG> illustrates a method <NUM> for training a convolutional neural network according to a further embodiment of the invention. The method comprises receiving a captured image <NUM>; generating a statistical noise model based on the captured image <NUM>; and training a convolutional neural network based on the captured image and the statistical model <NUM>.

<FIG> is an acquired image, for example a microscopy image, before being processed.

<FIG> is an image after being processing using a trained convolutional neural network trained according to an embodiment of the invention. It is apparent from the image in <FIG> that the SNR has been improved with respect to the acquired image.

<FIG> is a ground truth of the acquired image.

<FIG> shows a schematic illustration of a system <NUM> configured to perform a method described herein. The system <NUM> comprises a microscope <NUM> and a computer system <NUM>. The microscope <NUM> is configured to take images and is connected to the computer system <NUM>. The computer system <NUM> is configured to execute at least a part of a method described herein. The computer system <NUM> may be configured to execute a machine learning algorithm. The computer system <NUM> and microscope <NUM> may be separate entities but can also be integrated together in one common housing. The computer system <NUM> may be part of a central processing system of the microscope <NUM> and/or the computer system <NUM> may be part of a subcomponent of the microscope <NUM>, such as a sensor, an actor, a camera or an illumination unit, etc. of the microscope <NUM>.

The computer system <NUM> may be a local computer device (e.g. personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g. a cloud computing system <NUM> with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and/or one or more remote server farms and/or data centers). The computer system <NUM> may comprise any circuit or combination of circuits. In one embodiment, the computer system <NUM> may include one or more processors <NUM>, which can be of any type. As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor <NUM>, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g. camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system <NUM> may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system <NUM> may include one or more storage devices <NUM>, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and/or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system <NUM> may also include a display device <NUM>, one or more speakers, and a keyboard and/or controller <NUM>, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the computer system <NUM>.

Embodiments may be based on using a machine-learning model or machine-learning algorithm. Machine learning may refer to algorithms and statistical models that computer systems may use to perform a specific task without using explicit instructions, instead relying on models and inference. For example, in machine-learning, instead of a rulebased transformation of data, a transformation of data may be used, that is inferred from an analysis of historical and/or training data. For example, the content of images may be analyzed using a machine-learning model or using a machine-learning algorithm. In order for the machine-learning model to analyze the content of an image, the machine-learning model may be trained using training images as input and training content information as output. By training the machine-learning model with a large number of training images and/or training sequences (e.g. words or sentences) and associated training content information (e.g. labels or annotations), the machine-learning model "learns" to recognize the content of the images, so the content of images that are not included in the training data can be recognized using the machine-learning model. The same principle may be used for other kinds of sensor data as well: By training a machine-learning model using training sensor data and a desired output, the machine-learning model "learns" a transformation between the sensor data and the output, which can be used to provide an output based on non-training sensor data provided to the machine-learning model. The provided data (e.g. sensor data, meta data and/or image data) may be preprocessed to obtain a feature vector, which is used as input to the machine-learning model.

Machine-learning models may be trained using training input data. The examples specified above use a training method called "supervised learning". In supervised learning, the machine-learning model is trained using a plurality of training samples, wherein each sample may comprise a plurality of input data values, and a plurality of desired output values, i.e. each training sample is associated with a desired output value. By specifying both training samples and desired output values, the machine-learning model "learns" which output value to provide based on an input sample that is similar to the samples provided during the training. Apart from supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on a supervised learning algorithm (e.g. a classification algorithm, a regression algorithm or a similarity learning algorithm. Classification algorithms may be used when the outputs are restricted to a limited set of values (categorical variables), i.e. the input is classified to one of the limited set of values. Regression algorithms may be used when the outputs may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms but are based on learning from examples using a similarity function that measures how similar or related two objects are. Apart from supervised or semi-supervised learning, unsupervised learning may be used to train the machine-learning model. In unsupervised learning, (only) input data might be supplied and an unsupervised learning algorithm may be used to find structure in the input data (e.g. by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data comprising a plurality of input values into subsets (clusters) so that input values within the same cluster are similar according to one or more (pre-defined) similarity criteria, while being dissimilar to input values that are included in other clusters.

Reinforcement learning is a third group of machine-learning algorithms. In other words, reinforcement learning may be used to train the machine-learning model. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Based on the taken actions, a reward is calculated. Reinforcement learning is based on training the one or more software agents to choose the actions such, that the cumulative reward is increased, leading to software agents that become better at the task they are given (as evidenced by increasing rewards).

Furthermore, some techniques may be applied to some of the machine-learning algorithms. For example, feature learning may be used. In other words, the machine-learning model may at least partially be trained using feature learning, and/or the machine-learning algorithm may comprise a feature learning component. Feature learning algorithms, which may be called representation learning algorithms, may preserve the information in their input but also transform it in a way that makes it useful, often as a preprocessing step before performing classification or predictions. Feature learning may be based on principal components analysis or cluster analysis, for example.

In some examples, anomaly detection (i.e. outlier detection) may be used, which is aimed at providing an identification of input values that raise suspicions by differing significantly from the majority of input or training data. In other words, the machine-learning model may at least partially be trained using anomaly detection, and/or the machine-learning algorithm may comprise an anomaly detection component.

In some examples, the machine-learning algorithm may use a decision tree as a predictive model. In other words, the machine-learning model may be based on a decision tree. In a decision tree, observations about an item (e.g. a set of input values) may be represented by the branches of the decision tree, and an output value corresponding to the item may be represented by the leaves of the decision tree. Decision trees may support both discrete values and continuous values as output values. If discrete values are used, the decision tree may be denoted a classification tree, if continuous values are used, the decision tree may be denoted a regression tree.

Association rules are a further technique that may be used in machine-learning algorithms. In other words, the machine-learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. The machine-learning algorithm may identify and/or utilize one or more relational rules that represent the knowledge that is derived from the data. The rules may e.g. be used to store, manipulate or apply the knowledge.

Machine-learning algorithms are usually based on a machine-learning model. In other words, the term "machine-learning algorithm" may denote a set of instructions that may be used to create, train or use a machine-learning model. The term "machine-learning model" may denote a data structure and/or set of rules that represents the learned knowledge (e.g. based on the training performed by the machine-learning algorithm). In embodiments, the usage of a machine-learning algorithm may imply the usage of an underlying machine-learning model (or of a plurality of underlying machine-learning models). The usage of a machine-learning model may imply that the machine-learning model and/or the data structure/set of rules that is the machine-learning model is trained by a machine-learning algorithm.

For example, the machine-learning model may be an artificial neural network (ANN). ANNs are systems that are inspired by biological neural networks, such as can be found in a retina or a brain. ANNs comprise a plurality of interconnected nodes and a plurality of connections, so-called edges, between the nodes. There are usually three types of nodes, input nodes that receiving input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information, from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g. of the sum of its inputs). The inputs of a node may be used in the function based on a "weight" of the edge or of the node that provides the input. The weight of nodes and/or of edges may be adjusted in the learning process. In other words, the training of an artificial neural network may comprise adjusting the weights of the nodes and/or edges of the artificial neural network, i.e. to achieve a desired output for a given input.

Claim 1:
A computer implemented method (<NUM>) for training an image denoising convolutional neural network, the method comprising:
receiving (<NUM>) a captured image (<NUM>);
generating (<NUM>) a statistical noise model (<NUM>) based on the captured image; and
training (<NUM>) a convolutional neural network (<NUM>) based on the captured image and the statistical model;
wherein training the convolutional neural network comprises:
dividing the captured image into random patches (<NUM>);
swapping pixel values in each random patch with neighbour pixel values; and
training the convolutional neural network based on the random patches of the captured image and the random patches of the captured image comprising swapped pixel values (<NUM>);
further comprising:
receiving a further captured image;
updating the statistical model based on the captured image and the further captured image;
determining if there is a reduction in a variance or standard deviation value of the updated statistical model compared with the variance model; and
retraining the convolutional neural network based on the further captured image and the updated statistical model if it is determined that there is a reduction in a variance or standard deviation value of the updated statistical model compared with the statistical model.