Patent Description:
A medical electronic endoscope is a product implemented by the continuous development and integration of a conventional endoscope and computer, microelectronics and other technologies, and is currently a very widely used medical instrument. The endoscope can be used to perform effective in-vivo examination, photograph an image of a diseased region, and extract the diseased region as a sample, to provide more information for the treatment in a next step. However, in practical application, medical images are often affected by an imaging device and an external environment during collection, conversion and transmission, and there are inevitably some noise in endoscopic images. The noise may seriously reduce image quality, damage detailed information in the images, interfere with a doctor' correct judgment, or even cause misdiagnosis. Therefore, how to improve an endoscopic denoising technology so as to remove noise in an image well and retain effective information of the image to the greatest extent is of great value in practical applications.

Endoscopes usually adopt a complementary metal oxide semiconductor (CMOS) image sensor, with main noise sources being noise generated when a photodiode, a field-effect transistor and an image sensor of a pixel photosensitive unit operate. Noise produced by the photodiode includes thermal noise, shot noise, coincidence noise, and current noise. A metal oxide semiconductor (MOS) field-effect transistor includes a field-effect transistor in an amplifier and a field-effect transistor used for a row-column address selection analog switch, which mainly causes a thermal noise, an induced gate noise, and a current noise. Other noise, such as reset noise (kTC noise) and spatial noise, is further introduced into a CMOS image sensor composed of a photosensitive array and an MOS field-effect transistor.

Referring to <FIG>, some noise may be processed by an analog denoising method such as by using a correlated double sampling circuit, and a digital denoising processing part is mainly considered in the present disclosure.

Image denoising is faced with distinguishing and selection between details and noise. Because the goal of denoising is for medical images, it is necessary to investigate noise characteristics in detail in order to be rigorous during denoising and make denoised images truly clinically usable.

Currently, a bilateral filtering method is used to denoise an endoscope image. Bilateral filtering is implemented by a nonlinear filter, which can achieve effects of edge preserving and smooth denoising. Bilateral filtering is implemented by using a weighted average method, in which an intensity of a pixel is represented with a weighted average of brightness values of surrounding pixels, and the weighted average used is based on Gaussian distribution.

The bilateral filtering features that the weight not only takes a Euclidean distance between pixels (for example, ordinary Gaussian low-pass filtering only takes the impact of a position on a central pixel) into account, but also takes a radiation difference in a pixel range domain (such as a similarity between a pixel and a central pixel in a convolution kernel, a color intensity, and a depth distance) into account. During calculation of the central pixel, the above two contents are considered at the same time:.

In a flat region of an image, the pixel value changes little, and a corresponding pixel range domain weight is close to <NUM>. In this case, a spatial domain weight plays a major role, which is equivalent to Gaussian blur. In an edge region of the image, the pixel value changes greatly, and the weight of the pixel range domain becomes larger, thereby preserving edge information.

Although bilateral filtering performs well in preserving the edge part, the bilateral filtering cannot identify noise characteristics, and cannot well distinguish high-frequency noise from high-frequency details. When input noise parameters are too large, the details may be easily erased, otherwise the denoising effect is not obvious. In addition, bilateral filtering needs gray information of each central point domain to determine a parameter of the bilateral filtering, so that the bilateral filtering velocity is relatively low, and a growth rate of a calculated amount is the square of a kernel size.

Known denoising methods are disclosed in <NPL> and <NPL>.

Based on this, an objective of the present disclosure is to provide a method for denoising a medical endoscopic image based on a CBD-Net, which denoises an endoscopic image on the basis of preserving details. The invention is set out in de appended claims.

Roughly speaking, to achieve the above objective, the present disclosure provides a method for denoising a medical endoscopic image based on a CBD-Net, including:.

Further, the CBD-Net includes a noise estimation subnetwork and a non-blind denoising subnetwork;.

Further, in the improved CBD-Net, an initial noise model n(x)-N(<NUM>, σ(y)) is as follows:
<MAT> and <MAT>.

Further, in the improved CBD-Net, a loss function is as follows:
<MAT>.

Further, an imaging element of the endoscope is a CMOS photosensitive element, and the noise source is implemented in the following way: when the endoscope is in a human body, a light source only comes from supplementary light from the endoscope, which results in a dark environment, and when an aperture is unchanged, noise is increased by an excessive exposure gain.

Further, the obtaining a supplementary real training set based on the noise source includes:.

Further, the photographed objects include lean animal meat, fat animal meat, and animal viscera.

According to the specific embodiments of the present disclosure, the present disclosure discloses the following technical effects:.

To describe the technical solutions in embodiments of the present disclosure or in prior art more clearly, the accompanying drawings required in the embodiments are briefly described below. Obviously, the accompanying drawings in the following description describe only some embodiments of the present disclosure. A person of ordinary skill in the art may further obtain other accompanying drawings based on these accompanying drawings without creative efforts.

The technical solutions of the embodiments of the present disclosure are clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure while the scope of the invention is set out by the appended claims.

To make the above objectives, features, and advantages of the present disclosure clearer and more comprehensible, the present disclosure will be further described in detail below with reference to the accompanying drawings and the specific implementations.

Based on existing endoscopic image data, different algorithms are used to try to denoise for analyzing noise characteristics. Through observation of an existing endoscopic image, it can be learned that in practical application, the endoscopic image has low ambient brightness, and the noise is obvious, and is mainly Gaussian noise, so conventional methods such as Gaussian filtering, BM3D and bilateral filtering, which have good performance in Gaussian denoising, are used to make preliminary denoising attempts. Typical denoising convolutional neural network (DnCNN), DnCNN-B, FFDNet and CBD-Net models are used for denoising.

The conventional and neural network methods with good performance are selected for improvement and comparison: a conventional algorithm with good performance is the BM3D, and denoising networks with good performance are the FFDNet and the CBD-Net.

Based on the noise characteristics, the CBD-Net is selected and improved by a formula weight and training set expansion to meet denoising requirements: the CBD-Net performs well on some endoscopic negative samples, so this network is selected as the basis for improvement.

As shown in <FIG>, a method for denoising a medical endoscopic image based on a CBD-Net according to the present disclosure includes the following steps.

Step <NUM>: Construct and improve a CBD-Net, to obtain a medical endoscopic image denoising model.

Step <NUM>: Obtain a noise source based on an imaging principle of an endoscope.

Step <NUM>: Obtain a simulated training set based on noise modeling; and obtain a supplementary real training set based on the noise source.

Step <NUM>: Train the medical endoscopic image denoising model by using the simulated training set and the supplementary real training set.

Step <NUM>: Denoise a medical endoscopic image by using a trained medical endoscopic image denoising model.

Referring to <FIG>, in an embodiment, in step <NUM>, the CBD-Net includes a noise estimation subnetwork and a non-blind denoising subnetwork;.

On this basis, according to the present disclosure, improvements have been made in two aspects.

In this embodiment, in the CBD-Net improved in step <NUM>, an initial noise model n(x)-N(<NUM>, σ(y)) of the CBD-Net is as follows:
<MAT>
where in n(x) = ns(x) + nc herein, ns represents a signal-dependent noise part, which is often related to image brightness. nc represents a stationary noise part, and is often modeled as white Gaussian noise with a variance of <MAT>.

In addition, according to the present disclosure, a complex ISP process in the camera is considered, which produces a noise model that is expressed by the following formula and depends on both a signal and a color channel:
<MAT> where y represents a to-be-input noise image; f(·) represents a camera response function (CRF), which is used to restore an irradiance L generated by photographing in reality to convert the image into an original clean image x; M(·) represents a function that converts an sRGB image into a Bayer image, the Bayer format described herein may be unfamiliar to many people, but is actually the format of the most original picture, generally having a suffix of. raw; and M-<NUM>(·) represents a demosaicing function.

After this, further, considering that the image may undergo some compression more or less during transmission, the JPEG compression effect is also taken into account during noise modeling, and the formula is expressed as:
<MAT>.

Because most endoscopic images do not need high-level compression in the original text, the adding of the compression may affect the effect, so this method excludes the image compression effect from the noise model.

In this embodiment, in the CBD-Net improved in step <NUM>, the loss function comes from: <IMG>. The asymmetric loss function designed herein is used in noise estimation to eliminate asymmetric sensitivity of a mainstream non-blind denoising method which is sensitive to underestimated errors and robust to overestimated errors. Given an estimated noise level σ̂(yi) and a true value σ(yi) of a pixel i, when σ(yi) < σ(yi), more penalties should be introduced to a mean square error of the pixel i, which may be expressed as the following formula:
<MAT>
where when e<<NUM>, Ie = <NUM>, otherwise is <NUM>; and by setting <NUM> < α < <NUM>, more penalties may be introduced to underestimated errors.

In addition, a total variation regularization term (TV) is introduced for the smoothness of the constraint σ̂(yi):
<MAT>
where ∇h∇v represent gradients in a vertical direction and a horizontal direction respectively.

For x̂ of the denoising network input, the reconstruction error is defined:
<MAT>.

In conclusion, an ultimate target loss function of the CBD-Net is:
<MAT>.

Referring to non-blind denoising methods such as the BM3D and the FFDNet, an asymmetric learning method is adopted for Lasymm constructed by the method of the present disclosure, and a relatively large weight is given to encourage noise overestimation to achieve a good denoising effect. However, for a medical endoscopic image, overestimation may lead to the loss of details of the medical image, which may lead to misdiagnosis in a severe case, and the loss of details such as visceral texture is not desired. Therefore, the weight of asymmetric learning is reduced in the method of the present disclosure.

Considering parameters of asymmetric learning, in a medical image, a user prefers to preserve some noise rather than excessively smooth details of the image, so this method intends to lower the degree of asymmetric learning and change the original assignment α = <NUM> to α = <NUM> in the formula of asymmetric learning. Weights in the loss function are set to λasymm=<NUM>, and λTV = <NUM> respectively.

In this embodiment, in step <NUM>, an imaging element of the endoscope is a CMOS photosensitive element, and the noise source is implemented in the following way: when the endoscope is in a human body, a light source only comes from supplementary light from the endoscope, which results in a dark environment, and when an aperture is unchanged, noise is increased by an excessive exposure gain.

In this embodiment, in step <NUM>, the obtaining a supplementary real training set based on the noise source includes the following steps.

Step (<NUM>): Simulate and adapt to a plurality of different detail features in the human body by using photographed objects with different details.

Step (<NUM>): Fix a lens, photograph a video in an environment with adequate illumination, and change an exposure gain of the video.

Step (<NUM>): Capture images with noise and images without noise under the same brightness as positive and negative samples in the supplementary real training set.

In this embodiment, in step (<NUM>), the photographed objects include lean animal meat, fat animal meat, and animal viscera.

In this embodiment, the simulated training set in step <NUM> is <NUM> images of a public data set Waterloo, <NUM> images of BSD500, and <NUM> images of an MIT-Adobe FiveK database, which are modeled and added as positive and negative training data based on the explained noise model; and the real training set is the supplementary real training set described above, which is obtained after the test with the company endoscope.

In order to verify the effect of the present disclosure, the real noisy endoscope image is used to verify the improvement of the method of the present disclosure compared with conventional methods and other deep learning methods. Referring to <FIG>, the method of the present disclosure has little difference in denoising improvement effect from a conventional BM3D algorithm, but has obvious advantages compared with the FFDNet with good performance. Because the method of the present disclosure does not need to input a noise coefficient, and is a blind denoising method, the method is superior to a conventional algorithm in time and complexity and is of great significance to a clinical endoscopic surgery.

Each embodiment of the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and for the same and similar parts between the embodiments, reference may be made to each other.

Claim 1:
A computer-implemented method for denoising a medical endoscopic image based on a convolutional blind denoising network (CBD-Net), comprising:
constructing a CBD-Net, to obtain a medical endoscopic image denoising model;
obtaining a noise source based on an imaging principle of an endoscope;
wherein an imaging element of the endoscope is a complementary metal oxide semiconductor (CMOS) photosensitive element, and the noise source is implemented in the following way: when the endoscope is in a human body, a light source only comes from supplementary light from the endoscope, which results in a dark environment, and when an aperture is unchanged, noise is increased by an excessive exposure gain;
obtaining (<NUM>) a simulated training set based on the medical endoscopic image denoising model;
obtaining (<NUM>) a supplementary real training set based on the noise source; wherein the obtaining (<NUM>) a simulated training set based on the medical endoscopic image denoising model and the obtaining (<NUM>) a supplementary real training set based on the noise source comprise steps <NUM>) to <NUM>) as follows:
<NUM>) simulating and adapting to a plurality of different detail features in the human body by using photographed objects with different details;
<NUM>) fixing a lens, photographing a video in an environment with adequate illumination, and changing an exposure gain of the video; and
<NUM>) capturing images with noise and images without noise under the same brightness as positive and negative samples in the supplementary real training set;
training the medical endoscopic image denoising model by using the simulated training set and the supplementary real training set; and
denoising a medical endoscopic image by using a trained medical endoscopic image denoising model.