Patent Description:
Conventional self-supervised blind denoising shows poor quality in real-world images due to spatially correlated noise corruption. Recently, Pixel-shuffle Downsampling (PD) has been proposed to eliminate the spatial correlation of the noise.

A study combining asymmetric PD (AP) and a blind-spot network (BSN) successfully demonstrates that self-supervised blind denoising is applicable to real-world noisy images.

However, PD-based inference of the BSN may degrade texture details in the testing phase (denoising step) because high-frequency details (e.g., edges) are destroyed in the downsampled images.

To solve such a problem, a model capable of eliminating noise, without the PD process, is needed to allow texture details to be kept. Further, a new inference system is required to boost overall performance, while avoiding the use of an order-variant PD constraint, noise prior knowledge-based loss function, and the PD.

For example, in a publication in arxiv. org "Self-supervised Denoising for Real-World Images via Asymmetric PD and Blind-spot network" a blind denoising device is described. However, the claimed subject-matter of the independent claims of the present invention is not disclosed.

Accordingly, the present invention has been made in view of the above-mentioned problems occurring in the related art, and it is an object of the present invention to provide a blind denoising device and method that is capable of learning a denoising model only using a noisy image to eliminate noise, without a PD process, in a testing step, so that texture details of an original image are kept, without degrading.

To accomplish the above-mentioned objects, according to one aspect of the present invention, a blind denoising device may include: a receiving unit for receiving an original noisy image; a Pixel-shuffle Downsampling (PD) unit for performing PD for the original noisy image to produce at least one or more downsampled images; a predicted image producing unit for eliminating the noise from the original noisy image and the downsampled images to produce at least one or more predicted images restored to the shape of the original noisy image; and a learning unit for performing optimized learning for the predicted image producing unit, based on at least one or more self-supervised losses of the predicted images and the original noisy image.

Further, the PD unit may produce the at least one or more downsampled images from the original noisy image according to at least one or more stride factors to thus augment data of the original noisy image.

Furthermore, the PD unit may perform order-variant PD using transformation matrices selected randomly, and the transformation matrices may be shuffled in the sampling order.

Moreover, according to the present invention, the predicted image producing unit includes a first network for eliminating the noise from the original noisy image and the downsampled images, a second network for eliminating the noise from the downsampled images according to the stride factors, and a restoring part for restoring the shape of the original noisy image from the images from which the noise is eliminated through the first network and the second network to thus produce the predicted images.

Further, the learning unit may perform learning for the first network and the second network, based on a first self-supervised loss defining a loss of the second network, a second self-supervised loss for supporting residual noise learning of the first network, a third self-supervised loss for enhancing the similarity between the predicted images produced through the first network and the second network, and a fourth self-supervised loss for limiting the distribution of the predicted images produced through the first network.

Further, the first self-supervised loss may define the loss of the second network from a difference between the predicted image produced through the second network and the original noisy image.

Furthermore, the second self-supervised loss may be defined as a difference between a pseudo-noise map and the predicted image produced through the first network, the pseudo-noise map may be produced from a difference between the original noisy image and the predicted image produced through the second network, and the predicted image produced through the first network may be the predicted image from the original noisy image.

Moreover, the third self-supervised loss may be defined, based on a difference image between the original noisy image and the predicted image produced through the first network and a difference value between the predicted images produced through the second network, to enhance the similarity of low-frequency characteristics between the predicted image produced through the first network and the predicted images produced through the second network, and the predicted images produced through the first network and the second network may be the predicted images from the downsampled images according to the same stride factor.

Additionally, the fourth self-supervised loss may be noise prior loss that penalizes noise having higher size than a threshold value to allow the noise distribution of the predicted image produced through the first network to be close to the noise distribution of the original noisy image, and the predicted image produced through the first network may be the predicted image from the original noisy image.

To accomplish the above-mentioned objects, according to another aspect of the present invention, a blind denoising method includes the steps of: receiving an original noisy image; performing PD for the original noisy image to produce at least one or more downsampled images; eliminating the noise from the original noisy image and the downsampled images to produce at least one or more predicted images restored to the shape of the original noisy image; and performing optimized learning for a predicted image producing unit, based on at least one or more self-supervised losses of the predicted images and the original noisy image.

Further, the step of producing the at least one or more predicted images includes the steps of: eliminating the noise from the original noisy image and the downsampled images through a first network; eliminating the noise from the downsampled images according to different stride factors through a second network; and restoring the shape of the original noisy image from the images from which the noise is eliminated through the first network and the second network to thus produce the predicted images through a restoring part.

Further, the step of performing optimized learning for a predicted image producing unit may include the step of performing learning for the first network and the second network, based on a first self-supervised loss defining a loss of the second network, a second self-supervised loss for supporting residual noise learning of the first network, a third self-supervised loss for enhancing the similarity between the predicted images produced through the first network and the second network, and a fourth self-supervised loss for limiting the distribution of the predicted images produced through the first network.

Furthermore, the first self-supervised loss may define the loss of the second network from a difference between the predicted image produced through the second network and the original noisy image.

According to the present invention, the blind denoising device and method can perform denoising model learning only using the noisy image, restore high-frequency characteristics such as texture more effectively than the recent method, and eliminate the spatially correlated noise as well as the pixel-unit noise.

An explanation on embodiments of the present invention will be given in detail with reference to the attached drawing. The embodiments of the present invention as will be discussed later will be in detail described so that it may be carried out easily by those having ordinary skill in the art, and therefore, this does not limit the idea and technical scope of the invention. Before the present invention is disclosed and described, it is to be understood that the disclosed embodiments are merely exemplary of the invention, which can be embodied in various forms. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one of ordinary skill in the art to variously employ the present invention in virtually any appropriately detailed structure. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below. In the description, it should be noted that the parts corresponding to those of the drawings are indicated by corresponding reference numerals.

Hereinafter, embodiments of the present invention will now be described in detail with reference to the attached drawings.

<FIG> is a schematic block diagram showing a configuration of a blind denoising device according to the present invention, and <FIG> is a schematic block diagram showing a predicted image producing unit according to the present invention.

Referring to <FIG>, the blind denoising device <NUM> according to the present invention includes a receiving unit <NUM>, a PD unit <NUM>, a predicted image producing unit <NUM>, and a learning unit <NUM>.

According to the present invention, the receiving unit <NUM> receives an original noisy image. For example, the original image in the description represents an image that does not have any separate compression after produced and is not transmitted to a separate device or server, that is, an image where no loss is generated. The noise represents the damaged portions of image pixels that are generated by the compression applied to the original image or by the transmission of the original image. In the description, the term 'image' may include a still image or a unit image (e.g., frame) constituting a video.

According to the present invention, the PD unit <NUM> performs pixel-shuffle downsampling (PD) for the original noisy image to produce at least one or more downsampled images.

For example, the PD performed by the PD unit <NUM> is a process of augmenting the original noisy image to one or more pixel-shuffled images, more particularly a process of producing a plurality of downsampled images from the original noisy image according to at least one or more stride factors to thus augment data of the original noisy image. The process will be explained with reference to <FIG> and <FIG>.

<FIG> and <FIG> show a conventional method and a method according to the present invention for augmenting data of the original noisy image.

<FIG> and <FIG> show examples of order-invariant PD and order-variant PD for stride factor PD<NUM>.

First, <FIG> shows a conventional PD method for augmenting data of original noisy image to produce downsampled images.

The conventional PD method as shown in <FIG> shows the PD for the original noisy image with transformation matrices I<NUM>.

In specific, the conventional PD method as shown in <FIG> performs the PD for the original noisy image with spatially correlated noise only using self-supervised loss Ls.

The self-supervised loss Ls is represented by the following mathematical expression <NUM>. <MAT> wherein the x represents the original noisy image, the f BSN, the PD<NUM> a PD function having stride factor s (in this case, s is <NUM>), and I<NUM> the transformation matrices for the order-invariant PD.

The conventional order-invariant PD produces the downsampled images only using the self-supervised loss Ls in predetermined pixel sampling order, thereby resulting in the production of the limited number of sub images. For example, as shown in <FIG>, the order-invariant PD produces four 2x2 sub images from a 4x4 input image.

In specific, the conventional order-invariant PD is a method for extracting pixels regularly to produce downsampled images, so that it is performed for the original noisy image with spatially correlated noise only using the self-supervised loss Ls, thereby making it hard to effectively eliminate the spatially correlated noise.

Contrarily, the order-variant PD according to the present invention is performed even for irregular pixels, and an explanation of the order-variant PD will be given with reference to <FIG>.

<FIG> shows a PD method according to the present invention for augmenting data of the original noisy image to produce downsampled images.

The PD method according to the present invention as shown in <FIG> is the order-variant PD for the original noisy image with arbitrarily irregularly produced transformation matrices I ∈ I.

As shown in <FIG>, the PD according to the present invention is the order-variant PD for shuffling sampling order using the transformation matrices I selected randomly to augment the number of downsampled images (in this case, the transformation matrices may be shuffled in the sampling order). The self-supervised loss Ls used in the order-variant PD according to the present invention is represented by the following mathematical expression <NUM>. <MAT> wherein the x represents the original noisy image, the f BSN as a second network <NUM>, the I the transformation matrices for the order-variant PD, and the PD<NUM> a PD function having stride factor s (in this case, s is <NUM>).

The order-variant PD according to the present invention is performed even for irregular pixels, so that a process of segmenting the original noisy image into the plurality of downsampled images is performed to allow the spatial correlation of the noise to disappear, thereby making it very easy to eliminate the noise. Further, the PD according to the present invention is performed even for various images, thereby having no limitations in the types of images.

To upgrade denoising performance according to the order-variant PD of the present invention and to reduce visual artifacts of the AP-BSN, a post-processing method, random-replacing refinement (R<NUM>) has been proposed in the conventional practice.

The R<NUM> represents the average of the images restored from various images, with which noise is synthesized, produced by randomly replacing noise by pixel (selected in x) with initially predicted f (x;I<NUM>, PD<NUM>). However, the R<NUM> is likely to make the restored images excessively smooth through repeated denoising, thereby causing texture details to be lost. An explanation of such a problem will be given in detail later with reference to <FIG>.

Referring back to <FIG>, the predicted image producing unit <NUM> according to the present invention serves to eliminate the noise from the original noisy image and the downsampled images to thus produce at least one or more predicted images restored to the shape of the original noisy image. A configuration of the predicted image producing unit <NUM> will be explained with reference to <FIG> and <FIG>.

<FIG> is a schematic block diagram showing the predicted image producing unit according to the present invention, and <FIG> shows a learning process of the predicted image producing unit according to the present invention.

Referring to <FIG> and <FIG>, the predicted image producing unit <NUM> according to the present invention includes a first network <NUM>, the second network <NUM>, and a restoring part <NUM> to produce predicted images <NUM>. In specific, the first network <NUM> eliminates noise from the original noisy image <NUM> and the downsampled images <NUM>, the second network <NUM> eliminates noise from the at least one or more downsampled images <NUM> according to different stride factors, and the restoring part <NUM> restores the shape of the original noisy image <NUM> from the images from which noise is eliminated through the first network <NUM> and the second network <NUM> to thus produce the predicted images <NUM> as shown in a forward step of <FIG>.

Further, the second network <NUM> is limited in a structure thereof, unlike a recent network, and accordingly, the second network <NUM> is available when satisfying a given condition where the pixel shapes of input and output are independent of each other.

Contrarily, the first network <NUM> makes use of network learning even though such a given condition is not satisfied, and further, the PD process may be omitted to thus improve noise learning performance. If the PD process in a learning process is omitted, the PD process in a denoising process may be omitted, thereby resulting in the improvement of the denoising performance.

Further, the predicted images <NUM> include a first predicted image <NUM> produced by eliminating the noise from the original noisy image <NUM> through the first network <NUM>, a second predicted image <NUM> produced by eliminating the noise from the downsampled images <NUM> through the first network <NUM>, and third and fourth predicted images <NUM> and <NUM> produced by eliminating the noise from the plurality of downsampled <NUM> according to the different stride factors through the second network <NUM>.

In specific, the first predicted image <NUM> is an image produced by eliminating the noise detected from the original noisy image <NUM> through the first network <NUM>, while not performing the order-variant PD process. The second predicted image <NUM> is an image produced by eliminating the PD<NUM> noise detected from the downsampled images <NUM> according to the stride factor PD<NUM> through the first network <NUM>. The third predicted image <NUM> is an image produced by eliminating the PD<NUM> noise detected from the downsampled images <NUM> according to the stride factor PD<NUM> through the second network <NUM>. The fourth predicted image <NUM> is an image produced by eliminating the PD<NUM> noise detected from the downsampled images <NUM> according to the stride factor PD<NUM> through the second network <NUM>.

That is, the first predicted image <NUM> is an image that is produced by eliminating the noise from the original noisy image <NUM> where no PD is performed through the first network <NUM> and thus restored to the shape of the original noisy image <NUM> (in <FIG>, "PDS-<NUM>(·,·)"). The second to fourth predicted images <NUM> to <NUM> are images that are produced by eliminating the noise from the original noisy image <NUM> where the PD (in <FIG>, "PDS(x ,·)")is performed through the first and second networks <NUM> and <NUM> and thus restored to the shape of the original noisy image <NUM> (in <FIG>, "PDS-<NUM>(·,·)"). The first to fourth predicted images <NUM> to <NUM> are produced to the shapes shown in the forward step of <FIG>.

Referring to <FIG> and <FIG>, according to the present invention, the learning unit <NUM> serves to perform optimized learning for the predicted image producing unit <NUM>, based on at least one or more self-supervised losses of the predicted images <NUM> and the original noisy image <NUM>.

In specific, the learning unit <NUM> performs learning for the first network <NUM> and the second network <NUM>, based on a first self-supervised loss defining a loss of the second network <NUM> itself, a second self-supervised loss for supporting residual noise learning of the first network <NUM>, a third self-supervised loss for enhancing the similarity between the predicted images <NUM> produced through the first network <NUM> and the second network <NUM>, and a fourth self-supervised loss for limiting the distribution of the predicted images <NUM> produced through the first network <NUM>.

The first self-supervised loss Ls is a loss function defining a loss of the second network <NUM> itself from a difference between the third predicted image <NUM> as one of the predicted images <NUM> produced through the second network <NUM> and the original noisy image <NUM>. If the first self-supervised loss Ls is used for the residual noise learning of the first network <NUM>, it is possible that the noise learning is performed only with the noisy image produced therein, without any separate noisy image. In this case, the first self-supervised loss Ls is defined by the mathematical expression <NUM>. That is, the first self-supervised loss Ls is the same as the self-supervised loss Ls used in the PD process according to the present invention.

The second self-supervised loss Lr is defined as a difference between a pseudo-noise map and the first predicted image <NUM> as one of the predicted images <NUM> produced through the first network <NUM>. In this case, the pseudo-noise map is produced from a difference between the original noisy image <NUM> and the fourth predicted image <NUM> as one of the predicted images <NUM> produced through the second network <NUM>. The second self-supervised loss Lr is defined by the following mathematical expression <NUM>. <MAT> wherein the x represents the original noisy image <NUM>, the x - f(x;I<NUM>, PD<NUM>) the pseudo-noise map, and h(x) the first predicted image <NUM>.

The second self-supervised loss Lr solves learning-interference data distribution mismatch and blurring artifacts that are caused by the excessive downsampling in the process of PD<NUM>. In this case, it is important that if the downsampled images <NUM> according to the stride factor PD<NUM> are used in the learning step, they have to be used even in the denoising step.

In the second self-supervised loss Lr, the order-variant PD having the I<NUM> is advantageously available in minimizing aliasing artifacts of the first network <NUM>. The pseudo-noise map can consider the spatially correlated real-world noise and the aliasing artifacts caused by the downsampling. According to the present invention, it is found that the pseudo-noise map has a better quality of texture restoration than the predicted images <NUM> produced by the first network <NUM>.

Unlike the conventional network performing the learning only using the PD<NUM>, the original noisy image <NUM> as learning data can be a high-resolution noisy image. Further, J-invariant properties are not needed in a structure of the first network <NUM>. Accordingly, all types of recent network architecture for image restoration is available for the first network <NUM>.

The third self-supervised loss Lov is defined by the following mathematical expression <NUM>, based on a difference image between the original noisy image <NUM> and the predicted image <NUM> produced through the first network <NUM> and a difference value between the predicted images <NUM> produced through the second network <NUM>, to enhance the similarity of low-frequency characteristics between the predicted image <NUM> produced through the first network <NUM> and the predicted image <NUM> produced through the second network <NUM>. In this case, the predicted image <NUM> produced through the first network <NUM> is the second predicted image <NUM>, and the predicted image <NUM> produced through the second network <NUM> is the third predicted image <NUM>. Further, the second predicted image <NUM> and the third predicted image <NUM> are the predicted images <NUM> from the downsampled images <NUM> according to the same stride factor PD<NUM>. <MAT> wherein the x represents the original noisy image <NUM>, the f BSN as the second network <NUM>, the l and l' transformation matrices for order-variant PD, the PD<NUM> the PD function having stride factor s (in this case, s is <NUM>), the h noise extractor h as the first network <NUM>, the f(x;I, PD<NUM>) the third predicted image <NUM>, and the h(x; I', PD<NUM>) the second predicted image <NUM>.

The fourth self-supervised loss Lnp is noise prior loss that penalizes noise having higher size than a threshold value to allow the noise distribution of the predicted image <NUM> produced through the first network <NUM> to be close to the noise distribution of the original noisy image <NUM>, which is defined by the following mathematical expression <NUM>. In this case, the predicted image <NUM> produced through the first network <NUM> is the first predicted image <NUM>. <MAT> wherein the x represents the original noisy image <NUM>, and the h(x) the first predicted image <NUM>.

The first network <NUM> (as the noise extractor h) performs data distribution matching to thus improve denoising quality, but it causes some of texture details and color changes in a texture-abundant image to be lost. This is because the first network <NUM> overfits to the real noise of the second self-supervised loss as well as the aliasing artifacts of the PD<NUM>. The aliasing artifacts contribute to the texture details and the sizes thereof are larger than the real noise.

To upgrade the texture details, accordingly, the fourth self-supervised loss is proposed. The fourth self-supervised loss limits (that is, penalizes high noise) the distribution of the first predicted image <NUM> produced through the first network <NUM> using L1 regularization term. The L1 regularization term takes average absolute values by pixel according to mini batch and color axes to effectively suppress the outlier of the pseudo-noise map of the second self-supervised loss.

As a result, a total self-supervised loss for the first network <NUM> and the second network <NUM> is defined by the following mathematical expression <NUM>. <MAT> wherein the hyperparameters λs, λr, λov, and λnp represent the contribution weights of the first to fourth self-supervised losses.

The learning unit <NUM> performs the learning for the first network <NUM> and the second network <NUM>, based on the first to fourth self-supervised losses.

For example, the learning unit <NUM> applies the first self-supervised loss to the third predicted image <NUM> and the original noisy image <NUM>, applies the second self-supervised loss to the difference value among the fourth predicted image <NUM>, the original noisy image <NUM>, and the first predicted image <NUM>, applies the third self-supervised loss to the third predicted image <NUM> and the pseudo-noise map, and applies the fourth self-supervised loss to the first predicted image <NUM>, so that the learning unit <NUM> performs the learning for the first network <NUM> and the second network <NUM> to thus produce a denoising model.

<FIG> shows the learning process of the predicted image producing unit <NUM> according to the present invention.

Referring to <FIG>, the blind denoising device <NUM> performs the PD<NUM> or PD<NUM> process for the original noisy image <NUM> to produce at least one or more downsampled images <NUM>.

The blind denoising device <NUM> allows the predicted image producing unit <NUM> to eliminate the noise from the original noisy image <NUM> and the downsampled images <NUM> using the first network <NUM> and the second network <NUM>, as shown in the forward process of <FIG>, to thus produce at least one or more predicted images <NUM> restored to the shape of the original noisy image <NUM>. The restoring process to the shape of the original noisy image <NUM> is the process of the PDS-<NUM>(·,·) as shown in <FIG> that restores the images from which noise is eliminated according to the PD to the shape of the original noisy image <NUM> again.

The blind denoising device <NUM> allows the learning unit <NUM> to perform the optimized learning for the first network <NUM> and the second network <NUM> of the predicted image producing unit <NUM>, based on the first to fourth self-supervised losses, the first to fourth predicted images <NUM> to <NUM>, and the original noisy image <NUM>, as shown in the backward process of <FIG>.

After the learning for the first network <NUM> and the second network <NUM> has been performed to minimize the total loss according to the backward learning process as shown in <FIG>, some visual artifacts may remain in the baseline learning results due to the structural limitations of the first network <NUM> and the second network <NUM>. Accordingly, a denoising process as shown in <FIG> is performed.

<FIG> shows a denoising process of a denoising model <NUM> according to the present invention.

The R<NUM> proposed in the conventional AP-BSN is a technique that releases visual artifacts. However, if multiple prediction for various noisy samples are averaged, content similarity is increased and texture details are decreased. Further, the baseline of the R<NUM> is dependent on initial prediction where the texture details are decreased by the PD<NUM> process. To solve such disadvantages of the R<NUM>, according to the present invention, progressively random-replacing refinement (PR<NUM>) as shown in <FIG> is proposed as the denoising model <NUM>.

Referring to <FIG>, a baseline as a basic interference model that performs the ensemble of the predicted results of the denoising model and the BSN <NUM> and the PR<NUM> technique used for the denoising model <NUM> and the BSN <NUM> according to the present invention are shown.

In the conventional practice, the PD<NUM> process is additionally performed even in the denoising process, thereby causing resource increase due to denoising and bad-quality results, and contrarily, in the case of the PR<NUM> as the denoising model <NUM> according to the present invention, the images for which the PD process has been already performed are used to thus omit additional PD process, so that the text of the output is not crushed in the random replacing process, thereby ensuring high-quality outputs and effectively removing the structure artifacts of the image produced through the second network <NUM>.

Referring to <FIG>, a process in which the denoising model <NUM> eliminates noise from the original noisy image <NUM> is shown.

In specific, if the original noisy image x <NUM> with noise is inputted to the first network <NUM>, the denoising model <NUM> removes the first predicted image from the original noisy image <NUM> through the first network <NUM> and then produces P<NUM> as the x-h(x) image as shown in <FIG>, which is restored to the shape of the original noisy image <NUM>, through the random-replacing.

After that, the denoising model <NUM> inputs the produced P<NUM> to the second network <NUM>, performs denoising and random-replacing to produce P<NUM>, and inputs the produced P<NUM> to the first network <NUM>. Lastly, the denoising model <NUM> detects and removes the noise from the P<NUM> through the first network <NUM> and performs averaging between the P<NUM> from which the noise is removed and the x-h(x) image to have last denoising.

<FIG> is a flowchart showing learning steps of the blind denoising device <NUM> according to the present invention.

A blind denoising method according to the present invention is performed in the same configuration as the blind denoising device <NUM> as shown in <FIG> and the predicted image producing unit <NUM> as shown in <FIG>. Therefore, the corresponding parts to the parts as shown in <FIG> are given with corresponding reference numerals and the repeated explanation on the corresponding parts will be avoided.

The blind denoising method according to the present invention includes the steps of receiving an original noisy image (at step S10), performing PD for the original noisy image to produce at least one or more downsampled images (at step S20), removing noise from the original noisy image and the downsampled images to produce at least one or more predicted images restored to the shape of the original noisy image (at step S30), and performing optimized learning for the predicted image producing unit <NUM>, based on at least one or more self-supervised losses of the predicted images and the original noisy image (at step S40).

The step of receiving an original noisy image (at step S10) is performed by receiving the original noisy image from an external server or device.

The step of performing PD for the original noisy image to produce at least one or more downsampled images (at step S20) is performed by producing the at least one or more downsampled images from the original noisy image according to at least one or more stride factors.

The step of removing noise from the original noisy image and the downsampled images to produce at least one or more predicted images restored to the shape of the original noisy image (at step S30) includes the steps of removing the noise from the original noisy image and the downsampled images through a first network <NUM>, removing the noise from the downsampled images according to the stride factors through a second network <NUM>, and restoring the shape of the original noisy image from the images from which the noise is removed through the first network <NUM> and the second network <NUM> to produce the predicted images through a restoring unit <NUM>.

The step of performing optimized learning for the predicted image producing unit <NUM>, based on at least one or more self-supervised losses of the predicted images and the original noisy image (at step S40) includes the step of performing learning for the first network <NUM> and the second network <NUM>, based on a first self-supervised loss defining a loss of the second network <NUM> itself, a second self-supervised loss for supporting residual noise learning of the first network <NUM>, a third self-supervised loss for enhancing the similarity between the predicted images <NUM> produced through the first network <NUM> and the second network <NUM>, and a fourth self-supervised loss for limiting the distribution of the predicted images <NUM> produced through the first network <NUM>.

<FIG> is a table showing denoising performance according to the present invention, and <FIG> shows denoising results of blind denoising methods according to the present invention.

<FIG> is a table showing ablation study results of the self-supervised losses for smartphone image denoising dataset (SIDD). Referring to <FIG>, results where self-supervised losses do not exist are provided on columns, maximum and second-maximum results are underlined, and the maximum results are bolded.

If the second self-supervised loss Lr is omitted in <FIG>, details-based learning does not exist in the second network (BSN f), except the content information maintaining through the third self-supervised loss Lov, thereby resulting in overall performance degradation.

If the third self-supervised loss Lov is omitted, the performance of PSNR and SSIM is improved, but minimal performance degradation is observed in LPIPS and DISTS.

This is because the third self-supervised loss Lov promotes content similarity and provides higher PSNR and SSIM results, instead of improving texture restoration performance.

If the fourth self-supervised loss Lnp is omitted, overfitting is generated from the pseudo-noise map of the second network (BSN f) if there is no noise prior to the loss, thereby causing the LPIPS and DISTS results to be close to the AP-BSN results. That is, the fourth self-supervised loss Lnp prevents the first network (noise extractor h) to learn the aliasing artifacts on the pseudo-noise map to thus successfully reduce texture deformation.

Additionally, if the order-variant PD process in the first self-supervised loss Ls and the third self-supervised loss Lov is replaced with order-invariant PD (PDs(·, I<NUM>), overall performance degradation occurs.

According to the study results as shown in <FIG>, in the case where the respective self-supervised losses are omitted when I2V as the optimized learning of the predicted image producing unit and the basic learning process are used, some of performance may be degraded, but when all of the self-supervised losses are used, overall performance is upgraded.

<FIG> shows denoising results of blind denoising methods according to the present invention.

In <FIG>, completely self-supervised blind denoising tests are carried out, without external education dataset, to achieve wide adaptation of the I2V. In the tests, it is assumed that only target noisy images are used.

In the case of an image denoising method not paired with a supervised learning approach method, a self-supervised method is learned to allow the noise remover (e.g., the first network or the second network) to assume the true application scenario of the entire image using SIDD-Medium dataset.

The self-supervised learning-based method according to the present invention directly makes use of a subject noisy image to learn a deep learning model. If not so, the supervised denoising method or the image denoising method not paired may perform the learning with the clean and much noisy image included in different datasets. The different datasets have different texture or structure information from the subject noisy image. As a result, the method according to the present invention is similar in the LPIPS and DISTS to the supervised learning approach method or has better performance than the supervised learning approach method.

Referring to <FIG>, restoration transformer (Restormer) is a supervised learning-based denoising method, a clean-to-noisy image generation framework, namely C2N is an unpaired image denoising method requiring unpaired clean image data, and AP-BSN and I2V are methods only using noisy images. Bottom numbers represent the performance of PSNR/SSIM/LPIPS/DISTS in the order from the left.

As appreciated from <FIG>, the I2V according to the present invention shows the maximum effectiveness among state-of-the art self-supervised denoising methods for all matrices. Further, the I2V according to the present invention obtains better texture and LPIPS and DISTS results when compared with the most recent method, AP-BSN.

Further, the blind denoising method according to the present invention may be implemented in the form of a program instruction that can be performed through various computers, and may be recorded in a computer readable recording medium. The computer readable medium may include a program command, a data file, a data structure, and the like independently or in combination.

The program instruction recorded in the computer readable recording medium is specially designed and constructed for the present disclosure, but it may be well known to and may be used by those skilled in the art of computer software.

The computer readable recording medium may include a magnetic medium such as a hard disc, a floppy disc, and a magnetic tape, an optical recording medium such as a Compact Disc Read Only Memory (CD-ROM) and a Digital Versatile Disc (DVD), a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a Read Only Memory (ROM), a Random Access Memory (RAM), and a flash memory.

Further, the program command may include a machine language code generated by a compiler and a high-level language code executable by a computer through an interpreter and the like. The hardware device may be configured to operate as one or more software modules in order to perform operations of the present disclosure, and vice versa.

Claim 1:
A blind denoising device (<NUM>) comprising:
a receiving unit (<NUM>) for receiving an original noisy image (<NUM>);
a Pixel-shuffle Downsampling (PD) unit (<NUM>) for performing PD for the original noisy image (<NUM>) to produce at least one or more downsampled images (<NUM>);
a predicted image producing unit (<NUM>) for eliminating the noise from the original noisy image (<NUM>) and the downsampled images (<NUM>) to produce at least one or more predicted images (<NUM>) restored to the shape of the original noisy image (<NUM>); and
a learning unit (<NUM>) for performing optimized learning for the predicted image producing unit (<NUM>), based on at least one or more self-supervised losses of the predicted images (<NUM>) and the original noisy image (<NUM>),
characterized in that
the predicted image producing unit (<NUM>) comprises:
a first network (<NUM>) for eliminating the noise from the original noisy image (<NUM>) and the downsampled images (<NUM>);
a second network (<NUM>) for eliminating the noise from the downsampled images (<NUM>) according to at least one or more stride factors; and
a restoring part (<NUM>) for restoring the shape of the original noisy image (<NUM>) from the images from which the noise is eliminated through the first network (<NUM>) and the second network (<NUM>) to thus produce the predicted images (<NUM>).