Patent Description:
<NPL> discloses a modular denoising architecture.

Image noise refers to random variation in the color and/or brightness within images. Noise is a typical byproduct in images that are rendered using Monte Carlo rendering techniques. Images (i.e., photographs) captured by cameras can also contain noise due to the mechanics of camera sensors. For example, the quality of images captured in low-lighting conditions can be adversely affected by noise.

Image denoising is the process of removing noise from an image. Conventional approaches for image denoising utilize either specialized denoisers or "blind" denoisers. Specialized denoisers, such as deep denoising networks, can be trained to remove specific types of noise from images, where the noise is typically within narrow ranges of one or more noise parameters, such as additive white Gaussian noise having a certain magnitude. However, the performance of specialized denoisers drops rapidly for images having noise with noise parameters that are different than the training noise parameters.

In contrast to specialized denoisers, blind denoisers are typically trained using images having noise with diverse noise parameters. Although blind denoisers can be robust to different noise parameters, such robustness comes at the cost of overall denoising quality. In particular, the denoising quality of blind denoisers is generally lower than specialized denoisers for images having noise with the particular noise parameters that the specialized denoisers were trained to denoise.

Currently, there are few, if any, denoising techniques that optimally balance or combine the denoising quality of specialized denoisers with the denoising robustness of blind denoisers.

As the foregoing illustrates, what is needed in the art are more effective techniques for denoising images.

One example of the present application sets forth a computer-implemented method for denoising an image, according to claim <NUM>.

Another example of the present application sets forth a computer-implemented method for training denoisers. The method includes training each first denoiser included in a plurality of first denoisers using a respective set of images associated with at least one noise parameter and ground truth images corresponding to images included in the respective set of images. The method further includes training a second denoiser using an additional set of images, ground truth images corresponding to images included in the additional set of images, and a plurality of denoised images, where the plurality of denoised images is generated by processing the additional set of images using the plurality of first denoisers.

Other examples of the present disclosure include, without limitation, a computer-readable medium including instructions for performing one or more aspects of the disclosed techniques as well as a computing device for performing one or more aspects of the disclosed techniques.

The invention to which this European patent relates is defined in the appended claims.

At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques bypass the traditional tradeoff between the denoising quality of specialized denoisers and the generalizability and robustness of blind denoisers. In that regard, the disclosed techniques combine the performance of specialized denoisers with the generalizing capabilities of blind denoisers. Experience has shown that the disclosed techniques can achieve better denoising quality than conventional specialized denoisers when applied to images having noise with arbitrary noise parameters, such as different noise magnitudes. The disclosed techniques also can achieve better overall denoising quality than conventional blind denoisers. In addition, the disclosed techniques can denoise images more quickly than many conventional denoisers. These technical advantages represent one or more technological improvements over prior art approaches.

So that the manner in which the above recited features of the disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings.

In the following description, numerous specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to one of skill in the art that embodiments of the present invention may be practiced without one or more of these specific details.

<FIG> illustrates a system <NUM> configured to implement one or more aspects of the various embodiments. As shown, the system <NUM> includes a machine learning server <NUM>, a data store <NUM>, and a computing device <NUM> in communication over a network <NUM>, which may be a wide area network (WAN) such as the Internet, a local area network (LAN), or any other suitable network.

As shown, a model trainer <NUM> executes on a processor <NUM> of the machine learning server <NUM> and is stored in a system memory <NUM> of the machine learning server <NUM>. The processor <NUM> receives user input from input devices, such as a keyboard or a mouse. In operation, the processor <NUM> is the master processor of the machine learning server <NUM>, controlling and coordinating operations of other system components. In particular, the processor <NUM> may issue commands that control the operation of a graphics processing unit (GPU) that incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. The GPU may deliver pixels to a display device that may be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or the like.

The system memory <NUM> of the machine learning server <NUM> stores content, such as software applications and data, for use by the processor <NUM> and the GPU. The system memory <NUM> may be any type of memory capable of storing data and software applications, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash ROM), or any suitable combination of the foregoing. In some embodiments, a storage (not shown) may supplement or replace the system memory <NUM>. The storage may include any number and type of external memories that are accessible to the processor <NUM> and/or the GPU. For example, and without limitation, the storage may include a Secure Digital Card, an external Flash memory, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

It will be appreciated that the machine learning server <NUM> shown herein is illustrative and that variations and modifications are possible. For example, the number of processors <NUM>, the number of GPUs, the number of system memories <NUM>, and the number of applications included in the system memory <NUM> may be modified as desired. Further, the connection topology between the various units in <FIG> may be modified as desired. In some embodiments, any combination of the processor <NUM>, the system memory <NUM>, and a GPU may be replaced with any type of virtual computing system, distributed computing system, or cloud computing environment, such as a public, private, or a hybrid cloud.

As discussed in greater detail below, the model trainer <NUM> is configured to train machine learning models, including multiple specialized denoisers <NUM><NUM>-N, which are individually referred to as a specialized denoiser <NUM> and collectively referred to as specialized denoisers <NUM>, and a generalizer <NUM>. Each of the specialized denoisers <NUM> is trained to denoise images associated with specific ranges of noise parameters (i.e., images having noise with noise parameters that are within the specific ranges). For example, the specific ranges of noise parameters could indicate a particular type of camera sensor or a particular range of Gaussian noise magnitudes. By contrast, the generalizer <NUM> is trained to generate per-pixel kernels (i.e., individual kernels for each pixel) for denoising images associated with arbitrary noise parameters, as opposed to specific ranges of noise parameters. As described in greater detail below, the generalizer <NUM> receives as input an image to be denoised as well intermediate denoised images generated by the specialized denoisers <NUM> after the same image is input into the specialized denoisers <NUM>. Denoised images output by the specialized denoisers <NUM> are referred to herein as intermediate denoised images to distinguish them from the denoised images output by an overall denoising model <NUM> that includes both the specialized denoisers <NUM> and the generalizer <NUM>, discussed in greater detail below. The generalizer <NUM> is an example of a second denoiser (with the specialized denoisers <NUM> being the first denoisers) that can be trained and applied to denoise images given the images themselves, as well intermediate denoised images output by the specialized denoisers <NUM>, as input. Any technically feasible second denoiser may be used in other embodiments. For example, the second denoiser may generate outputs other than denoising kernels, such as denoised images, in some embodiments.

The architectures of the specialized denoisers <NUM> and the generalizer <NUM>, as well as techniques for training the same, are discussed in greater detail below. Training data and/or trained machine learning models, including the specialized denoisers <NUM> and the generalizer <NUM>, may be stored in the data store <NUM>. In some embodiments, the data store <NUM> may include any storage device or devices, such as fixed disc drive(s), flash drive(s), optical storage, network attached storage (NAS), and/or a storage area-network (SAN). Although shown as accessible over the network <NUM>, in some embodiments the machine learning server <NUM> may include the data store <NUM>.

The trained specialized denoisers <NUM> and generalizer <NUM> may be deployed to any suitable applications that denoise images. Illustratively, a denoising application <NUM> is stored in a memory <NUM>, and executes on a processor <NUM> of the computing device <NUM>. Components of the computing device <NUM>, including the memory <NUM> and the processor <NUM> may be similar to corresponding components of the machine learning server <NUM>.

As shown, the denoising application <NUM> includes the denoising model <NUM>, which itself includes the specialized denoisers <NUM> and the generalizer <NUM>. As described, each of the specialized denoisers <NUM> is a machine learning model trained to denoise images associated with a specific range of noise parameters. In some embodiments, the specialized denoisers <NUM> may be denoising convolutional neural networks (DnCNNs), discussed in greater detail below. In alternative embodiments, the specialized denoisers <NUM> may be other types of denoisers, such as U-net architecture networks. Any number and type of specialized denoisers <NUM> may be employed in embodiments. For example, three specialized denoisers may be used in some embodiments. In general, if the parameter space of the noise is more complex, then more specialized denoisers may be used, and vice versa if the parameter space of the noise is simpler. It should be noted that there is a trade-off between denoising performance and the number of specialized denoisers <NUM>, as adding specialized denoisers <NUM> generally increases computational costs.

As described, the generalizer <NUM> is trained to generate per-pixel denoising kernels for denoising images having noise with arbitrary noise parameters. The generalizer <NUM> takes as input noisy images, as well as intermediate denoised images generated by the specialized denoisers <NUM>, which are input into the generalizer <NUM> as additional features and also referred to herein as "denoised-image features. " The noisy image may generally be any image that a user wishes to denoise. Examples of such images include live-action images captured by a camera and images rendered via Monte Carlo rendering techniques. For example, a live-action image could be denoised before visual effects are added. As another example, denoising could be applied to a high dynamic range image, which can include a relatively large amount of noise. As yet another example, denoising could be used to accelerate rendering by denoising an initial noisy image generated via Monte Carlo rendering.

In some embodiments, the generalizer <NUM> is a kernel-generating machine learning model, such as a kernel predicting convolutional network (KPCN). As discussed in greater detail below in conjunction with <FIG> and <FIG>, the denoising application <NUM> can denoise a noisy image by first processing the noisy image using the specialized denoisers <NUM> to generate intermediate denoised images. Then, the denoising application <NUM> inputs the intermediate denoised images generated by the specialized denoisers <NUM> as features, in addition to the noisy image, into the generalizer <NUM>. In turn, the generalizer <NUM> outputs per-pixel denoising kernels, which the denoising application <NUM> normalizes and applies to the noisy image to generate a denoised image.

In the embodiments, discussed in greater detail below in conjunction with <FIG> and <FIG>, multi-scale denoising is performed to more effectively remove low-frequency components of noise. In such cases, a noisy image is downsampled one or more times to different scales prior to being input into the denoising model <NUM> that includes the specialized denoisers <NUM> and the generalizer <NUM>. Then, the denoised outputs of the denoising model <NUM> at the different scales are combined to generate a final denoised image. This is in contrast to simply processing the noisy image using the specialized denoisers <NUM> and then inputting the noisy image and the intermediate denoised images generated by the specialized denoisers <NUM> into the generalizer <NUM>, as described above.

Experience has shown that the denoising model <NUM> can achieve better denoising quality than conventional specialized denoisers. The denoising model <NUM> can also generalize better than conventional blind denoisers. In addition, the denoising model <NUM> can denoise images more quickly than some conventional denoisers. It should also be noted that the denoising model <NUM> does not require noise parameters to be explicitly estimated or a camera imaging pipeline to be modeled.

The number of machine learning servers and application servers may be modified as desired. Further, the functionality included in any of the applications may be divided across any number of applications or other software that are stored and execute via any number of devices that are located in any number of physical locations.

<FIG> is a more detailed illustration of the denoising model <NUM> of <FIG>, according to various embodiments. In <FIG> and <FIG>, rectangles with rounded edges represent software components and rectangles with square edges represent data, which may be input into and/or output from the software components.

As shown, the denoising model <NUM> is used by the denoising application <NUM> to process a noisy image <NUM> and obtain a denoised image <NUM>. As described, the noisy image <NUM> may be, e.g., a live-action image captured by a camera or a rendered image. Although described herein primarily with respect to receiving and denoising a single image at a time, in some embodiments, the denoising application <NUM> may process multiple image frames from a video at once, such as a number of consecutive image frames before and after a given image frame that is being denoised. Doing so allows the denoising application <NUM> to consider temporal neighborhoods as well as spatial neighborhoods of each pixel.

The denoising application <NUM> first inputs the noisy image <NUM> into the specialized denoisers <NUM>. As described, the specialized denoisers <NUM> output intermediate denoised images that are input as additional features <NUM><NUM>-N, along with the noisy image <NUM>, into a second denoiser, which as shown is the generalizer <NUM>. Each of the features <NUM><NUM>-N represents characteristics of the noisy image <NUM>, namely an intermediate denoised version of the noisy image <NUM>, that is input into the generalizer <NUM>. The features <NUM><NUM>-N are referred to herein individually as a feature <NUM> and collectively as features <NUM>. As the features <NUM> include denoised images, the features <NUM> are also referred to herein as "denoised-image features. " Although described herein with respect to denoised-image features for simplicity, some embodiments may actually employ three channels, namely the red, green, and blue channels, for each of the denoised-image features as well as for the noisy image.

As described, each of the specialized denoisers <NUM> is trained to denoise images associated with a specific range of noise parameters. The noise parameters could indicate, e.g., a particular type of camera sensor or a particular range of Gaussian noise magnitudes. In some embodiments, each of the specialized denoisers <NUM> is a DnCNN. The DnCNN architecture includes blocks of convolutional layers (including, e.g., <NUM> × <NUM> filters and <NUM> channels) followed by batch normalization and rectified linear unit (ReLU) activations, as well as a final layer (e.g., a <NUM> × <NUM> layer) that produces an output. In alternative embodiments, the specialized denoisers <NUM> may be other types of denoisers, such as U-net architecture networks.

More formally, each of the specialized denoisers <NUM> can be trained using noisy images having noise with a specific set of noise parameters {λ<NUM>,λ<NUM>,. Subsequent to training, the specialized denoisers <NUM> may generally perform well in denoising images associated with noise parameters similar to those used to train the specialized denoisers <NUM>. However, the denoising performance of the specialized denoisers <NUM> drops rapidly for images having noise with different noise parameters.

To alleviate the inability of the specialized denoisers <NUM> to generalize over noise parameters, the denoising application <NUM> further processes the noisy image <NUM> using the generalizer <NUM>. The generalizer <NUM> receives as inputs the noisy image <NUM> and intermediate denoised images, which are output by the specialized denoisers <NUM> and input as the features <NUM> into the generalizer <NUM>. As shown, the generalizer <NUM> outputs per-pixel denoising kernels <NUM>, which are individually referred to as a denoising kernel <NUM> and collectively referred to as per-pixel denoising kernels <NUM>. In some embodiments, the generalizer <NUM> may be a KPCN. For computational efficiency reasons, the number of denoised image features will generally be low, and the noise parameters associated with an image λa may not match the noise parameters {λ<NUM>, λ<NUM>,. , λs} that the specialized denoisers <NUM> were trained on. The generalizer <NUM> is therefore used to estimate suitable per-pixel denoising kernels <NUM> for the noisy image <NUM> associated with noise parameters λa of any input image given the image itself and denoised-image features output by the specialized denoisers <NUM>. Doing so produces a relatively consistent denoising quality over a wide range of noise parameters, making the denoising model <NUM> robust to image noise characteristics.

If the input noisy image <NUM> is treated as a vector, which can be denoted as <MAT>, then the denoised-image features are additional channels f, each of which includes an individual feature map {f<NUM>, f<NUM>,. The generalizer <NUM> takes as input the tuple {x, f}. The generalizer <NUM> is trained to minimize an average distance between estimated denoised images x̂ and corresponding noise-free ground truth images y in a training data set. The noise-free ground truth images in the training data set are also referred to herein as "clean" images and may be images to which noise (e.g., Gaussian noise) has been applied to generate noisy images. It should be understood that the training data may generally include such noisy images as well as the corresponding clean images. More formally, the generalizer <NUM> may be expressed as x̂ = d({x, f}; θ̂), where d denotes a denoiser with parameters θ̂. During training, the parameters θ̂ are determined in a supervised setting using the dataset {{x<NUM>, y<NUM>}, {x<NUM>, y<NUM>},. , {xn, yn}}, with the objective, or loss function, being: <MAT>.

The trained generalizer <NUM> can be used to estimate a k × k kernel of scalar weights around the neighborhood <IMG>(p) of a pixel location p, which is also referred to herein as the denoising kernel <NUM> for the pixel location p. The denoising application <NUM> may then use such a denoising kernel <NUM> to compute a weighted linear combination of the pixels in the neighborhood <IMG>(p) to obtain the final pixel color prediction at the pixel location p. As shown, a normalization module <NUM> of the denoising application <NUM> then normalizes each of the per-pixel denoising kernels <NUM> to generate normalized kernels <NUM>, after which a weighted reconstruction module <NUM> of the denoising application <NUM> applies the normalized kernels <NUM> to denoise the noisy image <NUM> via weighted reconstruction. In some embodiments, the denoising application <NUM> normalizes weights of the per-pixel denoising kernels <NUM> using a softmax function so that the weights have values within [<NUM>,<NUM>]. The denoised image <NUM> is then output from the denoising model <NUM>.

As described, the generalizer <NUM> is a KPCN in some embodiments. The KPCN architecture includes residual blocks, each of which can include <NUM> × <NUM> convolution layers bypassed by a skip connection. As described, rather than directly predicting pixel colors, KPCNs predict per-pixel denoising kernels, which can provide significant improvement in convergence speed during training. For example, the KPCN could output a <NUM> × <NUM> denoising kernel for each pixel. In some embodiments, the KPCN generalizer <NUM> estimates a denoising kernel at pixel location p using the following identity: <MAT> where wpq denotes a normalized estimated weight at location q belonging to the kernel at pixel location p.

<FIG> illustrate an exemplar noisy image and denoised-image features generated by the specialized denoisers <NUM>, according to various embodiments. As shown in <FIG>, a noisy image <NUM> depicts a wooden floor. <FIG> shows intermediate denoised image outputs <NUM>, <NUM>, and <NUM> generated by three specialized denoisers <NUM> given the noisy image <NUM> as input. As described, such intermediate denoised images <NUM>, <NUM>, and <NUM> can be input as additional denoised-image features into the generalizer <NUM>.

As the specialized denoisers <NUM> are trained for particular noise parameters, such as a specific noise type (e.g., noise produced by a specific type of camera sensor) or a range of noise parameters (e.g., additive Gaussian noise with a specific magnitude), the quality of each intermediate denoised image <NUM>, <NUM>, and <NUM> will vary depending on the noise parameters associated with the input noisy image <NUM>. As described, the generalizer <NUM> can be used to improve the final denoising quality by taking as input the intermediate denoised images <NUM>, <NUM>, and <NUM> as features in addition to the noisy image <NUM>.

<FIG> is a more detailed illustration of the denoising application <NUM> of <FIG>, according to various embodiments. This illustration assumes that multi-scale denoising is performed to more effectively remove low-frequency components of noise. As described, such multi-scale denoising is optional, and, in other embodiments, the denoising application <NUM> may simply process a noisy image using the specialized denoisers <NUM> and then process the noisy image and intermediate denoised images generated by the specialized denoisers <NUM> using the generalizer <NUM> to obtain per-pixel denoising kernels that can be used to denoise the original, noisy image.

As shown in <FIG>, downsampling modules <NUM> and <NUM> of the denoising application <NUM> downsample an input image <NUM> to generate downsampled images <NUM> and <NUM>, respectively. In some embodiments, the input image <NUM> is uniformly downsampled to half the original size by the downsampling module <NUM>, and then downsampled again to half the previously downsampled size (i.e., to a quarter of the original size) by the downsampling module <NUM>. Although one input image <NUM> is shown for illustrative purposes, in some embodiments, the denoising application <NUM> may process multiple image frames from a video, such as a number of consecutive image frames before and after a given image frame that is being denoised.

As shown, the denoising application separately inputs the image <NUM> and the downsampled images <NUM> and <NUM> into the denoising models <NUM><NUM>-<NUM>, to obtain an intermediate denoised image <NUM>, an intermediate denoised image <NUM>, and an intermediate denoised image <NUM>, respectively. Each of the denoising models <NUM><NUM>-<NUM> shown in <FIG> is the same as the denoising model <NUM>, and processing of the image <NUM> and the downsampled images <NUM> and <NUM> using the denoising models <NUM><NUM>-<NUM> may either be performed in parallel or serially. As described, the generalizer <NUM> of the denoising model <NUM> outputs, for each pixel location, a respective kernel (e.g., a fixed-size <NUM> × <NUM> kernel). By determining such per-pixel kernels at different scales obtained via downsampling, the effective kernel size is increased. Doing so increases the perceptive field and helps remove low-frequency components of noise, which can be more challenging to remove and require larger filter kernels.

The intermediate denoised images <NUM>, <NUM>, and <NUM> are combined, starting from a combination of the coarsest denoised image <NUM> with the finer denoised image <NUM> using a scale compositor <NUM> and proceeding to a combination of an output of the scale compositor <NUM> with the finest denoised image <NUM> using a scale compositor <NUM>. The output of the scale compositor <NUM> is a denoised image <NUM> that is output by the denoising application <NUM>. In some embodiments, each of the scale compositors <NUM> and <NUM> takes as input a pair of coarse and fine scale images (ic and if), such as the intermediate denoised image <NUM> and the intermediate denoised image <NUM> or the output of the scale compositor <NUM> and the intermediate denoised image <NUM>, as shown in <FIG>. In such cases, each of the scale compositors <NUM> and <NUM> extracts, for each pixel p, a scalar weight αp that is used to blend consecutive scales to generate a blended image as follows: <MAT> where D denotes downsampling (e.g., a <NUM> × <NUM> downsampling) and U denotes upsampling (e.g., a <NUM> × <NUM> upsampling). Such a blending replaces low frequencies of the fine scale image iƒ with low frequencies obtained from the coarse scale image ic and produces the output oƒ. The scalar weight αp balances low frequencies from the finer and coarser scales. In some embodiments, the scalar weight αp may be determined by taking the intermediate images produced by denoising two adjacent scales of a frame (or a sequence), namely a coarse-scale image ic and a fine-scale image iƒ, and inputting those intermediate denoised images into a convolutional network that extracts a per-pixel scalar weight αp, which can then be used to blend the coarse- and fine-scale intermediate denoised images according to equation (<NUM>). Although described herein primarily with respect to successive scale compositors that each combine a pair of coarse and fine scale intermediate denoised images, in alternative embodiments a single scale compositor that combines intermediate denoised images at all of the different scales (e.g., the denoised images <NUM>, <NUM>, and <NUM>) may be used.

<FIG> is a more detailed illustration of the model trainer <NUM> of <FIG>, according to various embodiments. As shown, a two-step training procedure is employed in which the model trainer <NUM> uses training data <NUM> that includes a set of noisy images and corresponding clean images to first train the specialized denoisers <NUM>. Then, the model trainer <NUM> uses the training data <NUM> and the trained specialized denoisers <NUM> to train the generalizer <NUM>. As described, each of the specialized denoisers <NUM> is trained using noisy images having noise with a different set of noise parameters and corresponding clean images. Accordingly, the training of each of the specialized denoisers <NUM> at <NUM> will generally only use some of the noisy images (and corresponding clean images) in the training data <NUM> having noise with the particular set of noise parameters for which the specialized denoiser <NUM> is being trained. By contrast, a generalizer training module <NUM> can use a more diverse set of images to train the generalizer <NUM>, such as all of the noisy images in the training data <NUM> and corresponding clean images, as well as intermediate denoised images output by the specialized denoisers <NUM> when given the noisy images from the training data <NUM>. For example, Gaussian noise of random sigma ranging from <NUM> to <NUM> could be added to each image in a data set, and such noisy images as well as intermediate denoised images that are generated by the specialized denoiser <NUM> can be fed into the generalizer <NUM> during training.

Embodiments may utilize any technically feasible training techniques to train the specialized denoisers <NUM> and the generalizer <NUM>. In some embodiments, both the generalizer <NUM> and the specialized denoisers <NUM> are trained using image patches of size <NUM> × <NUM>, using mini-batch size <NUM> and the Adam optimizer with an initial learning rate of <NUM>-<NUM>, as well as dataset specific schedulers to decay the learning rate during the course of training. In some embodiments, the training may use the mean absolute percentage error (MAPE) to assess the distance to a clean reference in the training data <NUM>: <MAT> where x̂ is the denoised image, y is the clean reference image, and ε = <NUM>-<NUM> is used to avoid division by zero. Other loss functions, such as L<NUM>, root mean squared error (RMSE), and structural similarity (SSIM) may be used in alternative embodiments.

<FIG> illustrate an exemplar noisy image <NUM>, a denoised image <NUM> generated using a conventional blind denoiser, and a denoised image <NUM> generated using the specialized denoisers <NUM> and the generalizer <NUM> of <FIG>, according to various embodiments. Illustratively, the noisy image <NUM> is a live-action image. As described, live-action images can be denoised before visual effects are added and in the case of high dynamic range images that include a relatively large amount of noise, among other things. Although a live-action image is shown for illustrative purposes, techniques disclosed herein can also be applied to denoise rendered images, such as those generated using Monte Carlo rendering techniques. In contrast to rendered images, no additional information (e.g., albedo, surface normal, depth, etc.) beyond colors are typically available for live-action images that a denoiser can take advantage of.

As shown in <FIG>, the denoised image <NUM> generated from the noisy image <NUM> using the specialized denoisers <NUM> and the generalizer <NUM> has higher quality than the denoised image <NUM> generated using the conventional blind denoiser, shown in <FIG>. Quality may be measured as, e.g., the average distance from a corresponding clean image. Experience has shown that the specialized denoisers <NUM> and the generalizer <NUM> can generate denoised images that are sharper and more robust to varying levels of noise magnitudes compared to conventional blind denoisers. In addition, denoised images generated using the specialized denoisers <NUM> and the generalizer <NUM> can be at least as good as specialized denoisers for images having noise with noise parameters that the specialized denoisers were trained to denoise.

<FIG> illustrate another exemplar noisy image <NUM>, a denoised image <NUM> generated using a conventional specialized denoiser, and a denoised image <NUM> generated using the specialized denoisers <NUM> and the generalizer <NUM> of <FIG>, according to various embodiments. Zoomed-in views <NUM> and <NUM> of regions <NUM> and <NUM>, respectively, within the noisy image are shown in <FIG>. Likewise, zoomed-in views <NUM> and <NUM> of regions <NUM> and <NUM>, respectively, within the denoised image <NUM> are shown in <FIG>, and zoomed-in views <NUM> and <NUM> of regions <NUM> and <NUM>, respectively, within the denoised image <NUM> are shown in <FIG>.

Similar to the noisy image <NUM>, the noisy image <NUM> is shown as a live-action image, but may alternatively be a rendered image. As shown in <FIG>, the denoised image <NUM> generated from the noisy image <NUM> using the specialized denoisers <NUM> and the generalizer <NUM> has higher quality than the denoised image <NUM> generated using the conventional specialized denoiser, shown in <FIG>. In generating the denoised image <NUM>, a specialized U-Net was used as the conventional specialized denoiser. In generating the denoised image <NUM>, five U-Nets and a KPCN were used as the specialized denoisers <NUM> and the generalizer <NUM>, respectively.

<FIG> sets forth a flow diagram of method steps for training the specialized denoisers <NUM> and the generalizer <NUM>, according to various embodiments. Although the method steps are described in conjunction with the system of <FIG>, persons of ordinary skill in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present disclosure.

As shown, a method <NUM> begins at step <NUM>, where the model trainer <NUM> trains the specialized denoisers <NUM> using respective training sets of images that are associated with different sets of noise parameters. As described, each of the specialized denoisers <NUM> is trained to denoise images associated with a specific range of noise parameters.

At step <NUM>, the model trainer <NUM> trains the generalizer <NUM> using an additional training set of images and denoised images output by the specialized denoisers <NUM> after the additional set of images is input into the trained specialized denoisers <NUM>. As described, the generalizer <NUM> is trained to take as inputs a noisy image and intermediate denoised images generated by the specialized denoisers <NUM> given the same noisy image, and to output per-pixel denoising kernels. In some embodiments, the generalizer <NUM> is trained using a more diverse set of images than the sets of images used to train each of the specialized denoisers <NUM> individually.

<FIG> sets forth a flow diagram of method steps for denoising an image, according to various embodiments. Although the method steps are described in conjunction with the system of <FIG>, persons of ordinary skill in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present disclosure.

As shown, a method <NUM> begins at step <NUM>, where the denoising application <NUM> receives a noisy image. The noisy image may be, e.g., a live-action image or a rendered image. Further, the noisy image may either be a stand-alone image or an image frame within a video. In some embodiments, the denoising application <NUM> may receive and process other images in addition to the noisy image, such as a number of consecutive image frames before and after the noisy image, as discussed above in conjunction with <FIG> and <FIG>.

At step <NUM>, the denoising application <NUM> downsamples the noisy image. As described, downsampling is performed in the multi-scale case to increase the perceptive field and remove low-frequency components of noise. In some embodiments, the noisy image may be uniformly downsampled to half the original size during each of one or more downsampling operations.

At step <NUM>, the denoising application <NUM> processes each of the noisy image and the downsampled images using the specialized denoisers <NUM> to generate intermediate denoised images that are to be used as denoised-image features. As described, each of the specialized denoisers <NUM> is trained to denoise images associated with a specific range of noise parameters. Intermediate denoised images output by the specialized denoisers <NUM> are then input as additional features, along with the original noisy image, into the generalizer <NUM>.

At step <NUM>, the denoising application <NUM> processes each of the noisy image and the downsampled images, along with associated denoised-image features, using the generalizer <NUM> to generate additional intermediate denoised images. As described, the generalizer <NUM> is a kernel predicting network, such as a KPCN, in some embodiments. In such cases, the denoising application <NUM> processes each of the noisy image and the downsampled images, along with associated denoised-image features, using the kernel predicting network to obtain respective per-pixel denoising kernels. Then, the denoising application <NUM> applies the respective denoising kernels to the noisy image and the downsampled images to generate the additional intermediate denoised images at step <NUM>. As described, other embodiments may employ a second denoiser other than the generalizer <NUM> that is trained and applied to denoise images given the images themselves, as well intermediate denoised images output by the specialized denoisers <NUM>, as input.

If the downsampling at step <NUM> is not performed, then the denoised image generated by processing the noisy image at step <NUM> is the final denoised image that is output by the denoising application <NUM>. Otherwise, the method <NUM> proceeds to step <NUM>.

At step <NUM>, the denoising application <NUM> combines the intermediate denoised images to generate a denoised image. This step assumes that the denoising application <NUM> downsampled the noisy image at step <NUM>. As described, in some embodiments, the denoising application <NUM> combines the intermediate denoised images using scale compositors, beginning with a coarsest intermediate denoised image and proceeding to finer intermediate denoised images.

In sum, techniques are disclosed for training and applying a denoising model that is capable of removing noise from images where attributes of the noise are not known a priori. The denoising model includes multiple specialized denoisers and a generalizer, each of which is a machine learning model. The specialized denoisers are trained to denoise images associated with specific ranges of noise parameters. The generalizer is trained to generate per-pixel denoising kernels for denoising images associated with arbitrary noise parameters using outputs of the trained specialized denoisers. Subsequent to training, a noisy image, such as a live-action image or a rendered image, can be denoised by inputting the noisy image into the specialized denoisers to obtain intermediate denoised images that are then input, along with the noisy image, into the generalizer to obtain per-pixel denoising kernels, which can be normalized and applied to denoise the noisy image.

Many modifications and variations will be apparent to those of ordinary skill in the art, and are encompassed by the present disclosure insofar as they fall within the scope of the appended claims.

Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "module" or "system. " Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable.

Claim 1:
A computer-implemented method for denoising an image (<NUM>), the method comprising:
processing the image using a plurality of first denoisers to generate a plurality of first denoised images, wherein each first denoiser comprises a specialized denoiser (<NUM>i) trained to denoise images associated with a specific respective range of noise parameters; and
processing the image and the plurality of first denoised images using a second denoiser to generate a second denoised image (<NUM>), wherein the second denoiser comprises a generalizer (<NUM>) trained for denoising images associated with arbitrary noise parameters;
the method further comprising:
downsampling the image to generate a downsampled image;
processing the downsampled image using the plurality of first denoisers to generate a plurality of third denoised images;
processing the downsampled image and the plurality of third denoised images using the second denoiser to generate a fourth denoised image; and
combining the second denoised image and the fourth denoised image to produce a first blended image.