METHOD AND APPARATUS WITH NOISE CONSIDERATION

A method and apparatus with noise consideration are provided. The method includes generating, using a noise model, a non-normal noise map corresponding to a noise of an input image, and generating an enhanced image of the input image by implementing an image enhancement model based on the input image and the non-normal noise map, where the noise of the input image follows a non-normal distribution.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2021-0132089 filed on Oct. 6, 2021, in the Korean Intellectual Property Office, the entire disclosures of which are incorporated herein by reference for all purposes.

BACKGROUND

The following description relates to method and apparatus with noise consideration.

2. Description of Related Art

Image enhancement is a technology that may include enhancing an image of low or degraded quality. A deep learning-based neural network may be used for performing image enhancement of an image.

After being trained based on deep learning for a special inference purpose, the neural network may perform an inference according to the special purpose, such as by intuitively mapping input data and output data that are in a nonlinear relationship with each other. The trained capability of such a neural network to intuitively generate such mappings may be referred to as a learning or trained capability of the neural network. Further, because of the specialized training, the neural network may thereby perform the image enhancement with a generalization capability to generate a relatively accurate image enhancement of an input pattern or image that the neural network may not have been trained for, for example.

As noted above, a deep learning-based neural network may be used for performing image enhancement of an image, but the training and inference operations may be based on a presumption that noise of the image follows a normal distribution and has a linear characteristic, but real image noise may not follow such a normal distribution. In addition, while theoretically the training of a deep learning-based model with noisy and clean image training pairs should be sufficiently accurate and straight forward, when dealing with real noise of real images it becomes onerous to obtain enough training pairs of captured noisy images and corresponding captured images without noise for training purposes, and there may be pixel position misalignments within training pairs. So, training may be difficult and less accurate for real images with real noise.

SUMMARY

In one general aspect, a processor-implemented method includes generating, using a noise model, a non-normal noise map corresponding to a noise of an input image, and generating an enhanced image of the input image by implementing an image enhancement model based on the input image and the non-normal noise map, where the noise of the input image follows a non-normal distribution.

The generating of the non-normal noise map may include extracting a first non-normal noise distribution information for a first pixel of the input image from first mapping information of the noise model based a first clean pixel value corresponding to the first pixel, pixel position information of the first clean pixel or of the first pixel, and image capturing parameters regarding a capturing of the input image, determining a first noise value of the first pixel from among plural noise information in the first non-normal noise distribution information for the first pixel, extracting a second non-normal noise distribution information for a second pixel of the input image from second mapping information of the noise model based a second clean pixel value corresponding to the second pixel, pixel position information of the second clean pixel or of the second pixel, and the image capturing parameters regarding the capturing of the input image, determining a second noise value of the second pixel from among plural noise information in the second non-normal noise distribution information for the second pixel, and generating the non-normal noise map based on the first noise value and the second noise value, and the generating of the enhanced image may be dependent at least on the first noise value and the second noise value.

The generating of the non-normal noise map may include, for each pixel of the input image, extracting a non-normal noise distribution information of a pixel of the input image from mapping information of the noise model based on a clean pixel data corresponding to the pixel of the input image, pixel position information of the pixel of the input image or of the clean pixel data corresponding to the pixel of the input image, and image capturing parameters regarding a capturing of the input image, determining a noise value of the pixel of the input image using the non-normal noise distribution information of the pixel, and generating the non-normal noise map based on the noise value of the pixel of the input image.

For each pixel of the input image, the mapping information may include a lookup table (LUT) that maps input data corresponding to the pixel, the pixel position information, and the image capturing parameters, and output data corresponding to the non-normal noise distribution information of the pixel.

The non-normal noise distribution information of the pixel may include first non-normal noise distribution information of the pixel, where the first non-normal noise distribution information may represent a first non-normal distribution for the pixel mapped to a clean pixel value of the pixel, the position information, and the image capturing parameters, and where the noise value of the pixel may be randomly determined from among plural information in the first non-normal distribution.

The non-normal noise distribution information of the pixel may include at least one of first distribution information may represent a distribution of non-normal noise data and second distribution information may represent a distribution of all normal distribution values of the non-normal noise data obtained by projecting the non-normal noise data into a normal distribution space.

The noise value of the pixel may be randomly determined from among the distribution of all normal distribution values.

The non-normal noise distribution information of the pixel may include at least one of first distribution information may represent a distribution of non-normal noise data and second distribution information may represent a distribution of a portion of all normal distribution values of the non-normal noise data obtained by projecting the non-normal noise data into a normal distribution space, and the second distribution information may include a maximum value and a minimum value as the portion of all of the normal distribution values.

The noise value of the pixel may be randomly determined from among a range of values between the maximum value and the minimum value.

The clean pixel data corresponding to the pixel of the input image may be of a clean image obtained through blur filtering performed on the input image.

The image capturing parameters may include at least one of an International Organization for Standardization (ISO) value, an exposure time, and Bayer pattern information.

The generating of the non-normal noise map may include, for each pixel of the input image, generating a normal noise data value corresponding to first partial input data of the pixel of the input image using a normal distribution-based first noise model, generating non-normal noise data value corresponding to second partial input data of the pixel of the input image using a non-normal distribution-based second noise model, and generating the non-normal noise map based on the normal noise data value and the non-normal noise data value.

In one aspect, only the first partial input data may include pixel value information with respect the pixel, and both of the first partial input data and the second partial input data may include some overlapping image capturing parameters regarding a capturing of the first noise modeling image.

The noise of the input image follows the non-normal distribution corresponding to image processing performed on raw image data of the input image.

The image enhancement model may be a machine learning model that may be trained in advance to provide intuitive mapping relationships, for each of a training input data and a training output data of respective plural training input data and plural training output data, between the respective plural training input data and plural training output data, the training input data may include a corresponding training image and a corresponding training non-normal noise map may be generated, using the noise model, corresponding to a noise of the corresponding training image, and the output data may include an enhanced image of the training image.

The image enhancement model may be trained in advance based on training images generated using the noise model.

The method may further include training the noise model based on plural modeling images having respective noises that follow non-normal distributions, including training the noise model based on a noise and pixel position information for each modeling image of the plural modeling images, and training the image enhancement model by, for each training data set of a plurality of training data sets, generating a degraded training image of a clean training image using the noise model, determining a training data set based on the clean training image and the degraded training image, and training the image enhancement model based on the training data set.

The method may further include training the noise model, including, for each of a noise modeling image of a plurality of noise modeling images whose noises each follow non-normal distributions, generating a clean modeling image from the noise modeling image, determining non-normal noise data of the noise modeling image based on a difference between the noise modeling image and the clean modeling image, and determining mapping information of the noise model based on a mapping relationship, for each pixel of the noise modeling image, that is based on distribution information of the non-normal noise data for the pixel, pixel data of the clean modeling image corresponding to the pixel, and pixel position information corresponding to the pixel.

In one general aspect, one or more embodiments include a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform any one, any combination, or all operations described herein.

In one aspect, a processor-implemented method includes training a noise model based on plural modeling images having respective noises that follow non-normal distributions, including training the noise model based on a noise and pixel position information for each modeling image of the plural modeling images, and training an image enhancement model by, for each training data set of a plurality of training data sets, generating a degraded training image of a clean training image using the noise model, determining a training data set based on the clean training image and the degraded training image, and training the image enhancement model based on the training data set.

The training of the noise model may include generating a first clean modeling image from a first noise modeling image that follows a non-normal distribution, determining non-normal noise data of the first noise modeling image based on a difference between the first noise modeling image and the first clean modeling image, and determining mapping information of the noise model based on a mapping relationship that may be based on distribution information of the non-normal noise data, pixel data of the first clean modeling image, and the pixel position information.

The distribution information may include at least one of first distribution information may represent a distribution of the non-normal noise data and second distribution information may represent a distribution of at least a portion of all normal distribution values of the non-normal noise data obtained by projecting the non-normal noise data into a normal distribution space.

The second distribution information may represent only the portion of the all normal distribution values, and the second distribution information may include a maximum value and a minimum value as the portion of all of the normal distribution values.

The noise model may include a normal distribution-based first noise model and a non-normal distribution-based second noise model, and the training of the noise model may include determining normal noise data corresponding to first partial input data for a first noise modeling image that follows a non-normal distribution, using the first noise model, determining residual data based on a difference between the normal noise data and non-normal noise data of the first noise modeling image, and determining mapping information of the second noise model based on a mapping relationship that may be based on second partial input data of the first noise modeling image and the residual data.

In one aspect, only the first partial input data may include pixel value information with respect the first noise modeling image, and both of the first partial input data and the second partial data may include some overlapping image capturing parameters regarding a capturing of the first noise modeling image.

The method may further include generating, using the noise model, a non-normal noise pixel information corresponding to a noise of an input image, and generating an enhanced image of the input image by implementing the image enhancement model based on the input image and the non-normal noise pixel information, wherein the noise of the input image follows a non-normal distribution.

In one general aspect, an electronic device includes a processor, and a memory storing instructions, which when executed by the processor configure the processor to generate, using a noise model, a non-normal noise pixel information corresponding to a noise of an input image, and generate an enhanced image of the input image by implementing an image enhancement model based on the input image and the non-normal noise pixel information, where the noise of the input image follows a non-normal distribution.

The device may further include a storage device, where the generated enhanced image may be stored in the storage device or the memory.

The device may further include a camera, and a display, where the input image may be an image captured by the camera, and the instructions further include display instructions, which when executed by the processor, configure the processor to control the display to display the enhanced image.

The instructions, which when executed by the processor, may further configure the processor to extract a first non-normal noise distribution information for a first pixel of the input image from first mapping information of the noise model based a first clean pixel value corresponding to the first pixel, pixel position information of the first clean pixel or of the first pixel, and image capturing parameters regarding a capturing of the input image, determine a first noise value of the first pixel from among plural noise information in the first non-normal noise distribution information for the first pixel, extract a second non-normal noise distribution information for a second pixel of the input image from second mapping information of the noise model based a second clean pixel value corresponding to the second pixel, pixel position information of the second clean pixel or of the second pixel, and the image capturing parameters regarding the capturing of the input image, determine a second noise value of the second pixel from among plural noise information in the second non-normal noise distribution information for the second pixel, and generate the non-normal noise pixel information based on the first noise value and the second noise value, where the generation of the enhanced image may be dependent at least on the first noise value and the second noise value.

Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same or like elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

Throughout the specification, when a component or element is described as being “connected to,” or “coupled to” another component or element, it may be directly “connected to,” or “coupled to” the other component or element, or there may reasonably be one or more other components or elements intervening therebetween. When a component or element is described as being “directly connected to,” or “directly coupled to” another component or element, there can be no other elements intervening therebetween. Likewise, expressions, for example, “between” and “immediately between” and “adjacent to” and “immediately adjacent to” may also be construed as described in the foregoing.

FIG.1illustrates an example of image enhancement, according to one or more embodiments. Referring toFIG.1, a computing apparatus100may receive or obtain an input image101, and generate an enhanced image103by enhancing a quality of the input image101using a noise model110and an image enhancement model120. The input image101may be an image captured or generated, or otherwise obtained, by a camera, an image received or otherwise obtained through a network interface, or an image stored or obtained in advance in a storage device, such as described in more detail further below with respect to the configuration of representative camera(s), network interface(s), and storage device(s) shown inFIG.19, noting that alternative examples of the obtaining of the input image are further available. The input image101may include a degradation characteristic, e.g., a noise, a blur, etc., and the computing apparatus100may remove the degradation characteristic by the collective operation of the noise model110and the image enhancement model120.

The computing apparatus100may estimate a noise distribution102of the input image101by implementing the noise model110dependent or based on the input image101, e.g., implementing the noise model110provided the input image101as an input to the noise model110. The input image101may have a noise that follows a non-normal distribution. For example, the noise of the input image101may follow any non-normal distribution, such as noise of the input image101having characteristics of real noise, and/or characteristics due to image processing having been performed on raw image data, for example, white balancing, demosaicing, noise reduction, sharpening, color space transform, tone reproduction, and/or compression when the input image101is generated from captured raw image information, as non-limiting examples. The noise model110may include mapping information, e.g., a lookup table (LUT), between input data corresponding to the input image101and output data corresponding to the noise distribution102. The noise distribution102may represent information on a distribution of noise values of a pixel of the input image101and/or a distribution of noise values of multiple pixels of the input image101, e.g., using such a LUT, and/or respective distributions of noise values of multiple pixels of the input image101, e.g., using multiple LUTs.

The computing apparatus100may generate the enhanced image103by implementing the image enhancement model120based on the input image101and the noise distribution102, e.g., implementing the image enhancement model120provided the input image101and the noise distribution102as respective inputs or as a collective input to the image enhancement model120. For example, the image enhancement model120may be a machine learning model that is trained in advance to provide intuitive mapped relationships between input data corresponding to input images and corresponding noise distributions and output data of corresponding enhanced images, such as between input data corresponding to one or more pixels of the input image101and corresponding noise distribution(s)102and output data corresponding to the enhanced image103. The image enhancement model120may include, for example, a deep learning network. The image enhancement model120may be a deep learning based neural network. The image enhancement model120may perform image enhancement by providing such intuitive mappings between such input data and output data that are in a nonlinear relationship. For example, the image enhancement model120may be trained to perform image enhancements through deep learning, as a non-limiting example. As an example, the deep learning may include supervised and/or unsupervised learning.

FIG.2illustrates an example estimating of a noise by a noise model, according to one or more embodiments. Referring toFIG.2, a noise model200may be trained in advance with mapping relationships between various input data210and corresponding output data220, e.g., using various modeling images. The mapping relationships may be stored in the memory of the corresponding computing apparatus, e.g., in respective LUTs. As a non-limiting example, the estimating of noise by the noise model ofFIG.2may correspond to the operation of the noise model110ofFIG.1. When the input data210of an input image is given, the noise model200may estimate the output data220corresponding to the input data210. For example, the noise model200may determine the output data220corresponding to the input data210using mapping information, e.g., stored in LUT(s), between the input data210and the output data220. As non-limiting example,FIGS.6and7each demonstrate such a LUT, where various different types of information of the input data210, as well as corresponding output noise distribution information. Such various different types of information of the input data210may include pixel data211of a pixel data type, pixel position information212of a pixel position information type, and image capturing parameters213of an image capturing parameters type. The output data220may include a noise distribution221, e.g., of a noise distribution type which may include noise distribution information for multiple noise values or select noise values, as non-limiting examples. When the pixel data211, the pixel position information212, and the image capturing parameters213are given, the noise model200may estimate the noise distribution221of the input image corresponding to the pixel data211, the pixel position information212, and the image capturing parameters213.

For example, when the input image is given, the noise model200may estimate the noise distribution221of the input image based on the pixel data211, the pixel position information212, and the image capturing parameters213of the input image. The pixel data211may include pixel values of pixels of the input image, e.g., of a pixel values type, and the pixel position information212may include pixel coordinate values and/or distances from a center of the input image as a pixel coordinate information type, as a non-limiting example. In one example, the pixel data211of the input image may be data of a clean version of the input image. For example, the clean version of the input image may be a pseudo-clean image generated from the input image, as a non-limiting example. The image capturing parameters213may include multiple image capturing information of multiple image capturing information types of the input image, for example, an International Organization for Standardization (ISO) value of an ISO value type, an exposure time of an exposure time type, and Bayer pattern information of a Bayer pattern information type, as non-limiting examples. The exposure time may correspond to a shutter speed. The Bayer pattern information may include information on a filter color, e.g., red, green, blue (RGB), used by a sensor, as only one example, for capturing or otherwise obtaining pixel data, e.g., of the input image. The noise model200may estimate the noise distribution221that is similar to a noise distribution of real noise by using various sets of information including, for example, the pixel data211, the pixel position information212, and the image capturing parameters213.

FIG.3illustrates an example training of a noise model, according to one or more embodiments. Referring toFIG.3, a clean modeling image320may be generated based on a noise modeling image310. For example, the noise modeling image310may be obtained by capturing an image of a flat plate of solid color, e.g., gray, and the clean modeling image320may be an image resulting from a blur filtering, for example, being performed on the noise modeling image310. In this example, the resulting image may also be referred to as a pseudo-clean image, as a non-limiting example. The noise modeling image310and the clean modeling image320may be collectively referred to as modeling images310and320. In an example, the noise modeling image310may be a version of the captured image that is obtained by applying some post-processing schemes to raw image data of the captured image. A noise of the noise modeling image310may follow a non-normal distribution. Noise data330and a noise distribution340of the noise modeling image310may be determined dependent on a difference between the noise modeling image310and the clean modeling image320. For example, the noise data330may include a noise value of each pixel of the noise modeling image310, while the noise distribution340may include distribution information, e.g., respective probability distributions, of plural noise values for each pixel of the noise modeling image310.

Input data301may be configured based on the modeling images310and320, and output data302may be configured dependent on noise information of the noise data330and the noise distribution340. For example, the input data301may be determined by clean pixel data of the clean modeling image320, pixel position information of the modeling images310and320, and image capturing parameters of the noise modeling image310. In addition, the output data302may be determined by the noise distribution340. The clean pixel data may include pixel values of pixels of the clean modeling image320. The modeling images310and320may have the same sizes or dimensions, and the pixel position information may include pixel coordinate values and/or distances from the center of the modeling images310and320, e.g., where matching pixels between the modeling images310and320may be aligned and have the same pixel coordinate values and/or distances from the respective centers because the clean modeling image320may be generated from the noise modeling image310. The image capturing parameters may include an ISO value, an exposure time, and Bayer pattern information that are used when the noise modeling image310was captured. References to the image capturing parameters from/of the capturing of the noise modeling image310may also be considered to correspond to the image capturing parameters at the time the raw image data of the noise modeling image310was captured, e.g., such as when image processing is performed on the raw image data to generate the noise modeling image310.

Training the noise model300may be repeatedly performed using various modeling mages. As an example, each of various paired modeling images310and320may have a different noise modeling image310and a clean modeling image320corresponding to the different noise modeling image310, e.g. a pseudo-clean modeling image320generated from the different noise modeling image310. Accordingly, different noise modeling images310may be generated by capturing respective images of different objects, e.g., different flat plates with the solid color and/or a flat plate in different solid colors, as well as in various image capturing environments, e.g., various illuminances, etc., under various image capturing conditions, e.g., various ISO values, various exposure times, etc., and corresponding clean modeling images, noise components, and noise distributions may be derived for each different noise modeling image310. When respective modeling pairs of the input data301, e.g., input data210ofFIG.2, and the output data302, e.g., noise distribution221ofFIG.2, corresponding to each of the different noise modeling images310are configured or determined, the noise model300may repeatedly learn or be trained with each mapping relationship of the respective modeling pairs.

FIG.4illustrates an example change in noise distribution of an image due to image processing of the image, according to one or more embodiments. Referring toFIG.4, a processed image403may correspond to a result of image processing performed on a raw image401. An image processing block410may represent a processor of the corresponding computing apparatus configured to perform such image processing, which includes, for example, white balancing, demosaicing, noise reduction, sharpening, color space transform, tone reproduction, and/or compression, as non-limiting examples. The processed image403may be, for example, an RGB image, a YUV image, a JPG image, and a PNG image, as non-limiting examples.

In graphs402and404, the horizontal axes represent respective pixel values, e.g., respectively, a clean pixel value and a processed pixel value, and the vertical axes may represent respective distribution values, e.g., a variance or standard deviation. In each of the graphs402and404, the different shades of points may represent different pixel positions. The shades may be darker as the positions of the pixels are closer to the center of a corresponding image frame. Pixel values and pixel positions of the raw image401may be included in input data to a noise model with respect to the raw image401, and pixel values and pixel positions of the processed image403may be included in input data to the noise model with respect to the processed image403. Likewise, distribution values for the raw image401may correspond to the output data of the noise model with respect to the raw image401, and distribution values for the processed image403may correspond to the output data of the noise model with respect to the processed image403. Each of the respective input data to the noise model for the raw image401and the processed image403may further include respective image capturing parameters for the raw image401and the processed image403. The respective image capturing parameters are not illustrated in the graphs402and404for the convenience of description.

The graph402may be derived by analyzing noise of the raw image401, and the graph404may be derived by analyzing noise of the processed image403. For example, in an example case of the noise model corresponding to the noise model300ofFIG.3, the raw image401may correspond to the noise modeling image310and the processes ofFIG.3may be performed to obtain the noise data330with respect to the raw image401, e.g., the graph402may be derived by analyzing clean pixel values of a corresponding clean image of the raw image401, e.g., a pseudo-clean image of the raw image401, positions, and a distribution characteristic of noise values of the noise data for the raw image401. Further, in a similar example case of the noise model corresponding to the noise model300ofFIG.3, the processed image403may correspond to the noise modeling image310and the processes ofFIG.3may be performed to obtain the noise data330with respect to the processed image403, e.g., the graph404may be derived by analyzing clean pixel values of a corresponding clean image of the processed image403, e.g., a pseudo-clean image of the processed image403, positions, and a distribution characteristic of noise values of the noise data for the processed image403.

As illustrated in graph402, noise of the raw image401may follow a normal distribution and have a linear characteristic. Thus, an Nth-order function may be derived through a regression analysis on the noise of the raw image401. When a pixel value and position of the corresponding pixel of the raw image401are specified, one distribution value of the output data can be specified through the Nth-order function.

Rather, graph440demonstrates that noise of the processed image403may follow a non-normal distribution due to the image processing, and thus, have a nonlinear characteristic. This nonlinear characteristic may be imposed by the image processing block410on the raw image401when the processed image403is generated. In this case of the processed image403, when the pixel value and corresponding position of the input data are specified, a plurality of distribution values and/or a range of the distribution values may be specified as the output data, compared to the one distribution value that can be specified for the raw image401because noise of the raw image401follows a normal distribution and has a linear characteristic. For example, a noise model may determine the output data corresponding to the input data of the processed image403using one-to-many mapping information, e.g., a LUT, for example, instead of an one-to-one Nth-order function that would be available for determining the output of the raw image401.

FIG.5illustrates an example mapping relationship between a target input and noise distribution information, according to one or more embodiments.

Considering the above example case of the noise model corresponding to the noise model300ofFIG.3, and the processed image403ofFIG.4corresponding to the noise modeling image, the processes ofFIG.3may be performed to obtain the noise data with respect to the processed image403, as illustrated in graph500ofFIG.5. In graph500, the noise of the processed image does not follow a normal distribution. Regardless, as demonstrated below with respect to graph520, a noise distribution510may still be generated even though the noise of the processing image does not follow a normal distribution.

Noise distribution information may be derived for the processed image by analyzing clean pixel values of a corresponding clean image, e.g., a pseudo-clean image of the processed image, and resulting positions of corresponding noise values of the noise data. For example, the noise values of the noise data, e.g., for each pixel of the processed image, can be classified based on the respective clean pixel values and their respective positions. In this example, first noise distribution information may be determined by first noise values corresponding to a first clean pixel value and a corresponding first position, and second noise distribution information may be determined by second noise values corresponding to the first clean pixel value and a corresponding second position.

Accordingly, as discussed above with respect toFIG.3, a noise model may be trained with mapping relationships between input data and output data, e.g., input data301and output data302ofFIG.3, and more particularly, the noise model may be trained with mapping relationships between input data, of a target pixel value501and a target position502, and output data of the noise distribution information. Herein, references to the ‘target’ pixel value and ‘target’ position are merely references to input data, e.g., pixel value and position of the clean image version of the processed image, and corresponding output data, in the context of the further below descriptions of the mapping information610and700ofFIGS.6and7, and the available use of such mapping information in the larger context of image enhancement, such as inFIGS.1,14,16-17, and19, as non-limiting examples.

Accordingly, in this context and as illustrated in graph500, the noise of the processed image does not follow a normal distribution, and thus the noise distribution information for the processed image may be mapped as the output data to the input data of the target pixel value501and the target position502. However, even though the noise of the processed image does not follow the normal distribution, the noise of the processed image may be represented through normal distribution values by projecting select noise data, e.g., with respect to the input data of the target pixel value501and the target position502, of the processed image in graph500into a normal distribution space, resulting in the illustrated noise distribution information510in graph520. The horizontal axis of graph520may represent multiple noise values or distribution values with respect to the input data of the target pixel value501and the target position502, while the vertical axis of graph520may represent numbers or probabilities. For example, the noise distribution information510in graph520may represent a probability distribution of multiple noise values or multiple distribution values, and each noise distribution information510for each target pixel value/position may similarly be represented by a similar graph520also representing a corresponding probability distribution of multiple noise values or multiple distribution values. This is distinguished from the case of a noise of an image following a normal distribution, where a single distribution value can be derived for a particular pixel value/position.

FIG.6illustrates an example extracting of noise distribution information from mapping information, according to one or more embodiments. Referring toFIG.6, mapping information610may store a multidimensional relationship between input data and output data, such as the input data and output data discussed above with respect toFIG.5. The mapping information610may correspond to a LUT. For example, the input data may include pixel data, pixel position information, and image capturing parameters, and the output data may include noise distribution information620. Although the mapping information610is illustrated inFIG.6as storing therein a four-dimensional (4D) mapping relationship, the mapping information610may store a mapping relationship of less than or greater than four dimensions. For example, when the image capturing parameters include an ISO value, an exposure time, and Bayer pattern information, the mapping information610may store a six-dimensional (6D) mapping relationship.

A noise model may be trained with multiple mapping information610, and may determine the respective output data corresponding to the respective input data using a corresponding mapping information610. For example, when particular input data of a target pixel value601, a target position602, and a target parameter603is provided, e.g., corresponding to any one target pixel of an input image, the computing apparatus may extract the noise distribution information620corresponding to the input data from the mapping information610of the noise model, and determine a noise value of the target pixel using the corresponding noise distribution information620for the particular input data. For example, when the noise distribution information620represents a probability distribution of multiple noise values for the particular input data, such as in the noise distribution information510ofFIG.5, the computing apparatus may randomly select a noise value from among the multiple noise values of the noise distribution information620. Here, such a random selection of a noise value from among the multiple noise values of the noise distribution information620is only an example, and alternate selection or collective consideration approaches of two or more of the multiple noise values may be implemented to derive the select one noise value. Likewise, as only an example, when the noise distribution information620represents a probability distribution of multiple distribution values for the particular input data, the computing apparatus may randomly select a distribution value from among the multiple distribution values of the noise distribution information620, and determine a noise value from the selected distribution value. The computing apparatus may determine respective noise values of each of the other target pixels of the input image using the mapping information610, and determine a noise image (or a noisy image) corresponding to the input image based on a noise map of the noise values.

Based on a characteristic of the mapping information610, sets of output data respectively corresponding to sets of input data may be discrete. Thus, an unnatural boundary may be formed in such a noise map. In an example, the computing apparatus may prevent such unnatural boundaries by interpolating the sets of output data. In this example, sets of output data present adjacent to each other may be used for the interpolation, and the interpolation may include averaging. For example, when determining a noise value of input data of the target position602, the computing apparatus may determine the noise value of the input data of the target position602by interpolating the noise distribution information620of the input data of the target position602and another noise distribution information of input data of at least another target position adjacent to the target position602. Alternatively, the corresponding selected noise values may also be interpolated. For example, the computing apparatus may determine the noise value of the input data of the target position602by interpolating a noise value determined from the noise distribution information620and another noise value determined from the other noise distribution information.

FIG.7illustrates an example mapping information including normal distribution values, according to one or more embodiments. Referring toFIG.7, mapping information700may store, as output data, a representative value of the distribution values or noise values, instead of all distribution values or all noise values. As described above, the output data of the mapping information700may include noise distribution information, and the noise distribution information may store a probability distribution of multiple noise values or a probability distribution of multiple distribution values for a particular input data. Rather, when the mapping information700is generated or stored, among the probability distribution of the multiple noise values or multiple distribution values, only some of this noise distribution information may be saved in the mapping information700. For example, the saved probability distribution may include only the representative value, e.g., a maximum value702and a minimum value703, of all of the distribution values or all of the noise values. In this case, the probability distribution may not be specified with a highest accuracy, e.g., as the entire probability distribution isn't provided, but a memory space for storing the mapping information700may be reduced.

For example, a computing apparatus may extract the maximum value702and the minimum value703corresponding to a target input701from the mapping information700. The target input701may specify a target pixel value, a target position, and a target parameter for a target pixel of an input image. A graph710may represent noise data of the input image, and a block711may represent the target pixel value. Among distribution values in the block711, target distribution values corresponding to the target position may be classified, and the representative value, for example, the maximum value702and the minimum value703, may be determined to be the representative values from the target distribution values and saved in the mapping information700. Upon extracting the maximum value702and the minimum value703, the computing apparatus may randomly select a value from the range of values between the maximum value702and the minimum value703, as a non-limiting example.

FIGS.8and9illustrate examples of multi-model training, respectively according to one or more embodiments. Referring toFIG.8, input data including pixel data811, pixel position information812, and image capturing parameters, e.g., a first capturing parameter813, a second capturing parameter814, and a third capturing parameter815, may be obtained from an input image810. A first noise model820may perform noise modeling based on first partial input data corresponding to a portion of the input data, and a second noise model830may perform noise modeling based on second partial input data corresponding to at least a remaining portion of the input data. In an example, the first partial input data may include the pixel data811, the pixel position information812, and the first capturing parameter813, and the second partial input data may include the first capturing parameter813, the second capturing parameter814, and the third capturing parameter815. The first partial input data and the second partial input data may share a same portion of the input data, such as the first capturing parameter813, but may not share any portion of the pixel data811, for example, e.g., only one of the multiple models may consider the pixel data811.

Non-normal noise data801may be derived from the input image810. The input image810may correspond to a processed image, e.g., the processed image403ofFIG.4, and the non-normal noise data801may represent a noise of the input image810that follows a non-normal distribution. For example, the noise data801may include noise values of the input image810or distribution values of the noise values.

Because the noise of the input image810may have a nonlinear characteristic, it may be difficult to derive a normal distribution-based Nth-order function that can accurately represent this noise with the nonlinear characteristic. Accordingly, the first noise model820may model a normal distribution-based noise through an Nth-order function that is specifically derived through rough fitting for noise of a non-normal distribution. Thus, the first noise model820may determine normal noise data802corresponding to the first partial input data using this specifically derived Nth-order function. Accordingly, the normal noise data802may still represent the non-normal noise corresponding to the first partial input data using a normal distribution value, though this value may not be fully accurate due to the rough fitting.

The second noise model830may compensate for the error of the derived roughly fitted Nth-order function using mapping information. For example, residual data803may be generated correspond to a difference between the non-normal noise data801and the normal noise data802, and the second noise model830may be trained with mapping information between the second partial input data and the residual data803.

After the training of the first noise model820and the second noise model830has completed, and the trained first noise model820and the trained second noise model830are implemented to determine a noise value for a corresponding input data, for example, output data of the trained first noise model820corresponding to the first partial input data and output data of the trained second noise model830corresponding to the second partial input data may be combined, and an estimation of the real noise data of the corresponding input image may be derived therefrom.

Referring toFIGS.8and9, a graph901may represent distribution values of the non-normal noise of the input image810. A first noise model910may train an approximately Nth-order function through rough fitting, and generate normal noise data911corresponding to first partial input data using the Nth-order function. For example, the first noise model910may correspond to the first noise model820.

In an example, two-dimensional (2D) maps respectively corresponding to pieces of the first partial input data, e.g., the pixel data811, the pixel position information812, and the first capturing parameter813, may be determined, and the maps may be input to the first noise model910. Each map component of a 2D map corresponding to the pixel data811may have a pixel value, e.g., a clean pixel value, corresponding to a pixel position of the pixel data811. Each component of a 2D map corresponding to the pixel position information812may have a distance value from the center of the clean image, for example. In addition, all map components of a 2D map corresponding to the first capturing parameter813may have the same parameter value. For example, all map components of a 2D map based on an ISO parameter may have the same value based on ISO settings. The 2D maps may be the same size as the input image810.

The normal noise data911may represent a noise value of each pixel of the input image810or a distribution value of the noise value as a 2D map according to a pixel position of the input image810. For example, the first noise model820may learn or be trained with a mapping relationship between the 2D maps respectively corresponding to the pixel data811, the pixel position information812, and the first capturing parameter813, and a 2D map corresponding to the normal noise data802to derive a specific Nth-order function, e.g., alike the specifically derived Nth-order function of the first noise model820. In this example, rather than directly mapping values of map components of each 2D map, the first noise model820may map square values of the values of the map components.

A non-normal distribution-based noise model, e.g. the second noise model830, may require more resources than a normal distribution-based noise model, e.g., the first noise model820, for storing and processing mapping information. As the dimensions of the mapping information increases, greater memory space may be occupied. In addition, the computation amount used to calculate the noise using the mapping information may increase significantly with increases in the dimensions of the mapping information. Specifically, a computation amount used for performing interpolation may increase more significantly with increases in the dimensions of the mapping information. However, by using a multi-model, e.g., the first noise model820and the second noise model830, where each of the models only consider respective portions of the input data regarding the input image, e.g., the aforementioned first and second partial input data, the dimensions of the mapping information provided to each model may be reduced since less than all dimensions corresponding to all input data can be provided to each model, which may reduce the use of the resources for the non-normal distribution-based noise model.

FIG.10illustrates an example generating of a noise image using noise distribution information, according to one or more embodiments.

Referring toFIG.10, noise distribution information, e.g., first noise distribution information1021, second noise distribution information1022, and third noise distribution information1023, corresponding to a pixel of a clean image1010may be determined based on mapping information of a non-normal distribution-based noise model. For example, the first noise distribution information1021may be determined based on pixel data, position information, and image capturing parameters of a first pixel of the clean image1010, the second noise distribution information1022may be determined based on pixel data, position information, and image capturing parameters of a second pixel of the clean image1010, and the third noise distribution information1023may be determined based on pixel data, position information, and image capturing parameters of a third pixel of the clean image1010. The noise distribution information, e.g., each of noise distribution information1021,1022, and1023, as well as the remaining noise distribution information with respect to each remaining pixel of the clean image1010, may represent respective probability distributions of noise values corresponding to multiple or all pixels of the clean image1010or respective probability distributions of distribution values corresponding to multiple or all pixels of the clean image1010.

From the noise distribution information, e.g., each of the noise distribution information1021,1022, and1023, as well as the remaining noise distribution information with respect to each remaining pixel of the clean image1010, respective noise values for each pixel of the clean image1010may be determined. When the noise distribution information, e.g., each of the noise distribution information1021,1022, and1023, each represent the probability distribution of noise values, respective noise values may be randomly selected from each of the noise distribution information, e.g., each of the noise distribution information1021,1022, and1023. For example, a first noise value1031for the first pixel may be randomly selected from the first noise distribution information1021, a second noise value1032for the second pixel may be randomly selected from the second noise distribution information1022, and a third noise value1033for the third pixel may be randomly selected from the third noise distribution information1023. When the noise distribution information, e.g., each of the noise distribution information1021,1022, and1023, each represent the probability distribution of distribution values, a distribution value may be randomly selected from each of the noise distribution information, e.g., each of the noise distribution information1021,1022, and1023. For example, a first distribution value for the first pixel may be randomly selected from the first noise distribution information1021, a second distribution value for the second pixel may be randomly selected from the second noise distribution information1022, and a third distribution value for the third pixel may be randomly selected from the third noise distribution information1023. In addition, when the noise distribution information, e.g., each of the noise distribution information1021,1022, and1023, each include only representative values, e.g., a minimum and maximum noise or distribution values, a noise or distribution value may be randomly selected from among the respective ranges between each minimum and maximum value, to select the respective noise or distribution values.

When the noise values1031,1032, and1033are determined or generated, a noise image1040may be generated by combining or adding the noise values1031,1032, and1033with the corresponding pixels of the clean image1010. In an example, a noise map may be determined based on the noise values1031,1032, and1033, and the noise image1040may be determined by combining or concatenating the clean image1010and the noise map.

FIG.11illustrates an example generating of a training image, according to one or more embodiments. Referring toFIG.11, a degradation model1120may include a blur model1121and a noise model1122. The blur model1121may add a blur effect to a clean training image1110, and the noise model1122may add a noise effect to the clean training image1110. A degraded training image1130may be generated based on the applications of the blur effect and the noise effect to the clean training image1110. For example, the blur model1121may generate a blurred image by adding the blur effect to the clean training image1110, the noise model1122may generate the degraded training image1130by adding the noise effect to the blurred image. However, the order in which the blur effect and the noise effect are added is not limited to the foregoing example.

The degradation model1120may generate the degraded training image1130based on a degradation condition1123. Each of one or more degradation elements may be set based on the degradation condition1123. For example, the degradation condition1123may include a blur condition controlling or setting a blur effect to be applied by the blur model1121, and a noise condition associated controlling or setting a noise effect to be applied by the noise model1122. For example, the blur condition may include motion information, and the blur model1121may generate the blurred image based on the clean training image1110and the motion information. The noise condition may include pixel position information and image capturing parameters, and the noise model1122may generate the degraded training image1130based on pixel data of the blurred image, pixel position information, and image capturing parameters of the blurred clean training image, such as in the non-limiting example where the blur mode1121first generates the blurred image, and the noise model1122generates the degraded training image1130from the blurred image.

A training data set may be determined based on the clean training image1110and the degraded training image1130, and an image enhancement model may be trained based on the training data set. For the training of the image enhancement model, various training data sets may be needed. Accordingly, the degradation model1120may generate many different versions of the degraded training images1130from the same clean training image1110by changing the degradation condition1123. Further, by using a different clean training image1110, with such varied degradation conditions1123, diverse versions of additional degraded training images1130may be generated with respect to the different clean training image1110. Additionally, for example, by adjusting an exposure time to be a long exposure time a long-exposure degraded training image1130may be generated, and by adjusting the exposure time to be a short exposure time a short-exposure degraded training image1130may be generated. In an example, a degraded training image1130may be generated with enhanced noise realism, e.g., with noise characteristics that mimic real noise very well, by implementing the blur model1121, and by implementing the noise model1122according to a long-exposure or short-exposure characteristic noise condition of the degradation condition1123. For example, a long or short exposure degraded training image1130may be generated with enhanced realism by implementing the blur model1121with respect to the clean training image1110, and then implementing the noise model1122according to the long-exposure or short-exposure characteristic noise condition of the degradation condition1123, or by implementing the noise model1122with respect to the clean training image1110according to the long-exposure or short-exposure characteristic noise condition of the degradation condition1123, and then implementing the blur model1121with respect to the result of the noise model1122.

FIG.12illustrates an example generating of a degraded training image by estimating a noise distribution by a noise model, according to one or more embodiments. Referring to FIG.12, a noise model1220may estimate a noise distribution1230based on a clean training image1210and a degradation condition1221. For example, the degradation condition1211may include pixel position information and image capturing parameters. Accordingly, in an example, the noise model1220may estimate the noise distribution1230based on pixel data, pixel position information, and image capturing parameters of the clean training image1210. In an example, a blur effect may be applied to the clean training image1210, and pixel data of the blurred image may be used instead of the pixel data of the clean training image1210. A degraded training image1240may be obtained based on the clean training image1210and the noise distribution1230. For example, a noise map may be generated from the noise distribution1230through random selection, and the degraded training image1240may be generated by applying, e.g., adding, the noise map to the clean training image1210. The clean training image1210, the noise distribution1230, and the degraded training image1240may be considered a training data set.

FIG.13illustrates an example training of an image enhancement model, according to one or more embodiments. Referring toFIG.13, an image enhancement model1310may be trained based on a degraded training image1301, a noise distribution1302, and a clean training image1303. The degraded training image1301, the noise distribution1302, and the clean training image1303may correspond to a training data set, such as discussed above with respect toFIG.12. As described above, a degradation model may generate the degraded training image1301and the noise distribution1302based on the clean training image1303. The image enhancement model1310may learn or be trained to intuitively map relationships between input data corresponding to the degraded training image1301and the noise distribution1302and output data corresponding to the clean training image1303. For example, the image enhancement model1310may learn or be trained to intuitively map such relationships using many respective data training sets of diverse degraded training images1301and corresponding noise distributions1302and clean training images1303. Thus, when an input image and a noise distribution of the input image are given, the trained image enhancement model1310may have an ability to infer or intuit an enhanced image of the input image.

The image enhancement model1310may be a machine learning model. The image enhancement model1310may include, as only an example, a deep learning network, such that the image enhancement model1310may perform image enhancement by intuitive mappings, e.g., by supervised and/or unsupervised learning in advance, of input data and output data that may be in nonlinear relationships with each other. Based on the degraded training image1301and the noise distribution1302, training input data may be determined. For example, the degraded training image1301and the noise distribution1302may be concatenated, and the training input data to the image enhancement model1310may thereby be determined. For example, when the image enhancement model1310is the machine learning model, a collective input to the machine learning model may be the concatenated degraded training image1301and noise distribution1302. The clean training image1303may be determined to be training output data. The image enhancement model1310may be repeatedly trained to generate the training output data based on the training input data. The image enhancement model1310may be repeatedly trained so as to generate each of many different training output, respectively based on corresponding training input data, e.g., until implementation of the resulting in-training image enhancement model has met a predetermined high accuracy threshold and/or meets a predetermined low inaccuracy threshold, for example.

FIG.14illustrates an example enhancing of an image using a noise model and an image enhancement model. Referring toFIG.14, a noise model1410may estimate a noise distribution1405of an input image1401based on the input image1401. For example, the noise model1410may estimate the noise distribution1405based on pixel data1402, pixel position information1403, and image capturing parameters1404of the input image1401.

The pixel data1402may include pixel values that are based on pixel values of the input image1401, and the pixel position information1403may include respective pixel coordinate values and/or distances from a center, e.g., corresponding to a center of an image including the pixel value included in the pixel data1402. The center may be a predetermined center dependent on the size or dimensions of an image, for example. A pseudo-clean image may be generated by applying blur filtering to the input image1401, and the pixel data1402may be clean pixel data1402determined from pixel values of the pseudo-clean image. The image capturing parameters1404may include image capturing information of the input image1401, e.g., sensor information or other information related to the capturing of the input image1401. The noise model1410may estimate the noise distribution1405based on a distance from the center of the input image1401to each pixel of the input image1401.

An image enhancement model1420may generate an enhanced image1406based on the input image1401and the noise distribution1405. For example, input data may be generated by combining or concatenating the input image1401and the noise distribution1405, and by implementation of the image enhancement model1420provided the generated input data, the image enhancement model1420may generate or output the enhanced image1406. The image enhancement model1420may be a machine learning model that learns or is trained in advance to intuitively map relationships between input data corresponding to the input image1401and the noise distribution1405and output data corresponding to the enhanced image1406, as well as to intuitively map respective relationships between many different input data corresponding to different input images and corresponding noise distributions and output enhanced images. In addition, the image enhancement model1420may be trained in advance based on training images generated through the noise model1410. Still further, because of this training of the image enhancement model1420, the image enhancement model1420may generate an enhanced image1406for an input image1401that the image enhancement model1420was not trained for.

FIG.15illustrates an example computing apparatus, according to one or more embodiments. Referring toFIG.15, a computing apparatus1500may include a processor1510and a memory1520. The memory1520may be connected to the processor1510, and store therein instructions executable by the processor1510, data to be processed by the processor1510, and/or data processed by the processor1510. For example, the memory1520may store a noise model1521, a training data set1522, and an image enhancement model1523. The memory1520may include a non-transitory computer-readable medium, for example, a high-speed random-access memory (RAM) and/or a nonvolatile computer-readable medium, e.g., one or more disk storage devices, flash memory devices, or other nonvolatile solid-state memory devices.

The processor1510may execute instructions to perform any one, any combination, or all operations described herein with respect toFIGS.1through14and16through19, e.g., where execution of the instructions by the processor1510configures the processor1510to perform any one, any combination, or all operations described herein with respect toFIGS.1through14and16through19. For example, the processor1510may train the noise model1521based on respective modeling images having noises that follows non-normal distributions and corresponding pixel position information, respectively generate different degraded training images of corresponding clean training images using the noise model1521, determine multiple training data sets1522based on the respectively generated different degraded training images and the corresponding clean training images, and train the image enhancement model1523based on the multiple training data sets1522. In addition, the computing apparatus1500may be configured to perform one or more or all operations described herein with respect toFIGS.1through14andFIGS.16through19. For example, in addition to training, the computing apparatus1500may further use the trained noise model and trained image enhancement model, to perform image enhancement of an input image, as described in any one, any combination, or all embodiments herein.

FIG.16illustrates an example computing apparatus, according to one or more embodiments. Referring toFIG.16, a computing apparatus1600may include a processor1610and a memory1620. The memory1620may be connected to the processor1610, and may store therein instructions executable by the processor1610, data to be processed by the processor1610, and/or data processed by the processor1610. For example, the memory1620may store a noise model1621and an image enhancement model1623. The processor1610may execute instructions to perform any one, any combination, or all operations described herein with respect toFIGS.1through15and17through19, e.g., where execution of the instructions by the processor1610configures the processor1610to perform any one, any combination, or all operations described herein with respect toFIGS.1through15and17through19. For example, the processor1610may receive or obtain an input image. For the input image that has a noise that follows a non-normal distribution, the processor may determine a non-normal noise map corresponding to the noise of the input image using the noise model1621, and generate an enhanced image by implementing the image enhancement model1623based or dependent on the input image and the non-normal noise map. In addition, the computing apparatus1600may be configured to perform one or more or all operations described herein with respect toFIGS.1through15and17through19. For example, in addition to performing image enhancement of an input image, the computing apparatus1600may further perform training to generate the noise model and/or training to generate the image enhancement model, as described in any one, any combination, or all embodiments herein.

FIG.17illustrates an example image enhancement method, according to one or more embodiments. Referring toFIG.17, a computing apparatus may receive or obtain an input image having a noise that follows a non-normal distribution in operation1710, determine a non-normal noise map corresponding to the noise of the input image using a noise model in operation1720, and generate an enhanced image by implementing an image enhancement model based or dependent on the input image and the non-normal noise map in operation1730.

Operation1720may include extracting non-normal noise distribution information of each pixel of the input image from mapping information of the noise model based on clean pixel data, pixel position information, and image capturing parameters of the input image, determining a noise value of each pixel of the input image using the non-normal noise distribution information, and determining the non-normal noise map based on the noise value of each pixel of the input image.

The mapping information may be a LUT that maps input data corresponding to the pixel data, the pixel position information, and the image capturing parameters, and output data corresponding to the non-normal noise distribution information. The non-normal noise distribution information may include first non-normal noise distribution information of a first pixel of the input image, and the first non-normal noise distribution information may represent a first non-normal distribution mapped to a clean pixel value of the first pixel, position information of the first pixel, and the image capturing parameters. A first noise value of the first pixel may be randomly determined based on the first non-normal distribution.

The non-normal noise distribution information may include at least one of first distribution information representing a distribution of non-normal noise data and second distribution information representing a distribution of normal distribution values of the non-normal noise data obtained by projecting the non-normal noise data into a normal distribution space. The second distribution information may include entire data of the normal distribution values. Alternatively, as a non-limiting example, the second distribution information may include a maximum value and a minimum value of the normal distribution values, instead of the entire data of the normal distribution values.

Operation1720may include determining normal noise data corresponding to first partial input data of the input image using a normal distribution-based first noise model, determining non-normal noise data corresponding to second partial input data of the input image using a non-normal distribution-based second noise model, and determining the non-normal noise map based on the normal noise data and the non-normal noise data.

The clean pixel data of the input image may be obtained through blur filtering performed on the input image. The image capturing parameters may include at least one of an ISO value, an exposure time, and Bayer pattern information. The noise of the input image may follow the non-normal distribution, e.g., due to image processing having been performed on raw image data to generate the input image. The image enhancement model may be a machine learning model that is trained in advance to intuitively map relationships between input data corresponding to the input image and the non-normal noise map and output data corresponding to the enhanced image. The image enhancement model may be trained in advance based on many training images generated using the noise model.

In addition, the image enhancement method ofFIG.17may further include operations described above with respect toFIGS.1through16and operations described hereinafter with respect toFIGS.18and19.

FIG.18illustrates an example training method, according to one or more embodiments. Referring toFIG.18, in operation1810, a computing apparatus may train a noise model based on modeling images having respective noises that each follow non-normal distributions and based on pixel position information. Operation1810may include generating a first clean modeling image from a first noise modeling image having a non-normal distribution, determining non-normal noise data of the first noise modeling image based on a difference between the first noise modeling image and the first clean modeling image, and determining mapping information of the noise model based on a mapping relationship that is based on distribution information of the non-normal noise data, pixel data of the first clean modeling image, and the pixel position information.

The distribution information may include at least one of first distribution information representing a distribution of the non-normal noise data, and second distribution information representing a distribution of normal distribution values of the non-normal noise data obtained by projecting the non-normal noise data into a normal distribution space. The second distribution information may include entire data of the normal distribution values. Alternatively, as a non-limiting example, the second distribution information may include a maximum value and a minimum value of the normal distribution values, instead of the entire data of the normal distribution values.

The noise model may include a normal distribution-based first noise model and a non-normal distribution-based second noise model. Operation1810may include determining normal noise data corresponding to first partial input data of the first noise modeling image that follows the non-normal distribution using the first noise model, determining residual data based on a difference between the normal noise data and non-normal noise data of the first noise modeling image, and determining mapping information of the second noise model based on a mapping relationship that is based on second partial input data of the first noise modeling image and the residual data.

In operation1820, the computing apparatus may generate a degraded training image of a clean training image using the noise model. In operation1830, the computing apparatus may determine a training data set based on the clean training image and the degraded training image. In operation1840, the computing apparatus may train an image enhancement model based on the training data set, e.g., based on the training data set and many other such training data sets.

In addition, the learning method ofFIG.18may further include operations described above with respect toFIGS.1through17and operations described below with respect toFIG.19.

FIG.19illustrates an example electronic device, according to one or more embodiments. Referring toFIG.19, an electronic device1900may include a processor1910, a memory1920, a camera1930, a storage device1940, an input device1950, an output device1960, and a network interface1970. These components may communicate with one another through a communication bus1980. Each of the processor1910, the memory1920, the camera1930, the storage device1940, the input device1950, the output device1960, the network interface1970, and the communication bus1980may be singular or plural. The electronic device1900may be, or embodied as at least a portion of, a mobile device, e.g., a mobile phone, a smartphone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, etc., a wearable device, e.g., a smartwatch, a smart band, smart eyeglasses, etc., a computer device, e.g., a desktop, a server, etc., a home appliance, e.g., a television (TV), a smart TV, a refrigerator, etc., a security device, e.g., a door lock, etc., or a vehicle, e.g., an autonomous vehicle, a smart vehicle, etc., as non-limiting examples.

The electronic device1900may structurally and/or functionally include the computing apparatus100ofFIG.1, the computing apparatus1500ofFIG.15, and/or the computing apparatus1600ofFIG.16. In addition, each of the computing apparatus100ofFIG.1, the computing apparatus1500ofFIG.15, and the computing apparatus1600ofFIG.16may each be such electronic devices, as non-limiting examples. For example, the processor1910may also be representative of the processor of computing apparatus100, and the memory1920may also be representative of the memory of computing apparatus100, the processor1510of the computing apparatus1500may correspond to the processor1910or the computing apparatus1500may further include the processor1910, and the memory1520of the computing apparatus1500may correspond to the memory1920or the computing apparatus1500may further include the processor1910, the processor1610of the computing apparatus1600may correspond to the processor1910or the computing apparatus1600may further include the processor1910, and the memory1620of the computing apparatus1600may correspond to the memory1920or the computing apparatus1600may further include the processor1910. Still further, the camera1930, the storage device1940, the input device1950, the output device1960, the network interface1970, and the communication bus1980are also respectively representative of any one or any combination of two or more of a camera, a storage device, an input device, an output device, a network interface, and a communication bus of the computing apparatus100, the computing apparatus1500, and/or the computing apparatus1600.

The processor1910may execute instructions and/or functions to be executed in the electronic device1900. For example, the processor1910may execute instructions stored in the memory1920or the storage device1940. The processor1910may perform any one or more, or all, of the operations described above with respect toFIGS.1through18. For example, the memory1920may store instructions, which when executed by the processor1910, configure the processor1910to perform any one or more, or all, of the operations described above with respect toFIGS.1through18. The memory1920may store related information of such operations as well as other software and/or applications that may be executed by the electronic device1900for additional functionalities of the electronic device1900. The memory1920is a computer-readable storage medium or a computer-readable storage device.

The camera1930may capture an image and/or a video. The image and/or the video may correspond to a modeling image, a training image, and an input image, as described above with respect toFIGS.1-18. The storage device1940may include a computer-readable storage medium or a computer-readable storage device. The storage device1940may store a greater amount of information than the memory1920and store the information for a long period of time. The storage device1940may include, for example, a magnetic hard disk, an optical disc, a flash memory, a floppy disk, or any other type of nonvolatile memory, as non-limiting examples. The memory1920and/or the storage device1940may store one or more of each of any or all of the noise models described herein, one or more of each of any or all of the blur models described herein, one or more of each of any or all of the image enhancement models described herein, one or more of each of any or all of the degradation models described herein, may store any or all of respective in-training noise models and/or in-training image enhancement models described herein, as well as pairs of modeling images, training data sets, and any other training data described herein, and may store one or more of each of any or all of the mapping information described herein, such as mapping information610ofFIG.6or mapping information700ofFIG.7, as non-limiting examples.

The input device1950may receive an input from a user through a traditional input method using a keyboard and a mouse, or a newer input method using, for example, a touch input, a voice input, and an image input, as non-limiting examples. The input device1950may include, for example, a keyboard, a mouse, a touchscreen, a microphone, or any other device that detects an input from a user and transmits the detected input to the electronic device1900, for example. The output device1960may provide an output of the electronic device1900to a user through a visual, auditory, or tactile channel, e.g., a visual, auditory, or tactile channel of the electronic apparatus1900. The output device1960may include, for example, a display, a touchscreen, a speaker, a vibration generating device, or any other device that provides an output of the electronic device1900to a user. The output device1960may also output information to an external display, touchscreen, speaker, vibration generating device, as non-limiting examples. The network interface1970may communicate with an external device through a wired or wireless network. In addition, the electronic device1900may include any one, any combination, or all hardware described herein with respect toFIGS.1through18that may be configured to perform any one, any combination, or all operations described herein with respect toFIGS.1through18.