Patent ID: 12230008

DETAILED DESCRIPTION

Throughout the disclosure, the expression “at least one of a, b, or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

Terms used herein will be briefly described and then the disclosure will be described in detail.

The terms used herein are those general terms currently widely used in consideration of functions in the disclosure, but the terms may vary according to the intentions of those of ordinary skill in the art, precedents, or new technology in the art. Also, in some cases, there may be terms that are optionally selected by the applicant, and the meanings thereof will be described in detail in the corresponding portions of the disclosure. Thus, the terms used herein should be understood not as simple names but based on the meanings of the terms and the overall description of the disclosure.

Throughout the specification, when something is referred to as “including” an element, another element may be further included unless specified otherwise. Also, as used herein, the terms such as “units” and “modules” may refer to units that perform at least one function or operation, and the units may be implemented as hardware or software or a combination of hardware and software.

Hereinafter, embodiments of the disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the embodiments. However, the disclosure may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Also, portions irrelevant to the description of the disclosure will be omitted in the drawings for a clear description of the disclosure, and like reference numerals will denote like elements throughout the specification.

FIG.1is a diagram illustrating an operation in which an image processing apparatus processes an image by using an image processing network, according to an embodiment of the disclosure.

Referring toFIG.1, an image processing network103according to an embodiment of the disclosure may receive a first image101and process the first image101to generate a second image102. In this case, the first image101may be an image including noise or may be a low-resolution image. An image processing apparatus100may use the image processing network103to perform denoising for removing noise while maintaining a fine edge and texture of the first image101, thereby generating the second image102. The second image102may be a higher-resolution image than the first image101and may be an image with an improved quality compared to the first image101.

An image processing performed by the image processing network103according to an embodiment of the disclosure will be described below in detail with reference to the drawings.

FIG.2is a diagram illustrating an image processing network according to an embodiment of the disclosure.

Referring toFIG.2, the image processing network103according to an embodiment of the disclosure may include a characteristic information generator210, an attention map generator220, a spatially variant kernel generator230, and a filter240.

According to an embodiment of the disclosure, the image processing network103may include a structure for receiving the first image101and outputting the second image102.

According to an embodiment of the disclosure, the characteristic information generator210may obtain characteristic information of the first image101. For example, the characteristic information of the first image101may include similarity information between each of the pixels included in the first image101and a neighboring pixel. In this case, the similarity information may be information representing the difference between a pixel value of each of the pixels and a pixel value of a neighboring pixel located near each of the pixels. A method of generating the similarity information will be described in detail with reference toFIGS.3to4B.

Also, the characteristic information of the first image101may include frequency characteristic information of the first image101. A method of generating the frequency characteristic information of the first image101will be described in detail with reference toFIG.5.

Also, the characteristic information of the first image101may include domain transformation information of the first image101, gradient characteristic information of the first image101, region characteristic information of the first image101, or the like, but embodiments are not limited thereto.

FIG.3is a diagram referenced to describe a method of generating similarity information, according to an embodiment of the disclosure.

According to an embodiment of the disclosure, the characteristic information generator210may generate similarity information by calculating the difference between each of the pixels included in the first image101and a neighboring pixel. For convenience of description, in embodiments of the disclosure, it will be assumed that the width of the first image101is W, the height thereof is H, and the number of channels is 1.

Referring toFIG.3, the characteristic information generator210may calculate a difference value between a first pixel310and each of K2pixels included in a first region301centered on the first pixel310among a plurality of pixels included in the first image101. In this case, K×K that is the size of the first region301may be determined based on the size of a spatial kernel described below.

The characteristic information generator210may obtain K2difference values with respect to the first pixel310by calculating a difference value between the first pixel310and each of K2pixels included in the first region301. For example, as illustrated inFIG.3, the characteristic information generator210may calculate a difference value between the first pixel310and a first neighboring pixel311, a difference value between the first pixel310and a second neighboring pixel, a difference value between the first pixel310and a third neighboring pixel313, and a difference value between the first pixel310and a fourth neighboring pixel314. In the same way, the characteristic information generator210may obtain K2difference values with respect to each of other pixels included in the first image101, other than the first pixel310. For example, the characteristic information generator210may obtain K2difference values from neighboring pixels by using each of other pixels other than the first pixel310as a center pixel.

The characteristic information generator210may arrange K2difference values for each of the pixels in similarity information350in the channel direction of the corresponding pixel, and accordingly, the size of the similarity information350may be W×H and the number of channels may be K2.

According to an embodiment of the disclosure, a first channel image of the similarity information350may represent a difference value between each of the pixels included in the first image101and a neighboring pixel having a first relative position with respect to each of the pixels (e.g., a pixel at a position shifted by (K−1)/2 pixels to the left side and by (K−1)/2 pixels to the upper side with respect to each of the pixels). Also, a second channel image of the similarity information350may represent a difference value between each of the pixels included in the first image101and a neighboring pixel having a second relative position with respect to each of the pixels (e.g., a pixel at a position shifted by (K−1)/2−1 pixels to the left side and by (K−1)/2 pixels to the upper side with respect to each of the pixels). However, the disclosure is not limited thereto.

FIGS.4A and4Bare diagrams referenced to describe methods of obtaining similarity information, according to another embodiment of the disclosure.

Referring toFIG.4A, the characteristic information generator210according to an embodiment of the disclosure may obtain K2images by horizontally shifting each of the pixels included in the first image101by p pixels (−(K−1)/2≤p≤(K−1)/K2images, where p is an integer) and vertically shifting each of the pixels by q pixels (−(K−1)/2≤q≤(K−1)/2, where q is an integer). In this case, each of the K2images410may have the same size (W×H) as the first image101.

The characteristic information generator210may obtain similarity information350by calculating a difference image between each of the K2images410and the first image101. Accordingly, as described with reference toFIG.3, the size of the similarity information350may be W×H and the number of channels may be K2.

Referring toFIG.4B, the characteristic information generator210according to an embodiment of the disclosure may obtain similarity information by performing mask processing on the first image101.

The mask processing may be performed through a convolution operation between the first image101and each of mask filters M1, M2, M3, . . . , Mn. In this case, “n” may be K2−1, and K2−1 channel images421,422,423, . . . ,429included in the similarity information350may be generated through the mask processing based on K2−1 mask filters. For example, the characteristic information generator210may generate a first channel image421of the similarity information350through a convolution operation between the first image101and a first mask filter M1and generate a second channel image422of the similarity information350through a convolution operation between the first image101and a second mask filter M2. Also, the characteristic information generator210may generate a third channel image423of the similarity information350through a convolution operation between the first image101and a third mask filter M3and generate a (K−1)th channel image429of the similarity information350through a convolution operation between the first image101and an n-th mask filter Mn.

Referring toFIG.4B, the characteristic information generator210may calculate a pixel value included in the similarity information350by respectively multiplying and summing K×K pixel values included in a first region401of the first image101and K×K pixel values (parameter values) included in each of the mask filters.

In this case, the parameter values included in the mask filter may be determined according to the position of a neighboring pixel for calculating the similarity information350. For example, the first mask filter M1may be a mask filter for calculating similarity information between a center pixel and a neighboring pixel having a first relative position with respect to the center pixel (e.g., a position shifted by (K−1)/2 pixels to the left side and by (K−1)/2 pixels to the upper side with respect to a reference pixel). Accordingly, in the first mask filter M1, a center pixel value may be ‘1’, a pixel value having a first relative position with respect to the center pixel may be ‘−1’, and the other pixel values may be ‘0’.

According to an embodiment of the disclosure, the characteristic information generator210may calculate a value of a second pixel431included in the first channel image421of the similarity information350by performing a convolution operation between the first mask filter M1and the first region401centered on a first pixel415. In this case, the position of the first pixel415in the first image101may be equal to the position of the second pixel431in the first channel image421of the similarity information350. The value of the second pixel431included in the first channel image421may be obtained by subtracting the value of a pixel411having a first relative position with respect to the first pixel415from the value of the first pixel415.

According to the convolution operation based on the first mask filter M1, as the first mask filter M1slides in the horizontal direction and the vertical direction, each of the pixels included in the first image101may be located at the center of the first mask filter M1. In this case, a reference pixel may be located at the center of a region covered by the first mask filter M1sliding and shifting over the first image101. The characteristic information generator210may calculate pixel values included in the first channel image421by performing a convolution operation between a changed region and the first mask filter M1.

Also, the second mask filter M2may be a mask filter for calculating similarity information between a center pixel and a neighboring pixel having a second relative position with respect to the center pixel (e.g., a position shifted by (K−1)/2−1 pixels to the left side and by (K−1)/2 pixels to the upper side with respect to a reference pixel). Accordingly, in the second mask filter M2, a center pixel value may be ‘1’, a pixel value having a second relative position with respect to the center pixel may be ‘−1’, and the other pixel values may be ‘0’.

The characteristic information generator210may calculate a value of a third pixel432included in the second channel image422of the similarity information350by performing a convolution operation between the second mask filter M2and the first region401centered on the first pixel415. In this case, the position of the first pixel415in the first image101may be equal to the position of the third pixel432in the second channel image422. Accordingly, the value of the third pixel432included in the second channel image422may be obtained by subtracting the value of a pixel412having a second relative position with respect to the first pixel415from the value of the first pixel415.

In the same way, a region to be subjected to a convolution operation may be changed such that each of the pixels included in the first image101may be located at the center of the region to be subjected to the convolution operation, and pixel values included in the second channel image422may be calculated by performing a convolution operation between the second mask filter M2and the changed region.

Also, the third mask filter M3may be a mask filter for calculating similarity information between a center pixel and a neighboring pixel having a third relative position with respect to the center pixel, and the n-th mask filter Mn may be a mask filter for calculating similarity information between a center pixel and a neighboring pixel having an n-th relative position with respect to the center pixel.

As illustrated and described inFIG.4B, by performing mask processing by using K2−1 mask filters, the characteristic information generator210according to an embodiment of the disclosure may obtain similarity information including difference values between each of the pixels included in the first image101and neighboring pixels having first to (K2−1)th relative positions with respect to each of the pixels. For example, the characteristic information generator210may generate the first to (K2−1)th channel images421,422,423, . . . ,429of the similarity information350by using the first to (K2−1)th mask filters M1, M2, M3, . . . , Mn.

Also, the similarity information350according to an embodiment of the disclosure may include a K2-th channel image representing similarity information with respect to itself with respect to each of the pixels included in the first image101. Thus, all pixel values of the K2-th channel image may be ‘0’.

Moreover, the methods of obtaining the similarity information350illustrated and described inFIGS.3,4A, and4Bare merely examples, and the characteristic information generator210may use various methods to obtain similarity information between each of the pixels included in the first image101and a neighboring pixel.

FIG.5is a diagram for describing a method of generating frequency characteristic information of a first image, according to an embodiment of the disclosure.

According to an embodiment of the disclosure, the characteristic information generator210may generate frequency characteristic information of the first image101. For example, the characteristic information generator210may obtain at least one piece of characteristic information by performing filtering on the first image101.

For example, the characteristic information generator210may obtain frequency characteristic information by performing filtering based on a Gaussian kernel or a discrete cosine transform (DCT) or a wavelet transform or the like.

Referring toFIG.5, the characteristic information generator210may perform a convolution operation by applying four filter kernels511,512,513, and514to the first image101. The characteristic information generator210may obtain first filtering information521by performing a convolution operation between the first image101and a first filter kernel511, may obtain second filtering information522by performing a convolution operation between the first image101and a second filter kernel512, may obtain third filtering information523by performing a convolution operation between the first image101and a third filter kernel513, and may obtain fourth filtering information524by performing a convolution operation between the first image101and a fourth filter kernel514. InFIG.5, four filter kernels are illustrated and described; however, the disclosure is not limited thereto. Also, the filter kernels may be Gaussian kernels but are not limited thereto.

According to an embodiment of the disclosure, weight values of the filter kernels511,512,513, and514for filtering the first image101may be preset values.

The characteristic information generator210may obtain first frequency characteristic information531by calculating the difference between the first filtering information521and the second filtering information522, may obtain second frequency characteristic information532by calculating the difference between the second filtering information522and the third filtering information523, and may obtain third frequency characteristic information533by calculating the difference between the third filtering information523and the fourth filtering information524.

In this case, the first frequency characteristic information531may be high frequency characteristic information, the second frequency characteristic information532may be middle frequency characteristic information, and the third frequency characteristic information533may be low frequency characteristic information.

Also, the characteristic information generator210according to an embodiment of the disclosure may extract at least one piece of sub-feature information from the first to third frequency characteristic information531,532, and533. For example, the characteristic information generator210may extract first sub-frequency feature information551through a convolution operation between the first frequency characteristic information531and a first sub-kernel541and may extract second sub-frequency feature information552through a convolution operation between the second frequency characteristic information532and a second sub-kernel542. Also, the characteristic information generator210may extract third sub-frequency characteristic information553through a convolution operation between the third frequency characteristic information533and a third sub-kernel543.

According to an embodiment of the disclosure, the characteristic information of the first image101may include a plurality of pieces of frequency characteristic information (e.g., the first to third frequency characteristic information531,532, and533) or a plurality of pieces of sub-frequency characteristic information (e.g., the first to third sub-frequency characteristic information551,552, and553).

Also, the characteristic information generator210may generate gradient characteristic information of the first image101. Gradient features according to an embodiment of the disclosure may be determined based on the feature of an edge appearing in each of the pixels included in the first image and may include at least one of strength feature, angle feature, or coherence. For example, the strength feature may be determined such that the strength may increase as the edge becomes sharper. The angle feature may represent the direction of the edge. The coherence may represent a measure of how directional the edge is. The coherence may be higher when the edge is straight than when the edge is curved.

According to an embodiment of the disclosure, the characteristic information generator210may determine the gradient feature of the first image based on the eigen values and eigen vectors calculated through the eigen component analysis for the gradient value of each of the pixels included in the first image.

According to an embodiment of the disclosure, the characteristic information of the first image may include a plurality of pieces of gradient characteristic information of the first image.

Referring back toFIG.2, the attention map generator220may generate an attention map based on the characteristic information. Hereinafter, for convenience of description, a case where the characteristic information of the first image is similarity information will be described as an example.

According to an embodiment of the disclosure, the attention map generator220may generate an attention map based on the similarity information. When the attention map is generated based on the similarity information, the image quality of an image-processed image may be improved by using the attention map generated to give a great weight to neighboring pixels having similar pixel values, for image processing.

The attention map generator220may generate an attention map representing weight information corresponding to each of the pixels based on the similarity information between each of the pixels included in the first image and neighboring pixels. A method of generating the attention map will be described in detail with reference toFIG.6.

FIG.6is a diagram referenced to describe a method of generating an attention map, according to an embodiment of the disclosure.

Referring toFIG.6, the attention map generator220may generate an attention map650by using a convolutional neural network610. The convolutional neural network610according to an embodiment of the disclosure may include one or more layers and may receive the characteristic information obtained by the characteristic information generator210according to an embodiment of the disclosure. In this case, the characteristic information may be divided into a plurality of groups, and the plurality of divided groups may be respectively input into different layers.

InFIG.6, the characteristic information will be described as similarity information. As illustrated inFIG.6, the similarity information350having K2channels may be divided into a first group, a second group, a third group, and a fourth group in units of channels and then may be input into different layers included in the convolutional neural network610. For example the first group may be input as first input information621(illustrated as Input1), the second group may be input as second input information622(illustrated as Input2), the third group may be input as third input information623(illustrated as Input3), and the fourth group may be input as fourth input information624(illustrated as Input4). Although the description has been given with respect to the similarity information350inFIG.6, even when the characteristic information is frequency characteristic information, gradient characteristic information, or the like, the characteristic information may be divided into a plurality of groups and then may be input into different layers included in the convolutional neural network610.

Moreover, a method of dividing the similarity information350into a plurality of groups will be described in detail with reference toFIG.7.

Also, referring toFIG.6, the attention map generator220according to an embodiment of the disclosure may input a plurality of pieces of input information621,622,623, and624in different layers included in the convolutional neural network610. For example, first input information621, second input information622, third input information623, and fourth input information624may be input into different layers. The attention map generator220may perform an operation on the plurality of pieces of input information621,622,623, and624by using the convolutional neural network610. Each of the layers included in the convolutional neural network610may have a structure for receiving the values output from the previous layer, performing an operation in the corresponding layer to obtain result values, and outputting the obtained result values to the next layer.

Also, referring toFIG.6, the attention map generator220may obtain a plurality of pieces of output information631,632,633, and634corresponding to the plurality of pieces of input information621,622,623, and624from different layers included in the convolutional neural network610. For example, first output information631(illustrated as Output1), second output information632(illustrated as Output2), third output information633(illustrated as Output3), and fourth output information634(illustrated as Output4) may be output from different layers.

The attention map generator220may generate the attention map650based on a plurality of pieces of output information. A method of generating the attention map650based on the plurality of pieces of output information will be described below in detail with reference toFIG.9. The size of the attention map650generated by the attention map generator220may be W×H, and the number of channels may be K2.

FIG.7is a diagram for describing a method of dividing similarity information into a plurality of groups, according to an embodiment of the disclosure.

Referring toFIG.7, the size of the similarity information350according to an embodiment of the disclosure may be W×H, and the number of channels may be K2. Also, the similarity information350may be divided into a plurality of groups according to the channel-wise characteristic value included in the similarity information350. For example, as illustrated inFIG.7, each of the channels included in the similarity information350may have an intensity value. In this case, the channel-wise intensity may be obtained by summing all pixel values included in one channel; however, the disclosure is not limited thereto.

The attention map generator220may divide the similarity information350into a plurality of groups in units of channels according to the channel-wise intensity. For example, the attention map generator220may divide the similarity information350into a first group, a second group, a third group, and a fourth group according to the channel-wise intensity level.

Referring toFIG.7, the channels included in the similarity information350according to an embodiment of the disclosure may include channel numbers (indexes) in order, and for example, a channel number ‘0’ may be assigned to the first channel included in the similarity information350and a channel number K2−1′ may be assigned to the last channel. In a two-dimensional image710in which the channels included in the similarity information350are arranged from left to right and from top to bottom in the order of channel numbers, the channels located in a first region711may be classified as a first group, the channels located in a second region712may be classified as a second group, the channels located in a third region713may be classified as a third group, and the channels located in a fourth region714may be classified as a fourth group. In embodiments, as discussed in greater detail below, the first group may correspond to first input information721, the second group may correspond to second input information722, the third group may correspond to third input information723, and the fourth group may correspond to fourth input information724.

When the similarity information350includes 169 channels, the first group may include 48 channels, the second group may include 40 channels, the third group may include 40 channels, and the fourth group may include 41 channels.

InFIG.7, an example in which the similarity information350is divided into a plurality of groups according to the channel-wise intensity level has been illustrated and described; however, the disclosure is not limited thereto and the similarity information350may be divided into a plurality of groups according to different characteristic values for each channel.

FIG.8is a diagram for describing a structure of a convolutional neural network according to an embodiment of the disclosure.

Referring toFIG.8, a convolutional neural network801according to an embodiment of the disclosure may include one or more convolution layers, one or more activation layers, one or more concatenation layers, and one or more split layers. Also, the convolutional neural network801may further include an element-wise summation layer. In embodiments, the convolutional neural network801may correspond to convolutional neural network610described above.

InFIG.8, for convenience of description, the convolutional neural network610according to an embodiment of the disclosure will be described as including 12 convolution layers.

Referring toFIG.8, the first input information721may be input into a first convolution layer811. The first input information721may include 48 channels, and in the first convolution layer811, first feature information may be obtained by performing a convolution operation between the first input information721and a first kernel included in the first convolution layer811. The first feature information may include 16 channels and may be input into a first activation layer831. In the first activation layer831, an operation of applying an activation function to the first feature information input into the first activation layer831may be performed. The value output from the first activation layer831may be input into a first concatenation layer851, and the value output from the first activation layer831may include 16 channels.

Also, the second input information722may also be input into the first concatenation layer851. The first concatenation layer851may concatenate the values input into the first concatenation layer851(which may be the value output from the first activation layer831and the second input information722) in the channel direction and output the same to a second convolution layer812. Accordingly, the value input into the second convolution layer812may include 56 (=40+16) channels.

In the second convolution layer812, second feature information may be obtained by performing a convolution operation between the value input into the second convolution layer812and a second kernel included in the second convolution layer812. The second feature information may include 20 channels and may be input into a second activation layer832. In the second activation layer832, an operation of applying an activation function to the second feature information input into the second activation layer832may be performed. The value output from the second activation layer832may be input into a second concatenation layer852, and the value output from the second activation layer832may include 20 channels.

Also, the third input information723may also be input into the second concatenation layer852. The second concatenation layer852may concatenate the values input into the second concatenation layer852(which may be the value output from the second activation layer832and the third input information723) in the channel direction and output the same to a third convolution layer813. Accordingly, the value input into the third convolution layer813may include 60 (=40+20) channels.

In the third convolution layer813, third feature information may be obtained by performing a convolution operation between the value input into the third convolution layer813and a third kernel included in the third convolution layer813. The third feature information may include 20 channels and may be input into a third activation layer833. In the third activation layer833, an operation of applying an activation function to the third feature information input into the third activation layer833may be performed. The value output from the third activation layer833may be input into a third concatenation layer853, and the value output from the third activation layer833may include 20 channels.

Also, the fourth input information724may also be input into the third concatenation layer853. The third concatenation layer853may concatenate the values input into the third concatenation layer853(which may be the value output from the third activation layer833and the fourth input information724) in the channel direction and output the same to a fourth convolution layer814. Accordingly, the value input into the fourth convolution layer814may include 61 (=41+20) channels. In the fourth convolution layer814, fourth feature information may be obtained by performing a convolution operation between the value input into the fourth convolution layer814and a fourth kernel included in the fourth convolution layer814. The fourth feature information may include 24 channels and may be input into a fourth activation layer834. In the fourth activation layer834, an operation of applying an activation function to the fourth feature information input into the fourth activation layer834may be performed. The value output from the fourth activation layer834may be input into a fifth convolution layer815.

In the fifth convolution layer815, fifth feature information may be obtained by performing a convolution operation between the value input into the fifth convolution layer815and a fifth kernel included in the fifth convolution layer815. The fifth feature information may include 24 channels and may be input into an element-wise summation layer860.

Also, the value output from the fourth activation layer834may also be input into the element-wise summation layer860. The element-wise summation layer860may perform an operation of element-wise-summing the fifth feature information and the value output from the fourth activation layer834.

The value output from the element-wise summation layer860may be input into a sixth convolution layer816. Sixth feature information may be obtained by performing a convolution operation between the value input into the sixth convolution layer816and a sixth kernel included in the sixth convolution layer816. The sixth feature information may include 32 channels. The sixth feature information may be input into a sixth activation layer836. In the sixth activation layer836, an operation of applying an activation function to the sixth feature information input into the sixth activation layer836may be performed. The value output from the sixth activation layer836may be input into a first split layer871, and the value output from the sixth activation layer836may include 32 channels.

The first split layer871may divide 32 channels in half (which may mean to divide by 2, or to multiply by ½), to output 16 channels to a seventh convolution layer817and output the other 16 channels to an eighth convolution layer818.

First output information881may be obtained by performing a convolution operation between the value input into the seventh convolution layer817and a seventh kernel included in the seventh convolution layer817. The first output information881may include 48 channels, and the first output information881may correspond to the first input information721.

Eighth feature information may be obtained by performing a convolution operation between the value input into the eighth convolution layer818and an eighth kernel included in the eighth convolution layer818. The eighth feature information may include 28 channels. The eighth feature information may be input into an eighth activation layer838. An operation of applying an activation function to the eighth feature information may be performed in the eighth activation layer838. The value output from the eighth activation layer838may be input into a second split layer872, and the value output from the eighth activation layer838may include 28 channels.

The second split layer872may divide 28 channels in in half to output 14 channels to a ninth convolution layer819and output the other 14 channels to a tenth convolution layer820.

Second output information882may be obtained by performing a convolution operation between the value input into the ninth convolution layer819and a ninth kernel included in the ninth convolution layer819. The second output information882may include 40 channels, and the second output information882may correspond to the second input information722.

Also, tenth feature information may be obtained by performing a convolution operation between the value input into the tenth convolution layer820and a tenth kernel included in the tenth convolution layer820. The tenth feature information may include 28 channels. The tenth feature information may be input into a tenth activation layer840. An operation of applying an activation function to the tenth feature information may be performed in the tenth activation layer840. The value output from the tenth activation layer840may be input into a third split layer873, and the value output from the tenth activation layer840may include 28 channels.

The third split layer873may divide 28 channels in half to output 14 channels to an eleventh convolution layer821and output the other 14 channels to a twelfth convolution layer822.

Third output information883may be obtained by performing a convolution operation between the value input into the eleventh convolution layer821and an eleventh kernel included in the eleventh convolution layer821. The third output information883may include 40 channels, and the third output information883may correspond to the third input information723.

Also, fourth output information884may be obtained by performing a convolution operation between the value input into the twelfth convolution layer822and a twelfth kernel included in the twelfth convolution layer822. The fourth output information884may include 41 channels, and the fourth output information884may correspond to the fourth input information724.

Moreover, the attention map generator220according to an embodiment of the disclosure may generate an attention map650based on the first to fourth output information881,882,883, and884.

An example of this will be described in detail with reference toFIG.9.

FIG.9is a diagram for describing an operation of generating an attention map, according to an embodiment of the disclosure.

Referring toFIG.9, based on the order of the channels included in the first output information881and the order of the channels included in the first input information721, the attention map generator220may assign the same channel number (which may be an index) to the channels included in the first output information881as to the channels included in the first input information721. For example, when the channel number of the first channel included in the first input information721is ‘0’, the attention map generator220may assign the channel number ‘0’ to the first channel included in the first output information881.

Also, the channel number may be assigned to the second to fourth output information882,883, and884in the same way. For example, when the channel number of the first channel included in the second input information722is ‘5’, the channel number ‘5’ may be assigned to the first channel included in the second output information882.

A two-dimensional image920ofFIG.9may represent an image in which the channels included in the first to fourth output information881,882,883, and884are arranged from left to right and from top to bottom in the order of channel numbers.

When a channel number is assigned to the first to fourth output information881,882,883, and884, the attention map generator220may generate an attention map650by arranging the channels included in the first to fourth output information881,882,883, and884in the order of channel numbers and then integrating the first to fourth output information881,882,883, and884.

FIGS.10A and10Bare diagrams for comparing a method of generating an attention map by using a first convolutional neural network and a method of generating an attention map by using a second convolutional neural network, according to an embodiment of the disclosure.

Referring toFIG.10A, a first convolutional neural network1010may include 14 convolution layers, 9 activation layers, and 4 element-wise summation layers. In this case, an attention map650may be generated by inputting the similarity information350including 169 channels into the first convolutional neural network1010and performing a convolution operation in each of the 14 convolution layers, an activation operation in each of the 9 activation layers, and an element-wise summation operation in each of the 4 element-wise summation layers in the first convolutional neural network1010.

When the similarity information350including 169 channels is image-processed by using the first convolutional neural network1010, the number of channels of the input information of a first convolution layer C1may be 169 and the number of channels of the output information thereof may be 134. Also, the number of channels of the input information of a second convolution layer C2may be 134, and the number of channels of the output information thereof may be 99. Also, the number of channels of the input information of a third convolution layer C3may be 99, and the number of channels of the output information thereof may be 64. Also, the number of channels of the input information of fourth to eleventh convolution layers C4, C5, C6, C7, C8, C9, C10, and C11may be 64, and the number of channels of the output information thereof may be 64. Also, the number of channels of the input information of a twelfth convolution layer C12may be 64, and the number of channels of the output information thereof may be 99. The number of channels of the input information of a thirteenth convolution layer C13may be 99, and the number of channels of the output information thereof may be 134. Also, the number of channels of the input information of a fourteenth convolution layer C14may be 134, and the number of channels of the output information thereof may be 169.

Moreover, referring toFIG.10B, a second convolutional neural network1020may be the convolutional neural network610illustrated and described inFIG.8.

When the similarity information350including 169 channels is image-processed by using the second convolutional neural network1020, the number of channels of the input information of a first convolution layer C′1may be 48 and the number of channels of the output information thereof may be 16. Also, the number of channels of the input information of a second convolution layer C′2may be 56, and the number of channels of the output information thereof may be 20. Also, the number of channels of the input information of a third convolution layer C′3may be 60, and the number of channels of the output information thereof may be 20. Also, the number of channels of the input information of a fourth convolution layer C′4may be 61 and the number of channels of the output information thereof may be 24, and the number of channels of the input information of a fifth convolution layer C′5may be 24 and the number of channels of the output information thereof may be 24. The number of channels of the input information of a sixth convolution layer C′6may be 24, and the number of channels of the output information thereof may be 32. Also, the number of channels of the input information of a seventh convolution layer C′7may be 16, and the number of channels of the output information thereof may be 48. Also, the number of channels of the input information of an eighth convolution layer C′8may be 16, and the number of channels of the output information thereof may be 28. The number of channels of the input information of a ninth convolution layer C′9may be 14, and the number of channels of the output information thereof may be 40. The number of channels of the input information of a tenth convolution layer C′10may be 14, and the number of channels of the output information thereof may be 28. The number of channels of the input information of an eleventh convolution layer C′11may be 14, and the number of channels of the output information thereof may be 40. The number of channels of the input information of a twelfth convolution layer C′12may be 14, and the number of channels of the output information thereof may be 41.

Moreover, when the size of the input information of the convolution layer is W×H, the number of channels is Ci, the size of the kernel included in the convolution layer is K×K, and the size of the output information of the convolution layer is W×H, and the number of channels is Co, the number of operations of the convolution operation performed in the convolution layer may be expressed as Equation 1 below.
Number of operations=W×H×K2×Ci×Co[Equation 1]

Accordingly, when the size of the similarity information350is 128×128 and the size of the kernel included in each of the first to fourteenth convolution layers C1, C2, . . . , C14included in the first convolutional neural network1010is 3×3, the number of operations of the entire convolution operation performed in the first convolutional neural network1010may be 17,291,280,384 as illustrated inFIG.10A. This may be referred to as a first number of operations.

Also, the size of the kernel included in each of the first to twelfth convolution layers C′1, C′2, C′12included in the second convolutional neural network1020is 3×3, the number of operations of the entire convolution operation performed in the second convolutional neural network1020may be 1,356,300,288 as illustrated inFIG.10B. This may be referred to as a second number of operations.

The second number of operations of the entire convolution operation performed in the second convolutional neural network1020may be 1/12.75 times the first number of operations of the entire convolution operation performed in the first convolutional neural network1010.

Moreover, in order to reduce the operation amount of the convolution operation, when the convolution operation is performed by being divided into depth-wise convolution and point-wise convolution, the number of operations of the entire convolution operation may be expressed as Equation 2 below.
Number of operations=W×H×(K2×Ci)+W×H×12×Ci×Co=W×H×Ci×(K2+Co)  [Equation 2]

Thus, when the convolution operation of the first convolutional neural network1010is lightened, the number of operations of the entire convolution operation may be 2,099,822,592 as illustrated inFIG.10A. This may be referred to as a third number of operations.

Accordingly, even when the convolution operation performed in the first convolutional neural network1010is lightened, the second number of operations of the second convolutional neural network1020that is not lightened may be smaller.

Thus, when the attention map is generated by using the second convolutional neural network by dividing the similarity information into a plurality of groups and inputting a plurality of divided groups into different layers, the operation amount may be reduced while similarly maintaining the image processing performance as in the case of generating the attention map by using the first convolutional neural network.

FIG.11is a diagram for describing an operation of generating an attention map, according to another embodiment of the disclosure.

According to an embodiment of the disclosure, the attention map generator220may downscale similarity information1110. For example, when the size of the similarity information1110is downscaled to 50% or to ½, the size of the downscaled similarity information may be W/2×H/2 and the number of channels may be K2. In this case, the attention map generator220may downscale the similarity information1110by using a bilinear interpolation method, a bicubic interpolation method, a nearest neighbor interpolation method, or the like. However, the disclosure is not limited thereto.

Also, the similarity information1110ofFIG.11may include 81 channels and may be divided into a plurality of groups according to the channel-wise characteristic value included in the downscaled similarity information. For example, the similarity information1110may be divided into first to fourth groups according to the intensity level of each of the channels included in the downscaled similarity information. In this case, 40 channels may be included in the first group, which may be referred to as first input information1121, 28 channels may be included in the second group, which may be referred to as second input information1122, 12 channels may be included in the third group, which may be referred to as third input information1123, and 1 channel may be included in the fourth group, which may be referred to as fourth input information1124.

According to an embodiment of the disclosure, the attention map generator220may obtain first to fourth output information1141,1142,1143, and1144by processing the first to fourth input information1121,1122,1123, and1124by using a convolutional neural network1130. Because the convolutional neural network1130has the same structure as the convolutional neural network610ofFIG.8and the structure and operation of the convolutional neural network610have been described in detail with reference toFIG.8, redundant descriptions of the structure and operation of the convolutional neural network1130will be omitted for conciseness.

Also, the attention map generator220may generate an attention map1050based on the first to fourth output information1141,1142,1143, and1144. Because an operation of the attention map generator220for generating an attention map based on a plurality of pieces of output information has been described in detail with reference toFIG.9, redundant descriptions thereof will be omitted for conciseness.

As illustrated and described inFIG.11, when an attention map1150is generated by downscaling the similarity information1110, the number of channels of the input information and the number of channels of the output information may be reduced in each of the convolution layers included in the convolutional neural network1130and thus the number of operations of the entire convolution operation may be reduced. Thus, when the similarity information1110is downscaled, the attention map1150may be generated with a smaller number of operations while similarly maintaining the feature of the similarity information1110.

FIG.12is a diagram for describing an operation of generating an attention map, according to another embodiment of the disclosure.

Referring toFIG.12, the image processing apparatus100according to an embodiment of the disclosure may further include a quality estimator1210for obtaining quality information of the first image101.

According to an embodiment of the disclosure, the quality estimator1210may obtain quality information corresponding to the first image101or each of a plurality of regions included in the first image101. The quality estimator1210may estimate the quality of the entire first image101or each region based on texture, edge, and/or noise information included in each of the plurality of regions included in the first image101. In this case, the quality estimator1210may obtain quality information for the entire first image or for each region based on a pre-trained quality estimation network. For example, the quality estimation network may be a network for receiving an entire image or a region of an image and outputting a quality value of the image or region; however, the disclosure is not limited thereto. Also, the quality estimator1210may obtain quality information for each pixel included in the first image101.

According to an embodiment of the disclosure, a convolutional neural network1220may have a structure similar to the convolutional neural network801described above with respect toFIG.8, and in addition each of the seventh convolution layer817, the ninth convolution layer819, the eleventh convolution layer821, and the twelfth convolution layer822includes a plurality of convolution layers.

For example, the seventh convolution layer817may include a structure in which N convolution layers ((7-1)th to (7-N)th convolution layers1230) are connected in parallel. The values output from the first split layer871may be input into each of the (7-1)th to (7-N)th convolution layers1230. In this case, the output values output from the N convolution layers may correspond to N pieces of preset quality information. For example, the value output from the (7-1)th convolution layer may be an output value corresponding to first quality information, and the value output from the (7-N)th convolution layer may be an output value corresponding to N-th quality information.

According to an embodiment of the disclosure, a weight determiner1215may determine the weights of the values output from each of the N convolution layers according to the pixel-wise quality information (the quality information for each of the pixels) obtained by the quality estimator1210.

The attention map generator220may obtain a weighted sum of the values output from each of the (7-1)th to (7-N)th convolution layers as first output information1271, based on the determined weights. In the same way, a weighted sum of the values output from each of (9-1)th to (9-N)th convolution layers1240may be obtained as second output information1272.

Also, a weighted sum of the values output from each of (11-1)th to (11-N)th convolution layers1250may be obtained as third output information1273, and a weighted sum of the values output from each of (12-1)th to (12-N)th convolution layers1260may be obtained as fourth output information1274.

The attention map generator220may generate an attention map based on the first to fourth output information1271,1272,1273, and1274. Because an operation of generating an attention map based on a plurality of pieces of output information has been described in detail with reference toFIG.9, redundant descriptions thereof will be omitted for conciseness.

Referring back toFIG.2, the spatially variant kernel generator230according to an embodiment of the disclosure may generate a spatially variant kernel based on the spatial kernel and the attention map generated by the attention map generator220. In this case, the spatial kernel may represent weight information according to the position relationship between each of the pixels included in the first image101and a neighboring pixel. A method of generating the spatially variant kernel will be described in detail with reference toFIG.13.

FIG.13is a diagram referenced to describe a method of generating a spatially variant kernel, according to an embodiment of the disclosure.

Referring toFIG.13, the spatially variant kernel generator230may generate a spatially variant kernel1350by using a spatial kernel1310and an attention map650. For example, the spatially variant kernel generator230may convert the spatial kernel1310into a one-dimensional vector1320. The spatial kernel1310may have a size of K×K, and as for the pixel values included in the spatial kernel1310, the center pixel value may be greatest and the pixel value may decrease away from the center pixel. The spatially variant kernel generator230may arrange the pixel values included in the spatial kernel1310in the channel direction and convert the spatial kernel1310into a weight vector1320having a size of 1×1×K2.

Moreover, the size of the attention map650generated by the attention map generator220may be W×H and the number of channels thereof may be K2.

The spatially variant kernel generator230may generate the spatially variant kernel1350by multiplying the attention map650and the weight vector1320. In this case, the spatially variant kernel generator230may generate the spatially variant kernel1350by performing element-wise multiplication between each of the one-dimensional vectors having a size of 1×1×K2included in the attention map650and the weight vector1320having a size of 1×1×K2.

As illustrated inFIG.13, a second vector1351may be generated by performing an element-wise multiplication operation between the weight vector1320and a first vector651included in the attention map650. In this case, the position of the first vector651in the attention map650and the position of the second vector1351in the spatially variant kernel1350may correspond to each other.

According to an embodiment of the disclosure, like the attention map650, the spatially variant kernel1350may have a size of W×H and may include K2channels.

Referring back toFIG.2, the spatially variant kernel generator230may output the generated spatially variant kernel1350to the filter240, and the filter240may generate the second image102by receiving the first image101and applying the spatially variant kernel1350to the first image101. A method of generating the second image102by applying the spatially variant kernel1350to the first image101will be described in detail with reference toFIG.14.

FIG.14is a diagram referenced to describe a method of applying a spatially variant kernel to a first image, according to an embodiment of the disclosure.

Referring toFIG.14, the spatially variant kernel1350according to an embodiment of the disclosure may include a kernel vector corresponding to each of the pixels included in the first image. For example, the spatially variant kernel1350may include a first kernel vector1351corresponding to a first pixel1410included in the first image101and may include a second kernel vector1352corresponding to a second pixel1420included in the first image101. Also, the spatially variant kernel1350may include a third kernel vector1353corresponding to a third pixel1430included in the first image101.

The filter240may convert a one-dimensional kernel vector having a size of 1×1×K2into a two-dimensional kernel having a size of K×K. For example, the first kernel vector1351may be converted into a first kernel1415, the second kernel vector1352may be converted into a second kernel1425, and the third kernel vector1353may be converted into a third kernel1435.

The filter240may calculate a value of a fourth pixel1440of the second image102by performing filtering by applying the first kernel1415to a first region centered on the first pixel1410included in the first image101. Also, the filter240may calculate a value of a fifth pixel1450of the second image102by performing filtering by applying the second kernel1425to a second region centered on the second pixel1420included in the first image101. Also, the filter240may calculate a value of a sixth pixel1460of the second image102by performing filtering by applying the third kernel1435to a third region centered on the third pixel1430included in the first image101.

In the same way, the filter240may calculate the pixel values included in the second image102by applying a kernel corresponding to each of the pixels included in the first image101to a region centered on each of the pixels included in the first image101.

FIG.15is a flowchart illustrating an operating method of an image processing apparatus according to an embodiment of the disclosure.

Referring toFIG.15, the image processing apparatus100according to an embodiment of the disclosure may obtain characteristic information of a first image at operation S1510.

For example, the characteristic information of the first image may include similarity information, frequency characteristic information, gradient characteristic information, region characteristic information, and/or the like. The similarity information may be information representing the similarity between each of the pixels included in the first image and a neighboring pixel. Because a method of generating the similarity information has been described in detail with reference toFIGS.3,4A, and4B, redundant descriptions thereof will be omitted for conciseness.

Also, the image processing apparatus100may obtain frequency characteristic information by performing filtering based on a Gaussian kernel or discrete cosine transform or wavelet transform or the like on the first image. An example of this has already been described in detail with reference toFIG.5, and thus, redundant descriptions thereof will be omitted for conciseness.

According to an embodiment of the disclosure, the image processing apparatus100may divide the characteristic information into a plurality of groups at operation S1520.

For example, when the characteristic information includes a plurality of channels, the image processing apparatus100may divide the characteristic information into a plurality of groups according to the channel-wise characteristic value. When the characteristic information is similarity information, the channels may be divided into a plurality of groups according to the strength of each of the channels included in the similarity information. This has already been described in detail with reference toFIG.7, and thus, redundant descriptions thereof will be omitted for conciseness.

According to an embodiment of the disclosure, the image processing apparatus100may obtain a plurality of pieces of output information corresponding to the plurality of groups by using a convolutional neural network at operation S1530.

For example, the image processing apparatus100may respectively input the plurality of groups divided in operation S1520to different layers included in the convolutional neural network. First input information corresponding to a first group may be input into a first convolution layer, and second input information corresponding to a second group may be input into a second convolution layer. In this case, the second convolution layer may be a layer located after the first convolution layer in the convolutional neural network. A convolution operation with one or more kernels may be performed on the input information. The image processing apparatus100may obtain a plurality of pieces of output information corresponding to the plurality of groups from different layers included in the convolutional neural network. For example, first output information corresponding to the first input information may be output from a third convolution layer, and second output information corresponding to the second input information may be output from a fourth convolution layer. In this case, the fourth convolution layer may be a layer located after the third convolution layer in the convolutional neural network.

According to an embodiment of the disclosure, the image processing apparatus100may generate an attention map based on the plurality of pieces of output information at operation S1540.

For example, the image processing apparatus100may determine the order of the channels of the output information based on the order of the channels of the input information. The image processing apparatus100may generate an attention map by integrating the plurality of pieces of output information in the channel direction according to the order of the channels. An example of this has already been described in detail with reference toFIG.9, and thus, redundant descriptions thereof will be omitted for conciseness.

According to an embodiment of the disclosure, the image processing apparatus100may generate a spatially variant kernel based on the spatial kernel and the attention map at operation S1550.

For example, the image processing apparatus100may convert the spatial kernel into a one-dimensional vector. The spatial kernel may have a size of K×K, and as for the pixel values included in the spatial kernel, the center pixel value may be greatest and the pixel value may decrease away from the center pixel. The image processing apparatus100may arrange the pixel values included in the spatial kernel in the channel direction and convert the spatial kernel into a weight vector having a size of 1×1×K2.

The size of the attention map generated in operation S1530may be W×H, and the number of channels thereof may be K2.

The image processing apparatus100may generate a spatially variant kernel by multiplying the attention map and the weight vector. In this case, the image processing apparatus100may generate a spatially variant kernel by performing element-wise multiplication between each of the one-dimensional vectors having a size of 1×1×K2included in the attention map and a weight vector having a size of 1×1×K2.

A method of generating the spatially variant kernel has already been described in detail with reference toFIG.13, and thus, redundant descriptions thereof will be omitted for conciseness.

According to an embodiment of the disclosure, the image processing apparatus100may generate a second image by applying the spatially variant kernel to the first image at operation S1560.

The spatially variant kernel generated in operation S1550may include a kernel vector corresponding to each of the pixels included in the first image. For example, the spatially variant kernel may include a first kernel vector corresponding to a first pixel included in the first image and may include a second kernel vector corresponding to a second pixel included in the first image.

The image processing apparatus100may convert a one-dimensional kernel vector having a size of 1×1×K2into a two-dimensional kernel having a size of K×K. For example, the first kernel vector may be converted into a two-dimensional first kernel, and the second kernel vector may be converted into a two-dimensional second kernel.

The image processing apparatus100may calculate a third pixel value included in the second image by performing filtering by applying the first kernel to a region centered on the first pixel and may calculate a fourth pixel value included in the second image by performing filtering by applying the second kernel to a region centered on the second pixel.

Accordingly, when filtering the first image, the image processing apparatus100may perform the filtering by applying different kernels according to the position of the center pixel.

FIG.16is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the disclosure.

Referring toFIG.16, the image processing apparatus100according to an embodiment of the disclosure may include a processor120and a memory130.

According to an embodiment of the disclosure, the processor120may overall control the image processing apparatus100. According to an embodiment of the disclosure, the processor120may execute one or more programs stored in the memory130.

According to an embodiment of the disclosure, the memory130may store various data, programs, or applications for driving and controlling the image processing apparatus100. The program stored in the memory130may include one or more instructions. The application or program (one or more instructions) stored in the memory130may be executed by the processor120.

According to an embodiment of the disclosure, the processor120may include at least one of a central processing unit (CPU), a graphic processing unit (GPU), or a video processing unit (VPU). Alternatively, according to an embodiment of the disclosure, the processor120may be implemented in the form of a system-on-chip (SoC) in which at least one of a CPU, a GPU, or a VPU is integrated. Alternatively, the processor120may further include a neural processing unit (NPU).

According to an embodiment of the disclosure, the processor120may use the image processing network103to generate an output image that has undergone denoising for maintaining the texture and fine edge processing while removing noise from an input image. For example, the processor120may perform at least one of the operations of the characteristic information generator210, the attention map generator220, the spatially variant kernel generator230, and the filter240illustrated and described with reference toFIGS.2to14.

The processor120may obtain characteristic information of a first image. For example, the characteristic information of the first image may include similarity information, frequency characteristic information, gradient characteristic information, region characteristic information, and/or the like. Because a method of generating the similarity information has already been described in detail with reference toFIGS.3,4A, and4Band a method of obtaining the frequency characteristic information has already been described in detail with reference toFIG.5, redundant descriptions thereof will be omitted for conciseness.

The processor120may divide the characteristic information into a plurality of groups. For example, when the characteristic information includes a plurality of channels, the processor120may divide the characteristic information into a plurality of groups according to the channel-wise characteristic value. When the characteristic information is similarity information, the channels may be divided into a plurality of groups according to the strength of each of the channels included in the similarity information. An example of this has already been described in detail with reference toFIG.7, and thus, redundant descriptions thereof will be omitted for conciseness.

The processor120may obtain a plurality of pieces of output information corresponding to the plurality of groups by using a convolutional neural network. For example, the processor120may respectively input a plurality of divided groups into different layers included in the convolutional neural network. First input information corresponding to a first group may be input into a first convolution layer, and second input information corresponding to a second group may be input into a second convolution layer. In this case, the second convolution layer may be a layer located after the first convolution layer in the convolutional neural network. A convolution operation with one or more kernels may be performed on the input information. The processor120may obtain a plurality of pieces of output information corresponding to the plurality of groups from different layers included in the convolutional neural network. For example, first output information corresponding to the first input information may be output from a third convolution layer, and second output information corresponding to the second input information may be output from a fourth convolution layer. In this case, the fourth convolution layer may be a layer located after the third convolution layer in the convolutional neural network.

The processor120may generate an attention map based on the plurality of pieces of output information. For example, the processor120may determine the order of the channels of the output information based on the order of the channels of the input information. The processor120may generate an attention map by integrating the plurality of pieces of output information in the channel direction according to the order of the channels. An example of this has already been described in detail with reference toFIG.9, and thus, redundant descriptions thereof will be omitted for conciseness.

Also, the processor120may generate a spatially variant kernel based on the spatial kernel and the attention map. For example, the processor120may convert the spatial kernel into a one-dimensional vector. The spatial kernel may have a size of K×K, and as for the pixel values included in the spatial kernel, the center pixel value may be greatest and the pixel value may decrease away from the center pixel. The processor120may arrange the pixel values included in the spatial kernel in the channel direction and convert the spatial kernel into a weight vector having a size of 1×3×K2. According to an embodiment of the disclosure, the size of the attention map may be W×H, the number of channels thereof may be K2, and the processor120may generate a spatially variant kernel by multiplying the attention map and the weight vector. In this case, the processor120may generate a spatially variant kernel by performing element-wise multiplication between each of the one-dimensional vectors having a size of 1×1×K2included in the attention map and a weight vector having a size of 1×1×K2.

The processor120may generate a second image by applying the spatially variant kernel to the first image. The spatially variant kernel may include a kernel vector corresponding to each of the pixels included in the first image. For example, the spatially variant kernel may include a first kernel vector corresponding to a first pixel included in the first image and may include a second kernel vector corresponding to a second pixel included in the first image.

The processor120may convert a one-dimensional kernel vector having a size of 1×1×K2into a two-dimensional kernel having a size of K×K. For example, the first kernel vector may be converted into a two-dimensional first kernel, and the second kernel vector may be converted into a two-dimensional second kernel. The processor120may calculate a third pixel value included in the second image by performing filtering by applying the first kernel to a region centered on the first pixel and may calculate a fourth pixel value included in the second image by performing filtering by applying the second kernel to a region centered on the second pixel.

Moreover, the image processing network103according to an embodiment of the disclosure may be a network trained by a server or an external device. The external device may train the image processing network103based on training data. In this case, the training data may include a plurality of data sets including image data including noise and image data in which an edge feature or a texture feature is preserved while noise is removed.

The server or the external device may determine parameter values included in the kernels used in each of a plurality of convolution layers included in the image processing network103. For example, the server or the external device may determine the parameter values in the direction of minimizing the difference (for example loss information) in the image data (for example training data) in which the edge feature is preserved while the noise and the image data generated by the image processing network103are removed.

According to an embodiment of the disclosure, the image processing apparatus100may receive the trained image processing network103from the server or the external device and store the same in the memory130. For example, the memory130may store the structure and parameter values of the image processing network103according to an embodiment of the disclosure, and the processor120may use the parameter values stored in the memory130to generate a second image in which the edge feature is preserved while the noise is removed from the first image according to an embodiment of the disclosure.

Moreover, the block diagram of the image processing apparatus100illustrated inFIG.16may be a block diagram for an embodiment of the disclosure. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the image processing apparatus100that are actually implemented. That is, when necessary, two or more components may be combined into one component, or one component may be divided into two or more components. Also, functions performed by the respective blocks are for describing embodiments, and particular operations or devices thereof do not limit the scope of the disclosure.

The operating method of the image processing apparatus according to an embodiment of the disclosure may be stored in a computer-readable recording medium by being implemented in the form of program commands that may be performed by various computer means. The computer-readable recording medium may include program instructions, data files, and data structures either alone or in combination. The program commands recorded on the computer-readable recording medium may be those that are especially designed and configured for the disclosure, or may be those that are known and available to computer programmers of ordinary skill in the art. Examples of the computer-readable recording medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks, and hardware devices such as ROMs, RAMs, and flash memories particularly configured to store and execute program commands. Examples of the program commands may include not only machine language code generated by a compiler but also high-level language code that may be executed by a computer by using an interpreter or the like.

Also, the image processing apparatus and the operating method thereof according to the described embodiments may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer.

The computer program product may include a S/W program and a computer-readable storage medium with a S/W program stored therein. For example, the computer program product may include products in the form of S/W programs (e.g., downloadable apps) electronically distributed through manufacturers of electronic devices or electronic markets (e.g., Google Play Store and App Store). For electronic distribution, at least a portion of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a server of a manufacturer, a server of an electronic market, or a relay server for temporarily storing the S/W program.

In a system including a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, when there is a third device (e.g., a smartphone) communicatively connected to the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself that is transmitted from the server to the client device or the third device or transmitted from the third device to the client device.

In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the described embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the described embodiments in a distributed manner.

For example, the server (e.g., a cloud server or an artificial intelligence server) may execute the computer program product stored in the server, to control the client device communicatively connected to the server to perform the method according to the described embodiments.

Although embodiments have been described above in detail, the scope of the disclosure is not limited thereto and various modifications and improvements made by those of ordinary skill in the art by using the basic concept of the disclosure defined in the following claims are also included in the scope of the disclosure.