Patent ID: 12190471

DETAILED DESCRIPTION

Hereinafter, example embodiments according to the disclosure will be described with reference to the accompanying drawings.

Throughout the disclosure, the expression “at least one of a, b or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

The terms used herein will be briefly described, and the disclosure will be described in detail.

The terms used herein are those general terms currently widely used in the art in consideration of functions in the disclosure but the terms may vary according to the intention of one of ordinary skill in the art, precedents, or new technology in the art. Also, some of the terms used herein may be arbitrarily chosen by the present applicant, and in this case, these terms are defined in detail below. Accordingly, the specific terms used herein should be defined based on the unique meanings thereof and the whole context of the disclosure.

It will be understood that when a certain part “includes” a certain component, the part does not exclude another component but may further include another component, unless the context clearly dictates otherwise. The terms “unit”, “ . . . er/or”, and “module” when used in this specification refers to a unit in which at least one function or operation is performed, and may be implemented as hardware, software, or a combination of hardware and software.

The disclosure will now be described more fully with reference to the accompanying drawings for one of ordinary skill in the art to be able to perform the disclosure without any difficulty. However, the disclosure may be embodied in many different forms and is not limited to the embodiments of the disclosure set forth herein. For clarity, portions irrelevant to the descriptions of the disclosure are omitted in the drawings, and like components are denoted by like reference numerals throughout the specification.

FIG.1is a diagram of an operation, performed by an image processing apparatus, of processing an input image, according to an embodiment of the disclosure.

According to an embodiment of the disclosure, an image processing apparatus100may be implemented as various types of electronic devices such as a mobile phone, a digital camera, a camcorder, a laptop computer, a desktop computer, an electronic book terminal, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, an MP3 player, an Internet protocol television (IPTV), a digital TV (DTV), a wearable device, etc.

Referring toFIG.1, according to an embodiment of the disclosure, the image processing apparatus100may perform a process of improving an image quality of an input image101. For example, the input image101may be an old photo or a portrait, but is not limited thereto.

According to an embodiment of the disclosure, the image processing apparatus100may include an old photo detector110, an old photo processor120, and a regular photo processor130.

The old photo detector110may receive the input image101and determine whether the input image101is an old photo. For example, the old photo detector110may perform image processing on the input image101to obtain characteristic information about the input image101including information indicating whether the input image101has faded, color distribution information about the input image101, noise information about the input image101, etc. The old photo detector110may determine whether the input image101is an old photo based on the obtained characteristic information about the input image101.

Alternatively, the old photo detector110may determine whether the input image101is an old photo by using a classification network. This will be described in detail with reference toFIG.2.

FIG.2is a diagram of a classification network according to an embodiment of the disclosure.

According to an embodiment of the disclosure, a classification network115may be a two-class classification model that classifies the input image101into two types or may be a model that classifies the input image101as an old photo or a regular photo. For example, referring toFIG.2, the classification network115may include a VGG network including a plurality of convolutional layers116, a plurality of pooling layers117, and a plurality of fully connected layers118. However, the disclosure is not limited thereto, and the classification network may include various structures.

Also, the old photo detector110may extract a portion of the input image101and input the portion of the input image101into the classification network115. Accordingly, an operation of detecting whether the input image101is an old photo may be quickly performed.

Referring back toFIG.1, when the old photo detector110classifies the input image101as an old photo, the old photo processor120may perform image processing on the input image101.

According to an embodiment of the disclosure, the old photo processor120may include an old photo restorer121, a first face region detector122, and a first face region restorer123. The input image101may be input into the old photo restorer121, and the old photo restorer121may perform image processing on the input image101by applying an old photo restoration model thereto. An image processed by the old photo restoration model may be input into the first face region detector122. The first face region detector122may detect a face region by using various algorithms and various models. For example, the first face region detector122may detect the face region by using a histogram-of-oriented-gradient (HoG)-based feature detection algorithm. The first face region detector122may divide an input image into regions having a certain size and calculate a gradient of pixels for each region. The first face region detector122may calculate, for each region, a histogram for directions of pixels having a gradient greater than or equal to a certain value among pixels included in one region and may determine whether the region is a face region based on the calculated histogram. Alternatively, the first face region detector122may detect the face region by using the classification network, but is not limited thereto.

When a first image includes the face region, the first face region restorer123may perform image processing for restoring an image quality of the face region included in the first image by using a face restoration model. Accordingly, the old photo processor120may obtain a restored old photo.

In contrast, when the old photo detector110classifies the input image101as a regular photo rather than an old photo, the regular photo processor130may perform image processing on the input image101.

According to an embodiment of the disclosure, the regular photo processor130may include a second face region detector131, a second face region restorer132, and a background restorer133. The input image101classified as a regular photo may be input into the second face region detector131, and the second face region detector131may detect whether the input image101includes a face region by using the same or similar method to that of the first face region detector122.

When the input image101includes the face region, the second face region restorer132may perform image processing for restoring an image quality of the face region included in the input image101by using the face restoration model. Also, the background restorer133may perform image processing for restoring an image quality of a background region rather than the face region included in the input image101by using a background restoration model.

The regular photo processor130may obtain a restored regular photo by synthesizing the face region whose image quality has been restored by the second face region restorer132and the background region whose image quality has been restored by the background restorer133.

Moreover, according to an embodiment of the disclosure, the old photo restoration model, the face restoration model, and the background restoration model may include an image processing network having the same or similar structure, and the image processing network may include at least one network (or at least one neural network). Hereinafter, the image processing network will be described with reference to the accompanying drawings.

FIG.3is a diagram for describing an image processing network according to an embodiment of the disclosure.

Referring toFIG.3, according to an embodiment of the disclosure, an image processing network50may include a downscale network200, a feature extraction network300, an image quality processing network400, and an upscale network500. According to an embodiment of the disclosure, the image processing apparatus100may perform image quality processing (e.g., image quality processing of an old photo, image quality processing on a face region, image quality processing on a background region, etc.) by using the image processing network50. The image quality processing may include various processing to enhance quality of an image such as, for example, denoising, de-lighting effects, and contrast and sharpness adjustment (such as improving contrast and sharpness).

For example, according to an embodiment of the disclosure, the downscale network200may generate a first image by receiving an input image20and downscaling the input image20. In this case, the downscale network200may hierarchically downscale the input image20. For example, the downscale network200may downscale a resolution of the input image20to ½ and downscale again the image whose resolution has been downscaled to ½, to obtain a first image in which the resolution of the input image20has been downscaled to ¼. An operation, performed by the downscale network200, of downscaling an input image will be described in detail with reference toFIG.4.

The downscaled first image may be input into the image quality processing network400and the feature extraction network300.

According to an embodiment of the disclosure, the feature extraction network300may receive the first image, extract at least one piece of feature information from the first image, and transmit the at least one piece of extracted feature information to the image quality processing network400. The image quality processing network400may perform a process of improving an image quality of the first image by using the at least one piece of feature information received from the feature extraction network300. For example, the image quality processing network400may remove noise or artifacts included in the first image and may perform a process of improving structural information such as edges of the first image, and detailed texture information. However, the disclosure is not limited thereto.

In a main network30including the feature extraction network300and the image quality processing network400, image quality processing is performed on a downscaled image, and accordingly, image processing speed may be increased and memory usage may be reduced.

A second image obtained by the image quality processing network400performing image quality processing may be input into the upscale network500.

According to an embodiment of the disclosure, the upscale network500may generate an output image40by upscaling the second image. The upscale network500may obtain a third image by upscaling the second image and may receive, from the downscale network200, image information before downscaling is performed. The upscale network500may perform image quality processing on the upscaled third image based on the image information received from the downscale network200and may obtain the output image40based on the third image on which image quality processing has been performed.

Image processing operations performed by the downscale network200, the feature extraction network300, the image quality processing network400, and the upscale network500, according to an embodiment of the disclosure, will be described with reference to the drawings below.

FIG.4is a diagram of structures of a downscale network and an upscale network, according to an embodiment of the disclosure.

Referring toFIG.4, according to an embodiment of the disclosure, the downscale network200may include at least one downscale block. Although it has been illustrated inFIG.4for convenience of description, that the downscale network200includes a first downscale block210and a second downscale block220, the disclosure is not limited thereto. The downscale network200may include one downscale block or may include at least three downscale blocks.

The first downscale block210and the second downscale block220may be sequentially connected, and the downscale network200may further include a convolutional layer201and an activation layer202, before the first downscale block210. However, the disclosure is not limited thereto.

According to an embodiment of the disclosure, the first downscale block210may include at least one convolutional layer211and at least one activation layer212. Also, the first downscale block210may include at least one space-to-depth layer213. For example, as illustrated inFIG.4, the first downscale block210may have a structure in which a convolutional layer211, a space-to-depth layer213, and an activation layer212are sequentially positioned, so that data output from the convolutional layer211is input into the space-to-depth layer213and data output from the space-to-depth213is input into the activation layer212. However, the disclosure is not limited thereto.

According to an embodiment of the disclosure, in the convolutional layer211, a convolutional operation between data input into the convolutional layer211and a kernel included in the convolutional layer211may be performed. In the space-to-depth layer213, an operation of reducing a resolution of the data input into the space-to-depth layer213and increasing the number of channels as much as the resolution is reduced may be performed.

The space-to-depth layer213may be a layer that changes resolution information into depth information. Accordingly, information included in the data input into the space-to-depth layer213may be equally included in the data output from the space-to-depth layer213.

In the activation layer212, an activation function operation of applying an activation function to the data input into the activation layer212may be performed. The activation function operation applies non-linear characteristics to an activation layer, and the activation function may include a sigmoid function, a Tan h function, a rectified linear unit (ReLU) function, a leaky ReLU function, etc. However, the disclosure is not limited thereto.

According to an embodiment of the disclosure, the first downscale block210may generate a first downscaled image by downscaling an image input into the first downscale block210by using the convolutional layer211, the space-to-depth layer213, and the activation layer212, which are included in the first downscale block210. In this regard, the first downscaled image may be an image obtained by reducing a resolution of the image input into the first downscale block210to 1/k1. For example, a value of k1 may be 2, and a resolution of the first downscaled image may be ½ of the resolution of the input image. However, the disclosure is not limited thereto.

The value of k1 may be variously determined according to the configuration of layers included in the first downscale block210. For example, the value of k1 may vary according to a kernel of each of the convolutional layers included in the first downscale block210.

The first downscaled image output from the first downscale block210may be input into the second downscale block220. However, the disclosure is not limited thereto.

According to an embodiment of the disclosure, the second downscale block220may include the same or similar structure to that of the first downscale block210. The second downscale block220may generate a second downscaled image by downscaling the first downscaled image. In this regard, the second downscaled image may be an image obtained by reducing the resolution of the first downscaled image to 1/k2. For example, k2 may have a value of 2, and a resolution of the second downscaled image may be ½ of the resolution of the first downscaled image, but is not limited thereto.

The value of k2 may be variously determined according to the configuration of layers included in the second downscale block220. For example, the value of k2 may vary according to a kernel of each of the convolutional layers included in the second downscale block220.

The second downscaled image output from the second downscale block220may be input into the image quality processing network400and the feature extraction network300. The image quality processing network400and the feature extraction network300will be described in detail below with reference toFIGS.5to10.

FIG.5is a diagram of structures of an image quality processing network and a feature extraction network, according to an embodiment of the disclosure.

Referring toFIG.5, according to an embodiment of the disclosure, the feature extraction network300may include at least one feature extraction block. Although it has been illustrated inFIG.5for convenience of description, that the feature extraction network300includes a first feature extraction block310and a second feature extraction block320, the feature extraction network300may include a larger number of feature extraction blocks. Also, the first feature extraction block310and the second feature extraction block320may be sequentially connected.

InFIG.5, it has been illustrated that a first image25is directly input into the first feature extraction block310, but the disclosure is not limited thereto. For example, a convolutional layer and an activation layer are positioned before the first feature extraction block310, and accordingly, an image obtained by performing a certain operation on the first image25may be input into the first feature extraction block310. Also, another feature extraction block, a convolutional layer, and an activation layer may be further positioned between the first feature extraction block310and the second feature extraction block320. Hereinafter, the structure and operations of the first feature extraction block310, according to an embodiment of the disclosure, will be described with reference toFIG.6.

FIG.6is a diagram of a structure of a first feature extraction block, according to an embodiment of the disclosure.

Referring toFIG.6, according to an embodiment of the disclosure, the first feature extraction block310may include a plurality of convolutional layers and may include an activation layer consecutively positioned in each of the plurality of convolutional layers. For example, the first feature extraction block310may include a first convolutional layer311and a second convolutional layer313, and a first activation layer312may be sequentially positioned after the first convolutional layer311, and a second activation layer314may be sequentially positioned after the second convolutional layer313. However, the disclosure is not limited thereto. For example, at least one convolutional layer and at least one activation layer may be further positioned between the first activation layer312and the second convolutional layer313.

In the first convolutional layer311, a first feature map F1may be generated through a convolutional operation between the first image25input into the first convolutional layer311and a first kernel included in the first convolutional layer311. The first feature map F1may be input into the first activation layer312, and an activation function operation of applying an activation function to the first feature map F1may be performed in the first activation layer312.

Also, a feature map output from the first activation layer312may be input into the second convolutional layer313, and in the second convolutional layer313, a second feature map may be generated through a convolutional operation between the feature map input into the second convolutional layer313and a second kernel included in the second convolutional layer313.

The second feature map may be input into the second activation layer314, and an activation function operation of applying an activation function to the second feature map may be performed in the second activation layer314.

Also, the first feature extraction block310may further include a normalization layer315. For example, according to an embodiment of the disclosure, when the image quality processing network400and the feature extraction network300are trained by the same loss function, a size of the feature extraction network300is smaller than a size of the image quality processing network400, and accordingly, gradient exploding occurs in the feature extraction network300. The gradient exploding means that a value of an updated gradient (slope) gradually increases (diverges) in a training process of a network, and as the value of the gradient (slope) gradually increases, parameter values (weight values) included in kernels also gradually increase, which results in a decrease in training speed and efficiency.

According to an embodiment of the disclosure, in order to prevent the occurrence of gradient exploding, the normalization layer315may adjust a range of values output from the second activation layer314. For example, the normalization layer315may normalize values output from the second activation layer314to values of 0 or more and 255 or less or to values of −1 or more and 1 or less. According to an embodiment of the disclosure, a range of normalization may be determined according to hardware performance of the image processing apparatus100.

Referring toFIG.6, the first feature extraction block310may include a skip connection317that skips a plurality of convolutional layers (e.g., the first convolutional layer311and the second convolutional layer313). The skip connection317may be a structure that connects an input of the first convolutional layer311to an output of the normalization layer315. For example, because the first feature extraction block310includes the skip connection317, an output image27may be generated by adding the first image25to a result of passing the first image25through the first convolutional layer311and the second convolutional layer313. Accordingly, the first feature extraction block310may refer to a network that has learned a residual between the first image25and the output image27.

The first feature extraction block310may further include a residual scaler316that adjusts sizes of skip-connected values. The residual scaler316may perform an operation of multiplying each of values included in the first image25by a preset constant value, to adjust sizes of values included in a feature of the first image25without losing feature information about the first image25.

Although only the first feature extraction block310has been illustrated and described inFIG.6, the second feature extraction block320may also have the same or similar structure as the first feature extraction block310and perform the same or similar operations as those of the first feature extraction block310.

FIG.7is a diagram of feature information extracted from a feature extraction network, according to an embodiment of the disclosure.

Referring toFIG.7, according to an embodiment of the disclosure, the feature extraction network300may extract feature information about the first image25by using at least one feature extraction block. In this case, the feature information extracted from the feature extraction network300may include at least one of edge information, shadow region information, luminance information, transform noise information, or texture information about the first image25.

Moreover, referring toFIG.5, feature information F extracted from the feature extraction network300may be transmitted to the image quality processing network400. For example, according to an embodiment of the disclosure, feature information generated through the first feature extraction block310and the second feature extraction block320may be transmitted to a first modulation block410and a second modulation block420, which are included in the image quality processing network400.

According to an embodiment of the disclosure, the image processing apparatus100may improve the image quality while maintaining edge information and texture information about the first image25, by using feature information about the first image25in the image quality processing network400, the feature information being extracted from the feature extraction network300.

Referring toFIG.5, according to an embodiment of the disclosure, the image quality processing network400may include at least one modulation block. Although it has been illustrated inFIG.5for convenience of description, that the image quality processing network400includes the first modulation block410and the second modulation block420, the disclosure is not limited thereto. The image quality processing network400may include a larger number of modulation blocks.

InFIG.5, it has been illustrated that the first image25is directly input into the first modulation block410, but the disclosure is not limited thereto. For example, a convolutional layer and an activation layer are positioned before the first modulation block410, and accordingly, an image obtained by performing a certain operation on the first image25may be input into the first modulation block410.

Hereinafter, the structure and operations of the first modulation block410, according to an embodiment of the disclosure, will be described with reference toFIG.8.

FIG.8is a diagram of a structure of a first modulation block, according to an embodiment of the disclosure.

Referring toFIG.8, according to an embodiment of the disclosure, the first modulation block410may include a plurality of convolutional layers and may include a modulation layer consecutively positioned in each of the plurality of convolutional layers. Also, the first modulation block410may include an activation layer consecutively positioned in the modulation layer. For example, a first convolutional layer411, a first modulation layer412, and a first activation layer413may be sequentially positioned, a second convolutional layer421, a second modulation layer422, and a second activation layer423may be sequentially positioned, and a third convolutional layer431, a third modulation layer432, and a third activation layer433may be sequentially positioned. Also, an output of the first activation layer413may be connected to an input of the second convolutional layer421, and an output of the second activation layer423may be connected to an input of the third convolutional layer431. However, the disclosure is not limited thereto.

In the first convolutional layer411, a first feature map may be generated through a convolutional operation between the first image25input into the first convolutional layer411and a first kernel included in the first convolutional layer411. The first feature map output from the first convolutional layer411may be input into the first modulation layer412.

Moreover, according to an embodiment of the disclosure, the first modulation block410may further include a weight map generator corresponding to each of a plurality of modulation layers.

For example, the first modulation block410may include a first weight map generator461corresponding to the first modulation layer412, a second weight map generator462corresponding to the second modulation layer422, and a third weight map generator463corresponding to the third modulation layer432.

According to an embodiment of the disclosure, each of the first to third weight map generators461,462, and463may generate a plurality of weight maps based on feature information F extracted from the feature extraction network300.

Operations of the weight map generators will be described in detail with reference toFIG.9.

FIG.9is a diagram of a structure of a first weight map generator, according to an embodiment of the disclosure.

Referring toFIG.9, the first weight map generator461may include at least one weight map generation block, and a weight map generation block may include a convolutional layer and an activation layer. For example, the first weight map generator461may include first to n-th weight map generation blocks610,620, . . . ,690, and the first to n-th weight map generation block610,620, . . . ,690may be connected in parallel. Also, the first weight map generation block610may generate a first weight map Fa based on the feature information F extracted from the feature extraction network300.

Referring toFIG.9, the first weight map generation block610may include a first convolutional layer611, an activation layer612, and a second convolutional layer613, which are sequentially positioned. However, the disclosure is not limited thereto. In the first convolutional layer611, a first feature map may be generated through a convolutional operation between the feature information F input into the first convolutional layer611and a first kernel included in the first convolutional layer611. The first feature map may be input into the activation layer612, and an activation function operation of applying the activation function to the first feature map may be performed in the activation layer612. A feature map output from the activation layer612may be input into the second convolutional layer613, and in the second convolutional layer613, the first weight map Fa may be generated through a convolutional operation between the feature map input into the second convolutional layer613and a second kernel included in the second convolutional layer613.

In the same manner as the first weight map generation block610, the second weight map generation block620may generate a second weight map Fb based on the feature information F, and the n-th weight map generation block690may generate an nth weight map Fn based on the feature information F.

Moreover, first to n-th weight maps Fa, Fb, Fn generated by the first weight map generator461may be input into the first modulation layer412ofFIG.8.

Referring back toFIG.8, in the first modulation layer412, a plurality of weight maps (e.g., the first to n-th weight maps Fa, Fb, Fn) received from the first weight map generator461may be applied to the first feature map output from the first convolutional layer411.

An operation of performing modulation by applying the plurality of weight maps to the first feature map in the first modulation layer412will be described in detail with reference toFIG.10.

FIG.10is a diagram for describing a first modulation layer according to an embodiment of the disclosure.

Referring toFIG.10, according to an embodiment of the disclosure, the first modulation layer412may receive the first feature map F1output from the first convolutional layer411and the first to n-th weighted values Fa, Fb, Fn output from the first weight map generator461. In the first modulation layer412, a second feature map F2may be generated by performing various operations based on the first feature map F1and the plurality of weight maps (e.g., the first to n-th weight maps Fa, Fb,

Fn).

For example, as illustrated inFIG.10, when the plurality of weight maps output from the first weight map generator461include the first weight map Fa and the second weight map Fb, in the first modulation layer412, the second feature map F2may be generated by performing a first operation between the first feature map F1and the first weight map Fa and performing a second operation between a value F1′ obtained by performing the first operation and the second weight map Fb.

In this case, the first operation and the second operation may be elementwise multiplication operations or elementwise summation operations, and sizes of the first feature map F1, the first weight map Fa, and the second weight map Fb need to be the same in order to perform an elementwise operation. The elementwise operation means that when each of values included in the first feature map F1and each of values included in a weight map are calculated, values in the same position are calculated.

For example, in the first modulation layer412, an elementwise multiplication operation of the first feature map F1and the first weight map Fa may be performed, and an elementwise summation operation of the second weight map Fb and a value F1′ obtained by performing the elementwise multiplication operation may be performed. Alternatively, the first operation may be an elementwise summation operation and the second operation may be an elementwise multiplication operation, but are not limited thereto.

Moreover, according to an embodiment of the disclosure, various operations may be applied in the first modulation layer412.

For example, as in any one of Equations 1 to 3 provided for illustrative purposes below, in the first modulation layer412, the second feature map F2may be generated by modulating the first feature map F1based on the plurality of weight maps (e.g., the first to n-th weight maps Fa, Fb, Fn).
F2=Fa×F1n+Fb×F1n-1+ . . . +Fn[Equation 1]
F2=Fa×log(F1n)+Fb×log(F1n-1)+ . . . +Fn[Equation 2]
F2=exp(Fa×F1)+exp(Fb×F1)+ . . . +exp(Fn×F1)  [Equation 3]

Referring back toFIGS.8and10, the second feature map F2output from the first modulation layer412may be input into the first activation layer413, and an activation function operation of applying the activation function to the second feature map may be performed in the first activation layer413.

Also, a feature map output from the first activation layer413may be input into the second convolutional layer421.

In addition, the first image25may be input into the second convolutional layer421. In this case, the feature map output from the first activation layer413and the first image25may be concatenated and input into the second convolutional layer421. For example, when the feature map output from the first activation layer413has 16 channels and the first image25has 16 channels, 32 channels may be input into the second convolutional layer421.

In the second convolutional layer421, a third feature map may be generated through a convolutional operation between the feature map input into the second convolutional layer421and a second kernel included in the second convolutional layer421. In this case, the number of channels of the output third feature map may be adjusted by adjusting the number of sub-kernels included in the second kernel. For example, the number of sub-kernels included in the second kernel is set to 16, and a convolutional operation between the second kernel and the feature map (including 32 channels) input into the second convolutional layer421is performed so that the third feature map including 16 channels may be generated.

Also, the third feature map may be input into the second modulation layer422, and a plurality of weight maps generated by the second weight map generator462may be input into the second modulation layer422. The second weight map generator462may generate the plurality of weight maps in the same manner as the first weight map generator461described with reference toFIG.9.

In the second modulation layer422, a fourth feature map may be generated by applying the plurality of weight maps to the third feature map and performing modulation.

In the second modulation layer422, the fourth feature map may be generated by modulating the third feature map in the same manner as the first modulation layer412described with reference toFIG.10.

The fourth feature map may be input into the second activation layer423, and an activation function operation of applying the activation function to the fourth feature map may be performed in the second activation layer423.

Also, a feature map output from the second activation layer423may be input into the third convolutional layer431. Also, the first image25and a feature map output from the first activation layer413may also be input into the third convolutional layer431. In this case, the feature map output from the second activation layer423, the first image25, and the feature map output from the first activation layer413may be concatenated and input into the third convolutional layer431. For example, when the feature map output from the second activation layer423has 16 channels, the first image25has 16 channels, and the feature map output from the first activation layer413has 16 channels, 48 channels may be input into the third convolutional layer431.

In the third convolutional layer431, a fifth feature map may be generated through a convolutional operation between the feature map input into the third convolutional layer431and a third kernel included in the third convolutional layer431. In this case, the number of channels of the output fifth feature map may be adjusted by adjusting the number of sub-kernels included in the third kernel. For example, the number of sub-kernels included in the third kernel is set to 16, and a convolutional operation between the third kernel and the feature map (including 48 channels) input into the third convolutional layer431is performed so that the fifth feature map including 16 channels may be generated.

The fifth feature map may be input into the third modulation layer432, and a plurality of weight maps generated by the third weight map generator463may be input into the third modulation layer432. The third weight map generator463may generate the plurality of weight maps in the same manner as the first weight map generator461described with reference toFIG.9.

In the third modulation layer432, a sixth feature map may be generated by applying the plurality of weight maps to the fifth feature map and performing modulation.

In the third modulation layer432, the sixth feature map may be generated by modulating the fifth feature map in the same manner as the first modulation layer412described with reference toFIG.10.

The sixth feature map may be input into the third activation layer433, and an activation function operation of applying the activation function to the sixth feature map may be performed in the third activation layer433.

Also, a feature map output from the third activation layer433may be input into a fourth convolutional layer440. Also, the first image25, the feature map output from the first activation layer413, and the feature map output from the second activation layer423may also be input into the fourth convolutional layer440. In this case, the feature map output from the third activation layer433, the first image25, the feature map output from the first activation layer413, and the feature map output from the second activation layer423may be concatenated and input into the fourth convolutional layer440.

For example, when the feature map output from the third activation layer433has 16 channels, and each of the first image25, the feature map output from the first activation layer413, and the feature map output from the second activation layer423has 16 channels, 64 channels may be input into the fourth convolutional layer440.

In the fourth convolutional layer440, a seventh feature map may be generated through a convolutional operation between the feature map input into the fourth convolutional layer440and a fourth kernel included in the fourth convolutional layer440. In this case, the number of channels of the output seventh feature map may be adjusted by adjusting the number of sub-kernels included in the fourth kernel. For example, the number of sub-kernels included in the fourth kernel is set to 16, and a convolutional operation between the fourth kernel and the feature map (including 64 channels) input into the fourth convolutional layer440is performed so that the seventh feature map including 16 channels may be generated.

Also, the first modulation block410may further include a feature scaler450that adjusts sizes of values included in the seventh feature map. The feature scaler450may perform an operation of multiplying each of the values included in the seventh feature map by a preset constant value, in order to adjust the sizes of the values included in the seventh feature map while maintaining feature information about the seventh feature map.

Also, the first modulation block410may include a skip connection470that skips a plurality of convolutional layers. The skip connection470may be a structure that connects an input (e.g., the first image25) of the first convolutional layer411to an output of the feature scaler450. For example, because the first modulation block410includes the skip connection470, an output image40may be generated by adding the first image25to a feature map output as a result of passing the first image25through the first to fourth convolutional layers411,421,431, and440. Accordingly, the first modulation block410may refer to a network that has learned a residual between the first image25and the output image40.

Also, the first modulation block410may efficiently restore edge information included in the first image25with a small number of layers by using channel concatenations and skip connections.

Moreover, although only the first modulation block410has been illustrated and described inFIG.8, other modulation blocks (e.g., second to n-th modulation blocks) included in the image quality processing network400may also have the same structure and perform the same or similar operations as those of the first modulation block410.

Referring back toFIG.3, according to an embodiment of the disclosure, the second image output from the image quality processing network400may be input into the upscale network500.

Referring toFIG.4, according to an embodiment of the disclosure, the upscale network500may include at least one upscale block and at least one spade block.

Although it has been illustrated inFIG.4for convenience of description, that the upscale network500includes a first upscale block510, a first spade block520, a second upscale block530, and a second spade block540, the upscale network500may include one upscale block and one spade block or may include at least three upscale blocks and at least three spade blocks.

Also, the first upscale block510, the first spade block520, the second upscale block530, and the second spade block540may be sequentially connected, and a convolutional layer501and an activation layer502may be further positioned after the second spade block540.

According to an embodiment of the disclosure, the first upscale block510may include at least one convolutional layer511and at least one activation layer512. Also, the first upscale block510may also include at least one depth-to-space layer513. For example, as illustrated inFIG.4, the first upscale block510may be configured in a manner in which a convolutional layer511, a depth-to-space layer513, and an activation layer512are sequentially positioned, so that data output from the convolutional layer511is input into the depth-to-space layer513and data output from the depth-to-space layer513is input into the activation layer512. However, the disclosure is not limited thereto. In the depth-to-space layer513, an operation of increasing a resolution of input data and reducing the number of channels as much as the resolution is increased may be performed. The depth-to-space layer513may be a layer that changes channel information into resolution information.

According to an embodiment of the disclosure, the first upscale block510may generate a first upscaled image by upscaling an image input into the first upscale block510by using the convolutional layer511, the depth-to-space layer513, and the activation layer512, which are included in the first upscale block510.

In this regard, the first upscaled image may be an image obtained by increasing a resolution of the image (the second image) input into the first upscale block510by k2 times. In this case, a value of k2 may be a value corresponding to the amount by which the second downscale block220downscales the resolution. For example, the value of k2 may be 2, and a resolution of the first upscaled image may be twice a resolution of the second image. However, the disclosure is not limited thereto.

Also, the value of k2 may be variously determined according to the configuration of layers included in the first upscale block510. For example, the value of k2 may vary according to a kernel of each of the convolutional layers included in the first upscale block510.

The first upscaled image output from the first upscale block510may be input into the first spade block520.

An operation of the first spade block520will be described in detail with reference toFIG.11.

FIG.11is a diagram of a structure of a first spade block, according to an embodiment of the disclosure.

Referring toFIG.11, the first spade block520may receive a first upscaled image801and a first downscaled image802input into the second downscale block220. The first upscaled image801input into the first spade block520may be an image output from the first upscale block510ofFIG.4. Also, the first downscaled image802input into the first spade block520may be an image output from the first downscale block210ofFIG.4. In this case, a resolution of the first upscaled image801and a resolution of the first downscaled image802may be ½ of a resolution of an input image20ofFIG.3. The first spade block520may process the first upscaled image801based on the first downscaled image802.

The first spade block520may include at least one spade layer and an activation layer consecutively positioned in each spade layer. Also, the first spade block520may include a convolutional layer consecutively positioned in the activation layer. For example, a first spade layer521, a first activation layer522, and a first convolutional layer523may be sequentially positioned, and a second spade layer524, a second activation layer525, and a second convolutional layer526may be sequentially positioned. Also, an output of the first convolutional layer523may be connected to an input of the second spade layer524. However, the disclosure is not limited thereto.

The first downscaled image802may be input into each of the spade layers included in the first spade block520. For example, the first spade layer521and the second spade layer524may receive the first downscaled image802from the downscale network200.

According to an embodiment of the disclosure, the first spade layer521may process the first upscaled image801based on the first downscaled image802.

An operation in which the first spade layer521processes the first upscaled image801based on the first downscaled image802will be described in detail with reference toFIG.12.

FIG.12is a diagram for describing an operation of a first spade layer, according to an embodiment of the disclosure.

Referring toFIG.12, according to an embodiment of the disclosure, the first spade layer521may generate a plurality of pieces of parameter information based on the first downscaled image802. The first spade layer521may include a first convolutional layer911and an activation layer912.

The first downscaled image802may be input into the first convolutional layer911, and first feature information may be generated through a convolutional operation in the first convolutional layer911. The first feature information may be input into the activation layer912, and an activation function operation of applying the activation function to the first feature information may be performed in the activation layer912. Second feature information output from the activation layer912may be input into each of a plurality of convolutional layers. For example, the second feature information may be input into a second convolutional layer921, and first parameter information P1may be generated through a convolutional operation in the second convolutional layer921. Also, the second feature information may be input into a third convolutional layer922, and second parameter information P2may be generated through a convolutional operation in the third convolutional layer922. However, the disclosure is not limited thereto, and at least three pieces of parameter information may be generated.

In the first spade layer521, image quality processing may be performed on the first upscaled image801by applying a plurality of pieces of parameter information to the first upscaled image801.

In the first spade layer521, a third image940may be generated by performing various operations based on the first upscaled image801and the plurality of pieces of parameter information.

For example, in the first spade layer521, image quality processing may be performed on the first upscaled image801based on the first parameter information P1and the second parameter information P2. In the first spade layer521, the third image940may be generated by performing a first operation between the first upscaled image801and the first parameter information P1and performing a second operation between a value930obtained by performing the first operation and the second parameter information P2.

In this case, the first operation and the second operation may be elementwise multiplication operations or elementwise summation operations.

For example, in the first spade layer521, an elementwise multiplication operation of the first upscaled image801and the first parameter information P1may be performed, and an elementwise summation operation of the second parameter information P2and values930obtained by performing the elementwise multiplication operation may be performed. Alternatively, the first operation may be an elementwise summation operation and the second operation may be an elementwise multiplication operation, but are not limited thereto. Moreover, the first operation and the second operation may include various operations.

Referring back toFIG.11, the third image940output from the first spade layer521may be input into the first activation layer522, and an activation function operation of applying the activation function to the third image940may be performed in the first activation layer522.

A fourth image output from the first activation layer522may be input into the first convolutional layer523. In the first convolutional layer523, a fifth image may be generated through a convolutional operation between the fourth image and a kernel included in the first convolutional layer523.

The fifth image may be input into the second spade layer524, and a sixth image may be generated in the second spade layer524by applying the plurality of pieces of parameter information to the fifth image based on the first downscaled image802.

The sixth image output from the second spade layer524may be input into the second activation layer525, and an activation function operation of applying the activation function to the sixth image may be performed in the second activation layer525.

A seventh image output from the second activation layer525may be input into the second convolutional layer526. An eighth image803may be generated in the second convolutional layer526through a convolutional operation between the seventh image and a kernel included in the second convolutional layer526.

Referring back toFIG.4, the eighth image803output from the first spade block520may be input into the second upscale block530. The second upscale block530may have the same or similar structure and perform the same or similar operations as those of the first upscale block510.

Accordingly, the second upscale block530may generate a second upscaled image by upscaling the eighth image803. A resolution of the second upscaled image may be twice a resolution of the first upscaled image or the eighth image803, but is not limited thereto.

According to an embodiment of the disclosure, the second upscaled image may be input into the second spade block540. Also, the second spade block540may receive an image input into the first downscale block210ofFIG.4. In this case, a resolution of an image equally input into the first downscale block210and the second spade block540may be the same as the resolution of the input image20. Also, the resolution of the second upscaled image output from the second upscale block530may also be the same as the resolution of the input image20. The second spade block540may have the same or similar structure and perform the same or similar operations to those of the first spade block520.

According to an embodiment of the disclosure, the upscale network500may further include the first convolutional layer501, the activation layer502, and a second convolutional layer503after the second spade block540.

A convolutional operation and an activation operation are performed on a ninth image output from the second spade block540in each of the first convolutional layer501, the activation layer502, and the second convolutional layer503, and accordingly, the output image40ofFIG.3may be obtained.

Accordingly, according to an embodiment of the disclosure, the image processing apparatus100may obtain the output image40in which an image quality of the input image20is improved by using the image processing network50described with reference toFIGS.3to12.

Also, according to an embodiment of the disclosure, the old photo restoration model, the face restoration model, and the background restoration model may include the image processing network50described with reference toFIGS.3to12.

Moreover, according to an embodiment of the disclosure, at least one of the first and second downscale blocks210and220, the first and second feature extraction blocks310and320, the first and second modulation blocks410and420, the first and second upscale blocks510and530, or the first and second spade blocks520and540may be manufactured in the form of a hardware chip and mounted in the image processing apparatus100. For example, according to an embodiment of the disclosure, at least one of downscale blocks, feature extraction blocks, upscale blocks, or spade blocks may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor) or a dedicated graphics processor (e.g., a graphics processing unit (GPU)) and mounted on the image processing apparatus100.

Also, at least one of the downscale blocks, the feature extraction blocks, the modulation blocks, the upscale blocks, or the spade blocks may be implemented as a software module. When at least one of the downscale blocks, the feature extraction blocks, the modulation blocks, the upscale blocks, or the spade blocks is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or a certain application. Alternatively, a part of at least one software module may be provided by an OS, and the remaining part may be provided by a certain application.

The structure of the image processing network50illustrated and described with reference toFIGS.3to12is provided for illustration of example embodiments of the disclosure. Each component of the image processing network50may be integrated, added, or omitted according to the specifications of the image processing apparatus100that is actually implemented. For example, two or more components may be combined into one component, or one component may be subdivided into two or more components as needed. In addition, the function performed in each block is for describing the embodiments of the disclosure, and the specific operation or apparatus does not limit the spirit and scope of the disclosure.

FIG.13is a diagram for describing an operation of determining weights of a downscale network and an upscale network included in a face restoration model, according to an embodiment of the disclosure.

According to an embodiment of the disclosure, the face restoration model may include the image processing network50described with reference toFIGS.3to12and may determine weight information (Wencoder, Wdecoder) about a downscale network and an upscale network included in the image processing network50, according to a size or a degree of deterioration of a face region included in an input image.

For example, the image processing apparatus100according to an embodiment may receive, from an external device, a plurality of pieces of weight information (Wencoder, Wdecoder) corresponding to the downscale network and the upscale network. The image processing apparatus100may store the plurality of pieces of received weight information (Wencoder, Wdecoder) and may determine, from among the plurality of pieces of stored weight information (Wencoder, Wdecoder), weight information (Wencoder, Wdecoder) about the downscale network and the upscale network included in the image processing network50, according to the size or the degree of deterioration of the face region included in the input image.

The external device may determine a plurality of pieces of weight information corresponding to the downscale network and the upscale network, by training the image processing network50(e.g., the face restoration model) by using training data sets including face images having a preset size. In this case, weights of the main network30included in the image processing network50may be fixed to preset values.

For example, the external device may obtain first weight information1310(Wencoder1, Wdecoder1) about the downscale network and the upscale network corresponding to a first size, by training the image processing network50by using first training data sets including face images having the first size. Also, the external device may obtain second weight information1320(Wencoder2, Wdecoder2) about the downscale network and the upscale network corresponding to a second size, by training the image processing network50by using second training data sets including face images having the second size.

In addition, the external device may determine the plurality of pieces of weight information corresponding to the downscale network and the upscale network, by training the face restoration model by using training data sets including face images having a preset degree of deterioration. In this case, weights of a main network included in the face restoration model may be fixed to preset values.

For example, the external device may obtain third weight information about the downscale network and the upscale network corresponding to a first degree of deterioration, by training the face restoration model by using training data sets including face images having the first degree of deterioration. Also, the external device may obtain fourth weight information about the downscale network and the upscale network corresponding to a second degree of deterioration, by training the face restoration model by using training data sets including face images having the second degree of deterioration.

Accordingly, the external device may obtain n pieces of weight information W1, W2, . . . , Wn corresponding to the downscale network and the upscale network according to a size of a face image or a degree of deterioration of the face image, and according to an embodiment of the disclosure, an electronic device may receive the n pieces of weight information W1, W2, . . . , Wn from the external device.

According to an embodiment of the disclosure, a face region restorer (e.g., the first face region restorer123or the second face region restorer132) may determine a weight of the downscale network and the upscale network of the image processing network50included in the face restoration model, based on the plurality of pieces of weight information, the size of the face region included in the input image, and the degree of deterioration of the face region.

For example, a weight W of the downscale network and the upscale network may be determined by using the following equation.

W=F⁡(W⁢1,W⁢2,…,Wn)=∑i=1nai×Wi[Equation⁢4]

In Equation 4, W denotes the weight of the downscale network and the upscale network included in the face restoration model, and ai denotes a coefficient corresponding to each of the plurality of pieces of weight information determined based on the size and degree of deterioration of the face region included in the input image. A method of determining the coefficient corresponding to each of the plurality of pieces of weight information will be described in detail with reference toFIG.14.

FIG.14is a diagram of coefficient information corresponding to a plurality of pieces of weight information about a downscale network and an upscale network, according to an embodiment of the disclosure.

Referring toFIG.14, according to an embodiment of the disclosure, the image processing network included in the face restoration model may be trained by using face images having a first size (e.g., 64×64) as training data, wherein a weight of the downscale network and the upscale network may be determined as a first weight (Wsize(64×64)). Also, the image processing network may be trained by using face images having a second size (e.g., 128×128) as training data, wherein a weight of the downscale network and the upscale network may be determined as a second weight (Wsize(128×128)). In addition, the image processing network may be trained by using face images having a third size (e.g., 192×192) as training data, wherein a weight of the downscale network and the upscale network may be determined as a third weight (Wsize(192×192)).

Also, coefficient information corresponding to the first weight, the second weight, and the third weight according to the size of the face region may be shown as a first graph1410, a second graph1420, and a third graph1430.

According to an embodiment of the disclosure, when the face region included in the input image has a fourth size (e.g., 96×96), a face restorer (e.g., the first face region restorer123or the second face region restorer132) may calculate a fourth weight (Wsize(96×96)) corresponding to the fourth size based on coefficient information about the first weight, the second weight, and the third weight, corresponding to the fourth size, the first weight, the second weight, and the third weight. For example, the fourth weight (Wsize(96×96)) may be determined as 0.5×Wsize(64×64)+0.5×Wsize(128×128)+0×Wsize(192×192), as illustrated inFIG.14.

Also, according to an embodiment of the disclosure, the image processing network included in the face restoration model is trained by using face images having a preset degree of deterioration as training data, and accordingly, a plurality of pieces of weight information about the downscale network and the upscale network according to a degree of deterioration may be determined.

According to an embodiment of the disclosure, the face restorer may calculate a weight according to the degree of deterioration of the face region, in the same manner as the method of calculating the weight according to the size of the face region. Also, the face restorer may obtain a final weight based on a weight (Wsize) according to the size of the face region and a weight (Wblur) according to the degree of deterioration. The final weight of the downscale network and the upscale network according to the size and degree of deterioration of the face region may be expressed by Equation 5 below.
W=a×Wsize+(1−a)×Wblur[Equation 5]

In Equation 5, an addition coefficient a may be determined by visual information fidelity (VIF) of the face region, but is not limited thereto. The addition coefficient a may have a value of 0 or more and 1 or less, and may have a value close to 1 as the degree of deterioration of the face region decreases.

FIG.15is a flowchart of an operating method of an image processing apparatus, according to an embodiment of the disclosure,FIG.16is a flowchart of operations included in operation1510(S1510) ofFIG.15, andFIG.17is a flowchart of operations included in operation1540(S1540) ofFIG.15.

Referring toFIG.15, according to an embodiment of the disclosure, the image processing apparatus100may obtain a first image by downscaling an input image by using a downscale network (S1510). Operation1510(S1510) will be described in detail with reference toFIG.16.

FIG.16is a flowchart of operations included in operation1510(S1510) ofFIG.15.

According to an embodiment of the disclosure, the downscale network may include at least one downscale block. For example, the downscale network may include a first downscale block and a second downscale block, but is not limited thereto.

Referring toFIG.16, according to an embodiment of the disclosure, the image processing apparatus100may obtain a first downscaled image by downscaling the input image by using the first downscale block (S1610).

For example, the image processing apparatus100may generate the first downscaled image obtained by downscaling the input image, by passing the input image through a convolutional layer, a space-to-depth layer, and an activation layer, which are included in the first downscale block. In this case, the first downscaled image may be an image obtained by reducing a resolution of the input image to 1/k1.

The first downscaled image output from the first downscale block may be input into the second downscale block.

According to an embodiment of the disclosure, the image processing apparatus100may obtain a second downscaled image by downscaling the first downscaled image by using the second downscale block (S1620).

For example, the second downscale block may have the same or similar structure to that of the first downscale block, and the second downscaled image may be an image obtained by reducing a resolution of the first downscaled image to 1/k2.

The image processing apparatus100may obtain a first image based on the second downscaled image (S1630) and input the first image to an image quality processing network and a feature extraction network.

Referring back toFIG.15, according to an embodiment of the disclosure, the image processing apparatus100may extract first feature information corresponding to the first image by using the feature extraction network (S1520).

For example, the feature extraction network may include at least one feature extraction block, and a feature extraction block may include a plurality of convolutional layers and an activation layer consecutively positioned in each of the plurality of convolutional layers. Also, the feature extraction block may include a skip connection that skips at least one convolutional layer. Also, the feature extraction block may further include a normalization layer and a residual scaler.

The image processing apparatus100may extract feature information by performing a convolutional operation, an activation operation, and normalization on the first image. Because this has been described in detail with reference toFIG.6, the same description will not be provided herein.

Also, according to an embodiment of the disclosure, the extracted feature information may include at least one of edge information, shadow region information, luminance information, transform noise information, or texture information about the first image.

According to an embodiment of the disclosure, the image processing apparatus100may obtain a second image, by performing image processing on the first image by using the image quality processing network.

According to an embodiment of the disclosure, the image quality processing network may include a plurality of modulation blocks, and a modulation block may include a plurality of convolutional layers and a modulation layer consecutively positioned in each of the plurality of convolutional layers. Also, according to an embodiment of the disclosure, the modulation block may include an activation layer consecutively positioned in the modulation layer. In addition, according to an embodiment of the disclosure, the modulation block may include a weight map generator corresponding to the modulation layer. Because the structure of the modulation block has been described in detail with reference toFIG.8, a detailed description thereof will not be provided herein.

The image processing apparatus100may generate a second feature map by applying a plurality of weight maps generated by the weight map generator to a first feature map obtained from the convolutional layer included in the modulation block. Also, the image processing apparatus100may generate the second image based on the second feature map. Because this has been described in detail with reference toFIGS.8to10, a detailed description thereof will not be provided herein.

According to an embodiment of the disclosure, the image processing apparatus100may obtain an output image by upscaling the second image and performing image quality processing on the upscaled second image, by using the upscale network (S1540). Operation1540(S1540) will be described in detail with reference toFIG.17.

FIG.17is a flowchart of operations included in operation1540(S1540) ofFIG.15.

According to an embodiment of the disclosure, the upscale network may include at least one upscale block and at least one spade block. For example, the upscale network may include a first upscale block, a first spade block, a second upscale block, and a second spade block, but is not limited thereto.

Referring toFIG.17, according to an embodiment of the disclosure, the image processing apparatus100may obtain a first upscaled image by upscaling the second image by using the first upscale block (S1710).

For example, the image processing apparatus100may obtain the first upscaled image upscaled by passing the second image through a convolutional layer, a depth-to-space layer, and an activation layer, which are included in the first upscale block. In this case, the first upscaled image may be an image obtained by increasing a resolution of the second image by k2 times.

The first upscaled image output from the first upscale block may be input into the first spade block.

According to an embodiment of the disclosure, the image processing apparatus100may obtain a third image by extracting first feature information corresponding to the first downscaled image and performing image quality processing on the first upscaled image based on the extracted first feature information, by using the first spade block (S1720).

According to an embodiment of the disclosure, the first spade block may include at least one spade layer and an activation layer consecutively positioned in each spade layer. Also, the first spade block may include a convolutional layer consecutively positioned in the activation layer. According to an embodiment of the disclosure, the image processing apparatus100may generate a plurality of pieces of parameter information in the spade layer based on the first downscaled image, and may obtain the third image by performing image quality processing on the first upscaled image based on the plurality of pieces of parameter information. Because this has been described in detail with reference toFIG.12, a detailed description thereof will not be provided herein.

According to an embodiment of the disclosure, the image processing apparatus100may obtain a second upscaled image obtained by upscaling the third image by using the second upscale block (S1730).

For example, the second upscale block may have the same or similar structure to that of the first upscale block, and the second upscaled image may be an image obtained by increasing a resolution of the first upscaled image by k1 times.

The second upscaled image output from the second upscale block may be input into the second spade block.

According to an embodiment of the disclosure, the image processing apparatus100may obtain an output image by extracting second feature information corresponding to the input image and performing image quality processing on the second upscaled image based on the extracted second feature information, by using the second spade block (S1740). For example, the second spade block may have the same or similar structure to that of the first spade block and may perform image quality processing on the second upscaled image by performing the same or similar operations to those of the first spade block. Also, the image processing apparatus100may obtain the output image based on the second upscaled image on which image quality processing has been performed.

FIG.18is a block diagram of a configuration of an image processing apparatus, according to an embodiment of the disclosure.

Referring toFIG.18, according to an embodiment of the disclosure, the image processing apparatus100may include a processor140and a memory150.

According to an embodiment of the disclosure, the processor140may control the image processing apparatus100overall. According to an embodiment of the disclosure, the processor140may execute at least one program stored in the memory150.

According to an embodiment of the disclosure, the memory150may store various data, programs, or applications for driving and controlling the image processing apparatus100. A program stored in the memory150may include one or more instructions. The program (one or more instructions) or an application stored in the memory150may be executed by the processor140.

According to an embodiment of the disclosure, the processor140may include at least one of a central processing unit (CPU), a graphic processing unit (GPU), or a video processing unit (VPU). Alternatively, according to an embodiment of the disclosure, the processor140may be implemented in the form of a system on chip (SoC) in which at least one of a CPU, a GPU, or a VPU is integrated. Alternatively, the processor140may further include a neural processing unit (NPU).

According to an embodiment of the disclosure, the processor140may detect whether an input image is an old photo by using the classification network115ofFIG.2.

Also, the processor140may perform image quality processing on the input image by using the image processing network50illustrated and described with reference toFIGS.3to12. For example, the processor140may remove compression artifacts included in the input image by using the image processing network50. Also, the processor140may perform a process of removing noise while maintaining detailed texture information and structural information such as edges of the input image.

Moreover, according to an embodiment of the disclosure, the image processing network50may be a network trained by a server or an external device. The external device may obtain weight information by training the image processing network50based on training data.

According to an embodiment of the disclosure, the processor140may receive the image processing network50, which has been trained, from the server or the external device, and store the image processing network50in the memory150. For example, according to an embodiment of the disclosure, the memory150may store structure and parameter values (weight information) of the image processing network50, and according to an embodiment of the disclosure, the processor140may generate an output image, in which edge characteristics or texture characteristics are maintained while noise is removed from the input image, by using the parameter values stored in the memory150.

Also, according to an embodiment of the disclosure, weight information corresponding to a downscale network and an upscale network included in the image processing network50may include a plurality of pieces of weight information according to a size or a degree of deterioration of a face image. The external device may determine a plurality of pieces of weight information corresponding to the downscale network and the upscale network, by training the image processing network50included in the face restoration model by using training data sets including face images having a preset size. In this case, weights of the main network30included in the image processing network50may be fixed to preset values.

Also, the external device may determine a plurality of pieces of weight information corresponding to the downscale network and the upscale network, by training the image processing network50included in the face restoration model by using training data sets including face images having a preset degree of deterioration. In this case, weights of the main network30included in the image processing network50may be fixed to preset values.

According to an embodiment of the disclosure, the processor140may determine the weights of the downscale network and the upscale network of the image processing network50included in the face restoration model, based on the plurality of pieces of weight information, a size of a face region included in the input image, and a degree of deterioration of the face region.

FIG.19is a block diagram of a configuration of an image processing apparatus, according to another embodiment of the disclosure. An image processing apparatus1900ofFIG.19may be another embodiment of the disclosure of the image processing apparatus100described with reference toFIG.1.

Referring toFIG.19, according to an embodiment of the disclosure, the image processing apparatus1900may include a sensing unit1910, a communicator (or communication interface)1920, a processor1930, an audio/video (NV) input unit1940, an output unit (or an output interface)1950, a memory1960, and a user input unit (or an user input interface)1970.

The processor1930ofFIG.19may correspond to the processor140ofFIG.18and the memory1960ofFIG.19may correspond to the memory150ofFIG.18, and thus, the same description will not be provided herein.

The sensing unit1910may include a sensor that detects a state of the image processing apparatus1900or a state around the image processing apparatus1900. Also, the sensing unit1910may transmit information detected by the sensor to the processor1930.

The communicator1920may include a short-range wireless communicator, a mobile communicator, etc. in response to the performance and structure of the image processing apparatus1900, but is not limited thereto.

The short-range wireless communicator may include a Bluetooth low energy (BLE) communicator, a near-field communicator, a wireless local area network (WLAN) (Wi-Fi) communicator, a Zigbee communicator, an infrared data association (IrDA) communicator, a Wi-Fi direct (WFD) communicator, an ultra-wideband (UWB) communicator, an Ant+ communicator, a microwave (uWave) communicator, etc., but is not limited thereto.

The mobile communicator may transceive wireless signals with at least one of a base station, an external terminal, or a server, through a mobile communication network. Here, the wireless signals may include a sound call signal, a video call signal, or various types of data according to transceiving of text/multimedia messages.

According to an embodiment of the disclosure, the communicator1920may receive a compressed image from an external device or transmit the compressed image.

According to an embodiment of the disclosure, the processor1930may include a single core, a dual core, a triple core, a quad core, and multiple cores thereof. Also, the processor1930may include a plurality of processors.

According to an embodiment of the disclosure, the memory1960may include at least one type of storage medium from among a flash memory-type storage medium, a hard disk-type storage medium, a multimedia card micro-type storage medium, a card-type memory (for example, a secure digital (SD) or extreme digital (XD) memory), a random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable-programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.

The NV input unit1940is a component for inputting an audio signal or a video signal and may include a camera1941and a microphone1942. The camera1941may obtain an image frame such as a still or moving image via an image sensor in a video call mode or capture mode. An image captured via the image sensor may be processed by the processor1930or a separate image processor (not illustrated).

The image frame processed by the camera1941may be stored in the memory1960or transmitted to the outside through the communicator1920. The camera1941may include two or more cameras according to the configuration of the image processing apparatus1900.

The microphone1942may receive an external sound signal and process the sound signal into electrical speech data. For example, the microphone1942may receive the sound signal from an external device or a speaker. The microphone1942may use various noise removal algorithms for removing noise occurring when the external sound signal is input.

The output unit1950is a component for outputting an audio signal, a video signal, or a vibration signal and may include a display1951, a sound output unit1952, a vibration unit1953, etc.

According to an embodiment of the disclosure, the display1951may display an image on which image quality processing has been performed by using the image processing network50.

The sound output unit1952may output audio data received from the communicator1920or stored in the memory1960. Furthermore, the sound output unit1952may also output sound signals (e.g., a call signal reception sound, a message reception sound, and a notification sound) associated with functions performed by the image processing apparatus1900. The sound output unit1952may include a speaker, a buzzer, etc.

The vibration unit1953may output a vibration signal. For example, the vibration unit1953may output a vibration signal corresponding to an output of video data or audio data (e.g., a call signal reception sound, a message reception sound, etc.). Also, the vibration unit1953may output a vibration signal when a touch is input on a touch screen.

The user input unit1970is a device via which a user inputs data necessary for controlling the image processing apparatus1900. For example, the user input unit1970may include a key pad, a dome switch, a touch pad (a touch capacitance method, a pressure-resistive layer method, an infrared sensing method, a surface ultrasonic conductive method, an integral tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, etc., but is not limited thereto.

Moreover, the block diagrams of the image processing apparatuses100and19300ofFIGS.18and19may be provided for illustration of embodiments of the disclosure. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the image processing apparatus100that is actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components as needed. In addition, the function performed in each block is for describing the embodiments of the disclosure, and the specific operation or apparatus does not limit the spirit and scope of the disclosure.

According to the embodiments of the disclosure, the operating method of the image processing apparatus may be embodied as program commands executable by various computer means and may be recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like separately or in combinations. The program commands recorded on the computer-readable recording medium may be specially designed and configured for the disclosure or may be well-known to and be usable by one of ordinary skill in the art of computer software. Examples of the computer-readable recording medium include a magnetic medium such as a hard disk, a floppy disk, or a magnetic tape, an optical medium such as a compact disc read-only memory (CD-ROM) or a digital versatile disc (DVD), a magneto-optical medium such as a floptical disk, and a hardware device such as a ROM, a RAM, or a flash memory which is specially configured to store and execute program commands. Examples of the program commands include high-level language codes that may be executed by a computer by using an interpreter or the like as well as machine codes that are generated by a compiler.

Also, the image processing apparatus and the operating method of the image processing apparatus according to the disclosed embodiments may be included in a computer program product and provided in that form. The computer program product is a product purchasable between a seller and a purchaser.

The computer program product may include an S/W program, and a computer-readable storage medium in which the S/W program is stored. For example, the computer program product may include an S/W program form of product (e.g., a downloadable application) electronically distributed through a manufacturing company of the electronic device or an electronic market (e.g., Google PlayStore™, or App Storer™). For electronic distribution, at least a portion of the SAN program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a server in the manufacturer or the electronic market or a relay server that temporarily stores the SAN program.

The computer program product may include a storage medium of a server or a storage medium of a client device in a system including the server and the client device. When there is a third device (e.g., a smartphone) communicating with the server or the client device, the computer program product may include a storage medium of the third device. Alternatively or additionally, the computer program product may include an SAN program transmitted from the server to the client device or the third device or from the third device to the client device.

In this case, one of the server, the client device, and the third device may execute a method according to disclosed embodiments of the disclosure by executing the computer program product. Alternatively, at least two of the server, the client device, and the third device may execute a method according to disclosed embodiments of the disclosure in a distributed fashion by executing the computer program product.

For example, the server (e.g., a cloud server or an artificial intelligence server) may execute the computer program product stored in the server, and may control the client device communicating with the server to perform a method according to disclosed embodiments of the disclosure.

At least one of the components, elements, modules or units (collectively “components” in this paragraph) represented by a block in the drawings may be embodied as various numbers of hardware, software and/or firmware structures that execute respective functions described above, according to an example embodiment. According to example embodiments, at least one of these components may use a direct circuit structure, such as a memory, a processor, a logic circuit, a look-up table, etc. that may execute the respective functions through controls of one or more microprocessors or other control apparatuses. Also, at least one of these components may be specifically embodied by a module, a program, or a part of code, which contains one or more executable instructions for performing specified logic functions, and executed by one or more microprocessors or other control apparatuses. Further, at least one of these components may include or may be implemented by a processor such as a central processing unit (CPU) that performs the respective functions, a microprocessor, or the like. Two or more of these components may be combined into one single component which performs all operations or functions of the combined two or more components. Also, at least part of functions of at least one of these components may be performed by another of these components. Functional aspects of the above exemplary embodiments may be implemented in algorithms that execute on one or more processors. Furthermore, the components represented by a block or processing steps may employ any number of related art techniques for electronics configuration, signal processing and/or control, data processing and the like.

Although the embodiments of the disclosure have been described in detail above, the scope of the disclosure is not limited thereto, and various modifications and improvements made by one of ordinary skill in the art by using the basic concept of the disclosure defined by the claims are also within the scope of the disclosure.