Patent Description:
Image restoration is a technique for restoring an image with a deteriorated quality into an image with an enhanced quality. For image restoration, a deep learning-based neural network may be used. A neural network may be trained based on deep learning and then perform inference for its purpose by mapping input data and output data in a nonlinear relationship. The ability to generate the above mapping may be expressed as a learning ability of the neural network. Also, the neural network trained for a specialized purpose, for example, image restoration, may have a generalization ability to generate a relatively accurate output for an input pattern that is not learned.

Known from the art is for example the network disclosed in "<NPL>. The authors developed a deep convolutional neural networks (CNNs) for moire artifacts removal by exploiting the complex properties of moire patterns in multiple complementary domains, i.e., the pixel and frequency domains. In the pixel domain, they employed multi-scale features to remove the moire artifacts associated with specific frequency bands using multi-resolution feature maps. In the frequency domain, they designed a network that processes discrete cosine transform (DCT) coefficients to remove moire artifacts. Next, they developed a dynamic filter generation network that learns dynamic blending filters. Finally, the results from the pixel and frequency domains were combined using the blending filters to yield moire-free images. In addition, they extended the proposed approach to arbitrary-length burst image demoireing. Specifically, they developed a new attention network to effectively extract useful information from each image in the burst and to align them with the reference image. They demonstrated the effectiveness of the proposed demoireing algorithm by evaluating on the test set in the NTIRE <NUM> Demoireing Challenge: Track <NUM> (Single image) and Track <NUM> (Burst).

Further technological background can be found in "<NPL>.

The present disclosure provides for image restoration methods, a computer program, a non-transatory computer-readable storage medium, and an image restoration apparatus according to the accompanying claims.

The following structural or functional descriptions of examples disclosed in the present disclosure are merely intended for the purpose of describing the examples and the examples may be implemented in various forms. The examples are not meant to be limited, but it is intended that various modifications, equivalents, and alternatives are also covered within the scope of the claims.

Although terms of "first" or "second" are used to explain various components, the components are not limited to the terms. These terms should be used only to distinguish one component from another component. For example, a "first" component may be referred to as a "second" component, or similarly, and the "second" component may be referred to as the "first" component within the scope of the right according to the concept of the present disclosure.

As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components or a combination thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Unless otherwise defined herein, all terms used herein including technical or scientific terms have the same meanings as those generally understood. Terms defined in dictionaries generally used should be construed to have meanings matching with contextual meanings in the related art and are not to be construed as an ideal or excessively formal meaning unless otherwise defined herein.

<FIG> schematically illustrates an example of an operation of an image restoration apparatus. Referring to <FIG>, an image restoration apparatus <NUM> may receive a burst image set <NUM>, generate a restored image <NUM> based on the burst image set <NUM>, and output the restored image <NUM>. The burst image set <NUM> may be generated by a camera. The burst image set <NUM> may include a plurality of images that are consecutively captured. The burst image set <NUM> may be, for example, a video generated through a video shooting function or consecutive still images generated through a continuous shooting function. Each of the plurality of images in the burst image set <NUM> may refer to an "individual image. " In an example of a video, each image frame may correspond to an individual image. In an example of a burst image, each still image may correspond to an individual image. The camera may be an image sensor.

For example, when it is assumed that the burst image set <NUM> is generated by capturing a target object using a camera, individual images of the burst image set <NUM> may have different characteristics due to the movement of the camera and/or the target object, and/or a change in ambient light (for example, an illuminance or color). In an example, when the burst image set <NUM> is obtained in a poor environment, such as a low illuminance environment, and/or when each individual image has a deteriorated quality, the restored image <NUM> with an enhanced quality may be derived by properly combining various characteristics of individual images. Thus, the restored image <NUM> with high quality may be derived through a process of restoring individual images with low qualities.

Since objects in each individual image are changed in position due to the movement of a camera and/or a target object, a pre-task of matching objects of each individual image by aligning center lines of each individual image may be required. The center line may not be an actual line represented in each individual image, and a virtual line may be used as a criterion to align individual images. If there is no such pre-task, a severe blur phenomenon may occur. However, to perform the pre-task, the calculation amount and calculation time may significantly increase. In addition, since the task of aligning center lines requires iterative processing, the calculation amount and calculation time may significantly increase as the number of individual images increases.

The image restoration apparatus <NUM> may determine an anchor image based on the individual images of the burst image set <NUM>, execute a neural network model while using anchor information of the anchor image, and generate the restored image <NUM>. For example, an operation of generating the restored image <NUM>, using the anchor information, may include an operation of generating the restored image <NUM> based on the anchor information by iteratively using (for example, emphasizing) the anchor information in an image restoration process. Such an image restoration operation of the image restoration apparatus <NUM> may be provided based on the anchor information of the anchor image instead of a center line. Accordingly, the restored image <NUM> with the enhanced quality may be derived without the task of aligning center lines. Since the task of aligning center lines is not necessary, the calculation amount and calculation time for image restoration may be reduced, and the tendency of the calculation amount and calculation time to significantly increase based on the number of individual images may be resolved.

The image restoration apparatus <NUM> may select an anchor image among the individual images of the burst image set <NUM> or generate an anchor image based on the image information of the individual images. In an example, the image restoration apparatus <NUM> may select an anchor image, among the individual images, based on any one or any combination of any two or more of a quality-based selection, a time interval-based selection, and a random selection. In another example, the image restoration apparatus <NUM> may assign a weight to each of the individual images based on a criterion, such as image quality, and may generate an anchor image by applying the weight to each of the individual images.

The image restoration apparatus <NUM> may execute the neural network model based on the burst image set <NUM>, and generate the restored image <NUM>. For example, the neural network model may include a feature extraction network configured to extract a feature from an individual image of the burst image set <NUM>, and an image restoration network configured to convert the extracted feature to the restored image <NUM>. At least a portion of each of the feature extraction network and the image restoration network may correspond to a deep neural network (DNN), including a plurality of layers. The plurality of layers may include an input layer, at least one hidden layer, and an output layer.

The DNN may include any one or any combination of a fully connected network (FCN), a convolutional neural network (CNN), and a recurrent neural network (RNN). For example, at least a portion of a plurality of layers in a neural network may correspond to a CNN, and another portion may correspond to an FCN. In this example, the CNN may be referred to as a convolution layer, and the FCN may be referred to as a fully connected layer.

In the CNN, data input to each layer may be referred to as an "input feature map," and data output from each layer may be referred to as an "output feature map. " The input feature map and the output feature map may also be referred to as activation data. When a convolutional layer corresponds to an input layer, an input feature map of the input layer may correspond to an input image.

The neural network may be trained based on deep learning, and may perform inference suitable for training, by mapping input data and output data in a nonlinear relationship. Deep learning may be a machine learning scheme for solving an issue such as image or voice recognition from a big data set. Deep learning may be understood as a process of solving an optimization issue to find a point at which energy is minimized while training the neural network based on prepared training data.

Through supervised or unsupervised learning of deep learning, a structure of the neural network or a weight corresponding to a model may be obtained, and input data and output data may be mapped to each other through the weight. For example, when a width and a depth of the neural network are sufficiently large, the neural network may have a capacity large enough to implement an arbitrary function. When the neural network is trained on a sufficiently large quantity of training data through an appropriate training process, optimal performance may be achieved.

In the following description, the neural network may be expressed as being "pre-trained," where "pre-" may indicate a state before the neural network is "started. " The "started" neural network may indicate that the neural network may be ready for inference. For example, the "start" of the neural network may include a loading of the neural network in a memory, or input of input data for inference to the neural network after the neural network is loaded in the memory.

The image restoration apparatus <NUM> may execute a neural network model while using anchor information of an anchor image. For example, the image restoration apparatus <NUM> may emphasize the anchor information when performing any one or any combination of an operation of inputting an input image to the neural network model, an operation of extracting a feature from the input image using the neural network model, and an operation of outputting the extracted feature. The anchor information may include, for example, image information of the anchor image and/or feature information extracted from the anchor image. The anchor information may provide a geometric criterion for image restoration. Thus, image information of corresponding positions may be combined without a need to align center lines. Accordingly, it is possible to enhance image quality while preventing an occurrence of a blur.

<FIG> illustrate examples of operations of selecting an anchor image. Referring to <FIG>, an image restoration apparatus may select an anchor image <NUM>, among a plurality of individual images <NUM> to <NUM>, of a burst image set <NUM>. In an example, the image restoration apparatus may select the anchor image <NUM> based on image quality. In this example, the image restoration apparatus may determine the quality of each of the individual images <NUM> to <NUM> based on any one or any combination of noise, a blur, a signal-to-noise ratio (SNR), and sharpness, and may select an image with the highest quality as the anchor image <NUM>. The image restoration apparatus may use a deep learning network and/or a calculation module to calculate the above quality.

In another example, the image restoration apparatus may select the anchor image <NUM> based on an image order. In this example, the individual images <NUM> to <NUM> may be captured in order. The image restoration apparatus may select the first individual image, the individual image <NUM>, as the anchor image <NUM>. In another example, the image restoration apparatus may select an arbitrary image among the individual images <NUM> to <NUM> as the anchor image <NUM>. This is because an image quality may be enhanced based on image information of the individual images <NUM> to <NUM> even though the anchor image <NUM> for providing a criterion for image restoration has relatively low quality.

Referring to <FIG>, the image restoration apparatus may select an anchor image <NUM> among individual images corresponding to a predetermined time interval. For example, a first time interval <NUM> may cover a period of time from the start of capturing, and the image restoration apparatus may select the anchor image <NUM> in the first time interval <NUM>. In another example, a plurality of time intervals may be used. For example, a second time interval <NUM> and a third time interval <NUM> may cover different capturing times, and the image restoration apparatus may select the anchor image <NUM> in the second time interval <NUM> and the third time interval <NUM>.

Referring to <FIG>, the image restoration apparatus may determine a weight set <NUM> for a burst image set <NUM>, and may generate an anchor image <NUM> by applying weights W<NUM> to W<NUM> to individual images <NUM> to <NUM> of the burst image set <NUM>. For example, the image restoration apparatus may determine the weights W<NUM> to W<NUM> based on image qualities of the individual images <NUM> to <NUM>, and may generate the anchor image <NUM> by reflecting image information of the individual images <NUM> to <NUM> based on the weights W<NUM> to W<NUM>. In this example, when a weight applied to an image increases, the amount of image information of the image to be provided to the anchor image <NUM> may increase.

<FIG> illustrates an example of generating a restored image based on anchor information. Referring to <FIG>, an image restoration apparatus may determine an anchor image based on individual images <NUM> to <NUM> of a burst image set <NUM>, may execute a neural network model <NUM> while emphasizing anchor information <NUM>, and may generate a restored image <NUM>. The burst image set <NUM> may include the individual images <NUM> to <NUM>, and the image restoration apparatus may determine an anchor image based on the individual images <NUM> to <NUM> according to various criteria. In the example of <FIG>, the individual image <NUM> is selected as an anchor image. Although an example of four individual images, for example, the individual images <NUM> to <NUM>, is described below, the number of individual images may be greater than or less than "<NUM>.

The image restoration apparatus may sequentially input the individual images <NUM> to <NUM> to the neural network model <NUM>, and execute the neural network model <NUM> while emphasizing the anchor information <NUM>. For example, the image restoration apparatus may emphasize the anchor information <NUM> when performing any one or any combination of an operation of inputting the individual images <NUM> to <NUM> to the neural network model <NUM>, an operation of extracting features from the individual images <NUM> to <NUM> using the neural network model <NUM>, and an operation of outputting the extracted features. The anchor information <NUM> may include, for example, image information of an anchor image and/or feature information extracted from the anchor image.

The neural network model <NUM> may include a feature extraction network <NUM> and an image restoration network <NUM>. The feature extraction network <NUM> may extract features from the individual images <NUM> to <NUM> in response to inputs of the individual images <NUM> to <NUM>. For example, the feature extraction network <NUM> may extract local features from the individual images <NUM> to <NUM> and may extract global features from the local features. The image restoration network <NUM> may convert the features to the restored image <NUM>. The feature extraction network <NUM> may correspond to, for example, an encoder configured to convert image information into feature information, and the image restoration network <NUM> may correspond to, for example, a decoder configured to convert feature information into image information.

<FIG> illustrates an example of a configuration and an operation related to a neural network model. Referring to <FIG>, a feature extraction network <NUM> may include a local feature extractor <NUM> and a global feature extractor <NUM>. The local feature extractor <NUM> may extract a local feature from each individual image of a burst image set <NUM>, and the global feature extractor <NUM> may extract a global feature from the local features. An image restoration network <NUM> may convert the global feature to a restored image <NUM>. The feature extraction network <NUM> and the image restoration network <NUM> may each include a neural network, and may be pre-trained to perform an extraction operation and a conversion operation.

An image restoration apparatus may execute the feature extraction network <NUM> while iteratively using and/or emphasizing anchor information <NUM>. For example, the image restoration apparatus may emphasize the anchor information <NUM> when performing any one or any combination of an operation of inputting the individual images to the feature extraction network <NUM>, an operation of extracting features from the individual images using the feature extraction network <NUM>, and an operation of outputting the extracted features. An example of an operation related to the use of the anchor information <NUM> will be further described below.

<FIG> illustrates an example of an operation of using anchor information in the process of inputting an input image. Referring to <FIG>, an image restoration apparatus may use anchor information of an anchor image in the process of inputting individual images <NUM> to <NUM> to a feature extraction network. The feature extraction network may correspond to, for example, a local feature extractor. In the example of <FIG>, the individual image <NUM> may be assumed as an anchor image, and the image restoration apparatus may fuse image information of the individual image <NUM> as anchor information with the individual images <NUM> to <NUM>.

For example, fusion may include concatenation and/or addition. The concatenation may be linking elements together, and the addition may be the summing of elements. Thus, the concatenation may influence a dimension, whereas the addition may not influence the dimension. The concatenation may be performed in a channel direction. For example, when each of the individual images <NUM> to <NUM> has a dimension "W × H × C," a concatenation result may have a dimension "W × H × 2C", and an addition result may have a dimension "W × H × C.

The image restoration apparatus may input a fusion result as an input image to the feature extraction network, and may extract local features in operations <NUM> to <NUM>. For example, the image restoration apparatus may extract a local feature in operation <NUM> by inputting a result of the fusion of the individual image <NUM> and anchor image information to the feature extraction network, and as a result, a local feature map <NUM> may be obtained. Similarly, the image restoration apparatus may extract local features in operations <NUM> to <NUM> by sequentially inputting results of the fusion of the other individual images <NUM> to <NUM> and the anchor image information to the feature extraction network, and as a result, local feature maps <NUM> to <NUM> may be obtained.

<FIG> illustrates an example of an operation of using anchor information in a process of outputting an output feature map. Referring to <FIG>, an image restoration apparatus may extract a global feature map <NUM> from local feature maps <NUM> to <NUM> in operation <NUM> of extracting a global feature, and may use anchor information of an anchor image in a process of outputting the global feature map <NUM>. The image restoration apparatus may perform operation <NUM> using a feature extraction network. The feature extraction network may correspond to, for example, a global feature extractor. In the example of <FIG>, the local feature map <NUM> may be assumed to be extracted from the anchor image, and the image restoration apparatus may fuse feature information of the local feature map <NUM> as anchor information with the global feature map <NUM>. In this example, the fusing may include concatenation and/or addition. A result of the fusing may correspond to an output feature map of the feature extraction network, and the image restoration apparatus may convert the output feature map to a restored image using an image restoration network.

<FIG> illustrates an example of an operation of using anchor information in a process of extracting a global feature. Referring to <FIG>, an image restoration apparatus may extract a global feature map <NUM> from local feature maps <NUM> to <NUM> in operation <NUM> of extracting a global feature, and anchor information of an anchor image may be used as guide information. In the example of <FIG>, the local feature map <NUM> may be assumed to be extracted from the anchor image, and the image restoration apparatus may use feature information of the local feature map <NUM> as anchor information, that is, guide information. For example, when the local feature map <NUM> is assumed as an anchor local feature, and when the local feature maps <NUM> to <NUM> are assumed as neighboring local features, the image restoration apparatus may assign a higher weight to the anchor local feature than the neighboring local features and may perform operation <NUM>. Thus, information of the anchor local feature may have a great influence on the global feature map <NUM> in comparison to the neighboring local features.

<FIG> illustrates an example of the operation of <FIG>. As shown in <FIG>, various weight assignment schemes may be used to use anchor information as guide information. <FIG> illustrates a scheme of emphasizing anchor information through operation <NUM> of assigning a weight and operation <NUM> of performing weighted fusion. Since anchor information corresponding to a geometric criterion for image restoration is emphasized in the above scheme, the scheme may be more effective in combining image information corresponding to each other through a pooling operation (for example, max pooling or average pooling) in comparison to a scheme of extracting a global feature from a local feature. Referring to <FIG>, an image restoration apparatus may assign different weights to local feature maps <NUM> to <NUM>. The image restoration apparatus may fuse the local feature maps <NUM> to <NUM> with each other while taking into consideration a weight set <NUM> in operation <NUM>, and as a result, a global feature map <NUM> may be generated. For example, the image restoration apparatus may assign weights W<NUM> to W<NUM> to the local feature maps <NUM> to <NUM> through Softmax and may sum the local feature maps <NUM> to <NUM> based on the weights W<NUM> to W<NUM>, to generate the global feature map <NUM>.

For example, it may be assumed that the local feature map <NUM> is extracted from an anchor image and the local feature maps <NUM> to <NUM> are extracted from the other individual images. In this example, the image restoration apparatus may set the weight W<NUM> of the local feature map <NUM> to be higher than the weights W<NUM> to W<NUM> of the local feature maps <NUM> to <NUM>. Accordingly, anchor information may be emphasized through feature information of the local feature map <NUM>. In another example, the image restoration apparatus may determine a similarity between the local feature map <NUM> and each of the local feature maps <NUM> to <NUM>, and may assign a relatively high weight to a feature map similar to the local feature map <NUM> as well as the local feature map <NUM>. In this example, when a similarity between the local feature map <NUM> and the local feature map <NUM> is high, and a similarity between each of the local feature maps <NUM> and <NUM> and the local feature map <NUM> is low, the image restoration apparatus may set the weights W<NUM> and W<NUM> of the local feature maps <NUM> and <NUM> to be higher than the weights W<NUM> and W<NUM> of the local feature maps <NUM> and <NUM>. Thus, the anchor information may be strengthened through feature information of the local feature maps <NUM> and <NUM>.

<FIG> illustrates an example of an operation of using anchor information in a process of extracting local features. Referring to <FIG>, an image restoration apparatus may extract local feature maps <NUM> to <NUM> from individual images <NUM> to <NUM> of a burst image set <NUM> through operations <NUM> to <NUM> of extracting local features. The image restoration apparatus may emphasize anchor information in the process of extracting the local feature maps <NUM> to <NUM>. For example, the image restoration apparatus may select the individual image <NUM> as an anchor image and may generate anchor information <NUM> based on the individual image <NUM>. The anchor information <NUM> may include image information and/or feature information of the individual image <NUM>.

A feature extraction network (for example, a local feature extractor) may include a plurality of layers. The plurality of layers may be classified into layer groups, each including a portion of the layers. For example, each layer group may include a convolution layer and/or a pooling layer. The image restoration apparatus may extract local features while using the anchor information for each of a plurality of layer groups of the feature extraction network. The image restoration apparatus may extract local features through each layer group, and may fuse the anchor information <NUM> with the extracted local features. The image restoration apparatus may iteratively perform the above process on all the layer groups, to generate the local feature maps <NUM> to <NUM>.

For example, the image restoration apparatus may extract a primary local feature from the individual image <NUM> through operation <NUM> of extracting features using a first layer group, and may transform the primary local feature by fusing the anchor information <NUM> with the primary local feature. The image restoration apparatus may extract a secondary local feature from the transformed primary local feature through operation <NUM> of extracting features using a second layer group, and may transform the secondary local feature by fusing the anchor information <NUM> with the secondary local feature. Also, the image restoration apparatus may extract a tertiary local feature from the transformed secondary local feature through operation <NUM> of extracting features using a third layer group, and may transform the tertiary local feature by fusing the anchor information <NUM> with the tertiary local feature. When operation <NUM> of extracting features using a last layer group is completed, the local feature map <NUM> may be generated as a result of operation <NUM>. Operations <NUM> to <NUM> associated with the other individual images <NUM> to <NUM> may also correspond to operation <NUM> for the individual image <NUM>, and as a result, the local feature maps <NUM> to <NUM> may be generated.

In this example, the image restoration apparatus may fuse the same anchor information <NUM> with an output of each layer group, or may fuse the anchor information <NUM> specialized for each layer group. Fusion through common anchor information <NUM> is described below. The common anchor information <NUM> may be image information and/or feature information of an anchor image. To obtain the feature information, an operation of extracting features from the anchor image may be pre-emptively performed. For example, the above pre-emptive operation may be performed using a feature extraction network (for example, the feature extraction network <NUM> of <FIG>, or the local feature extractor <NUM> of <FIG>) used for operations <NUM> to <NUM>, or using a separate feature extraction network. When common image information and/or common feature information are provided, the common image information and/or common feature information may be fused with an output, for example, a local feature, of each layer group according to operations <NUM> to <NUM>.

Subsequently, fusion through specialized anchor information <NUM> is described below. The specialized anchor information <NUM> may be information processed to be suitable for each layer group, unlike the common anchor information <NUM>. The specialized anchor information <NUM> may include stepwise local features of an anchor image extracted through each layer group. For example, when a first local feature to a third local feature of the anchor image are extracted through the first layer group to the third layer group, the first local feature to the third local feature may be used as anchor information <NUM> specialized for the first layer group to the third layer group. Thus, the first local feature may be fused with each of the local features extracted through operations <NUM> and <NUM>, the second local feature may be fused with each of the local features extracted through operations <NUM> and <NUM>, and the third local feature may be fused with each of the local features extracted through operations <NUM> and <NUM>.

<FIG> illustrates an example of an image restoration method. Referring to <FIG>, in operation <NUM>, an image restoration apparatus may determine an anchor image based on individual images of a burst image set. The image restoration apparatus may select an anchor image among the individual images based on the qualities of the individual images. Also, the image restoration apparatus may select an arbitrary image among the individual images as an anchor image.

In operation <NUM>, the image restoration apparatus may execute a feature extraction network based on the burst image set while using anchor information of the anchor image. The image restoration apparatus may extract a primary local feature from a first individual image among the individual images using a first layer group of the feature extraction network, may transform the primary local feature by fusing the anchor information with the primary local feature, and may extract a secondary local feature from the transformed primary local feature using a second layer group of the feature extraction network. Also, the image restoration apparatus may transform the secondary local feature by fusing the anchor information with the secondary local feature, may extract a tertiary local feature from the transformed secondary local feature using a third layer group of the feature extraction network, and may determine a global feature based on the tertiary local feature.

In an example, the image restoration apparatus may extract an anchor local feature from the anchor image, may extract a local feature from an image other than the anchor image among the individual images, and may extract a global feature from the anchor local feature and the local feature of the image while using the anchor local feature. In this example, the image restoration apparatus may extract the global feature from the anchor local feature and the local feature of the image by assigning a higher weight to the anchor local feature than the local feature of the image.

In another example, the image restoration apparatus may extract an anchor local feature from the anchor image, may extract a local feature from an image other than the anchor image among the individual images, may extract a global feature from the anchor local feature and the local feature of the image and may fuse the anchor local feature with the global feature. Also, the image restoration apparatus may generate input images of a neural network model by fusing anchor information with each of the individual images.

In operation <NUM>, the image restoration apparatus may generate a restored image based on a feature map corresponding to an output of the feature extraction network. The image restoration apparatus may execute an image restoration network based on the feature map. The description of <FIG> may apply to the image restoration method.

<FIG> illustrates an example of a configuration of an image restoration apparatus. Referring to <FIG>, an image restoration apparatus <NUM> may include a processor <NUM>, and a memory <NUM>. The memory <NUM> may be connected to the processor <NUM>, and may store instructions executable by the processor <NUM>, data to be computed by the processor <NUM>, or data processed by the processor <NUM>. The memory <NUM> may include, for example, a non-transitory computer-readable storage medium, for example, a high-speed random access memory (RAM) and/or a non-volatile computer-readable storage medium (for example, at least one disk storage device, a flash memory device, or other non-volatile solid state memory devices).

The processor <NUM> may execute instructions to perform the operations described above with reference to <FIG>. For example, the processor <NUM> may determine an anchor image based on individual images of a burst image set, execute a feature extraction network based on the burst image set while using anchor information of the anchor image, and generate a restored image based on a feature map corresponding to an output of the feature extraction network. In addition, the description of <FIG> is also applicable to the image restoration apparatus <NUM>.

<FIG> illustrates an example of a configuration of an electronic apparatus. Referring to <FIG>, the electronic apparatus <NUM> may include a processor <NUM>, a memory <NUM>, a camera/image sensor <NUM>, a storage device <NUM>, an input device <NUM>, an output device <NUM>, and a network interface <NUM>. The processor <NUM>, the memory <NUM>, the camera/image sensor <NUM>, the storage device <NUM>, the input device <NUM>, the output device <NUM>, and the network interface <NUM> may communicate with each other via a communication bus <NUM>. For example, the electronic apparatus <NUM> may be implemented as at least a portion of, for example, a mobile device such as a mobile phone, a smartphone, a personal digital assistant (PDA), a netbook, a tablet computer or a laptop computer, a wearable device such as a smartwatch, a smart band or smart glasses, a computing device such as a desktop or a server, home appliances such as a television (TV), a smart TV or a refrigerator, a security device such as a door lock, or a vehicle such as an autonomous vehicle or a smart vehicle. The electronic apparatus <NUM> may structurally and/or functionally include the image restoration apparatus <NUM> of <FIG> and/or the image restoration apparatus <NUM> of <FIG>.

The processor <NUM> may execute instructions and functions in the electronic apparatus <NUM>. For example, the processor <NUM> may process instructions stored in the memory <NUM> or the storage device <NUM>. The processor <NUM> may perform at least one of the operations described above with reference to <FIG>. The memory <NUM> may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The memory <NUM> may store instructions that are to be executed by the processor <NUM>, and may also store information associated with software and/or applications when the software and/or applications are being executed by the electronic apparatus <NUM>.

The camera/image sensor <NUM> may capture a photo and/or a video. For example, the camera/image sensor <NUM> may consecutively capture photos or film a video to generate a burst image set. When the burst image set includes successive photos, each individual image of the burst image set may correspond to each photo. When the burst image set is a video, each individual image of the burst image set may correspond to each frame of the video. The storage device <NUM> may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. In an example, the storage device <NUM> may store a greater amount of information than that of the memory <NUM> for a relatively long period of time. For example, the storage device <NUM> may include magnetic hard disks, optical disks, flash memories, floppy disks, or other forms of non-volatile memories known in the art.

The input device <NUM> may receive an input from a user through a traditional input scheme using a keyboard and a mouse, and through a new input scheme such as a touch input, a voice input and an image input. The input device <NUM> may include, for example, a keyboard, a mouse, a touch screen, a microphone, or other devices configured to detect an input from a user and transmit the detected input to the electronic apparatus <NUM>. The output device <NUM> may provide a user with an output of the electronic apparatus <NUM> through a visual channel, an auditory channel, or a tactile channel. The output device <NUM> may include, for example, a display, a touchscreen, a speaker, a vibration generator, or any other device configured to provide a user with the output. The network interface <NUM> may communicate with an external device via a wired or wireless network.

The image restoration apparatus <NUM>, image restoration apparatus <NUM>, processor <NUM>, and memory <NUM> in <FIG> that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
A computer-implemented image restoration method comprising:
determining (<NUM>) an anchor image based on individual images of a burst image set (<NUM>);
executing (<NUM>) a feature extraction network (<NUM>) based on the burst image set (<NUM>) while using anchor information (<NUM>) of the anchor image (<NUM>; <NUM>), comprising extracting a local feature while using the anchor information (<NUM>) for each of a plurality of layer groups of the feature extraction network (<NUM>), and wherein the executing of the local feature comprises:
extracting a primary local feature from a first individual image of the individual images using a first layer group of the plurality of layer groups;
transforming the primary local feature by fusing the anchor information (<NUM>) with the primary local feature;
extracting a secondary local feature from the transformed primary local feature using a second layer group of the plurality of layer groups; and
determining a global feature based on the secondary local feature;
generating (<NUM>) a restored image (<NUM>) based on a feature map corresponding to an output of the feature extraction network (<NUM>).