Patent Description:
The development of optical technology and image processing technology has led to the widespread use of an image capturing device in extensive fields of application, for example, multimedia contents, security, and object recognition. The image capturing device may be provided in a mobile device, a camera, a vehicle, a computer, or the like, to capture an image, recognize an object, or obtain data for controlling a device. A size of the image capturing device may be determined by the size of a lens, a focal length of a lens, the size of a sensor, and the like. To reduce the size of the image capturing device, a multi-lens including compact and small lenses may be used. <CIT> discloses a method and apparatus for restoring image. An image restoration apparatus generates a plurality of warped images by warping each of a plurality of input images based on respective depths corresponding to a plurality of disparities, and generates an output image with a high resolution based on the plurality of input images and the plurality of warped images using an image restoration model. <CIT> discloses a method and apparatus for restoring image. The image restoration apparatus may store an image restoration model including a convolutional layer corresponding to kernels having various dilation gaps, and may restore an output image from a target image obtained by rearranging a compound eye vision, CEV, image by the image restoration model. The publication of <NPL>) discloses a Spatial-Angular Attention Network for Light Field Reconstruction. A spatial-angular attention network is discussed to perceive correspondences in the light field non-locally, and reconstruction high angular resolution light field in an end-to-end manner. The publication of <NPL> discloses attention-based view selection networks for light-field disparity estimation and introduces a deep network for estimating depth maps from a light field image. A view selection module is discussed that generates an attention map indicating die importance of each view and its potential for contributing to accurate depth estimation.

One or more example embodiments may address at least the above problems and/or disadvantages and other disadvantages not described above. Also, the example embodiments are not required to overcome the disadvantages described above, and an example embodiment may not overcome any of the problems described above. It is the object of the present invention to provide an improved image restoration method and an improved image restoration device.

The above and/or other aspects will be more apparent by describing certain example embodiments, taken in conjunction with the accompanying drawings, in which:.

Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout.

The following structural or functional descriptions are merely examples to describe embodiments, and the scope of the example embodiments is not limited to the descriptions provided in the disclosure. Various changes and modifications can be made thereto by those of ordinary skill in the art.

Although terms of "first" or "second" are used to explain various components, the components are not limited to the terms. These terms should be used only to distinguish one component from another component. For example, a "first" component may be referred to as a "second" component, or similarly, and the "second" component may be referred to as the "first" component.

As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms "comprises," "comprising," "includes," and/or "including," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components or a combination thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Unless otherwise defined herein, all terms used herein including technical or scientific terms have the same meanings as those generally understood by one of ordinary skill in the art. Terms defined in dictionaries generally used should be construed to have meanings matching with contextual meanings in the related art and are not to be construed as an ideal or excessively formal meaning unless otherwise defined herein.

<FIG> illustrates an example of an overall process of image restoration according to an example embodiment.

According to an example embodiment, an image restoration device may restore an image based on information sensed from a scene. The image restoration device may be embodied by an imaging device <NUM>, for example. The imaging device <NUM> may include a lens array and an image sensor. A quality of an image captured and restored by the imaging device <NUM> may be determined by the number of sensing elements included in the image sensor and an amount of light incident on the sensing elements. A sensing element described herein may be an image sensing element that is configured to sense light passing through the lens array and includes, for example, a complementary metal-oxide-semiconductor (CMOS), a charge-coupled device (CCD), or a photodiode.

The image sensor may include a sensing array <NUM> in which a plurality of sensing elements is arranged on a plane. For example, a resolution of an image may be determined by the number of the sensing elements included in the sensing array <NUM>, and a sensitivity of the image may be determined by an amount of light incident on the sensing elements. In turn, the amount of light incident on the sensing elements may be determined based on a size of the sensing elements. Thus, as the size of each sensing element increases, the amount of light may also increase and a dynamic range of the sensing array <NUM> may increase. Thus, as the number of the sensing elements included in the sensing array <NUM> increases, the imaging device <NUM> may capture a high-resolution image. In addition, as the size of the sensing elements increases, the imaging device <NUM> may operate more effectively in capturing a high-sensitivity image at a low light level.

A size of the imaging device <NUM> may be determined by a focal length of a lens element <NUM>. In detail, the size of the imaging device <NUM> may be determined by an interval between the lens element <NUM> and the sensing array <NUM>. This is because, to collect light refracted by the lens element <NUM>, the lens element <NUM> and the sensing array <NUM> need to be arranged separately from each other by a distance equal to the focal length of the lens element <NUM>. The focal length of the lens element <NUM> may be determined by a viewing angle of the imaging device <NUM> and a size of the lens element <NUM>. For example, when the viewing angle is fixed, the focal length may increase in proportion to the size of the lens element <NUM>. As the size of the sensing array <NUM> increases to capture an image of a certain viewing angle range, the size of the lens element <NUM> may also need to increase. As described above, to increase a sensitivity of an image while maintaining a viewing angle and a resolution, the size of the imaging device <NUM> may increase. For example, to increase the sensitivity while maintaining the resolution, a size of each sensing element may need to increase while maintaining the number of the sensing elements included in the sensing array <NUM>, and thus the size of the sensing array <NUM> may increase. In this example, to maintain the viewing angle, the size of the sensing array <NUM> may increase, and the size of the lens element <NUM> may also increase. Thus, the focal length of the lens element <NUM> may increase, and the size of the imaging device <NUM> may increase accordingly.

According to an example embodiment, to reduce the size of the imaging device <NUM>, the lens array may include a plurality of lens elements corresponding to a plurality of viewpoints. The lens elements may be arranged on a plane of the lens array. The sensing array <NUM> may be divided into sensing regions corresponding to the lens elements. The plane of the lens array may be parallel to the plane of the sensing array <NUM>, and separated from the plane of the sensing array <NUM> by the focal length of the lens element <NUM>. The lens array may also be referred to as a multi-lens array (MLA). To decrease a size of each of the lens elements included in the lens array, that is, to increase the number of lenses to be included in a same area in the lens array, the focal length of the lens element <NUM> may decrease and the thickness of the imaging device <NUM> may thereby decrease. Thus, it is possible to embody a thin camera. In such a case, the imaging device <NUM> may realign and concatenate low-resolution viewpoint images <NUM> captured by the lens elements to restore a high-resolution output image <NUM>.

The lens element <NUM> which is an individual lens element of the lens array may cover a sensing region <NUM> of the sensing array <NUM> corresponding to a lens size of the lens element <NUM>. The sensing region <NUM> of the sensing array <NUM> that is covered by the lens element <NUM> may be determined by the lens size of the lens element <NUM>. The sensing region <NUM> may be a region in the sensing array <NUM> in which rays in a certain viewing angle range arrive after passing through the lens element <NUM>. The size of the sensing region <NUM> may be represented by a distance from a center of the sensing region <NUM> to an outermost point or a diagonal length, and the lens size may correspond to a diameter of a corresponding lens.

Each of the sensing elements of the sensing array <NUM> may generate sensed information based on a ray passing through the lenses of the lens array. For example, a sensing element may sense, as the sensed information, an intensity of light received through the lens element <NUM>. The imaging device <NUM> may determine intensity information corresponding to an original signal associated with points included in a view of the imaging device <NUM> based on the sensed information output from the sensing array <NUM>, and restore an image based on the determined intensity information.

When the diversity of the sensed information is sufficiently secured and a full rank relationship is established between the sensed information and original signal information corresponding to the points included in the view of the imaging device <NUM>, an image corresponding to a maximum resolution of the sensing array <NUM> may be obtained. The diversity of the sensed information may be obtained based on parameters of the imaging device <NUM> such as the number of the lenses included in the lens array and the number of the sensing elements included in the sensing array <NUM>.

In addition, the sensing region <NUM> covered by the lens element <NUM> may include a non-integer number of sensing elements. An MLA structure may be embodied in a fractional alignment structure. For example, when the lens elements included in the lens array have the same lens size, the number of the lens elements included in the lens array and the number of the sensing elements included in the sensing array <NUM> may have a relatively prime relationship. For example, a ratio P/L between the number L of the lens elements of the lens array and the number P of sensing elements in a single row or column of the sensing array <NUM> may be determined to be a non-integer number. Each of the lens elements may cover the number of sensing elements which is the same as a pixel offset corresponding to P/L.

As described above, through the fractional alignment structure, an optical center axis (OCA) of the lens elements of the imaging device <NUM> may be slightly differently arranged with respect to the sensing array <NUM>. That is, the lens element <NUM> may be eccentrically arranged with respect to a sensing element. Thus, the lens elements of the lens array may receive different light field (LF) information. An LF may refer to a field indicating a direction and intensity of a ray that is radiated from an arbitrary target point and reflected from an arbitrary point on a subject. The LF information may indicate information in which a plurality of LFs is combined. A direction of a chief ray of each of the lens elements may vary, and thus the sensing regions may receive different LF information and the sensing regions may obtain slightly different input information. Through the slightly different input information, the imaging device <NUM> may obtain an optically greater amount of sensed information.

Although only the sensing array <NUM> of the image sensor is illustrated for the convenience of description, examples are not limited thereto. The image sensor may further include a condensing lens, a color filter (CF), and the like. The color filter may allow a ray of a wavelength corresponding to a color desired by a sensing element to pass through to be sensed, and to be transmitted to the sensing element. Thus, the sensing element may generate, as the sensed information, a color intensity value corresponding to the color. The color filter may be arranged in a Bayer pattern, but is not limited thereto. The condensing lens may be provided as a plurality of condensing lenses respectively corresponding to a plurality of sensing elements and respectively arranged on the sensing elements, and be configured to collect rays incident on the sensing elements. A condensing lens array may include the condensing lenses arranged on the plane of the sensing array <NUM>.

According to an example embodiment, the imaging device <NUM> may include a plurality of sensing units. A number of sensing units may be equal to a number of lenses included in an MLA. For example, a sensing unit may include the lens element <NUM> and sensing elements of the sensing region <NUM> covered by the lens element <NUM>. In the imaging device <NUM>, a position at which each lens element is arranged may correspond to a viewpoint. A viewpoint may indicate a point from which a subject is observed and/or captured. As described above, the lens elements are arranged in the fractional alignment structure with respect to the sensing array <NUM>, and thus information obtainable for each viewpoint may differ slightly. The imaging device <NUM> may capture an image (hereinafter "viewpoint image") corresponding to a viewpoint based on light received through the lens element arranged at that viewpoint.

According to an example embodiment, the imaging device <NUM> may capture a plurality of viewpoint images through lenses arranged at different positions. For example, the imaging device <NUM> may generate a viewpoint image from sensed information obtained for each sensing region corresponding to each lens. That is, each of the sensing units may obtain a viewpoint image. As described above, the sensing units may obtain different LF information, and thus viewpoint images captured by the sensing units may include slightly different scenes. For example, the imaging device <NUM> may include N lenses corresponding to N sensing units. The N sensing units may individually capture viewpoint images, and thus the imaging device <NUM> may obtain N viewpoint images. Here, N denotes an integer greater than or equal to <NUM>. For example, as illustrated in <FIG>, an MLA may include <NUM> (N =<NUM>×<NUM>=<NUM>) lenses, and an image sensor may capture <NUM> low-resolution viewpoint images <NUM>. For another example, a multi-lens image may include <NUM> (N=<NUM>×<NUM>=<NUM>) viewpoint images. Although it is described above that one imaging device <NUM> includes a plurality of sensing units, examples are not limited thereto. For example, a sensing unit may be an independent image sensing module, for example, a camera sensor. In this example, each sensing unit may be arranged at a position different from that of another sensing unit. For example, a sensing unit may be provided in each of a plurality of drones, and a positional relationship between sensing units may dynamically change as the drones move.

Referring to <FIG>, the imaging device <NUM> may generate a plurality of low-resolution viewpoint images <NUM> from various sets of sensed information obtained as described above, and restore a high-resolution output image <NUM> based on a target image <NUM> among the generated low-resolution viewpoint images <NUM>. The target image <NUM> may be captured by a target sensing unit corresponding to a target viewpoint, for example, a lens element <NUM> corresponding to the target viewpoint and a corresponding sensing region <NUM>.

Although it is illustrated in <FIG> that a center image among the viewpoint images <NUM> is determined to be the target image <NUM>, examples are not limited thereto. For example, another viewpoint image, for example, a viewpoint image <NUM>, may also be used as a target image. In addition, the imaging device <NUM> may use an image of another additional image sensor as the target image. The additional image sensor may be an independent sensor different from the imaging device <NUM>, for example, a camera sensor configured to capture a higher-resolution image compared to the imaging device <NUM>.

<FIG> illustrates a flowchart of an example of an image restoration method according to an example embodiment. <FIG> illustrates an example of image restoration using a warping model and an image restoration model according to an example embodiment.

Referring to <FIG> and <FIG>, in operation <NUM>, an image restoration device obtains input data including input image information <NUM> for each of a plurality of viewpoints. The input data may refer to data to be input to a warping model <NUM>, and may include a plurality of sets of the input image information <NUM> corresponding to the different viewpoints. The input data may be data in which the sets of the input image information <NUM> of the viewpoints are concatenated. For example, the input image information <NUM> corresponding to one of the viewpoints may include a viewpoint image <NUM> that is captured from the viewpoint, or a feature map corresponding to the viewpoint that is extracted from the viewpoint image <NUM>. In this example, the input data may be data in which feature maps are concatenated. The viewpoint image <NUM> may refer to an image captured at a viewing angle determined by an optical system at the corresponding viewpoint, and may correspond to a set of sensed values from sensing elements that sense rays passing through a lens corresponding to the viewpoint. An input feature map <NUM> may indicate feature data and/or a feature vector that is extracted from the viewpoint image <NUM> sensed through the lens for capturing a corresponding scene. However, the input image information <NUM> is not limited to what is described in the foregoing. For example, the input image information <NUM> corresponding to a viewpoint may include the viewpoint image <NUM> itself corresponding to the viewpoint instead of the input feature map <NUM>. The input feature map <NUM> will be described hereinafter as an example of the input image information <NUM>, and extracting the input feature map <NUM> will be described in detail with reference to <FIG>.

In operation <NUM>, the image restoration device estimates a global transformation parameter of each of the viewpoints from the obtained input data. In other words, each of the viewpoints may correspond to a respective global transformation parameter. A global transformation parameter corresponding to one of the viewpoints may be a parameter for globally transforming the viewpoint of the viewpoint image <NUM> into a target viewpoint, and may include a rotation parameter, a translation parameter, and a scale parameter, for example. The rotation parameter corresponding to one viewpoint may indicate a degree of a rotation between a sensing unit of the viewpoint and a sensing unit of the target viewpoint, for example, rotations on three axes. The translation parameter corresponding to one viewpoint may indicate a level of translation between a sensing unit of the viewpoint and the sensing unit of the target viewpoint, for example, translation along each of the three axes. The scale parameter corresponding to one viewpoint may indicate a difference in scale between a sensing unit of the viewpoint and the sensing unit of the target viewpoint. The estimating of the global transformation parameter will be described in detail with reference to <FIG>.

In operation <NUM>, the image restoration device estimates disparity information of each of the viewpoints from the obtained input data. In other words, each of the viewpoints may correspond to respective disparity information. The disparity information may refer to information for correcting a disparity error between globally transformed image information and target image information, which will be described in operation <NUM>. The disparity information may also be referred to herein as disparity correction information. The estimating of the disparity information will be described in detail with reference to <FIG>.

In operation <NUM>, the image restoration device generates warped image information <NUM> for each of the viewpoints by warping the input image information <NUM> using the global transformation parameter and the disparity information. In other words, each of the viewpoints may correspond to respective warped image information. For example, the warped image information <NUM> may include a warped feature map obtained by transforming the input feature map <NUM> extracted from an input image into a pixel coordinate system of a target sensing unit capturing a target image. However, the warped image information <NUM> is not limited to the foregoing. For example, the warped image information <NUM> may be a warped image itself obtained by transforming the input image into the pixel coordinate system of the target image. The warping performed using the global transformation parameter and the disparity information will be described in detail with reference to <FIG>.

In operation <NUM>, the image restoration device generates an output image <NUM> for the viewpoints using an image restoration model <NUM> from the generated warped image information <NUM>. For example, first, the image restoration device may generate image information that is realigned according to a single viewpoint by performing a pixel shuffle on the generated warped image information <NUM>. The pixel shuffle may refer to an operation of realigning or rearranging pixels indicating same and/or similar points in feature maps of the viewpoints and/or viewpoint images of the viewpoints to be near to one another. By the pixel shuffle, the image information that is realigned according to the single viewpoint may be generated. Then, the image restoration device may generate the output image <NUM> having a target resolution by applying the image restoration model <NUM> to the realigned image information. The output image <NUM> may be, for example, an image obtained through image registration of pixels of the viewpoint images based on the target image, and the target resolution of the output image <NUM> may be greater than or equal to a resolution of each of the individual viewpoint images. The output image <NUM> may be an image <NUM> obtained as the viewpoint images are integrated according to the single viewpoint.

According to an example embodiment, the image restoration model <NUM> may be a model that is trained to output the output image <NUM> from the warped image information <NUM>. The warped image information <NUM> input to the image restoration model <NUM> may be the realigned image information that is obtained by the pixel shuffle for realignment according to the single viewpoint. The image restoration model <NUM> may be, for example, a neural network of a machine learning structure. The neural network may perform image registration-based image restoration and the like by mapping input data and output data that are in a nonlinear relationship to each other based on deep learning. The deep learning may be a machine learning method used to solve an issue of image registration from a big dataset, and may map the input data and output data through supervised or unsupervised learning. For example, as illustrated, the neural network may include an input layer <NUM>, a plurality of hidden layers <NUM>, and an output layer <NUM>. Data input through the input layer <NUM> may be propagated through the hidden layers <NUM> and then be output from the output layer <NUM>. However, data may be input directly to the hidden layers <NUM> or data may be output directly from the hidden layers <NUM>, instead of the input layer <NUM> and the output layer <NUM>. Although the example that the image restoration model <NUM> includes the neural network is described above, examples are not limited to the foregoing. A portion of a warping model and a feature extraction model that will be described hereinafter may also include a neural network. The neural network may be trained through backpropagation, for example. Hereinafter, the training of the image restoration model <NUM> will be described in detail with reference to <FIG>.

The image restoration model <NUM> may be embodied by a convolutional neural network (CNN). The CNN may be a neural network including convolutional layers, and the hidden layers <NUM> of the CNN may include a convolutional layer that applies convolution filtering to input data. For example, the CNN may include a convolutional layer of which nodes are connected through a kernel. The CNN may be a network trained in advance to output the output image <NUM> of a high resolution from a plurality of sets of the warped image information <NUM>. The image restoration device may extract feature data by performing convolution filtering on data input to the convolutional layer. The feature data may be data of which a feature of an image is abstracted, and may indicate a value resulting from a convolution operation based on a kernel of the convolutional layer. The image restoration device may perform the convolution operation on a pixel at a position and neighboring pixels of the pixel using values of elements of the kernel. The image restoration device may calculate a convolution operation value of each of pixels by sweeping a kernel across the pixels of an image. The convolutional layers of the image restoration model may be connected in a U-net architecture, but are not limited thereto.

According to an example embodiment, the image restoration device may reconstruct feature maps of each of the plurality of viewpoint images by performing warping using a global transformation parameter and disparity information. The image restoration device may align the feature maps with a target viewpoint, without information given in advance, for example, a camera parameter and depth information.

<FIG> illustrates an example of a feature extraction model for generating input image information according to an example embodiment.

Referring to <FIG>, an image restoration device captures a plurality of viewpoint images <NUM> through lenses arranged at different positions. In the example of <FIG>, a resolution of each of the viewpoint images <NUM> is H × W, in which H denotes the number of pixels arranged along a height of each of the viewpoint images <NUM> and W denotes the number of pixels arranged along a width of each of the viewpoint images <NUM>. Each of H and W may be an integer greater than or equal to <NUM>. For example, as illustrated, the image restoration device captures the viewpoint images <NUM> including a first viewpoint image c<NUM> through a ninth viewpoint image c<NUM>. Each of the viewpoint images <NUM> may include a color image which includes three channel images-a red channel image, a green channel image, and a blue channel image-and may correspond to data having a <NUM>×H×W dimension. In other words, each viewpoint image <NUM> may include data for a number of pixels equal to <NUM>×H×W.

The image restoration device obtains input image information corresponding to each of the viewpoints from the captured viewpoint images <NUM>. For example, the image restoration device extracts, as a plurality of sets of the input image information, a plurality of input feature maps <NUM> from the viewpoint images <NUM> using a feature extraction model <NUM>.

The feature extraction model <NUM> includes one or more convolutional layers. The feature extraction model <NUM> includes convolutional layers connected in a form of a residual block (ResBlock as illustrated). In the example of <FIG>, the feature extraction model <NUM> includes one convolutional layer and M residual blocks for each viewpoint. Here, M denotes an integer greater than or equal to <NUM>. A residual block includes one or more convolutional layers. Through the convolutional layers, the residual block may extract feature data from data input to the residual block, and output a result of adding the extracted feature data and the data input to the residual block. For example, an output of a jth residual block may be input to a j+1th residual block. In this example, j denotes an integer greater than or equal to <NUM> and less than or equal to M-<NUM>. The image restoration device extracts, as the input feature maps <NUM>, results obtained by applying convolution filtering by one or more residual blocks to the viewpoint images <NUM>. The feature extraction model <NUM> is illustrated in <FIG> as including a total of V×M residual blocks that are independently distinguished according to each viewpoint. Here, V denotes the number of viewpoints, which is an integer greater than or equal to <NUM>. In the example of <FIG>, V is <NUM>. However, examples are not limited to the foregoing. For example, the image restoration device may apply the M residual blocks of the feature extraction model <NUM> to respective viewpoints in sequential order.

For example, an input feature map <NUM> of a viewpoint may include feature data and/or a feature vector that is extracted from one of the viewpoint images <NUM> corresponding to the viewpoint based on the feature extraction model <NUM>. The image restoration device generates the input feature maps <NUM> corresponding to the viewpoints, and generates input data by concatenating the generated input feature maps <NUM>. For example, a feature map extracted from a viewpoint image of a viewpoint by a convolutional layer and a residual block of the feature extraction model <NUM> may be augmented by a factor of D. In this example, D is an integer greater than or equal to <NUM>. For example, each of the input feature maps <NUM> may have a H×W×D dimension. In this example, a dimension of the total input data may then be H×W×D×V. Here, an example that D = <NUM> is described.

<FIG> illustrates an example of a warping model according to an example embodiment.

Referring to <FIG>, a warping model includes a global transformation parameter estimator <NUM>, a disparity estimator <NUM>, and a warping operation <NUM>. The warping model may include a plurality of convolution operations and a plurality of pooling operations.

An image restoration device generates feature data extracted from input data <NUM>. For example, the feature data may be pooling data that is pooled from the input data <NUM>. For example, the image restoration device applies convolution filtering <NUM> to the input data <NUM>. In this example, the feature data having dimensions of H×W×<NUM> may be extracted from the input data <NUM> having dimensions of V×H×W×<NUM>. The image restoration device applies a pooling layer <NUM> and a convolutional layer <NUM> to the feature data for downsampling. The dimension of the feature data may then be reduced to H/<NUM> × W/<NUM> × <NUM>. The image restoration device applies a pooling layer <NUM> again for downsampling, and thus generates pooling data having dimensions of H/<NUM> × W/<NUM> × <NUM>. Although it is illustrated in <FIG> that a pooling layer reduces a height and a width of data by half, examples are not limited to the foregoing.

The image restoration device estimates a global transformation parameter through the global transformation parameter estimator <NUM>. The image restoration device obtains information in which a spatial dimension component is removed from the input data <NUM> through a global pooling operation <NUM>. The global pooling operation <NUM> may be a global average pooling operation. As a result of the global average pooling operation, the information in which the spatial dimension component is removed is obtained having dimensions of <NUM>×<NUM>. The image restoration device calculates the global transformation parameter from the information in which the spatial dimension component is removed. For example, the image restoration device applies, to the information in which the spatial dimension component is removed, an operation associated with one or more fully-connected (FC) layers <NUM> of, for example, a convolutional neural network. The FC layers <NUM> may be provided as a plurality of FC layers in number. The global transformation parameter may have dimensions of <NUM>×(V×param). Here, "param" denotes a dimension of a global transmission parameter for each viewpoint, and is <NUM> because a total of seven parameters is required. The seven parameters are calculated for each viewpoint and include a rotation parameter for each of three axes, a translation parameter for each of the three axes, and a scale parameter. In the example of <FIG>, V=<NUM>, and the dimension of the global transformation parameter is therefore <NUM>×<NUM>. However, examples are not limited to the foregoing, and the required number of parameters (param) may vary according to a global transformation algorithm. For example, when, as another model for a global transformation, a two-dimensional (2D) affine transformation model is used, six parameters may be required instead of seven.

The image restoration device estimates disparity information for each pixel with respect to each of the viewpoints by performing convolution filtering one or more times on the pooling data that is pooled from the input data <NUM>. For example, the image restoration device performs convolution filtering twice at <NUM> and <NUM> on the pooling data downsampled from the input data <NUM>. In the example of <FIG>, a result of the convolution filtering is feature data of an H/<NUM> × W/<NUM> ×<NUM> dimension. The image restoration device calculates the disparity information for each pixel with a same resolution as that of the input data <NUM> while upsampling <NUM> the result of the convolution filtering. For example, a dimension of the disparity information is H×W×(<NUM>×V). In the example of <FIG>, V=<NUM>, and thus the dimension of the disparity information is H×W×<NUM>. In this example, a total of two disparity values including a disparity on a horizontal axis (e.g., an x axis) and a disparity on a vertical axis (e.g., a y axis) for each of H×W pixels are calculated for each of V viewpoints.

A configuration of each layer of the warping model is not limited to the example illustrated in <FIG>, and the number of layers and dimensions may change according to design.

The image restoration device generates warped image information <NUM> by applying the warping operation <NUM> to input feature maps of the input data <NUM> using the global transformation parameter and the disparity information that are calculated as described above. Hereinafter, the warping operation <NUM> will be described in detail with reference to <FIG>.

<FIG> illustrates an example of a warping operation according to an example embodiment.

According to an example embodiment, an image restoration device may generate transformed image information by transforming input image information for each of viewpoints into a pixel coordinate system of target image information corresponding to a target viewpoint, using a global transformation parameter. The image restoration device may correct a disparity of the transformed image information with respect to the target image information, using disparity information. Hereinafter, a global transformation and a disparity correction will be described with reference to <FIG>.

The image restoration device may perform backward warping to warp backward all pixels of the input image information to a pixel coordinate system corresponding to a target image, using a single depth corresponding to a reference disparity. Hereinafter, an example of backward warping will be described in detail. For example, the image restoration device may calculate a coordinate in the input image information corresponding to a position that is based on the pixel coordinate system of the target image information, using a rotation parameter, a translation parameter, and a scale parameter that are included in the global transformation parameter. The image restoration device may determine a pixel value of the coordinate calculated in the input image information to be a pixel value at the position in the transformed image information. Hereinafter, the global transformation will be described in detail.

A disparity may indicate a difference between two images in terms of a position of a same target point, for example, a difference in pixel coordinate. According to an example embodiment, a disparity with the target image may be set to be the reference disparity for each input image. The reference disparity may be set as an arbitrary value. Based on the reference disparity, a virtual distance (e.g., a depth value) from an image sensor to a target point may be determined.

For example, as illustrated in <FIG>, image information of an ith viewpoint image among N viewpoint images may be warped to a pixel coordinate system corresponding to image information of a target viewpoint image, and thereby warped image information may be generated. As described above, the image restoration device may generate a warped feature map from the warped image information by warping a feature map extracted from the ith viewpoint image. In this case, the image information of the ith viewpoint image may indicate an ith feature map extracted from the ith viewpoint image by an ith sensing unit Ci <NUM>, and the image information of the target viewpoint image may indicate a target feature map extracted from the target viewpoint image by a target sensing unit CT <NUM>.

The world coordinate system described above may indicate a three-dimensional (3D) coordinate system having, as a reference, a point in a world. In addition, the camera coordinate system described above may indicate a 3D coordinate system having, as a reference, a position of a camera. For example, a principal point of a sensing unit may be used as an original point or a starting point, an optic axis direction of the sensing unit may be indicated by a z axis, a vertical direction of the sensing unit may be indicated by a y axis, and a horizontal direction of the sensing unit may be indicated by an x axis. A pixel coordinate system described herein may also be referred to as an image coordinate system and indicate a 2D coordinate of a pixel in an image.

For example, as illustrated, a world coordinate of a target point separated from an image sensor may be (X, Y, Z). In this example, a pixel coordinate corresponding to the target point that is sensed by the ith sensing unit Ci <NUM> among the N sensing units may be (u', v'), and a pixel coordinate corresponding to the target point that is sensed by the target sensing unit CT <NUM> may be (u, v). However, an accurate distance to the target point may not be readily determined only with a pixel value sensed by each sensing unit. According to an example embodiment, the image restoration device may assume that the input image information already has the reference disparity with respect to the target image information, and may warp the input image information to the pixel coordinate system of the target image information using a distance value corresponding to the assumed disparity. Here, backward warping may be performed.

The image restoration device may calculate a normalized coordinate (xT, yT) of the target sensing unit CT <NUM> by normalizing a pixel coordinate (uT, vT) of each pixel of the target image information, as represented by Equation <NUM> below.

In Equation <NUM>, <MAT> denotes a principal point of the target sensing unit CT <NUM> on the x axis and the y axis, respectively. <MAT> denotes a focal length of the target sensing unit CT <NUM> with respect to the x axis and they axis, respectively. As represented by Equation <NUM>, the image restoration device may normalize each pixel of the target sensing unit CT <NUM> by using a principal point of the target sensing unit CT <NUM> as an original point and dividing it by a focal length. Here, when principal point and focal length information of each sensing unit are unknown, a center position of an image may be used as a principal point and an arbitrary value may be used as a focal length.

In addition, the image restoration device may calculate a 3D camera coordinate (XT, YT, ZT) of the target sensing unit CT <NUM> using a single depth z corresponding to the reference disparity with respect to the normalized coordinate (xT, yT), as represented by Equation <NUM> below.

By multiplying each of the normalized coordinate (xT, yT) by the depth z, the image restoration device may obtain XT and YT as represented by Equation <NUM> above. Thus, the image restoration device may calculate the 3D camera coordinate having an optic axis of the target sensing unit CT <NUM> as a reference.

The image restoration device may transform the 3D camera coordinate of the target image information that is transformed using the disparity as represented by Equation <NUM> above into a 3D camera coordinate (Xi, Yi, Zi) of the ith sensing unit Ci <NUM>, as represented by Equation <NUM> below.

In Equation <NUM>, Ri denotes rotation information between the target sensing unit CT <NUM> and the ith sensing unit Ci <NUM> in the world coordinate system. Ti denotes parallel translation information between the target sensing unit CT <NUM> and the ith sensing unit Ci <NUM> in the world coordinate system. The rotation information may also be referred to as a rotation parameter, and the translation information may also be referred to as a translation parameter. As represented by Equation <NUM> above, the image restoration device may calculate the 3D camera coordinate (Xi, Yi, Zi) of an ith sensing unit Ci <NUM> corresponding to each pixel of the target sensing unit CT <NUM> by transforming the 3D camera coordinate (XT, YT, ZT) using a rotation parameter Ri and a translation parameter Ti.

The rotation parameter Ri may indicate a rotation matrix between the target sensing unit CT <NUM> and the ith sensing unit Ci <NUM> in the world coordinate system. The translation parameter Ti may indicate a translation matrix between the target sensing unit CT <NUM> and the ith sensing unit Ci <NUM> in the world coordinate system. The translation parameter Ti may include, for example, three parameters as represented by <MAT>. The rotation parameter Ri may be determined, for example, from three parameters <MAT>. The rotation parameter Ri may be calculated from the three parameters <MAT>, as represented by Equation <NUM> below.

In Equation <NUM>, α denotes a yawing rotation angle with respect to the z axis. β denotes a pitch rotation angle with respect to the y axis. γ denotes a rolling rotation angle with respect to the x axis.

In addition, the image restoration device may normalize the 3D camera coordinate (Xi, Yi, Zi) that is based on the ith sensing unit Ci <NUM> calculated from each pixel coordinate of the target image information, as represented by Equation <NUM> below.

As represented by Equation <NUM> above, the image restoration device may Xi and Yi by a depth Ziin the 3D camera coordinate that is based on the ith sensing unit Ci <NUM>, and thus obtain a normalized coordinate (xi, yi) for the ith sensing unit Ci <NUM>. Here, the image restoration device may consider a case in which sensing units have different focal lengths, and may multiply the normalized coordinate (xi, yi) by a scale parameter as represented by Equation <NUM> below.

The image restoration device may then calculate the pixel coordinate (u', v') of the pixel coordinate system corresponding to the ith image information from the normalized coordinate (xi, yi) of the ith sensing unit Ci <NUM>, as represented by Equation <NUM> below.

In Equation <NUM>, cx(i) and cy(i) denote principal points of the ith sensing unit Ci <NUM> with respect to the x axis and the y axis, respectively. In addition, fx(i) and fy(i) denote focal lengths of the ith sensing unit Ci <NUM> with respect to the x axis and the y axis, respectively.

Through Equations <NUM> through <NUM> above, the image restoration device may transform the pixel coordinate (u, v) of the targeting sensing unit CT <NUM> into the pixel coordinate (u', v') of the ith sensing unit Ci <NUM>. The image restoration device may determine a pixel value of a pixel coordinate (u, v) of warped image information Bi <NUM> to be the pixel value of the pixel coordinate (u', v') of the ith image information. In other words, the warped image information Bi <NUM> may associate the pixel value of the pixel coordinate (u', v') of the ith image information with the pixel coordinate (u, v) of the target sensing unit CT <NUM>. When the coordinate transformed from the pixel coordinate (u, v) of the target sensing unit CT <NUM> is not present in the ith image information, the image restoration device may perform padding, for example, zero padding. A series of coordinate transformations and operations of determining a pixel value in a transformed coordinate to be a value of warped image information based on Equations <NUM> through <NUM> above may be referred to as backward warping. Although the backward warping is described in a time series manner for the convenience of description, examples are not limited thereto. For example, an operation in which operations based on Equations <NUM> through <NUM> are combined, for example, a unified matrix operation, may be used.

Although the global transform is described above as being performed using a single depth, a depth of each pixel of input image information may differ. The image restoration device may additionally perform disparity correction in addition to the global transform (e.g., global warping). For example, the image restoration device may correct a disparity in each pixel with respect to the x axis and the y axis, as represented by Equation <NUM> below.

In Equation <NUM>, dT→i([xT,yT]T) denotes a disparity value from a current coordinate (xT, yT) to an ith sensing unit. An entire warping operation including Equation <NUM> above may be represented by Equation <NUM> below.

In Equation <NUM>, G([xT, yT]T) indicates a series of operations of Equations <NUM> through <NUM>.

<FIG> illustrates an example of training a feature extraction model, a warping model, and an image restoration model according to an example embodiment.

Referring to <FIG>, a feature extraction model <NUM>, a warping model <NUM>, and an image restoration model <NUM> may be a machine learning model and include a neural network, for example. For example, the feature extraction model <NUM> may include a neural network of a structure illustrated in <FIG>, the warping model <NUM> may include a neural network of a structure illustrated in <FIG>, and the image restoration model <NUM> may include a neural network of a U-net structure. The feature extraction model <NUM>, the warping model <NUM>, and the image restoration model <NUM> before training is completed may be referred to as a temporary feature extraction model <NUM>, a temporary warping model <NUM>, and a temporary image restoration model <NUM>, respectively.

A training device may train the temporary feature extraction model <NUM>, the temporary warping model <NUM>, and the temporary image restoration model <NUM> based on training data. The training data may include a pair of a training input and a training output. The training input may be low-resolution viewpoint images captured from a plurality of viewpoints, and the training output may be a high-resolution image of a single viewpoint corresponding to the training input. The training device may train the temporary feature extraction model <NUM>, the temporary warping model <NUM>, and the temporary image restoration model <NUM> such that the training output is output from the training input. For example, the training device may calculate at least one loss value while propagating the training input to the temporary feature extraction model <NUM>, the temporary warping model <NUM>, and the temporary image restoration model <NUM>, and update parameters (e.g., connection weights) of the models based on the calculated loss. The training device may update the connection weights of the models such that the loss value is reduced. The training device may repeatedly perform the training until the loss value becomes less than or equal to a threshold loss value. When the loss value is less than or equal to the threshold loss value , the training device may complete the training.

For example, as illustrated, the training device may apply, to the feature extraction model <NUM>, training viewpoint images <NUM> as the training input. The training device may propagate, to the warping model <NUM>, temporary feature maps extracted from the training viewpoint images <NUM> using the feature extraction model <NUM>. Here, a positional relationship between viewpoint images captured by a multi-camera or a multi-lens camera may be fixed or predetermined in general. Thus, a global transform on each viewpoint may need to be invariantly performed on an input image. As a constraint that allows a global transformation parameter for each viewpoint to have a same or similar value for each batch during the training, a loss value may be defined as represented by Equation <NUM> below.

Equation <NUM> above may define a geometric consistency loss lGeo <NUM>. In Equation <NUM>, B denotes a size of a batch used during the training. v denotes an index indicating a viewpoint, and is an integer greater than or equal to <NUM> and less than or equal to N. N denotes a total number of viewpoints. <MAT> denotes a global transformation parameter of a bth batch corresponding to a vth viewpoint image, in which b denotes an index indicating a batch and is greater than or equal to <NUM> and less than or equal to B. pv denotes a mean global transformation parameter of the vth viewpoint images of all batches. Thus, lGeo <NUM> may be a variance of global transformation parameters of the batches. The training device may update a parameter of the warping model <NUM> by minimizing the variance lGeo <NUM> such that a consistent global transformation parameter is output for each batch.

The training device may calculate temporary warped feature maps <NUM> from the temporary feature maps using the warping model <NUM>. The training device may transform the temporary warped feature maps <NUM> into low-resolution warped images <NUM>, for example, RGB images as color images, using a convolutional layer. The training device may calculate a low-resolution loss lLR <NUM> using the low-resolution color images and a training output image downsampled to a low resolution, as represented by Equation <NUM> below.

In Equation <NUM>, <MAT> denotes a temporary low-resolution warped image aligned from a vth viewpoint to a target viewpoint. ILR denotes an experimentally determined (i.e., ground truth) low resolution image obtained by downsampling a training output image. ∥ ∥ denotes an L2 norm. N denotes a total number of viewpoints. Through the training using the loss of Equation <NUM>, parameters of the models may be updated such that viewpoint images are aligned with a target viewpoint.

The training device may generate a temporary output image <NUM> having a target resolution from the temporary warped feature maps <NUM>. For example, the training device may perform a pixel shuffle <NUM> on the temporary warped feature maps <NUM> to obtain the target resolution, and generate the temporary output image <NUM> using the temporary image restoration model <NUM> of the U-net structure. The training device may calculate a high-resolution loss lHR <NUM> between the restored temporary output image <NUM> and the ground truth value image, as represented by Equation <NUM> below.

In Equation <NUM>, IHR denotes a training output image given as a ground truth value, and ISR denotes the temporary output image <NUM> restored to have a target resolution. Equation <NUM> may represent an L2 loss between the training output image and the temporary output image <NUM>.

A total loss of an entire image restoration network may be calculated using the losses represented in Equations <NUM> through <NUM>, as represented by Equation <NUM>.

In Equation <NUM>, λLR and λgeo denote a weight of lLR <NUM> of Equation <NUM> and a weight of the geometric consistency loss lGeo <NUM>, respectively. The training device may update the parameters of the feature extraction model <NUM>, the warping model <NUM>, and the image restoration model <NUM> such that the loss to be calculated based on Equation <NUM> is reduced. For example, the training device may minimize the total loss by updating a parameter of each model. For another example, the training device may repeatedly perform the training until the total loss becomes less than or equal to a threshold loss value by updating a parameter of each model, and complete the training when the total loss is less than or equal to the threshold loss value. Here, by the losses based on Equations <NUM> through <NUM> above, the training device may train the models such that viewpoint images are aligned without a camera parameter and a ground truth value of a depth value.

When the training is completed, the convolutional layer for transforming the temporary warped feature maps <NUM> into the low-resolution warped images <NUM> may be removed.

<FIG> illustrates examples of a result of image restoration according to an example embodiment. <FIG> shows input images <NUM>, globally transformed images <NUM>, and images <NUM> obtained through disparity correction.

Referring to <FIG>, there may be a misalignment between input images <NUM>, for example, viewpoint images, due to a disparity between sensing units. For example, as illustrated, a point <NUM> on a chessboard pattern in an image captured by a target sensing unit CT is a corner of a rectangular space. In this example, points <NUM> and <NUM> of a same pixel coordinate in images captured by a first sensing unit C<NUM> and a ninth sensing unit C<NUM> are different points on the chessboard pattern.

After the input images <NUM> are transformed to the globally transformed images <NUM>, a difference between a point <NUM> in an image captured by the target sensing unit CT and points <NUM> and <NUM> of a same pixel coordinate in images captured by the first sensing unit C<NUM> and the ninth sensing unit C<NUM> is reduced, but not eliminated, compared to that in the input images <NUM>.

In images <NUM> obtained through disparity correction in addition to the global transformation, a point <NUM> in an image captured by the target sensing unit CT and points <NUM> and <NUM> of a same pixel coordinate in images captured by the first sensing unit C<NUM> and the ninth sensing unit C<NUM> are matched. In other words, the remaining difference present in the globally transformed images <NUM> is eliminated by disparity correction. Although images are illustrated in <FIG> for the convenience of description, points of feature maps may be matched for each viewpoint in the examples described above with reference to <FIG>.

For example, when the global transformation is applied without the disparity correction, results of image restoration show that a peak signal-to-noise ratio (PSNR) is <NUM> decibels (dB) and a structural similarity (SSIM) is <NUM>. When the disparity correction is applied without the global transformation, results of image restoration show that PSNR = <NUM>. 50dB and SSIM = <NUM>. However, when both the global transformation and the disparity correction are applied to image restoration using models trained without a geometric consistency loss, results of image restoration show that PSNR = <NUM>. 37dB and SSIM = <NUM>. When both the global transformation and the disparity correction are applied to image restoration using models trained with a geometric consistency loss, results of image restoration show that PSNR = <NUM>. 67dB and SSIM = <NUM>. Compared to results of image restoration with residual channel attention networks (RCAN) showing that PSNR = <NUM>. 34dB and SSIM = <NUM>, and results of video restoration with enhanced deformable convolutional networks (EDVR) showing that PSNR = <NUM>. 66dB and SSIM = <NUM>, the image restoration method described herein may restore a clearer and higher-resolution image.

<FIG> illustrates an example of an image restoration device according to an example embodiment.

Referring to <FIG>, an image restoration device <NUM> includes an image sensor <NUM>, a processor <NUM>, and a memory <NUM>. However, the image restoration device <NUM> is not limited to the foregoing, and may be integrated into an image sensor or embodied by an imaging device including a lens array and an image sensor.

The image sensor <NUM> may obtain a plurality of sets of input image information. The image sensor <NUM> may obtain, as the sets of input image information, a plurality of viewpoint images captured through lenses arranged at different positions. The image sensor <NUM> may include sensing units configured to respectively obtain the sets of input image information. For example, the image sensor <NUM> may include N sensing units to obtain N sets of input image information. For example, the image sensor <NUM> including an MLA may capture a multi-lens image including a plurality of viewpoint images respectively corresponding to a plurality of viewpoints. The processor <NUM> may generate input data from the multi-lens image captured by the image sensor <NUM>. However, examples are not limited to the foregoing example that the single image sensor <NUM> includes the N sensing units, but each of N image sensors may include a single sensing unit. In such a case, each of the image sensors may capture a viewpoint image, and the processor <NUM> may generate input data from the captured viewpoint image. The image sensors may be embodied in a single device. However, examples are not limited to the foregoing, and the image sensors may be respectively provided in independently movable devices, for example, drones.

The processor <NUM> may obtain the input data including the input image information for each of the viewpoints. The processor <NUM> may estimate a global transformation parameter of each of the viewpoints from the obtained input data. The processor <NUM> may estimate disparity information of each of the viewpoints from the obtained input data. The processor <NUM> may generate warped image information for each of the viewpoints by warping the input image information using the global transformation parameter and the disparity information. The processor <NUM> may generate an output image for the viewpoints from the generated warped image information, using an image restoration model. The processor <NUM> may generate the output image, without sensing a depth to a target point corresponding to each pixel. The output image may have a target resolution that is greater than a resolution of each of the individual viewpoint images. For example, the target resolution may be greater than or equal to a resolution that is obtained by adding up all resolutions of the individual viewpoint images. However, the target resolution is not limited to the foregoing example, and may be less than or equal to the resolution that is obtained by adding all the resolutions of the viewpoint images.

The processor <NUM> is not limited to the operations described in the foregoing, but may perform simultaneously or sequentially one or more, or all of, the operations and methods described above with reference to <FIG>.

Cameras having different capturing directions and capturing positions may capture different scenes. For example, when CNN-based image restoration, for example, an RCAN, is performed on sets of viewpoint image information that are not aligned, an artifact may occur in a restored image. According to an example embodiment, the image restoration device <NUM> may restore an image based on a non-alignment of viewpoint images. Regions in an image may have different depths, and thus there may be a disparity for each of the regions in the image. Thus, a global transformation based on a single depth may not enable an accurate image alignment. The processor <NUM> may use the estimated disparity information in addition to the estimated global transformation parameter.

The memory <NUM> may temporarily or permanently store data required for performing the image restoration method described herein. For example, the memory <NUM> may store the viewpoint images, the input image information, the warped image information, and the output image. In addition, the memory <NUM> may store a feature extraction model and its parameters, a warping model and its parameters, and an image restoration model and its parameters. The parameters of each model may be trained in advance. As described above, the image restoration model may be a neural network including at least one convolutional layer that applies convolution filtering to input data, and a parameter of the model may be a connection weight.

<FIG> illustrates an example of a computing device according to an example embodiment.

A computing device <NUM> may be a device configured to generate a high-resolution image using the image restoration method described above. According to an example embodiment, the computing device <NUM> may correspond to the image restoration device <NUM> described above with reference to <FIG>. The computing device <NUM> may be a mobile terminal, for example, an image processing device, a smartphone, a wearable device, a tablet computer, a netbook, a laptop, a desktop, a personal digital assistant (PDA), a head-mounted display (HMD), a camera device, and the like. The computing device <NUM> may also be embodied by a vision camera device provided in, for example, a vehicle, a drone, a closed-circuit television (CCTV), and the like. In addition, the computing device <NUM> may be embodied by, for example, a webcam device for a video call, a virtual reality (VR) camera device for capturing a <NUM> degree (°) image, a VR and augmented reality (AR) camera device, and the like.

Referring to <FIG>, the computing device <NUM> includes a processor <NUM>, a storage <NUM>, a camera <NUM>, an input device <NUM>, an output device <NUM>, and a network interface <NUM>. The processor <NUM>, the storage <NUM>, the camera <NUM>, the input device <NUM>, the output device <NUM>, and the network interface <NUM> may communicate with one another through a communication bus <NUM>.

The processor <NUM> may execute functions and instructions in the computing device <NUM>. For example, the processor <NUM> may process instructions stored in the storage <NUM>. The processor <NUM> may perform one or more, or all, of operations described above with reference to <FIG>.

The storage <NUM> may store information or data needed for the processor <NUM> to process. The storage <NUM> may include a non-transitory computer-readable storage medium or device. The storage <NUM> may store instructions to be executed by the processor <NUM>, and store related information while software or an application is being executed by the computing device <NUM>.

The camera <NUM> may capture a plurality of input images. Although a still image is used herein to refer to an image, an image is not limited to the still image. The camera <NUM> may capture a plurality of images each including one or more image frames. For example, the camera <NUM> may generate a frame image corresponding to each of a plurality of lenses. In this example, the computing device <NUM> may generate a high-resolution output image of each frame from a plurality of input images respectively corresponding to individual frames, using a feature extraction model, a warping model, and an image restoration model that are described above.

The input device <NUM> may receive an input from a user through a tactile input, a video input, an audio input, or a touch input. The input device <NUM> may include, for example, a keyboard, a mouse, a touchscreen, a microphone, and other devices that may detect the input from the user and transmit the detected input.

The output device <NUM> may provide an output of the computing device <NUM> to a user through a visual, auditory, or tactile channel. The output device <NUM> may include, for example, a display, a touchscreen, a speaker, a vibration generator, and other devices that may provide the user with the output. The network device <NUM> may communicate with an external device through a wired or wireless network. According to an example embodiment, the output device <NUM> may provide the user with a result obtained by processing data using at least one of visual information, auditory information, or haptic information. For example, the computing device <NUM> may visualize a generated high-resolution output image through a display.

The units described herein may be implemented using hardware components and software components. For example, the hardware components may include microphones, amplifiers, band-pass filters, audio to digital convertors, non-transitory computer memory and processing devices. A processing device may be implemented using one or more general-purpose or special purpose computers, such as, for example, a processor, a controller and an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications that run on the OS. The processing device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processing device is used as singular; however, one skilled in the art will appreciate that a processing device may include multiple processing elements and multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such as parallel processors.

The software may include a computer program, a piece of code, an instruction, or some combination thereof, to independently or collectively instruct or configure the processing device to operate as desired. The non-transitory computer readable recording medium may include any data storage device that can store data which can be thereafter read by a computer system or processing device.

Example embodiments include non-transitory computer-readable media including program instructions to implement various operations embodied by a computer. The media may also include, alone or in combination with the program instructions, data files, data structures, tables, and the like. The media and program instructions may be those specially designed and constructed for the purposes of example embodiments, or they may be of the kind well known and available to those having skill in the computer software arts. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks; magneto-optical media such as floptical disks; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM) and random-access memory (RAM). The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described example embodiments, or vice versa.

The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner and/or replaced or supplemented by other components or their equivalents.

Claim 1:
An image restoration method comprising:
obtaining (<NUM>) input data comprising a respective input image (<NUM>) for each of a plurality of viewpoints;
estimating (<NUM>) a respective global transformation parameter of each of the plurality of viewpoints based on the obtained input data;
estimating (<NUM>) respective disparity of each of the plurality of viewpoints based on the obtained input data;
generating (<NUM>) respective warped image (<NUM>) for each of the plurality of viewpoints by warping the respective input image (<NUM>) using the respective global transformation parameter and the respective disparity; and
generating (<NUM>), using an image restoration model (<NUM>), an output image (<NUM>) from the generated respective warped image (<NUM>) for each of the plurality of viewpoints, wherein the estimating (<NUM>) the respective global transformation parameter comprises performing a global pooling operation (<NUM>) for removing a spatial dimension component from the input data, and obtaining the respective global transformation parameter based on the input data having removed the spatial dimension component.