Method and apparatus for image processing

At least one example embodiment discloses an imaging apparatus that may receive input images with different viewpoints, obtain information associated with the input images, and generate a new viewpoint image based on the received input images and the obtained information and thus, reduce an information loss of the new viewpoint image for the input images.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of Korean Patent Application No. 10-2014-0009078, filed on Jan. 24, 2014, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference.

BACKGROUND

Example embodiments of the following description relate to image processing, and more particularly, to a method and an apparatus for image processing that may generate a new viewpoint image based on input images.

2. Description of the Related Art

A new viewpoint image of multiview input images may be generated through interpolation or extrapolation performed on the input images.

Cameras capturing the input images may be positioned in different environments. Thus, to generate the new viewpoint image through interpolation or extrapolation using the input images, calibration and warping may be performed on each input image based on the environments in which the cameras capture the input images. The new viewpoint image may be generated from the calibrated and warped images based on the environments in which the cameras capture the input images.

The calibrated and the warped images may include a black region. To remove the black region, the new viewpoint image generated from the calibrated and the warped images may be cropped. However, the cropping may cause information loss when generating the new viewpoint image.

SUMMARY

The foregoing and/or other aspects are achieved by providing an image processing method including calculating a motion vector of points corresponding to input images with different viewpoints based on homography information and disparity information on the input images, the homography information being based on calibration information and rectification information on the input images, and generating a new viewpoint image from the input images based on the motion vector.

The calibration information may include rotation information for rectification of the input images.

The rectification information may include parameters associated with rectified input images.

The input images may include a left input image and a right input image.

A viewpoint part of the new viewpoint image is between a viewpoint part of the left input image and a viewpoint part of the right input image.

The input images may include a first input image and a second input image.

The homography information includes first homography information on the first input image and second homography information on the second input image.

The calculating of the motion vector includes calculating a second point of a first rectified image corresponding to a first point of the first input image based on the first homography information.

The calculating of the motion vector may include calculating a third point of a second rectified image corresponding to the second point based on the disparity information.

The calculating of the motion vector may include calculating a fourth point of the second input image corresponding to the third point based on the second homography information.

The calculating of the motion vector may include calculating the motion vector, the motion vector indicating a motion between the first point and the fourth point.

The image processing method may further include obtaining the calibration information, the rectification information, and the disparity information.

The obtaining the calibration information, the rectification information, and the disparity information may include receiving at least one set of information among the calibration information, the rectification information, and the disparity information, and generating remaining information, excluding the at least one received set of information, based on the at least one received set of information and the input images.

The obtaining the calibration information, the rectification information, and the disparity information may include receiving the calibration information.

The obtaining the calibration information, the rectification information, and the disparity information may include generating the rectification information by performing the rectification on the input images based on the calibration information and the input images.

The obtaining the calibration information, the rectification information, and the disparity information may include obtaining the disparity information by calculating disparities based on the rectification.

The image processing method may further include calculating an information loss of cropped images for the input images.

The cropped images may be obtained by cropping performed concurrently with the rectification.

The calculating the motion vector and the generating the new viewpoint image may be performed when the information loss is greater than or equal to a value.

The foregoing and/or other aspects are achieved by providing an imaging apparatus including a motion vector calculator configured to calculate a motion vector of points corresponding to input images with different viewpoints based on homography information and disparity information on the input images, the homography information being based on calibration information and rectification information on the input images, and an image generator to generate a new viewpoint image from the input images based on the motion vector.

The calibration information may include rotation information for rectification of the input images.

The input images may include a first input image and a second input image.

The motion vector calculator is configured to calculate a second point of a first rectified image corresponding to a first point of the first input image based on a part of the homography information corresponding to the first input image.

The motion vector calculator calculates a third point of a second rectified image corresponding to the second point based on the disparity information.

The motion vector calculator may calculate a fourth point of the second input image corresponding to the third point based on another part of the homography information corresponding to the second input image.

The motion vector calculator may calculate the motion vector indicating a motion between the first point and the fourth point. The imaging apparatus may further include an information acquirer configured to obtain the homography information on the input images based on the calibration information and the rectification information.

The information acquirer is configured to obtain the calibration information, the rectification information, and the disparity information.

The information acquirer may obtain the homography information based on the obtained calibration information and the obtained rectification information.

The information acquirer is configured to receive at least one set of information among the calibration information, the rectification information, and the disparity information.

The information acquirer may generate remaining information, excluding the at least one received set of information, based on the at least one received set of information and the input images.

The information acquirer is configured to receive the calibration information.

The information acquirer may generate the rectification information by performing rectification on the input images based on the calibration information and the input images.

The information acquirer is configured to obtain the disparity information by calculating disparities of images generated by performing the rectification.

The information acquirer is configured to calculate an information loss of cropped images for the input images.

The cropped images based on cropping performed concurrent with the rectification.

If the information loss is greater than or equal to a value, the motion vector calculator is configured to calculate the motion vector, and the image generator is configured to generate the new viewpoint image.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference to the accompanying drawings. Many alternate forms may be embodied and example embodiments should not be construed as limited to example embodiments set forth herein. In the drawings, like reference numerals refer to like elements.

As disclosed herein, the term “storage medium”, “computer readable storage medium” or “non-transitory computer readable storage medium” may represent one or more devices for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other tangible machine readable mediums for storing information. The term “computer-readable medium” may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other mediums capable of storing, containing or carrying instruction(s) and/or data.

An image or images described herein may refer to data of the image or the images. For example, receiving, processing, and outputting the image or the images may indicate receiving, processing, and outputting the data of the image or the images, respectively.

FIG. 1is a diagram illustrating an imaging apparatus100according to example embodiments.

The imaging apparatus100may include an image receiver110, an image processor120, and an image outputter130.

The imaging apparatus100may receive input images with different viewpoints and obtain information associated with the input images.

The imaging apparatus100may generate a new viewpoint image based on the received input images and the obtained information associated with the input images. A viewpoint of the new viewpoint image may differ from the viewpoints of the input images. Thus, the imaging apparatus100may generate an image with a new viewpoint.

The generated new viewpoint image and the input images may be used to generate a three-dimensional (3D) image. For example, the imaging apparatus100may be included in a multiview 3D display device or a portion of the multiview 3D display device.

The imaging apparatus100may generate the new viewpoint image based on original input images on which calibration and warping are not performed, namely, not based on calibrated and/or wrapped input images. The warping of the input images may indicate rectification of the input images. The rectification may indicate a method of processing the input images to parallelize epipolar lines of the input images.

The image receiver110may receive the input images with different viewpoints. The input images may be received from an external device in lieu of the imaging apparatus100. For example, the input images may be received from a stereo camera or at least one camera. Alternatively, although not illustrated, the input images may refer to images stored in a storage (not shown) included in the imaging apparatus100.

The input images with different viewpoints may refer to images captured by the stereo camera. Also, the input images may refer to images captured by cameras positioned at different locations.

The image receiver110may transmit the received input images to the image processor120.

Although not illustrated, the image receiver110may receive the information associated with the input images from an external source. The image receiver110may transmit the information associated with the received input images to the image processor120. The information associated with the input images may include at least one set of information among calibration information on the input images, rectification information on the input images, and disparity information on the input images. For example, the information associated with the input images may include the calibration information on the input images, the rectification information on the input images, and the disparity information on the input images.

The image processor120may obtain the information associated with the input images and generate the new viewpoint image based on the input images and the information associated with the input images. The image processor120may obtain the information associated with the input images. The information may be received and generated based on the input images. Thus, the obtaining of the information may include reception of the information and generation of the information.

The image processor120may process calculations required to obtain the information associated with the input images and calculations required to generate the new viewpoint image.

The image outputter130may output the new viewpoint image generated by the image processor120. The image outputter130may be a hardware module used to output the new viewpoint image. The image outputter130may include a port used to output the new viewpoint image. For example, the image outputter130may output the new viewpoint image to the multiview 3D display device.

Components included in the image processor120and functions of the components will be further described with reference toFIGS. 2 through 9.

FIG. 2is a diagram illustrating the image processor120including an information acquirer210, a motion vector calculator220, and an image generator230according to example embodiments.

Referring toFIG. 2, the image processor120includes the information acquirer210, the motion vector calculator220, and the image generator230.

The information acquirer210, the motion vector calculator220and the image generator230may be hardware, firmware, hardware executing software or any combination thereof. When at least one of the information acquirer210, the motion vector calculator220and the image generator230is hardware, such existing hardware may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits (ASICs), field programmable gate arrays (FPGAs) computers or the like configured as special purpose machines to perform the functions of the at least one of the information acquirer210, the motion vector calculator220and the image generator230. CPUs, DSPs, ASICs and FPGAs may generally be referred to as processors and/or microprocessors.

In the event where at least one of the information acquirer210, the motion vector calculator220and the image generator230is a processor executing software, the processor is configured as a special purpose machine to execute the software, stored in a storage medium, to perform the functions of the at least one of the information acquirer210, the motion vector calculator220and the image generator230. In such an embodiment, the processor may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits (ASICs), field programmable gate arrays (FPGAs) computers.

The information acquirer210may obtain information associated with input images. For example, the information associated with the input images may include calibration information on the input images, rectification information on the input images, and disparity information on the input images. Alternatively, the information acquirer210may obtain homography information on the input images based on the calibration information and the rectification information on the input images.

The motion vector calculator220may calculate a motion vector of points corresponding to the input images based on the homography information obtained based on the calibration information and the rectification information on the input images and the disparity information on the input images.

The image generator230may generate an image with a new viewpoint image from the input images based on the generated motion vector.

The generated new viewpoint image may be output through the image outputter130.

The information associated with the input images obtained by the information acquirer210and a method of generating the new viewpoint image by the motion vector calculator220and the image generator230will be further described with reference toFIGS. 3 through 9.

FIG. 3is a flowchart illustrating an image processing method according to example embodiments.

The image processing method to be described with reference toFIG. 3may be performed by the image processor120of the imaging apparatus100described with reference toFIG. 1.

In operation310, the image receiver110receives input images with different viewpoints. For example, the input images may be multiview input images. The input images may include a left input image and a right input image. A viewpoint of the left input image among the input images may be present further to a leftward direction than a viewpoint of the right input image.

In operation320, the information acquirer210of the image processor120obtains information associated with the input images, for example, calibration information on the input images, rectification information on the input images, and disparity information on the input images.

The information acquirer210may obtain calibration information on each input image.

Calibration of an input image may be an operation performed to obtain calibration parameters required to project an image captured by a camera on a two-dimensional (2D) plane.

The calibration information obtained by the information acquirer210may include the calibration parameters.

The calibration parameters may include extrinsic camera parameters and intrinsic camera parameters.

The intrinsic camera parameters may include parameters associated with at least one of a focal distance, an aspect ratio, and a central point of a camera capturing each input image.

The extrinsic camera parameters may include parameters associated with a geometric position of each camera in a space in which each input image is captured, for example, a height and a direction of the camera.

For example, the calibration information obtained by the information acquirer210may include rotation information for rectification of the input images. The rotation information for the rectification of the input images may include parameters required to perform the rectification, for example, parameters associated with an angle at which each input image is required to rotate. The rotation information on each input image may be determined based on the geometric position of each camera in the space in which each input image is captured.

The calibration information on each input image may be a matrix including the calibration parameters as elements. For example, the calibration information on each input image may indicate a rotation matrix including the parameters associated with the angle at which each input image is required to rotate as the elements. The rotation matrix of each input image may be used as the calibration information to obtain a homography matrix used to generate the new viewpoint image to be described hereinafter.

The rectification information may be obtained by performing the rectification on the input images. The rectification of the input images may be performed based on the input images and the calibration information.

The rectification information may include parameters associated with rectified input images. For example, the rectification information may include rotation information shared by the rectified input images. The rotation information may include rotation parameters shared by the rectified input images.

The rectification information may be a rectification matrix including the rotation parameters shared by the rectified input images as elements. The rectification matrix of the input images may be used as the rectification information to obtain the homography matrix used to generate the new viewpoint image to be described hereinafter.

The calibration information and the rectification information obtained by the information acquirer210will be further described with reference toFIG. 6.

The disparity information on the input images obtained by the information acquirer210may be obtained based on the rectified input images. For example, a dense stereo matching algorithm, a local matching algorithm, or a global matching algorithm may be used to calculate disparities for obtaining the disparity information.

Alternatively, the disparity information on each input image may be obtained by a sensor when generating the input images.

The information acquirer210may obtain at least one set of information among the calibration information on the input images, the rectification information on the input images, and the disparity information on the input images by receiving the at least one set of information from an external source. For example, the at least one set of information among the calibration information, the rectification information, and the disparity information may be generated from the external source, for example, an external device, and transmitted to the information acquirer210subsequent to the generation.

The information acquirer210may obtain one or two sets of information among the calibration information, the rectification information, and the disparity information by receiving the one or two sets of the information. The information acquirer210may obtain remaining information, excluding the received one or two sets of information, by generating the remaining information based on the received one or two sets of information and the input images. For example, the image processor120may externally receive the calibration information on the input images, and generate the rectification information and the disparity information based on the received calibration information and the input images.

A method of obtaining the rectification information and the disparity information on the input images by the information acquirer210will be further described with reference toFIG. 5.

In operation325, the motion vector calculator220of the image processor120calculates a motion vector of points corresponding to the input images based on the homography information obtained based on the calibration information and the rectification information on the input images and the disparity information on the input images. The calibration information and the rectification information may be obtained by the information acquirer210. Operation325may include operations330and340to be described hereinafter.

In operation330, the information acquirer210obtains homography information on each of the input images based on the calibration information and the rectification information on the input images.

The homography information on each input image may be the homography matrix used to convert an input image to a rectified image. A relationship among the homography matrix, a point of the input image, and a point of the rectified image corresponding to the input image may be represented by Equation 1.

In Equation 1, “H” may denote a homography matrix of an input image. “x” and “y” may denote an x coordinate and a y coordinate, respectively, of an arbitrary point of the input image projected on a two-dimensional (2D) plane. “xrec” and “yrec” may denote an x coordinate and a y coordinate, respectively, of a point of a rectified image corresponding to the coordinates of the arbitrary point of the input image. “c” may denote a scalar value, and a value that may allow 3×1 matrices on both sides indicating coordinate values of the point of the input image and coordinate values of the point of the rectified image to correspond to one another. For example, the homography matrix may transform the coordinate of the point of the input image into the coordinate of the point of the rectified image.

When the input images received by the image receiver110include a left input image and a right input image, a homography matrix of the left input image and a homography matrix of the right input image may be calculated based on Equations 2 and 3.

In Equation 2, “RL” may indicate calibration information on the left input image. Also, RLmay denote a rotation matrix for rectification of the left input image. “R” may indicate rectification information on the left input image and the right input image. Also, R may denote a rectification matrix. “HL” may denote the homography matrix of the left input image.

In Equation 3, “RR” may indicate calibration information on the right input image. Also, RRmay denote a rotation matrix for rectification of the right input image. “R” may indicate the rectification information on the left input image and the right input image. Also, R may denote a rectification matrix. “HR” may denote the homography matrix of the right input image.

The homography matrix obtained based on Equation 2 and Equation 3 may be used as the homography information to calculate a motion vector for generating the new viewpoint image to be described along with the disparity information on the input images.

In operation340, the motion vector calculator220calculates the motion vector of the points corresponding to the input images based on the homography information obtained in operation330and the input images. The motion vector calculator220calculates the motion vector of the coordinates of the points corresponding to the input images.

The points corresponding to the input images may indicate positions of corresponding objects in the input images, or coordinates of the corresponding objects in the images. The corresponding objects may indicate a single identical subject in the input images. The single identical subject may appear at different positions and in different forms in the input images based on viewpoints of the input images.

The points corresponding to the input images may correspond to positions or coordinates of pixels at which the input images are output. The points corresponding to the input images may indicate pixels of the input images that may indicate a single position in a scene. The single position in the scene or the points may be indicated as the pixels at different positions and of different colors in the input images based on the viewpoints of the input images. For example, the motion vector may be calculated based on the positions of the pixels at which the input images indicating the corresponding objects are output.

The motion vector may indicate a vector of a motion of the points corresponding to the input images. For example, the motion vector may be the vector of the motion of the positions of the corresponding objects in the input images or the coordinates of the corresponding objects in the input images. The motion of the positions may indicate a change of the positions.

For example, when the input images include the left input image and the right input image, the motion vector may be a vector indicating the change of the positions, or the motion, of corresponding objects in the left input image and the right input image, for example, the change of the positions from a position of the corresponding objects in the left input image to a position of the corresponding objects in the right input image. Alternatively, the motion vector may be a vector indicating a change of the coordinates, or the motion, of corresponding pixels in the left input image and the right input image, for example, the change of the coordinates from a coordinate of the corresponding pixels in the left input image to a coordinate of the corresponding pixels in the right input image.

Relationships among the foregoing images, the positions, and the pixels may be identically applicable to the following descriptions.

A method of calculating the motion vector of the points corresponding to the input images by the motion vector calculator220based on the homography information and the input images will be further described with reference toFIGS. 4 and 7.

In operation350, the image generator230of the image processor120generates the new viewpoint image from the input images based on the motion vector calculated in operation340. When the input images include the left input image and the right input image, a viewpoint of the new viewpoint image may be present between a viewpoint of the left input image and a viewpoint of the right input image. The viewpoint of the new viewpoint image may be present further toward a leftward direction of the viewpoint of the left input image or present further toward a rightward direction of the viewpoint of the right input image.

A method of generating the new viewpoint image by the image generator230will be further described with reference toFIG. 8.

In operation360, the image outputter130outputs the new viewpoint image generated in operation350.

The new viewpoint image may be used to generate a 3D image. For example, the image outputter130may output the new viewpoint image and the input images to the multiview 3D display device. A viewer of the multiview 3D display device may recognize the 3D image through the new viewpoint image and multiview images of the input images.

Technical descriptions provided with reference toFIGS. 1 and 2may be applicable hereto and thus, repeated descriptions will be omitted here for brevity.

FIG. 4is a flowchart illustrating a method of calculating a motion vector according to example embodiments.

Operation340described with reference toFIG. 3may include operations410through440to be described hereinafter with reference toFIG. 4.

Input images received by the image receiver110may include a first input image and a second input image. One of the first input image and the second input image may correspond to the left input image described with reference toFIG. 3and the other may correspond to the right input image described with reference toFIG. 3.

In operation330, the information acquirer210obtains first homography information on the first input image and second homography information on the second input image based on calibration information on the input images and rectification information on the input images. For example, the obtained homography information may include homography information on the first input image and homography information on the second input image.

The first homography information may be a homography matrix of the first input image, and the second homography information may be a homography matrix of the second input image.

The motion vector calculator220may calculate a motion vector indicating a motion between a first point of the first input image and a fourth point of the second image based on the obtained first homography information and the obtained second homography information and the first input image and the second input image. The fourth point may be a point corresponding to the first point.

In operation410, the motion vector calculator220calculates a second point of a first rectified image corresponding to the first point of the first input image based on the first homography information. The first rectified image may be a rectified image of the first input image. For example, the motion vector calculator220may calculate a coordinate of the second point of the first rectified image by multiplying a matrix indicating a coordinate of the first point of the first input image and a first homography matrix together. The second point may be a point corresponding to the first point.

In operation420, the motion vector calculator220calculates a third point of a second rectified image corresponding to the second point based on the disparity information on the first input image and the second input image. For example, the motion vector calculator220may calculate a coordinate of the third point of the second rectified image from the coordinate of the second point based on a disparity between the first rectified image and the second rectified image included in the disparity information. The third point may be a point corresponding to the second point.

Epipolar lines of the first rectified image and the second rectified image may be parallel to one another and thus, the second point and the third point may be different from each other only in x-axis coordinate values.

The second rectified image may correspond to a rectified image of the second input image.

In operation430, the motion vector calculator220calculates the fourth point of the second input image corresponding to the third point based on the second homography information on the second input image. For example, the motion vector calculator220may calculate the fourth point of the second input image by multiplying an inverse matrix of a second homography matrix and a matrix indicated by the coordinate of the third point together.

The first point and the fourth point may indicate points corresponding to the first input image and the second input image, respectively. Also, the first point and the fourth point may be points indicating each corresponding objects in the first input image and the second input image.

In operation440, the motion vector calculator220calculates the motion vector indicating a motion between the first point and the fourth point.

The method of calculating the motion vector will be further described with reference toFIG. 7.

The image generator230may generate a new viewpoint image based on the motion vector calculated based on the first input image, the second input image, and the motion between the first point and the fourth point.

Technical descriptions provided with reference toFIGS. 1 through 3may be applicable hereto and thus, repeated descriptions will be omitted here for brevity.

FIG. 5is a flowchart illustrating a method of obtaining calibration information, rectification information, and disparity information according to example embodiments.

Operation320described with reference toFIG. 3may include operations510through530to be described hereinafter with reference toFIG. 5.

As described with reference toFIG. 3, one or two sets of information among the calibration information, the rectification information, and the disparity information may be obtained by receiving the one or two sets of information, and remaining information excluding the one or two sets of information may be obtained by generating the remaining information based on the received one or two sets of information and input images.

In operation510, the information acquirer210receives the calibration information. For example, the information acquirer210may receive the calibration information from an external device.

Also, the calibration information may be transmitted to the image receiver110along with the input images from the external device or to the information acquirer210separately from the input images from the external device.

In operation520, the information acquirer210generates the rectification information by performing rectification on the input images based on the calibration information and the input images. For example, the information acquirer210may perform the rectification on the input images based on calibration parameters included in the calibration information, and perform the rectification to obtain a rectification matrix which is the rectification information.

In operation530, the information acquirer210obtains the disparity information by calculating disparities of images generated by performing the rectification. The images generated by performing the rectification may be rectified input images. Epipolar lines of the rectified images may be parallel to one another and thus, a line by line disparity between the rectified images may be calculated.

However, when the information acquirer210receives the disparity information on the input images from the external device, for example, a sensor, operation530may not be performed.

The calibration information and the rectification information obtained by the information acquirer210will be further described with reference toFIG. 6.

Technical descriptions provided with reference toFIGS. 1 through 4may be applicable hereto and thus, repeated descriptions will be omitted here for brevity.

FIG. 6illustrates examples of calibration information on input images and rectification information on rectified images according to example embodiments.

As described with reference toFIG. 3, the input images may include a left input image610and a right input image620. The left input image610and the right input image620may be tilted at “θL” and “θR” respectively, compared to rectified images, for example,630and640. θLand θRmay be determined based on extrinsic camera parameters of each camera capturing the left input image610and the right input image620.

The calibration information on the left input image610may correspond to a rotation matrix “RL” used to rotate the left input image610at −θL. Similarly, the calibration information on the right input image620may correspond to a rotation matrix “RR” used to rotate the right input image620at θR.

A rectified left input image630may be a rectified image of the left input image610. Also, a rectified right input image640may be a rectified image of the right input image620.

Subsequent to the rectification being performed as illustrated, the epipolar lines of the rectified left input image530and the rectified right input image640may become parallel to one another.

The rectification information on the left input image610and the right input image620may include a rectification matrix “R.” The rectification matrix R may indicate a matrix in which the rotation matrix RLis rotated at −θLor the rotation matrix RRis rotated at θR.

Calibration information “RL” on the left input image610, calibration information “RR” on the right input image620, and the rectification information R may be used to obtain homography information on each of the left input image610and the right input image620. The homography information may be used to calculate a motion vector used to generate a new viewpoint image.

Technical descriptions provided with reference toFIGS. 1 through 5may be applicable hereto and thus, repeated descriptions will be omitted here for brevity.

FIG. 7illustrates a method of calculating a motion vector according to example embodiments.

An example of the method of calculating the motion vector is described in operations410through430provided with reference toFIG. 4.

One of the left input image610ofFIG. 6and the right input image620ofFIG. 6may correspond to the first input image ofFIG. 4, and a remaining image between the left input image610and the right input image620may correspond to the second input image ofFIG. 4.

The method of calculating the motion vector described in operations410through430may be represented by Equation 4.
imL(xL,yL)→imL(xLrec,yLrec)→imL(xLrec+DisLrec(xLrec,yLrec),yLrec)→imR(xR,yR)  [Equation 4]

In Equation 4, “imL(xL, yL)” may indicate a coordinate of a first point710. imL(xLrec,yLrec) may indicate a coordinate of a second point720. An arrow between imL(xL,yL) and imL(xLrec, yLrec) may indicate calculation performed in operation410described with reference toFIG. 4. imL(xLrec+DisLrec(xLrec, yLrec), yLrec) may indicate a coordinate of a third point730. An arrow between imL(xLrec, yLrec) and imL(xLrec+DisLrec(xLrec, yLrec), yLrec) may indicate calculation performed in operation420. DisLrec(xLrec, yLrec) may indicate a disparity between points corresponding to the rectified left input image630ofFIG. 6and the rectified right input image640ofFIG. 6, respectively, which is included in disparity information on the left input image610and the right input image620. “imR(xR,yR)” may indicate a coordinate of a fourth point740. An arrow between imL(xLrec+DisLrec(xLrec, yLrec), yLrec) and imR(xR,yR) may indicate calculation performed in operation430.

The arrows in Equation 4 may be inversely indicated. For example, a coordinate of a point of the left input image610corresponding to a coordinate of an arbitrary point of the right input image620may be calculated. A motion vector of the points corresponding to the left input image610and the right input image620may be calculated based on the calculation of the coordinates.

The motion vector calculated based on Equation 4 may be represented by Equation 5.

The image generator230may generate a new viewpoint image based on 1) the motion vector represented by Equation 5 and 2) the left input image610and the right input image620. A method of generating the new viewpoint image will be further described with reference toFIG. 8.

The motion vector calculator220may calculate the motion vector indicating a motion between the coordinate of the first point710and the coordinate of the fourth point740. The motion vector calculator220may calculate coordinates of the right input image620corresponding to all coordinates of the left input image610, and calculate motion vectors of the all points of the left input image610and the points of the right input image620corresponding to the all points of the left input image610.

The rectified left input image630and the rectified right input image640may be illustrated only to describe the method of calculating the motion vector performed in operations410through430. Although not illustrated, the rectified left input image630and the rectified right input image640may not be output through the imaging apparatus100ofFIG. 1.

Technical descriptions provided with reference toFIGS. 1 through 6may be applicable hereto and thus, repeated descriptions will be omitted here for brevity.

FIG. 8illustrates a method of generating a new viewpoint image according to example embodiments.

The method of generating the new viewpoint image based on a motion vector represented by Equation 5 provided with reference toFIG. 7will be further described hereinafter.

As described with reference toFIG. 3, a viewpoint of the new viewpoint image may be present between a viewpoint of the left input image610ofFIG. 6and a viewpoint of the right input image620ofFIG. 6. Alternatively, the viewpoint of the new viewpoint image may be present on a more left side than the viewpoint of the left input image610or present on a more right side than the viewpoint of the right input image620.

The image generator230may generate, based on the motion vector, an interpolated image and an extrapolated image of the left input image610and the right input image620of input images.

An image plane810may indicate a plane of a screen to which each of the input images and new viewpoint images is output or a plane of a screen of a display device. As illustrated, objects corresponding to the input images and the new viewpoint images may be output at different positions on the image plane810based on a viewpoint of each of the input images and the new viewpoint images.

Objects marked with a line may indicate the left input image610and the right input image620of the input images. Also, objects marked with a dotted line may indicate the new viewpoint images generated by interpolation and extrapolation performed on the left input image610and the right input image620.

As illustrated, a coordinate of an interpolation point820of a new viewpoint image corresponding to the first point710ofFIG. 7and the fourth point740ofFIG. 7may be located at a distance “α” apart from a coordinate imL(xL, yL) of the first point710and a distance “β” apart from a coordinate imR(xR, yR) of the fourth point740. The coordinate of the interpolation point820may be represented by Equation 6.

The motion vector calculator220may use motion vectors of all points corresponding to the left input image610and the right input image620, and calculate all interpolation points of the new viewpoint image corresponding to the all points as represented by Equation 6. The image generator230may generate the interpolated image of the left input image610and the right input image620based on all the calculated interpolation points.

Similarly, a coordinate of an extrapolation point830of the new viewpoint image corresponding to the first point710may be located at a distance “α” apart from the coordinate imL(xL, yL) of the first point710. The coordinate of the extrapolation point830of the new viewpoint image corresponding to the first point710may be represented by Equation 7.

Also, a coordinate of an extrapolation point840of the new viewpoint image corresponding to the fourth point740may be located at a distance “α” apart from the coordinate imR(xR, yR) of the fourth point740. The coordinate of the extrapolation point840of the new viewpoint image corresponding to the fourth point740may be represented by Equation 8.

The motion vector calculator220may use motion vectors of all points corresponding to the left input image610and the right input image620, and calculate all the extrapolation points of the new viewpoint image corresponding to the all points as represented by Equations 7 and 8. The image generator230may generate the extrapolated images of the left input image610and the right input image620based on all the calculated extrapolation points.

Technical descriptions provided with reference toFIGS. 1 through 7may be applicable hereto and thus, repeated descriptions will be omitted here for brevity.

FIG. 9is a flowchart illustrating an image processing method according to example embodiments.

In operation910, subsequent to operation320described with reference toFIG. 3being performed, the information acquirer210calculates an information loss of cropped images for input images. The cropped images may be generated by cropping performed concurrent with rectification performed on the input images. The cropped images may be obtained from rectified input images and used to generate a new viewpoint image.

For example, when the new viewpoint image is generated based on the cropped images of the rectified input images, the information acquirer210may calculate the information loss of the cropped images for the input images. The information loss of the cropped images in operation910may indicate an information loss occurring between the input images and the cropped images.

The information loss of the cropped images for the input images may be calculated based on a size of a black region generated by performing the rectification on the input images or information quantity of a portion cropped from the rectified images.

In operation920, the information acquirer210determines whether the information loss of the cropped images is greater than or equal to a predetermined and/or selected value.

Operations325through350described with reference toFIG. 3may be selectively performed when the information loss of the cropped images is greater than or equal to the predetermined and/or selected value.

In operation930, when the information loss of the cropped images is less than the predetermined and/or selected value, the image generator230generates a new viewpoint image based on the cropped images.

Operations910through930may be selectively performed. Operations910through930may not be performed to prevent the information loss of the new viewpoint image for the input images.

Technical descriptions provided with reference toFIGS. 1 through 8may be applicable hereto and thus, repeated descriptions will be omitted here for brevity.

The above-described example embodiments may be recorded in non-transitory computer-readable media including program instructions to implement various operations embodied by a computer. The media may also include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the media may be those specially designed and constructed for the purposes of example embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. The non-transitory computer-readable media may also be a distributed network, so that the program instructions are stored and executed in a distributed fashion. The program instructions may be executed by one or more processors. The non-transitory computer-readable media may also be embodied in at least one application specific integrated circuit (ASIC) or Field Programmable Gate Array (FPGA), which executes (processes like a processor) program instructions. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The above-described devices may be configured to act as one or more software modules in order to perform the operations of the above-described example embodiments, or vice versa.