IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND PROGRAM

With a wide angle-of-view image acquired by an image pickup unit 21-1 as a reference, a signal processing unit 30 performs super-resolution processing by using a plurality of narrow angle-of-view images acquired by an image pickup unit 21-2 that uses a lens having a higher MTF (Modulation Transfer Function) than the image pickup unit 21-1. A control unit 60 controls the signal processing unit 30 so as to select an image having a range of angle of view according to a zoom magnification indicated by user's operation from the image subjected to the super-resolution processing. In the super-resolution processing, according to a detection result of parallax from a narrow angle-of-view image and the wide angle-of-view image acquired at the same time and a motion detection result for each of the plurality of narrow angle-of-view images, parallax compensation and motion compensation are performed on the plurality of narrow angle-of-view images. A captured image beyond the performance of the image pickup unit can be acquired.

TECHNICAL FIELD

This technique relates to an image processing device, an image processing method, and a program, and enables a zoom operation to be performed seamlessly from a wide-angle view to a telephoto view without degrading image quality.

BACKGROUND ART

In the past, in an information processing terminal such as a portable electronic device like a smartphone, due to downsizing and thinning, the image quality of an image pickup unit is lower than that of a single-lens reflex camera or the like. For this reason, for example, PTL 1 discloses that a plurality of image pickup units is provided to simultaneously generate a plurality of images having different image qualities such as images of a first angle of view and a second angle of view narrower than the first angle of view.

CITATION LIST

Patent Literature

SUMMARY

Technical Problem

Meanwhile, a captured image that exceeds the performance of the image pickup unit cannot be acquired simply by providing a plurality of image pickup units as in PTL 1.

Therefore, an object of this technology is to provide an image processing device, an image processing method, and a program that enable a captured image that exceeds the performance of the image pickup unit to be acquired.

Solution to Problem

A first aspect of this technology is an image processing device including a signal processing unit that performs super-resolution processing using a plurality of narrow angle-of-view images having a narrow angle of view within a range of an angle of view of a wide angle-of-view image, with the wide angle-of-view image as a reference.

In this technique, the signal processing unit performs super-resolution processing, with a wide angle-of-view image (for example, color image) acquired by the first image pickup unit as a reference, by using a plurality of narrow angle-of-view images (for example, black-and-white images) acquired by a first image pickup unit using a lens having a higher MTF (Modulation Transfer Function) than that of the first image pickup unit, that is, using images having a narrow angle-of-view within the angle of view of the wide angle-of-view image. The super-resolution processing uses images of the region of interest set in the wide angle-of-view image and the narrow angle-of-view image according to the zoom magnification.

The control unit controls the signal processing unit so as to select an image in the range of angle of view corresponding to the zoom magnification indicated by the user operation from the image after the super-resolution processing. Further, the signal processing unit extracts an image in the range of angle of view corresponding to the zoom magnification from the wide angle-of-view image at the time of preview.

In the super-resolution processing, parallax compensation and movement compensation are performed on a plurality of narrow angle-of-view images according to a parallax detection result from the wide angle-of view image and the narrow angle-of-view image acquired at the same time and a movement detection result for each of a plurality of narrow angle-of-view images.

Moreover, the signal processing unit may use a plurality of wide angle-of-view images in the super-resolution processing. In this case, the parallax is detected from the wide angle-of-view image and the narrow angle-of-view image acquired at the same time, and the movement of the plurality of wide angle-of-view images is detected, and parallax compensation and movement compensation are performed on a plurality of narrow angle-of-view images according to the detection result, regarding the movement of the plurality of narrow angle-of-view images as the movement of the wide angle-of-view image at the same time, and movement compensation is performed on a plurality of wide angle-of-view images according to the detection result, in the super-resolution processing.

A second aspect of this technology is a method for processing an image including performing super-resolution processing using a plurality of narrow angle-of-view images that have an angle of view which is narrower than that of a wide angle-of-view image and within a range of an angle of view of the wide angle-of-view image, with the wide angle-of-view image as a reference, by using a signal processing unit.

A third aspect of this technology is a program that causes a computer to execute processing of an image generated by an image pickup unit, the processing including

acquiring a wide angle-of-view image, and

performing super-resolution processing using a plurality of narrow angle-of-view images that have an angle of view which is narrower than that of the wide angle-of-view image and within a range of an angle of view of the wide angle-of-view image, with the wide angle-of-view image as a reference.

Advantageous Effect of Invention

According to this technique, super-resolution processing is performed using a plurality of narrow angle-of-view images having a narrow angle of view within the range of angle of view of the wide angle-of-view image, with the wide angle-of-view image as a reference. Therefore, a captured image that exceeds the performance of the image pickup unit can be acquired. It should be noted that the effects described in the present specification are merely examples and the effect is not limited thereto and may have additional effects.

DESCRIPTION OF EMBODIMENTS

Hereinafter, modes for carrying out the present technology will be described. Incidentally, the description will be given in the following order.

1. Configuration of a device to which an image processing device is applied

2. Embodiment of the image processing device2-1. Configuration of a first embodiment2-2. Operation of the first embodiment2-3. Configuration of a second embodiment2-4. Operation of the second embodiment2-5. Third embodiment2-6. Other embodiments

3. Application example

1. Configuration of Device to which Image Processing Device is Applied

FIG. 1illustrates an external appearance of a device to which an image processing device according to this technique is applied. Note that, in the following description, the image processing device is applied to the information processing terminal, for example. A subfigure (a) ofFIG. 1depicts the front side of an information processing terminal10, and a display unit53, a touch panel54, and an operation unit55are provided on the front side. A subfigure (b) ofFIG. 1depicts the back side of the information processing terminal10, and a plurality of image pickup units, for example, two image pickup units21-1and21-2are provided on the back side.

FIG. 2illustrates the configuration of the information processing terminal. The information processing terminal10includes a plurality of image pickup units, for example, the two image pickup units21-1and21-2, a signal processing unit30, a sensor unit51, a communication unit52, the display unit53, the touch panel54, the operation unit55, and a storage unit56, and a control unit60. The signal processing unit30constitutes an image processing device of this technology.

The image pickup units21-1and21-2are provided on the same surface side of the information processing terminal10as depicted in the subfigure (b) ofFIG. 1. The image pickup units21-1and21-2are configured by using an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and perform photoelectric conversion of light taken in by a lens (not illustrated), thereby generating the image data of the captured images to output the data to the signal processing unit30. Further, the image pickup units21-1and21-2have characteristic differences, the image pickup unit21-1has a wider angle of view than the image pickup unit21-2, and the image pickup unit21-2has higher quality for images than the image pickup unit21-1. Moreover, the imaging area of the image pickup unit21-2is configured to be included in the imaging area of the image pickup unit21-1.FIG. 3illustrates imaging areas, and an imaging area AR-2of the image pickup unit21-2is configured to be located at the center of an imaging area AR-1of the image pickup unit21-1. Incidentally, in the following description, the captured image acquired by the image pickup unit21-1is referred to as a wide angle-of-view image, and the captured image acquired by the image pickup unit21-2is referred to as a narrow angle-of-view image.

FIG. 4illustrates pixel arrays of the image pickup units. A subfigure (a) ofFIG. 4depicts the pixel array of the image pickup unit21-1. The image pickup unit21-1is configured using a color filter in which red (R) pixels, blue (B) pixels, and green (G) pixels are arranged in a Bayer array, for example. In the Bayer array, in a pixel unit of 2×2 pixels, two pixels at diagonal positions are green (G) pixels, and the remaining pixels are a red (R) pixel and a blue (B) pixel. That is, the image pickup unit21-1is so configured that the pixel array depicted in the subfigure (a) ofFIG. 4is repeated and each pixel outputs an electric signal based on the incident light amount of any one of the color components of red, blue, or green. Therefore, the image pickup unit21-1generates image data of a color captured image in which each pixel indicates one of the three primary color (RGB) components.

A subfigure (b) ofFIG. 4depicts the pixel array of the image pickup unit21-2. In the image pickup unit21-2, the pixel array depicted in the subfigure (b) ofFIG. 4is repeated, and each pixel is configured as a W (white) pixel that outputs an electric signal based on the amount of incident light in all wavelength regions of visible light. Therefore, the image pickup unit21-2generates image data of black-and-white captured images. Note that the image pickup unit21-2is not limited to the case of generating image data of black-and-white images as image data of captured images having higher image quality than that of the image pickup unit21-1and may generate image data of color images.

The signal processing unit30performs super-resolution processing, with a wide angle-of-view image acquired by the image pickup unit21-1as a reference, by using a plurality of narrow angle-of-view images acquired by the image pickup unit21-2, that is, a plurality of narrow angle-of-view images having a narrow angle of view within the range of angle of view of the wide angle-of-view image. Further, the signal processing unit30uses an image in the range of angle of view corresponding to the zoom magnification from the image after the super-resolution processing to generate a seamless zoom image from a wide-angle view to a telephoto view, and outputs the image to the display unit53and the storage unit56. Incidentally, the details of the configuration and operation of the signal processing unit30will be described later.

The sensor unit51is configured by using a gyro sensor or the like, and detects a shake generated to the information processing terminal10. The sensor unit51outputs information regarding the detected shake to the control unit60.

The communication unit52communicates with devices on a network such as a LAN (Local Area Network) or the Internet.

The display unit53displays a captured image on the basis of image data supplied from the signal processing unit30, and displays a menu screen, various application screens, and the like on the basis of information signals from the control unit60. Further, the touch panel54is mounted on the display surface side of the display unit53and configured such that the GUI function can be used.

The operation unit55is configured using operation switches and the like and generates operation signals according to a user operation and outputs the operation signals to the control unit60.

The storage unit56stores information generated by the information processing terminal10, for example, image data supplied from the signal processing unit30and various types of information used for executing communication and applications in the information processing terminal10.

The control unit60includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory) (not illustrated), and the like. The control unit60executes the programs stored in the ROM or the RAM and controls the operation of each part such that the information processing terminal10performs operation corresponding to a user operation on the user interface unit, which is the touch panel54or the operation unit55. In addition, the control unit60generates information regarding the user operation, for example, zoom information indicating a zoom magnification set by the user or the like and outputs the information to the signal processing unit30.

It should be noted that the information processing terminal10is not limited to the configuration depicted inFIG. 2, and for example, may be provided with an encoding processing unit configured to encode image data to store the encoded image data in the storage unit56, a resolution converting unit that adjusts the image data to the resolution of the display unit, and the like.

2. Embodiment of Image Processing Device

2-1. Configuration of First Embodiment

In the first embodiment, a case will be described in which, with a color captured image acquired by the image pickup unit21-1as a reference, super resolution is performed using this color captured image and a plurality of frames of black-and-white captured images acquired by the image pickup unit21-2to generate a high-resolution color image.

FIG. 5illustrates the configuration of the first embodiment. The signal processing unit30includes region-of-interest (RIO) determining sections31-1and31-2, a parallax/motion vector detecting section32, and a super-resolution processing section36.

The region-of-interest (RIO) determining section31-1determines an area necessary for display (region of interest) in a wide angle-of-view color captured image acquired by the image pickup unit21-1on the basis of the zoom magnification indicated from the control unit60. The region-of-interest determining section31-1outputs a color image Ic1t0of the region of interest to the parallax/motion vector detecting section32and the super-resolution processing section36.

The region-of-interest (RIO) determining section31-2determines an area necessary for display (region of interest) in the plurality of frames of black-and-white captured images acquired by the image pickup unit21-2on the basis of the zoom magnification indicated from the control unit60. The region-of-interest determining section31-2outputs black-and-white images Ic2t0to Ic2tnof the region of interest to the parallax/motion vector detecting section32and the super-resolution processing section36.

By setting the regions of interest in the region-of-interest determining sections31-1and31-2in this manner, the super-resolution processing to be described later can be performed more efficiently than the case where the super-resolution processing using the entire image is performed.

The parallax/motion vector detecting section32detects the parallax of the image pickup unit21-2with respect to the image pickup unit21-1from the image of the region of interest determined by the region-of-interest determining section31-1and the image of the region of interest determined by the region-of-interest determining section31-2. Further, with respect to the plurality of frames of images of the region of interest determined by the region-of-interest determining section31-2, a motion vector based on the image acquired by the image pickup unit21-1is detected. The parallax/motion vector detecting section32outputs the detected parallax and motion vector to the super-resolution processing section36.

The super-resolution processing section36performs super-resolution processing using a plurality of narrow angle-of-view images acquired by the image pickup unit21-2having a narrower angle of view than the image pickup unit21-1within the range of angle of view of the image pickup unit21-1, with the wide angle-of-view image acquired by the image pickup unit21-1, which has a wider angle of view than the image pickup unit21-2as a reference. In the super-resolution processing, a high-resolution image is generated by additively feeding back a plurality of low-resolution images at different times.

FIG. 6illustrates the configuration of the super-resolution processing section. The super-resolution processing section36includes a compensating section361, a spatial filter362, a downsampling section363, a subtracting section364, an upsampling section365, an inverse spatial filter366, an adding section367, a buffer368, and an image output section369.

The compensating section361outputs the color captured image to be a reference to the subtracting section364. Further, the compensating section361performs parallax compensation and motion compensation on the plurality of frames of monochrome captured images on the basis of the detection result of the parallax/motion vector detecting section and outputs the images to the subtracting section364.

The spatial filter362performs a process of simulating deterioration of spatial resolution on the image stored in the buffer368. Here, the point spread function measured beforehand is used as a filter to apply convolution to the image.

The downsampling section363performs downsampling processing on the image supplied from the spatial filter362to the same resolution as the monochrome image of the region of interest.

The subtracting section364subtracts the image from the downsampling section363from the image from the compensating section361for each pixel to generate a difference image. The subtracting section364outputs the generated difference image to the upsampling section365.

The upsampling section365converts the difference image supplied from the subtracting section364to an image having the same resolution as that before downsampling performed in the downsampling section363, which is higher than that of the color captured image or the monochrome captured image of the region of interest, and outputs the image to the inverse spatial filter366.

The inverse spatial filter366performs a filter processing having a characteristic opposite to that of the spatial filter362on the difference image supplied from the upsampling section365and outputs the filtered difference image to the adding section367.

The adding section367adds the image stored in the buffer368to the difference image output from the inverse spatial filter366and outputs the result to the buffer368and the image output section369.

The buffer368stores the image supplied from the adding section367. In addition, the buffer368outputs the stored image to the spatial filter362and the adding section367.

The image output section369outputs an image in the range of angle of view corresponding to the zoom magnification set by the user or the like from the image after the super-resolution processing to the display unit53, the storage unit56, or the like, on the basis of the zoom information from the control unit60so as to perform a seamless zoom operation from a wide-angle view to a telephoto view.

2-2. Operation of First Embodiment

FIG. 7is a flowchart depicting the operation of the signal processing unit according to the first embodiment. In step ST1, the signal processing unit acquires zoom information. The signal processing unit acquires the zoom information from the control unit60and proceeds to step ST2.

In step ST2, the signal processing unit sets a region of interest. The region-of-interest determining section31-1of the signal processing unit30determines the region of interest that is a region necessary for outputting an image of the zoom magnification indicated by the zoom information in the wide angle-of-view color captured image acquired by the image pickup unit21-1. In addition, the region-of-interest determining section31-2determines the region of interest that is a region necessary for outputting the image of the zoom magnification indicated by the zoom information in the narrow angle-of-view monochrome image captured by the image pickup unit21-2. The region-of-interest determining sections31-1and31-2determine the regions of interest, and the processing proceeds to step ST3.

In step ST3, the signal processing unit detects a parallax/motion vector. The parallax/motion vector detecting section32of the signal processing unit30detects the parallax of the image pickup unit21-2with respect to the image pickup unit21-1, on the basis of the image of the region of interest determined by the region-of-interest determining section31-1and the image of the region of interest determined by the region-of-interest determining section31-2. Further, the motion vector is detected for each of the plurality of frame images of the region of interest determined by the region-of-interest determining section31-2, and the processing proceeds to step ST4.

In step ST4, the signal processing unit performs super-resolution processing. The super-resolution processing section36of the signal processing unit30performs super-resolution processing, with the color image as a reference, using this color captured image and a plurality of frames of black-and-white captured images, and generates a color image in which the resolution of the imaging area of the image pickup unit21-2has been made high and proceeds to step ST5.

In step ST5, the signal processing unit performs image output processing. The super-resolution processing section36of the signal processing unit30outputs an image in the range of angle of view corresponding to the zoom magnification set by the user or the like to the display unit53, the storage unit56, and the like from the image generated in step ST4, on the basis of the zoom information from the control unit60.

FIG. 8depicts operation examples of the first embodiment. Incidentally, for the sake of simplifying the description, the region of interest is set to be the entire image. As depicted in a subfigure (a) ofFIG. 8, the signal processing unit30performs super-resolution processing using the six monochrome images Ic2t0to Ic2t5acquired by the image pickup unit21-2while using the color image Ic1t0acquired by the image pickup unit21-1as a reference, for example. Note that, in the super-resolution processing, position correction and addition feedback processing of the monochrome images Ic2t0to Ic2t5acquired by the image pickup unit21-2are performed on the basis of the parallax and motion vectors Wc1t0, c2t0to Wc1t0, c2t5. Therefore, the imaging area AR-2of the image pickup unit21-2has high resolution. That is, as depicted a subfigure (b) of inFIG. 8, when the zoom magnification is 1, a wide angle-of-view color image in which the resolution of the image in the imaging area AR-2of the image pickup unit21-2has been made high is output. Further, a high resolution color image is output when the zoom magnification becomes higher and the zoom range is Za times, namely when the zoom range coincides with the imaging area AR-2of the image pickup unit21-2. Further, when the zoom magnification becomes Zb times that is higher than Za times, the image in the region corresponding to the zoom magnification is output from the area where the imaging area AR-1of the image pickup unit21-1and the imaging area AR-2of the image pickup unit21-2overlap. That is, since the image of the overlapping area of the imaging area of the image pickup unit21-1and the imaging area of the image pickup unit21-2is an image generated by the super-resolution processing, a color image having a higher resolution than that of the related art can be output.

Therefore, according to the first embodiment, the zoom operation can be performed seamlessly from a wide-angle view to a telephoto view without degrading the image quality.

2-3. Configuration of Second Embodiment

In the second embodiment, a case will be described in which a high-resolution color image is generated by super-resolution by using a plurality of frames of color images and a plurality of frames of monochrome images having different visual points.

FIG. 9illustrates the configuration of the second embodiment. The signal processing unit30includes region-of-interest (RIO) determining sections31-1and31-2, a motion detecting section33, a parallax detecting section34, a registration vector calculating section35, super-resolution processing sections37and38.

The region-of-interest (RIO) determining section31-1determines the area required for display (region of interest) in the plurality of frames of color images having a wide angle of view acquired by the image pickup unit21-1on the basis of the zoom magnification indicated from the control unit60. The region-of-interest determining section31-1outputs color images Ic1t0to Ic1tnof the region of interest to the motion detecting section33, the parallax detecting section34, and the super-resolution processing section37.

The region-of-interest (RIO) determining section31-2determines an area necessary for display (region of interest) in the plurality of frames of black-and-white captured images acquired by the image pickup unit21-2on the basis of the zoom magnification indicated from the control unit60. The region-of-interest determining section31-2outputs monochrome images Ic2t0to Ic2tnof the region of interest to the parallax detecting section34and the super-resolution processing section38.

The motion detecting section33detects a motion vector for the color image Ic1t0for each frame from the plurality of frames of images of region of interest determined by the region-of-interest determining section31-1. The motion detecting section33outputs the detected motion vector to the registration vector calculating section35and the super-resolution processing section37.

The parallax detecting section34detects the parallax of the image pickup unit21-2with respect to the image pickup unit21-1from the image of the region of interest determined by the region-of-interest determining section31-1and the image of the region of interest determined by the region-of-interest determining section31-2. The parallax detecting section34detects the parallax on the basis of the color image Ic1t0and the black-and-white image Ic2t0of the region of interest, for example, and outputs the detected parallax to the registration vector calculating section35.

The registration vector calculating section35calculates a motion vector in the spatiotemporal direction that adjusts the positions of the black-and-white images Ic2t0to Ic2tnwith respect to the color image Ic1t0that is a reference. The registration vector calculating section35calculates the motion vector of each frame and outputs to the super-resolution processing section38, with the monochrome images Ic2t0to Ic2tnas the visual point of the image pickup unit21-1, by using the motion vector detected by the motion detecting section33and the parallax detected by the parallax detecting section34. Here, for the plurality of frames of black-and-white images Ic2t0to Ic2tn, detection of the motion vector with respect to the reference color image Ic1t0makes the calculation cost high. Accordingly, the motions of the black-and-white images Ic2t1to Ic2tnare regarded as equal to the motions of the color images Ic1t1to Ic2tn, and motion vectors Wc1t0, c2t0to Wc1t0, c2tnare calculated and output to the super-resolution processing section38.

The super-resolution processing sections37and38are configured similarly to the above-described super-resolution processing section36. Note that, for simplification of description, the super-resolution processing sections37and38use the reference numerals of the super-resolution processing section36.

The super-resolution processing section37stores a high-resolution color image calculated by performing upsampling or inverse spatial filter processing on the color image Ic1t0, in the buffer368as an accumulated image Ic1s. Next, the accumulated image Ic1sin the buffer368is subjected to spatial filter processing and downsampling and supplied to the subtracting section364as an image Ic1sa.

The color image Ic1t1is subjected to motion compensation by the compensating section361on the basis of the motion vector Wc1t0t1detected by the motion detecting section33and is supplied to the subtracting section364.

The subtracting section364calculates a difference image between an image Ic1t1aafter motion compensation and the image Ic1sathat has undergone the spatial filter processing or downsampling. This difference image is added to the accumulated image Ic1sin the buffer368after being subjected to upsampling and inverse spatial filter processing, and the image after addition is accumulated in the buffer368as a new accumulated image Ic1s.

After that, similar processing is repeatedly performed up to the final color image Ic1tnfor the plurality of frames, so that a difference image between the image Ic1tnaafter motion compensation and the image Ic1sasubjected to spatial filter processing and downsampling is calculated, and the accumulated image Ic1sin the buffer368is added to the image calculated by performing upsampling or inverse spatial filter processing on this difference image, so that the image after the addition is output to the super-resolution processing section38as a super-resolution image SRc1t0from the super-resolution processing section37.

While using the super-resolution image SRc1t0supplied from the super-resolution processing section37as a reference, the super-resolution processing section38performs super-resolution processing by using a plurality of narrow angle-of-view images (monochrome images) Ic2t0to Ic2tnacquired by the image pickup unit21-2having a narrower angle of view than the image pickup unit21-1within the range of angle of view of the image pickup unit21-1and motion vectors Wc1t0, c2t0to Wc1t0, c2tncalculated by the registration vector calculating section35.

2-4. Operation of Second Embodiment

FIG. 10is a flowchart depicting the operation of the signal processing unit according to the second embodiment. In step ST11, the signal processing unit acquires zoom information. The signal processing unit acquires the zoom information from the control unit60and proceeds to step ST12.

In step ST12, the signal processing unit sets a region of interest. The region-of-interest determining section31-1of the signal processing unit30determines the region of interest that is a region required for display in the wide angle-of-view color captured image acquired by the image pickup unit21-1, on the basis of the zoom magnification indicated from the control unit60. In addition, the region-of-interest determining section31-2determines the region of interest that is a region necessary for display in the plurality of frames of black-and-white captured images acquired by the image pickup unit21-2, on the basis of the zoom magnification indicated from the control unit60, and then, the processing proceeds to step ST13.

In step ST13, the signal processing unit performs motion detection. The motion detecting section33of the signal processing unit30detects the motion for each frame from the plurality of frames of color images of region of interest determined by the region-of-interest determining section31-1and the processing proceeds to step ST14.

In step ST14, the signal processing unit performs super-resolution processing. The super-resolution processing section37of the signal processing unit30performs addition feedback of the plurality of frames of color images and the like to generate a color image having a higher resolution than that of the color image acquired by the image pickup unit21-1, and then, the processing proceeds to step ST15.

In step ST15, the signal processing unit performs parallax detection. The parallax detecting section34of the signal processing unit30detects the parallax of the image pickup unit21-2with respect to the image pickup unit21-1from the image of region of interest determined by the region-of-interest determining section31-1and the image of region of interest determined by the region-of-interest determining section31-2. The parallax detecting section34detects the parallax on the basis of the color image Ic1t0and the black-and-white image Ic2t0of the region of interest, for example, and the processing proceeds to step ST15.

In step ST16, the signal processing unit calculates a registration vector. The registration vector calculating section35of the signal processing unit30calculates the motion vector of each frame while using the black and white images Ic2t0to Ic2tnas the visual point of the image pickup unit21-1, on the basis of the motion vector detected in step ST13and the parallax detected in step ST15and outputs the vector to the super-resolution processing section38.

In step ST17, the signal processing unit performs super-resolution processing. The super-resolution processing section38of the signal processing unit30performs addition feedback and the like of the plurality of frames of monochrome images on the color image generated by the super-resolution processing of step ST14, and generates a higher-resolution color image than the color image generated in step ST14, and the processing proceeds to step ST18.

In step ST18, the signal processing unit performs image output processing. The super-resolution processing section36of the signal processing unit30outputs an image in the range of angle of view according to the zoom magnification set by the user or the like to the display unit53, the storage unit56, and the like, from the image generated in step ST17, on the basis of the zoom information from the control unit60.

Note that the operation of the second embodiment is not limited to the step order depicted inFIG. 10, and for example, the processing of step ST14and step ST17may be performed after the processing of step ST15and step ST16.

FIG. 11depicts operation examples of the second embodiment. For the sake of simplifying the description, the region of interest is set to be the entire image. As depicted in a subfigure (a) ofFIG. 11, the signal processing unit30performs super-resolution processing, with the color image Ic1t0acquired by the image pickup unit21-1as a reference, for example, by using five color images Ic1t1to Ic1t5acquired by the image pickup unit21-1thereafter. Note that, in the super-resolution processing, position correction and addition feedback processing and the like of the color images Ic1t1to Ic1t5are performed on the basis of motion vectors Wc1t0, c1t1to Wc1t0, c1t5detected by the motion detecting section33.

After that, the signal processing unit30performs super-resolution processing by using six monochrome images Ic2t0to Ic2t5acquired by the image pickup unit21-2with the color image Ic1t0acquired by the image pickup unit21-1as a reference, for example. Note that in the super-resolution processing, position correction and addition feedback processing and the like of the monochrome images Ic2t0to Ic2t5are performed on the basis of motion vectors Wc1t0, c2t0to Wc1t0, c2t5calculated by the registration vector calculating section35.

Accordingly, the imaging area AR-1of the image pickup unit21-1and the imaging area AR-2of the image pickup unit21-2have high resolution. Therefore, as depicted in a subfigure (b) ofFIG. 11, a high-resolution color image can be output regardless of the zoom magnification, and the zoom operation can be performed seamlessly from a wide-angle view to a telephoto view without degrading the image quality.

Hence, according to the second embodiment, the zoom operation can be performed seamlessly from a wide-angle view to a telephoto view without degrading the image quality, as in the first embodiment.

Further, since the registration vector calculating section35calculates the registration vector on the assumption that the motions of the image pickup unit21-2and the image pickup unit21-1are equal, the calculation cost can be reduced as compared with the case where the motion is detected individually for the image pickup unit21-2and the image pickup unit21-1.

Further, in the second embodiment, although the case where the super-resolution processing is performed using a plurality of frames of monochrome images after the super-resolution processing is performed using a plurality of frames of color images is exemplified, the super-resolution processing is not limited to the above order. For example, the super-resolution processing may be performed using a color image and a monochrome image according to the order of the frames.

Next, a third embodiment will be described. In the third embodiment, an MTF (Modulation Transfer Function) lens that is less affected by aliasing distortion is used in an image pickup unit that does not acquire a plurality of images used for super-resolution processing. Further, in the image pickup unit that acquires a plurality of images used for the super-resolution processing, a lens having a higher MTF is used than the image pickup unit that does not acquire the plurality of images, and a high-resolution image that is not affected by aliasing distortion is generated by the super-resolution processing.

For example, in the first embodiment, since the super-resolution processing is performed using the black and white images Ic2t0to Ic2tnfor the plurality of frames, the image pickup unit21-2that generates the black-and-white images Ic2t0to Ic2tnuses a lens with a high MTF. Further, the image pickup unit21-1that generates the color image Ic1t0uses a lens having a lower MTF than the image pickup unit21-2such that the influence of aliasing distortion is small.

FIG. 12illustrates the spectral distribution. The image pickup unit21-1uses a lens having a low MTF such that the influence of aliasing distortion is small. Therefore, the image acquired by the image pickup unit21-1is an image in which aliasing distortion is not noticeable, as depicted in a subfigure (a) ofFIG. 12. Note that a subfigure (b) ofFIG. 12illustrates the spectral distribution of the lens used in the image pickup unit21-1and does not have a component with a frequency higher than the Nyquist frequency. Incidentally, the Nyquist frequency is determined by the pixel size of the image sensor used in the image pickup unit.

The image pickup unit21-2uses a lens having a higher MTF than the image pickup unit21-1. Therefore, the image acquired by the image pickup unit21-2is an image in which aliasing distortion has occurred, as depicted in a subfigure (c) ofFIG. 12. Note that a subfigure (d) ofFIG. 12illustrates the spectral distribution of the lens used in the image pickup unit21-2, which has a frequency component higher than the Nyquist frequency. Here, if a plurality of images that has moved differently from the pixel unit is aligned and added by the super-resolution processing, an image in which aliasing distortion is removed is obtained as depicted in a subfigure (e) ofFIG. 12. Note that a subfigure (f) ofFIG. 12illustrates the spectrum distribution after the super-resolution processing.

Further, in the second embodiment, a lens having a high MTF may be used not only in the image pickup unit21-2but also in the image pickup unit21-1.

In this way, when a lens with a high MTF is used in the image pickup unit that acquires a plurality of images used for super-resolution processing, a color image with high resolution can be obtained as compared with the case where a lens with a low MTF is used.

Further, for separating a lens that is less affected by the aliasing distortion and a lens for generating the plurality of images to be used for the super-resolution processing, it is only required to use the result of comparison with the threshold value set for the MTF, for example. Here, the threshold value is set to a predetermined multiplying factor of the Nyquist frequency (for example, a value larger than the Nyquist frequency and smaller than the double thereof, preferably approximately 1.3 to 1.5 times).

2-6. Other Embodiments

Meanwhile, the processing cost becomes high in the case where super-resolution processing is performed by using a plurality of images to generate an image having a higher resolution than that of the plurality of images. Therefore, at the time of preview, an image in the range of angle of view corresponding to the zoom magnification is extracted from a wide angle-of-view color image. In addition, when an image is recorded or output to an external equipment or the like, super-resolution processing is performed to generate a high-resolution image, and an image having the range of angle of view corresponding to the zoom magnification is extracted from the high-resolution image. By doing so, the processing cost at the time of preview can be reduced.

3. Application Example

The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure is not limited to information processing terminals and thus may be achieved as a device mounted on any one type of mobile bodies such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility, airplanes, drones, ships, robots, construction machines, agricultural machines (tractors), etc.

FIG. 14is a diagram depicting an example of the installation position of the imaging section12031.

In the vehicle control system12000described above, the imaging sections12031,12101,12102,12103,12104, and12105are configured to use a plurality of image pickup units as necessary, for example, the image pickup units21-1and21-2depicted inFIG. 2. Further, the signal processing unit30is provided in the integrated control unit12010of the application example depicted inFIG. 13. With such a configuration, even when the imaging sections12031,12101,12102,12103,12104, and12105are made small and thin, a high-quality and wide-angle-view captured image or zoom image can be obtained, and thus, the obtained captured image can be used for driving support and driving control. Note that the signal processing unit30may be achieved in a module (for example, an integrated circuit module configured by one die) for the integrated control unit12010depicted inFIG. 13.

The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. In the case of executing the processes by software, a program having recorded a processing sequence therein is installed in a memory in a computer incorporated in dedicated hardware and executed. Alternatively, the program can be installed and executed in a general-purpose computer that can execute various processes.

For example, the program can be recorded in advance in a hard disk, an SSD (Solid State Drive), or a ROM (Read Only Memory) as a recording medium. Alternatively, the program can be stored (recorded) temporarily or permanently in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disc, a DVD (Digital Versatile Disc), a BD (Blu-Ray Disc (registered trademark)), a magnetic disc, a semiconductor memory card. Such a removable recording medium can be provided as so-called package software.

Further, in addition to installing the program from the removable recording medium into the computer, the program may be transferred wirelessly or by wire from a download site to the computer via a network such as a LAN (Local Area Network) or the Internet. In the computer, the program thus transferred is received and can be installed in a recording medium such as a built-in hard disk.

It should be noted that the effects described in the present specification are merely examples and the effect is not limited thereto, and there may be additional effects not described. Further, the present technology should not be construed as being limited to the above-described embodiments of the technology. The embodiments of this technology disclose the present technology in the form of exemplification, and it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present technology. That is, in order to determine the gist of the present technology, the claims should be taken into consideration.

Further, the image processing device of the present technology can also have the following configurations.

(1) An image processing device including:

a signal processing unit that performs super-resolution processing using a plurality of narrow angle-of-view images having a narrow angle of view within a range of an angle of view of a wide angle-of-view image, with the wide angle-of-view image as a reference.

(2) The image processing device described in item (1), in which

the signal processing unit takes out an image in a range of an angle of view corresponding to a zoom magnification from an image subjected to the super-resolution processing.

(3) The image processing device described in item (2), in which

the signal processing unit sets a region of interest in each of the wide angle-of-view image and the narrow angle-of-view image according to the zoom magnification, and performs the super-resolution processing by using an image of the region of interest.

(4) The image processing device described in item (2) or (3), in which

the signal processing unit takes out the image in the range of the angle of view corresponding to the zoom magnification from the wide angle-of-view image at a time of preview.

(5) The image processing device described in any one of item (1) or (4), in which

the signal processing unit performs detection of a parallax from the wide angle-of-view image and the narrow angle-of-view image which are acquired simultaneously, and performs motion detection of the plurality of narrow angle-of-view images, so as to perform parallax compensation and motion compensation on the plurality of narrow angle-of-view images according to the parallax and a result of the motion detection in the super-resolution processing.

(6) The image processing device described in any one of items (1) to (5), in which

the signal processing unit uses a plurality of the wide angle-of-view images in the super-resolution processing.

(7) The image processing device described in item (6), in which

the signal processing unit performs detection of a parallax from the wide angle-of-view image and the narrow angle-of-view image which are acquired simultaneously, and performs motion detection of the plurality of the wide angle-of-view images, so as to perform parallax compensation and motion compensation on the plurality of narrow angle-of-view images according to a result of the detection, and to perform the motion compensation on the plurality of the wide angle-of-view images according to the result of the detection, regarding a motion of the plurality of narrow angle-of-view images as a motion of the wide angle-of-view image at a same time, in the super-resolution processing.

(8) The image processing device described in any one of items (1) to (7), in which

the plurality of narrow angle-of-view images is acquired by using a lens having a higher MTF (Modulation Transfer Function) than the wide angle-of-view image.

(9) The image processing device described in item (8), in which

a predetermined multiplying factor of Nyquist frequencies of an image pickup unit that acquires the narrow angle-of-view image and an image pickup unit that acquires the wide angle-of-view image is used as a threshold value, the wide angle-of-view image is acquired by using a lens having an MTF lower than the threshold value, and the narrow angle-of-view image is acquired by using a lens having an MTF which is the threshold value or higher.

(10) The image processing device described in any one of items (1) to (9), further including:

a first image pickup unit that acquires the wide angle-of-view image; and

a second image pickup unit that acquires the narrow angle-of-view image by using a lens having a higher MTF (Modulation Transfer Function) than the first image pickup unit.

(11) The image processing device described in any one of items (1) to (10), further including:

a control unit that controls the signal processing unit so as to select an image in a range of an angle of view corresponding to a zoom magnification indicated by a user operation from an image subjected to the super-resolution processing.

(12) The image processing device described in any one of items (1) to (11), in which

the wide angle-of-view image includes a color image, and the narrow angle-of-view image includes a black-and-white image.

INDUSTRIAL APPLICABILITY

With the image processing device, image processing method, and program of this technology, super-resolution processing using a plurality of narrow-angle images having a narrow angle of view within the angle-of-view range of a wide angle-of-view image is performed with the wide angle-of-view image as a reference. Accordingly, a captured image that exceeds the performance of the image pickup unit can be acquired. Therefore, the technology is suitable for equipment that uses an image pickup unit and the like and that requires downsizing and thinning of the image pickup unit.

REFERENCE SIGN LIST