Image processing apparatus, image capturing apparatus, image processing method, and storage medium for obtaining information on focus control of a subject

Conventionally, on a subject in which range, refocus can be performed at the time of image capturing or at the time of editing an image is not specified clearly, and therefore, it is difficult for a user to capture an image or to edit an image in a manner the user intends. An image processing apparatus has an acquisition unit configured to acquire an image including a plurality of subject areas and distance information corresponding to the plurality of subject areas and a generation unit configured to generate a shifted image in which positions of the plurality of subject areas are shifted in the image based on the distance information.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a storage medium for obtaining information on focus control of a subject.

2. Description of the Related Art

As an image capturing apparatus, such as a digital camera, there is known an apparatus that makes focus adjustment of a subject included in a captured image by a user specifying the subject by a touch panel operation etc. (for example, Japanese Patent Laid-Open No. 2011-39457).

Further, the technique is also proposed, which makes it possible for a user to easily grasp an object in focus the image of which is to be captured (subject) by displaying a map image for indicating a depth position representing a position of the object the image of which is to be captured in the depth direction and a focus position representing the position of focus (for example, Japanese Patent Laid-Open No. 2010-177741).

In recent years, the light field photography technique is also known, which acquires information of the orientation and intensity of a light ray (light field data) and makes adjustment of the focus position (refocus) and adjustment of the depth of field by the later image processing (for example, Japanese Patent Laid-Open No. 2008-515110).

By using this technique, there is an advantage that a failure in focus adjustment at the time of image capturing can be made up by image processing because it is possible to make focus adjustment after image capturing. Further, there is also an advantage that it is possible to obtain a plurality of images in which arbitrary subjects are brought into focus in an image from one captured image by changing the image processing method, and therefore, it is possible to reduce the number of times of image capturing.

In light field photography, from data of images captured from a plurality of viewpoints, the direction and intensity of a light ray that passes through each position (light field) in a plurality of positions in the space are calculated. Then, by using the information of the obtained light field, an image on the assumption that light passes through a virtual optical system and forms the image on a virtual sensor is calculated. By appropriately setting such a virtual optical system and a virtual sensor, refocus described previously is enabled. As an image capturing apparatus for acquiring the light field, a plenoptic camera (for example, Japanese Patent Laid-Open No. 2009-124213) in which a microlens array is arranged behind a main lens and a camera array in which compact cameras are arranged side by side are known. It is possible for both to obtain data of subject images from a plurality of viewpoints captured from different directions by one-time image capturing. In other words, it is possible to represent the light field photography as calculation of an image to be acquired by a virtual sensor under virtual optical conditions from data of images from a plurality of viewpoints. In the following, the processing to calculate an image acquired by the virtual sensor is referred to as “refocus processing”. As refocus processing, there is known a method in which acquired data of images from a plurality of viewpoints is subjected to projective transformation onto a virtual sensor, and added and averaged (for example, WO2008050904).

As a method for displaying an image having been subjected to refocus processing (hereinafter, a refocus image) while changing the focus position, mention is made of, for example, the method disclosed in United States Patent No. 2008/0131019. In the method disclosed in United States Patent No. 2008/0131019, the user interface (UI) to adjust the focus position is prepared on the screen on which a refocus image is displayed and the focus position is changed via the UI. Further, in United States Patent No. 2008/0131019, the method is disclosed, which displays a refocus image whose focus position is adjusted to a subject which a user desires to put to a focus position and has given instructions through a screen on which the refocus image is displayed.

In the display of captured image data or light field data, an object to be brought into focus exists on a plane at a fixed distance from the position of image capturing. However, in the case where focusing is performed by specifying a specific subject, it is not possible for a user to grasp which subject other than the specific subject is brought into focus.

Further, by the technique disclosed in Japanese Patent Laid-Open No. 2010-177741, it is possible to grasp which subject is brought into focus by the map image in which marks indicating objects the image of which is to be captured are arranged in order in the depth direction, however, the map image is quite different from the captured image (actual image), and therefore, it is difficult to grasp the subject intuitively.

That is, by the conventional methods disclosed in the above-described Patent Literatures etc., on a subject in which range, refocus can be performed is not specified clearly at the time of photographing or at the time of editing an image, and therefore, there is such a problem that it is difficult for a user to perform image capturing or image edition in a manner the user intends.

SUMMARY OF THE INVENTION

The image processing apparatus according to the present invention has an acquisition unit configured to acquire an image including a plurality of subject areas and distance information corresponding to the plurality of subject areas, and a generation unit configured to generate a shifted image in which positions of the plurality of subject areas are shifted in the image.

Alternatively, the image processing apparatus has an acquisition unit configured to acquire an image including a plurality of subject areas and distance information corresponding to the plurality of subject areas, a first generation unit configured to generate a rearranged image in which the plurality of subject areas is rearranged in the image based on the distance information, and a second generation unit configured to generate an image in which information on a focus state is reflected in the rearranged image at the time of performing image combination processing to change the focus state of the image including the plurality of subject areas.

According to the present invention, it is possible for a user to intuitively grasp the focus state of a subject at the time of photographing or at the time of editing an image.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, with reference to drawings, embodiments of the present invention are explained in detail.

First Embodiment

Here, an aspect in which an image processing apparatus according to the present invention is applied to a digital camera is explained as a first embodiment.

FIG. 1is a diagram showing an example of an external appearance of a digital camera.

In a display unit101, for example, a liquid crystal display is used and images and various kinds of information are displayed. The display unit101has a touch screen function and is capable of detecting a touch on the display unit101. For example, a touch panel is configured so that the transmittance of light does not affect the display of the display unit101and the touch panel is attached to an upper layer of the display surface of the display unit101. Then, the input coordinates on the touch panel and the display coordinates on the display unit101are associated with each other. Due to this, a GUI with which it seems possible for a user to directly operate a screen displayed on the display unit101can be configured.

A shutter button102is an operation unit for giving instructions to perform image capturing.

A mode dial103is an operation unit for switching various kinds of modes.

A connector104is a connector between a connection cable111and a digital camera100.

An operation unit105includes various kinds of switches, buttons, etc., to receive various kinds of operations from a user.

A controller wheel106is an operating member included in the operation unit105and capable of being operated to rotate.

A power source switch107switches between on and off of a power source of the digital camera100.

A storage medium108is an information storage medium, such as a memory card and a hard disc, for storing captured image data and light field data.

A storage medium slot109is a slot for housing the storage medium108. It is made possible for the storage medium108housed in the storage medium slot109to communicate with the digital camera100.

A lid110is a lid of the storage medium slot109.

FIG. 2is a block diagram showing an internal configuration of the digital camera100.

An image capturing lens201includes a focus lens.

A shutter includes an aperture function and controls exposure in an interlocking manner with the shutter button102.

An image capturing unit203includes a CCD that converts an optical image into an electric signal, a CMOS element, etc.

An A/D converter204converts an analog signal output from the image capturing unit203into a digital signal.

A barrier205prevents stain and breakage of the image capturing system by covering the image capturing unit203including the image capturing lens201and the shutter202.

An image processing unit206performs resize processing, such as predetermined pixel interpolation and reduction, and color conversion processing on image data from the A/D conversion unit204or image data from a memory control unit207. In the image processing unit206, predetermined arithmetic operation processing is performed using captured image data and a system control unit208performs exposure control and distance measuring control based on the obtained arithmetic operation results. Due to this, TTL (Through The Lens) AF (Auto Focus) processing, AE (Auto Exposure) processing, and EF (pre-flash) processing are performed. In the image processing unit206, predetermined arithmetic operation processing is further performed using captured image data and TTL AWB (Auto White Balance) processing is also performed based on the obtained arithmetic operation results. The image data from the A/D converter204is directly written to a memory209via the image processing unit206and the memory control unit207, or via the memory control unit207.

The memory209stores image data obtained by the image capturing unit203and converted into digital data by the A/D converter204, and image data to be displayed on the display unit101. The memory209includes a storage capacity sufficient to store a predetermined number of still images, moving images for a predetermined period of time, and voice. By sequentially displaying captured image data using the display unit101, it is possible to implement an electronic finder function (through-image display). Further, the memory209serves also as a memory for displaying images (video memory). A D/A converter210converts the data for displaying images stored in the memory209into analog signals and supplies the signals to the display unit101. In this manner, the image data to be displayed written to the memory209is displayed on the display unit101via the D/A converter210.

The display unit101produces a display on a display, such as an LCD, in accordance with the analog signal from the D/A converter210.

A nonvolatile memory211is an electrically erasable/programmable memory and, for example, an EEPROM etc. is used. The nonvolatile memory211stores constants for the operation of the system control unit208, programs, etc. Programs referred to here are programs used to perform various kinds of flowcharts, to be described later.

The system control unit208controls the whole of the digital camera100. The system control unit208implements each piece of processing, to be described later, by executing the programs stored in the nonvolatile memory211described previously. As a system memory212, a RAM is used and on which constants, variables for the operation of the system control unit208, programs read from the nonvolatile memory211, etc., are developed. Further, the system control unit208also performs display control by controlling the memory209, the D/A converter210, the display unit101, etc. Furthermore, it is possible for the system control unit208to detect operations below to a touch panel.

Operation to touch the touch panel with a finger or a pen (hereinafter, referred to as touch down)

State where a finger or a pen is in touch with the touch panel (hereinafter, referred to as touch on)

Operation to move a finger or a pen in the state where it is kept in touch with the touch panel (hereinafter, referred to as move)

Operation to touch the touch panel with two fingers and to move the fingers so as to narrow the interval therebetween (hereinafter, referred to as pinch in)

Operation to touch the touch panel with two fingers and to move the fingers so as to increase the interval therebetween (hereinafter, referred as pinch out)

Operation to let go a finger or a pen in touch with the touch panel (hereinafter, referred to as touch up)

State where nothing is in touch with the touch panel (hereinafter, referred to as touch off)

The above-mentioned various kinds of operations and the coordinates of the position where a finger or a pen is in touch with the touch panel are notified to the system control unit208and the system control unit208determines which operation is performed on the touch panel based on the notified information. With regard to move, it is also possible to determine the moving direction of the finger or the pen that moves on the touch panel for each of the vertical composition and the horizontal composition on the touch panel based on the change in the coordinates of the position. It is assumed that a stroke is described in the case where the series of operations from touch down on the touch panel to touch up after constant move is performed. The operation to quickly describe a stroke is called flick. Flick is the operation to quickly move a finger in touch with the touch panel a certain distance and then let the finger go. In other words, flick is the operation to quickly slide a finger on the touch panel so as to flick it. In the case where move of a predetermined distance or more at a predetermined speed or higher is detected and then touch up is detected, it is possible to determine that flick is performed. It is assumed that in the case where move of a predetermined distance or more at less than a predetermined speed is detected, it is determined that drag is performed. As a touch panel, it may also be possible to use any of various kinds of systems, such as the resistance film system, the electrostatic capacitance system, the surface acoustic wave system, the infrared system, the electromagnetic induction system, the image recognition system, and the optical sensor system.

The mode dial103switches the operation mode of the system control unit208to any of the image capturing mode capable of recording still images and moving images, the reproduction mode, etc. The shutter button102includes a first shutter switch and a second shutter switch, not shown schematically. The first shutter switch turns on during the operation of the shutter button102, i.e. by the so-called half-press operation (instructions to perform the image capturing preparation) and generates a first shutter switch signal SW1. The first shutter switch signal SW1 causes the operations, such as AF (Auto Focus) processing, AE (Auto Exposure) processing, AWB (Auto White Balance) processing, and EF (pre-flash) processing (hereinafter, at least one of these operations is referred to as an image capturing preparation operation). These pieces of processing are performed under control of the system control unit208. The second shutter switch turns on at the time of completion of the operation of the shutter button102, i.e. by the so-called full-press operation (instructions to perform image capturing) and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit208starts a series of operations of image capturing processing from reading a signal from the image capturing unit203to writing image data to the storage medium108.

To each operating member of the operation unit105, a function is allocated appropriately for each scene by performing the operation to select various kinds of function icons displayed on the display unit101and each operating member acts as each of the various kinds of function buttons. Function buttons include, for example, an end button, a return button, an image forward button, a jump button, a narrow-down button, an attribute change button, etc. For example, in the case where a menu button is pressed, a menu screen on which various kinds of settings can be performed is displayed on the display unit101. It is possible for a user to intuitively perform various kinds of settings using the menu screen displayed on the display unit101, a four-direction button, and a SET button. The controller wheel106is an operating member included in the operation unit105and capable of being operated to rotate and is used at the time of specifying a selected item together with the direction button.

A power source control unit213includes a battery detection circuit, a DC-DC converter, a switch circuit that switches blocks to be energized, etc., and detects whether or not a battery is attached, the kind of a battery, and the remaining battery life. Further, the power source control unit213controls the DC-DC converter based on the detection results and the instructions of the system control unit208and supplies a necessary voltage for a necessary period of time to each unit including the storage medium108.

A power source unit214includes a primary battery, such as an alkali battery and a lithium battery, a secondary battery, such as a NiCd battery, a NiMH battery, and a Li battery, an AC adaptor, etc.

An interface215is an interface with the storage medium108, such as a memory card and a hard disc.

FIGS. 3A to 3Care flowcharts showing a flow of image processing performed in the digital camera according to the present embodiment. The series of processing is implemented by the system control unit208developing programs stored in the nonvolatile memory211in the system memory212and executing the programs.

By a user turning on the power source, the system control unit208starts image capturing under predetermined image capturing conditions (focus, exposure, white balance, etc.) at step301. The image capturing conditions depend on the specifications of the apparatus, such as that the value before the power source is turned off is recovered and used.

At step302, the system control unit208displays a two-dimensional image acquired by the image capturing unit203on the display unit101.FIG. 4is a diagram showing an example of a two-dimensional image displayed on the display unit101. The two-dimensional image inFIG. 4is an image captured outdoors and the sky is photographed, and therefore, the distance to the farthest subject is an infinity. At this time, in the case where the “sky”, the farthest subject, is brought into focus (the focus distance is an infinity), the forward depth of field thereof becomes equal to ½ of the so-called “hyperfocal distance”. The value of the hyperfocal distance is obtained by expression (1) below.
(f^2)/(δ·F)  expression (1)
Here, f denotes the focal length (mm) of the lens, F denotes the f-stop of the lens, and δ denotes the diameter of the permissible circle of confusion (mm).

For example, in the case where the focal length of the lens is taken to be 50 mm, the f-stop of the lens at the time of full-aperture to be 2.8, and the diameter of the permissible circle of confusion to be 0.03328 mm, the hyperfocal distance is about 26.8 m and the forward depth of field is about 13.4 m. In this case, an area whose distance is greater than about 13.4 m is the “background area”. In the case where the farthest subject is brought into focus with full-aperture and a certain subject in the forward depth of field is brought into focus, all the subjects in this range are brought into focus as a result. Because of this, by displaying the areas of these subjects as one “background area”, it is possible to clearly demonstrate to a user that the subjects in this area cannot be classified into those which are brought into focus and those which are not brought into focus.

At step303, in the case where there is a predetermined user's operation as to the various kinds of settings, such as forced flash/flash inhibited/automatic, the system control unit208performs the setting.

At step304, the system control unit208determines whether there are instructions to change the display mode from a user. In the case where there are instructions to change the display mode, the procedure proceeds to step305. On the other hand, in the case where there are not instructions to change the display mode, the procedure returns to step302.

At step305, the system control unit208detects subjects included in a captured image. As the method for detecting subjects within a captured image, it may be possible to apply various kinds of methods, such as, for example, the method disclosed in Japanese Patent Laid-Open No. 1997-186931.FIG. 5is a diagram showing a state where a person, an automobile in profile, a traffic sign, and a building are detected, respectively, as subjects from the two-dimensional image shown inFIG. 4. InFIG. 5, each subject of a clump of trees, a mailbox, and an automobile facing the front is regarded to be at a distance of ½ of the hyperfocal distance at the time of the aperture being fully open or farther, and therefore, is included in the above-described “background area” together with the “sky”, which is the farthest subject.

As step306, the system control unit208acquires distance information (information of the distance to each subject detected from the two-dimensional image) of the captured two-dimensional image. Here, the distance information is information of the distance to each subject in the two-dimensional image recorded in association with the coordinates of the two-dimensional image, and for example, corresponds to a “distance image (distance map)” in which the magnitude of the depth of the coordinates (the distance from the camera) are represented by a shade of gray. As the method for obtaining distance information of a captured image, it may be possible to apply the various kinds of methods, such as, for example, the method disclosed in Japanese Patent Laid-Open No. 2001-169308.FIG. 6shows an example of a distance image generated for the two-dimensional image shown inFIG. 4. In the distance image inFIG. 6, the areas of the four subjects of the person, the automobile in profile, the traffic sign, and the building are expressed by images in a gray scale different in density according to each distance (the person whose distance is the shortest is expressed in the darkest shade of gray and the building in the lightest shade of gray). Those such as a clump of trees farther than the building are included in part of the background as described above, and therefore, they are not treated as “subjects” here.

At step307, the system control unit208performs processing to extract subject areas to be subjected to shift processing at the next step308based on the subjects detected at step305and the distance information acquired at step306. The subject area extraction processing can also be said as processing to separate subject areas for each fixed distance. For example, among the subjects detected at step305but not included in the background area, the portion within a range of a fixed distance (for example, 1 m±50 cm) is extracted as one of units of subject areas.FIG. 7is a diagram showing a state where the subject areas are extracted from the two-dimensional image shown inFIG. 4and the portion (each area of the person, the automobile in profile, the traffic sign, and the building) indicated by the broken line represents each extracted subject area.

At step308, the system control unit208generates a “shifted image” in which each subject area is shifted in a fixed direction by performing processing to change (shift) the position (coordinates) of each subject area extracted at step307in accordance with the distance information acquired at step306. For example, in the case where the distances to the four subject areas shown inFIG. 7correspond to 1 m, 3 m, 5 m, and m, respectively, an image is generated in which the coordinates of each subject area are shifted in the horizontal direction and in the vertical direction by predetermined amounts, respectively, (for example, 100 pixels, 33 pixels, 20 pixels, and 13 pixels, respectively). The amount of shift in the “shifted image generation” does not necessarily needs to be in inverse proportion to the distance and they may be in a logarithmic ratio or a correspondence table of the distance and the amount of shift is prepared in advance and the amount of shift may be determined by applying the table. At this time, in order to make the same the shift direction of each area, the rate between the amount of shift in the horizontal direction and the amount of shift in the vertical direction is made constant. Alternatively, it may also be possible to determine the rate of amount of shift between the horizontal direction and the vertical direction by processing as shown below.

FIG. 8is a flowchart showing a flow of processing to determine a rate of amount of shift between the horizontal direction and the vertical direction.

At step801, the system control unit208generates an image in which a shift is caused at a predetermined rate. In the stage immediately after the processing starts, a shifted image in which a shift is caused by a predetermined initial value (for example at a rate of 1:1) is generated.

At step802, the system control unit208determines whether there is an overlap of subject areas within the shifted image obtained at step801. In the case where there is an overlap of subject areas, the procedure proceeds to step803. On the other hand, in the case where there is no overlap of subject areas, the procedure proceeds to step805.

At step803, the system control unit208determines whether the ratio between the area of the hidden portion of a subject area to be displayed behind another subject area overlapping thereon and the area of the subject area to be displayed behind the other subject area exceeds a predetermined threshold value (for example, 50%). In this case, the smaller the ratio of the area of the hidden portion of the subject area to be displayed behind, the more appropriately it is made possible for a user to grasp the subjects. The predetermined threshold value is determined in advance in view of this point. In the case where the ratio of the area of the hidden portion exceeds the threshold value, the procedure proceeds to step804. On the other hand, in the case where the ratio of the area of the hidden portion does not exceeds the threshold value, the procedure proceeds to step805.

At step804, the system control unit208changes the rate at which the shift is caused in the horizontal and vertical directions and the procedure returns to step801. That is, the rate is changed so that the ratio between the area of the hidden portion of the subject area to be displayed behind the other subject area overlapping thereon and the area of the subject area to be displayed behind becomes smaller and then a shifted image in which a shift is caused in accordance with the changed rate is generated.

At step805, the system control unit208determines the rate in the case where the ratio of the hidden area of the subject area to be displayed behind becomes lower than the threshold value as the formal rate for display. The amount of shift in accordance with the rate determined as described above is determined as the amount of shift in shifted image generation.

In the shifted image generation processing, it is desirable to shift subject areas in the direction in which the subject area corresponding to the subject at the short distance (first subject area) appears lower than the subject area corresponding to the subject at the great distance (second subject area). The reason is that, due to this, it is possible to produce a display in which the scene is looked down upon by a user from the front and which is suitable to the feeling of the user. Further, the above-described background area (area included in the forward depth of field at the time of bringing the subject at the greatest distance in the two-dimensional image into focus with the aperture of the shutter being fully open) is taken to be a reference at the time of shifting the position (coordinates). The shift is caused with the “background area” as a reference because of the following. The object of the present invention is to make it easy for a user to grasp which subject is brought into focus and the area included in the depth of field at the time of the aperture being fully open in the case where the subject at the greatest distance is in focus is recognized to be in focus even in the case where the f-stop is changed.FIG. 9is a diagram showing the results of performing the shifted image generation processing on the image in which the four subject areas are extracted (seeFIG. 7) and it is known that the position (coordinates) of each subject area is shifted in accordance with its distance. In the example inFIG. 9, in order to make the same the direction in which each subject area is shifted, the rate of amount of shift between the horizontal direction and the vertical direction is made constant.

Explanation is returned to the flowchart inFIG. 3.

At step309, the system control unit208performs shear processing (oblique deformation processing) on each subject area in the “shifted image” generated by the shifted image generation processing at step308so that the horizontal direction is inclined. By performing the shear processing, it is made possible to produce a display more suitable to the feeling of a user that the user views the scene from the oblique front. To which degree the oblique deformation is applied is determined by a shear angle determined in advance.FIG. 10is a diagram showing the results of performing the shear processing on each subject area in the shifted image (seeFIG. 9). By the shear processing, it is made possible for a user to grasp the distance interval between subjects more easily. The shear angle may be a given angle determined in advance or it may also be possible to enable a user to specify the shear angle arbitrarily.

At step310, the system control unit208performs processing to superimpose (combine) a lattice pattern based on the shift direction and the shear angle. For example, a ground lattice pattern including segments in the same direction as the shift direction and segments in the same direction as the shear (oblique deformation) direction is generated and superimposed on the image.FIG. 11is a diagram showing the results of superimposing a lattice pattern on the image inFIG. 10having been subjected to the shear processing. Due to this, it is made possible for a user to grasp the distance interval between subjects still more easily. In the example inFIG. 11, two kinds of lattice patterns in total, i.e. a ground lattice pattern1101and a wall surface lattice pattern1102are superimposed and displayed. Due to this, it is made possible to produce a display still more suitable to the feeling that the user views the scene from the oblique front. It may also be possible to display the “background area” as a “wall” and not to display the lattice pattern corresponding to the greater distances. Due to this, it is made possible for a user to intuitively grasp the limit of the change in the depth of field etc. (that is, the fact that the focus distance or the depth of field cannot be changed for the portion displayed as the “wall”).

At step311, the system control unit208derives the forward depth of field and the backward depth of field from the image capturing conditions (focus, exposure, white balance, etc.) of a captured image. For example, by arithmetic operations using expression (2) and expression (3) below, the forward depth of field and the backward depth of field are derived, respectively.
Forward depth of field(mm)=(R^2·δ·F)/(f^2+R·δ·F)  expression (2)
Backward depth of field(mm)=(R^2·δ·F)/(f^2−R·δ·F)  expression (3)
In the above-mentioned expression (2) and expression (3), R denotes the distance (mm) to the subject in focus (however, not the infinity), f denotes the focal length (mm) of the lens, F denotes the f-stop of the lens, and δ denotes the diameter of the permissible circle of confusion (mm).

The diameter of the permissible circle of confusion is a characteristic value derived from the diagonal length of the image sensing element and for example, for the image sensing element having a width of 36 mm, a height of 24 mm, and a diagonal length of 43.27 mm, a value of 0.03328 mm is used.

At step312, the system control unit208performs processing to superimpose (combine) indexes indicative of the distance to the subject in focus (hereinafter, referred to as a “focus distance”) and indicating the forward depth of field and the backward depth of field derived at step311on the captured image. That is, in the positions corresponding to these distances on the ground lattice pattern superimposed at step310, the images that serve as indexes are superimposed.FIG. 12is a diagram showing the results of superimposing the index of the focus distance and the index of depth of field (the index of the forward depth of field and the index of the backward depth of field) on the image inFIG. 11with which the lattice pattern is combined. InFIG. 12, an index1201indicates the distance (focus distance) in the depth direction of the position where the person (subject P) is located along the ground lattice pattern, representing that the person (subject P) is in focus. Indexes1202and1203indicate the forward depth of field and the backward depth of field, respectively, along the ground lattice pattern, representing that the automobile (subject C) in profile is in the depth of field (it is possible to recognize that the subject C is in focus in the captured image).

At step313, the system control unit208displays the captured image with which each index is combined at step312on the display unit101. Then, as will be described later, by a user giving instructions and performing operations on the combined indexes, it is possible to control the image capturing lens201, the aperture of the shutter202, etc., to change the focus distance and the depth of field. At this time, it may also be possible to produce a highlighted display of the lattice line corresponding to the position of the index or to display the area of the subject corresponding to the position differently from the display of the areas of the other subjects in accordance with the user's operation to move the index. Due to this, it is possible for a user to grasp which subject is brought into focus or which subject is in the depth of field more clearly.

At step314, the system control unit208determines whether there is a touch operation on the display unit101by a user. In the case where there is a touch operation by a user, the procedure proceeds to step315. On the other hand, in the case where there is no touch operation by a user, the procedure returns to step305.

At step315, the system control unit208determines whether the touch operation by the user is the operation “to perform touch down in the vicinity of the index of the focus distance and then perform move”. In the case where the touch operation by the user is the operation “to perform touch down in the vicinity of the index of the focus distance and then perform move”, the procedure proceeds to step316. On the other hand, in the case where the touch operation by the user is the operation other than the operation “to perform touch down in the vicinity of the index of the focus distance and then perform move”, the procedure proceeds to step318.

At step316, the system control unit208adjusts the focus by controlling the image capturing lens201so that the subject at the distance corresponding to the position touched by a user is brought into focus.

At step317, the system control unit208moves the display position of the index indicative of the focus distance to the position touched by the user.

At step318, the system control unit208determines whether the touch operation by the user is the operation “to perform touch down in the vicinity of the index of the depth of field and then perform pinch in or pinch out”. In the case where the touch operation by the user is the operation “to perform touch down in the vicinity of the index of the depth of field and then perform pinch in or pinch out”, the procedure proceeds to step319. On the other hand, in the case where the touch operation by the user is the operation other than the operation “to perform touch down in the vicinity of the index of the depth of field and then perform pinch in or pinch out”, the procedure proceeds to step321.

At step319, the system control unit208adjusts the depth of field by controlling the image capturing lens201and the shutter202so that the subject between the distances corresponding to the user's two fingers (fingers in touch), respectively, is brought into focus. It is known that the depth of field increases by reducing the aperture in the case where the conditions, such as the focal length and the diameter of the permissible circle of confusion, are fixed. It is possible to adjust an increase or a reduction in the quantity of light due to the change in the f-stop by using the ND (Neutral Density) filter or by increasing or reducing the sensitivity of the image sensing element.

At step320, the system control unit208moves the display positions of the two indexes indicative of the depth of field to the positions of the user's two fingers in touch, respectively.

At step321, the system control unit208performs processing corresponding to the contents of the touch operation of the user. For example, in the case where any of the areas extracted from the captured image is touched by the user, the system control unit208controls the image capturing lens201so that the distance corresponding to the area is brought into focus and moves the display position of the index indicative of the focus distance to the corresponding position.

At step314, the system control unit208exits the present processing in accordance with a predetermined end operation, such as turning off of the power source by the user.

As described above, by the digital camera to which the image processing apparatus according to the present embodiment is applied, it is made possible for a user to intuitively grasp which subject is brought into focus on the UI thereof in the state where the image is close to the actual image.

Second Embodiment

Next, an aspect is explained as a second embodiment, in which the image processing apparatus according to the present invention is applied to an information processing apparatus that handles light field data. Explanation of the parts common to those of the first embodiment is simplified or omitted and here, different points are explained mainly.

First, light field data is explained.

Light field data (light ray information) is data recording the quantity of light for a light ray path. Specifically, in the data, for a light ray passing through coordinates on two parallel planes (coordinates (u, v) on a first plane and coordinates (x, y) on a second plane), a quantity of light L of the light ray is expressed by L (u, v, x, y) as a function of u, v, x, and y. In contrast to this, a two-dimensional image is expressed by the quantity of light of a light ray at the coordinates (x, y) on the single plane. For example, that which records the quantity of light at the coordinates (x, y) of the image sensing element is the two-dimensional image of a captured image. In other words, that in which the two-dimensional images are bundled in the number corresponding to the number of coordinates on the first plane is light filed data.

As a method for acquiring light field data by an image capturing apparatus, mentions is made of a method in which a microlens array is arranged in front of the image sensing element, a method in which the opening position of the aperture is controlled, etc. In summary, it is possible to obtain light field data by recording the quantity of light at the coordinates of the image sensing element in correspondence to the coordinates on a plane parallel to this (for example, the coordinates of the position of each microlens).

By integrating (calculating the sum of) the light field data L (u, v, x, y) with respect to part or the whole of the coordinates (u, v) on the first plane described above, a two-dimensional image that can be displayed on a two-dimensional display is obtained. That is, it is made possible to display a plurality of two-dimensional images corresponding to the coordinates in a fixed range on the above-described first plane configuring the light field data on a two-dimensional display by superimposing the plurality of two-dimensional images.

By superimposing two-dimensional images configuring light field data so that the position of a subject at a specific distance agrees with itself at the time of superimposing the two-dimensional images, it is possible to obtain a two-dimensional image in which the subject at the distance is in focus. On the contrary, the two-dimensional images configuring light field data each have a parallax corresponding to the distance of the separation between the coordinates on the first plane described above, and therefore, subjects at distances other than the specific distance are superimposed with positions shifted and their images are blurred on the obtained two-dimensional image as a result.

By superimposing the two-dimensional images configuring light field data by the amount corresponding to the coordinates in a small range on the first plane described above, the parallax of each two-dimensional image is limited, and therefore, it is possible to obtain a two-dimensional image with a large depth of field. That is, by changing the range on the above-described first plane corresponding to the two-dimensional images to be superimposed, it is made possible to change the depth of field of the two-dimensional image obtained by superimposition.

It is also possible to obtain a distance image from light field data as below by applying the “image correlation method”. That is, two coordinates are selected from the above-described first plane and the two two-dimensional images corresponding thereto are compared. For a small area around an arbitrary pixel in one of the two two-dimensional images, a small area having a similar pattern is specified in the other two-dimensional image, and an amount of movement of the two corresponding small areas between the two two-dimensional images is calculated. This amount of movement is the so-called “parallax” and is in inverse proportion to the distance of the subject displayed at the coordinates. Consequently, by finding the distance to the subject from the parallax for each coordinate and reflecting the distance in the color of the pixel at the coordinate (for example, a shade of gray), a distance image is obtained.

FIG. 13is a diagram showing an example of an internal configuration of an apparatus that performs information processing, such as, for example, a general personal computer (hereinafter, PC) and a tablet terminal, which handles light field data.

InFIG. 13, an apparatus1300includes a CPU1301, a hard disk (HD)1302, a memory1303, a display control unit1304, an input unit1305, a drive device1306, and a communication I/F1307and these units are connected to one another by an internal bus1308. It is possible for each unit connected to the internal bus1308to transmit and receive data to and from one another via the internal bus1308.

In the hard disk1302, image data, other data, various kinds of programs for the CPU1301to operate, etc., are stored. The memory1303includes, for example, a RAM. The CPU1301as a display control unit, a change unit, and a calculation unit controls each unit of the apparatus1300using the memory1303as a work memory in accordance with programs, for example, stored in the hard disk1302. The programs for the CPU1301to operate are not limited to those stored in the hard disk1302and it may also be possible to store the programs in advance, for example, in a ROM, not shown schematically.

The input unit1305receives a user's operation and generates a control signal in accordance with the operation, and then supplies the signal to the CPU1301. For example, the input unit1305has a character information input device, such as a keyboard, a pointing device, such as a mouse and a touch panel, etc., as an input device configured to receive a user's operation. The touch panel is an input device by which coordinate information in accordance with a touched position on the input unit1305configured, for example, into a planar shape is output. The CPU1301controls each unit of the apparatus1300in accordance with programs based on the control signal generated and supplied by the input unit1305in response to a user's operation performed to the input device. Due to this, it is possible to cause the apparatus1300to perform the operation in accordance with the user's operation.

The display control unit1304outputs a display signal for causing a display1310to display an image. For example, to the display control unit1304, a display control signal generated by the CPU1301in accordance with programs is supplied. The display control unit1304generates a display signal based on the display control signal and outputs the signal to the display1310. For example, the display control unit1304causes the display1310to display a GUI screen configuring a GUI (Graphical User Interface) based on the display control signal generated by the CPU1301.

In the case where a touch panel is used as the input unit1305, it may also be possible to integrally configure the input unit1305and the display1310.

On the drive device1306, an external storage medium1320, such as a CD and a DVD, can be mounted and the drive device1306reads data from the external storage medium1320mounted thereon and writes data to the external storage medium1320based on the control of the CPU1301. The external storage medium1320that can be mounted on the drive device1306is not limited to the disc storage medium, such as a CD and a DVD, and a nonvolatile semiconductor memory, such as, for example, a memory card, may be mounted on the drive device1306. The communication interface (I/F)1307communicates with a network1330, such as a LAN and the Internet, based on the control of the CPU1301.

FIG. 14is a flowchart showing a flow of image processing performed in the apparatus that handles light field data according to the present embodiment. The series of processing is implemented by the CPU1301developing programs recorded in the hard disk1302or a ROM, not shown schematically, onto the memory1303and executing the programs.

First, from the light field data obtained by the image capturing apparatus, a two-dimensional image is generated (at step1401) and the generated two-dimensional image is displayed on the display1310by the display control unit1304(step1402).

Then, whether there are instructions to change the display mode from a user is determined and in the case where there are instructions to change the display mode, the procedure proceeds to step1404and in the case where there are not instructions to change the display mode, the procedure returns to step1402(step1403).

Each piece of processing at step1404to step1412that follow corresponds to that at step305to step313in the first embodiment. That is, each piece of processing below is performed.

Processing to detect subjects included in the two-dimensional image (step1404)

Processing to acquire distance information (distance image) of the two-dimensional image (step1405)

Processing to extract subject areas based on the subjects detected at step1404and the distance information acquired at step1405(step1406)

Processing to generate a shifted image in accordance with the distance information for each extracted subject area (step1407)

Processing to perform shear processing to cause the horizontal direction to incline on the generated shifted image (step1408)

Processing to superimpose a lattice pattern based on the shift direction and a shear angle (step1409)

Processing to derive the forward depth of field and the backward depth of field of the two-dimensional image (step1410)

Processing to combine indexes each indicative of the distance to the subject in focus (focus distance), the forward depth of field, and the backward depth of field with the two-dimensional image (step1411)

Processing to display the two-dimensional image with which the indexes are combined on the display1310(step1412)

After each piece of processing described above, at step1413, whether there are user's instructions/operation for each index within the image displayed on the display1310is determined. In this case, the user's instructions/operation for the index are input using a mouse in the case of a general PC and are input by the same touch operation as that in the first embodiment in the case of a tablet terminal having a touch panel. In the case where there are user's instructions/operation for the index, the procedure proceeds to step1414and in the case where there are not user's instructions/operation, the procedure returns to step1404.

Then, in accordance with a new position of the index specified by a user, processing to change the focus distance and/or the depth of field is performed using the previously-described light field data (step1414). Accompanying this, the display position of the index indicative of the focus distance or the depth of field is moved to the new position specified by the user (step1415).

In the case where a predetermined exit operation is performed by a user, for example, where instructions to exit the application are given, the present processing is exited (step1416).

In the present embodiment also, at step1405described above, as distance information, for example, the distance image as inFIG. 6, is generated and by the shifted image generation processing at step1407, the “shifted image” in which the position of each subject area is shifted as inFIG. 9is obtained. That is, different from an image in which the viewpoint is only changed based on light field data (in this case, each subject is represented in a stereoscopic manner), an image in which each subject area is represented as a portable painting used as part of the backdrop (stage setting) (each subject area has no depth and is represented in a planar manner) is obtained. Due to this, it is made easier for a user to grasp the feeling of distance to each subject.

As above, by the apparatus according to the present embodiment also, it is made possible for a user to intuitively grasp which subject is brought into focus on the display of a PC etc. in the state where the image is close to the actual image.

Third Embodiment

In the first and second embodiments, it is made possible to intuitively grasp the range in which refocus can be performed by the image (shifted image) in which the position of each subject area is shifted in accordance with the distance of each subject area and which is close to the actual image. In a third and subsequent embodiments, an aspect is explained in which it is made possible to intuitively grasp the range in which refocus can be performed using an image in which the coordinates (arrangement) of each subject area are changed in accordance with the distance of each subject area.

FIG. 15is a block diagram showing an internal configuration of a camera array image capturing apparatus (also referred to simply as a “camera array”, as known as a camera array system, multiple lens camera, and the like) according to the present embodiment. An image capturing unit1500acquires image data (digital data) by receiving light information of a subject by an image sensing element and by A/D-converting the received signal. The image data is saved in a storage medium, such as an SD card, as captured image data upon receipt of instructions to perform image capturing from a user. The image data acquired by the image capturing unit1500according to the present embodiment is data of an image in which a subject space is captured from a plurality of viewpoints and which has parallax (hereinafter, referred to as “parallax image data”). The image data acquired by the image capturing unit1500is also utilized in the so-called live view function to sequentially display images in real time on a display unit1506provided at the rear side of the image capturing apparatus. In the following, an image saved in a storage medium in response to instructions to perform image capturing from a user is referred to as a record image and an image displayed in real time in the live view display is referred to as a live view image.

A central processing unit (CPU)1501comprehensively controls each unit described below. A RAM1502functions as a maim memory, a work area, etc. A ROM1503stores control programs etc. executed in the CPU1501. A bus1504is a transfer path of various kinds of data and for example, digital data acquired by the image capturing unit1500is sent to a predetermined processing unit via the bus1504. An operation unit1505configured to receive instructions of a user includes buttons, a mode dial, etc. In the display unit1506configured to display images and characters, for example, a liquid crystal display is used. The display unit1506may have the touch screen function and in such a case, it is also possible to handle user's instructions using the touch screen as inputs to the operation unit1505. In the present embodiment, in the case where the range in which refocus can be performed is displayed and the focus position at the time of refocus is specified, such a position is specified by an input of a user via such a touch screen.

A display control unit1507performs display control of images and characters displayed on the display unit1506. An image capturing unit control unit1508performs control of an image capturing system based on instructions from the CPU1501, such as focusing, shutter opening/closing, and aperture stop adjustment, etc. A digital signal processing unit1509performs various kinds of processing, such as white balance processing, gamma processing, and noise reduction processing, on digital data received via the bus1504. An encoder unit1510performs processing to convert digital data into a file format, such as JPEG and MPEG. An external memory control unit1511is an interface configured to connect to a PC and other media (for example, hard disk, memory card, CF card, SD card, USB memory). An image processing unit1512performs image processing, such as refocus processing, to be described later, on image data acquired by the image capturing unit1500or image data output from the digital signal processing unit1509. Details of the image processing unit1512will be described later. An exposure state prediction unit1513predicts an exposure state of the image capturing unit1500at the time of image capturing. There exist components of the image capturing apparatus other than those described above, however, they are not the main purpose of the present embodiment, and therefore, explanation is omitted.

First, a configuration of the image capturing unit1500is explained in detail.

As described above, the image capturing unit1500acquires parallax image data, i.e. light field data. A configuration example of the image capturing unit1500is shown inFIG. 16toFIG. 19. Among these,FIG. 16toFIG. 18each show a configuration in which a lens array is arranged on the image side of an image forming optical system andFIG. 19shows a configuration in which a plurality of image forming optical systems is arranged (camera array). As the configuration of the image capturing unit, a camera array configuration capable of obtaining images from a plurality of viewpoints at the same time as shown inFIG. 16toFIG. 19is desirable. This is because by a method of performing image capturing a plurality of times while changing the position using a single camera image capturing apparatus, what is obtained is parallax image data of images of a subject space captured at different times and in the case where there exists a moving object in the subject space, it is not possible to obtain accurate parallax information.

By performing processing to extract, rearrange, combine, etc., pixels, it is possible to perform refocus, depth of field control, viewpoint changing, etc., of the parallax image data acquired by the image capturing unit1500having the configuration shown inFIG. 16toFIG. 19. In the following, the processing as described above is referred to as image combination processing, and an image generated by the image combination processing is referred to as a combined image. It may also be possible to perform processing such as noise reduction, and processing such as depth of field control on a combined image. Further, a range in which refocus can be performed in a subject space is referred to as a “focus control range”.

Here, an internal configuration of the image capturing unit1500shown inFIG. 16is explained in detail.

A lens array1603is arranged on an image side conjugate surface of an image forming optical system1602with respect to a subject surface1601. Further, the lens array1603is configured so that an exit pupil of the image forming optical system1602and an image sensing element1604establish substantially a conjugate relationship. Light rays from the subject surface1601enter different pixels of the image sensing element1604in accordance with the position and angle of the light ray on the subject surface1601via the image forming optical system1602and the lens array1603. Due to this, parallax image data (light field data) is acquired. Here, the lens array1603has a role to prevent light rays having passed through different positions on the subject surface1601from entering the same pixel. As a result of that, an image in which a group of pixels of the same area on the subject surface1601captured from a plurality of viewpoints is arranged side by side is acquired in the image sensing element1604.

An object, such as a person and building, does not necessarily need to exist on the subject surface1601inFIG. 16toFIG. 19. This is because it is possible to focus a person or building that exists behind or in front of the subject surface1601by refocus after image capturing.

Hereinafter, refocus processing is explained. Refocus is explained in detail in “Fourier Slice Photography” by Ren Ng, 2005 ACM Trans. Graph. 24, p 735 to 744, and therefore, simple explanation is given here. The basic principles of refocus are common in each configuration inFIG. 16toFIG. 19. Here, the configuration inFIG. 16is explained as an example. InFIG. 16, the pupil of the image forming optical system is divided two-dimensionally into nine pupils (three in one dimension), and therefore, images from nine viewpoints are acquired as a result. Here, an image corresponding to a certain divided pupil is called a single-viewpoint image. The nine single-viewpoint images have parallax with respect to one another, and therefore, a relative positional relationship of the subjects on the image changes in accordance with the subject distance. In the case where single-viewpoint images are combined so that a certain subject overlaps itself, subjects located at different subject distances are combined in the shifted state. Because of this shift, subjects located at different subject distances are blurred. The blurring at this time is determined by the pupil corresponding to the single-viewpoint image used for combination and in the case where all the nine single-viewpoint images are combined, it is possible to reproduce the blurring of the image acquired by the image forming optical system1602. The subject overlapped on itself in combination of the single-viewpoint images is arbitrary, and therefore, it is possible to reproduce an image in which an arbitrary subject is brought into focus in the image forming optical system1602. This is the focus control after image capturing, that is, the principles of refocus.

Here, a method for generating a single-viewpoint image in the present embodiment is explained.FIG. 20is a diagram depicting a relationship between the lens array1603and the image sensing element1604inFIG. 16. A broken line circle2001represents an area of a pixel that a light ray having passed through one lens (microlens) configuring the lens array1603enters.FIG. 20corresponds to the case where a plurality of microlenses is arranged in the form of a lattice, however, the arrangement of microlenses is not limited to this. For example, the arrangement may have hexagonal symmetry (honeycomb structure). It may also be possible to slightly shift each microlens from a regular arrangement. A slashed area2002inFIG. 20represents a pixel that a light ray having passed through the same pupil area of the image forming optical system1602enters. Consequently, by extracting pixels in the slashed areas1602, it is possible to generate a single-viewpoint image in the case where a subject space is viewed from under the pupil of the image forming optical system1602. Similarly, by extracting pixels whose relative position with respect to the broken line circle2001is the same, another single-viewpoint image can also be generated.

Next, a focus control range in which refocus can be performed is explained.

Refocus is performed by superimposing single-viewpoint images, and therefore, it is not possible to refocus a subject blurred within each single-viewpoint image. This is because even by superimposing blurred images, a high frequency component is not obtained and the images remain blurred.

As the pupil is divided into smaller pupils, the depth of field in each single-viewpoint image becomes deeper, and therefore, the focus control range is extended. That is, the focus control range depends on the divided pupils of the image forming optical system1602. However, the depth of field in the single-viewpoint image does not necessarily agree with the focus control range. This is because the focus control range changes depending on the resolution ratio between the single-viewpoint images and the combined image obtained by combining them. For example, in the case where the resolution of the combined image is lower than the resolution of the single-viewpoint image from each viewpoint, the sampling pitch of the spatial component in the combined image becomes large relative to that in the single-viewpoint image. Because of this, the depth of field of the combined image becomes deeper than that of the single-viewpoint image and accompanying that, the focus control range is extended. Conversely, in the case where the resolution of the combined image is higher than that of the single-viewpoint image, the focus control range becomes narrower than the depth of field of the single-viewpoint image.

As a method for making the resolution of the combined image higher than that of the single-viewpoint image, the use of the super-resolution by pixel shifting etc. can be thought of. As described previously, in refocus, combination is performed by shifting the single-viewpoint images relative to one another. In the case where the amount of shift at this time is not an integer multiple of the pixel pitch of the single-viewpoint image, it is possible to perform the super-resolution by pixel shifting, and the resolution of the combined image can be increased.

From the discussion hitherto, it is known that it is necessary to take into consideration the conditions at the time of combining the single-viewpoint images in order to acquire an accurate focus control range in the combined image.

Next, how the focus control range of a combined image is derived is explained. The focal depth corresponding to the depth of field of a combined image is considered. A permissible circle of confusion of the focal depth is taken to be ε and the sampling pitch of the angular component of a light ray is taken to be Δu. At this time, coefficients α±of refocus are given by expression (4) below.

A range (range conjugate with respect to the image forming optical system1602) conjugate with a range α+s2to α_s2in which refocus can be performed on the image side expressed using the expression (4) is the focus control range, which is a range in which refocus can be performed on the object side.FIG. 21schematically shows the relationship of the expression (4) and the center position of the range in which refocus can be performed is the focus position of the image forming optical system1602(in this case, the position of the lens array1603is the center position of the range in which refocus can be performed). Here, s2is an interval between the image side main plane of the image forming optical system1602and the image side conjugate surface of the image forming optical system1602with respect to the subject surface1601. InFIG. 21, the image side focus control range refers to a range conjugate with the focus control range via the image forming optical system1602. Further, Δy is the sampling pitch of the two-dimensional intensity distribution of light and in the configuration inFIG. 16, it is equal to a pitch ΔLAof the lens array1603. The relationship of the expression (4) holds also in each of the configurations inFIG. 16toFIG. 19.

Then, because a pixel pitch Δ of the image sensing element1604is sufficiently small compared to an exit pupil distance P of the image forming optical system1602, and therefore, the expression (4) can be approximated as expression (5) below.
α±s2=s2∓NFεexpression (5)

Here, the exit pupil distance P of the image forming optical system1602is an interval between an exit pupil surface of the image forming optical system1602and the image side conjugate surface of the image forming optical system1602with respect to the subject surface1601. Further, N is the number of one-dimensionally divided pupils of the image forming optical system1602and F is the F-number of the image forming optical system1602. It is possible to derive the F-number by expression (6) below from a focal length f and a lens effective diameter D of the image forming optical system1602.
F=f/Dexpression (6)

That is, the range in which refocus can be performed (focus control range) is determined according to the number of one-dimensionally divided pupils of the image forming optical system and the range of the field angle defined by the focal length and the lens effective diameter of the image forming optical system. Further, according to the resolution of the combined image data, to be described later, the range in which refocus can be performed is determined.

Next, a configuration of the image processing unit is explained in detail.

FIG. 22is a block diagram showing an internal configuration of the image processing unit1512according to the present embodiment.

A parallax image data acquisition unit2201acquires parallax image data supplied from the image capturing unit1500etc.

The distance derivation unit2203drives information of the distance to the subject in the scene. The distance to the subject is derived using the parallax image data supplied from the parallax image data acquisition unit2201and the optical parameters supplied from the optical parameter acquisition unit2202. Details of distance derivation will be described later.

A subject area extraction unit2204performs processing to extract a subject area to be subjected to processing in a rearranged image generation unit2205based on the subject in the scene and the distance information supplied from the distance derivation unit2203. The subject area extraction processing can also be said as processing to separate subject areas for each fixed distance.

The rearranged image generation unit2205performs processing to change the position (coordinates) of each subject area extracted in the subject area extraction unit2204in accordance with the distance information supplied from the distance derivation unit2203. By this processing, an image is generated, in which each subject area is rearranged in accordance with the distance from the image capturing apparatus (hereinafter, referred to as a “rearranged image”) and by which it is made easy to grasp the feeling of distance for each subject area.

An image combination parameter acquisition unit2206acquires parameters (resolution after combination, viewpoint, focus position, depth of field, etc.) necessary for image combination.

The focus control range derivation unit2207derives a focus control range using optical parameters supplied from the optical parameter acquisition unit2202and image combination parameters supplied from the image combination parameter acquisition unit2206.

A focus position acquisition unit2208acquires a focus position at the time of image combination based on a user's input via the operation unit1505. In the case of the present embodiment, the distance specified by a user via the touch screen as the operation unit1505based on the rearranged image generated by the rearranged image generation unit2205is specified as the focus position. Information on the specified focus position is sent to a display image generation unit2209.

The display image generation unit2209generates an image in which the information on the focus state (focus position, focus control range, depth of field, etc.) is reflected (hereinafter, referred to as a “focus information display image”) for the rearranged image generated in the rearranged image generation unit2205. At the time of generation, the optical parameters, the image combination parameters, the focus control range, and the focus position supplied from each unit described above are used. The generation of the focus information display image may be performed on a live view image before image capturing or on a record image after image capturing. Further, the focus information display image may be based on an arbitrary single-viewpoint image included in the parallax image or may be based on a combined image obtained by combining a plurality of single-viewpoint images. The data of the generated focus information display image is output to the display unit1506. It is supposed that the display image generation unit2209according to the present embodiment also performs image combination processing in accordance with the image combination parameters using the parallax image data, however, it may also be possible to provide a processing unit configured to perform image combination processing separately from the display image generation unit2209.

FIG. 23is a flowchart showing a flow of processing in the image processing unit1512according to the present embodiment. The series of processing is performed by the CPU1501executing a computer executable program in which a procedure shown below is described after reading the program from the ROM1503onto the RAM1502.

At step2301, the parallax image data acquisition unit2201acquires parallax image data acquired by the image capturing unit1500. The parallax image data acquired here may be parallax image data as a live view image or may be parallax image data as a record image as described above.

At step2303, the distance derivation unit2203derives distance information in the whole of the scene. Specifically, the distance derivation unit2203derives the distance of the scene by a method or the like of performing stereo matching between the single-viewpoint images included in the parallax image. For example, in the stereo matching method, first, a single-viewpoint image that serves as a reference (for example, a single-viewpoint image corresponding to the light ray passing through the vicinity of the center of the pupil) is divided into a plurality of blocks and among the other single-viewpoint images, a block position nearest to the target block is detected. Then, by using the positional relationship between the two blocks, the positional relationship between the corresponding pupils, and the field angle of each pupil, the distance of the target block is derived by the triangulation. As the single-viewpoint image that serves as a reference, it may also be possible to select a single-viewpoint image the viewpoint position of which is other than the center. Further, as a method for acquiring distance information, there are a method, such as DFD (Depth From Defocus), and a method of performing measurement using a range finding unit that utilizes infrared rays etc., besides the stereo matching method described above.

FIG. 24Ais a diagram showing an example of a single-viewpoint image andFIG. 24Bshows a distance map as distance information derived for the single-viewpoint image inFIG. 24A. In the scene obtained in the single-viewpoint image shown inFIG. 24A, three kinds of subjects (person, building, mountain) exist at different distances with the image capturing apparatus as a reference. The distance map shown inFIG. 24Bis displayed by the shade in accordance with the distance from the image capturing apparatus, and the subject “person” in the vicinity of the image capturing apparatus is displayed by the color in the darkest shade, the subject “mountain” far from the image capturing apparatus by the color in the lightest shade, and the subject “building” located between the person and the mountain by the color in an intermediate shade. As described above, at this step, the distance information of the subjects in the scene is derived.

Explanation is returned to the flowchart inFIG. 23.

At step2304, the subject area extraction unit2204performs processing to extract subject areas based on the distance map of the whole of the scene derived at step2303. The subject area extraction processing can also be said as processing to separate subject areas for each fixed distance. Image areas having substantially the same distance are grouped and the grouped image area is extracted as a subject area. As the method for extracting subject areas, mention is made of, for example, methods as below besides the method that uses the distance information of subjects.

1) Areas in which a person, animal, plant, etc., exist are identified by applying a technique, such as face recognition, to the parallax image data and the identified person etc. is extracted as a subject area.

2) Data of images of representative objects of interest, such as a person, animal, and plant, is prepared in advance as a template and a subject area is extracted by pattern matching with the template.

3) Prior learning is performed as in a neural network and a subject area is extracted by recognizing main objects using the learning result.

Extraction of a subject area in the present embodiment can be implemented by applying a variety of publicly-known methods as mentioned in 1) to 3) described above besides the method that uses the distance information of subjects.

FIGS. 25A and 25Bare each a diagram showing an example of the results of extracting subject areas from the distance map shown inFIG. 24B, andFIG. 25Ashows the diagram in the case where the subject areas are extracted along the contour of each subject andFIG. 25Bshows the diagram in the case where rectangular areas circumscribing each subject are extracted as subject areas. InFIGS. 25A and 25B, each area surrounded by the dotted line is the subject area corresponding to “person”, “building”, and “mountain”, respectively. As the method for extracting subject areas, other methods can also be applied as long as areas are set so as to include main subjects. For example, it may also be possible to define a subject area by an arbitrary shape, such as a circle, ellipse, triangle, and polygon.

Explanation is returned to the flowchart inFIG. 23.

At step2305, the rearranged image generation unit2205generates a rearranged image of the scene using the distance information in the whole of the scene derived at step2303and the subject areas in the scene extracted at step2304.

FIGS. 26A to 26Dare each a diagram showing an example of a rearranged image in the case where the scene of the single-viewpoint image shown inFIG. 24Ais looked down upon from directly above. The “subject distance” inFIGS. 24A to 24Ddoes not necessarily need to be the scaled distance to the actual subject and, for example, the subject distance may be represented in a reciprocal ratio or a logarithmic ratio, or it may also be possible to apply a correspondence table by preparing in advance the table in which actual distances and corresponding values are associated with each other. In the case where the distant view, such as “mountain”, the actual distance of which may reach several kilometers or more, and “person” and “building” the actual distances of which are several meters to tens of meters are displayed by rearrangement with the actual distances being scaled, subjects in the close-range view are displayed densely in a small area compared to the subjects in the distant view. However, the “subject distance” serves as a guide for a user to grasp the focus state of a subject or to change the focus state of a subject, and therefore, the case where the subjects in the close-range view whose focus states are generally different are displayed densely is not appropriate because it becomes difficult for a user to grasp or change the focus state of a subject. Further, the difference between the focus states due to the distances of the subjects in the distant view is very slight compared to that of the subjects at short distances, and therefore, there will not arise any problem for a user to grasp the focus states of the subjects in the distant view even in the case where the subjects in the distant view are displayed densely. Consequently, in the case where the number of pixels of one side that can be displayed on the display unit of the image capturing apparatus is 640 and the subjects, such as “mountain”, “building”, and “person”, are located, for example, at distances of 10 km, 8 m, and 2 m, respectively, from the image capturing apparatus, by producing a display in which the subjects are rearranged in positions 0 pixels, 80 pixels, and 320 pixels, respectively, moved from the end part of the display area, the shorter distances the subjects are located at, the more sparsely the subjects are displayed on the display unit, and therefore, this display is convenient for a user to grasp or change the focus state.

In each ofFIGS. 26A to 26D, the position nearer to the image capturing apparatus is displayed at the lower part and the position farther from the image capturing apparatus is displayed at the upper part, and therefore, the “person” is arranged at the lower part because of being in the nearest position to the image capturing apparatus and the “mountain” is arranged at the upper part because of being in the farthest position from the image capturing apparatus.FIG. 26Ais a rearranged image in which the image areas cut out from the above-described single-viewpoint image along the subject areas extracted at step2304(seeFIG. 25Adescribed above) are rearranged in the positions corresponding to each subject distance. In the case of this rearranged image, a portion of the single-viewpoint image is arranged in association with the subject distance, and therefore, this is the rearranged image the closest to the actual image amongFIGS. 26A to 26D.FIG. 26Bis a rearranged image in which the image areas extracted along the above-described subject areas from the distance map derived at step2303(seeFIG. 24B) are rearranged in the positions corresponding to each subject distance. In the case of this rearranged image, each image area is displayed in a shade corresponding to the distance, and therefore, in this rearranged image, the subject distances are enhanced.FIGS. 26C and 26Dare each a rearranged image in which the frames of subject areas extracted at step2304(seeFIGS. 25A and 25Bdescribed above) are rearranged in the positions corresponding to each subject distance. In the case of these rearranged images, the rearranged image are more simplified.

Explanation is returned to the flowchart inFIG. 23.

At step2306, the focus control range derivation unit2207derives a focus control range in the case where image combination of the parallax image acquired at step2301is performed by using the optical parameters and the image combination parameters acquired at step2302. Detailed explanation is given below.

As described previously, the focus control range changes depending on the resolution ratio between each single-viewpoint image and the combined image, however, here, for simplification of explanation, a case where the resolution ratio is 1 is considered. In the case where the image side focus control range is within the range expressed by the above-described expression (5), it can be concluded that an area in which focus control can be performed is already acquired. Consequently, the image side conjugate surface of the image forming optical system1602with respect to the subject surface1601and an image side focus control range drefocusare required only to satisfy expression (7) below.
−NFε≦drefocus≦NFεexpression (7)

FIG. 27is a diagram showing an optical arrangement in the configuration of the image capturing unit1500shown inFIG. 16, and σ is an interval between the image side main plane of the lens array1603and the image sensing element1604. InFIG. 27, two parallel broken lines extending from both ends of the microlens at the center represent an area of the image sensing element1604corresponding to the microlens and pixels with slashes within the image sensing element1604represent a dead zone that no light rays enter. In the present embodiment, the lens array1603is configured so as to prevent a dead zone from being produced and in this case, ΔLA=ΔN holds. However, the configuration is not limited to this and there may exist a dead zone. FromFIG. 27, it is known that NF=σ/Δ holds in terms of geometry. By characterizing the size of the permissible circle of confusion that determines the focal depth by the sampling pitch Δy=ΔLAof the spatial component, the expression (7) can be rewritten as expression (8) below.

Next, a general case is considered, where the resolution ratio between the single-viewpoint image and the combined image is other than 1. The field angle of the combined image and the single-viewpoint image used for combination is the same, and therefore, in the case where the resolution ratio is other than 1, Δy differs between both. In general, the smaller Δy, the smaller the permissible circle of confusion is, and the larger Δy, the larger the permissible circle of confusion is. Because of this, it is possible to extend the above-mentioned expression (8) as expression (9) below by taking the ratio of Δy between the single-viewpoint image and the combined image.

Here, Rmonodenotes the resolution of the single-viewpoint image used for combination and Rcombdenotes the resolution of the combined image. The ratio of Δy is obtained by calculating the square root of the ratio between Rmonoand Rcomb. It is known that the resolution Rmonoof the single-viewpoint image is represented by expression (10) below fromFIG. 27.

Here, Rtotaldenotes the number of effective pixels of the image sensing element1604. From the expression (9) and the expression (10), conditional expression (11) below that the image side focus control range needs to satisfy is obtained.

In the above-mentioned conditional expression (11), the range is set between ±10.0, however, by setting the range between ±6.0, it is possible to obtain a clearer combined image. More desirably, by setting the range between ±3.0, it is possible to obtain a still clearer combined image.

In the following, a specific example of each value is shown.

Number Rtotalof effective pixels of the image sensing element1604: 46.7×106(pix)

Interval σ between the image side main plane of the lens array1603and the image sensing element1604: 0.0374 (mm)

Pixel pitch Δ of the image sensing element1604: 0.0043 (mm)

Focal length f of the image forming optical system1602: 14.0 (mm)

Number N of one-dimensionally divided pupils: 3

The range of the conditional expression (11), the resolution Rcombof the combined image corresponding to each range of the conditional expression (11), and drefocuscorresponding to each resolution of the combined image are, for example, as shown in Table (1) below.

By a user's input via the operation unit1505, the resolution Rcombof the combined image is selected, for example, from among the above-mentioned three kinds.

In the above-mentioned example, it is known that in order to generate a combined image of, for example, 8.0×106pix, the resolution needs to be increased by the super-resolution by pixel shifting etc. because the resolution Rmonoper single-viewpoint image is 5.2×106pix. The above-described value of each resolution Rcombof the combined image is an example and the number of kinds is not limited to three. It is only necessary to appropriately determine Rcomband drefocusthat satisfy the above-mentioned conditional expression (11).

It is possible to calculate the focus control range from the image forming formula by using the image side focus control range and the focal length and the focus position of the image forming optical system1602. The information of the focus control range acquired in this manner may be attached to the parallax image data or the combined image data as tag information and stored in the ROM1503. Alternatively, it may also be possible to create a table of the focus control range corresponding to a variety of conditions and store it in the ROM1503, and then to read data corresponding to an input condition instead of performing the above-described arithmetic operation to derive the focus control range.

As another method for acquiring a focus control range, a method can be thought of, which evaluates the contrast etc. of a subject existing in the focus position by actually generating a combined image for which refocus is performed. However, by this method, it is necessary to generate a combined image while moving the focus position and to sequentially determine whether refocus can be performed successfully, and therefore, it takes time to perform the processing. Further, in the case where no subject exists in the focus position where refocus is performed, evaluation cannot be carried out, and therefore, it is not possible to acquire an accurate focus control range. Consequently, it is desirable to use the previously-described method in order to acquire a focus control range.

Explanation is returned to the flowchart inFIG. 23.

At step2307, the display image generation unit2209generates a focus information display image, which is the rearranged image generated at step2305and in which information on the focus state (focus position, focus control range, depth of field, etc.) is reflected. At the time of generation, the optical parameters and the image combination parameters acquired at step2302and the focus control range derived at step2306are used.

FIGS. 28A and 28Bare each an example of the focus information display image generated based on the rearranged image shown inFIG. 26Aand each piece of information of a focus position2801, a focus control range2802, and a depth of field2803is displayed. The focus position2801indicated by a rectangular black mark on a slider bar indicates the focus position of the lens at the time of image capturing included in the optical parameters, or the focus position specified by a user's input, to be described later. In the focus information display image inFIG. 28A, the focus position2801is set in a position at a subject distance d0 (position of the person), and in the focus information display image inFIG. 28B, the focus position2801is set in a position at a subject distance d0′ (position of the building), respectively. The focus control range2802indicated by slashes on the slider bar indicates a range from a subject distance d1 (d1′) to a subject distance d2 (d2′) and in the present embodiment, it is known that the “person” and the “building” are located at distances in this focus control range. In the following, it is assumed that in the case where the “focus control range” is simply referred to, it refers to this object side focus control range. The depth of field2803indicated by slashes on the rearranged image is a range from a subject distance d3 (d3′) to a subject distance d4 (d4′) and in the focus information display image inFIG. 28A, the depth of field in the focus position d0 where the person is in focus is shown, and in the focus information display image inFIG. 28B, the depth of field in the focus position d0′ where the building is in focus is shown.

By displaying the focus information display image as described above, it is made possible for a user to intuitively grasp both pieces of information on the subject in the scene and on the focus state. InFIGS. 28A and 28B, for convenience of explanation, a schematic diagram of the image capturing apparatus and a viewing field range (field angle)2804where image capturing can be performed by the image capturing apparatus are also shown, however, it is not necessarily required to include these pieces of information in the focus information display image generated at this step.

Explanation is returned to the flowchart inFIG. 23.

At step2308, the display image generation unit2209outputs the data of the focus information display image generated at step2307to the display unit1506.

At step2309, the CPU1501determines whether a user's input relating to the focus position at the time of refocus processing is newly done. The user's input in this case is done via the touch screen or the operation unit1505of the image capturing apparatus. For example, there can be thought of a method for directly specifying a subject desired to be newly brought into focus on the rearranged image, a method for directly specifying the subject distance to a new focus position, or a method for specifying by operating a mark indicative of the focus position2801on the slider bar. In the case where an input of a new focus position is done, the procedure returns to step2307and the series of processing at step2307to step2309is repeated. For example, in a case where a user's input to set the “building” as a new focus position is done in the state where the focus information display image (FIG. 28A) in which the “person” is set in the focus position is displayed, the focus information display image shown inFIG. 28Bdescribed previously is newly displayed as a result. As described previously, inFIG. 28B, the depth of field2803has also changed in accordance with the newly set focus position (the subject distance d0′). That is, in the case where a new focus position at the time of refocus processing is input, the focus control range2802does not change but the focus position2801and the depth of field2803in the focus position change mainly as a result.

There can be thought of a case where the newly input focus position is beyond the focus control range. In this case, it may also be possible to notify an alarm to prompt a user to input a focus position within the focus control range. Alternatively, it may also be possible to limit the range a user can input so as not to receive a focus position beyond the focus control range.

On the other hand, in the case where there is no input of a new focus position, the present processing is exited.

In the example described above, the focus information display image based on the rearranged image in which the scene is looked down upon from directly above is explained. However, the focus information display image is not limited to the example described above and there can be though of a variety of aspects.FIGS. 29A to 29Ceach show an example of the variation of the focus information display image.FIG. 29Ais a focus information display image based on a rearranged image in which the scene is looked down upon from the lateral direction.FIG. 29Bis a focus information display image based on a rearranged image in which the scene is looked down upon from the obliquely above direction.FIG. 29Cis a focus information display image based on a rearranged image in which the scene is looked down upon from the obliquely lateral direction. As described above, the rearranged image that serves as a base of the focus information display image may be any image in which subjects are sequentially arranged in order in the depth direction based on distance information.

At step2308, the focus information display image is displayed, and at this time, it may also be possible to simultaneously display the single-viewpoint image acquired by the image capturing unit1500and/or the combined image combined in the display image generation unit2209.FIGS. 30A to 30Ceach show an example of the case where a single-viewpoint image and/or a combined image is displayed together in the focus information display image.

FIG. 30Ashows a case where a single-viewpoint and two kinds of combined images are displayed in addition to the focus information display image (seeFIG. 29Adescribed previously) based on the rearranged image in which the scene is looked down upon from the lateral direction. A combined image 1 inFIG. 30Ais a combined image in the state where the focus position is set by taking the “person” to be a subject of interest (the “building” and the “mountain” that are subjects other than the subject of interest are in the blurred state because of being outside the range of the depth of field). A combined image 2 inFIG. 30Ais a combined image in the state where the focus position is set by taking the “building” to be a subject of interest (the “person” and the “mountain” that are subjects other than the subject of interest are in the blurred state because of being outside the range of the depth of field). In the case where the combined image after refocus such as this is displayed at the same time, by the combined image in which only the subject of interest is in focus, it is made possible for a user to more intuitively check the specified subject of interest. InFIG. 30A, the rearranged image in which the scene is looked down upon from the lateral direction is used as a base, however, it is needless to say that any type of rearranged image may be used as a base. Further, in the example inFIG. 30A, two kinds of combined images are displayed, however, only one kind of combined image, or three or more kinds of combined images may be displayed.

FIG. 30Bshows a case where a combined image is displayed in the case where the focus position is set by taking the “person” to be the subject of interest, in addition to the focus information display image based on the rearranged image in which the scene is looked down upon from directly above. By producing the display in this manner, it is made possible to efficiently display the rearranged image, information on the focus state, and the combined image within the limited display screen. The focus information display image inFIG. 30Bis based on the rearranged image shown inFIG. 26Ain which the scene is looked down upon from directly above, from which information in the lateral direction is omitted, and in which only information in the depth direction is left.FIG. 30Cshows a case where a combined image is displayed in addition to the focus information display image based on the rearranged image in which the scene is looked down upon from directly above. By simultaneously displaying the single-viewpoint image in this manner, it is possible to intuitively grasp the focus state while checking the image at the time of image capturing. The focus information display image inFIG. 30Cis based on the rearranged image shown inFIG. 26Din which the scene is looked down upon from directly above, in which the subject areas are surrounded by rectangles, from which information in the lateral direction is omitted, and in which only information in the depth direction is left.

As above, it may also be possible to display the single-viewpoint image and the combined image together with the focus information display image.

In the focus information display images shown inFIG. 28toFIG. 30, it is possible to know each piece of information of the focus position, the focus control range, and the depth of field at the same time as the information on the focus state. However, it is not necessary to show all the pieces of information at the same time and they may be displayed separately. For example, it may also be possible to display only the focus position and the depth of field at first and then to display information of the focus control range in response to the instructions of a user.

As explained above, according to the present embodiment, at the time of image capturing or at the time of editing an image, information on the focus state, such as the focus position, the focus control range, and the depth of field at the time of image combination, is displayed based on the rearranged image in which subjects are arranged in order in the depth direction. Due to this, it is made possible for a user to intuitively grasp the focus state of a subject at the time of image capturing and/or at the time of editing the image.

Fourth Embodiment

In the third embodiment, explanation is given by premising the image capturing apparatus having the image capturing unit with the configuration shown inFIG. 16. Next, an aspect that premises an image capturing apparatus having the image capturing unit with the configuration shown inFIG. 17andFIG. 18is explained as a fourth embodiment. In the following, the points peculiar to the present embodiment are explained mainly.

With the configuration shown inFIG. 17andFIG. 18, data of a parallax image in which a plurality of small images from different image capturing viewpoints and with different image capturing ranges is arranged in order is acquired by the image sensing element1604. In the configuration inFIG. 17, the lens array1603is arranged between the image forming optical system and the image side conjugate surface (left side of the image side conjugate surface). In contrast to this, in the configuration inFIG. 18the lens array1603is arranged outside (right side of the image side conjugate surface), not between the image forming optical system and the image side conjugate surface. Because of this, in the case of the configuration inFIG. 18, the lens array1603regards the image formed by the image forming optical system1602as a real object and forms an image again on the image sensing element1604. However, in both the configurations, the lens array1603regards the image formed by the image forming optical system1602as an object and forms an image of the object on the image sensing element1604, and therefore, both the configurations are the same essentially. In the following, the configuration inFIG. 17is explained as an example, however, the contents of the explanation also apply to the configuration inFIG. 18.

First, image combination (refocus) processing in the present embodiment is explained.

Qualitatively, the processing is similar to that in the third embodiment and in the processing, images of divided pupils of the image forming optical system1602are superimposed on one another with the amount of shift corresponding to the distance of the subject desired to be brought into focus.

FIG. 31is an enlarged view of the portion of the lens array1603and the image sensing element1604as a main portion in the configuration inFIG. 17. In the present embodiment, the lens array1603is configured by microlenses whose surface on the object side is flat and whose surface on the image side is convex. Of course, the shape of the lens array1603is not limited to this. InFIG. 31, the alternate long and short dash line represents the field angle of each microlens. By projecting pixel values obtained by the image sensing element1604onto a virtual image forming surface via the microlens corresponding to each pixel and combining them, it is possible to generate a combined image in which the virtual image forming surface is in focus. Here, the virtual image forming surface refers to a surface conjugate with the surface on the side of the object desired to be brought into focus by image combination (surface conjugate via the image forming optical system1602). For example, in order to generate an image in which the subject surface1601is in focus inFIG. 17, a virtual image forming surface is set to an image side conjugate surface1701. InFIG. 31, for easier understanding, the pixels projected at the time of combined image generation are represented by the broken line being shifted in units of field angle of each microlens. It may also be possible to generate a combined image by a method for combining images by translating each pixel as long as the pixels are superimposed on one another in the same way as that by the above-described method (combination method in which the pixel values obtained by the image sensing element1604are projected onto a virtual image forming surface via the microlens corresponding to each pixel). At this time, in the case where the area of the lens array1603through which the luminous flux incident on pixels has passed is the same, the amount of translation of the pixels is the same. That is, the operation of the pixel at the time of combined image generation inFIG. 17andFIG. 18is determined in accordance with the area of the lens array1603through which the luminous flux incident on pixels has passed.

Next, the focus control range in the present embodiment is explained.

The focus control range in the present embodiment is also described by the same expression (4) as in the third embodiment. That is, the range (range conjugate with respect to the image forming optical system1602) conjugate with the refocus range α+s2to α−s2on the image side expressed using the expression (4) described previously is the focus control range, which is the refocus range on the object side.FIG. 32is a diagram corresponding toFIG. 21according to the third embodiment. Δy inFIG. 32represents the sampling pitch of the two-dimensional intensity distribution of light and Δy=Δτ1/σ2(σ1: interval between the image side conjugate surface1701and the object side main plane of the lens array1603, σ2: interval between the image side main plane of the lens array1603and the image sensing element1604). This is because the lens array1603regards the image formed by the image forming optical system1602as a virtual object, and therefore, forms a reduced image on the image sensing element1604with a magnification of σ2/σ1. Then, in the present embodiment also, the expression (4) can be approximated as the expression (5) because Δ<<P (P: exit pupil distance of the image forming optical system1602).

The flow of the processing in the image processing unit1512according to the present embodiment is the same as that in the flowchart inFIG. 23according to the third embodiment, however, the configuration of the image capturing unit1500is different, and therefore, the focus control range is derived as follows (step2306).

As is obvious fromFIG. 32, NF=σ1/ΔLAand Δy=Δσ1/σ2hold in terms of geometry, and therefore, expression (12) below holds.

From the expression (12) etc., conditional expression (13) that the image side focus control range drefocusin the present embodiment should satisfy is found.

As in the third embodiment, by replacing the range, which is ±10.0 in the above conditional expression (13), with ±6.0 or ±3.0, it is possible to obtain a clearer combined image.

In the following, specific examples are shown.Number Rtotalof effective pixels of the image sensing element1604: 150.0×106(pix)σ1: 0.3712 (mm)σ2: 0.0740 (mm)Pixel pitch Δ of the image sensing element1604: 0.0024 (mm)Pitch ΔLAof the lens array1603: 0.0256 (mm)Focal length fwat the wide-angle end of the image forming optical system1602: 72.2 (mm)Focal length fTat the telephoto end of the image forming optical system1602: 194.0 (mm)F-number (from the wide-angle end to the telephoto end): 2.9Number N of one-dimensionally divided pupils: 5Resolution Rmonoper single-viewpoint image: 6.0×106pix

The range of the conditional expression (13), the resolution Rcomdof the combined image corresponding to each range expressed by the conditional expression (13), and drefocuscorresponding to each resolution of the combined image are, for example, as shown in Table (2) below.

As in the third embodiment, by a user's input via the operation unit1505, the resolution Rcomdof the combined image is selected, for example, from among the above-mentioned three kinds.

In the above-mentioned example, in order to generate a combined image of, for example, 10.0×106pix, the resolution Rmonoper single-viewpoint image is 6.0×106pix, and therefore, it is known that it is necessary to increase the resolution by the super-resolution by pixel shifting etc.

Fifth Embodiment

Next, an aspect that premises an image capturing apparatus having an image capturing unit with the configuration (camera array) shown inFIG. 19is explained as a fifth embodiment. In the following, the points peculiar to the present embodiment are explained mainly.

FIG. 33is a diagram of the image capturing unit1500having the configuration of the camera array inFIG. 19according to the present embodiment in the case where the image capturing unit1500is viewed from the front (object side) and an arrangement of image forming optical systems1602ato1602gand image sensing elements1604ato1604gin each of seven image capturing modules is shown. In the case of the image capturing unit1500, the camera array has hexagonal symmetry with an optical axis of the image forming optical system1602bas an axis of rotation. However, the configuration of the camera array is not limited to this and the number and array of the image forming optical systems are arbitrary. Further, each of the image sensing elements1604ato1604gis arrayed in correspondence to each of the image forming optical systems1602ato1602g, however, in the case where it is possible to acquire the image formed by each of the image forming optical systems1602ato1602g, the number of image sensing elements may be one.

In the image capturing unit1500with the configuration shown inFIG. 33, the light ray refracted by each of the image forming optical systems1602ato1602gis received by each of the image sensing elements1604ato1604gcorresponding thereto. A plurality of images acquired by the image sensing elements1604ato1604gare parallax images of a subject space observed from different viewpoints. By performing image combination using the plurality of parallax images, it is possible to obtain the light field of the subject space.

FIG. 34is a diagram (sectional view) of the image forming optical system1602aand the image sensing element1604aviewed from the side. The other image forming optical systems1602bto1602gand image sensing elements1604bto1604gare the same, however, the configuration of each image forming optical system may differ from one another. The image forming optical system1602ais a single focus lens and performs refocus by changing the interval between the image forming optical system1602aand the image sensing element1604a.

As in the third embodiment, in the image combination processing in the present embodiment also, the image from each viewpoint is superimposed on one another with the amount of shift corresponding to the distance of the subject desired to be brought into focus. The focus control range is also the same as that expressed by the expression (4) described previously.FIG. 35is a diagram corresponding toFIG. 21of the third embodiment, and in the case of the present embodiment, Δy=Δ and Δu=Pmono/Fmonohold. Here, Fmonorepresents the F-number in each of the image forming optical systems1602ato1602gand Pmonorepresents the exit pupil distance corresponding to each image forming optical system. Then, in the present embodiment, Δ<<Pmono, and therefore, the expression (4) can be approximated as expression (14) below.
α±s2=s2∓FmonoΔy=s2∓FmonoΔ  expression (14)

The flow of the processing in the image processing unit1512according to the present embodiment is the same as that in the flowchart inFIG. 23according to the third embodiment, however, the configuration of the image capturing unit1500is different, and therefore, the focus control range is derived as follows (step2302and step2306).

In the case of the present embodiment, each of the image forming optical systems1602ato1602gconfiguring the image capturing unit1500has an aperture the f-stop of which is variable. Because of this, at step2302, the f-stop of each image forming optical system at the time of image capturing is acquired as an optical parameter.

At step2306, based on the same concept as that of the third embodiment, the F-number of each of the image forming optical systems1602ato1602gat the time of image capturing obtained at step2302is substituted for Fmonoand the focus control range is derived. Here, by taking the resolution of the image formed by the image forming optical system having the F-number of Fmonoto be Rmono, conditional expression (15) that drefocusshould satisfy is found.

As in the third embodiment, by replacing the range, which is ±10.0 in the above-mentioned conditional expression (15), with ±6.0 or ±3.0, it is possible to obtain a clearer combined image.

In the following, specific examples are shown.Number Rmonoof effective pixels of each of the image sensing elements1604ato1604g: 19.3×106(pix)Pixel pitch Δ: 0.0012 (mm)Focal length f of each of the image forming optical systems1602ato1602g: 50.0 (mm)F-number (Fmono): 1.8 mThe range of the conditional expression (15), the resolution Rcombof the combined image corresponding to each range expressed by the conditional expression (15), and drefocuscorresponding to each resolution of the combined image are, for example, as shown in Table (3) below.

Then, as in the third embodiment, by a user's input via the operation unit1505, the resolution Rcombof the combined image is selected, for example, from among the above-mentioned three kinds.

In the case where the F-number at the time of image capturing is a different one, drefocusis determined so as to satisfy the conditional expression (15).

Sixth Embodiment

The image combination processing explained in the third to fifth embodiments is implemented by using the parallax images acquired by the image capturing apparatus having the image capturing unit shown inFIG. 16toFIG. 19and obtained by performing image capturing from a plurality of different viewpoints. The image combination processing is not limited to the image combination processing using parallax images. It may also be possible to adjust the focus position and the depth of field of the data of a captured image from a single viewpoint later by image processing.

For example, as a method for changing the focus position by image processing, the method for performing filter processing on captured image data in accordance with the relative distance from the focus position is known. In this case, first, data of a captured image with a deep depth of field and distance information of the scene are acquired. The distance information is acquired by providing a distance sensor separately from the image capturing apparatus or by analyzing the captured image data of the image capturing apparatus. Next, for a subject (or subject distance) desired to be brought into focus, the original captured image data is used without performing filter processing, and for the areas other than the subject desired to be brought into focus, the relative distance from the subject is found from the distance information. Then, for the area whose relative distance from the subject desired to be brought into focus is short, filter processing to reduce the amount of blur is performed (for example, the setting to relatively reduce the filter size). On the other hand, for the area whose relative distance from the subject desired to be brought into focus is great, filter processing to increase the amount of blur is performed (for example, the setting to relatively increase the filter size). By the method such as this, the focus position may be changed.

As described above, by the method for performing filter processing on captured image data in accordance with the relative distance from the focus position for the single-viewpoint image, the focus control range is the depth of field at the time of acquisition of the single-viewpoint image.

It is also possible to adjust the degree of depth of field by image processing using a method similar to that described above. That is, in the filtering processing on the captured image data, by the setting to increase the filter size, the amount of blur increases and it is possible to generate data of an image with a narrow depth of field. On the contrary, in order to generate data of an image with a deep depth of field, the filter size is reduced.

In the third to sixth embodiments, an image in which the focus position etc. is changed is generated by performing image combination processing on an image from a plurality of viewpoints or an image from a single viewpoint. However, it is possible to obtain an image in which the focus position etc. is changed without the need to perform image combination processing.

For example, the method for acquiring a plurality of pieces of captured image data is known, which acquires a plurality of images by bracket image capturing while changing the image capturing parameter little by little. Specifically, data of captured images in which a focus adjustment ring and an aperture adjustment ring included in the image capturing apparatus are changed stepwise is acquired. It is assumed that in the captured image data by bracket image capturing, the parameters at the time of image capturing are held integrally with the captured images. That is, in the third to fifth embodiments, from one piece of captured image data including images from a plurality of viewpoints, a combined image in which the focus position etc. is changed is generated by an arithmetic operation, however, in the present embodiment, from a plurality of pieces of captured image data obtained by bracket image capturing, a desired captured image is selected and displayed.

In the bracket image capturing in which the focus position is changed stepwise (focus bracket image capturing), image capturing is performed a plurality of times by changing the focus position stepwise while using the operation unit configured to make focus adjustment included in the image capturing apparatus. InFIG. 26Adescribed previously, the three kinds of subjects (person, building, mountain) exist at distances different from the image capturing apparatus and the “person” is located in the nearest position from the image capturing apparatus, the “building” in the intermediately distant position, and the “mountain” in the farthest position, respectively. In the present embodiment, captured image data in each focus position is acquired by continuously changing the focus position so that the subjects located from the nearest position to the farthest position are included. Due to this, it is possible to obtain captured images in which each subject of the “person”, the “building”, and the “mountain” is in focus. In the actual focus bracket image capturing, an image is also captured, in which the portion in which no main subject exists (for example, between the person and the building) is in focus. Then, in the obtained captured image data, the information of the focus position at the time of image capturing is recorded as distance information in association with the captured image. As in the sixth embodiment, it may also be possible to acquire the distance information of the scene by providing a distance sensor separately from the image capturing apparatus or by analyzing the captured image data of the image capturing apparatus.

As described above, by the bracket image capturing, data of captured images in which the focus position is changed stepwise is displayed without performing image combination processing. In this case, the focus control range is from the nearest focus position to the image capturing apparatus to the farthest focus position among the focus positions changed at the time of bracket image capturing. Further, it is possible to handle the depth of field substantially the same as the depth of field determined by the aperture adjustment ring of the image capturing apparatus at the time of bracket image capturing.

Furthermore, in the case where images in which the focus position is changed are displayed in addition to the focus information display image as inFIGS. 30A and 30Bdescribed previously, a desired captured image is selected and displayed in accordance with the focus position supplied from the focus position acquisition unit2208.

It may also be possible to obtain data of images in which the focus position is changed using a plurality of pieces of captured image data obtained by the bracket image capturing as explained above.

Other Embodiments

This application claims the benefit of Japanese Patent Application Nos. 2013-134111, filed Jun. 26, 2013, and 2013-043264, filed Mar. 5, 2013 which are hereby incorporated by reference herein in their entirety.