Patent Publication Number: US-8970714-B2

Title: Image capturing apparatus, image processing apparatus, and method thereof

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to an imaging process applied to image data captured by an image capturing apparatus such as a multi-eye camera. 
     2. Description of the Related Art 
     There exists a technique of generating an image from a plurality of images captured at different positions by changing the focus or the depth of field. In this technique, a plurality of images are deformed in accordance with the capturing position and the distance to the object to be in focus. The deformed images are composited to generate an image having a shallow depth of field (refocus process). 
     There also exists a technique of changing the depth of field so as to make all objects fall in focus. In the invention of Japanese Patent Laid-Open No. 2011-022796, the refocus process is performed based on a plurality of images from different viewpoints. If there are a plurality of objects, focus on them is obtained by pan-focus. However, since these techniques attain focus on the plurality of objects by deepening the depth of field or using pan-focus, an image having a shallow depth of field cannot be obtained. 
     In addition, a technique has been proposed to composite an image having focus on a plane that does not face a camera. However, if a plurality of objects to be in focus are not placed on the same plane but interspersed, an image having a shallow depth of field and focus on the objects cannot be generated even by this technique. 
     SUMMARY OF THE INVENTION 
     In one aspect, an image processing method comprising the step of compositing multi-view image data obtained by capturing from a plurality of viewpoints so as to generate synthetic image data having focus on a curved focus surface. 
     According to the aspect, it is possible to generate image data having focus on an arbitrary focus surface from a plurality of image data obtained by capturing an object from different capturing positions. 
     Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic view for explaining the schematic arrangement of an image capturing apparatus according to an embodiment. 
         FIG. 2  is a block diagram for explaining the arrangement of an image processing apparatus according to the embodiment. 
         FIG. 3  is a block diagram for explaining the functional arrangement of the image processing apparatus. 
         FIG. 4  is a flowchart for explaining image processing. 
         FIGS. 5A and 5B  are views for explaining the process of a focus coordinate acquisition unit. 
         FIG. 6  is a flowchart for explaining the process of a distance estimation unit. 
         FIG. 7  is a view for explaining distance calculation. 
         FIG. 8  is a flowchart for explaining the process of a virtual focus surface setting unit. 
         FIG. 9  is a view for explaining the relationship between focus points and focus coordinates. 
         FIGS. 10A to 10E  are views for explaining virtual focus surface generation. 
         FIG. 11  is a flowchart for explaining the process of a virtual focus surface generation unit. 
         FIGS. 12A to 12C  are views for explaining a virtual focus surface (curved surface) and a virtual focus surface image. 
         FIG. 13  is a flowchart for explaining the process of an image composition unit. 
         FIGS. 14A and 14B  are views for explaining weighting coefficients. 
         FIG. 15  is a block diagram for explaining the functional arrangement of an image processing apparatus according to the second embodiment. 
         FIG. 16  is a flowchart for explaining image processing according to the second embodiment. 
         FIG. 17  is a flowchart for explaining the process of a virtual focus surface setting unit. 
         FIG. 18  is a view for explaining grouping focus points. 
         FIG. 19  is a view showing an example of an object region recognition result. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     Image processing according to the embodiments of the present invention will now be described in detail with reference to the accompanying drawings. 
     First Embodiment 
     Schematic Arrangement of Image Capturing Apparatus 
     The schematic arrangement of an image capturing apparatus  140  according to the embodiment will be described with reference to the schematic view of  FIG. 1 . 
     As shown in  FIG. 1 , a plurality of image capturing units  131  to  139  are arranged on the front surface of the image capturing apparatus  140 . A release button  130  to instruct the start of image capturing is arranged on the upper surface. The image capturing units  131  to  139  are uniformly arranged in a square matrix. The vertical axes, horizontal axes, and optical axes of these image capturing units are directed in the same directions. 
     The image capturing apparatus  140 , which is called a multi-view camera or a multi-eye camera, composites a plurality of images captured by the image capturing units  131  to  139  to generate an image having a shallow depth of field. At this time, the apparatus generates an image having a shallow depth of field in which focus is obtained on all arbitrary objects that are not on a plane, and a blur occurs in regions outside that portion. 
     When the user presses the release button  130 , each of the image capturing units  131  to  139  extracts, from an image capturing device, an analog signal corresponding to the light information of an object whose image is formed on the image capturing device through a capturing lens, a diaphragm, and the like. The analog signal undergoes analog/digital conversion and image processing such as demosaicing so as to output image data. 
     The image capturing apparatus  140  can obtain an image data group by capturing a single object from a plurality of viewpoints. In this example, the image capturing apparatus includes nine image capturing units. The present invention is applicable as far as the apparatus includes a plurality of image capturing units, and the number of image capturing units is arbitrary. The image capturing units need not always be arranged uniformly in a square matrix and can be arranged arbitrarily. For example, the image capturing units may be arranged radially or linearly, or completely at random. 
     The present invention can also be applied to a plurality of image data obtained by capturing an object from different viewpoints while moving one image capturing unit using a pan head, a robot arm, or the like. 
     [Image Processing Apparatus] 
     Arrangement 
     The arrangement of an image processing apparatus according to the embodiment will be described with reference to the block diagram of  FIG. 2 . Note that the image processing apparatus can be either incorporated in the image capturing apparatus  140  or separated from the image capturing apparatus  140  so as to communicate with the image capturing apparatus  140  via an interface to be described later and thus perform image processing to be described later. 
     A microprocessor (CPU)  101  executes various processes including image processing to be described later by executing programs stored in a read only memory (ROM)  103  and a hard disk drive (HDD)  105  using a random access memory (RAM)  102  as a work memory and controlling components to be described later through a system bus  100 . 
     An image capturing interface (I/F)  104  is a serial bus interface such as a USB (Universal Serial Bus) or IEEE1394 and is connected to the image capturing apparatus  140 . The CPU  101  can control the image capturing apparatus  140  via the image capturing I/F  104  to perform image capturing and receive captured data from the image capturing apparatus  140 . 
     An HDD I/F  119  is, for example, a serial ATA interface and is connected to, for example, a secondary storage device such as the HDD  105  or an optical disk drive. The CPU  101  can read out data from the HDD  105  or write data to the HDD  105  through the HDD I/F  119 . The CPU  101  can also expand, on the RAM  102 , data stored in the HDD  105  or conversely store, in the HDD  105 , data expanded on the RAM  102 . 
     A general-purpose I/F  106  is a serial bus interface such as a USB or IEEE1394 and is connected to an input device  107  such as a keyboard, a mouse, or a touch panel, or a printer (not shown). A video I/F  108  is an interface such as a digital visual interface (DVI) or a high-definition multimedia interface (HDMI®) and is connected to a monitor  109  or the like. The CPU  101  displays a user interface (UI) on the monitor  109 . 
     Functional Arrangement 
     The functional arrangement of the image processing apparatus will be described with reference to the block diagram of  FIG. 3 . Note that the functional arrangement shown in  FIG. 3  is implemented by causing the CPU  101  to execute an image processing program. Details of each process of the functional arrangement will be described later. 
     An image input unit  110  inputs captured data and image capturing apparatus information representing the information of the apparatus (for example, the image capturing apparatus  140 ) that has captured the data and image capturing information (for example, shutter speed and stop number) from the image capturing apparatus  140 , the HDD  105 , or the like. Note that the captured data includes a plurality of image data captured by the plurality of image capturing units (for example, the image capturing units  131  to  139  of the image capturing apparatus  140 ). The image capturing apparatus information also represents the angle of view of the image capturing apparatus and the capturing positions (relative positional relationship) of the image capturing units. 
     A focus coordinate acquisition unit  111  receives reference image data and outputs focus coordinates representing the position where focus is obtained based on a user instruction. Note that the focus coordinate acquisition unit  111  may output focus coordinates input from the HDD  105  or the like. The reference image data is one of the plurality of image data. The reference image data can be any of the plurality of image data. For example, image data captured by the image capturing unit  135  that is located at the center out of the image capturing units  131  to  139  of the image capturing apparatus  140  is used as the reference image data. 
     A distance estimation unit  112  receives the plurality of image data and the image capturing apparatus information and estimates the depth value of a captured scene by stereo matching, thereby generating a depth image (distance map). A virtual focus surface setting unit  113  receives the focus coordinates and the depth image and calculates a curved surface parameter and a focus point representing three-dimensional coordinates. 
     A virtual focus surface generation unit  114  receives the curved surface parameter and the focus point and generates a virtual focus surface image. An image composition unit  115  receives the plurality of image data, the image capturing apparatus information, the depth image, and the virtual focus surface image and generates synthetic image data. Note that the CPU  101  displays the synthetic image data on the monitor  109  or stores it in the HDD  105 . 
     Image Processing 
     Image processing will be described with reference to the flowchart of  FIG. 4 . Note that the image processing shown in  FIG. 4  is processing executed by the CPU  101  when, for example, a user instruction is input via the UI. Details of each process will be described later. 
     The CPU  101  causes the image input unit  110  to input captured data designated by the user and image capturing apparatus information (step S 101 ) and the focus coordinate acquisition unit  111  to acquire focus coordinates Pn from reference image data (step S 102 ). 
     Next, the CPU  101  causes the distance estimation unit  112  to execute distance estimation processing and thus generate a depth image (step S 103 ) and the virtual focus surface setting unit  113  to execute virtual focus surface setting processing and thus calculate a curved surface parameter S and a focus point pn (step S 104 ). 
     The CPU  101  then causes the virtual focus surface generation unit  114  to execute virtual focus surface generation processing and thus generate a virtual focus surface image (step S 105 ). The CPU  101  causes the image composition unit  115  to execute image composition processing and thus generate synthetic image data (step S 106 ) and output the synthetic image data (step S 107 ). 
     Focus Coordinate Acquisition Unit 
     The process (step S 102 ) of the focus coordinate acquisition unit  111  will be described with reference to  FIGS. 5A and 5B . 
     As indicated by the example illustrated in  FIG. 5B , the focus coordinate acquisition unit  111  displays an image (to be referred to as a reference image hereinafter)  503  represented by reference image data on a UI  501  and causes the user to input positions (to be referred to as focus positions hereinafter) where focus is to be obtained. The user designates the focus positions by using a pointing device or touching the screen of the UI  501 . 
     When the focus positions are designated, the focus coordinate acquisition unit  111  displays the coordinates of the positions in the reference image  503  on a display unit  502  of the UI  501 . Referring to  FIGS. 5A and 5B , a cross “+” indicates a focus position designated by the user. The focus coordinates are expressed as coordinates Pn(xn, yn) (n=1, 2, . . . ) in a reference image  160 , as shown in  FIG. 5A . 
     Distance Estimation Unit 
     The distance estimation unit  112  estimates the distance of an object included in a captured scene based on a plurality of image data from different viewpoints. The stereo method is used to estimate the distance. A multi-baseline stereo method or the like is also usable. The process (step S 103 ) of the distance estimation unit  112  will be described with reference to the flowchart of  FIG. 6 . 
     The distance estimation unit  112  receives two image data from different viewpoints out of captured data (step S 301 ). For example, assume that image data captured by the image capturing unit  135  arranged at the center and image data captured by the image capturing unit  134  adjacent to the image capturing unit  135  in the horizontal direction are used. An image represented by the image data captured by the image capturing unit  135  will be referred to as a “reference image”, and an image represented by the image data captured by the image capturing unit  134  will be referred to as a “non-reference image” hereinafter. Note that the image capturing unit of the non-reference image is not limited to the image capturing unit  134 , and image data captured by any other image capturing unit may be used as the non-reference image. 
     Next, the distance estimation unit  112  sets the pixel of interest at, for example, the origin of the reference image (step S 302 ) and sets a region of interest including the pixel of interest and peripheral pixels (step S 303 ). Block matching is performed between the region of interest and the non-reference image, thereby deciding a pixel (corresponding pixel) of the non-reference image corresponding to the pixel of interest (step S 304 ). 
     The distance estimation unit  112  then calculates a distance p corresponding to the pixel of interest based on the angle of view of the image capturing unit  134  that has captured the non-reference image and the relative position with respect to the image capturing unit  135 , which are represented by the image capturing apparatus information, and the pixel of interest and the corresponding pixel (step S 305 ). 
     Calculation of the distance p will be explained with reference to  FIG. 7 . An angle α is calculated from the horizontal angle of view of the image capturing unit  135 , the capturing position of the reference image, and the coordinates of the pixel of interest. An angle β is calculated from the horizontal angle of view of the image capturing unit  134 , the capturing position of the non-reference image, and the coordinates of the corresponding pixel. A distance s is the distance between the image capturing unit  135  and the image capturing unit  134  and is calculated from the capturing positions of the reference image and the non-reference image. The distance p from an object  141  is calculated by
 
 p ={sin α·sin β/sin(π−α−β)} s   (1)
 
     The distance estimation unit  112  determines whether the distances p corresponding to all pixels of the reference image are calculated (step S 306 ). If an uncalculated pixel remains, the pixel of interest is moved (step S 307 ), and the process returns to step S 303 . If the distances p corresponding to all pixels of the reference image are calculated, a depth image is generated by associating the distances p corresponding to all pixels with the pixel values (step S 308 ). 
     Virtual Focus Surface Setting Unit 
     The virtual focus surface setting unit  113  calculates the focus point pn, that is, focus coordinates in a three-dimensional space from the received focus coordinates Pn. Next, the virtual focus surface setting unit  113  sets the order of the equation of the surface (to be referred to as a focus surface hereinafter) where focus is obtained in accordance with the calculated object position and outputs the order as the curved surface parameter S. In other words, the virtual focus surface setting unit  113  sets the shape of a focus surface according to the number and positions of the focus coordinates Pn. 
     The process (step S 104 ) of the virtual focus surface setting unit  113  will be described with reference to the flowchart of  FIG. 8 . The virtual focus surface setting unit  113  receives the depth image and the focus coordinates Pn (step S 131 ) and calculates the focus points pn corresponding to the focus coordinates Pn from the depth image (step S 132 ). 
     The relationship between the focus points pn and the focus coordinates Pn will be described with reference to  FIG. 9 . The focus coordinates Pn are expressed as the xy coordinates of the reference image  160 . As for the focus point pn, a focus point in the actual three-dimensional space corresponding to the focus coordinates Pn is expressed as xyz coordinates. Let Dmap be the two-dimensional array of the depth image, and Pn=(xn, yn). The distance of the focus coordinates Pn from the image capturing apparatus  140  is represented by Dmap(Pn). Hence, the focus point is represented by pn(x, y, z)=(xn, yn, Dmap(xn, yn)). 
     Next, the virtual focus surface setting unit  113  compares the number N of focus points pn with a predetermined number Gth (step S 133 ). If N≦Gth, (N−1) is set to the curved surface parameter S (step S 134 ). If N&gt;Gth, the predetermined number Gth is set to the curved surface parameter S (step S 135 ). Then, the curved surface parameter S and the focus points pn are output (step S 136 ). 
     For example, when three focus points pn are present, the curved surface parameter S=2. The virtual focus surface generation unit  114  generates a focus surface from a quadratic curve. That is, a curve having a lower order can uniquely be decided by controlling the curved surface parameter S. 
     If the number of focus points pn is enormous, the order also becomes large and the focus surface becomes complex. As a result, the calculation cost for generating a focus surface may increase, or it may be impossible to generate a focus surface. Hence, if the number of focus points pn exceeds the predetermined number Gth, the order is limited to Gth. 
     When Gth=3, the curved surface parameter S=3, and the order of the focus surface is 3 even if the number of focus points pn exceeds 3. Even if the number N of focus points pn is as large as, for example, 10 or 20, the focus surface can be generated more simply by approximating it by a cubic curve. If the order is less than the number N of focus points pn, some focus points pn may fall outside the focus surface. Gth is appropriately decided in consideration of the number N of focus points pn and the calculation cost or made adjustable as needed. 
     Virtual Focus Surface Generation Unit 
     Virtual focus surface generation will be described with reference to  FIGS. 10A to 10E . In  FIGS. 10A to 10E , the origin corresponds to the position of the image capturing apparatus  140 .  FIGS. 10A to 10E  show the three-dimensional space including objects observed from the y-axis direction. Hence, the z-axis in  FIGS. 10A to 10E  corresponds to the depth (capturing distance). Gth=3 in the following explanation. 
       FIG. 10A  shows a case in which one focus point pn exists. The curved surface parameter S=0, and a virtual focus surface corresponding to a zero order curve is generated. That is, when N=1, a virtual focus surface perpendicular to the z-axis is generated. 
       FIG. 10B  shows a case in which two focus points pn exist. The curved surface parameter S=1, and a virtual focus surface corresponding to a linear curve passing through focus points p 1  and p 2  is generated. 
       FIG. 10C  shows a case in which three focus points pn exist. The curved surface parameter S=2, and a virtual focus surface corresponding to a quadratic curve passing through focus points p 1 , p 2 , and p 3  is generated. 
       FIG. 10D  shows a case in which four focus points pn exist (N&gt;Gth). The curved surface parameter S=Gth=3, and a virtual focus surface corresponding to a cubic curve passing through focus points p 1 , p 2 , p 3 , and p 4  is generated. 
       FIG. 10E  shows a case in which five focus points pn exist (N&gt;Gth). The curved surface parameter S=Gth=3, and a virtual focus surface corresponding to a cubic curve is generated. In this case, although not all the focus points p 1  to p 5  are necessarily placed on the virtual focus surface, a simple smooth curved surface is generated as the virtual focus surface. 
     The process (step S 105 ) of the virtual focus surface generation unit  114  will be described with reference to the flowchart of  FIG. 11 . The virtual focus surface generation unit  114  receives the curved surface parameter S and the focus points pn (step S 141 ) and selects a polynomial corresponding to the curved surface parameter S (step S 142 ). When Gth=3, the polynomials are given by
 
if ( S== 0) z=a;  
 
if ( S== 1) z=ax+b;  
 
if ( S== 2) z=ax   2   +bx+c;  
 
if ( S== 3) z=ax   3   +bx   2   +cx+d;   (2)
 
     Next, the virtual focus surface generation unit  114  calculates the coefficients of the polynomial from the focus points pn (step S 143 ). The coefficients of the polynomial take a form (a, b), (a, b, c), or (a, b, c, d) in accordance with equation (2) selected. The focus point pn is expressed as xyz coordinate values. The polynomial is solved by the xz coordinates, thereby calculating the curve on the xz plane. The coefficient calculation method is known. For example, the coefficients can be calculated either by obtaining an approximate curve using the least squares method or by a matrix operation. 
     The virtual focus surface generation unit  114  then generates a virtual focus surface image from the curve represented by the polynomial using the calculated coefficients (step S 144 ). That is, a two-dimensional image representing the z-coordinate values of the pixels of the reference image  160  on the virtual focus surface is generated from the curve represented by the polynomial and an angle θw of view contained in the image capturing apparatus information. The two-dimensional image is output as the virtual focus surface image (step S 145 ). 
     The virtual focus surface (curved surface) and the virtual focus surface image will be explained with reference to  FIGS. 12A to 12C . As shown in  FIG. 12A , a curve  1201  passing through the three focus points pn is obtained and expanded in the y-axis direction so as to generate a virtual focus surface  1202 . As shown in  FIG. 12B , a grayscale value is set in correspondence with each depth of the virtual focus surface  1202 , thereby generating a virtual focus surface image  1203 . For example, a region where the density of the virtual focus surface image  1203  is high represents a region having a deep (far) focus surface, and a region where the density is low represents a region having a shallow (close) focus surface. 
     The curve  1201  may be calculated by a spline function passing through the xz coordinates of the focus points pn, which is given by
 
 z=Σ   i   C   i   B   i ( x )  (3)
 
where
 
     x is the input value, 
     z is the output value, 
     B i (x) is the basis function, 
     C i  is the spline coefficient, and 
     i is the basis function number. 
     However, the focus points are not always placed on the virtual focus surface. 
     To obtain an interpolation function passing through the desired focus points pn using the spline function, the spline coefficients are calculated from the given focus points pn. To calculate the spline coefficients, the vector of the focus point pn is set to X={x 0 , . . . , x N } (x 0 ≦ . . . ≦x N ). The vector of the corresponding output value is set to Z={z 0 , . . . , z N }. 
     Next, nodes are set for the focus points pn. The nodes can arbitrarily be set as long as they satisfy Schoenberg-Whitney conditions. For example, the midpoints of the focus points pn or the same points as the focus points pn may be set as the nodes. The basis function is calculated from the set nodes. To calculate the basis function, the general DeBoor-Cox algorithm is used. 
     Next, from the focus points pn and the basis function, a coefficient matrix M given by 
                   M   =     [             B   0     ⁡     (     x   0     )               B   1     ⁡     (     x   0     )           …           B   N     ⁡     (     x   0     )                   B   0     ⁡     (     x   1     )               B   1     ⁡     (     x   1     )           …           B   N     ⁡     (     x   1     )               ⋮       ⋮                   ⋮           ⋮       ⋮                   ⋮               B   0     ⁡     (     x   N     )               B   1     ⁡     (     x   N     )           …           B   N     ⁡     (     x   N     )             ]             (   4   )               
is generated.
 
     In addition, letting C be the vector of the spline coefficients, the spline function is represented by a determinant given by
 
 Z=MC   (5)
 
     When the determinant (5) is solved for the spline coefficients using LU decomposition or the like, the spline coefficient vector C can be obtained. 
     An example in which B-spline is used as the basis function has been described above. However, any other piecewise function such as a non uniform rational B-spline or a Bezier can be used as the basis function. 
     In the above-described example, the virtual focus surface  1202  have the same y-coordinate value. However, a curved virtual focus surface freely deformed even in the y-axis direction can be generated. In this case, the coefficients of a three-dimensional spline function are calculated. The three-dimensional spline function is represented by expanding the above-described linear spline function as
 
 X=Σ   i Σ j Σ k α ijk   B   i ( y ) B   j ( z ) B   k   z ( x )
 
 Y=Σ   i Σ j Σ k β ijk   B   i ( y ) B   j ( z ) B   k   z ( x )
 
 Z=Σ   i Σ j Σ k γ ijk   B   i ( y ) B   j ( z ) B   k   z ( x )  (6)
 
where
 
     the calculation range of Σ i  is i=0 to N, 
     the calculation range of Σ j  is j=0 to N, and 
     the calculation range of Σ k (is k=0 to N. 
     Like the linear spline function, the spline coefficients of each dimension are calculated.  FIG. 12C  shows an example of a curved virtual focus surface  1204  when there are four focus points pn. 
     Image Composition Unit 
     The image composition unit  115  sets a weighting coefficient for each image data. The image represented by each image data is shifted based on the virtual focus image and the image capturing apparatus information. In addition, the image data multiplied by the weighting coefficients are added to generate synthetic image data. 
     The process (step S 106 ) of the image composition unit  115  will be described with reference to the flowchart of  FIG. 13 . The image composition unit  115  receives the captured data, the image capturing apparatus information, and the virtual focus surface image (step S 151 ) and sets the weighting coefficients of the image data to be used to composite the images (step S 152 ). 
     The weighting coefficients will be described with reference to  FIGS. 14A and 14B .  FIG. 14A  shows positions Pcm (cm=0 to 8) of the image capturing units  131  to  139 . Let Im be an image represented by image data corresponding to the position Pcm, and A(m) be the weighting coefficient corresponding to the image Im. In this example, weighting coefficients according to a Gaussian function are set based on the weighting coefficient A( 4 ) of the image capturing unit  135  arranged at a center P 4  and normalized so that the sum of the weighting coefficients becomes 1 (see  FIG. 14B ). This allows to smoothly increase the defocus amount in the region outside the virtual focus surface. In other words, the images captured by the image capturing units  131  to  139  are composited while placing emphasis on the image captured by the image capturing unit  135 . 
     Next, the image composition unit  115  calculates the shift amount of each image based on the virtual focus surface image and the image capturing apparatus information (step S 153 ). Letting d be the capturing distance corresponding to the value of the pixel of interest of the virtual focus surface image, a horizontal shift amount Δi(m, d) and a vertical shift amount Δj(m, d) of the image Im are calculated by
 
Δ i ( m,d )=( s   m   −s ′) W/{ 2 d ·tan(θ w/ 2)}
 
Δ j ( m,d )=( t   m   −t ′) H/{ 2 d ·tan(θ h/ 2)}  (7)
 
where d is the capturing distance represented by the virtual focus surface image,
 
     W is the image size in the horizontal direction, 
     H is the image size in the vertical direction, 
     θw is the horizontal view angle of the image capturing unit, 
     θh is the vertical view angle of the image capturing unit, 
     (s m , t m ) are the coordinates of Pcm on the xy plane, and 
     (s′, t′) are the coordinates of the position Pc 4  of the image capturing unit  135  on the xy plane. 
     The image composition unit  115  then shifts the pixels of each image data by the shift amounts, multiplies each image data by the corresponding weighting coefficient, and calculates the sum of the shifted and weighted pixel values (step S 154 ), that is, performs weighted addition represented by
 
 H ( i,j )=Σ m   A ( m ) Im ( I+Δi ( m,d ), j+Δj ( m,d ))  (8)
 
where H is synthetic image data. When the calculation has ended for all pixels, the image composition unit  115  outputs the synthetic image data (step S 155 ).
 
     In the above description, the weighting coefficient A(m) is a fixed value. However, the weighting coefficient A(m) may be input via the general-purpose I/F  106  as a process parameter. 
     As described above, when generating an image from a plurality of image data from different viewpoints, it is possible to easily generate an image having a shallow depth of field in which focus is obtained on an object located on a focus surface (or curved focus surface) that is not always a plane, and the blur smoothly increases in regions outside the focus surface. 
     Second Embodiment 
     Image processing according to the second embodiment of the present invention will be described next. Note that the same reference numerals as in the first embodiment denote the same parts in the second embodiment, and a detailed description thereof will be omitted. 
     In the first embodiment, an example has been described in which the shape of the virtual focus surface is decided from the number N of focus points pn. According to this method, when the number N increases, the shape of the virtual focus surface becomes complex, and the calculation cost drastically increases. To prevent this, in the second embodiment, an example will be described in which the shape of the virtual focus surface is decided from the number of groups into which focus points pn are grouped. 
     The functional arrangement of an image processing apparatus according to the second embodiment will be described with reference to the block diagram of  FIG. 15 . Note that the functional arrangement shown in  FIG. 15  is implemented by causing a CPU  101  to execute an image processing program. The functional arrangement of the second embodiment is different from that of the first embodiment in that a depth of field acquisition unit  201  is added. 
     The depth of field acquisition unit  201  acquires the depth of field at the time of image data capturing from the stop number represented by image capturing apparatus information and outputs information representing the depth of field. Note that the relationship between stop numbers and depths of field of an image capturing apparatus  140  is stored in a table stored in, for example, an HDD  105 . Alternatively, information representing the depth of field added to image capturing apparatus information may be acquired, or the user may input the depth of field. A virtual focus surface setting unit  113  receives focus coordinates, a depth image, and the information representing the depth of field and calculates a curved surface parameter and a focus point. 
     An example of image processing according to the second embodiment will be described with reference to the flowchart of  FIG. 16 . Note that the image processing shown in  FIG. 16  is processing executed by the CPU  101  when, for example, a user instruction is input via the UI. The processes of steps S 101  to S 103  and S 105  to S 107  are the same as in the first embodiment, and a detailed description thereof will be omitted. 
     After acquiring focus coordinates Pn (step S 102 ), the CPU  101  causes the depth of field acquisition unit  201  to acquire depth of field (step S 201 ). After generating a depth image (step S 103 ), the CPU  101  causes the virtual focus surface setting unit  113  to execute virtual focus surface setting processing and thus calculate a curved surface parameter S and the focus point pn (step S 202 ). 
     The process (step S 202 ) of the virtual focus surface setting unit  113  will be described with reference to the flowchart of  FIG. 17 . Note that the processes of steps S 132 , S 135 , and S 136  are the same as in the first embodiment, and a detailed description thereof will be omitted. 
     The virtual focus surface setting unit  113  receives the depth image, the focus coordinates Pn, and the information representing the depth of field (step S 211 ), calculates the focus points pn (step S 132 ), and groups the focus points pn on the xz plane based on the depth of field (step S 212 ). The virtual focus surface setting unit  113  compares the number N G  of groups with a predetermined number Gth (step S 213 ). If N G ≦Gth, (N G −1) is set to the curved surface parameter S (step S 214 ). If N G &gt;Gth, the predetermined number Gth is set to the curved surface parameter S (step S 135 ). 
     Objects included in the depth of field can be handled as one object. Hence, grouping objects corresponding to the focus points pn included in the depth of field makes it possible to obtain an image having focus on the focus points pn even using a surface (or curved surface) with a lower order. 
     Grouping of the focus points pn will be described with reference to  FIG. 18 .  FIG. 18  shows the three-dimensional space including objects observed from the y-axis direction, like  FIGS. 10A to 10E . A distance D between the focus points pn on the xz coordinates is compared with a depth of field (Dc-Df). The focus points pn whose distance D falls within the range of (Dc-Df) (depth of field) are grouped. 
     In the example shown in  FIG. 18 , focus points p 1  and p 2  are grouped because the distance between them is included within the range of (Dc-Df). Focus points p 3  and p 4  are not grouped because the distance between them is not included within the range of (Dc-Df). Hence, the number N G  of groups is 3. 
     The subsequent processing is executed while defining the average coordinates of the grouped focus points as the focus point of the group. For example, when the focus points p 1  and p 2  are grouped, a representative focal point Gi(x, y, z) of the group is calculated by
 
 Gi ( x )={ p 1( x )+ p 2( x )}/2
 
 Gi ( y )={ p 1( y )+ p 2( y )}/2
 
 Gi ( z )={ p 1( z )+ p 2( z )}/2  (9)
 
     Note that the representative focal point of a group including only one focus point equals that focus point, as a matter of course. 
     [Modification] 
     In the above-described examples, the user designates the focus coordinates Pn via the UI. However, an object or a human face in a captured image may be recognized as an object region, and the position of the object region may be used as the focus coordinates Pn. 
     That is, the focus coordinate acquisition unit  111  recognizes an object region from a reference image. Note that the object recognition is performed using an existing technique. For example, to detect an object such as a flower vase from a reference image, a method disclosed in Japanese Patent Laid-Open No. 2011-053959 is used. To recognize a human face, a method disclosed in Japanese Patent Laid-Open No. 2009-060379 is used. 
       FIG. 19  shows an example of an object region recognition result. The focus coordinate acquisition unit  111  extracts, for example, regions surrounded by broken lines in  FIG. 19  from the reference image  160  as object regions, and outputs the coordinates at the center of each object region as the focus coordinates Pn. 
     Other Embodiments 
     Aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment(s), and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment(s). For this purpose, the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (for example, computer-readable medium). 
     While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions. 
     This application claims the benefit of Japanese Patent Application No. 2011-122597, filed May 31, 2011, which is hereby incorporated by reference herein in its entirety.