Abstract:
The present invention aims to simplify stitching algorithm which generates horizontal panoramic image. The image pickup device of the present invention comprises a plurality of lenses and positioning means. Said positioning means positions each lens so that the FOV (Field Of View) intersection points of all lenses are aligned in vertical direction. Accordingly, the horizontal parallax does not exist in the image picked up by the camera system and the stitching point remains the same for the objects at different distances.

Description:
FIELD OF THE INVENTION  
       [0001]     The present invention relates generally to an image pickup device. More specifically, the present invention relates to an image pickup device of multiple lens camera system for generating panoramic image. The image pickup device can position a plurality of lenses in a multiple camera system so that a simple stitching algorithm is implemented in an ASIC (Application Specific Integrated Circuit) solution.  
       DESCRIPTION OF THE PRIOR ARTS  
       [0002]     The generation of a panoramic image usually requires taking pictures concurrently by a plurality of cameras and then composing an image by an image processor. On the other hand, a static panoramic image may be formed by using a single camera combined with a panning motor to shoot multiple times and then stitching the images captured each time. For example, Japan Patent No. 11-008845 and No. 11-018003 involve panning motors to capture wide angle images. However, the panning motor increases the cost and size of the camera system. Accordingly, it is desired to generate a panoramic image by a simpler mechanism and a simpler stitching algorithm.  
       SUMMARY OF THE INVENTION  
       [0003]     The image pickup device of the invention aligns the FOV (Field Of View) intersection points of all lenses to provide a system with fixed stitching points of the captured image so that simple stitching algorithm can be implemented in a low-cost ASIC solution to generate panoramic video.  
         [0004]     To achieve the above purpose, the present invention provides an image pickup device of multiple lens camera system, comprising: N lenses, wherein the horizontal field of view for each lens is HFOV i  (i=1, 2, . . . , N); positioning means, wherein said positioning means positions each lens on top of the other by rotation of  i degrees (0&lt; i &lt;HFOV i , i=1, 2, . . . , N−1) in horizontal direction, and said positioning means positions each lens so that the FOV intersection points of all lenses are aligned in vertical direction.  
         [0005]     According to an aspect of the present invention, the above-mentioned positioning means tilts each lens with an angle of φ i  degrees (0&lt;φ i &lt;VFOV i , i=1, 2, . . . , N) in vertical direction.  
         [0006]     According to another aspect of the present invention, the above-mentioned  1 = 2 = 3 = . . . = N− 1.  
         [0007]     According to yet another aspect of the present invention, the total field of view obtained by the above-mentioned N lenses is equal to  
           ∑     i   =   1     N     ⁢     HFOV   i       -       ∑     i   =   1     N     ⁢       (       HFOV     i   +   1       -         i       )     .           
 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0008]      FIG. 1  is an illustrative diagram of the N lenses of an image pickup device according to the present invention.  
         [0009]      FIG. 2  is an illustrative diagram of lens rotation and HFOV (horizontal field of view).  
         [0010]      FIG. 3  is a diagram showing the FOV intersection point of a single lens.  
         [0011]      FIG. 4  is a diagram showing the horizontal parallax caused by the misalignment of FOV intersection points.  
         [0012]      FIG. 5 ( a ) and  FIG. 5 ( b ) are diagrams showing overlapping portions of the images of near objects and far objects, respectively, in the case of misalignment.  
         [0013]      FIG. 6  is a diagram showing the case in which the FOV intersection points are aligned.  
         [0014]      FIG. 7  is a diagram showing the image shift without tilting the camera in vertical direction.  
         [0015]      FIG. 8  is a diagram showing the case in which the images are aligned by tilting the camera in vertical direction.  
         [0016]      FIG. 9  shows a block diagram of the multiple lens camera system according to the present invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0017]      FIG. 1  shows an image pickup device according to the present invention by the examples of 2, 3 and N lenses. This lens arrangement is achieved by positioning means according to the present invention. This positioning means can be a part of a video phone system which creates wide angle images beyond the angle limitation of a single lens. This multiple lens camera system together with a simple ASIC where a simple stitching algorithm is implemented are adapted to provide a low-cost, small-size and wide-angle camera system.  
         [0018]     The principle of the present invention is described with reference to  FIGS. 2-8  as follows.  
         [0019]     The image pickup device according to the present invention comprises N lenses and positioning means. Said positioning means positions each lens on top of the other by rotation of  i degrees (0&lt; i &lt;HFOV i , i=1, 2, . . . , N−1) in horizontal direction.  
         [0020]      FIG. 2  is an illustrative diagram of lens rotation and HFOV (horizontal field of view). Providing N lenses with horizontal view angle of HFOV i  for each lens (i=1, 2, 3, . . . , N) and lens rotation angle of θ i  (i=1, 2, . . . , N−1) in the camera system, the total HFOVt of the system is equal to  
           ∑     i   =   1     N     ⁢     HFOV   i       -       ∑     i   =   1     N     ⁢       (       HFOV     i   +   1       -     θ   i       )     .             
 In case the HFOV i  of each lens is equal to HFOV and all rotation angles θ i  are equal to θ, the total HFOVt of the system will be equal to HFOV*N−(HFOV−θ)*(N−1). For example, N=2, HFOV 1 =HFOV 2 =60, and θ 1 =30° result in a total HFOVt=90°; and N=11 (11 lenses in total), HFOV i =60° (i=1, 2, 3, . . . 11) and θ i =30° (i=1, 2, 3, . . . 10) result in a total HFOVt=360°. 
 
         [0021]     The importance of the invention is to capture images for a simple stitching algorithm which can be implemented in a low-cost ASIC for video stitching. The alignment of the FOV intersection point of each lens provides constant stitching point for the objects at different distance and the rotation angle between each lens is fixed for the camera system. Hence the stitching point can be calculated during camera calibration. It is not necessary for the ASIC to calculate the stitching point dynamically at every frame due to the distance change of the objects. Therefore the computation power for stitching can be much reduced and the ASIC cost can be saved.  
         [0022]     In the following description, the relation between the stitching point and the FOV intersection point alignment is explained.  
         [0023]      FIG. 3  shows the FOV intersection point of a single lens.  FIG. 4  shows the stitching problem caused by the misalignment of FOV intersection points. In the figure, stpn represents the stitching point of near objects; stpf represents the stitching point of far objects; Dn represents the distance between the FOV intersection point and near objects; Df represents the distance between the FOV intersection point and far objects; Dth represents the distance between the FOV intersection point and the FOV cross point; Wn represents viewable width of near objects; Wf represents viewable width of far objects; α represents the angle between overlapped boundary and the stitching point; and HFOV represents horizontal field of view. As shown in  FIG. 4 , in the case of misalignment, there is no image overlapping for the objects within the distance of Dth. Providing the definition of stitching point is center of the overlapped images, the stitching points shift when the distance between the object and the camera changes.  
         [0024]      FIG. 5 ( a ) and  FIG. 5 ( b ) show overlapping portions of the images of near objects and far objects, respectively, in the case of misalignment. Comparing these two figures, it can be seen that the overlapping portion (shadowed portion) of the images of near objects in  FIG. 5 ( a ) is obviously smaller than the overlapping portion (shadowed portion) of the images of far objects in  FIG. 5 ( b ).  
         [0025]     The stitching point change can be derived from the following equations:  
         [0026]     For near objects:  
       stpn   =       2   ⁢           ⁢   Dn   *   tan   ⁢           ⁢     (       HFOV   i     2     )       -       (     Dn   -   Dth     )     *   tan   ⁢           ⁢   α           
       Wn   =     2   ⁢   Dn   *   tan   ⁢           ⁢     (       HFOV   i     2     )           
 
         [0027]     The stitching point percentage of near objects within the image is:  
         stpn   Wn     =         2   ⁢   Dn   *   tan   ⁢           ⁢     (       HFOV   i     2     )       -       (     Dn   -   Dth     )     *   tan   ⁢           ⁢   α         2   ⁢   Dn   *   tan   ⁢           ⁢     (       HFOV   i     2     )             
 
         [0028]     For far objects:  
       stpf   =       2   ⁢           ⁢   Df   *   tan   ⁢           ⁢     (       HFOV   i     2     )       -       (     Df   -   Dth     )     *   tan   ⁢           ⁢   α           
       Wf   =     2   ⁢   Df   *   tan   ⁢           ⁢     (       HFOV   i     2     )           
 
         [0029]     The stitching point percentage of far objects within the image is:  
         stpf   Wf     =         2   ⁢           ⁢   Df   *   tan   ⁢           ⁢     (       HFOV   i     2     )       -       (     Df   -   Dth     )     *   tan   ⁢           ⁢   α         2   ⁢   Df   *   tan   ⁢           ⁢     (       HFOV   i     2     )             
 
         [0030]     Therefore,  
         stpn   Wn     ≠       stpf   Wf     ⁢           ⁢     (       since   ⁢           ⁢   Dth     ≠   0     )           
 
         [0031]      FIG. 6  shows the case in which the FOV intersection points are aligned. In this case, the stitching points remain the same regardless of the object distances. This can be explained by the following equations:  
         [0032]     For near objects:  
       stpn   =       2   ⁢           ⁢   Dn   *   tan   ⁢           ⁢     (       HFOV   i     2     )       -     Dn   *   tan   ⁢           ⁢   α           
       Wn   =     2   ⁢   Dn   *   tan   ⁢           ⁢     (       HFOV   i     2     )           
 
         [0033]     The stitching point percentage of near objects within the image is:  
         stpn   Wn     =           2   ⁢   Dn   *   tan   ⁢           ⁢     (       HFOV   i     2     )       -     Dn   *   tan   ⁢           ⁢   α         2   ⁢   Dn   *   tan   ⁢           ⁢     (       HFOV   i     2     )         =         2   ⁢           ⁢   tan   ⁢           ⁢     (       HFOV   i     2     )       -     tan   ⁢           ⁢   α         2   ⁢           ⁢   tan   ⁢           ⁢     (       HFOV   i     2     )               
 
         [0034]     For far objects:  
       stpf   =       2   ⁢           ⁢   Df   *   tan   ⁢           ⁢     (       HFOV   i     2     )       -     Df   *   tan   ⁢           ⁢   α           
       Wf   =     2   ⁢   Df   *   tan   ⁢           ⁢     (       HFOV   i     2     )           
 
         [0035]     The stitching point percentage of far objects within the image is:  
         stpf   Wf     =           2   ⁢           ⁢   Df   *     tan   ⁡     (       HFOV   i     2     )         -     Df   *   tan   ⁢           ⁢   α         2   ⁢   Df   *     tan   ⁡     (       HFOV   i     2     )           =         2   ⁢           ⁢     tan   ⁡     (       HFOV   i     2     )         -     tan   ⁢           ⁢   α         2   ⁢           ⁢     tan   ⁡     (       HFOV   i     2     )                 
 
         [0036]     Therefore,  
         stpn   Wn     =     stpf   Wf         
 
         [0037]     Besides, the images captured by each lens are shifted due to the vertical displacement of FOV  FIG. 7  explains the image non-coinciding caused by the FOV displacement. The non-coinciding portions have to be cropped in the final panoramic image. The larger the N is, the more portions are cropped. To solve this problem, the present invention provides positioning means for tilting each lens by φ i  degrees (0&lt;φ i &lt;VFOV i , i=1, 2, . . . , N) in vertical direction.  FIG. 8  explains the result obtained by tilting each lens in vertical direction. It should be noted that the FOV intersection points are always aligned while tilting the lenses.  
         [0038]     Accordingly, the image pickup device of the present invention is able to provide the images with constant stitching points, thereby simplifying the complexity of the stitching algorithm.  
         [0039]     In the following, an embodiment of the multiple lens camera system according to the present invention is described with reference to  FIG. 9 . For conciseness, the following description is focused on the lens part and the related image processing procedure with the detailed description of other parts of the camera system omitted.  
         [0040]     As shown in  FIG. 9 , a lens part  110  includes three lenses  110 A,  110 B and  110 C, wherein the lens  110 B is arranged on top of the lens  110 A with a counterclockwise rotation of θ degrees (not shown in the figure) in horizontal direction; and the lens  110 C is arranged on top of the lens  110 B with a further counterclockwise rotation of θ degrees in horizontal direction. The image signals captured by the lenses  110 A,  110 B and  110 C are passed through FFC (Flexible Flat Cable)  120 A,  120 B and  120 C, respectively, to an image processing logic block  130  for further processing. The image processing logic block  130  includes a multi-lens ISP (image signal processor)  131 , stitching logic  132 , an ISP  133 , a video encoder  134 , a MPEG encoder  135  and a network interface  136 .  
         [0041]     At first, the multi-lens ISP  131  performs preliminary processing of the image signals passed from the lenses  110 A,  110 B and  110 C so that the differences between the images captured by respective lenses are reduced. The image signals after the preliminary processing are respectively passed to the stitching logic  132 . The stitching logic  132  performs transformation and positional calculation on the image signals so that the images are put seamlessly together as one single image. Said one single image is then passed to the ISP  133  for traditional image processing. At this point, the processed image can be encoded by the video encoder  134  and then displayed on any display device. Alternatively, the processed image can also be compressed for storing in any storage device. Further, the compressed image data can be passed through the network interface  136  to the Internet.  
         [0000]     Effects of the Invention  
         [0042]     The stitching algorithm is the part which consumes most computational power when generating a panoramic image. For high frame rate video (e.g. 30 fps), a low-cost ASIC solution is not powerful enough to achieve the performance of updating stitching point for every 1/30 second. The present invention discloses a simple and feasible mechanism for positioning multiple lenses to capture images with constant stitching points, and thus provides a low-cost, small-size and wide-angle camera system.