Patent Publication Number: US-2020288059-A1

Title: Image processor, image processing method and program, and imaging system

Description:
CROSS REFERENCE TO RELATED APPLICATION 
     The present application is based on and claims priority to U.S. patent application Ser. No. 14/022,987, filed on Sep. 10, 2013, which claims priority to Japanese Patent Application No. 2012-199320, filed on Sep. 11, 2012 and No. 2013-124397, filed on Jun. 13, 2013, the disclosure of which is hereby incorporated by reference in its entirety. 
    
    
     BACKGROUND Of THE INVENTION 
     Field of the Invention 
     The present invention relates to an image processor, image processing method and program, and an imaging system for connecting input images formed by a lens system. 
     Description of the Related Art 
     There is a known omnidirectional imaging system which includes multiple wide-angle lenses such as fish eye lens or super wide-angle lens to capture an image in omnidirections at once. It is configured to project images from the lenses onto a sensor surface and combine the images through image processing to thereby generate an omnidirectional image. For example, by use of two wide-angle lenses with angle of view of over 180 degrees, omnidirectional images can be generated. 
     In the image processing a partial image captured by each lens system is subjected to distortion correction and projection conversion on the basis of a certain projection model with a distortion from an ideal model taken into account. Then, the partial images are connected on the basis of an overlapping portion of the partial images to form a single omnidirectional image. The positions at which subject images overlap in the overlapping portion are detected by pattern matching or the like. 
     However, partial images with a large amount of distortion, for example, ones captured with the fisheye lens, contain connecting areas having different kinds or amounts of distortion even when the same subject is captured. Accordingly, it is very difficult to accurately detect the overlapping positions of the images by pattern matching. Thus, partial images cannot be connected properly and a high-quality omnidirectional image cannot be generated accordingly. 
     There are various known techniques to combine partial images captured with multiple cameras. For example, Japanese Patent Application Publication No. 2010-130628 (Reference 1) discloses an imaging device comprising partial cameras having an overlapping photographic area and capturing a partial area of a photographic area of a subject and a reference camera having a photographic area including a part of an image captured by each partial camera. It corrects a distortion of a captured image of each camera using a camera parameter, detects an image area in which the corrected images of the partial cameras and the corrected image of the reference camera coincide with each other, calculates a relative position, and connects the images. 
     Further, Japanese Patent Application Publication No. 2009-104323 (Reference 2) discloses a camera system which uses multiple cameras arranged to have overlapping photo graphic areas and generates a high-precision mapping table not to cause a displacement in connecting points according to an actually captured image without the estimation of set positions of the cameras. Japanese Patent Application No. 2013-81479 (Reference 3) discloses an image processor which converts only X-coordinates of a fisheye image of a vehicle on the road captured with an in-vehicle camera, to generate a virtual view image with no vanishing points. It intends to convert fisheye images into images such that parking lot lines on a road surface appear to be approximately linear and parallel. 
     However, Reference 1 relates to connecting images represented in plane coordinates and cannot detect connecting positions accurately when applied to an imaging device using a lens with a large distortion such as a fisheye lens. Also, Reference 2 teaches the generation of mapping tables by use of an existing target board but cannot align the positions of images accurately. Reference 3 teaches the correction of fisheye images but does not concern connecting a plurality of images. 
     SUMMARY OF THE INVENTION 
     The present invention aims to provide an image processor, image processing method and program, and an imaging system which can accurately connect captured images even with use of a lens system having a large amount of distortion. 
     According to one aspect of the present invention, an image processor comprises a first converter to convert input images into images in a different coordinate system from that of the input images according to first conversion data based on a projection model, a position detector to detect a connecting position of the images converted by the converter, a corrector to correct the first conversion data on the basis of a result of the detection by the position detector, and a data generator to generate second conversion data for image synthesis from the conversion data corrected by the corrector on the basis of coordinate conversion, the second conversion data defining the conversion of the input images. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Features, embodiments, and advantages of the present invention will become apparent from the following detailed description with reference to the accompanying drawings: 
         FIG. 1  is a cross section view of an omnidirectional imaging system according to a first embodiment of the present invention; 
         FIG. 2  shows the hardware configuration of the omnidirectional imaging system in  FIG. 1 ; 
         FIG. 3  shows a flow of the entire image processing of the omnidirectional imaging system in  FIG. 1 ; 
         FIG. 4  is a function block diagram for omnidirectional image synthesis in the omnidirectional imaging system; 
         FIG. 5  is a flowchart for the image synthesis of an omnidirectional image executed by the omnidirectional imaging system; 
         FIGS. 6A, 6B  show the projection of an omnidirectional imaging system using a fisheye lens; 
         FIGS. 7A, 7B  show the data structure of image data in an omnidirectional image format according to the first embodiment; 
         FIGS. 8A, 8B  show conversion data to which a first distortion corrector for position detection and a second distortion corrector for image synthesis refer; 
         FIG. 9  shows mapping of two partial images captured by two fisheye lenses in a spherical coordinate system for position detection; 
         FIG. 10  is a flowchart for connecting position detection executed by the omnidirectional imaging system; 
         FIGS. 11A, 11B  show the connecting position detection in the first embodiment; 
         FIG. 12  shows the data structure of data generated by the position detector; 
         FIG. 13  is a flowchart for generating an image-synthesis conversion table by the omnidirectional imaging system according to the first embodiment; 
         FIG. 14  shows mapping of two partial images captured by two fisheye lenses in a spherical coordinate system for image synthesis; 
         FIG. 15  schematically shows the structure of an omnidirectional imaging system according to a second embodiment; and 
         FIG. 16  is a flowchart for omnidirectional image synthesis by the omnidirectional imaging system according to the second embodiment. 
     
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
     Hereinafter, embodiments of an image processor and an imaging system will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. 
     Referring to  FIGS. 1 to 3 , the overall configuration of an omnidirectional imaging system  10  is described. The omnidirectional imaging system  10  comprises two fisheye lenses to capture two partial images and an image processing function to perform distortion correction and projection conversion to the partial images, connect the partial images and generate an omnidirectional image, by way of example.  FIG. 1  is a cross section view of the omnidirectional imaging system  10  (hereinafter, simply imaging system). It comprises a camera unit  12 , a housing  14  accommodating the camera unit  12  and elements as controller, batteries, and a shutter button  18  provided on the housing  14 . The camera unit  12  in  FIG. 1  comprises two lens systems  20 A,  20 B and two solid-state image sensors  22 A,  22 B as CCD (charge coupled device) sensor or CMOS (complementary metal oxide semiconductor). Herein, each of the pairs of the lens systems  20  and solid-state image sensors  22  are referred to as imaging unit. The lens systems  20 A,  20 B are each comprised of 6 groups of 7 lenses as a fisheye lens, for instance. In the present embodiment the fisheye lens has total angle of view of 180 degrees (360 degrees/n, n=2) or more, preferably 185 degrees or more, more preferably 190 degrees or more. 
     The optical elements as lenses, prisms, filters, aperture stops of the lens systems  20 A,  20 B are positioned relative to the solid-state image sensors  22 A,  22 B so that the optical axes of the optical elements are orthogonal to the centers of the light receiving areas of the corresponding solid-state image sensors  22  as well as the light receiving areas become the imaging planes of the corresponding fisheye lenses. The solid-state image sensors  22  are area image sensors on which photodiodes are two-dimensionally arranged, to convert light gathered by the lens systems  20  to image signals. 
     In the present embodiment the lens systems  20 A,  20 B are the same and disposed opposite to each other so that their optical axes coincide. The solid-state image sensors  22 A,  22 B convert light distribution to image signals and output them to a not-shown image processor on the controller. The image processor combines partial images from the solid-state image sensors  22 A,  22 B to generate an image with solid angle of 4π in radian or an omnidirectional image. The omnidirectional image is captured in all the directions which can be seen from a shooting point. Instead of the omnidirectional image, a panorama image which is captured in a 360-degree range only on a horizontal plane can be generated. 
     To form an omnidirectional image with use of the fisheye lenses with total angle of view of more than 180 degrees, an overlapping portion of the captured images by the imaging units is used as reference data representing the same image and for connecting images. Generated omnidirectional images are output to, for instance, a display provided in or connected to the camera unit  12 , a printer or an external storage medium such as SD card®, compact flash®. 
       FIG. 2  shows the structure of hardware of the imaging system  10  according to the present embodiment. The imaging system  10  comprises a digital still camera processor  100  (hereinafter, simply processor), a lens barrel unit  102 , and various elements connected with the processor  100 . The lens barrel unit  102  includes the two pairs of leas systems  20 A,  20 B and solid-state image sensors  22 A,  22 B. The solid-state image sensors  22 A,  22 B are controlled by a command from a CPU  130  of the processor  100 . 
     The processor  100  comprises ISPs (image signal processors)  108 A,  108 B, a DMAC (direct memory access controller)  110 , an arbiter (ARBMEMC)  112  for memory access, a MEMC (memory controller)  114  for memory access, and a distortion correction and image synthesis black  118 . The ISPs  108 A,  108 B set white balance and gamma balance of image data signal processed by the solid-state image sensors  22 A,  22 B. The MEMC  114  is connected to an SDRAM  116  which temporarily stores data used in the processing of the ISP  108 A,  108 B and distortion correction and image synthesis block  118 . The distortion correction and image synthesis block  118  performs distortion correction and vertical correction on the two partial images from the two imaging units on the basis of information from a triaxial acceleration sensor  120  and synthesizes them. 
     The processor  100  further comprises a DMAC  122 , an image processing block  124 , a CPU  130 , an image data transferrer  126 , an SDRAMC  128 , a memory card control block  140 , a USB block  146 , a peripheral block  150 , an audio unit  152 , a serial block  158 , an LCD (Liquid Crystal Display) driver  162 , and a bridge  168 . 
     The CPU  130  controls the operations of the elements of the imaging system  10 . The image processing block  124  performs various kinds of image processing on image data together with a resize block  132 , a JPEG block  134 , an H. 264 block  136 . The resize block  132  enlarges or shrinks the size of image data by interpolation. The JPEG block  134  is a codec block to compress and decompress image data in JPEG. The H. 264 block  136  is a codec block to compress and decompress video data in H.264. The image data transferrer  126  transfers the images processed by the image processing block  124 . The SDRAMC  128  controls the SDRAM  138  connected to the processor  100  and temporarily storing image data during image processing by the processor  100 . 
     The memory card control block  140  controls data read and write to a memory card and a flash ROM  144  inserted to a memory card throttle  142  in which a memory card is detachably inserted. The USB block  146  controls USB communication with an external device such as personal computer connected via a USB connector  148 . The peripheral block  150  is connected to a power switch  166 . The audio unit  152  is connected to a microphone  156  for receiving an audio signal from a user and a speaker  154  for outputting the audio signal, to control audio input and output. The serial block  158  controls serial communication with the external device and is connected to a wireless NIC (network interface card)  160 . The LCD driver  162  is a drive circuit for the LCD  164  and converts the image data to signals for displaying various kinds of information on an LCD  164 . 
     The flash ROM  144  contains a control program written in readable codes by the CPU  130  and various kinds of parameters. Upon power-on of the power switch  166 , the control program is loaded onto a main memory. The CPU  130  controls the operations of the units and elements of the image processor in compliance with the control program on the main memory, and temporarily stores necessary control data in the SDRAM  138  and a not-shown local SRAM. 
       FIG. 3  shows a flow of the entire image processing of the imaging system  10  according to the present embodiment. In step S 101 A,  101 B the solid-state image sensors  22 A,  22 B capture images. In steps S 102 A,  102 B the ISPs  108  perform optical black correction, defective pixel correction, linear correction, shading, and area division and averaging onto Bayer RAW images and in steps S 103 A,  103 B the images are stored in the memory. In steps S 104 A,  104 B the ISPs  108  further perform white balance, gamma correction. Bayer interpolation, YUV conversion, edge enhancement and color correction to the images, and the images are stored in the memory in steps S 105 A,  105 B. 
     Upon completion of the above processing to the images captured on the solid-state image sensors  22 A,  22 B, in step S 106  each partial image is subjected to distortion correction and image synthesis. In step S 107  a generated omnidirectional image is added with a tag properly and stored in a file in the internal memory or an external storage. Alternatively, inclination correction can be additionally performed on the basis of the information from the triaxial acceleration sensor  120  or a stored image file can be subjected to compression when appropriate. 
     By use of a fisheye lens with a wide angle of view, an overlapping portion of partial images as a reference for image connection includes a large amount of distortion. Due to the distortion inherent to the fisheye lens, partial images may not be accurately connected in the distortion correction and image synthesis in step S 106 . In view of this, the imaging system  10  uses different parameters for image synthesis and connecting position detection before image synthesis, for the purpose of properly synthesizing partial images captured by the fisheye lenses with a larger amount of distortion than a general lens. 
     In the following the omnidirectional image synthesis function of the imaging system  10  will be described in detail, referring to  FIG. 4  to  FIG. 14 .  FIG. 4  shows a function block  200  for the omnidirectional image synthesis of the imaging system  10 . The distortion correction and image synthesis block  118  in  FIG. 4  comprises a distortion corrector for position detection  202 , a position detector  204 , a table corrector  206 , a table generator  208 , a distortion corrector for image synthesis  210 , and an image synthesizer  212 . For the sake of simplicity, the distortion corrector for position detection and the distortion corrector for image synthesis will be referred to as first and second distortion correctors, respectively. 
     Two partial images are input from the solid-state image sensors  22 A,  22 B to the distortion correction and image synthesis block  118  via the ISPs  108 A,  108 B. Herein, the solid-state image sensors  22 A,  22 B are referred to as 0 th  and 1 st  image sensors and a partial image from the solid-state image sensors  22 A is referred to as 0th partial image while that from the solid-state image sensor  22 B is referred to as 1 st  partial image. The distortion correction and image synthesis block  118  is provided with a conversion table for position detection  220  which is prepared by a manufacturer in compliance with a certain projection model on the basis of design data about the lens systems. 
     The first distortion corrector  202  corrects distortion of the 0th and 1 st  partial images before connecting position detection, referring to the conversion table  220  to generate 0 th  and 1 st  corrected images. The 0 th  and 1 st  partial images are captured on the two-dimensional solid-state image sensors and image data represented in a plane coordinate system (x, y). Meanwhile, the 0 th  and 1 st  corrected images are image data in a different coordinate system from the partial images, more specifically, they are image data in an omnidirectional image format represented in a spherical coordinate system which is a polar coordinate system having a radius of 1 and two arguments θ, φ. 
       FIGS. 6A, 6B  show the projection of the imaging system incorporating a fisheye lens. In the present embodiment a single fisheye lens captures an image in directions in about a hemisphere from a photographic point, and generates an image with image height h corresponding to an incidence angle φ relative to the optical axis. The relation between the image height h and incidence angle φ is determined by a projection function according to a certain projection model. The projection function differs depending on the property of a fisheye lens and that of on equisolid angle projection type fisheye lens is expressed by the following equation (1): 
     
       
      
       h=f*φ 
      
     
     where f is focal length. 
     The projection model exemplifies a perspective projection (h=f*tan φ), a stereo projection (h=2f*tan(φ/2)), an equisolid angle projection (h=2f*sin(φ/2)), an orthographic projection (h=f*sin φ). In any of the projection models the image height h is determined according to the incidence angle φ and focal length f. In the present embodiment a circular fisheye lens having an image circle diameter smaller than an image diagonal line is used and an obtained partial image is a planar image including the entire image circle in which about the hemisphere of a photographic area is projected, as shown in  FIG. 6B . 
       FIGS. 7A, 7B  show the data structure of image data in omnidirectional image format according to the present embodiment. As shown in the drawings, the image data in omnidirectional image format is represented by the arrays of pixel values at coordinates of vertical angle φ corresponding to an angle relative to a certain axis and horizontal angle θ corresponding to a rotational angle around the certain axis. The horizontal angle θ is from 0 to 360 degrees or −180 to +180 degrees while the vertical angle φ is from 0 to 180 degrees or −90 to +90 degrees. Each coordinate value (θ, φ) is associated with each point on the spherical surface representing omnidirections from a photographic point and the omnidirections are mapped on an omnidirectional image. The plane coordinates of an image captured by a fisheye lens can be related to the spherical coordinates in omnidirectional image format by the projection function described in  FIGS. 6A, 6B . 
       FIGS. 8A, 8B  show conversion data to which the first and second distortion correctors  202 ,  210  refer. The conversion tables  220 ,  224  define projection from a partial image represented in a plane co-ordinate system to an image represented in a spherical coordinate system. The conversion tables  220 ,  224  contain, for each fisheye lens, for all the coordinate values (θ, φ) (θ=0 to 360 degrees, φ=0 to 180 degrees), information to associate the coordinate values of a corrected image with the coordinate values (x, y) of a partial image before correction to be mapped on the coordinate values (θ, φ). In  FIGS. 8A, 8B , for instance, the angle of one pixel is a 1/10 degree in both φ and θ directions. The conversion tables  220 ,  224  contain information indicating 3,600*1,800 relations for each fisheye lens. 
     The data in the conversion table for position detection  220  are calculated in advance by a manufacturer or else on the basis of lens design data and the lens projection shewn in  FIGS. 6A, 6B  with distortion from an ideal lens model due to radial distortion or eccentric distortion corrected. In contrast the conversion table for image synthesis  224  is generated from the conversion table  220  by a certain conversion. Herein, the conversion data is a table indicating a relation between the two coordinate values. Alternatively, it can be coefficient data of one or more functions to define projection from a partial image (x, y) expressed in a plane coordinate system to an image (θ, φ) expressed in a spherical coordinate system. 
     Referring back to  FIG. 4 , the first distortion corrector  202  converts 0 th  and 1 st  partial images into 0 th  and 1 st  corrected images with reference to the conversion table  220 . Specifically, it finds, for all the coordinate values (θ, φ) of the corrected images, the coordinate values (x, y) of a partial image before the conversion and the pixel values thereof at the found coordinates, referring to the conversion table  220 . Thereby, a corrected image is generated. 
       FIG. 9  shows the mapping of two partial images, 0 th  and 1 st , captured with the two fisheye lenses, 0 th  and 1 st , into data in a spherical coordinate system. Processed by the first distortion corrector  202 , the 0 th  and 1 st  partial images are laid out on the omnidirectional image format, as shown in  FIG. 9 . Generally, the 0 th  partial image by the 0 th  fisheye lens is mapped in the upper hemisphere while the 1 st  partial image by the 1 st  fisheye lens is mapped in the lower hemisphere. The 0 th  and 1 st  corrected images in the omnidirectional image format are spread beyond the respective hemispheres since the total angle of view of the fisheye lenses exceeds 180 degrees. As a result, when the two corrected images are superimposed, an overlapping photographic area therebetween will occur. 
     After the distortion correction by the first distortion corrector  202 , the position detector  204  detects a connecting, position of the overlapping areas. However, in the spherical coordinate system the closer the vertical angle φ to the pole, 0 or 180 degrees, the larger a difference between the number of pixels along the horizontal angle θ and an actual distance. At the vertical angle being 0 or 180 degrees, the distance in θ-direction becomes zero and all the pixels in this direction represent the same direction. Also, the amount of variation in the distance in θ-direction increases as the vertical angle φ approaches 0 or 180 degrees, and it is smallest at the vertical angle φ being 90 degrees. It signifies a change amount of distortion when an image is shifted in θ-direction. Thus, the closer to 0 or 180 degrees the vertical angle φ is, the lower the accuracy of connecting position detection is. 
     In view of the above, the conversion table  220  is created so that the optical axes of the two lens systems are projected on the two poles (φ=0, 180 degrees) of the spherical surface as well as an overlapping area between the images is projected near the equatorial line (φ=90 degrees±((total angle of view −180 degrees)/2)) on the spherical surface, as shown in  FIG. 9 . The axis defining the vertical and horizontal angles in the omnidirectional image format matches the optical axis of one of the images, 0 th  image in this example. Thereby, a connecting position is detected around the vertical angle of 90 degrees with a small distortion when the image is shifted in θ-direction, improving the accuracy at which connecting positions are detected. Thus, it is made possible to precisely detect connecting positions of images captured by a lens system having a large distortion. 
     Returning to  FIG. 4 , the position detector  204  detects a connecting position of the 0 th  and 1 st  corrected images converted by the first distortion corrector  202  to generate resultant detection data  222 . The table corrector  206  corrects the conversion data in the conversion table  220  on the basis of the detection data  222  and sends it to the table generator  208 . The table generator  208  generates the conversion table for image synthesis  224  from the corrected conversion data by rotational coordinate conversion. 
     The second distortion corrector  210  corrects distortion of the 0 th  and 1 st  partial images before image synthesis, referring to the conversion table  224  and generates 0 th  and 1 st  corrected images for image synthesis. The corrected images are represented in a spherical coordinate system as the corrected images for position detection but the definition of the coordinate axis is different because of the rotational coordinate conversion. The image synthesizer  212  synthesizes the 0 th  and 1 st  corrected images to generate a synthetic image in omnidirectional image format. The operations of the position detector  204 , table corrector  206 , table generator  208 , second distortion corrector  210 , and image synthesizer  212  will be described in detail later. 
     The function block  200  in  FIG. 4  can additionally include a display image generator  214 . The generated synthetic image in the omnidirectional image format cannot be displayed as it is on a planar display device as a display because the closer to the vertical angle 0 and 180 degrees, the larger the distortion of an image on the display. The display image generator  214  performs image processing to the omnidirectional image for display on a planar display device. For example, it can convert a synthetic image in the omnidirectional image format in a spherical coordinate system to one in a plane coordinate system having a specific direction and a specific angle of view, so as to project an image with a certain angle of view in a specific field of view designated by a user. 
     In the following the omnidirectional image synthesis according to the present embodiment is described with reference to  FIGS. 5, 10, and 13 .  FIG. 5  is a flowchart for the omnidirectional image synthesis executed by the imaging system  10 . The operation starts when the CPU  130  issues an instruction for shooting in response to press-down of the shutter button  18 , for example. 
     In step S 201  the first distortion corrector  202  performs distortion correction or the 0 th  and 1 st  partial images acquired by the two solid-state image sensors  22 A,  22 B, referring to the conversion table  220 , to acquire the 0 th  and 1 st  corrected images in the omnidirectional image format as shown in  FIG. 9 . In step S 202  the position detector  204  detects a connecting position in an overlapping area between the 0 th  and 1 st  corrected images. 
     Now, the connecting position detection executed by the imaging system  10  is described, referring to  FIG. 10 .  FIG. 11  shows connecting position detection and  FIG. 12  shows the data structure of detection data generated by the position detector  204 . The operation in  FIG. 10  starts, following step S 202  in  FIG. 5 . In steps S 301  in S 304  each pixel (θ, φ) in the overlapping area of the 1 st  corrected image is subjected to the processing in steps S 302  and S 303 . The overlapping area is defined by the horizontal angles 0 to 360 degrees and the vertical angles φs to φe which are start and end points of the vertical angle of the overlapping area preset in accordance with the total angle of view of the lens systems. Each pixel is set in order by pattern matching of all the pixels of the overlapping area. 
     In step S 302  the position detector  204  sets a certain pixel block around each pixel (θ, φ) as a pattern image as shown in  FIG. 11A .  FIG. 11A  shows, by way of example, a pixel block  300  of 11 by 11 including the pixel (θ, φ) in question indicated by the star mark. Both ends (0 and 360 degrees) of θ coordinates of the omnidirectional image format are connected. Because of this, a pattern image including the periphery of the pixel in question is set, with a next to a right-side end considered as left end and vice versa. 
     In step S 303  the position detector  204  performs pattern matching of the 0 th  corrected image and the pattern image while moving the pattern image vertically and horizontally, to find a connecting position. Pattern matching can be template matching using correlation coefficient, city block distance, Euclidean distance, error sum of squares as similarity. In  FIG. 11B  the pattern image  300  of the 1 st  corrected image does not match the 0 th  corrected image at the original coordinate values but matches a 1 st  corrected image  310  at the coordinate values (θ+Δθ, φ+Δφ) shifted by a certain amount (Δθ, Δφ). That is, the 0 th  corrected image is appropriately aligned in position when the pixel value is at the coordinate (θ, φ) instead of (θ+Δθ, φ+Δφ). Accordingly, herein, a shift amount of (Δθ, Δφ) relative to the coordinate value (θ, φ) of the 0 th  corrected image for position detection is maintained. 
     The operation ends after all the pixels in the overlapping area are subjected to the processings in steps S 302  and S 303 . Thereby, the detection data  222  containing, for all the coordinate values, information about the converted coordinate values (θ, φ) associated with the shift amounts (Δθ, Δφ) is acquired. If there is a coordinate value for which the shift amount cannot be found in the above connecting position detection, the shift amount can be set to zero. Alternatively, the shift amount corresponding to each coordinate value can be calculated by interpolation according to all the sets of shift amounts obtained and the projection model. 
     Referring back to  FIG. 5 , in step S 203  the table corrector  206  corrects the data in the conversion table  220  on the basis of the detection data  222  so that the images are aligned in position on the spherical coordinates. As shown in  FIG. 12 , a shift amount is found for each coordinate value of the omnidirectional image format. In step S 203 , specifically, a 0 th  distortion correction table used for the 0 th  partial image is corrected so that an input coordinate value (θ, φ) in replace of (θ+Δθ, φ+Δφ) becomes associated with (x, y). Note that it is unnecessary to correct a 1 st  distortion correction table for the 1 st  partial image. 
     In step S 204  the table generator  208  generates the conversion table  224  for image synthesis from the corrected conversion table  220  by rotational coordinate conversion. 
       FIG. 13  is a flowchart for generating the conversion table for image synthesis by the imaging system  10 .  FIG. 14  shows the mapping of two partial images captured by two fisheye lenses into the spherical coordinate system for image synthesis. The operation in  FIG. 13  starts, following in step S 204  in  FIG. 5 . In steps S 401  to S 406  the table generator  208  performs processing in steps S 402  to S 405  for each coordinate value (θ g , φ g ) of the spherical coordinate system for image synthesis. A range of the coordinate values are defined by the entire ranges of the horizontal angles (0 to 360 degrees) and the vertical angles (0 to 180 degrees). All the coordinate values as input values are converted and set in order. 
     In step S 402  the table generator  208  finds a coordinate value (θ d , φ d ) of a spherical coordinate system for connecting position detection in association with the coordinate value (θ g , φ g ) by rotational coordinate conversion. By rotational coordinate conversion, the coordinate axes defined by the horizontal angle θ d  and vertical angle φ d  relative to the axis of one of the lens system in  FIG. 9  is convened to one defined by the horizontal angle θ g  and vertical angle φ g  relative to the axis orthogonal to the optical axis in  FIG. 14 . The coordinate (θ d , φ d ) corresponding to that (θ g , φ g ) can be calculated on the basis of the rotational coordinate conversion at radios being 1.0 by the following equations, using a three-dimensional Cartesian coordinate (x g , y g , z g ) corresponding to the spherical coordinate system (θ g , φ g ) for image synthesis and the three-dimensional Cartesian coordinate (x d , y d , z d ) corresponding to the spherical coordinate system (θ d , φ d ) for position detections. 
     
       
         
           
             
               x 
               g 
             
             = 
             
               
                 sin 
                  
                 
                   ( 
                   
                     φ 
                     g 
                   
                   ) 
                 
               
                
               
                 cos 
                  
                 
                   ( 
                   
                     θ 
                     g 
                   
                   ) 
                 
               
             
           
         
       
       
         
           
             
               y 
               g 
             
             = 
             
               
                 sin 
                  
                 
                   ( 
                   
                     φ 
                     g 
                   
                   ) 
                 
               
                
               
                 sin 
                  
                 
                   ( 
                   
                     θ 
                     g 
                   
                   ) 
                 
               
             
           
         
       
       
         
           
             
               z 
               g 
             
             = 
             
               cos 
                
               
                 ( 
                 
                   φ 
                   g 
                 
                 ) 
               
             
           
         
       
       
         
           
             
               ( 
               
                 
                   
                     
                       x 
                       d 
                     
                   
                 
                 
                   
                     
                       y 
                       d 
                     
                   
                 
                 
                   
                     
                       z 
                       d 
                     
                   
                 
               
               ) 
             
             = 
             
               
                 ( 
                 
                   
                     
                       1 
                     
                     
                       0 
                     
                     
                       0 
                     
                   
                   
                     
                       0 
                     
                     
                       
                         cos 
                          
                         
                             
                         
                          
                         β 
                       
                     
                     
                       
                         sin 
                          
                         
                             
                         
                          
                         β 
                       
                     
                   
                   
                     
                       0 
                     
                     
                       
                         
                           - 
                           sin 
                         
                          
                         
                             
                         
                          
                         β 
                       
                     
                     
                       
                         cos 
                          
                         
                             
                         
                          
                         β 
                       
                     
                   
                 
                 ) 
               
                
               
                 ( 
                 
                   
                     
                       
                         x 
                         z 
                       
                     
                   
                   
                     
                       
                         y 
                         z 
                       
                     
                   
                   
                     
                       
                         z 
                         z 
                       
                     
                   
                 
                 ) 
               
             
           
         
       
       
         
           
             
               φ 
               d 
             
             = 
             
               Arccos 
                
               
                 ( 
                 
                   z 
                   d 
                 
                 ) 
               
             
           
         
       
       
         
           
             
               θ 
               d 
             
             = 
             
               Arctan 
                
               
                 ( 
                 
                   
                     y 
                     d 
                   
                   
                     x 
                     d 
                   
                 
                 ) 
               
             
           
         
       
     
     where β is a rotational angle about X axis of the three-dimensional Cartesian coordinate and set to 90 degrees in the present embodiment. 
     According to the conversion table  220  the optical axis is projected on the poles of the spherical surface and the overlapping portion between the images is projected on near the equatorial line of the spherical surface. Therefore, a vertical direction of the omnidirectional image format does not match a zenith direction of a captured scene. Meanwhile, according to the conversion table  224 , by the rotational coordinate conversion the optical axis is projected on the equatorial line and a vertical direction of the omnidirectional image format matches a zenith direction of a captured scene. 
     In steps S 403  to S 405  the table generator  208  executes the processing in step S 404  to each of the 0 th  and 1 st  images. In step S 404  the table generator  208  finds the coordinate values (x, y) of the 0 th  and 1 st  partial images corresponding to (θ d , φ d ), referring to the corrected conversion table  220 . The conversion tables  220 ,  224  contain θ d  and φ d  together with a corresponding coordinate value (x, y) for each pixel. The coordinate values (θ d , φ d ) calculated by the conversion are typically values less than a decimal point. The coordinate values (x, y) can be simply ones corresponding to coordinate values in the most recent conversion table of the calculated coordinate values (θ d , φ d ). Preferably, the coordinate values (x, y) of the 0 th  and 1 st  partial images can be calculated by weighted interpolation in accordance with a distance from a calculated coordinate (θ d , φ d ), referring to coordinate values (x, y) corresponding to a most recent coordinate value and coordinate values around the most recent coordinate value in the conversion table. 
     The operation ends when the calculations for both the partial images are completed in step S 403  to step S 405  and the calculations of all the coordinate values for the correction table are completed in steps S 402  to S 406 . Thereby, all the items of data for the conversion table for image synthesis  224  are generated. 
     Returning to  FIG. 5 , in step S 205  the second distortion corrector  210  corrects distortion of the original 0 th  and 1 st  partial images to obtain the 0 th  and 1 st  corrected images for image synthesis, referring to the conversion table  224 . Thereby, the two partial images captured by the 0 th  and 1 st  fisheye lenses are laid out on the omnidirectional image format in  FIG. 14 . Generally, the 0 th  partial image is mapped on the left hemisphere while the 1 st  partial image is mapped on the right hemisphere. 
     In comparison with  FIG. 9 , the 0 th  and 1 st  partial images are mapped at different positions on the omnidirectional image format and the zenith direction of a captured scene coincides with the vertical direction φ of an image in  FIG. 14 . The centers of the 0 th  and 1st partial images are mapped on the equatorial line with less distortion and the overlapping area between the 0 th  and 1 st  corrected images are mapped in the vicinity of the vertical angles 0 and 180 degrees and the horizontal angles 0 and 180 degrees in  FIG. 14 . 
     In step S 206  the image synthesizer  212  generates 0 th  and 1 st  corrected images for image synthesis. In the synthesis process the overlapping areas are blended. If there is an area with pixel values of only one of the images, the pixel values are used without a change. Thus, a single omnidirectional image is created from two partial images captured with the fisheye lenses. 
     As described above, the conversion table used for image synthesis differs from that used for position detection in the present embodiment. This makes it possible to prevent a decrease in the accuracy at which images are connected due to a distortion even if the images are captured with a wide-angle lens as a fisheye lens which causes a larger distortion in the image overlapping area than a general lens. Thus, the imaging system can generate high quality omnidirectional images. 
     Second Embodiment 
     The first embodiment has described the imaging system  10  as an example of the image processor and the imaging system which includes the imaging unit to capture an omnidirectional still image and the distortion correction and image synthesis block to synthesize images. Alternatively, the image processor and imaging system can be configured as an omnidirectional video imaging system. The image processor can be a camera and processor to generate an omnidirectional still or video image, upon receiving partial still or video images captured by plural imaging units, a data processor such as a personal computer, a work station, a virtual machine on a physics computer system, or a portable data terminal such as a smart phone or tablet to synthesize an omnidirectional still or video image from input partial images captured with a dedicated omnidirectional imaging unit. The omnidirectional imaging system can be one including an image processor as the above-described camera and processor, data processor, or portable data terminal and an imaging unit separately. 
     In the following an omnidirectional imaging system comprising an omnidirectional imaging unit and an external computer unit to generate a synthetic omnidirectional image from input partial images captured with the omnidirectional imaging unit is described with reference to  FIG. 15  and  FIG. 16 .  FIG. 15  schematically shows the structure of an omnidirectional imaging system  300 . 
     The omnidirectional imaging system  300  in  FIG. 15  comprises an omnidirectional imaging unit  310  dedicated for capturing images and a computer unit  330  dedicated for image processing connected with the omnidirectional imaging unit  310 . Note that  FIG. 15  omits showing detailed structures. Also, the difference between the omnidirectional imaging system  300  in  FIGS. 15, 16  and the imaging system  10  in  FIG. 1  to  FIG. 14  is in that image processing for synthesizing an omnidirectional image is executed exclusively by the computer unit  330 . In the following the difference is mainly described. 
     The omnidirectional imaging unit  310  in  FIG. 15  comprises a digital still camera and processor  100 , a lens barrel unit  102 , and a triaxial acceleration sensor  120  connected to the processor  100 . The lens barrel unit  102  and the processor  100  are the same as those in  FIG. 2 . 
     The processor  100  comprises ISPs  108 A,  108 B, a USB block  146 , and a serial block  158 , to control USB communication wish the computer unit  330  connected via a USB connector  148 . The serial block  158  is connected with a wireless NIC  160  to control wireless communication with the computer unit  330  connected via a network. 
     The computer unit  330  in  FIG. 15  can be a general-purpose computer such as a desktop personal computer, work station and comprises hardware components as a processor, a memory, an ROM, and a storage medium. It includes a USB interface  332  and a wireless NIC  334  to connect with the omnidirectional imaging unit  316  via a USB bus or a network. 
     The computer unit  330  further comprises a first distortion corrector  202  for position detection, a position detector  204 , a table corrector  206 , a table generator  208 , a second distortion corrector  210  for image synthesis, and an image synthesizer  212 . In the present embodiment two partial images captured by multiple imaging units of the lens barrel unit  102  and a position-detection conversion table of the omnidirectional imaging unit  310  are transferred to the computer unit  330  outside via a USB bus or a network. 
     In the computer unit  330  the first distortion corrector  202  performs distortion correction to 0 th  and 1 st  partial images transferred from the omnidirectional imaging unit  310 , referring to the conversion table, to generate 0 th  and 1 st  corrected images for position detection. The position detector  204  detects a connecting position of the 0 th  and 1 st  corrected images and generates resultant detection data. The table corrector  206  corrects the conversion data in the conversion table on the basis of the detection data. The table generator  208  generates a conversion table for image synthesis from the corrected conversion data by rotational coordinate conversion. 
     The second distortion corrector  210  corrects distortion of the 0 th  and 1 st  partial images before image synthesis, referring to the image-synthesis conversion table and generates 0 th  and 1 st  corrected images for image synthesis. The image synthesizer  212  synthesizes the 0 th  and 1 st  corrected images to generate a synthetic image in omnidirectional image format. 
     The computer unit  330  can additionally include a display image generator  214  which executes image processing to project an omnidirectional image onto a planar display device. The computer unit  330  is configured to read a program from a ROM or HDD and expand it on a workspace of a RAM to thereby execute the above described functions and later-described processing under the control of the CPU. 
       FIG. 16  is a flowchart for the omnidirectional image synthesis of the omnidirectional imaging system  300  according to the present embodiment, describing from an input of a captured image from the omnidirectional imaging unit  310  to storing of the image in the computer unit  330 . 
     The operation starts when a shooting instruction is issued in response to press-down of the shutter button of the omnidirectional imaging unit  310 , for example. First, the omnidirectional imaging unit  310  executes the processing. 
     In step S 501  the two solid-state image sensors  22 A,  22 B of the omnidirectional imaging unit  310  capture 0 th  and 1 st  partial images. In stop S 502  the omnidirectional imaging unit  310  transfers the 0 th  and 1 st  partial images together with the position-detection conversion table to the computer unit  330  via the USB bus or network. In addition inclination information obtained by the triaxial acceleration sensor  120  is transferred to the computer unit  330  if the computer unit  330  executes inclination correction. 
     The above conversion table can be transferred once when the omnidirectional imaging unit  310  and the computer unit  330  recognize each other. That is, it is unnecessary to transfer the conversion table to the computer unit  330  every time images are transferred. For example, the position-detection conversion table is stored in a not-shown SDRAM and read and transferred therefrom. This completes the operation of the omnidirectional imaging unit  310 . 
     In step S 503  the computer unit  330  executes distortion correction to the 0 th  and 1 st  partial images transferred from the first distortion corrector  202 , referring to the conversion table, to acquire 0 th  and 1 st  corrected images. If the computer unit  530  executes inclination correction, the conversion data in the conversion table can be corrected in advance according to transferred inclination data relative to a vertical direction. In step S 504  the position detector  204  detects a connecting position of the overlapping area between the 0 th  and 1 st  corrected images to obtain resultant detection data. In step S 505  the table corrector  206  corrects the data in the conversion table according to the detection data so that the images are aligned in position on the spherical coordinates. In step S 506  the table generator  208  generates a conversion table for image synthesis from the corrected conversion table by rotational coordinate conversion. 
     In step S 507  the second distortion corrector  210  executes distortion correction to the original 0 th  and 1 st  partial images, referring to the image-synthesis conversion table, to acquire the 0 th  and 1 st  corrected images for image synthesis. In step S 508  the image synthesizer  212  synthesizes the 0 th  and 1 st  corrected images. Thus, a single omnidirectional image is generated from two partial images captured by the fisheye lenses. In step S 509  the computer unit  330  stores a generated omnidirectional image in an external storage, completing the operation. 
     Note that the operation in the flowchart of  FIG. 16  can be executable by a program on the computer. That is, the CPU controlling the operation of the omnidirectional imaging unit  310  and the CPU controlling the operation of the computer unit  330  each read a program from a storage as ROM or RAM and expand it on the memory to execute their respective processings of the omnidirectional image synthesis.  FIGS. 15 and 16  show a separate type omnidirectional imaging system by way of example and the present embodiment should not be limited thereto. The functions of the omnidirectional imaging system can be distributed to 1 or more imaging units and 1 or more computer systems in various types of configuration. 
     According to the above-described embodiments it is made possible to realize no image processor, image processing method and program, and an imaging system which can accurately connect captured images even with use of a lens system having a large amount of distortion. 
     The above embodiments have described an example where partial images are captured almost concurrently by different lens systems. Alternatively, partial images can be captured by the same lens system at different points of time from a certain shooting point in different orientations. Further, the present invention is applicable to the synthesis of three or more overlapped partial images captured by one or more lens systems in replace of the two overlapped partial images captured by the lens system having an angle of view over 180 degrees. Moreover, the present invention is applicable to an omnidirectional imaging system incorporating a super wide-angle lens instead of the imaging system using the fisheye lenses. 
     The functions of the omnidirectional imaging system can be realized by a computer-executable program written in legacy programming language such as assembler, C, C++, C#, JAVA® or object-oriented programming language. Such a program can be stored in a storage medium such as ROM, EEPROM, EPROM, flash memory, flexible disc, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, blue ray disc, SD card, or MO and distributed through an electric communication line. Further, a part or all of the above functions can be implemented on, for example, a programmable device (PD) as field programmable gate array (FPGA) or implemented as application specific integrated circuit (ASIC). To realize the functions on the PD, circuit configuration data as bit stream data and data written in HDL (hardware description language), VHDL (very high speed integrated circuits hardware description language), and Verilog-HDL stored in a storage medium can be distributed. 
     Although the present invention has been described in terms of exemplary embodiments, it is not limited thereto. It should be appreciated that variations or modifications may be made in the embodiments described by persons skilled in the art without departing from the scope of the present invention as defined by the following claims.