Scene imaging method using a portable two-camera omni-imaging device for human-reachable environments

A scene imaging method using a portable two-camera omni-imaging device for human-reachable environments is disclosed. Firstly, an upper omni-image and a lower omni-image of at least one scene are captured. Next, image inpainting and image dehazing perform on the upper omni-image and the lower omni-image. Next, image unwrapping performs on the upper omni-image and the lower omni-image by a pano-mapping table, so as to form an upper panoramic image and a lower panoramic image respectively, and the upper panoramic image and the lower panoramic image respectively have an upper part and a lower part overlapping each other. Finally, a data item on the upper part and the lower part is obtained by view points of the scene and the pano-mapping table, thereby stitching the upper and lower panoramic images.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an imaging method, particularly to a scene imaging method using a portable two-camera omni-imaging device for human-reachable environments.

2. Description of the Related Art

In recent years, many street view systems have been developed. To exploit a certain street scene, one may just browse a street view system to see the scene online without going there. This provides great value to applications like city touring, real estate sales, environmental management, etc. To collect street scenes, Google Street View Cars were used. To enter narrow lanes, a Street View Trike was designed later. Also launched was a Street View Snowmobile for use on rough terrains. Aboard each of these vehicles is an imaging device with eight digital cameras, one fish-eye camera, and three laser range finders for scene collection. The device weighs about 150 kg.

Though the use of the Street View Trek overcomes the incapability of the Street View Car in reaching narrow alleys, it is hard for a rider to pedal the heavy tricycle on steep ways. Also, although the Snowmobile has better mobility, it still cannot be ridden to some spaces like indoor stairways, mountains tracks, garden paths, etc. Below are briefly cited some prior arts of the scene imaging method. Among them, the Taiwan patent No. M373507 uses two cameras to capture a scene at different angles, and stitches a plurality of images by calculating setting values of the cameras. Besides, the Taiwan patent No. 201200960 uses a camera to record directions and angles of images in combination with other equipments, wherein the directions and angles are helpful in stitching the images. For the prior arts, processing a large amount of data would result in time delay.

In view of the problems and shortcomings of the prior art, the present invention provides a scene imaging method using a portable two-camera omni-imaging device for human-reachable environments, so as to solve the afore-mentioned problems of the prior art.

SUMMARY OF THE INVENTION

The primary objective of the present invention is to provide a scene imaging method using a portable two-camera omni-imaging device for human-reachable environments, which captures an upper omni-image and a lower omni-image and stitch them. The method uses properties of the omni-imaging device to greatly reduce the image amount for matching in the stitching process, thereby reducing time for the stitching process. Besides, the omni-imaging device has light weight and high mobility, and overcomes the limit for equipments and environments to take images of environments that a person reaches.

To achieve the above-mentioned objectives, the present invention proposes a scene imaging method using a portable two-camera omni-imaging device for human-reachable environments. Firstly, an upper omni-image and a lower omni-image of at least one scene are captured. Next, image inpainting and image dehazing perform on the upper omni-image and the lower omni-image. Next, image unwrapping performs on the upper omni-image and the lower omni-image by a pano-mapping table, so as to form an upper panoramic image and a lower panoramic image respectively, and the upper panoramic image and the lower panoramic image respectively have an upper part and a lower part overlapping each other. Then, a data item on the upper part and the lower part is obtained by a plurality of view points of the scene and the pano-mapping table. Finally, the upper panoramic image and the lower panoramic image are stitched to form a total panoramic image according to the data.

Below, the embodiments are described in detailed in cooperation with the attached drawings to make easily understood the technical contents, characteristics, and accomplishments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

As an improvement on conventional street view systems like the Google Map and the Google Earth, which cannot cover some special environment areas, such as indoor space with stairways, mountains with steep roads, gardens or parks with narrow tracks, etc. The present invention uses a portable omni-imaging device, which is light and can be carried by a person on his/her back. The omni-imaging device can be used to take an upper omni-image and a lower omni-image simultaneously. The two images cover respectively the upper and lower hemi-spherical fields of view of the environment with an overlapping image band. In other words, the omni-imaging device with high mobility can overcome the limit for equipments and environments to take images of environments that a person reaches.

Refer toFIG. 1andFIG. 2. The present invention, the omni-imaging device, uses a first omni-camera10and a second omni-camera12aligned coaxially in a back-to-back fashion and disposed on a stabilizer14. The stabilizer14is carried by a person on his/her back to overcome the vibration problem for the first omni-camera10and the second omni-camera12. The first omni-camera10comprises a first projective camera16and a first hyperboloidal-shaped mirror18, and the second omni-camera12comprises a second projective camera20and a second hyperboloidal-shaped mirror22, and wherein the first hyperboloidal-shaped mirror18and the second hyperboloidal-shaped mirror22are aligned coaxially in the back-to-back fashion, and the first projective camera16is disposed above the first hyperboloidal-shaped mirror18, and the second projective camera22is disposed below the second hyperboloidal-shaped mirror20. If the mirror bottom plane is designed, as usual, to go through a focal point of the hyperboloidal-shaped of the mirror, then when two omni-cameras of such a design are not connected seamlessly as in the usual case, a circular blind region, which is not observable by either camera, will appear undesirably. Therefore, in the present invention, the mirror bottom plane has to be lower than the focal point as shown inFIG. 2. Moreover, after two cameras with such mirrors are attached back to back, a circular overlapping region will appear in both omni-images as shown inFIG. 2. The overlapping region is helpful in building a total panoramic image.

Refer toFIG. 1andFIG. 3. Firstly, in Step S10, the first omni-camera10and the second omni-camera12respectively capture an upper omni-image and a lower omni-image of at least one scene. The upper omni-image and the lower omni-image are transmitted to a processor. Next, in Step S12, the processor performs image inpainting and image dehazing on the upper omni-image and the lower omni-image, and the resulted upper and lower omni-images are shown inFIGS. 4(a)-4(b). Next, a pano-mapping table is obtained by the rotation-invariant property (which means that an azimuth angle of a point in space is the same to that of a corresponding image point) of the first omni-camera10and the second omni-camera12, and a relationship between an elevation angle of a projecting point in space and radiuses of mapping points on the first hyperboloidal-shaped mirror18and the second hyperboloidal-shaped mirror22. The pano-mapping table can show a pair of azimuth angle and elevation angle of a real-world point, and coordinates of a mapping point formed by the real-world point on the omni-image. Next, in Step S14, the processor performs image unwrapping on the upper omni-image and the lower omni-image by the pano-mapping table, so as to form an upper panoramic image and a lower panoramic image respectively. The upper panoramic image and the lower panoramic image respectively have an upper part and a lower part overlapping each other. The upper part and the lower part are formed by the overlapping region. According to the optical principle and the space-mapping approach, a light ray of a point in space can form a matching line on the omni-image. Next, in Step S16, the processor establishes a plurality of first matching lines24on the upper omni-image26and the lower omni-image28by a plurality of sight points of the scene and the rotation-invariant property of the first omni-camera10and the second omni-camera12, as shown inFIGS. 5(a)-5(b). Next, in Step S18, the processor uses the pano-mapping table to transform the first matching lines24, so as to obtain a plurality of corresponding second matching lines30on the upper panoramic image32and the lower panoramic image34, and the second matching lines30comprise a plurality of second upper matching lines36on the upper part and a plurality of second lower matching lines38on the lower part, and the second upper matching lines36respectively correspond to the second lower matching lines38, and each second upper matching line corresponds to one view point, as shown inFIG. 6.

Next, in Step S20, the processor performs edge detection on the second upper matching lines36and the second lower matching lines38, so as to obtain a plurality of upper feature points on the second upper matching lines36and a plurality of lower feature points on the second lower matching lines38, as shown inFIGS. 7(a)-7(b).

Due to differences in noise, distortion, and scaling between the two panoramic images, the feature points appearing in the second upper matching line might not appear in the second lower matching line, and vice versa, resulting in insertions and deletions of feature points in the matching lines. Regarding feature points which match mutually as substitutions. As a result, in Step S22, the processor then defines that the upper feature points and the lower feature points matching each other respectively have substitution costs, and that the upper feature points that the lower feature points don't match respectively have insertion costs, and that the lower feature points that the upper feature points don't match respectively have deletion costs. For example, the substitution cost is expressed by |H(u)−H(l)|/Hmax, wherein H(u) is a hue value of the upper feature point, H(l) is a hue value of the lower feature point, and Hmaxis a maximum hue value, and wherein the insertion cost is 0.5, and the deletion cost is 0.5.

Next, in Step S24, the processor performs the dynamic programming technique on the upper feature points and the lower feature points according the substitution costs, the insertion costs, and the deletion costs, so as to obtain a minimum cost sequence as an optimal matching point sequence, and the optimal matching point sequence comprises parts of the upper feature points and parts of the lower feature points matching each other, and the parts of upper feature points are used as upper matching points, the parts of lower feature points are used as lower matching points, as shown inFIGS. 8(a)-8(b).

Next, in Step S26, the processor determines whether light rays of each upper matching point and each lower matching point matching each other intersect on the region that a first view of the first omni-camera10overlaps a second view of the second omni-camera12. If the answer is no, the process proceeds to Step S28for deleting the determined upper matching point and the determined lower matching point, and then Step S30is performed. If the answer is yes, as shown inFIG. 9, the process proceeds to Step S30.

As shown inFIG. 10, in Step S30, the processor chooses two upper matching points a and b on the same second upper matching line36and two lower matching points c and d on the same second lower matching line38, which match each other, and determines whether the difference of a first interval11and a second interval12is less than a first predetermined value, and the first interval11is the largest interval between the two upper matching points a and b, and the second interval12is the largest interval between the two lower matching points c and d. If the answer is no, the process proceeds to Step S32for deleting the two upper matching points a and b and the two lower matching points c and d, and then Step S34is performed. If the answer is yes, the process proceeds to Step S34.

As shown inFIG. 11, in Step S34, the processor chooses at least two adjacent upper matching points a and e on different second upper matching lines36and at least two adjacent lower matching points c and g on different second lower matching lines38, which match each other, and determines whether the difference of a first distance h and a second distance h′ is less than a second predetermined value, and the first distance h is a vertical distance between the two adjacent upper matching points a and e, and the second distance h′ is a vertical distance between the two adjacent lower matching points c and g. If the answer is no, the process proceeds to Step S36for deleting the determined upper matching points corresponding to the first distance h and the lower matching points corresponding to the second distance h′, and then Step S38is performed. If the answer is yes, the process proceeds to Step S38.

In Step S38, the processor determines whether at least one upper matching point and at least one lower matching point are deleted. If the answer is no, the upper matching points and the lower matching points are regarded as a data item, and Step S40is performed. In Step S40, the processor performs the homographic transformation technique to stitch the upper panoramic image and the lower panoramic image to form a total panoramic image according to the upper matching points and the lower matching points on the upper part and the lower part. If the answer is yes, the process proceeds to Step S39. As shown inFIG. 10andFIG. 12, if the two deleted upper matching points a and b are used as first replaceable matching points, and the two deleted lower matching points c and d are used as second replaceable matching points. In Step S39, the processor chooses two neighboring upper matching points n1and n3, or n2and n4at two sides of the first replaceable matching point a or b and two neighboring lower matching points n5and n7, or n6and n8at two sides of the second replaceable matching point c or d, which match each other, builds a first connection line connecting with the two neighboring upper matching points n1and n3, or n2and n4and intersects one second upper matching line36to generate a first intersection a′ or b′, and builds a second connection line connecting with the two neighboring lower matching points n5and n7, or n6and n8and intersects one second lower matching line38to generate a second intersection c′ or d′. The above intersections become the new matching points for image stitching. Next, in Step S40, the processor performs the homographic transformation technique to stitch the upper panoramic image and the lower panoramic image to form a total panoramic image according to the remaining upper matching points and the remaining lower matching points on the upper part and the lower part, the first intersection, and the second intersection, which are all used as a data item. Step S39is used to maintain the matching relationship for the matching lines.

Alternatively, Steps S26-S39can be replaced with a step of obtaining the data on the upper part and the lower part by the view points of the scene and the pano-mapping table.

And, the total panoramic image of Step S40is shown inFIG. 13. The present invention uses the dynamic programming technique and the properties of the omni-camera to greatly reduce the image amount for matching in the stitching process, thereby reducing time for the image stitching process.

If there is a plurality of scene spot, a process is performed after Step S40, as shown inFIG. 14. In order to dynamically display in a scene browsing system and immediately retrieve the image information, in Step S42, the processor firstly records the upper matching points, the lower matching points, the first intersections, the second intersections, the upper omni-image, the lower omni-image, the upper panoramic image, the lower panoramic image, the total panoramic image, and geographical coordinates of each scene spot, so as to avoid repeating the image process. Next, in Step S44, the processor transforms each total panoramic image into a perspective-view image by a perspective-mapping table in order to browse a real environment image. The perspective-mapping table can generate the perspective-view image for any of directions without complicated computations, so as to achieve the goal of real-time display. Specifically, the entries of perspective-mapping table are recorded by the corresponding coordinates of the pano-mapping table. When a user wants to change the direction of the scene image, the entries of the perspective-mapping table can be shifted to obtain the perspective-view image required according to the changes of azimuth angle and elevation angle. Then, the processor uses optical properties and recording rules for images to calculate the coordinates on the omni-images and the perspective-view image of each scene spot by the geographical coordinates, thereby forming a walking path for browsing. Next, in Step S46, the processor establishes an environment three-view diagram by the geographical coordinates of each scene. The processor is coupled to a display and an operation interface such as a mouse, and transmits the total panoramic image, the perspective-view image, and the environment three-view diagram to the display for showing.

Finally, in Step S48, the processor changes a viewpoint through the total panoramic image, the perspective-view image, or the environment three-view diagram. For example, as shown inFIG. 15, the user may navigate the environment further at the current scene spot by 6 ways: (1) changing the viewpoint though the perspective-view image (by clicking and dragging the image to turn into a desired view); (2) changing the viewpoint through the total panoramic image (by clicking a point on the image to select a new view); (3) going forward or backward by navigating the walking path (by clicking on an arrow of the path); (4) going forward or backward step by step on the walking path (by clicking on the arrow); (5) changing the scene spot through the three-view diagram (by clicking on a point any of the three views to see the closest scene spot); (6) zooming in or out the current view (by clicking on the perspective-view image).

In the process ofFIG. 14, Step S48can be omitted whereby the display effect of the present invention is also achieved.

The present invention has several merits: (1) it is light to carry to any indoor/outdoor place; (2) it can be used to collect images in narrow, non-planar, or slanted paths where existing street view vehicles cannot navigate; (3) only two omni-images are acquired to construct various images for each scene spot; (4) perspective-view images can be generated for scene observation from any view direction; (5) six ways of environment browsing via a perspective-view image, a panoramic image, a walking path, and a three-view diagram are provided.

In conclusion, the present invention can greatly reduce the image amount and time for the image stitching process, and take images of environments that a person can reach.

The embodiments described above are only to exemplify the present invention but not to limit the scope of the present invention. Therefore, any equivalent modification or variation according to the shapes, structures, characteristics and spirit of the present invention is to be also included within the scope of the present invention.