Abstract:
A method for restoration of building structure using infinity homographies calculated based on parallelograms includes: calculating, using two or more parallelograms, an infinity homography between those cameras which refer to an arbitrary camera; restoring cameras and the building structure on an affine space using the computed infinity homography and homologous points between images; and transforming the restored result onto the metric space using constraints on orthogonality of vectors joining the restored three-dimensional points, the ratio of lengths of the vectors and intrinsic camera parameters. As a result, intrinsic camera parameters, camera positions on the metric space and the structure of the building are restored. All the restoration is possible even when intrinsic camera parameters corresponding to all the images are not constant.

Description:
CROSS-REFERENCE(S) TO RELATED APPLICATION(S) 
     The present invention claims priority of Korean Patent Application No. 10-2008-0131370, filed on Dec. 22, 2008, which is incorporated herein by reference. 
     FIELD OF THE INVENTION 
     The present invention relates to a method that restores a building structure, and, in particular, to a method that calculates infinity homographies using parallelograms obtained from photographed building images and restores the building structure using the infinity homographies. 
     BACKGROUND OF THE INVENTION 
     Calculating a building structure and a camera position using pictures of a building photographed by a camera can be described as a practice in imaged-based modeling. Not only such a practice has been carried out in a number of research works in the field of computer vision but also it can be used widely in computer graphics, virtual reality and robotics. 
     The conventional methods of image-based modeling can be categorized into two groups. First, methods based on features extract corresponding feature points from many images and restore the three-dimensional positions of these points and cameras. Second, methods based on primitives take geometric primitives like balls, cylinders, cones or cubes as building blocks for more complex models and carry out a restoration procedure by projecting them again on images as parameters of the models are varied to minimize errors. Primary benefits are precise results with a relatively smaller number of images. 
     These conventional methods have shortcomings difficult to overcome. The feature-based methods need a relatively great number of images in order to achieve a certain degree of accuracy, whereas the primitive-based methods cannot be applied with ease to general scenes where the primitives are absent. 
     SUMMARY OF THE INVENTION 
     In view of the above, the present invention provides a method which calculates infinity homographies using parallelograms when there are only a few images of a building that is subject to restoration and more than two parallelograms are given as the only geometric information. The other object of the present invention is to provide a method that restores the structure of the building using the calculation method. 
     In accordance with one embodiment of the present invention, there is provided a calculation method for an infinity homography based on parallelograms, including: transforming parallelograms in the three-dimensional space of images photographed by cameras to respective rectangles and calculating rectifying homographies of the respective images; calculating homographies of new images transformed by using the calculated rectifying homographies; calculating a linear constraint for the infinity homography of the respective images by using the rectifying homographies and the calculated homographies of new images; calculating a linear constraint for the infinity homography from a vanishing point by using a relationship between the infinity homography of the images and the vanishing points corresponding to two images; and calculating the infinity homography by using the calculated linear constraints. 
     In accordance with another embodiment of the present invention, there is provided a method for restoration of a building structure using an infinity homography calculation method based on a parallelogram, including: calculating an infinity homography of the images based on respective parallelograms in the three-dimensional space of images photographed by cameras; restoring the structure of a building and the cameras on an affine space by using the calculated infinity homography; and transforming the affine space to a metric space by using intrinsic parameters of the respective cameras or a constraint on a space recognized in the respective images. 
     The present invention provides a method which allows not only calibration and restoration of positions of cameras and restoration of a building structure at the same time but also restoration of a building structure with a few images photographed at random or found on the internet, when there are only a few images of a building and parallelograms are the only geometric information for the building subject to restoration. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The above features of the present invention will become apparent from the following description of embodiments given in conjunction with the accompanying drawings, in which: 
         FIG. 1  shows a flowchart of a restoration process of a building structure using a method that calculates an infinity homography based on parallelograms in accordance with an embodiment of the present invention. 
         FIG. 2  illustrates a flowchart of a process that calculates an infinity homography in accordance with an embodiment of the present invention. 
         FIGS. 3A to 3D  depict an example of rectifying camera images converted by a rectifying homography. 
         FIGS. 4A to 4D  show an example of a situation for the parallelograms not used in  FIGS. 3A to 3D . 
         FIG. 5  depicts an image of a building used to verify the contents of the present invention. 
         FIG. 6  illustrates an image of a building used to verify the contents of the present invention. 
         FIG. 7  shows an image of a building used to verify the contents of the present invention. 
         FIG. 8  depicts an image of a building used to verify the contents of the present invention. 
         FIG. 9  depicts the restored result of cameras and building obtained by applying the method of the present invention to the images in  FIGS. 5 to 8 . 
     
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
     Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. 
       FIG. 1  shows a flowchart of a three-dimensional restoration procedure of an object in accordance with an embodiment of the present invention. As shown in  FIG. 1 , the procedure starts with calculation S 100  of infinity homographies H ∞  between images. 
       FIG. 2  illustrates a flowchart of a procedure that calculates an infinity homography H ∞  in accordance with an embodiment of the present invention. 
     A procedure in which rectifying homographies are calculated for a parallelogram present on an arbitrary object in the three-dimensional space is depicted in  FIGS. 3A to 3D . As shown in  FIG. 3A , a parallelogram A present on a building, which is an image projected by a camera at a chosen position, can be transformed into an arbitrary rectangle in S 200 , as depicted in  FIG. 3B . The relationship between the rectangle and the parallelogram is given by a two-dimensional rectifying homography H rA1 , a 3×3 matrix calculated in S 202 . A similar procedure can be applied to transform same parallelogram, as illustrated in  FIG. 3C , obtained by a projected image photographed by another camera into an arbitrary rectangle, as shown in  FIG. 3D . 
     A two-dimensional homography for the transformation between the images in  FIGS. 3C to 3D  shall be denoted by H rA2 . 
     A rectangle used in the calculation of a rectifying homography can be chosen in an arbitrary way, to be of any size or aspect ratio, as long as it remains a rectangle.  FIGS. 4A to 4D  show that a same procedure depicted in the above is adopted for calculation of rectifying homographies H rB1  and H rB2  for another parallelogram B present in the image. 
     If there are new images transformed from original camera images by using rectifying homographies, a homography between the new images is calculated S 204 , i.e. a homography H ∞   rA  between the images in  FIGS. 3B and 3D  and a homography H ∞   rB  between the images in  FIGS. 4B and 4D . Note that the first two diagonal components, the (1,1) and (2,2) components, have identical values in homographies H ∞   rA  and H ∞   rB , respectively. 
     By denoting by H ∞  the infinity homography between original images, e.g. the images in  FIGS. 3A and 3C  or those in  FIGS. 4A and 4C , the relationships between infinity homography H ∞  and rectifying homographies, H rA1 , H rA2 , H rB1 , and H rB2 , for parallelograms A and B are given respectively by
 
 H   ∞   =H   rA2   −1   H   ∞   rA   H   rA1 ,
 
 H   ∞   =H   rB2   −1   H   ∞   rB   H   rB1 .  Equation (1)
 
These relationships are arranged to yield homographies H ∞   rA  and H ∞   rB  between the respective new images:
 
 H   ∞   rA   =H   rA2   H   ∞   H   rA1   −1 ,
 
 H   ∞   rB   =H   rB2   H   ∞   H   rB1   −1 .  Equation (2)
 
     A linear constraint can be derived for infinity homography H ∞  between original images S 206  by equating, for example, the first two diagonal components of homography H ∞   rA  between the new images for parallelogram A in  FIGS. 3A to 3D . Such a constraint can also be derived for all parallelograms that can be present in addition to parallelogram A. 
     As is well known, in general, two vanishing points v 1  and v 2  corresponding to their respective images are related linearly by infinity homography H ∞  between the images as
 
 v   2   =H   ∞   v   1 .  Equation (3)
 
This relationship indicates that from a vanishing point two linear constraints for infinity homography H ∞  can be derived S 208 .
 
     Since there are more than three vanishing points, v 1 , v 2 , v 3  and v 4 , as shown in  FIGS. 3A and 4A , eight linear constraints can be obtained for infinity homography H ∞  if more than two parallelograms are employed. Because infinity homography H ∞  has eight degrees of freedom, it is always possible to calculate the components of infinity homography H ∞  with the eight linear constraints S 210 . 
     It is of great importance to note that, in the procedure described in the above, the parallelograms depicted in  FIGS. 3A and 3B , while they need to be identical on the space, can be used to form a constraint by regarding them as being identical if they can be overlapped by translation. For example, although parallelograms D and E in  FIGS. 5 to 8  are different parallelograms on the space, they can be overlapped by translation and hence are regarded as being identical so that they can be used to form the linear constraint described in the above. This is of great use in relaxing the requirement that the fields of views of cameras be overlapped as much as possible in the example addressed in  FIGS. 5 to 8 . 
     The next step is to restore the cameras and the structure of a building S 110  on the affine space using infinity homography H ∞ , which will be explained in what follows. 
     To start with, an image is chosen for a reference and then projection matrix P o  for the reference camera that corresponds to the reference image is given by
 
 P   o   =[I|O],   Equation (4)
 
where I is the 3×3 identity matrix and 0 is the 1×3 zero column vector. By denoting by H ∞   i  infinity homography between the i th arbitrary camera among the rest and the reference camera, the projection matrix for the i th camera can be given by
 
 P   i   =[H   ∞   i   |t   i ],  Equation (5)
 
where t i  is a 1×3 vector to be solved for.
 
     A point x i   j  projected on the ith camera of jth point x j  that is to be restored on the space is given by
 
 x   i   j   =P   i   X   j .  Equation (6)
 
     Applying this equation for all the points and cameras under consideration leads to a set of simultaneous linear equations. Solving the equations gives restored cameras and building structure on the affine space. 
     The procedure S 120  that transforms the restored result on the affine space into the metric space using the spatial constraints is described henceforth. Since the spatial constraints are extracted from the images by the cameras, these can be information that can be verified directly from the images. 
     The procedure starts with finding absolute conic Ω ∞   A  on the affine space. If two vectors d A1  and d A2  joining the restored points are orthogonal on the real metric space due to the spatial constraints, they are related by absolute conic Ω ∞   A  as
 
 d   A1   T Ω ∞   A   d   A2 =0.  Equation (7)
 
If the ratio of lengths of two vectors, d A1  and d A2 , is given by r based on the constraints, they are related by absolute conic Ω ∞   A  as
 
 d   A1   T Ω ∞   A   d   A1   =r   2   d   T   A2   d   A2 .  Equation (8)
 
If the ratio of lengths of two vectors, d A1  and d A2 , and the angle they make are given respectively by r and θ, then they are related by absolute conic Ω ∞   A  as
 
 d   A1   T Ω ∞   A   d   A2   =r  cos θ d   T   A2 Ω ∞   A   d   A2 .  Equation (9)
 
Equations (7) to (9) are results that can be obtained without difficulty for cases of general buildings. A system of homogeneous linear equations can be yielded by using all or part of them. In other words, the relationships between absolute conic Ω ∞   A  and vectors d A1  and d A2  in the above Equations (7) to (9) are applied selectively depending on the acknowledged constraints and by the applied equations homogeneous linear equations are yielded.
 
     The procedure to get absolute conic Ω ∞   A  using the constraints of the intrinsic camera parameters is to be addressed in the below. If the intrinsic camera parameters are constant, the relationship between absolute conic Ω ∞   A  and infinity homography H ∞   i  is given by
 
Ω ∞   A   =H   ∞   i     −T   Ω ∞   A   H   ∞   i     −1   .  Equation (10)
 
If the pixels are square, the first off-diagonal component of absolute conic Ω ∞   A  is zero:
 
( H   ∞   i     −1   Ω ∞   A   H   ∞   i     −1   ) 12 =0,  Equation (11)
 
where (Σ) ij  denotes the (i,j) th component of an arbitrary matrix Σ. If the optic axis of the camera passes through the center of the image, the other off-diagonal components of absolute conic Ω ∞   A  vanish, yielding two other constraints:
 
( H   ∞   i     −1   Ω ∞   A   H   ∞   i     −1   ) 13 =0,
 
( H   ∞   i     −1   Ω ∞   A   H   ∞   i     −1   ) 23 =0.  Equation (12)
 
     If the pixels of the image have an aspect ratio r, then the first two diagonal components of absolute conic Ω ∞   A  are related by
 
 r   2 ( H   ∞   i     −T   Ω ∞   A   H   ∞   i     −1   ) 11 =( H   ∞   i     −T   Ω ∞   A   H   ∞   i     −1   ) 22 .  Equation (13)
 
Equations (11) to (13) are used to obtain homogeneous linear equations for absolute conic Ω ∞   A , which, after all the available constraints applied, are solved to give absolute conic Ω ∞   A .
 
     Now, an A matrix can be calculated S 130  by a Choleski factorization of absolute conic Ω ∞   A :
 
Ω ∞   A   =A   −T   A   −1 .  Equation (14)
 
And three-dimensional homography H E   A  is obtained S 140  by the definition given by
 
                     H   E   A     =       (         A       0           0       1         )     .             Equation   ⁢           ⁢     (   15   )                 
Finally, camera projection P i   E  and points X j   E  on the metric space are obtained by transforming corresponding P i  and X j  on the affine space with application of three-dimensional homography H E   A :
 
 P   i   E   =H   E   A   P   i ,
 
 X   j   E   =H   E   A     −1     X   j .  Equation (16)
 
       FIGS. 5 to 8  are images that are used to verify the embodiments of the present invention.  FIG. 9  shows the restored result of cameras and building obtained by applying the method of the present invention to the images in  FIGS. 5 to 8 . Parallelograms C, D and E are used for the procedure and, as mentioned in the above, parallelograms D and E are regarded as being identical when infinity homography H ∞  is calculated S 100 . The spatial constraints used here are that vectors a and b in  FIG. 5  are orthogonal and that the length ratio of the vectors c and d is 1:6 and that the edges of the building are orthogonal. 
     The restoration procedure of a three-dimensional structure of a building in accordance with an embodiment of the present invention can be realized in the form of a computer-readable code on a computer-readable record medium. Computer-readable record media include all sorts of record media on which are stored data that can be read by a computer system. Typical examples are ROM, RAM, CD-ROM, magnetic tape, hard disks, floppy diskettes, flash memory, and optical data storage devices as well as those realizable in the form of carrier waves (e.g. transmission over the internet). Computer-readable record media can be stored and executed as a code readable in a distributive manner on a distributed computer system interconnected on a communication network. The font ROM data structure of the present invention can also be realized as a computer-readable code on record media such as ROM, RAM, CD-ROM, magnetic tape, hard disks, floppy diskettes, flash memory, optical storage devices. 
     While the invention has been shown and described with respect to the embodiments, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the scope of the invention as defined in the following claims.