# Calibration Method for Central Catadioptric Camera Using Multiple Groups of Parallel Lines and Their Properties

Authors: Yue Zhao; Xin Yang
Journal: Journal of Sensors
 (2021)

---

## Abstract

This paper presents an approach for calibrating omnidirectional single-viewpoint sensors using the central catadioptric projection properties of parallel lines. Single-viewpoint sensors are widely used in robot navigation and driverless cars; thus, a high degree of calibration accuracy is needed. In the unit viewing sphere model of central catadioptric cameras, a line in a three-dimensional space is projected to a great circle, resulting in the projections of a group of parallel lines intersecting only at the endpoints of the diameter of the great circle. Based on this property, when there are multiple groups of parallel lines, a group of orthogonal directions can be determined by a rectangle constructed by two groups of parallel lines in different directions. When there is a single group of parallel lines in space, the diameter and tangents at their endpoints determine a group of orthogonal directions for the plane containing the great circle. The intrinsic parameters of the camera can be obtained from the orthogonal vanishing points in the central catadioptric image plane. An optimization algorithm for line image fitting based on the properties of antipodal points is proposed. The performance of the algorithm is verified using simulated setups. Our calibration method was validated though simulations and real experiments with a catadioptric camera.


---

## Body

## 1. Introduction

With the rapid development of computer vision technology, requirements for visual performance have become increasingly stringent. Increasing the field of view of a camera usually improves its visual performance [1–3]. A traditional camera has a limited field of view; using mirrors, as originally proposed by Hecht [4], offers an effective method to enhance the field of view. This camera–mirror system is referred to as a catadioptric camera system and can be divided into two types, central and noncentral, depending on the presence of a unique effective viewpoint [5]. There are four principal types of mirror shapes for central catadioptric cameras: paraboloidal, hyperboloidal, ellipsoidal, and planar. The image captured by the central catadioptric camera can be easily converted into three-dimensional (3D) coordinates of basic feature points using a projection inverse algorithm. This technique is widely used in the field of computer vision.This paper addresses calibration issues in central catadioptric cameras with a generalized projection model proposed by Geyer and Daniilidis [6]. The results demonstrate that the imaging process is equivalent to two-step mapping with a unit viewing sphere. Previous calibration methods for central catadioptric cameras can be divided into five types: self-calibration [7, 8], calibration based on two-dimensional (2D) points [9–11], calibration based on 3D points [12], calibration based on lines [13–18], and calibration based on spheres [19–21]. Li and Zhao [22] calibrated a catadioptric camera by obtaining orthogonal directions and circular points only through the image of a single sphere. This demonstrates that the analysis of orthogonal directions under the catadioptric projection model is important and useful.First, we present a brief review of the methods proposed by others for central catadioptric camera calibration. Deng et al. [11] proposed a simple calibration method based on a 2D calibration pattern for central catadioptric camera. They proposed a nonlinear constraint on camera intrinsic parameters from the projections of any three collinear points on the viewing sphere. Geyer and Daniilidis [13] proposed a calibration method for a paracatadioptric camera using the images of three lines and demonstrated rectification of an image using at least two groups of parallel lines. Ying and Hu [14] used the projections of lines or balls to calibrate cameras. They showed that the line and the ball had three and two limitations, respectively, when it came to their use in calibrating cameras; moreover, the method involved was nonlinear. When the noise level is very large, the robustness degree of the methods decreases as singularity approaches.This paper studies the geometry of the central catadioptric projections of multiple groups of parallel lines. A set of projective properties are described and proved. These properties support the geometric constructions proposed for calibration. In addition, this study considers the applications of the projection properties of parallel lines in the unit viewing sphere model to calibrate central catadioptric cameras; hence, these methods could apply to any of the mirror-type cameras. The main contribution of this study can be summarized as follows:(I)
Two groups of parallel lines in different directions are projected to great circles and intersect at four points. According to the properties of antipodal points, the orthogonal vanishing points can be determined on the image plane to obtain the intrinsic parameters of the camera(II)
A group of parallel lines is projected to a group of great circles on the unit viewing sphere. The diameter of the circles and tangents at their endpoints determine a group of orthogonal directions for the support plane containing each great circle to obtain the intrinsic parameters of the camera(III)
An optimization algorithm for line image fitting is proposed based on the relationship between the antipodal pointsThe remainder of this paper is organized as follows. Section2 presents a review of the unit sphere model for central catadioptric cameras. Section 3 describes two calibration algorithms for central catadioptric cameras that employ the projection properties of parallel lines as well as an optimization algorithm for line image fitting. Simulation and real data were used to verify the effectiveness of the algorithms, and this is discussed in Section 4. Finally, conclusions regarding the two calibration algorithms are presented in Section 5.


## 2. Preliminaries

In this section, we briefly review the projection process of points and lines under central catadioptric cameras. Moreover, we define antipodal image points and discuss their properties.


### 2.1. Central Catadioptric Projection Model

Geyer and Daniilidis [6] proved that the central catadioptric imaging process is equivalent to the following two-step mapping with a unit viewing sphere (Figure 1). First, a point M in 3D space is projected to two points M± on the surface of the unit viewing sphere centered at the focal point O of a reflective mirror. The unit viewing sphere is called the unit sphere, and O is the center of the projection. Second, with the 3D point Oc at the center, M± is projected to two points m± on image plane Π, where Oc is called the virtual optical center, Π is perpendicular to the line OOc, and their intersection is the principal point p.Figure 1
Projections of a three-dimensional pointM and parallel lines L1 and L2 under the unit viewing sphere model of a central catadioptric camera.As shown in Figure1, the world coordinate system is adjusted to the unit viewing sphere coordinate system O−xwywzw. Let M=xyz1T denote the homogeneous coordinate of a point in O−xwywzw; the projections M± of M on the unit viewing sphere can then be represented as M±=±x/M∧±y/M∧±z/M∧1T, where M^=x2+y2+z2 and M^ denotes the nonhomogeneous coordinate of M. The unit viewing sphere coordinate system O−xwywzw traverses ξ units along the ‐zw direction to establish the virtual camera coordinate system Oc−xcyczc. Thus, the rotation matrix R and the translation vector t between them can be represented as R=I and t=00ξT, respectively, where I is a three-identity matrix and ξ=O−Oc is the mirror parameter. The type of mirror selected depends on the value of ξ: the mirror is a plane if ξ=0, an ellipsoid or hyperboloid if 0<ξ<1, and a paraboloid if ξ=1.Let the intrinsic parameter matrix of the virtual camera be(1)K=fusu00fvv0001,where fu and fv represent the focal length along the u and v directions, respectively, on the 2D image plane. u0v01T are the homogeneous coordinates of the principal point p, and s is the skew factor. Then, with the unit viewing sphere, the imaging process from point M± to m± can be described as
(2)λ1,2m±=KM~±+00ξT,where λ1 and λ2 are two nonzero scale factors and M~±=±x/M∧±y/M∧±z/M∧Tdenote the nonhomogeneous coordinates of points M±.


### 2.2. Antipodal Image Points

For a central catadioptric camera, Wu et al. [16] defined the antipodal image points and their properties as follows:Definition 1.
In Figure1, M+,M− is called a pair of antipodal points if they are two endpoints of the diameter of the viewing sphere.Definition 2.
In Figure1, m+,m− is called a pair of antipodal image points if they are images of two endpoints of the diameter of the viewing sphere.Proposition 3.
In Figure1, if m+,m− is a pair of antipodal image points obtained using a central catadioptric camera, then
(3)1+1+τm+Tωm+m+Tωm+m++1+1+τm−Tωm−m−Tωm−m−=2p,where τ=1−ξ2/ξ and ω=K−TK−1. In the case of a paraboloid mirror, Equation (3) can be simplified to
(4)1m+Tω¯m+m++1m−Tω¯m−m−=p.


## 2.1. Central Catadioptric Projection Model

Geyer and Daniilidis [6] proved that the central catadioptric imaging process is equivalent to the following two-step mapping with a unit viewing sphere (Figure 1). First, a point M in 3D space is projected to two points M± on the surface of the unit viewing sphere centered at the focal point O of a reflective mirror. The unit viewing sphere is called the unit sphere, and O is the center of the projection. Second, with the 3D point Oc at the center, M± is projected to two points m± on image plane Π, where Oc is called the virtual optical center, Π is perpendicular to the line OOc, and their intersection is the principal point p.Figure 1
Projections of a three-dimensional pointM and parallel lines L1 and L2 under the unit viewing sphere model of a central catadioptric camera.As shown in Figure1, the world coordinate system is adjusted to the unit viewing sphere coordinate system O−xwywzw. Let M=xyz1T denote the homogeneous coordinate of a point in O−xwywzw; the projections M± of M on the unit viewing sphere can then be represented as M±=±x/M∧±y/M∧±z/M∧1T, where M^=x2+y2+z2 and M^ denotes the nonhomogeneous coordinate of M. The unit viewing sphere coordinate system O−xwywzw traverses ξ units along the ‐zw direction to establish the virtual camera coordinate system Oc−xcyczc. Thus, the rotation matrix R and the translation vector t between them can be represented as R=I and t=00ξT, respectively, where I is a three-identity matrix and ξ=O−Oc is the mirror parameter. The type of mirror selected depends on the value of ξ: the mirror is a plane if ξ=0, an ellipsoid or hyperboloid if 0<ξ<1, and a paraboloid if ξ=1.Let the intrinsic parameter matrix of the virtual camera be(1)K=fusu00fvv0001,where fu and fv represent the focal length along the u and v directions, respectively, on the 2D image plane. u0v01T are the homogeneous coordinates of the principal point p, and s is the skew factor. Then, with the unit viewing sphere, the imaging process from point M± to m± can be described as
(2)λ1,2m±=KM~±+00ξT,where λ1 and λ2 are two nonzero scale factors and M~±=±x/M∧±y/M∧±z/M∧Tdenote the nonhomogeneous coordinates of points M±.


## 2.2. Antipodal Image Points

For a central catadioptric camera, Wu et al. [16] defined the antipodal image points and their properties as follows:Definition 1.
In Figure1, M+,M− is called a pair of antipodal points if they are two endpoints of the diameter of the viewing sphere.Definition 2.
In Figure1, m+,m− is called a pair of antipodal image points if they are images of two endpoints of the diameter of the viewing sphere.Proposition 3.
In Figure1, if m+,m− is a pair of antipodal image points obtained using a central catadioptric camera, then
(3)1+1+τm+Tωm+m+Tωm+m++1+1+τm−Tωm−m−Tωm−m−=2p,where τ=1−ξ2/ξ and ω=K−TK−1. In the case of a paraboloid mirror, Equation (3) can be simplified to
(4)1m+Tω¯m+m++1m−Tω¯m−m−=p.


## 3. Calibrating Catadioptric Cameras Using Projections of Multiple Groups of Parallel Lines

In the unit sphere model, a line in 3D space is projected to a great circle, where the center of the great circle is the center of the viewing sphere; thus, two great circles intersect at two points.Theorem 4.
In Figure1, the great circles, from which a group of parallel lines on one plane are projected on the unit viewing sphere, only intersect at two points: a pair of antipodal pointsM+,M−.Proof.
In Figure1, a group of parallel lines in one plane intersect at a point at infinity. With a central catadioptric projection model, the point at infinity projected onto the unit sphere forms two points. Hence, according to Definition 1, the points of intersection of the great circles of the projections of the group of parallel lines are a pair of antipodal points M+,M−.☐Corollary 5.
In Figure1, for the catadioptric camera, the images show a group of parallel lines in one plane intersecting at two points, called a pair of antipodal image points.Proof.
Based on Theorem4 and geometric invariance under a central projective transformation, the images of the group of parallel lines have only two points of intersection, the images of the points of intersection of the great circles.☐In this study, the 3D lines are not parallel to thezw-axis of the catadioptric camera coordinate system. If the 3D lines are parallel to the zw-axis, the line images coincide into a straight line.


### 3.1. Camera Calibration Using Two Groups of Parallel Lines

Letl1i and l2ii=1,2,3,4 be two groups of great circles on the unit viewing sphere corresponding to projections of two groups of parallel lines in two directions, where i=1,2,3,4, as shown in Figure 2(a). The center of the unit viewing sphere is O. Points M1+ and M1− are the points of intersection of l1i, points M2+ and M2− are the points of intersection of l2i. Points M1± and M2± are two pairs of antipodal points. According to Definition 1, the quadrilateral with points M1+,M2+,M1−,M2− as vertices is a rectangle; thus, lines M1+M2+ and M1+M2−are orthogonal, and lines M1+M2+ and M1−M2− are parallel and intersect at point D1∞ at infinity. Similarly, lines M1+M2− and M1−M2+ are parallel and intersect at point D2∞ at infinity.Proposition 6.
In a central catadioptric system, if the images of two groups of parallel lines in different directions are given, a group of orthogonal vanishing points can be obtained.Proof.
As shown in Figure2(b), based on geometric invariance under central projective transformation, the images C1i and C2i of the great circles l1i and l2i also intersect at two points m1± and m2±, respectively, which are images of the points M1± and M2±. The line passing through points m1+ and m2+ intersects the line passing through m1− and m2− at the vanishing point d1, which is the image of the point D1∞ at infinity. Similarly, two lines m1+m2− and m1−m2+intersect at the vanishing point d2, which is the image of the point D2∞at infinity. Thus,
(5)d1=m1+×m2+×m1−×m2−,d2=m1+×m2−×m1−×m2+.
According to the properties of the projection, pointsd1 and d2 are a group of orthogonal vanishing points, hence, proving Proposition 6.☐Figure 2
(a) Projections of two groups of parallel linesl1i and l2i (i=1,2,3,4) in different directions on the unit viewing sphere; (b) images of two groups of parallel lines in different directions in a central catadioptric system.
(a)(b)


### 3.2. Camera Calibration Using a Group of Parallel Lines

Proposition 7.
In a central catadioptric system, if the images of a group ofi parallel lines (i=4, in Figure 3) are given, i groups of the orthogonal vanishing points can be obtained.Proof.
As shown in Figure2(a), a group of circles l1i is the projection of a group of parallel lines on the unit viewing sphere. Points M1+ and M1− are the points of intersection of circles l1i, which are the two endpoints of the diameters of circles l1i. According to the properties of the tangent of a circle, the tangents of points M1+ and M1− with respect to circles l1ii≥2 are parallel and orthogonal to the diameter M1−M1+.
As shown in Figure3, images C1i of circles l1i intersect at two points m1±, which are the images of points M1±.Without losing generality, to simplify the representation, we can take images C1i of circles l1i with i=1 as examples, and let C1ii≥2 be Cii≥2. The tangents of points m1+ and m1− with respect to conics Cii≥2 intersect at point d1, the vanishing point in the direction of the tangents calculated by
(6)d1=C1m1−×C1m1+.
d2 denotes the fourth harmonic element of points m1−,m1+, and p, where p is the image of the midpoint of the diameter of circles l1i. Using crossratio invariance [23],
(7)m1+,m1−;p,d2=−1,and d2 is the vanishing point in the direction of diameter m1− and m1+. The principal point p can be obtained by the equation of the projection contour of the mirror [14]. Points d1 and d2 are a pair of orthogonal vanishing points for the image of the plane including circle l1i with i=1.☐Figure 3
Vanishing points in the direction of the tangents and the direction of the diameter at the points of intersection of line images with respect to conicC11.


### 3.3. Line Image Fitting in Central Catadioptric Cameras

The projection of a 3D line is a conic in a central catadioptric system. However, only a section of the conic is visible in the central catadioptric image plane. Therefore, conic fitting is inaccurate when only a section of it is known.Proposition 8.
In a central catadioptric system, knowing theNN≥5 image points and their antipodal image points, conic fitting can be accomplished.Proof.
Let conicC be the image of a 3D line with a central catadioptric camera; consider NN≥5 points mi+ on the visible section of the conic, where i=1,2,3,⋯,N. The projective contour of the mirror can be completely extracted and the equation of the projective contour of the mirror can be formulated by least-squares fitting to initialize the camera’s intrinsic parameters K~ [14]. Hence, according to Proposition 3, the antipodal image point mi− of point mi+ can be determined. Let conic C be
(8)C=ax2+bxy+cy2+dx+ey+f=0.
We need to minimize the algebraic distanceCxj,yj from the corresponding points to conic C [24]; then,
(9)Γ=∑j=12NC2xj,yj,where xj,yj are the coordinates of image points mi+ or its antipodal image points mi−. We then minimize the objective function Γ (9), to which there always exists the trivial solution a=b=c=d=e=d=0. To avoid the existence of zero solution, we normalize Cx,y with a=1, b=1, c=1, d=1, e=1, and f=1. Whena=1, let X1=bcdefT; thus, the equation Cxj,yj=0 becomes ajTX1−bj=0, where aj=xjyjyj2xjyj1T and bj=−xj2. Given 2N points, we obtain the following equation:
(10)AX1=b,where A=a1a2⋯a2NT and b=b1b2⋯b2NT. By minimizing objective function (9), we obtain
(11)ΓX1=AX1−bTAX1−b.
We then minimize the objective function—specifically, we obtain the partial derivative of (11) with respect to X1, making the partial derivative equal to zero:
(12)2ATAX1−b=0.
The solution of (12) is
(13)X1=ATA−1ATb,and the coefficients of the conic are X11=1X1TT.
Similarly, by normalizing withb=1, c=1, d=1, e=1, and f=1, the coefficients of the conic X21,X31,X41,X51,X61 can be obtained. Thus, the equation of the line image can be obtained by
(14)X=16∑i=16Xi1.☐


### 3.4. Calibration Algorithms Using Orthogonal Vanishing Points

According to Sections3.1 and 3.2, multiple groups of orthogonal vanishing points can be obtained from images of two groups of parallel lines. A pair of orthogonal vanishing points d1 and d2 can be represented with the homogeneous coordinates d1=x1y11T and d2=x2y21T, respectively. From the relationship between the orthogonal vanishing points and the image of the absolute conic (IAC) [23], it follows that
(15)d1Tωd2=0,where ω=K−TK−1 represents the IAC and K denotes the intrinsic parameter matrix of the camera. ω with five degrees of freedom can be solved linearly with at least five groups of orthogonal vanishing points. K can then be determined using the Cholesky factorization of ω.The intrinsic parameters of the camera can thus be obtained from two groups of parallel lines in different directions (Method 1) or a group of parallel lines (Method 2). The two algorithms can be summarized as follows:Step 1.
Inputkk≥5 views, each containing projections of at least two groups of parallel lines in different directions.Step 2.
Fit the equation of the projective contour of the mirror by least-squares fitting; obtain the initial value of the intrinsic parameters of the camera.Step 3.
TakeNN≥5 points for each projection of the line, and calculate their antipodal image points; estimate the equation of the line image using Proposition 8.Step 4.
Calculate the two common points of intersection of the images in each group of parallel lines using the “solve” function in MATLAB.Step 5.
Solve the tangents at the points of intersection corresponding to each conicl=Cm.Step 6.
Obtain the orthogonal vanishing points using Proposition6 or 7.Step 7.
When the orthogonal vanishing points are known, solveω using (15). Determine K using the Cholesky factorization of ω.


## 3.1. Camera Calibration Using Two Groups of Parallel Lines

Letl1i and l2ii=1,2,3,4 be two groups of great circles on the unit viewing sphere corresponding to projections of two groups of parallel lines in two directions, where i=1,2,3,4, as shown in Figure 2(a). The center of the unit viewing sphere is O. Points M1+ and M1− are the points of intersection of l1i, points M2+ and M2− are the points of intersection of l2i. Points M1± and M2± are two pairs of antipodal points. According to Definition 1, the quadrilateral with points M1+,M2+,M1−,M2− as vertices is a rectangle; thus, lines M1+M2+ and M1+M2−are orthogonal, and lines M1+M2+ and M1−M2− are parallel and intersect at point D1∞ at infinity. Similarly, lines M1+M2− and M1−M2+ are parallel and intersect at point D2∞ at infinity.Proposition 6.
In a central catadioptric system, if the images of two groups of parallel lines in different directions are given, a group of orthogonal vanishing points can be obtained.Proof.
As shown in Figure2(b), based on geometric invariance under central projective transformation, the images C1i and C2i of the great circles l1i and l2i also intersect at two points m1± and m2±, respectively, which are images of the points M1± and M2±. The line passing through points m1+ and m2+ intersects the line passing through m1− and m2− at the vanishing point d1, which is the image of the point D1∞ at infinity. Similarly, two lines m1+m2− and m1−m2+intersect at the vanishing point d2, which is the image of the point D2∞at infinity. Thus,
(5)d1=m1+×m2+×m1−×m2−,d2=m1+×m2−×m1−×m2+.
According to the properties of the projection, pointsd1 and d2 are a group of orthogonal vanishing points, hence, proving Proposition 6.☐Figure 2
(a) Projections of two groups of parallel linesl1i and l2i (i=1,2,3,4) in different directions on the unit viewing sphere; (b) images of two groups of parallel lines in different directions in a central catadioptric system.
(a)(b)


## 3.2. Camera Calibration Using a Group of Parallel Lines

Proposition 7.
In a central catadioptric system, if the images of a group ofi parallel lines (i=4, in Figure 3) are given, i groups of the orthogonal vanishing points can be obtained.Proof.
As shown in Figure2(a), a group of circles l1i is the projection of a group of parallel lines on the unit viewing sphere. Points M1+ and M1− are the points of intersection of circles l1i, which are the two endpoints of the diameters of circles l1i. According to the properties of the tangent of a circle, the tangents of points M1+ and M1− with respect to circles l1ii≥2 are parallel and orthogonal to the diameter M1−M1+.
As shown in Figure3, images C1i of circles l1i intersect at two points m1±, which are the images of points M1±.Without losing generality, to simplify the representation, we can take images C1i of circles l1i with i=1 as examples, and let C1ii≥2 be Cii≥2. The tangents of points m1+ and m1− with respect to conics Cii≥2 intersect at point d1, the vanishing point in the direction of the tangents calculated by
(6)d1=C1m1−×C1m1+.
d2 denotes the fourth harmonic element of points m1−,m1+, and p, where p is the image of the midpoint of the diameter of circles l1i. Using crossratio invariance [23],
(7)m1+,m1−;p,d2=−1,and d2 is the vanishing point in the direction of diameter m1− and m1+. The principal point p can be obtained by the equation of the projection contour of the mirror [14]. Points d1 and d2 are a pair of orthogonal vanishing points for the image of the plane including circle l1i with i=1.☐Figure 3
Vanishing points in the direction of the tangents and the direction of the diameter at the points of intersection of line images with respect to conicC11.


## 3.3. Line Image Fitting in Central Catadioptric Cameras

The projection of a 3D line is a conic in a central catadioptric system. However, only a section of the conic is visible in the central catadioptric image plane. Therefore, conic fitting is inaccurate when only a section of it is known.Proposition 8.
In a central catadioptric system, knowing theNN≥5 image points and their antipodal image points, conic fitting can be accomplished.Proof.
Let conicC be the image of a 3D line with a central catadioptric camera; consider NN≥5 points mi+ on the visible section of the conic, where i=1,2,3,⋯,N. The projective contour of the mirror can be completely extracted and the equation of the projective contour of the mirror can be formulated by least-squares fitting to initialize the camera’s intrinsic parameters K~ [14]. Hence, according to Proposition 3, the antipodal image point mi− of point mi+ can be determined. Let conic C be
(8)C=ax2+bxy+cy2+dx+ey+f=0.
We need to minimize the algebraic distanceCxj,yj from the corresponding points to conic C [24]; then,
(9)Γ=∑j=12NC2xj,yj,where xj,yj are the coordinates of image points mi+ or its antipodal image points mi−. We then minimize the objective function Γ (9), to which there always exists the trivial solution a=b=c=d=e=d=0. To avoid the existence of zero solution, we normalize Cx,y with a=1, b=1, c=1, d=1, e=1, and f=1. Whena=1, let X1=bcdefT; thus, the equation Cxj,yj=0 becomes ajTX1−bj=0, where aj=xjyjyj2xjyj1T and bj=−xj2. Given 2N points, we obtain the following equation:
(10)AX1=b,where A=a1a2⋯a2NT and b=b1b2⋯b2NT. By minimizing objective function (9), we obtain
(11)ΓX1=AX1−bTAX1−b.
We then minimize the objective function—specifically, we obtain the partial derivative of (11) with respect to X1, making the partial derivative equal to zero:
(12)2ATAX1−b=0.
The solution of (12) is
(13)X1=ATA−1ATb,and the coefficients of the conic are X11=1X1TT.
Similarly, by normalizing withb=1, c=1, d=1, e=1, and f=1, the coefficients of the conic X21,X31,X41,X51,X61 can be obtained. Thus, the equation of the line image can be obtained by
(14)X=16∑i=16Xi1.☐


## 3.4. Calibration Algorithms Using Orthogonal Vanishing Points

According to Sections3.1 and 3.2, multiple groups of orthogonal vanishing points can be obtained from images of two groups of parallel lines. A pair of orthogonal vanishing points d1 and d2 can be represented with the homogeneous coordinates d1=x1y11T and d2=x2y21T, respectively. From the relationship between the orthogonal vanishing points and the image of the absolute conic (IAC) [23], it follows that
(15)d1Tωd2=0,where ω=K−TK−1 represents the IAC and K denotes the intrinsic parameter matrix of the camera. ω with five degrees of freedom can be solved linearly with at least five groups of orthogonal vanishing points. K can then be determined using the Cholesky factorization of ω.The intrinsic parameters of the camera can thus be obtained from two groups of parallel lines in different directions (Method 1) or a group of parallel lines (Method 2). The two algorithms can be summarized as follows:Step 1.
Inputkk≥5 views, each containing projections of at least two groups of parallel lines in different directions.Step 2.
Fit the equation of the projective contour of the mirror by least-squares fitting; obtain the initial value of the intrinsic parameters of the camera.Step 3.
TakeNN≥5 points for each projection of the line, and calculate their antipodal image points; estimate the equation of the line image using Proposition 8.Step 4.
Calculate the two common points of intersection of the images in each group of parallel lines using the “solve” function in MATLAB.Step 5.
Solve the tangents at the points of intersection corresponding to each conicl=Cm.Step 6.
Obtain the orthogonal vanishing points using Proposition6 or 7.Step 7.
When the orthogonal vanishing points are known, solveω using (15). Determine K using the Cholesky factorization of ω.


## 4. Experiments

To test the validity and feasibility of the proposed calibration methods, we performed a number of experiments using simulation data in addition to real images. We compared the results with the calibration methods of Deng et al. [11], Geyer and Daniilidis [18], and Li and Zhao [22].


### 4.1. Experiment for Conic Fitting

The simulated catadioptric camera had the following parameters:fu=600, fv=550, s=0.8, u0=400, and v0=350. The parameter of the hyperboloidal mirror ξ=0.966. The image resolution was 800×700. The 3D lines were not parallel with the zw-axis of the catadioptric camera coordinate system.In the simulation experiment, the conic of the line image was fitted by the method stated in Proposition8 and the least-squares method, respectively, and the intrinsic parameters of the camera were calibrated. One hundred points were chosen on each line image. Gaussian noise with a zero mean and standard deviation σ was added to each pixel point. The noise level σ varied from 0 to 3.5 pixels. For each noise level, we performed 200 independent trials to solve the absolute error of the intrinsic parameters fu,fv,u0,v0, and s of the camera and compared the results for the two conic fitting methods. The changes in the absolute errors of fu,fv,u0,v0, and s with different noise levels are shown in Figures 4(a)–4(e). The line image of the least-squares method obtained through fitting is denoted by Method 1-1 and Method 2-1. The proposed methods in this study are denoted by Method 1 and Method 2; Li and Zhao’s [22] methods are denoted by Li-1 (Proposition 6), Li-2 (Proposition 7), and Li-3 (Proposition 8), respectively.Figure 4
Variations in the absolute errors of (a)fu, (b) fv, (c) u0, (d) v0, and (e) s with different fitting line image methods under varying levels of Gaussian noise.
(a)(b)(c)(d)(e)Typically, the accuracy of the obtained calibration results highly depends on the accuracy of the extracted conics. Figures4(a)–4(e) demonstrate that the accuracy of all methods suffers from degradation in the presence of increasing noise. However, Method 1 and Method 2 perform better than the other methods. The comparison of the experimental results indicated that in the center catadioptric camera, the accuracy of fitting the line image using the antipodal constraint was greater than that observed with the least-squares method.


### 4.2. Experiment Using Simulation Data

The line image was fitted using the method proposed in Section3.3. In addition, with the assumption of Gaussian noise, the absolute errors in the intrinsic parameters of the camera, fu,fv,u0,v0, and s with different noise levels were calculated and compared with the reference methods [18, 11, 22], as shown in Figures 5(a)–5(e), respectively. Furthermore, the run time of the seven methods were compared using the MATLAB R2014b platform; the results obtained after 200 simulations are listed in Table 1.Figure 5
Variations in the absolute errors of (a)fu, (b) fv, (c) u0, (d) v0, and (e) s with different calibration methods under varying levels of Gaussian noise.
(a)(b)(c)(d)(e)Table 1
Average runtime of the methods (unit: second).
Method 1Method 2Geyer [18]Deng [11]Li-1 [22]Li-2 [22]Li-3 [22]Runtime0.1570.1260.3720.4360.2080.2780.241It is well recognized that the linear method is less affected by noise. Figures5(a)–5(e) demonstrate that as the noise level σ increased, the absolute errors of Deng et al.’s [11], Geyer and Daniilidis’ [18], and Li and Zhao’s [22] methods, including three different ways denoted by Li-1 (Proposition 6), Li-2 (Proposition 7), and Li-3 (Proposition 8), respectively, increased linearly. The errors from our methods showed a slower rate of increase, indicating the effectiveness and feasibility of the proposed method. Furthermore, Table 1 reveals that the average runtimes of our methods and Li and Zhao’s [22] were similar, whereas the runtime of Deng et al.’s [11] and Geyer and Daniilidis’ [18] methods was higher than our methods. This result is likely as Deng et al.’s [11] and Geyer and Daniilidis’ [18] methods were nonlinear.Only a section of the projection of a straight line for the central catadioptric system is visible. The angle between the directions of the 3D lines influences their projections and affects the calibration accuracy. In the simulation experiment, the angle of the direction of the first group of parallel lines was set to be perpendicular to thezw-axis, and the direction of the second group of parallel lines was set to be parallel to the zw-axis as the initial value. The direction of the second group of parallel lines rotates the angle θ from 0° to 90°around the xw-axis. The line image was fitted using the method proposed in Section 3.3. For each angle, 200 independent experiments are carried out. The absolute errors in the intrinsic parameters of the camera, fu,fv,u0,v0, and s with different angles were calculated and compared with the reference methods [18], as shown in Figures 6(a)–6(e). Li and Zhao [22] calibrated the camera using the projection image of a sphere, and Deng et al.’s [11] method was based on a 2D calibration pattern of points. The angle of lines had a small effect on their methods and, thus, were not considered in this experiment.Figure 6
Variations in the absolute errors of (a)fu, (b) fv, (c) u0, (d) v0, and (e) s with different calibration methods under different angles between one of the parallel lines and zw-axis.
(a)(b)(c)(d)(e)When one of the groups of parallel lines rotates around thexw-axis, the angle between the directions of the two parallel lines is changed accordingly. Figure 7 shows that calibration accuracy is high and stable when their angle is between 20° and 70°. When the direction of one of the groups of parallel lines is nearly parallel to the zw-axis, the image of this group of parallel lines tends to be a straight line with a large error in calibration results. When the angle between the directions of two groups of parallel lines becomes smaller, the interference between the two groups of parallel lines becomes larger under noisy conditions, resulting in a larger absolute error. When two groups of parallel lines approach coincidence, Method 1 in this study and Geyer and Daniilidis’ [18] method will degenerate and have large errors. Therefore, the angle between the directions of two parallel lines used for calibration is selected as 20°≤θ≤70° in the real situation.Figure 7
Image of a checkerboard used in the experiments.


### 4.3. Experiment Using Images

The angle between the directions of the two parallel lines is selected according to the influence of the angle on the calibration results in the simulation experiment, and the angle is20°≤θ≤70° in the real experiment. In general, the noise in a real environment is unknown; random Gaussian noise is the main source. Gaussian noise was used to simulate the real noise in the simulation experiment. Figure 5 shows that the proposed method has robustness in the presence of noise. Next, we used real images to verify the accuracy and applicability of the methods in this study. In the real experiment, a checkerboard was used as a calibration model to replace the line. We used a central catadioptric camera with a hyperboloidal mirror having parameter ξ=0.966, designed by the Center for Machine Perception at Czech Technical University. Five pictures of the checkerboard were taken using the catadioptric camera with its position changed. One of the five images is shown in Figure 7. The resolutions of these images were 1320×1170 pixels. The equations of the line images were obtained using the method described in Section 3.3.The intrinsic parameters of the camera were obtained with the two calibration methods proposed in this study in addition to Deng et al. [11], Geyer and Daniilidis [18], and Li and Zhao’s [22] calibration methods. Table 2 lists the calibration results on real data.Table 2
Calibration results with real data (unit: pixels).
Calibration methodsCamera’s intrinsic parametersfufvu0v0sMethod 11076.44291072.85371.0815650.7238585.5324Method 21084.75011081.50821.0763648.9626587.7246Geyer [18]1093.07191089.80251.1583653.6927586.8968Deng [11]1086.25771082.79281.2128651.4500587.7956Li-1 [22]1076.44291472.85371.0584650.7238585.5324Li-2 [22]1094.75011489.60820.9763644.9626589.7246Li-3 [22]1062.87431441.34501.5256648.0849580.4329To check the calibration results in Table2, the test image in Figure 7 was rectified to its projective image, as shown in Figures 8(a)–8(g).Figure 8
Rectified images based on the intrinsic parameters in Table2. Rectified image using (a) Method 1, (b) Method 2, (c) Geyer and Daniilidis [18], (d) Deng et al. [11], (e) Li-1 (Proposition 6), (f) Li-2 (Proposition 7), and (g) Li-3 (Proposition 8) from Li and Zhao [22].
(a)(b)(c)(d)(e)(f)(g)The results of all methods in Table2 were similar to each other, indicating that the proposed methods are feasible. The straight line in the space is changed into a curve due to lens distortion in the original image shown in Figure 7. After using the data in Table 2 to rectify the image, the straight line in the space is still straight in the images shown in Figures 8(a)–8(g). The experiments showed that the proposed calibration methods in this study are effective.To further verify the accuracy of the method, the data in Table2 was used to rectify the image shown in Figure 7, and the checkerboard reconstruction results were extracted. The reconstruction information in parallel and orthogonal directions of the checkerboard is shown in Figures 9(a)–9(g). First, we extracted the points on each row and each column to fit the equation of the line. Next, we calculated the angles between two lines in all parallel directions and obtained their average value. Similarly, the average value of angles between all orthogonal lines could also be obtained. Table 3 lists the angle results with real data as shown in Figures 9(a)–9(g).Figure 9
Reconstruction information of the checkerboard based on the intrinsic parameters in Table2. Reconstructed using (a) Method 1, (b) Method 2, (c) Geyer and Daniilidis [18], (d) Deng et al. [11], (e) Li-1 (Proposition 6), (f) Li-2 (Proposition 7), and (g) Li-3 (Proposition 8) from Li and Zhao [22].
(a)(b)(c)(d)(e)(f)(g)Table 3
Verifying the parallelity and orthogonality of 3D reconstruction result with real data after rectification (unit: degree).
Method 1Method 2Geyer [18]Deng [11]Li-1 [22]Li-2 [22]Li-3 [22]Parallel0.62330.60230.72660.62510.62630.71320.6840Orthogonal89.377289.308289.126589.254789.193589.170089.1506The ground truth data were 0° for the parallel lines and 90° in the orthogonal direction on the checkerboard, as already known. Using our methods in the reconstruction results, the angle between the parallel and orthogonal directions is closer to the ground truth, which proves the accuracy of the method in this study.


## 4.1. Experiment for Conic Fitting

The simulated catadioptric camera had the following parameters:fu=600, fv=550, s=0.8, u0=400, and v0=350. The parameter of the hyperboloidal mirror ξ=0.966. The image resolution was 800×700. The 3D lines were not parallel with the zw-axis of the catadioptric camera coordinate system.In the simulation experiment, the conic of the line image was fitted by the method stated in Proposition8 and the least-squares method, respectively, and the intrinsic parameters of the camera were calibrated. One hundred points were chosen on each line image. Gaussian noise with a zero mean and standard deviation σ was added to each pixel point. The noise level σ varied from 0 to 3.5 pixels. For each noise level, we performed 200 independent trials to solve the absolute error of the intrinsic parameters fu,fv,u0,v0, and s of the camera and compared the results for the two conic fitting methods. The changes in the absolute errors of fu,fv,u0,v0, and s with different noise levels are shown in Figures 4(a)–4(e). The line image of the least-squares method obtained through fitting is denoted by Method 1-1 and Method 2-1. The proposed methods in this study are denoted by Method 1 and Method 2; Li and Zhao’s [22] methods are denoted by Li-1 (Proposition 6), Li-2 (Proposition 7), and Li-3 (Proposition 8), respectively.Figure 4
Variations in the absolute errors of (a)fu, (b) fv, (c) u0, (d) v0, and (e) s with different fitting line image methods under varying levels of Gaussian noise.
(a)(b)(c)(d)(e)Typically, the accuracy of the obtained calibration results highly depends on the accuracy of the extracted conics. Figures4(a)–4(e) demonstrate that the accuracy of all methods suffers from degradation in the presence of increasing noise. However, Method 1 and Method 2 perform better than the other methods. The comparison of the experimental results indicated that in the center catadioptric camera, the accuracy of fitting the line image using the antipodal constraint was greater than that observed with the least-squares method.


## 4.2. Experiment Using Simulation Data

The line image was fitted using the method proposed in Section3.3. In addition, with the assumption of Gaussian noise, the absolute errors in the intrinsic parameters of the camera, fu,fv,u0,v0, and s with different noise levels were calculated and compared with the reference methods [18, 11, 22], as shown in Figures 5(a)–5(e), respectively. Furthermore, the run time of the seven methods were compared using the MATLAB R2014b platform; the results obtained after 200 simulations are listed in Table 1.Figure 5
Variations in the absolute errors of (a)fu, (b) fv, (c) u0, (d) v0, and (e) s with different calibration methods under varying levels of Gaussian noise.
(a)(b)(c)(d)(e)Table 1
Average runtime of the methods (unit: second).
Method 1Method 2Geyer [18]Deng [11]Li-1 [22]Li-2 [22]Li-3 [22]Runtime0.1570.1260.3720.4360.2080.2780.241It is well recognized that the linear method is less affected by noise. Figures5(a)–5(e) demonstrate that as the noise level σ increased, the absolute errors of Deng et al.’s [11], Geyer and Daniilidis’ [18], and Li and Zhao’s [22] methods, including three different ways denoted by Li-1 (Proposition 6), Li-2 (Proposition 7), and Li-3 (Proposition 8), respectively, increased linearly. The errors from our methods showed a slower rate of increase, indicating the effectiveness and feasibility of the proposed method. Furthermore, Table 1 reveals that the average runtimes of our methods and Li and Zhao’s [22] were similar, whereas the runtime of Deng et al.’s [11] and Geyer and Daniilidis’ [18] methods was higher than our methods. This result is likely as Deng et al.’s [11] and Geyer and Daniilidis’ [18] methods were nonlinear.Only a section of the projection of a straight line for the central catadioptric system is visible. The angle between the directions of the 3D lines influences their projections and affects the calibration accuracy. In the simulation experiment, the angle of the direction of the first group of parallel lines was set to be perpendicular to thezw-axis, and the direction of the second group of parallel lines was set to be parallel to the zw-axis as the initial value. The direction of the second group of parallel lines rotates the angle θ from 0° to 90°around the xw-axis. The line image was fitted using the method proposed in Section 3.3. For each angle, 200 independent experiments are carried out. The absolute errors in the intrinsic parameters of the camera, fu,fv,u0,v0, and s with different angles were calculated and compared with the reference methods [18], as shown in Figures 6(a)–6(e). Li and Zhao [22] calibrated the camera using the projection image of a sphere, and Deng et al.’s [11] method was based on a 2D calibration pattern of points. The angle of lines had a small effect on their methods and, thus, were not considered in this experiment.Figure 6
Variations in the absolute errors of (a)fu, (b) fv, (c) u0, (d) v0, and (e) s with different calibration methods under different angles between one of the parallel lines and zw-axis.
(a)(b)(c)(d)(e)When one of the groups of parallel lines rotates around thexw-axis, the angle between the directions of the two parallel lines is changed accordingly. Figure 7 shows that calibration accuracy is high and stable when their angle is between 20° and 70°. When the direction of one of the groups of parallel lines is nearly parallel to the zw-axis, the image of this group of parallel lines tends to be a straight line with a large error in calibration results. When the angle between the directions of two groups of parallel lines becomes smaller, the interference between the two groups of parallel lines becomes larger under noisy conditions, resulting in a larger absolute error. When two groups of parallel lines approach coincidence, Method 1 in this study and Geyer and Daniilidis’ [18] method will degenerate and have large errors. Therefore, the angle between the directions of two parallel lines used for calibration is selected as 20°≤θ≤70° in the real situation.Figure 7
Image of a checkerboard used in the experiments.


## 4.3. Experiment Using Images

The angle between the directions of the two parallel lines is selected according to the influence of the angle on the calibration results in the simulation experiment, and the angle is20°≤θ≤70° in the real experiment. In general, the noise in a real environment is unknown; random Gaussian noise is the main source. Gaussian noise was used to simulate the real noise in the simulation experiment. Figure 5 shows that the proposed method has robustness in the presence of noise. Next, we used real images to verify the accuracy and applicability of the methods in this study. In the real experiment, a checkerboard was used as a calibration model to replace the line. We used a central catadioptric camera with a hyperboloidal mirror having parameter ξ=0.966, designed by the Center for Machine Perception at Czech Technical University. Five pictures of the checkerboard were taken using the catadioptric camera with its position changed. One of the five images is shown in Figure 7. The resolutions of these images were 1320×1170 pixels. The equations of the line images were obtained using the method described in Section 3.3.The intrinsic parameters of the camera were obtained with the two calibration methods proposed in this study in addition to Deng et al. [11], Geyer and Daniilidis [18], and Li and Zhao’s [22] calibration methods. Table 2 lists the calibration results on real data.Table 2
Calibration results with real data (unit: pixels).
Calibration methodsCamera’s intrinsic parametersfufvu0v0sMethod 11076.44291072.85371.0815650.7238585.5324Method 21084.75011081.50821.0763648.9626587.7246Geyer [18]1093.07191089.80251.1583653.6927586.8968Deng [11]1086.25771082.79281.2128651.4500587.7956Li-1 [22]1076.44291472.85371.0584650.7238585.5324Li-2 [22]1094.75011489.60820.9763644.9626589.7246Li-3 [22]1062.87431441.34501.5256648.0849580.4329To check the calibration results in Table2, the test image in Figure 7 was rectified to its projective image, as shown in Figures 8(a)–8(g).Figure 8
Rectified images based on the intrinsic parameters in Table2. Rectified image using (a) Method 1, (b) Method 2, (c) Geyer and Daniilidis [18], (d) Deng et al. [11], (e) Li-1 (Proposition 6), (f) Li-2 (Proposition 7), and (g) Li-3 (Proposition 8) from Li and Zhao [22].
(a)(b)(c)(d)(e)(f)(g)The results of all methods in Table2 were similar to each other, indicating that the proposed methods are feasible. The straight line in the space is changed into a curve due to lens distortion in the original image shown in Figure 7. After using the data in Table 2 to rectify the image, the straight line in the space is still straight in the images shown in Figures 8(a)–8(g). The experiments showed that the proposed calibration methods in this study are effective.To further verify the accuracy of the method, the data in Table2 was used to rectify the image shown in Figure 7, and the checkerboard reconstruction results were extracted. The reconstruction information in parallel and orthogonal directions of the checkerboard is shown in Figures 9(a)–9(g). First, we extracted the points on each row and each column to fit the equation of the line. Next, we calculated the angles between two lines in all parallel directions and obtained their average value. Similarly, the average value of angles between all orthogonal lines could also be obtained. Table 3 lists the angle results with real data as shown in Figures 9(a)–9(g).Figure 9
Reconstruction information of the checkerboard based on the intrinsic parameters in Table2. Reconstructed using (a) Method 1, (b) Method 2, (c) Geyer and Daniilidis [18], (d) Deng et al. [11], (e) Li-1 (Proposition 6), (f) Li-2 (Proposition 7), and (g) Li-3 (Proposition 8) from Li and Zhao [22].
(a)(b)(c)(d)(e)(f)(g)Table 3
Verifying the parallelity and orthogonality of 3D reconstruction result with real data after rectification (unit: degree).
Method 1Method 2Geyer [18]Deng [11]Li-1 [22]Li-2 [22]Li-3 [22]Parallel0.62330.60230.72660.62510.62630.71320.6840Orthogonal89.377289.308289.126589.254789.193589.170089.1506The ground truth data were 0° for the parallel lines and 90° in the orthogonal direction on the checkerboard, as already known. Using our methods in the reconstruction results, the angle between the parallel and orthogonal directions is closer to the ground truth, which proves the accuracy of the method in this study.


## 5. Conclusions

The main contribution of this study to the field of omnidirectional camera calibration is the use of the projection properties of parallel lines in a unit viewing sphere model to calibrate a central catadioptric camera. A line in 3D space is projected to a great circle on the unit viewing sphere, where its center is consistent with that of the unit viewing sphere. A group of parallel lines is projected to a group of big circles with only two points of intersection: the antipodal points. In Method 1, two groups of parallel lines in different directions are projected onto two groups of big circles with four points of intersection on the unit viewing sphere. According to the definition of antipodal points, the quadrilateral with these points as vertices is a rectangle, and the orthogonal directions can be obtained by the four points of intersection. In Method 2, the tangent at a point on a circle is orthogonal to the diameter passing through that point on the circle, using which the orthogonal vanishing points for the support plane containing each great circle can be determined on the image plane. Finally, the intrinsic parameters of the camera can be linearly determined by five groups of orthogonal vanishing points. This indicates that five images are required and all five intrinsic parameters are recovered without any assumptions, such as zero skews=0 or a unitary aspect ratio fu=fv.The two methods proposed in this study provide new insights into the calibration of central catadioptric cameras, especially from the aspect of constraints provided by the projection properties of parallel lines. Two linear calibration approaches using parallel lines are derived from its projection properties. Experimental results on both simulated data and real image data validated the effectiveness of our methods and demonstrated that the two linear calibration methods maintain comparable accuracy. As only a section of the conic projected by a line in the central catadioptric projection is visible, according to the relationship between the antipodal points, an optimization algorithm for line image fitting to improve the accuracy of calibration was proposed in this study. The simulated results show that the proposed optimization algorithm can improve the precision of a fitting line image in center catadioptric cameras. Although the algorithm proposed in this study improves the calibration accuracy to some extent, a more thorough investigation may be required. The shortcoming of the method used in this study is that the projection contour of the mirror for the center catadioptric camera is required to obtain images of antipodal points. In our future research, the method of directly obtaining the images of antipodal points from the projective properties of geometric elements will be considered.



---

*Source: 6675110-2021-07-02.xml*