# Review of Calibration Methods for Scheimpflug Camera

Authors: Cong Sun; Haibo Liu; Mengna Jia; Shengyi Chen
Journal: Journal of Sensors
 (2018)

---

## Abstract

The Scheimpflug camera offers a wide range of applications in the field of typical close-range photogrammetry, particle image velocity, and digital image correlation due to the fact that the depth-of-view of Scheimpflug camera can be greatly extended according to the Scheimpflug condition. Yet, the conventional calibration methods are not applicable in this case because the assumptions used by classical calibration methodologies are not valid anymore for cameras undergoing Scheimpflug condition. Therefore, various methods have been investigated to solve the problem over the last few years. However, no comprehensive review exists that provides an insight into recent calibration methods of Scheimpflug cameras. This paper presents a survey of recent calibration methods of Scheimpflug cameras with perspective lens, including the general nonparametric imaging model, and analyzes in detail the advantages and drawbacks of the mainstream calibration models with respect to each other. Real data experiments including calibrations, reconstructions, and measurements are performed to assess the performance of the models. The results reveal that the accuracies of the RMM, PLVM, PCIM, and GNIM are basically equal, while the accuracy of GNIM is slightly lower compared with the other three parametric models. Moreover, the experimental results reveal that the parameters of the tangential distortion are likely coupled with the tilt angle of the sensor in Scheimpflug calibration models. The work of this paper lays the foundation of further research of Scheimpflug cameras.


---

## Body

## 1. Introduction

The Scheimpflug camera, adopting the Scheimpflug condition by tilting the lens with respect to the sensor [1, 2], can optimize the distribution of depth of field (DOF) without zooming the aperture, which has been widely used in the field of PIV [3–5], line structured light [6], and portable 3D laser scanner [7, 8]. At the same time, it has also been introduced into the medical field, greatly facilitating cataract and other ophthalmic surgery [9, 10]. So far, more and more scholars have also devoted themselves to the related research [6, 7, 11–18].The Scheimpflug principle, traditionally credited to Theodor Scheimpflug in 1902, states that the object plane (the plane that is in focus), the thin lens’s plane, and the image plane must all meet in a single line, Scheimpflug line. The principle is applicable to both thin prism and thick prism models, which only needs to be modified accordingly [1, 2]. As for the case of a real optical system, it is likely to find that for different object points, the optical system have different f-numbers and different aberrations; thus, the image generated by the real optical system will not be completely sharp [7]. Yet, this influence usually can be ignored in the Scheimpflug camera research.Scheimpflug camera calibration is a necessary preliminary step to ensure its further high-quality measurement. The conventional calibration methods are not valid in this case because the assumptions used by classical calibration methodologies are not valid anymore for cameras undergoing Scheimpflug condition [19–22]. Although Legarda et al. [23], Nocerino et al. [16], and Cornic et al. [17] point out that when the lens-tilted angle is small (≤6°), the existing calibration methods can compensate, to a certain extent, the error generated by tilting the sensor via tangential distortion parameters to solve the calibration of Scheimpflug cameras. However, when the tilted angle is greater, the conventional camera calibration methods are weak.Therefore, more and more researchers have carried out related researches [14, 17, 18, 23–33]. Up to now, a variety of methods has been proposed to accommodate various applications. Generally, these calibration methods can be classified into parametric (defined by several intrinsic parameters) [14, 17, 18, 23–33] and general nonparametric calibration methods according to the imaging model [33–35]. In fact, the kernels of the parametric methods are basically identical, which firstly develop an ideal image plane parallel to the lens plane as a bridge, then the different form of relationships between the ideal and actual image plane can be established according to different models. As the projection between the ideal image plane and calibration checkerboard can be easily obtained via conventional imaging model, the projection between the tilted image plane and calibration checkerboard is established, on the basis of which the calibration algorithm is developed [14, 17, 18, 23–31].Generally, the parametric calibration methods of Scheimpflug camera fall into two categories according to the dimension of the tilted angle, the literature [11, 24, 25] as the representative of the one-dimensional angle calibration model and the literature [13–15, 17, 18, 23, 27, 28, 30, 31] as the two-dimensional angle calibration model. Further, as the lens is supposed to be planar and symmetric about optical axis, two-angle parameters should be sufficient to express the homography between the tilted image plane and ideal image plane [23, 27].Furthermore, the parametric calibration methods with two-dimensional angles are supposed to be divided into two main categories: (1) modified pinhole imaging model (MPIM), as the name implies, is developed on the basis of the conventional pinhole imaging model and taking the imaging characteristics of Scheimpflug camera into account, see literature [14, 15, 17, 23, 27, 28, 30, 31] to cite a few. (2) Pupil centric imaging model (PCIM) modeled the different ray angles in object and image space, and the actual projection center are the center of the exit and entrance pupils. There is a new mapping from pupil centric imaging to a geometrically equivalent pinhole imaging; thus, the Scheimpflug camera calibration can be carried out in a conventional framework, see literature [13, 18, 29, 36] to cite a few.In general, the mainstream parametric camera calibration algorithms can be attributed to the nonlinear parameter optimization problem and the appropriate initialization is the key to the fast convergence and global minimum. The initialization acquisition methods can be classified into the three categories: (1) taking the nominal parameters of the camera as the initial values of the optimization algorithm [14, 25], (2) adopting the results of Zhang et al.’s calibration algorithm as initial values [17, 23, 27, 30], (3) obtaining initialization intrinsic parameters of the Scheimpflug camera by means of auxiliary tools, for example, adopts a collimator to obtain the principal point [31], and (4) initializing by complex analytic solution, as illustrated in literature [13, 29, 32, 33].While perspective projection serves as the dominant imaging model in nowadays’ computer vision, conventional camera calibration techniques are taylor made for specific camera model which may not suffice for an unknown imaging system (a black box). Thus, the general imaging models accommodating a wide range of devices have been proposed [33–35]. The mentality of the general imaging model is that all types of imaging systems perform a mapping from the scene rays to a set of associated image pixels and the image pixels measure light traveling along the associated half-rays from the scene in various directions. As illustrated in literature [33–35], the calibration undergoing general imaging models simply refers to the computation of the projection between the pixels and associated 3D rays.Aiming to offer a comprehensive review that provides an insight into recent calibration methods of Scheimpflug cameras with perspective lens, this paper presents a survey of recent calibration methods of Scheimpflug cameras and analyzes in detail the advantages and drawbacks of the mainstream calibration models with respect to each other. Besides, the general nonparametric imaging model is novelly introduced to solve the problem. Furthermore, Real data experiments are performed to validate the performance of different calibration models which therefore lays the foundation of further research of the Scheimpflug cameras.


## 2. Modified Pinhole Imaging Model

Modified pinhole imaging model [14, 17, 23, 26–31, 37], as the name implies, is developed on the basis of the conventional pinhole imaging model, and the imaging characteristics of Scheimpflug camera is taken into account. Generally, the projection center of MPIM is supposed to be the optical center.According to the different ways of describing the tilt effect of the lens, the MPIM can be simply divided into three categories: (1) extended distortion model (EDM), which extends the pinhole imaging model by considering the inclination of the sensor as an additional distortion [14, 26, 29]; (2) rotation matrix model (RMM), which models the lens-sensor configuration by an explicit rotation matrix about the optic axis and includes it as a part of intrinsic calibration parameter set [17, 30, 31, 38]; (3) and point-line vector model (PLVM), in which the transformation between the tilted and ideal image plane is established by the intersection of the light ray, determined by the optical center, the spatial point, and two image planes [23, 27, 28].


### 2.1. Extended Distortion Model (EDM)

Extended distortion model considers the inclination of the sensor as an additional distortion and develops a Scheimpflug camera calibration model based on the classical pinhole imaging model [14, 26]. As the work of Wang et al. [26] is only useful for very small tilt angles, Peng et al. [14] extend their studies to the condition with greater tilt angles based on the geometric optics theory. As illustrated in Figure 1, OXCYCZC is the camera coordinate frame with the origin at the aperture stop. ZC is the optical axis of the lens, and I0 and IS denote the ideal and real tilted image plane, respectively. The symbols α and β indicate the angles between the image plane IS and the x-axis and y-axis, respectively, while symbol d represents the distance from the origin OC to the image plane I0. The 3D world point PW(XW,YW,ZW) projects on the I0 at point pi, and the light ray intersects the IS at point pS.Figure 1
Schematic of imaging system applied with Scheimpflug condition.The section views of the camera geometry when the image plane, respectively, tiltsα with respect to the x- and y-axis are illustrated in Figure 2, where A(x,y) is an arbitrary point on I0 while point B is the back projection of A on the IS. The symbols θx and θy, respectively, denote the projection angle of the angle between the light ray and the optical axis.Figure 2
Projection view of Scheimpflug camera lens [14]. (a) View from the direction of x-axis; (b) View from the direction of y-axis.According to the derivation of the literature [14], the increment (∆xα,∆yα) and (∆xβ,∆yβ) along the x-axis or y-axis caused by rotating the angles α and β can be expressed as follows,
(1)Δxα=−xydtanα,Δyα=−y2dtanα,Δxβ=x2dtanβ,Δyβ=xydtanβ.Then, on the basis of conventional distortion model [39], the extended distortion model can be obtained as follows,
(2)Δx=xk1r2+k2r4+xtanβdx−tanαdy,Δy=yk1r2+k2r4+ytanβdx−tanαdy,where (∆x,∆y) are the total distortions caused by the Scheimpflug camera, (xn,yn)T are the normalized image coordinates and r2=xn2+yn2. (k1,k2) are the radial distortion coefficients. And it deserves noting that the tangential distortion coefficients are ignored here, as in many of the Scheimpflug camera calibration methods [11, 13, 23, 25, 27–29, 40].The tangential distortion mainly consists of decentering distortion and thin prism distortion. The thin prism distortion usually results from the defects of lens design and manufacture and the slight tilt of sensor plane [26]. It also implicitly confirms the conclusions in literature [16, 17, 23] that the conventional calibration methods can compensate the error generated by tilting the sensor via tangential distortion parameters, to a certain extent.In conclusion, the calibration method based on EDM is logically explicit, and the model is simple and easy to operate which can completely inherit the existing calibration process of the standard camera [21]. However, in spite of the fact that Peng et al. [14] have extended the model to suit larger tilt angles, the range of application of the model is still limited within a relatively small tilt angle. Also, the model described in this section is mainly based on the assumptions of the telecentric lens [14], which cannot be applied directly to the standard perspective lens condition.Moreover, the model illustrated here requires that the length of the lens should be provided in advanced [14], which in practice is not easy to obtain and the nominal parameters of the lens might not be accurate enough. Thus, the method needs to be further improved.


### 2.2. Rotation Matrix Model (RMM)

Rotation matrix model models the lens-sensor configuration by an explicit rotation matrix about the optic axis and includes it as a part of intrinsic calibration parameter set [17, 30, 31, 38]. One of the most representative methods is that Kumar and Ahuja [29] introduce the rotation transformation between the lens and image plane and extends the RAC (radial alignment constraint) [19] to accommodate the Scheimpflug camera, which is defined as the gRAC (generalized radial alignment constraint).As illustrated in Figure3(a), OWXWYWZW, OLXLYLZL, and OSXSYSZS denote the world, lens, and imaging plane coordinate frames, respectively. The lens coordinate frame is assumed to be parallel to the image plane coordinate frame with a common Z-axis. OS is the principal point, and Pu(xu,yu), and Pd(xd,yd) indicate the ideal and the distorted image points of the 3D spatial point PW, respectively. The RAC [29] points out that, assuming only radial lens distortion exists, the position vector OSPd→ determined by the ideal image point and the distorted image point should be aligned with the position vector POZPW→, which is the vertical line from the point PW to the optical axis, and there is a relationship between two position vectors as OSPd→×POZPW→=0.Figure 3
Schematic of imaging system applied with Scheimpflug condition [29]. (a) Standard camera; (b) Scheimpflug camera.
(a)
(b)Nevertheless, as shown in Figure3(b), the triplet OS, Pu, and Pd might not be collinear due to lens misalignment or intentional sensor tilt, which makes RAC difficult to be directly applied to Scheimpflug condition [29]. Thus, a more generic imaging model is proposed where the nonalignment of lens and image plane is explicitly modeled via a rotation matrix.As illustrated in Figure4, Kumar and Ahuja [29] derive a rotation matrix R(ρ,σ) which aligns the coordinate frame of the image plane with the lens’s by two dimensional angles (ρ,σ) indicating clockwise rotation about its x- and y-axis, respectively. Consider the coordinate frames described in Figure 3, the 3D world point PW is projected on the image plane at point PS, and the ray of light intersects the ideal image plane at point PN. Let the rotation matrix as
(3)Rρ,σ=cosσsinρsinσcosρsinσ0cosρ−sinρ−sinσsinρcosσcosρcosσ=r11r12r13r21r22r23r31r32r33,wherein rij represents the ith row and jth column entry of R. Given that the relative rotation between two planes is known, the relationship between image points in tilted image plane and ideal image plane can be obtained according to the derivation in literature [29]:
(4)xNyN=−r11xS+r21ySλr13xS+r23yS−λ−r12xS+r22ySλr13xS+r23yS−λ,where (xN,yN)T and (xS,yS)T denote the image point in ideal image plane and tilted image plane, respectively, and λ indicates the distance between the lens and ideal image plane.Figure 4
Schematic of imaging system applied with Scheimpflug condition [29].The world pointPW is projected on the optical axis with the coordinate POZ=0,0,Zl. And it is supposed that the projected ideal image point PN aligns with the world point PW, which means the location vectors OPPN→ and POZPW→ should be coplanar and also parallel to each other. Thus, we can obtain the radial alignment constraint as follows:
(5)OPPN→×POZPW→=0.The gRAC with the rotation matrix can be established by combining formulas (4) and (5), on the basis of which a two-step calibration algorithm is developed. The algorithm proposes a new analytical solution to solve the gRAC for a subset of calibration parameters, followed by the nonlinear refinement of calibration parameters [29].As far as the paper is concerned, the calibration method based on RMM is flexible, accurate, and robust, which is also easy to operate and initialize. Inevitably, there are still some drawbacks of the method. The analytical solution, as proposed in literature [29], is tedious and involving more sign ambiguous estimation, and the computational efficiency of the calibration algorithm should make further efforts.


### 2.3. Point-Line Vector Model (PLVM)

Point-line vector model shows that the transformation between the tilted image plane and ideal image plane is established on the basis of the intersections of the light ray and two image planes (ideal image plane and real tilted image plane) (see literature [23, 27, 28] to cite a few).As illustrated in Figure5, the symbol O represents the optical center and OXCYCZC is the coordinate frame of the Scheimpflug camera. The image planes S and P denote the tilted image plane and ideal image plane, respectively. The origins of the ideal and real tilted image planes are assumed to be same, as the ideal image plane can be arbitrarily placed [23, 27].Figure 5
Schematic of imaging system applied with Scheimpflug condition.The symbolr indicates a straight ray which goes from the 3D world point PW(XW,YW,ZW) to the optical center O. r intersect with the tilted image plane S and the ideal image plane P at the points PS and PP, respectively. (m,n) are the base vectors in the plane S, and the vectors m and n, respectively, lie in the planes XOZ and YOZ. Further, according to the definition in the literature [23], m = (cosβ, 0, sinβ) and the vector n is perpendicular to the m and has an angle with the Y-axis; hence, the vector n = (−sinαsinβ, cosα, sinαcosβ).The formula of the line r is
(6)r:λPW=λXW,YW,ZWT.And the symbolAT denotes the transpose of the vector A. The following transposed vectors are noted the same way. And λ here is an arbitrary nonzero scale factor. The intersections of r and two image planes can be expressed as
(7)PP=PPx,PPy,fT,PS=0,0,f′+PSxm+PSyn.Substituting formula (7) into (6) yields
(8)PPx=λPSxcosβ−PSysinαcosβ,PPy=λPSycosα,wherein λ=f/PPxsinβ+PPysinαcosβ+f, assuming that the (CPx,CPy) and (CSx,CSy) represent the principal points in planes P and S, respectively, and (dx,dy) denote the pixel sizes on the image plane while (Fx,Fy) are the equivalent focal length in pixel units. Then, the formula (6) can be converted to (9) as
(9)uv1T=λfKRSKS−1uSvS1T,where (u,v) and (uS,vS), respectively, denote the image points on planes P and S and
(10)K=fdx0CPx0fdyCPy001,KS=1dx0CSx01dyCSy001,RS=cosβ−sinαsinβ00cosα0sinβsinαcosβf.Without a loss of generality, assuming that the calibration target plane is onZ ≡ 0 of the world coordinate frame, we have
(11)uv1T=sKr1r2TXY1T,where s is an arbitrary scale factor, (R,T) represent the rotation and transformation which relate the world coordinate frame to the ideal image plane coordinate frame, and r1 and r2 are the first two columns of the rotation matrix R. The homography between the tilted image plane and the calibration target plane can be obtained by combining the formulas (9) and (11).To summarise, the calibration method based on PLVM has the advantage of a robust and simple model, as well as convenient calibration process. However, the application of the model in literature [27] still strongly depends on the quality of the calibration target. In addition, the initial estimation of the calibration parameters is obtained using Zhang’s method [21] directly rather than taking the unique imaging characteristics of Scheimpflug camera into account.


## 2.1. Extended Distortion Model (EDM)

Extended distortion model considers the inclination of the sensor as an additional distortion and develops a Scheimpflug camera calibration model based on the classical pinhole imaging model [14, 26]. As the work of Wang et al. [26] is only useful for very small tilt angles, Peng et al. [14] extend their studies to the condition with greater tilt angles based on the geometric optics theory. As illustrated in Figure 1, OXCYCZC is the camera coordinate frame with the origin at the aperture stop. ZC is the optical axis of the lens, and I0 and IS denote the ideal and real tilted image plane, respectively. The symbols α and β indicate the angles between the image plane IS and the x-axis and y-axis, respectively, while symbol d represents the distance from the origin OC to the image plane I0. The 3D world point PW(XW,YW,ZW) projects on the I0 at point pi, and the light ray intersects the IS at point pS.Figure 1
Schematic of imaging system applied with Scheimpflug condition.The section views of the camera geometry when the image plane, respectively, tiltsα with respect to the x- and y-axis are illustrated in Figure 2, where A(x,y) is an arbitrary point on I0 while point B is the back projection of A on the IS. The symbols θx and θy, respectively, denote the projection angle of the angle between the light ray and the optical axis.Figure 2
Projection view of Scheimpflug camera lens [14]. (a) View from the direction of x-axis; (b) View from the direction of y-axis.According to the derivation of the literature [14], the increment (∆xα,∆yα) and (∆xβ,∆yβ) along the x-axis or y-axis caused by rotating the angles α and β can be expressed as follows,
(1)Δxα=−xydtanα,Δyα=−y2dtanα,Δxβ=x2dtanβ,Δyβ=xydtanβ.Then, on the basis of conventional distortion model [39], the extended distortion model can be obtained as follows,
(2)Δx=xk1r2+k2r4+xtanβdx−tanαdy,Δy=yk1r2+k2r4+ytanβdx−tanαdy,where (∆x,∆y) are the total distortions caused by the Scheimpflug camera, (xn,yn)T are the normalized image coordinates and r2=xn2+yn2. (k1,k2) are the radial distortion coefficients. And it deserves noting that the tangential distortion coefficients are ignored here, as in many of the Scheimpflug camera calibration methods [11, 13, 23, 25, 27–29, 40].The tangential distortion mainly consists of decentering distortion and thin prism distortion. The thin prism distortion usually results from the defects of lens design and manufacture and the slight tilt of sensor plane [26]. It also implicitly confirms the conclusions in literature [16, 17, 23] that the conventional calibration methods can compensate the error generated by tilting the sensor via tangential distortion parameters, to a certain extent.In conclusion, the calibration method based on EDM is logically explicit, and the model is simple and easy to operate which can completely inherit the existing calibration process of the standard camera [21]. However, in spite of the fact that Peng et al. [14] have extended the model to suit larger tilt angles, the range of application of the model is still limited within a relatively small tilt angle. Also, the model described in this section is mainly based on the assumptions of the telecentric lens [14], which cannot be applied directly to the standard perspective lens condition.Moreover, the model illustrated here requires that the length of the lens should be provided in advanced [14], which in practice is not easy to obtain and the nominal parameters of the lens might not be accurate enough. Thus, the method needs to be further improved.


## 2.2. Rotation Matrix Model (RMM)

Rotation matrix model models the lens-sensor configuration by an explicit rotation matrix about the optic axis and includes it as a part of intrinsic calibration parameter set [17, 30, 31, 38]. One of the most representative methods is that Kumar and Ahuja [29] introduce the rotation transformation between the lens and image plane and extends the RAC (radial alignment constraint) [19] to accommodate the Scheimpflug camera, which is defined as the gRAC (generalized radial alignment constraint).As illustrated in Figure3(a), OWXWYWZW, OLXLYLZL, and OSXSYSZS denote the world, lens, and imaging plane coordinate frames, respectively. The lens coordinate frame is assumed to be parallel to the image plane coordinate frame with a common Z-axis. OS is the principal point, and Pu(xu,yu), and Pd(xd,yd) indicate the ideal and the distorted image points of the 3D spatial point PW, respectively. The RAC [29] points out that, assuming only radial lens distortion exists, the position vector OSPd→ determined by the ideal image point and the distorted image point should be aligned with the position vector POZPW→, which is the vertical line from the point PW to the optical axis, and there is a relationship between two position vectors as OSPd→×POZPW→=0.Figure 3
Schematic of imaging system applied with Scheimpflug condition [29]. (a) Standard camera; (b) Scheimpflug camera.
(a)
(b)Nevertheless, as shown in Figure3(b), the triplet OS, Pu, and Pd might not be collinear due to lens misalignment or intentional sensor tilt, which makes RAC difficult to be directly applied to Scheimpflug condition [29]. Thus, a more generic imaging model is proposed where the nonalignment of lens and image plane is explicitly modeled via a rotation matrix.As illustrated in Figure4, Kumar and Ahuja [29] derive a rotation matrix R(ρ,σ) which aligns the coordinate frame of the image plane with the lens’s by two dimensional angles (ρ,σ) indicating clockwise rotation about its x- and y-axis, respectively. Consider the coordinate frames described in Figure 3, the 3D world point PW is projected on the image plane at point PS, and the ray of light intersects the ideal image plane at point PN. Let the rotation matrix as
(3)Rρ,σ=cosσsinρsinσcosρsinσ0cosρ−sinρ−sinσsinρcosσcosρcosσ=r11r12r13r21r22r23r31r32r33,wherein rij represents the ith row and jth column entry of R. Given that the relative rotation between two planes is known, the relationship between image points in tilted image plane and ideal image plane can be obtained according to the derivation in literature [29]:
(4)xNyN=−r11xS+r21ySλr13xS+r23yS−λ−r12xS+r22ySλr13xS+r23yS−λ,where (xN,yN)T and (xS,yS)T denote the image point in ideal image plane and tilted image plane, respectively, and λ indicates the distance between the lens and ideal image plane.Figure 4
Schematic of imaging system applied with Scheimpflug condition [29].The world pointPW is projected on the optical axis with the coordinate POZ=0,0,Zl. And it is supposed that the projected ideal image point PN aligns with the world point PW, which means the location vectors OPPN→ and POZPW→ should be coplanar and also parallel to each other. Thus, we can obtain the radial alignment constraint as follows:
(5)OPPN→×POZPW→=0.The gRAC with the rotation matrix can be established by combining formulas (4) and (5), on the basis of which a two-step calibration algorithm is developed. The algorithm proposes a new analytical solution to solve the gRAC for a subset of calibration parameters, followed by the nonlinear refinement of calibration parameters [29].As far as the paper is concerned, the calibration method based on RMM is flexible, accurate, and robust, which is also easy to operate and initialize. Inevitably, there are still some drawbacks of the method. The analytical solution, as proposed in literature [29], is tedious and involving more sign ambiguous estimation, and the computational efficiency of the calibration algorithm should make further efforts.


## 2.3. Point-Line Vector Model (PLVM)

Point-line vector model shows that the transformation between the tilted image plane and ideal image plane is established on the basis of the intersections of the light ray and two image planes (ideal image plane and real tilted image plane) (see literature [23, 27, 28] to cite a few).As illustrated in Figure5, the symbol O represents the optical center and OXCYCZC is the coordinate frame of the Scheimpflug camera. The image planes S and P denote the tilted image plane and ideal image plane, respectively. The origins of the ideal and real tilted image planes are assumed to be same, as the ideal image plane can be arbitrarily placed [23, 27].Figure 5
Schematic of imaging system applied with Scheimpflug condition.The symbolr indicates a straight ray which goes from the 3D world point PW(XW,YW,ZW) to the optical center O. r intersect with the tilted image plane S and the ideal image plane P at the points PS and PP, respectively. (m,n) are the base vectors in the plane S, and the vectors m and n, respectively, lie in the planes XOZ and YOZ. Further, according to the definition in the literature [23], m = (cosβ, 0, sinβ) and the vector n is perpendicular to the m and has an angle with the Y-axis; hence, the vector n = (−sinαsinβ, cosα, sinαcosβ).The formula of the line r is
(6)r:λPW=λXW,YW,ZWT.And the symbolAT denotes the transpose of the vector A. The following transposed vectors are noted the same way. And λ here is an arbitrary nonzero scale factor. The intersections of r and two image planes can be expressed as
(7)PP=PPx,PPy,fT,PS=0,0,f′+PSxm+PSyn.Substituting formula (7) into (6) yields
(8)PPx=λPSxcosβ−PSysinαcosβ,PPy=λPSycosα,wherein λ=f/PPxsinβ+PPysinαcosβ+f, assuming that the (CPx,CPy) and (CSx,CSy) represent the principal points in planes P and S, respectively, and (dx,dy) denote the pixel sizes on the image plane while (Fx,Fy) are the equivalent focal length in pixel units. Then, the formula (6) can be converted to (9) as
(9)uv1T=λfKRSKS−1uSvS1T,where (u,v) and (uS,vS), respectively, denote the image points on planes P and S and
(10)K=fdx0CPx0fdyCPy001,KS=1dx0CSx01dyCSy001,RS=cosβ−sinαsinβ00cosα0sinβsinαcosβf.Without a loss of generality, assuming that the calibration target plane is onZ ≡ 0 of the world coordinate frame, we have
(11)uv1T=sKr1r2TXY1T,where s is an arbitrary scale factor, (R,T) represent the rotation and transformation which relate the world coordinate frame to the ideal image plane coordinate frame, and r1 and r2 are the first two columns of the rotation matrix R. The homography between the tilted image plane and the calibration target plane can be obtained by combining the formulas (9) and (11).To summarise, the calibration method based on PLVM has the advantage of a robust and simple model, as well as convenient calibration process. However, the application of the model in literature [27] still strongly depends on the quality of the calibration target. In addition, the initial estimation of the calibration parameters is obtained using Zhang’s method [21] directly rather than taking the unique imaging characteristics of Scheimpflug camera into account.


## 3. Pupil Centric Imaging Model (PCIM)

The imaging models as described above assume that the angle of the chief ray in object space and the angle of the chief ray in image space are identical, which is incorrect. And this difference employs a significant influence on the Scheimpflug camera, which in general can be ignored in pinhole imaging model. So Kumar and Ahuja [13] propose a generalized PCIM which models the exact relationship between these rays. Besides, Steger [18] reduces the number of the model parameters and develops a simpler PCIM.Figure6 shows a classical geometry of a thick lens, a ray that enters the principal plane P and exits the principal plane P′ at the same position with respect to the optical axis (however, generally not at the same angle) [18]. The ray geometry of the real Scheimpflug camera model is illustrated in Figure 7.Figure 6
The geometry of standard thick lens [18].Figure 7
The ray geometry of the real Scheimpflug camera model [18].First, the object points are projected to the ideal image plane lying at a distancec from the projection center O, where the ray angles in object space and image space are identical ω = ω″. Then, move the ideal image plane to a distance d, resulting in the difference of angles ω ≠ ω′. Next, the real tilted image plane can be obtained by tilting the ideal image plane at the correct distance d by the angle τ.Consider the definition of coordinate frames in literature [18], as shown in Figure 8. Oxuyuzu and Oxtytzt denote the coordinate frame of ideal image plane and tilted image plane, respectively. n is the rotation axis in a plane orthogonal to the optical axis which forms an angle ρ with axis xu. And a new coordinate frame OxSySzS is constructed in the tilted image plane parallel to coordinate frame Oxtytzt which lies at the perpendicular projection of the projection center. The coordinate frames of Oxuyuzu and OxSySzS can be considered as two cameras that are rotated around their common projection center O. Hence, the transformation from Oxuyuzu to OxSySzS can be given by KSRKu−1, in which KS and Ku are the calibration matrices corresponding to two cameras, respectively, and R denotes the rotation matrix that relates to the two cameras [18]. It is apparent that there is a relationship between the OxSySzS and Oxtytzt defined by a translation matrix T. Thus, the complete projection can be expressed as follows:
(12)HP=TKSRKu−1,where
(13)Ku=d000d0001,KS=dr33000dr330001,T=10−dr3101−dr32001,and rij indicates the ith row and jth column entry of R defined as
(14)Rt=r11r12r13r21r22r23r31r32r33=cρ21−cτcρsρ1−cτsρsτcρsρ1−cτsρ21−cτ+cτ−cρsτ−sρsτcρsτcτ,where cθ = cosθ, sθ = sinθ, and τ(0 ≤ τ < π/2). Thus, the projection relation between coordinate frame Oxuyuzu and Oxtytzt can be expressed as
(15)HP=r11r33−r13r31r21r33−r23r310r12r33−r13r32r22r33−r23r320r13dr23dr33.Figure 8
The projection model of the Scheimpflug camera [18].Furthermore, the projection from the world coordinate frame to theOxuyuzu can be easily obtained via perspective projection, so that the complete projection from the world frame to Oxtytzt is established.In comparison with the aforementioned three calibration models, PCIM proposed in literature [13, 18] gives full consideration to the difference of the ray angles in object space and image space, which is more closely related to the actual imaging. In addition, the model allows the lens to be tilted in an arbitrary direction which is much more flexible and has more explicit physical meaning than the above three. The calibration method base on the PCIM in literature [18] is accurate, easy to initialize, and as well as fast convergence. However, the model also inevitably has some imperfections that the PCIM cannot be directly applied to the situation where the tilt is in the horizontal or vertical direction, which requires the pixel aspect ratio to be known in advance [18].


## 4. General Nonparametric Imaging Model (GNIM)

Great progress has been made during last decade in exploiting the general nonparametric imaging model for camera calibration [32, 33, 35, 41–43]. In comparison with the perspective imaging model, the GNIM is rather flexible and accommodates a wide range of imaging devices. Moreover, as has been demonstrated, given that the distortion is quite high, the GNIM may offer better calibration results than the parametric methods [44].As proposed in literature [33], Ramalingam and Sturm model any camera as a set of image pixels and their associated camera rays in space, and the image pixels measure light traveling along the associated half-rays from the scene in various directions. Therefore, the calibration undergoing GNIM simply refers to the computation of the projection between the pixels and their associated 3D rays. The mentality of GNIM calibration is quite simple, as illustrated in Figure 9, and three (or more) different points Pi1,Pi2,Pi3 corresponding to the same single pixel pi(u,v) on the image plane, as sampled on three (or more) different calibration checkerboards with unknown relative position, must be collinear. Hence, the relative position among the calibration checkerboards can be obtained on the basis of the collinearity constraint of the checkerboard points. Then, the calibration can be completed by computing rays going through the associated sampled checkerboard points.Figure 9
Schematic of general nonparametric imaging model.We consider here a Scheimpflug camera with perspective lens, to validate the performance of GNIM in the Scheimpflug camera calibration. As shown in Figure9, let us denote several points on different planar calibration checkerboards corresponding to the same image pixel by homogeneous coordinates PikXik,Yik,0,Wik, for checkerboards k=1,2,…,n. The unknown poses with respect to different girds can be expressed as rotation matrices RK and translation vectors TK. Thus, the point Pik relative to the corresponding local coordinate frame can be projected to the global reference frame via
(16)RkTk0T1Xik,Yik,0,Wik.Furthermore, the optical center of the camera is defined asO(O1, O2, O3, 1). And without the loss of generality, the first calibration checkerboard is adopted as the global coordinate reference frame.As the collinearity constraint proposed in literature [32, 33], the checkerboard points corresponding to the one single pixel, after transforming into the global reference coordinate frame via formula (16), must be collinear. Thus, the 4×n+1 matrix including the coordinates of collinear checkerboard points as follows should be of rank smaller than 3.
(17)O1X11R112X12+R122Y12+T12W42⋯O2Y21R212X12+R222Y12+T22W42⋯O30R312X12+R322Y12+T32W42⋯1W41W42⋯,where Rij indicates the ith row and jth column entry of R. Therefore, the determinants of all the 3 × 3 submatrix of formula (17) must vanish. Hence, a new submatrix is constructed by selecting the first column and any other two columns, and the bilinear equations in accordance with checkerboard points Pij and Pik can be obtained as follows’
(18)PijTMjkPik=0,where Mjk represents the bifocal matching tensor with respect to the specific 3×3 submatrix associated with the optical center and checkerboard pose. Referring to the approach proposed in literature [32, 33], the optical center and checkerboards’ poses can be determined with at least three different views. Then, we must fit the camera rays to the single optical center O. Let Λj denotes the direction of the camera ray passing through the checkerboard point Pij and λi represents the parameter corresponding to the closest point on the camera ray to the given checkerboard point Pij. Thus, the vector Λj can be calculated by minimizing the following cost function through nonlinear optimization algorithms.
(19)min∑inO+λiΛj−Pij2,ΛjTΛj=1.On the whole, as has been noted, the general nonparametric imaging model is rather flexible and accommodates a wider range of imaging devices than the specific parametric imaging model. As in the case of Scheimpflug camera calibration, no matter whether the lens is tilted or not, the Scheimpflug camera can be regarded as the same with the ordinary camera in the general nonparametric imaging model. And the concise GNIM avoids the problem that the parameters in the conventional camera calibration methods easily converge to the local optimums.Nonetheless, the nonparametric nature of GNIM results in the obscure physical meaning of the model. Moreover, the major drawback in generic calibration model, as presented in literature [32, 33, 35], is that the same motion variables can be computed from two different coupled variables, which leads to discrepancies in the computation of motion variables and some way of averaging should be used to obtain a consistent solution. Besides, the calibration method involves solving a large number of linear equations which is much more complicated than the former calibration methods.


## 5. Experiment

### 5.1. Experimental Apparatus

Two Scheimpflug imaging systems are calibrated to demonstrate the performance of different calibration models. One imaging system is a DSRL camera (Nikon D300s, 4288 × 2848 pixels, pixel size of 5.5μm) with a tilt-shift lens (Nikon PC-E Micro NIKKOR 45 mm f/2.8 D ED). Owing to the fact that the tilt range of first imaging system is limited within ±8.5°, we construct a custom-made Scheimpflug imaging system on the basis of a MVI-D2048 camera (2048 × 2048 pixels) with a perspective lens (kowa-LM25JC5M2 25 mm/f), and the sensor is intentionally tilted with respect to the lens by ≈10°. A ceramic checkerboard with 9 × 12 squares, each having dimension of 10 × 10 mm, is employed to model known control points, and 15 calibration images are taken at different locations, from different angles.As for the first imaging system, two sets of experiments are carried out to quantify the performance of calibration models. One set is to adjust the tilt-shift lens to the angle approximate 0°, in the case of which the Scheimpflug camera model reduces to standard camera model. The other set rotates the tilt-shift lens to the angle approximate 7°. In view of the fact that both Scheimpflug imaging systems employed here just simply rotate around one single axis, the tilt angle rotating around the other axis is assumed to be 0°.


### 5.2. Results and Discussion

Tables1 and 2 show the calibration results of the first imaging system’s intrinsic parameters and reprojection errors using different calibration models with tilt angles of 0° and 7°, respectively. The calibration results of the second imaging system are illustrated in Table 3. In calibration model (1), only the first two order radial distortion coefficients (k1,k2) are considered, while calibration model (2) includes both the radial and tangential distortion coefficients (k1,k2,k3,k4). As illustrated in the following tables, (α,β) represent the two dimensional tilt angles corresponding to the specific calibration models, (Cx,Cy) and (fx,fy), respectively, denote the principal point and focal length, and the reprojection error, denoted by RMSE, is indicated via the root of mean squared error in pixels, between the detected image corners and the projected ones.Table 1
Calibration results of the first imaging system with tilt angle of 0° approximately.
Calibration model
Pinhole (1)
Pinhole (2)
RMM (1)
RMM (2)
PLVM (1)
PLVM (2)
PCIM (1)
PCIM (2)
α (°)
—
—
0.1928
0.1214
0.2149
0.7449
0.3175
0.2436
β (°)
—
—
−0.5407
2.7449
0.7623
−3.1642
0.7844
0.6752
Cx (pixel)
1984.8581
2044.4909
2045.3706
2041.9526
2045.6208
2077.7127
2040.3889
2044.8924
Cy (pixel)
1499.5190
1461.1779
1461.8725
1446.3428
1462.4681
1447.5127
1458.2369
1462.3325
fx (mm)
48.0293
48.0073
48.0056
47.8958
47.9954
48.1980
48.1031
47.9975
fy (mm)
47.9819
47.9663
48.0024
47.9726
47.9954
48.1980
48.1031
47.9975
k1
−0.0777
−0.0785
−0.0837
−0.0793
−0.0852
−0.0806
−0.0798
−0.0802
k2
0.1294
0.1255
0.2126
0.1560
0.2329
0.0922
0.2330
0.1121
k3
—
−0.0018
—
0.0022
—
0.0015
—
0.0019
k4
—
0.0025
—
0.0036
—
0.0028
—
0.0027
RMSE (pixel)
0.1896
0.1620
0.1623
0.1614
0.1630
0.1608
0.1622
0.1619Table 2
Calibration results of the first imaging system with tilt angle of 7° approximately.
Calibration model
Pinhole (1)
Pinhole (2)
RMM (1)
RMM (2)
PLVM (1)
PLVM (2)
PCIM (1)
PCIM (2)
α (°)
—
—
6.4126
5.5814
6.3463
5.1486
6.2329
5.4688
β (°)
—
—
0.9920
1.7603
0.8197
2.6659
1.2703
2.2456
Cx (pixel)
1986.0064
2099.2407
2055.3751
2098.9543
2058.8085
2121.2302
2049.6208
2140.7542
Cy (pixel)
1883.7801
1935.0271
1978.2631
1869.6842
1984.6398
1784.6398
1974.4682
1776.5569
fx (mm)
48.2265
48.2806
48.2024
48.5231
48.3472
48.4253
48.1954
48.1151
fy (mm)
48.4051
48.4798
48.1955
48.5018
48.3472
48.4253
48.1954
48.1151
k1
−0.0854
−0.0824
−0.0755
−0.0836
−0.0694
−0.0663
−0.0542
−0.0945
k2
0.2769
0.2271
0.0795
0.0725
0.0252
−0.0085
−0.0167
0.4497
k3
—
0.0022
—
−0.0022
—
0.0107
—
−0.9368
k4
—
0.0041
—
0.0098
—
0.0019
—
0.0023
RMSE (pixel)
0.2021
0.1928
0.1701
0.1697
0.1693
0.1690
0.1685
0.1683Table 3
Calibration results of the second imaging system with tilt angle of 10° approximately.
Calibration model
Pinhole (1)
Pinhole (2)
RMM (1)
RMM (2)
PLVM (1)
PLVM (2)
PCIM (1)
PCIM (2)
α (°)
—
—
9.2544
3.0078
9.1981
2.9987
9.0974
4.6926
β (°)
—
—
−0.9193
0.0248
1.0680
−0.0337
268.2745
261.3949
Cx (pixel)
1011.6078
1015.8023
1017.8545
1010.8408
1014.2669
1011.9323
1014.2182
1009.2238
Cy (pixel)
950.4627
373.6203
945.0634
439.4258
921.0598
684.9546
907.9720
424.2683
fx (mm)
25.2562
25.8743
26.3688
26.1073
26.4134
26.1090
26.4266
26.3146
fy (mm)
29.1527
27.1610
26.0862
26.1204
26.4134
26.1090
26.4266
26.3146
k1
−0.0643
−0.0023
−0.0695
0.0010
−0.0748
0.0003
−0.0454
0.0011
k2
0.8235
−1.2101
0.6203
−1.4850
0.8104
−1.4587
0.6705
−1.4361
k3
—
0.0046
—
0.0044
—
0.0044
—
0.0005
k4
—
0.0005
—
0.0004
—
0.0002
—
0.0089
RMSE (pixel)
0.1556
0.1548
0.1076
0.1071
0.1078
0.1076
0.1080
0.1078From the calibration results of conventional pinhole model in the above three tables, it can be seen that pinhole model works well with the tilt angle of approximate 0°, yet when the tilt angle increases several degrees (such as 7° and 10°), the reprojection errors increase significantly whereas the aforementioned models almost remain constant. This is also consistent with the conclusion made in literature [23] that when the lens-tilted angle is greater than 7°, the tangential distortion cannot compensate the error generated by tilting the sensor.In terms of tilt angles in above tables, the calibration results of RMM (1), PLVM (1), and PCIM (1) are quite close to corresponding tilt angles measured manually in the adapter, in spite of minor deviations resulting from measurement noise. Furthermore, comparing the calibration results of models (1) and models (2), respectively, it is obvious to find that the formers give better results and smaller angle errors than the latters. More comprehensive distortion model in RMM (2), PLVM (2), and PCIM (2) does not necessarily result in better calibration results, and together with the previous analysis, it can be concluded that the parameters of tangential distortion are likely coupled with the tilt angles of the image plane in RMM, PLVM, and PCIM.It is difficult to obtain high accuracy ground truth that serves as absolute reference for the Scheimpflug camera’s calibration results. Hence, this paper borrows the idea from the literature [19–22] that assesses the accuracy of the calibration results via the uncertainties of the calibration parameters.Supposem = f(P,Q) indicates the projection from the point set Q(Xi,Yi,Zi) i=1,2,…,M in the world to the image feature points muik,vik, k=1,2,…,n from n different views, where the parameter set P consists of the intrinsic parameter set PA = {α,β,fx,fy,Cx,Cy,k1,k2} and extrinsic parameter set PB = {omk,Tk} of the Scheimpflug camera. Considering the fact that the nonlinear least square problem such as camera calibration generally employs the iterative method (e.g., Levenberg-Marquardt method), the linear iteration of the parameters can be modeled as Pi+ 1 = Pi + ΔPi. As for the minor deviation ΔP, we have
(20)Δm=JmPΔP,where the Jacobian matrix is defined by JmP=∂m/∂P.
(21)JmP=JmPA1JmPB10⋯0JmPA20JmPB2⋯0⋮⋮⋮⋯⋮JmPAn0⋯0JmPBn.And the sub-Jacobian matrixJmPAk and JmPBk are, respectively, defined as follows:
(22)JmPAk=∂u1k∂α∂u1k∂β∂u1k∂fx∂u1k∂fy⋯∂v1k∂α∂v1k∂β∂v1k∂fx∂v1k∂fy⋯⋮⋮⋮⋮⋯∂uMk∂α∂uMk∂β∂uMk∂fx∂uMk∂fy⋯∂vMk∂α∂vMk∂β∂vMk∂fx∂vMk∂fy⋯,JmPBk=∂u1k∂om1∂u1k∂om2∂u1k∂om3∂u1k∂T1⋯∂v1k∂om1∂v1k∂om2∂v1k∂om3∂v1k∂T1⋯⋮⋮⋮⋮⋯∂uMk∂om1∂uMk∂om2∂uMk∂om3∂uMk∂T1⋯∂vMk∂om1∂vMk∂om2∂vMk∂om3∂vMk∂T1⋯.Assuming thatΛm refers to the covariance matrix of m which can be defined as Λm = σ2I (σ2 is the standard deviation of m, and in order to simplify the analysis, it can be assumed that each feature point is independently observed). Hence, the best linear unbiased estimate of ΔP can be calculated via
(23)ΔP=JmPTΛm−1JmP−1JmPTΛm−1Δm.Thus, we can calculate the covariance matrix of the parameter setP by
(24)ΛP=σm2JmPTJmP−1.Therefore, the uncertainties of the second imaging system’s calibration parameters using four kinds of models (1) can be obtained by the formula (25), and the results are shown in Table 4.
(25)ΓPPA=3σdiagΛP.Table 4
The uncertainties of the intrinsic parameters of the second imaging system using models (1).
Calibration model
α (°)
β (°)
Cx (pixel)
Cy (pixel)
fx (mm)
fy (mm)
Pinhole
—
—
2.4277
78.7451
0.0995
0.4525
RMM
0.0781
0.0126
0.7514
2.1001
0.0308
0.0649
PLVM
0.0776
0.0133
0.7766
2.1731
0.0462
0.0462
PCIM
0.0811
0.0135
0.7840
2.1417
0.0505
0.0505As shown in Table4, the uncertainties of the calibration parameters recovered from RMM, PLVM, and PCIM are obviously smaller than those from the conventional pinhole imaging model. Besides, the uncertainties of recovered parameters from RMM, PLVM, and PCIM are almost consistent with each other, which mean the accuracies of the three models are approximately equal.Moreover, the experiments examine the performance of different calibration models with respect to the number of the planes utilized to recover the camera parameters. To facilitate comparison, assuming that the tilt angles around the two axes are 10° and 0°, respectively, the results of tilt angles in different models are transformed into the deviations with respect to the reference values. Figure10 shows the results of the second Scheimpflug imaging system’s intrinsic parameters versus the number of images for different calibration models.Figure 10
Intrinsic parameters results versus the number of the images for different calibration models.
(a)
(b)
(c)
(d)
(e)
(f)From Figures10(a) and 10(d), it can be seen that three calibration models can quickly converge to the global optimization and the tilt angles α recovered from the three models are almost consistent with an accuracy of 1°, while the tilt angles β vary from one model to another with the accuracies of approximate 1°, 0.5°, and 1.5°, respectively. As far as the experiment is concerned, the PCIM gives slightly better results than the other two models with respect to the tilt angles.As the Figures10(b) and 10(e) illustrated, the results of focal length via the three Scheimpflug calibration models are consistent with each other. Choosing the results of conventional pinhole imaging model as the reference, it is evident to find that the fx recovered from the three Scheimpflug calibration models is larger, while the fy is smaller. Besides, it is revealed that when the lens is tilted with respect to the sensor, the effective focal length increases accordingly, which is also in accord with the conclusion in literature [1, 2].Figures10(c) and 10(f) show the results of principal points using four different models. It is evident to observe that the results of Cx and Cy via the three Scheimpflug calibration models are quite similar with each other. And it is worthy of noting that the Cx derived from the Scheimpflug calibration models and pinhole imaging model almost has the same results with little difference. However, the results of Cy recovered from the two main types of calibration models are quite different. Obviously, the principal points are supposed to move in one direction on the image plane, since the lens is only tilted with respect to one single axis.In addition, as depicted in Figure11, it can be seen that the calibration models undergoing Scheimpflug condition give smaller RMSE than the pinhole imaging model. In spite of the fact that full calibration of Scheimpflug camera using a checkerboard requires at least three different views mathematically, four views at least are required to obtain accurate and robust results in practice. Together with the above calibration results, it can be concluded that calibration models undergoing Scheimpflug condition are the better description of Scheimpflug camera imaging.Figure 11
RMSE versus the number of the images for different calibration models.On account of the fact that the GNIM does not have explicit intrinsic parameters used for comparison and the limit of pages, we choose the calibration results of any two checkerboards’ motion parameters, as illustrated in Table5, to verify the performance of the GNIM to a certain extent. To facilitate comparison, the rotation matrix elements are expressed as the Rodrigues parameter forms.Table 5
Calibration results of motion parameters among calibration checkerboards using different calibration models.
Calibration model
(R11, R21, R31)
(T11, T21, T31)
(R12, R22, R32)
(T12, T22, T32)
RMM (1)
(1.5067, 1.5080, −0.8159)
(−54.7739, −45.2021, 442.8741)
(1.9628, 0.7965, −0.3941)
(−66.5666, −26.0286, 400.8027)
PLVM (1)
(1.5392, 1.4766, −0.8186)
(−53.7829, −48.3303, 441.7981)
(1.9208, 0.7914, −0.4100)
(−67.8423, −25.1659, 399.2397)
PCIM (1)
(1.4928, 1.4434, −0.9046)
(−52.5684, −44.1141, 441.0225)
(1.8824, 0.7641, −0.4607)
(−65.8903, −25.5468, 397.2328)
GNIM
(1.5123, 1.4682, −0.8481)
(−53.3728, −46.9573, 441.6816)
(1.9031, 0.7725, −0.4432)
(−66.7914, −25.6247, 398.4627)As shown in Table5, the motion parameter calibration results of GNIM are consistent with the other three models despite of minor deviations, which indirectly prove the effectiveness of the GNIM in Scheimpflug camera calibration. That is, the GNIM accommodates a wide range of imaging system including the Scheimpflug imaging system.Furthermore, the 3D coordinates and structure of the checkerboards are reconstructed with the help of calibrated parameters and obtained image points. As far as the experimental results indicate, the reconstruction results of the checkerboard points using the four different Scheimpflug calibration models are basically in accord with each other, as illustrated in Figure12. Figure 12(a) depicts the reconstructed 3D checkerboard points and the fitted plane while the error distribution of reconstructed checkerboard points is shown in Figure 12(b). Besides, the RMSEs of the distance from the reconstructed points to the fitted plane using four models are about 0.0300 mm despite of slight fluctuations, as given in Table 6.Figure 12
The reconstructed results and error distribution of the 3D checkerboard points. (a) The reconstructed 3D checkerboard points and the fitted 3D plane. (b) The error distribution of reconstructed points with respect to the fitted plane.
(a)
(b)Table 6
The RMSE of reconstructed points with respect to the fitted plane.
Calibration model
RMM
PLVM
PCIM
GNIM
RMSE (mm)
0.0300
0.0287
0.0295
0.0311According to the reconstruction results in Figure12, the reconstructed 3D points agree well with the fitted plane in spite of minor deviations, which might result from the following factors: (1) the distortion parameters are not absolutely accurate to describe the lens distortion, and (2) the image point coordinates obtained by corner extraction also have errors inevitably. Consequently, the error of image points will be enlarged in the process of 3D reconstruction.As shown in Figure12(b), the errors of reconstructed points are approximately symmetrical about the plane center, while the errors of the reconstructed points at the edge are more significant. Nonetheless, the above reconstructed errors (about 0.030 mm) are still very small compared to the depth of the checkerboard.To further verify the accuracy, another two sets of experiments are carried out with the help of two-axis electric rotary table (SLT-2MA) and three-axis electric translation table (ZG14TA) as shown in Figures13 and 14, respectively. The cooperated mark is successively mounted on the two-axis electric rotary table and three-axis electric translation table; therefore, the motions of cooperated mark are pure rotations and translations, respectively, in the two sets of experiments.Figure 13
Two-axis electric rotary table (SLT-2MA).Figure 14
Three-axis electric translation table.In the first experiment, the two-axis electric rotary table is controlled to a rotation of no less than 5° around the two axes for each rotation. As for the second experiment, we control the three-axis electric translation table to a translation of more than 10 mm for each movement, and both the operations are repeated ten times.Meanwhile, we estimate the pose change of the cooperated mark via the first Scheimpflug imaging system with different imaging models. In this way, the spatial relationship between the coordinate frames of the checkerboard and the two-axis electric rotary table or three-axis electric translation table can be converted into the hand-eye calibration problem [45–48], which already has a very mature solution. Hence, the reference pose of the cooperated mark can be obtained via the two auxiliary equipment. And the pose estimation errors with respect to the reference values of the two sets of experiments are shown in Table 7.Table 7
The pose estimation errors of the two sets of experiments.
Experiments
Calibration model
RMM
PLVM
PCIM
GNIM
1st
RMSE of rotation (°)
0.0387
0.0409
0.0400
0.0417
RMSE of translation (mm)
1.4659
1.5109
1.4905
1.5305
2nd
RMSE of rotation (°)
0.0412
0.0428
0.0424
0.0432
RMSE of translation (mm)
1.5562
1.6042
1.5732
1.6655It can be seen in Table7 that the pose estimation errors vary little in first three parametric Scheimpflug imaging models, while they are slightly smaller than those in GNIM, regardless of whether the cooperated mark is merely rotation or translation. As revealed in Table 7, the measurement results of rotation and translation can be obtained to an accuracy of about 0.040° and 1.5 mm, respectively. Given that the camera employed here is an ordinary DSRL camera, rather than an industrial camera with high precision and robustness, the pose estimation errors of the two sets of experiments are relatively low, which also validates the accuracy and effectiveness of the calibration models.


## 5.1. Experimental Apparatus

Two Scheimpflug imaging systems are calibrated to demonstrate the performance of different calibration models. One imaging system is a DSRL camera (Nikon D300s, 4288 × 2848 pixels, pixel size of 5.5μm) with a tilt-shift lens (Nikon PC-E Micro NIKKOR 45 mm f/2.8 D ED). Owing to the fact that the tilt range of first imaging system is limited within ±8.5°, we construct a custom-made Scheimpflug imaging system on the basis of a MVI-D2048 camera (2048 × 2048 pixels) with a perspective lens (kowa-LM25JC5M2 25 mm/f), and the sensor is intentionally tilted with respect to the lens by ≈10°. A ceramic checkerboard with 9 × 12 squares, each having dimension of 10 × 10 mm, is employed to model known control points, and 15 calibration images are taken at different locations, from different angles.As for the first imaging system, two sets of experiments are carried out to quantify the performance of calibration models. One set is to adjust the tilt-shift lens to the angle approximate 0°, in the case of which the Scheimpflug camera model reduces to standard camera model. The other set rotates the tilt-shift lens to the angle approximate 7°. In view of the fact that both Scheimpflug imaging systems employed here just simply rotate around one single axis, the tilt angle rotating around the other axis is assumed to be 0°.


## 5.2. Results and Discussion

Tables1 and 2 show the calibration results of the first imaging system’s intrinsic parameters and reprojection errors using different calibration models with tilt angles of 0° and 7°, respectively. The calibration results of the second imaging system are illustrated in Table 3. In calibration model (1), only the first two order radial distortion coefficients (k1,k2) are considered, while calibration model (2) includes both the radial and tangential distortion coefficients (k1,k2,k3,k4). As illustrated in the following tables, (α,β) represent the two dimensional tilt angles corresponding to the specific calibration models, (Cx,Cy) and (fx,fy), respectively, denote the principal point and focal length, and the reprojection error, denoted by RMSE, is indicated via the root of mean squared error in pixels, between the detected image corners and the projected ones.Table 1
Calibration results of the first imaging system with tilt angle of 0° approximately.
Calibration model
Pinhole (1)
Pinhole (2)
RMM (1)
RMM (2)
PLVM (1)
PLVM (2)
PCIM (1)
PCIM (2)
α (°)
—
—
0.1928
0.1214
0.2149
0.7449
0.3175
0.2436
β (°)
—
—
−0.5407
2.7449
0.7623
−3.1642
0.7844
0.6752
Cx (pixel)
1984.8581
2044.4909
2045.3706
2041.9526
2045.6208
2077.7127
2040.3889
2044.8924
Cy (pixel)
1499.5190
1461.1779
1461.8725
1446.3428
1462.4681
1447.5127
1458.2369
1462.3325
fx (mm)
48.0293
48.0073
48.0056
47.8958
47.9954
48.1980
48.1031
47.9975
fy (mm)
47.9819
47.9663
48.0024
47.9726
47.9954
48.1980
48.1031
47.9975
k1
−0.0777
−0.0785
−0.0837
−0.0793
−0.0852
−0.0806
−0.0798
−0.0802
k2
0.1294
0.1255
0.2126
0.1560
0.2329
0.0922
0.2330
0.1121
k3
—
−0.0018
—
0.0022
—
0.0015
—
0.0019
k4
—
0.0025
—
0.0036
—
0.0028
—
0.0027
RMSE (pixel)
0.1896
0.1620
0.1623
0.1614
0.1630
0.1608
0.1622
0.1619Table 2
Calibration results of the first imaging system with tilt angle of 7° approximately.
Calibration model
Pinhole (1)
Pinhole (2)
RMM (1)
RMM (2)
PLVM (1)
PLVM (2)
PCIM (1)
PCIM (2)
α (°)
—
—
6.4126
5.5814
6.3463
5.1486
6.2329
5.4688
β (°)
—
—
0.9920
1.7603
0.8197
2.6659
1.2703
2.2456
Cx (pixel)
1986.0064
2099.2407
2055.3751
2098.9543
2058.8085
2121.2302
2049.6208
2140.7542
Cy (pixel)
1883.7801
1935.0271
1978.2631
1869.6842
1984.6398
1784.6398
1974.4682
1776.5569
fx (mm)
48.2265
48.2806
48.2024
48.5231
48.3472
48.4253
48.1954
48.1151
fy (mm)
48.4051
48.4798
48.1955
48.5018
48.3472
48.4253
48.1954
48.1151
k1
−0.0854
−0.0824
−0.0755
−0.0836
−0.0694
−0.0663
−0.0542
−0.0945
k2
0.2769
0.2271
0.0795
0.0725
0.0252
−0.0085
−0.0167
0.4497
k3
—
0.0022
—
−0.0022
—
0.0107
—
−0.9368
k4
—
0.0041
—
0.0098
—
0.0019
—
0.0023
RMSE (pixel)
0.2021
0.1928
0.1701
0.1697
0.1693
0.1690
0.1685
0.1683Table 3
Calibration results of the second imaging system with tilt angle of 10° approximately.
Calibration model
Pinhole (1)
Pinhole (2)
RMM (1)
RMM (2)
PLVM (1)
PLVM (2)
PCIM (1)
PCIM (2)
α (°)
—
—
9.2544
3.0078
9.1981
2.9987
9.0974
4.6926
β (°)
—
—
−0.9193
0.0248
1.0680
−0.0337
268.2745
261.3949
Cx (pixel)
1011.6078
1015.8023
1017.8545
1010.8408
1014.2669
1011.9323
1014.2182
1009.2238
Cy (pixel)
950.4627
373.6203
945.0634
439.4258
921.0598
684.9546
907.9720
424.2683
fx (mm)
25.2562
25.8743
26.3688
26.1073
26.4134
26.1090
26.4266
26.3146
fy (mm)
29.1527
27.1610
26.0862
26.1204
26.4134
26.1090
26.4266
26.3146
k1
−0.0643
−0.0023
−0.0695
0.0010
−0.0748
0.0003
−0.0454
0.0011
k2
0.8235
−1.2101
0.6203
−1.4850
0.8104
−1.4587
0.6705
−1.4361
k3
—
0.0046
—
0.0044
—
0.0044
—
0.0005
k4
—
0.0005
—
0.0004
—
0.0002
—
0.0089
RMSE (pixel)
0.1556
0.1548
0.1076
0.1071
0.1078
0.1076
0.1080
0.1078From the calibration results of conventional pinhole model in the above three tables, it can be seen that pinhole model works well with the tilt angle of approximate 0°, yet when the tilt angle increases several degrees (such as 7° and 10°), the reprojection errors increase significantly whereas the aforementioned models almost remain constant. This is also consistent with the conclusion made in literature [23] that when the lens-tilted angle is greater than 7°, the tangential distortion cannot compensate the error generated by tilting the sensor.In terms of tilt angles in above tables, the calibration results of RMM (1), PLVM (1), and PCIM (1) are quite close to corresponding tilt angles measured manually in the adapter, in spite of minor deviations resulting from measurement noise. Furthermore, comparing the calibration results of models (1) and models (2), respectively, it is obvious to find that the formers give better results and smaller angle errors than the latters. More comprehensive distortion model in RMM (2), PLVM (2), and PCIM (2) does not necessarily result in better calibration results, and together with the previous analysis, it can be concluded that the parameters of tangential distortion are likely coupled with the tilt angles of the image plane in RMM, PLVM, and PCIM.It is difficult to obtain high accuracy ground truth that serves as absolute reference for the Scheimpflug camera’s calibration results. Hence, this paper borrows the idea from the literature [19–22] that assesses the accuracy of the calibration results via the uncertainties of the calibration parameters.Supposem = f(P,Q) indicates the projection from the point set Q(Xi,Yi,Zi) i=1,2,…,M in the world to the image feature points muik,vik, k=1,2,…,n from n different views, where the parameter set P consists of the intrinsic parameter set PA = {α,β,fx,fy,Cx,Cy,k1,k2} and extrinsic parameter set PB = {omk,Tk} of the Scheimpflug camera. Considering the fact that the nonlinear least square problem such as camera calibration generally employs the iterative method (e.g., Levenberg-Marquardt method), the linear iteration of the parameters can be modeled as Pi+ 1 = Pi + ΔPi. As for the minor deviation ΔP, we have
(20)Δm=JmPΔP,where the Jacobian matrix is defined by JmP=∂m/∂P.
(21)JmP=JmPA1JmPB10⋯0JmPA20JmPB2⋯0⋮⋮⋮⋯⋮JmPAn0⋯0JmPBn.And the sub-Jacobian matrixJmPAk and JmPBk are, respectively, defined as follows:
(22)JmPAk=∂u1k∂α∂u1k∂β∂u1k∂fx∂u1k∂fy⋯∂v1k∂α∂v1k∂β∂v1k∂fx∂v1k∂fy⋯⋮⋮⋮⋮⋯∂uMk∂α∂uMk∂β∂uMk∂fx∂uMk∂fy⋯∂vMk∂α∂vMk∂β∂vMk∂fx∂vMk∂fy⋯,JmPBk=∂u1k∂om1∂u1k∂om2∂u1k∂om3∂u1k∂T1⋯∂v1k∂om1∂v1k∂om2∂v1k∂om3∂v1k∂T1⋯⋮⋮⋮⋮⋯∂uMk∂om1∂uMk∂om2∂uMk∂om3∂uMk∂T1⋯∂vMk∂om1∂vMk∂om2∂vMk∂om3∂vMk∂T1⋯.Assuming thatΛm refers to the covariance matrix of m which can be defined as Λm = σ2I (σ2 is the standard deviation of m, and in order to simplify the analysis, it can be assumed that each feature point is independently observed). Hence, the best linear unbiased estimate of ΔP can be calculated via
(23)ΔP=JmPTΛm−1JmP−1JmPTΛm−1Δm.Thus, we can calculate the covariance matrix of the parameter setP by
(24)ΛP=σm2JmPTJmP−1.Therefore, the uncertainties of the second imaging system’s calibration parameters using four kinds of models (1) can be obtained by the formula (25), and the results are shown in Table 4.
(25)ΓPPA=3σdiagΛP.Table 4
The uncertainties of the intrinsic parameters of the second imaging system using models (1).
Calibration model
α (°)
β (°)
Cx (pixel)
Cy (pixel)
fx (mm)
fy (mm)
Pinhole
—
—
2.4277
78.7451
0.0995
0.4525
RMM
0.0781
0.0126
0.7514
2.1001
0.0308
0.0649
PLVM
0.0776
0.0133
0.7766
2.1731
0.0462
0.0462
PCIM
0.0811
0.0135
0.7840
2.1417
0.0505
0.0505As shown in Table4, the uncertainties of the calibration parameters recovered from RMM, PLVM, and PCIM are obviously smaller than those from the conventional pinhole imaging model. Besides, the uncertainties of recovered parameters from RMM, PLVM, and PCIM are almost consistent with each other, which mean the accuracies of the three models are approximately equal.Moreover, the experiments examine the performance of different calibration models with respect to the number of the planes utilized to recover the camera parameters. To facilitate comparison, assuming that the tilt angles around the two axes are 10° and 0°, respectively, the results of tilt angles in different models are transformed into the deviations with respect to the reference values. Figure10 shows the results of the second Scheimpflug imaging system’s intrinsic parameters versus the number of images for different calibration models.Figure 10
Intrinsic parameters results versus the number of the images for different calibration models.
(a)
(b)
(c)
(d)
(e)
(f)From Figures10(a) and 10(d), it can be seen that three calibration models can quickly converge to the global optimization and the tilt angles α recovered from the three models are almost consistent with an accuracy of 1°, while the tilt angles β vary from one model to another with the accuracies of approximate 1°, 0.5°, and 1.5°, respectively. As far as the experiment is concerned, the PCIM gives slightly better results than the other two models with respect to the tilt angles.As the Figures10(b) and 10(e) illustrated, the results of focal length via the three Scheimpflug calibration models are consistent with each other. Choosing the results of conventional pinhole imaging model as the reference, it is evident to find that the fx recovered from the three Scheimpflug calibration models is larger, while the fy is smaller. Besides, it is revealed that when the lens is tilted with respect to the sensor, the effective focal length increases accordingly, which is also in accord with the conclusion in literature [1, 2].Figures10(c) and 10(f) show the results of principal points using four different models. It is evident to observe that the results of Cx and Cy via the three Scheimpflug calibration models are quite similar with each other. And it is worthy of noting that the Cx derived from the Scheimpflug calibration models and pinhole imaging model almost has the same results with little difference. However, the results of Cy recovered from the two main types of calibration models are quite different. Obviously, the principal points are supposed to move in one direction on the image plane, since the lens is only tilted with respect to one single axis.In addition, as depicted in Figure11, it can be seen that the calibration models undergoing Scheimpflug condition give smaller RMSE than the pinhole imaging model. In spite of the fact that full calibration of Scheimpflug camera using a checkerboard requires at least three different views mathematically, four views at least are required to obtain accurate and robust results in practice. Together with the above calibration results, it can be concluded that calibration models undergoing Scheimpflug condition are the better description of Scheimpflug camera imaging.Figure 11
RMSE versus the number of the images for different calibration models.On account of the fact that the GNIM does not have explicit intrinsic parameters used for comparison and the limit of pages, we choose the calibration results of any two checkerboards’ motion parameters, as illustrated in Table5, to verify the performance of the GNIM to a certain extent. To facilitate comparison, the rotation matrix elements are expressed as the Rodrigues parameter forms.Table 5
Calibration results of motion parameters among calibration checkerboards using different calibration models.
Calibration model
(R11, R21, R31)
(T11, T21, T31)
(R12, R22, R32)
(T12, T22, T32)
RMM (1)
(1.5067, 1.5080, −0.8159)
(−54.7739, −45.2021, 442.8741)
(1.9628, 0.7965, −0.3941)
(−66.5666, −26.0286, 400.8027)
PLVM (1)
(1.5392, 1.4766, −0.8186)
(−53.7829, −48.3303, 441.7981)
(1.9208, 0.7914, −0.4100)
(−67.8423, −25.1659, 399.2397)
PCIM (1)
(1.4928, 1.4434, −0.9046)
(−52.5684, −44.1141, 441.0225)
(1.8824, 0.7641, −0.4607)
(−65.8903, −25.5468, 397.2328)
GNIM
(1.5123, 1.4682, −0.8481)
(−53.3728, −46.9573, 441.6816)
(1.9031, 0.7725, −0.4432)
(−66.7914, −25.6247, 398.4627)As shown in Table5, the motion parameter calibration results of GNIM are consistent with the other three models despite of minor deviations, which indirectly prove the effectiveness of the GNIM in Scheimpflug camera calibration. That is, the GNIM accommodates a wide range of imaging system including the Scheimpflug imaging system.Furthermore, the 3D coordinates and structure of the checkerboards are reconstructed with the help of calibrated parameters and obtained image points. As far as the experimental results indicate, the reconstruction results of the checkerboard points using the four different Scheimpflug calibration models are basically in accord with each other, as illustrated in Figure12. Figure 12(a) depicts the reconstructed 3D checkerboard points and the fitted plane while the error distribution of reconstructed checkerboard points is shown in Figure 12(b). Besides, the RMSEs of the distance from the reconstructed points to the fitted plane using four models are about 0.0300 mm despite of slight fluctuations, as given in Table 6.Figure 12
The reconstructed results and error distribution of the 3D checkerboard points. (a) The reconstructed 3D checkerboard points and the fitted 3D plane. (b) The error distribution of reconstructed points with respect to the fitted plane.
(a)
(b)Table 6
The RMSE of reconstructed points with respect to the fitted plane.
Calibration model
RMM
PLVM
PCIM
GNIM
RMSE (mm)
0.0300
0.0287
0.0295
0.0311According to the reconstruction results in Figure12, the reconstructed 3D points agree well with the fitted plane in spite of minor deviations, which might result from the following factors: (1) the distortion parameters are not absolutely accurate to describe the lens distortion, and (2) the image point coordinates obtained by corner extraction also have errors inevitably. Consequently, the error of image points will be enlarged in the process of 3D reconstruction.As shown in Figure12(b), the errors of reconstructed points are approximately symmetrical about the plane center, while the errors of the reconstructed points at the edge are more significant. Nonetheless, the above reconstructed errors (about 0.030 mm) are still very small compared to the depth of the checkerboard.To further verify the accuracy, another two sets of experiments are carried out with the help of two-axis electric rotary table (SLT-2MA) and three-axis electric translation table (ZG14TA) as shown in Figures13 and 14, respectively. The cooperated mark is successively mounted on the two-axis electric rotary table and three-axis electric translation table; therefore, the motions of cooperated mark are pure rotations and translations, respectively, in the two sets of experiments.Figure 13
Two-axis electric rotary table (SLT-2MA).Figure 14
Three-axis electric translation table.In the first experiment, the two-axis electric rotary table is controlled to a rotation of no less than 5° around the two axes for each rotation. As for the second experiment, we control the three-axis electric translation table to a translation of more than 10 mm for each movement, and both the operations are repeated ten times.Meanwhile, we estimate the pose change of the cooperated mark via the first Scheimpflug imaging system with different imaging models. In this way, the spatial relationship between the coordinate frames of the checkerboard and the two-axis electric rotary table or three-axis electric translation table can be converted into the hand-eye calibration problem [45–48], which already has a very mature solution. Hence, the reference pose of the cooperated mark can be obtained via the two auxiliary equipment. And the pose estimation errors with respect to the reference values of the two sets of experiments are shown in Table 7.Table 7
The pose estimation errors of the two sets of experiments.
Experiments
Calibration model
RMM
PLVM
PCIM
GNIM
1st
RMSE of rotation (°)
0.0387
0.0409
0.0400
0.0417
RMSE of translation (mm)
1.4659
1.5109
1.4905
1.5305
2nd
RMSE of rotation (°)
0.0412
0.0428
0.0424
0.0432
RMSE of translation (mm)
1.5562
1.6042
1.5732
1.6655It can be seen in Table7 that the pose estimation errors vary little in first three parametric Scheimpflug imaging models, while they are slightly smaller than those in GNIM, regardless of whether the cooperated mark is merely rotation or translation. As revealed in Table 7, the measurement results of rotation and translation can be obtained to an accuracy of about 0.040° and 1.5 mm, respectively. Given that the camera employed here is an ordinary DSRL camera, rather than an industrial camera with high precision and robustness, the pose estimation errors of the two sets of experiments are relatively low, which also validates the accuracy and effectiveness of the calibration models.


## 6. Conclusion

This paper presents a comprehensive survey of recent calibration methods of the Scheimpflug camera with perspective lens. The general nonparametric imaging model is novelly employed to deal with the problem as well. All the calibration models are briefly recalled and compared in detail with respect to each other, with some highlights on their respective advantages and drawbacks. Real data experiments including calibrations, reconstructions, and measurements are performed to validate the performance of the calibration models.As the experimental results indicated, compared with the classic pinhole imaging model, the models undergoing Scheimpflug condition are the better description of Scheimpflug camera imaging, especially when the tilt angle is greater than 7°. Moreover, although the imaging models and the parameter forms are various, the accuracies of the four calibration models (RMM, PLVM, PCIM, and GNIM) are basically equal, while the accuracy of GNIM is slightly lower compared with the other three parametric models in view of the errors in reconstruction and pose estimation. Given that the PLVM and PCIM require the pixel aspect ratio to be known in advance, the calibration model of RMM is rather flexible. Besides, the experimental results reveal that the parameters of tangential distortion are likely coupled with the tilt angles of the image plane in the calibration models of RMM, PLVM, and PCIM. In the actual calibration task, the appropriate calibration model is supposed to be chosen according to the specific implementation condition.



---

*Source: 3901431-2018-03-27.xml*