Plane dimension detection with a single camera based on a flat refractive camera model

The Journal of EngineeringVolume 2019, Issue 3 p. 647-651 Research ArticleOpen Access Plane dimension detection with a single camera based on a flat refractive camera model Dong Xu, Corresponding Author Dong Xu xudong@ustb.edu.cn National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorKedong Liu, Kedong Liu National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorQuan Yang, Quan Yang National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorFeng Dong, Feng Dong National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorXiaochen Wang, Xiaochen Wang National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorZhenyang Dai, Zhenyang Dai National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this author Dong Xu, Corresponding Author Dong Xu xudong@ustb.edu.cn National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorKedong Liu, Kedong Liu National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorQuan Yang, Quan Yang National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorFeng Dong, Feng Dong National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorXiaochen Wang, Xiaochen Wang National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this authorZhenyang Dai, Zhenyang Dai National Engineering Research Center of Flat Rolling Equipment, University of Science and Technology Beijing, 30 Xueyuan Road, Beijing, People's Republic of ChinaSearch for more papers by this author First published: 25 February 2019 https://doi.org/10.1049/joe.2018.5124AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract It is common that a pinhole camera captures object images through a transparent plate with two parallel planar faces. In this case, the refraction leads to a non-linear error using the simple pinhole model. To address this issue, a modified pinhole camera model, which is called flat refractive camera model, is established based on the analysis of light refraction in the transparent plate placed between the scene and the camera. Then, an effective backward-projection algorithm is proposed to realise plane dimension detection and eliminate the plate refraction effect. Unlike other models, this proposed model has a form similar to the pinhole camera model. Therefore, it is easy to employ conventional vision techniques in the plate imaging system. At the same time, experiments are designed to demonstrate the plane dimension detection based on the proposed method. Experimental results show that the proposed method can obtain higher accuracy than the traditional method without considering the plate refraction effect. 1 Introduction Dimension detection of components is a routine job in mechanical industries. Recently, applying machine vision to that field has become a necessary trend in modern manufacturing [1-4]. Some cameras for dimension measurement usually have glasses as the visual window to keep the cameras away from the dust and steam environment. In this case, the light refraction in glasses leads to a non-linear error using the simple pinhole model. The correction of that non-linear errors is a key algorithm for this type of technology, also widely used in robot vision, smart cameras, and face recognition [5-7]. The plate imaging system, a kind of non-single viewpoint imaging system, is widely used in real applications such as explosion-proof cameras. These cameras, used in mining or metallurgy factories, usually have glasses with a certain thickness as the visual window to protect the camera from the dusty and steamy environment. The plate imaging system is composed of a pinhole camera and a transparent plate with two parallel planar faces (plate for short) between the camera and the target scene, which introduce a refraction effect in the imaging. In general, the refraction effect is considered as a radial distortion or simply a negligible refraction effect when a camera captures the images through a transparent plate [8-12]. Therefore, it is necessary to analyse the effect of plate refraction on the detection of the plane dimension in order to ensure its accuracy. Gao et al. [13] proposed an algorithm to reconstruct 3D structures and recover high-resolution images by a novel depth-sensing system composed of a single camera, and a transparent plate which is placed in front of the camera and rotates about the optical axis of the camera. Chen et al. [14] introduced a novel method for depth acquisition based on the refraction of light with the first image captured directly and the second captured by placing a transparent medium between the scene and the camera. Agrawal et al. [15] developed a calibration theory and derived forward-projection equations by analysing the geometry of a perspective camera imaging through multiple flat refractive layers. Shimizu and Okutomi [16, 17] proposed stereo reflection that records scenes from a fixed viewpoint, with and without the reflection light paths. Their set-up uses a reflective or refractive medium for the implementation. However, that method requires a complex calibration set-up, as described in [18]. In addition, in underwater imaging systems, light is refracted by water, glass, and air in the propagation process, which is similar to the plate imaging system. Underwater imaging systems have been studied extensively, and some camera models have been proposed to address the effects of refraction, which are instructive in solving problems in plate imaging systems [9-12, 19]. However, all the above models are not in a neat form similar to the pinhole camera model, but rather use a complicated process involving the computation of the refraction vector and refraction point twice in backward-projection or complex calibration settings. In this paper, a practical camera model is established based on the study of light refraction in plate imaging systems, eliminates the influence of the refraction effect. Then, based on the flat refractive camera model, an effective backward-projection algorithm is proposed to realise the plane detection. Finally, experiments are conducted to evaluate the accuracy of the proposed method. This proposed model has a form similar to the pinhole camera model so that is easy to employ conventional vision techniques in the imaging system. 2 Light refraction in a plate imaging system In the actual application, the transparent plate is located between the scene and the camera in a planer imaging system. The parallel planar faces of the transparent plate are parallel to the image plane in the specific case of this study. Moreover, the case where the faces are not parallel to the image plane but with a known orientation can be converted to the special parallel case we studied with a rotation matrix [14]. Based on the assumption that the imaging system has a radial symmetry around the optical axis, the meridian plane can be selected to analyse the imaging process, as shown in Fig. 1, which includes the optical axis and given space points and image points. Fig. 1Open in figure viewerPowerPoint Diagram of projection on the image plate through a parallel plate Consider the object point A, (X c, Y c, Z c) in camera coordinates, observed by a camera centred on O c. The direct projection of A on the image plane is the image point S ′, (u ′, v ′) in the pixel coordinates. Assuming that a transparent plate having two parallel planes placed between the camera and object point A, so that the two parallel planes parallel to the image plane. As shown in Fig. 1, the plate imaging systems are not suitable for the pinhole camera model due to the refraction effect of the plates. Then, the object point A is no longer projected to the image point S ′. It is assumed that the light reflected from the object point A passes through the plate and intersects the surface of the plate at points B and C, and forms the image point S (u, v) in the pixel coordinates. In addition, assume that the virtual object plane is perpendicular to the optical axis and passes through the object point A. The virtual object plane point A ′, in camera coordinates, may be considered as the object point from the given image point S by backward projection without the plate. Refraction occurs twice when light passes through the plate. According to Snell's law [20] (1) where α and β are the incident angle in the air and the refracting angle in the plate, respectively, and n is the refractive index of the transparent plate relative to air. Furthermore, from the geometry shown in Fig. 1(2) (3) where Δ1 and Δ2 are the length of SS ′ and AA ′, respectively, and f and d are the focal length and the thickness of the transparent plate, respectively. Through (1)–(3), displacements Δ1 and Δ2 can be obtained as (4) (5) Light passing through the parallel plates causes a lateral displacement Δ2 without any change in the direction, as shown in Fig. 1. According to (5), the displacement Δ2 is related to the thickness d, the refractive index n of the plate, and the incident angle α. From (5), the variation of displacement Δ2 increases as the incident angle α increases, as shown in Fig. 2. It is demonstrated that when the incident angle α increases to 30°, displacement Δ2 increases gently in any case of the plate thickness and refractive index, which corresponds to the gently varying region in Fig. 2. Conversely, when the incident angle is >60°, displacement Δ2 sharply increases, corresponding to the region of significant variation in Fig. 2. In addition, as the refractive index or the thickness of the plate increases, the refraction becomes more pronounced, resulting in a larger displacement Δ2. More specifically, displacement Δ2 increases linearly with the plate thickness and increases logarithmically with the refractive index. Fig. 2Open in figure viewerPowerPoint Variation of displacement Δ2 as incident angle α increases (a) Refractive index n = 1.5 with different plate thickness, (b) Plate thickness d = 4 mm with different refractive index 3 Flat refractive camera model In this section, based on the pinhole camera system, an improved model called a flat refractive camera model is established, which represents the projection of the planer imaging system between object points and image points, as shown in Fig. 1. As described above, the displacement Δ2 can be written in terms of the camera coordinates of points A and A ′ as follows: (6) Also, based on the assumptions about the virtual object plane (7) Through (6) and (7), (8) where . (u, v) are the pixel coordinates of point S, and (u 0, v 0) are the coordinates of the principal point. Based on the pinhole camera mapping, the relationship between the object point A ′ and the image point S without the plate can be obtained: (9) where are the world coordinates of point A ′. R and T are the rotation matrix and translation vector between the world coordinates and the camera coordinates, respectively. Through (8) and the extrinsic camera parameters, (10) where (X w, Y w, Z w) are the world coordinates of point A. The flat refractive matrix M g can be defined as (11) Substituting (10) into (9), the relationship between the object point A and the image point S in the plate imaging system with light refraction can be expressed as (12) where M in and M ex are the matrix of intrinsic camera parameters and extrinsic camera parameters, respectively. Therefore, except for the difference in the flat refractive matrix, the flat refractive camera model has the same form as the pinhole camera model. This means that some of the advantages of the pinhole camera model can be used in the plate refractive imaging system. 4 Plane dimension detection algorithm Using the proposed camera model, plane dimension could be detected via the image captured by the single camera through the transparent plate. The procedure could be expressed as follows. First, the camera intrinsic parameters M in and distortion parameters are calibrated without transparent plate using Zhang's method [21]. The homography matrix is then estimated. After that, the extrinsic parameters M ex of the checkerboard in the measurement plane can be obtained. Second, depending on the image captured by the calibrated single camera through the transparent plate, feature points can be detected by the edge or feature point detection methods. Then, the most important issue with the proposed camera model is to calculate Δ2 in camera coordinates. Δ2 can be obtained through (4) with known n and d. It is a function of the incident angle α rather than a fixed term in the model. There are two ways to determine the incident angle, namely backward projection and forward projection. In the backward projection, the given image point and its corresponding object point are unknown, which is the same as the size detection condition using a single camera: (13) In the forward projection, it is difficult to provide an effective analytical solution of the incident angle α for the reason that the object world coordinates are known, but the corresponding image point is not. However, it transforms the problem of solving the incident angle into solving a fourth-order algebraic equation [22-24]. After determining the incident angle α, the flat refractive camera model could be determined as well. The proposed camera model is not a constant matrix, but varies according to the position of the image point in the backward projection, that is, distinct from the traditional pinhole camera model. According to every single obtained feature point, M g can be obtained through (11). For each single feature point, the new homography matrix can be obtained: (14) Expanding (14), we have (15) Eliminating the variable Zc in (15), we have (16) In the planer size detection process, it can be assumed that the measurement plane is generally perpendicular to the Z w axis of the world coordinates, which means Z w = 0. Therefore, (16) can be expressed as (17) According to (16), after obtaining the image point (u, v), the corresponding object point (X w, Y w, 0) in the measurement plane can be calculated. Then, the plane dimension of the components can be obtained as well. This procedure is presented in Table 1. Table 1. Algorithm of plane dimension detection Input 1: Feature points (u, v) detected from the image captured by the calibrated single camera through the transparent plate 2: The transparent plate parameters: n and d Output 3: The position of the object point (X w, Y w, 0) in the measurement plane Procedure 4: Calibrate camera parameters M in and M ex by Zhang's method without plate 5: For every single image point (u, v) 6: Calculate the displacement Δ2 by (4) 7: Compute the flat refractive matrix M g by (11) 8: Compute the new homography matrix by (14) 9: Obtain object points (X w, Y w, 0) in the measurement plane, corresponding to image point (u, v), by (17) 10: End for 5 Experimental evaluation The method of plane dimension detection described in Section 4 has been implemented in the experiments. The experiments are designed to evaluate the accuracy of planer size detection based on the proposed method. In the experiments, a single camera is selected as the imaging system which is calibrated without the transparent plate using Zhang's method, as shown in Fig. 3. The parameters of the imaging system are shown in Table 2. Fig. 3Open in figure viewerPowerPoint Calibration process of the single camera Table 2. Parameters of the imaging system The image resolution 3376 × 2704 the intrinsic parameters the extrinsic parameters Normal transparent glasses with the refractive index n = 1.52, parallel planar faces and different thicknesses (d = 3, 5, 10, 15, 20, 25, 30, 35, 40, 45 mm) are chosen to verify the accuracy of plane dimension detection based on the proposed method. The images are captured twice using a single camera, with the first captured directly and the second being captured by placing a glass with a certain thickness in front of the camera. For directly captured images, the object points are reconstructed using the traditional method. For the images captured by placing a glass in front of the camera, the proposed method of considering the plate refraction effect and the traditional method of not considering that are applied to reconstructing these selected points, respectively. The corner points on the checkerboard reconstructed with the traditional method can be used for the true position detection of the object points based on images captured directly. Then, the distance between these corresponding object points is reconstructed using the traditional method and the proposed method based on images captured by placing a glass in front of the camera, and the true position is computed as the measurement error. In the case of d = 3 mm, the measurement error of corner points on the checkerboard reconstructed by the different methods is shown in Fig. 4. The blue and red points are the object points reconstructed using the traditional method and the proposed method, respectively, where all the real positions of the object points are translated to the origin. Compared with the traditional method, the position of the corner points measured by the proposed method is closer to the true position, and the average error is reduced from 0.1502–0.0261 mm. Fig. 4Open in figure viewerPowerPoint Measurement error reconstructed by different methods In order to study the effect of the thickness of the glass on the accuracy of the proposed method, four checkerboards are arranged at the four corners of the camera's view field, where the refraction has the most significant impact because of the largest incident angle. The relationships between the measurement error and the thickness of the glass are shown in Fig. 5, where the average errors for all corner points are calculated. As shown in Fig. 5, the measurement error of the traditional model without considering the plate refraction effect is linear with the plate thickness, which is consistent with the conclusion above. More specifically, when the thickness of the plate is in the range of 3–45 mm, the measurement error using the traditional model is increased from 0.2902–to 4.7333 mm. In contrast, in the same condition of plate thickness, the measurement error of the proposed model is stable and remains <0.21 mm. This means that the proposed model can reconstruct the object points with high accuracy compared with the traditional model, under the influence of the plate refraction effect. Fig. 5Open in figure viewerPowerPoint Relationship between the measurement error and the glass thickness It can be summarised that when there is a plate refraction effect in the imaging system, the proposed model eliminates that effect to solve the plane dimension detection problem and can achieve significantly more accurate results. Finally, there are other factors that affect the accuracy of plane dimension detection, such as the accuracy of the refractive index, thickness, degree of parallelism, and orientation of the plate. So, in practical application, it is very important to ensure that these parameters are as accurate as possible. 6 Conclusion In this paper, a practical camera model is established for an imaging system with the plate refraction effect, whose form is similar to the pinhole camera model. Based on the model, an effective backward-projection algorithm is proposed to realise plane dimension detection with the plate refraction effect. The experiment is designed to evaluate the accuracy of plane dimension detection based on the proposed method. The results show that the proposed method can obtain higher accuracy than the traditional method without considering the plate refraction effect. It is demonstrated that the method can eliminate the influence of the refraction effect caused by the transparent plate with parallel planar faces. Hence, the model can mainly be used to improve the precision of plane dimension detection by the plate imaging system applied in harsh environments. 7 Acknowledgments The work was supported by National Natural Science Foundation of China (grant number 51604024), the Tribology Science Fund of State Key Laboratory of Tribology (grant number SKLTKF16B11), Beijing Natural Science Foundation (grant number 3182026), and Fundamental Research Funds for the Central Universities (grant number FRF-TP-17-002A2). 8 References 1Ngo N.-V., Hsu Q.-C., Hsiao Q.-C. et al.: ‘Development of a simple three-dimensional machine-vision measurement system for in-process mechanical parts’, Adv. Mech. Eng., 2017, 9, (10), pp. 1 – 11 2Sun Q., Hou Y., Tan Q. et al.: ‘A planar-dimensions machine vision measurement method based on lens distortion correction’, Scientific World J., 2013, 2017, Article ID 963621, p. 6 3Zhao D., Feng W., Sun G. et al.: ‘High precision measurement system of micro-electronic connector based on machine vision’, J. Appl. Sci., 2013, 13, (22), pp. 5363 – 5369 4Shi L., Ren H., Wang J. et al.: ‘Latin square design for chip length machine vision measurement system analysis’, Qual. Eng., 2016, 28, (4), pp. 381 – 387 5Xu G., Zhang Z.: ‘ Epipolar geometry in stereo, motion and object recognition: a unified approach’ ( Springer Science & Business Media, New York, NY, USA, 2013) 6Hall E.L., Tio J.B.K., McPherson C.A. et al.: ‘Measuring curved surfaces for robot vision’, Computer. (Long. Beach. Calif), 1982, 15, (12), pp. 42 – 54 7Raghavendra R., Yang B., Raja K.B. et al.: ‘A new perspective – face recognition with light-field camera’. 2013 Int. Conf. IEEE Biometrics (ICB), Madrid, Spain, 2013, pp. 1 – 8 8Huang L., Liu Y.: ‘Camera calibration for plate refractive imaging system’. Proc. 22nd Int. Conf. Pattern Recognition (ICPR), Stockholm, Sweden, August 2014, pp. 4068 – 4073 9Kwon Y.-H., Casebolt J.B.: ‘Effects of light refraction on the accuracy of camera calibration and reconstruction in underwater motion analysis’, Sports Biomech., 2006, 5, (2), pp. 315 – 340 10Treibitz T., Schechner Y., Kunz C. et al.: ‘Flat refractive geometry’, IEEE Trans. Pattern Anal. Mach. Intell., 2012, 34, (1), pp. 51 – 65 11Yamashita A., Fujii A., Kaneko T.: ‘Three dimensional measurement of objects in liquid and estimation of refractive index of liquid by using images of water surface with a stereo vision system’. Proc. 2008 IEEE Int. Conf. Robotics and Automation, Pasadena, California, USA, May 2008, pp. 974 – 979 12Gedge J., Gong M., Yang Y.-H.: ‘Refractive epipolar geometry for underwater stereo matching’. Proc. 2011 Canadian Conf. Computer and Robot Vision (CRV), St. Johns, NL, Canada, May 2011, pp. 146 – 152 13Gao C., Ahuja N.: ‘A refractive camera for acquiring stereo and super-resolution images’. Proc. 2006 IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), New York, USA, June 2006, vol. 2, pp. 2316 – 2323 14Chen Z., Wong K.-Y.K., Matsushita Y. et al.: ‘Depth from refraction using a transparent medium with unknown pose and refractive index’, Int. J. Comput. Vis., 2013, 102, (1–3), pp. 3 – 17 15Agrawal A., Ramalingam S., Taguchi Y. et al.: ‘A theory of multi-layer flat refractive geometry’. Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA, June 2012, pp. 3346 – 3353 16Shimizu M., Okutomi M.: ‘Reflection stereo-novel monocular stereo using a transparent plate’. Proc. 3rd Canadian Conf. Computer and Robot Vision (CRV), Quebec, Canada, June 2006, pp. 14 17Shimizu M., Okutomi M.: ‘Monocular range estimation through a double-sided half-mirror plate’. Proc. 4th Canadian Conf. Computer and Robot Vision (CRV), Quebec, Canada, May 2007, pp. 347 – 354 18Shimizu M., Okutomi M.: ‘Calibration and rectification for reflection stereo’. Proc. 2008 IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Anchorage, AK, USA, June 2008, pp. 1 – 8 19Shortis M., Harvey E., Seager J.: ‘A review of the status and trends in underwater video metric measurement’. Proc. IS&T/SPIE Electronic Imaging, San Jose, CA, USA, February 2007, pp. 64 – 91 20Parazzoli C G., Greegor R B., Li K. et al.: ‘Experimental verification and simulation of negative index of refraction using Snell's law’, Phys. Rev. Lett., 2003, 90, (10), p. 107401 21Zhang Z.: ‘A flexible new technique for camera calibration’, IEEE Trans. Pattern Anal. Mach. Intell., 2000, 22, (11), pp. 1330 – 1334 22Chang Y.-J., Chen T.: ‘Multi-view 3D reconstruction for scenes under the refractive plane with known vertical direction’. Proc. 2011 Int. Conf. Computer Vision, Barcelona, Spain, November 2011, pp. 351 – 358 23Glaeser G., Schröcker H.P.: ‘Reflections on refractions’, J. Geom. Graphics, 2000, 4, (1), pp. 1 – 18 24Huang L., Zhao X., Cai S. et al.: ‘Plate refractive camera model and its applications’, J. Electron. Imaging, 2017, 26, (2), Article ID 023020-1-16 Volume2019, Issue3March 2019Pages 647-651 FiguresReferencesRelatedInformation