# FastQR: Fast Pose Estimation of Objects Based on Multiple QR Codes and Monocular Vision in Mobile Embedded Devices

Authors: Yuheng Yan; Yiqiu Liang; Zihan Zhou; Bin Jiang; Jian Xiao
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

In recent years, the pose estimation of objects has become a research hotspot. This technique can effectively estimate the pose changes of objects in space and is widely used in many mobile devices, such as AR/VR. At present, mainstream technologies can achieve high-precision pose estimation, but the problem of that of multiple irregular objects in mobile and embedded devices under limited resource conditions is still challenging. In this paper, we propose a FastQR algorithm that can estimate the pose of multiple irregular objects on Renesas by utilizing homography method to solve the transformation matrix of a single QR code and then establish the spatial constraint relationship between multiple QR codes to estimate the posture of irregular objects. Our algorithm obtained a competitive result in simulation and verification on the RZ/A2M development board of Renesas. Moreover, the verification results show that our method can estimate the spatial pose of the multiobject accurately and robustly in distributed embedded devices. The average frame rate calculated on the RZ/A2M can reach 28 fps, which is at least 37 times faster than that of other pose estimation methods.


---

## Body

## 1. Introduction

With the development of AR/VR and control technology, object pose estimation is needed in manifold scenes. A large number of low-cost small embedded platforms also need to carry out object spatial pose estimation, such as in the exhibition, education, and other fields [1, 2]. At present, most of the estimation of object pose depends on the binocular camera [3] or RGB-D camera [4]. This type of camera is large and requires a computing platform with a certain amount of computing power, which is not conducive to the installation on small devices. There are also inertial sensors on objects to complete pose detection [5] or the comprehensive use of IMU, binocular camera, and RGB-D camera [6], but this method has certain difficulties in data fusion and filtering. Moreover, using convolution neural network to estimate the position and pose of objects [7] has also become a research focus, which brings accuracy, but requires high computational power of the platform, and even needs to be accelerated by GPU.In many scenarios, like AR, it is necessary to realize the fast pose estimation of objects on small portable devices, so the pose estimation of objects needs a convenient, fast, and low-cost solution. With the development and popularization of QR code technology [8] in recent years, QR code has been widely used in various aspects, such as QR code payment and QR code business cards. Most of these applications use the function of QR codes carrying text information, but at the same time, QR codes can also provide accurate pose information. At present, in the existing research, it has been possible to use QR codes as visual markers, utilizing the position, number, and other information stored in the QR code to complete the navigation [9, 10] or using the calculated position and attitude information to estimate the position and pose of the camera [11].This paper proposes a FastQR algorithm that can estimate the pose of multiple irregular objects on Renesas by utilizing QR codes and a monocular camera and then estimate the posture of irregular objects. This method only requires low resolution camera, which reduces the amount of computation with the characteristics of low cost, high accuracy, good robustness, low computing power requirements, fast speed, and so on, and making the multiobject pose estimation has a wider range of application scenarios. The results also verify the reliability of our method. At the same time, the highest frame rate calculated on the Renesas RZ/A2M development board can reach 45 fps, and the average frame rate can reach 28 fps.The contributions of this paper are as follows:(1)
It is proposed to use QR codes to realize a high frame rate and high accuracy pose estimation on equipment with limited resources(2)
It is possible to estimate the pose of multiple objects simultaneously under the premise of ensuring the frame rate and accuracy(3)
The proposed algorithm is tested on the simulation platform and the real object, and the result is competitive


## 2. The Principle of Position and Pose Estimation of Objects

### 2.1. Identification of QR Codes

The recognition and decoding algorithm of the QR code used in our system is AprilTag [12]. To further improve the recognition speed, the Dynamically Reconfigurable Processor (DRP) [13] on RZ/A2M is used to further accelerate the image processing steps, including adaptive binarization, Gaussian blur, and edge detection in the process of QR code recognition.


### 2.2. Solution of Position and Pose of QR Codes

Using the AprilTag recognition algorithm, the coordinates of the QR code in the image coordinate system can be found out. Meanwhile, the transformation matrixH=h00h01h02h10h11h12h20h21h22 [14] from space coordinates of the QR code to plane coordinates can be obtained.The projection matrix of the camera can be expressed asP=KR∣T, which means point X in the space can be transformed into a pixel on the image through the matrix. The rotation matrix is R=r00r01r02r10r11r12r20r21r22. The translational vector is T=txtytz. The camera internal parameter matrix is K=fx0cx0fycy001.P is the projection matrix. In camera internal parameter matrix K, fx and fy represent the focal length of the camera in the x and y directions. cx and cy represent the position of the intersection of the camera’s principal axis on the image plane. At the same time, it is set that the plane where the QR code is located is on plane X−YZ=0, of which the center is the origin of the coordinate. Hence, the following is concluded as:
(1)x=KR∣TXY01.To simplify the calculation, removing the third column ofR in formula (1) can have
(2)x=Kr0r1TXY1.r0 and r1 in formula (2) is the first and second columns of R.The homography matrixH from the point on the space plane to the point on the image is constructed.Suppose(3)XY1=λ000λ000λX′Y′1,where λ represents the scaling ratio, of which X′Y′1T is the corner of the zoom unit QR code; the following can be concluded:
(4)x=Kr0r1Tλ000λ0001XY1.It is known that the internal parameter matrix of the camera isK and the single transformation matrix is H. Let Kλr0λr1T=Kr0′r1′T, the equation can be concluded as the following:
(5)fxr00′+cxr20′=h00,fxr01′+cxr21′=h01,fxtx+cxtz=h02,fyr10′+cyr20′=h10,fyr11′+cyr21′=h11,fyty+cytz=h12,r20′=h20,r21′=h21,tz=h22.r0′r1′T can be got, and then, the unit is dealt with the following steps:
(6)r0′′=r0′r0′r1′,r1′′=r1′r0′r1′,T′′=Tr0′r1′.It is shown thatr0″ and r1″ after the dealt are the same as r0 and r1, while T and T′ are different from each other. For the same QR code on the camera image, T represents the translational vector of the actual size of the QR code in the direction of R, and T represents the translational vector of the QR code whose actual size is a square in the same direction. Because of the proportional scaling of the same QR code square in the direction, the translational vector from the camera to the actual QR code can be calculated by the size of the actual one. If the width of the QR code is ω, the translational vector from the camera to the actual QR code is T=ωT′. The translational vector T is also the coordinator of the QR code center in the camera coordinate system.


### 2.3. Position and Pose Transformation from Multiple QR Codes Coordinate System to Object Coordinate System

By pasting multiple QR codes on an object, the omnidirectional pose estimation of the object can be realized, and the QR code can be avoided being blocked from vision. Meanwhile, according to the position relation of multiple QR codes on the same object, the overall pose information of a single QR code can be obtained from one of a single QR code. The rotation matrixR and translational vector T mentioned in Section 2.2 are the rotation and shift transformation of the QR code coordinate system. In order to map the pose of the QR code to the pose of the object, it is necessary to convert the QR code into the object coordinate base vector group.As for the pose of the object, it is assumed that the six faces of a hexahedral are affixed with QR codes, respectively, and the center of the QR code coincides with the center of each surface, as shown in Figure1 below. Suppose the basis vectors of the object coordinate system X, Y, and Z are ex⇀,ey⇀,ez⇀ and only consider the change of the direction of the base vector of the coordinate system.Figure 1
The model of tag cube. (a) Schematic diagram of the QR code and object coordinate system. (b) The model used for testing the pose measurement.
(a)(b)When the coordinate system of the QR code is consistent with the direction of axisX, Y, and Z of the object coordinate system, as shown in the front of the stereo in Figure 1, suppose the base vectors of the three axes under the object coordinate system are ex0⇀,ey0⇀,ez0⇀, the rotation matrix can directly represent the pose of the object, that is
(7)ex0⇀ex0⇀ex0⇀=100010001ex⇀ex⇀ex⇀.When the coordinate system of QR code is inconsistent with the direction of axisX, Y, and Z of the object coordinate system, as shown on the right of the stereo in Figure 1, suppose the base vectors of the three axes under the object coordinate system are the direction of the Y-axis of the QR code coordinate system is consistent with the direction of the Y-axis of the object coordinate system; the direction of the X-axis of the QR code coordinate system is opposite to the one of the Z-axis of the object coordinate system, and the direction of the Z-axis of the QR code coordinate system is consistent with the direction of the X-axis of the object coordinate system, that is
(8)ex1⇀ex1⇀ex1⇀=001010−100ex⇀ey⇀ez⇀.Similarly, when the other surfaces of the QR code are recognized, the pose of the object can be obtained from any angle according to the relationship between the known QR code coordinate system and the object coordinate system.As for the position of the object, suppose the coordinate of a vector in the QR code coordinate system isx0,y0,z0T, and the coordinate in the camera coordinate system is x0′,y0′,z0′T, the convert relationship can be obtained as follows:
(9)x0′y0′z0′=Rx0y0z0+T.The convert relationship can be obtained as follows ifx0′,y0′,z0′T is transformed into x0,y0,z0T:
(10)x0y0z0=R−1x0′y0′z0′−T.The coordinate of the object center in the camera coordinate system can be obtained by formula (9). For example, in Figure 1, the coordinate of the object center in the QR code coordinate system can be regarded as 0,0,−ω/2T, of which ω stands for the edge length of the pasted QR code. The coordinate of the object center in the camera coordinate system can be acquired by the transformation of formula (11). The transformation from any point in the QR code coordinate system to the camera coordinate system can also be obtained by the transformation of formula (9), so the transformation from the base vector set of the QR code coordinate system to the camera coordinate system can be obtained.
(11)x0′y0′z0′=R00−ω/2+T.From the analysis above, what is known is that the spatial pose of a three-dimensional object can be estimated through the relative position of the position and pose information of the identified QR code and multiple QR codes on the known object.


### 2.4. Multiple QR Code Pose Estimation of Irregular Rigid Bodies

Considering that it is difficult to get the transformation relationship from QR code coordinate system to object coordinate system on an irregular rigid body, the system can calculate the transformation relationship among QR codes according to the QR code pasted on the object, and the concrete principle is as follows.By pasting multiple QR codes on an object, it is necessary to ensure that each QR code can be recognized along with another QR code at a certain angle of view, which is called “connection,” and that a QR code can be “connected” to all QR codes according to the above conditions, and the coordinate system of one QR code is used as the object coordinate system. The transformation relationship between the coordinate system base vector group of the QR code connected to the QR code and the base vector group of the QR code coordinate system is calculated by taking one of the QR codes in a coordinate system as the object coordinate system.Again, taking the cube in Figure1 as an example, remember that the QR code on the front is A and the QR code on the right, back, left, top, and bottom are B, C, D, E, and F, respectively. Firstly, the camera faces the QR code A on the front of the cube, where the coordinate system is regarded as the object coordinate system to rotate the cube to the left. When the camera rotates to about 45 degrees, the QR codes marked A, and B can be both recognized, and the transformation relationship from QR code A coordinate system to QR code A coordinate system can be got. To simplify the calculation, the pose and position transformations are considered separately, which means only the change of the base vector direction of the coordinate system is considered when the position transformation is carried out, and only the change of the origin of the coordinate system is considered when the position transformation is carried out.As for the origin of the coordinate system, the coordinate of the origin of QR codeA coordinate system under the coordinate system of QR code B is T0−T1. For the coordinate system, the following transformation formula can be obtained:
(12)ex0⇀=R0R1−1ex1⇀,ey0⇀=R0R1−1ey1⇀,ey0⇀=R0R1−1ez1⇀.R0 and T0 represent the rotation matrix and shift vector of QR code A, and ex0⇀,ey0⇀,ez0⇀ is the representation of the base vector group of QR code A coordinate system in the camera coordinate system; R1 and T1 represent the rotation matrix and shift vector of QR code B, and ex1⇀,ey1⇀,ez1⇀ is the representation of the base vector group of QR code B coordinate system in the camera coordinate system.Keeping rotating the cube until QR codeC and QR code B can be both recognized. For the origin of the coordinate system, the coordinate of the origin of the coordinate system of the QR code B under the coordinate system of the QR code C is T1′−T2. Combined with T0−T1, the coordinate of the origin of the coordinate system of QR code 0 under the coordinate system of the QR code B and the coordinate of the origin of the coordinate system of QR code A under the coordinate system of the QR code C can be obtained: T1′−T2+T0−T1. According to formula (9), the coordinate of the QR code A coordinate system in the camera coordinate system can be acquired. Similarly, the transformation relationship from QR code C coordinate system to the QR code B coordinate system can be obtained from formula (13).
(13)ex1⇀′=R1′R2−1ex2⇀,ey1⇀′=R1′R2−1ey2⇀,ez1⇀′=R1′R2−1ez2⇀.R1′ and T1′ are the rotation matrix and shift vector of QR code B, and ex1⇀′,ey1⇀′,ez1⇀′ is the representation of the base vector group of QR code B coordinate system in the camera coordinate system. Combined with formula (12), the transformation relationship from QR code C coordinate system to QR code B coordinate system can be obtained, and the rotation matrix is R0R1−1R1′R2−1. Continue to rotate the cube, the transformation relationship from all the QR code coordinate systems on the object to QR code A coordinate system can be obtained.The specific initialization process goes as follows: firstly, enter the number of QR codes posted on the object and the number of objects, then initialize each object one by one. Align any QR code on the first object with the camera, and the system will record the QR code coordinate system as the object coordinate system; rotate the object slowly so that the system can recognize all the QR codes posted on the object, and each new QR code needs to be recognized at the same time as an existed QR code. According to the coordinate system transformation relationship between each new QR code and the existed QR code that has been identified with it as well as the transformation relationship between the coordinate system of the existed QR code and the object coordinate system, the transformation relationship between the new QR code and the object coordinate system is obtained and recorded in the system through the principle of the above transformation relationship, and the initialization of an object is completed.


## 2.1. Identification of QR Codes

The recognition and decoding algorithm of the QR code used in our system is AprilTag [12]. To further improve the recognition speed, the Dynamically Reconfigurable Processor (DRP) [13] on RZ/A2M is used to further accelerate the image processing steps, including adaptive binarization, Gaussian blur, and edge detection in the process of QR code recognition.


## 2.2. Solution of Position and Pose of QR Codes

Using the AprilTag recognition algorithm, the coordinates of the QR code in the image coordinate system can be found out. Meanwhile, the transformation matrixH=h00h01h02h10h11h12h20h21h22 [14] from space coordinates of the QR code to plane coordinates can be obtained.The projection matrix of the camera can be expressed asP=KR∣T, which means point X in the space can be transformed into a pixel on the image through the matrix. The rotation matrix is R=r00r01r02r10r11r12r20r21r22. The translational vector is T=txtytz. The camera internal parameter matrix is K=fx0cx0fycy001.P is the projection matrix. In camera internal parameter matrix K, fx and fy represent the focal length of the camera in the x and y directions. cx and cy represent the position of the intersection of the camera’s principal axis on the image plane. At the same time, it is set that the plane where the QR code is located is on plane X−YZ=0, of which the center is the origin of the coordinate. Hence, the following is concluded as:
(1)x=KR∣TXY01.To simplify the calculation, removing the third column ofR in formula (1) can have
(2)x=Kr0r1TXY1.r0 and r1 in formula (2) is the first and second columns of R.The homography matrixH from the point on the space plane to the point on the image is constructed.Suppose(3)XY1=λ000λ000λX′Y′1,where λ represents the scaling ratio, of which X′Y′1T is the corner of the zoom unit QR code; the following can be concluded:
(4)x=Kr0r1Tλ000λ0001XY1.It is known that the internal parameter matrix of the camera isK and the single transformation matrix is H. Let Kλr0λr1T=Kr0′r1′T, the equation can be concluded as the following:
(5)fxr00′+cxr20′=h00,fxr01′+cxr21′=h01,fxtx+cxtz=h02,fyr10′+cyr20′=h10,fyr11′+cyr21′=h11,fyty+cytz=h12,r20′=h20,r21′=h21,tz=h22.r0′r1′T can be got, and then, the unit is dealt with the following steps:
(6)r0′′=r0′r0′r1′,r1′′=r1′r0′r1′,T′′=Tr0′r1′.It is shown thatr0″ and r1″ after the dealt are the same as r0 and r1, while T and T′ are different from each other. For the same QR code on the camera image, T represents the translational vector of the actual size of the QR code in the direction of R, and T represents the translational vector of the QR code whose actual size is a square in the same direction. Because of the proportional scaling of the same QR code square in the direction, the translational vector from the camera to the actual QR code can be calculated by the size of the actual one. If the width of the QR code is ω, the translational vector from the camera to the actual QR code is T=ωT′. The translational vector T is also the coordinator of the QR code center in the camera coordinate system.


## 2.3. Position and Pose Transformation from Multiple QR Codes Coordinate System to Object Coordinate System

By pasting multiple QR codes on an object, the omnidirectional pose estimation of the object can be realized, and the QR code can be avoided being blocked from vision. Meanwhile, according to the position relation of multiple QR codes on the same object, the overall pose information of a single QR code can be obtained from one of a single QR code. The rotation matrixR and translational vector T mentioned in Section 2.2 are the rotation and shift transformation of the QR code coordinate system. In order to map the pose of the QR code to the pose of the object, it is necessary to convert the QR code into the object coordinate base vector group.As for the pose of the object, it is assumed that the six faces of a hexahedral are affixed with QR codes, respectively, and the center of the QR code coincides with the center of each surface, as shown in Figure1 below. Suppose the basis vectors of the object coordinate system X, Y, and Z are ex⇀,ey⇀,ez⇀ and only consider the change of the direction of the base vector of the coordinate system.Figure 1
The model of tag cube. (a) Schematic diagram of the QR code and object coordinate system. (b) The model used for testing the pose measurement.
(a)(b)When the coordinate system of the QR code is consistent with the direction of axisX, Y, and Z of the object coordinate system, as shown in the front of the stereo in Figure 1, suppose the base vectors of the three axes under the object coordinate system are ex0⇀,ey0⇀,ez0⇀, the rotation matrix can directly represent the pose of the object, that is
(7)ex0⇀ex0⇀ex0⇀=100010001ex⇀ex⇀ex⇀.When the coordinate system of QR code is inconsistent with the direction of axisX, Y, and Z of the object coordinate system, as shown on the right of the stereo in Figure 1, suppose the base vectors of the three axes under the object coordinate system are the direction of the Y-axis of the QR code coordinate system is consistent with the direction of the Y-axis of the object coordinate system; the direction of the X-axis of the QR code coordinate system is opposite to the one of the Z-axis of the object coordinate system, and the direction of the Z-axis of the QR code coordinate system is consistent with the direction of the X-axis of the object coordinate system, that is
(8)ex1⇀ex1⇀ex1⇀=001010−100ex⇀ey⇀ez⇀.Similarly, when the other surfaces of the QR code are recognized, the pose of the object can be obtained from any angle according to the relationship between the known QR code coordinate system and the object coordinate system.As for the position of the object, suppose the coordinate of a vector in the QR code coordinate system isx0,y0,z0T, and the coordinate in the camera coordinate system is x0′,y0′,z0′T, the convert relationship can be obtained as follows:
(9)x0′y0′z0′=Rx0y0z0+T.The convert relationship can be obtained as follows ifx0′,y0′,z0′T is transformed into x0,y0,z0T:
(10)x0y0z0=R−1x0′y0′z0′−T.The coordinate of the object center in the camera coordinate system can be obtained by formula (9). For example, in Figure 1, the coordinate of the object center in the QR code coordinate system can be regarded as 0,0,−ω/2T, of which ω stands for the edge length of the pasted QR code. The coordinate of the object center in the camera coordinate system can be acquired by the transformation of formula (11). The transformation from any point in the QR code coordinate system to the camera coordinate system can also be obtained by the transformation of formula (9), so the transformation from the base vector set of the QR code coordinate system to the camera coordinate system can be obtained.
(11)x0′y0′z0′=R00−ω/2+T.From the analysis above, what is known is that the spatial pose of a three-dimensional object can be estimated through the relative position of the position and pose information of the identified QR code and multiple QR codes on the known object.


## 2.4. Multiple QR Code Pose Estimation of Irregular Rigid Bodies

Considering that it is difficult to get the transformation relationship from QR code coordinate system to object coordinate system on an irregular rigid body, the system can calculate the transformation relationship among QR codes according to the QR code pasted on the object, and the concrete principle is as follows.By pasting multiple QR codes on an object, it is necessary to ensure that each QR code can be recognized along with another QR code at a certain angle of view, which is called “connection,” and that a QR code can be “connected” to all QR codes according to the above conditions, and the coordinate system of one QR code is used as the object coordinate system. The transformation relationship between the coordinate system base vector group of the QR code connected to the QR code and the base vector group of the QR code coordinate system is calculated by taking one of the QR codes in a coordinate system as the object coordinate system.Again, taking the cube in Figure1 as an example, remember that the QR code on the front is A and the QR code on the right, back, left, top, and bottom are B, C, D, E, and F, respectively. Firstly, the camera faces the QR code A on the front of the cube, where the coordinate system is regarded as the object coordinate system to rotate the cube to the left. When the camera rotates to about 45 degrees, the QR codes marked A, and B can be both recognized, and the transformation relationship from QR code A coordinate system to QR code A coordinate system can be got. To simplify the calculation, the pose and position transformations are considered separately, which means only the change of the base vector direction of the coordinate system is considered when the position transformation is carried out, and only the change of the origin of the coordinate system is considered when the position transformation is carried out.As for the origin of the coordinate system, the coordinate of the origin of QR codeA coordinate system under the coordinate system of QR code B is T0−T1. For the coordinate system, the following transformation formula can be obtained:
(12)ex0⇀=R0R1−1ex1⇀,ey0⇀=R0R1−1ey1⇀,ey0⇀=R0R1−1ez1⇀.R0 and T0 represent the rotation matrix and shift vector of QR code A, and ex0⇀,ey0⇀,ez0⇀ is the representation of the base vector group of QR code A coordinate system in the camera coordinate system; R1 and T1 represent the rotation matrix and shift vector of QR code B, and ex1⇀,ey1⇀,ez1⇀ is the representation of the base vector group of QR code B coordinate system in the camera coordinate system.Keeping rotating the cube until QR codeC and QR code B can be both recognized. For the origin of the coordinate system, the coordinate of the origin of the coordinate system of the QR code B under the coordinate system of the QR code C is T1′−T2. Combined with T0−T1, the coordinate of the origin of the coordinate system of QR code 0 under the coordinate system of the QR code B and the coordinate of the origin of the coordinate system of QR code A under the coordinate system of the QR code C can be obtained: T1′−T2+T0−T1. According to formula (9), the coordinate of the QR code A coordinate system in the camera coordinate system can be acquired. Similarly, the transformation relationship from QR code C coordinate system to the QR code B coordinate system can be obtained from formula (13).
(13)ex1⇀′=R1′R2−1ex2⇀,ey1⇀′=R1′R2−1ey2⇀,ez1⇀′=R1′R2−1ez2⇀.R1′ and T1′ are the rotation matrix and shift vector of QR code B, and ex1⇀′,ey1⇀′,ez1⇀′ is the representation of the base vector group of QR code B coordinate system in the camera coordinate system. Combined with formula (12), the transformation relationship from QR code C coordinate system to QR code B coordinate system can be obtained, and the rotation matrix is R0R1−1R1′R2−1. Continue to rotate the cube, the transformation relationship from all the QR code coordinate systems on the object to QR code A coordinate system can be obtained.The specific initialization process goes as follows: firstly, enter the number of QR codes posted on the object and the number of objects, then initialize each object one by one. Align any QR code on the first object with the camera, and the system will record the QR code coordinate system as the object coordinate system; rotate the object slowly so that the system can recognize all the QR codes posted on the object, and each new QR code needs to be recognized at the same time as an existed QR code. According to the coordinate system transformation relationship between each new QR code and the existed QR code that has been identified with it as well as the transformation relationship between the coordinate system of the existed QR code and the object coordinate system, the transformation relationship between the new QR code and the object coordinate system is obtained and recorded in the system through the principle of the above transformation relationship, and the initialization of an object is completed.


## 3. Verification Method and Environment

### 3.1. Verification Preparation

The system platform adopts Renesas RZ/A2M development board, uses rza_linux-4.19 provided by Renesas with C programming language, which connects to the RGB display through the HDM interface on the board, and sends the pose data to another PC through the network interface on the board to display the pose model. The camera has a Raspberry pi cameraV2 with a resolution of 640∗ 480. The QR code adopts Tag36H11 code in AprilTag. Camera calibration and postdata processing use Matlab2019, and the pose system model display uses Processing3. The system structure diagram is shown in Figure 2.Figure 2
Systematic structure diagram.The validation environment is built, as shown in Figure3 below. The validation environment includes a Raspberry pi camera V2, Renesas RZ/A2M development board, an LCD display, Processing3 display, and the objects pasted with QR codes.Figure 3
Verification environment.The software flow chart of the system platform is shown in Figure4. The software mainly includes camera calibration, QR code detection, transformation matrix calculation, and object pose estimation.Figure 4
Software flow chart.


### 3.2. Means of Verification

When checking the position measurement, the QR code is 4 cm∗ 4 cm, and it is pasted on a regular hexahedral surface of 6 cm∗6 cm. The measurement accuracy is obtained by measuring the position coordinates of the object in the camera coordinate system and comparing it with the position coordinates obtained by the system. It is also necessary to calibrate the position of the camera before testing the attitude measurement, display a QR code in the center of the display, and place the camera in the position of 1.2 m in front of the display. Adjust the angle of the camera until the angle of the measured shaft angle is 0.In the process of checking the pose measurement, in light of the spatial pose of the object is difficult to measure accurately, Processing3 is used to display a regular hexahedral model with 6 cm the edge length of the model. Different QR codes are displayed on the six faces of the regular hexahedral, and the axial angle of the model pose is brought out while adjusting the model. The accuracy of the measurement is obtained by comparing the pose of both the model and the measurement. The model is shown on Processing3, rotating the model around the arrow direction shown in Figure1.


## 3.1. Verification Preparation

The system platform adopts Renesas RZ/A2M development board, uses rza_linux-4.19 provided by Renesas with C programming language, which connects to the RGB display through the HDM interface on the board, and sends the pose data to another PC through the network interface on the board to display the pose model. The camera has a Raspberry pi cameraV2 with a resolution of 640∗ 480. The QR code adopts Tag36H11 code in AprilTag. Camera calibration and postdata processing use Matlab2019, and the pose system model display uses Processing3. The system structure diagram is shown in Figure 2.Figure 2
Systematic structure diagram.The validation environment is built, as shown in Figure3 below. The validation environment includes a Raspberry pi camera V2, Renesas RZ/A2M development board, an LCD display, Processing3 display, and the objects pasted with QR codes.Figure 3
Verification environment.The software flow chart of the system platform is shown in Figure4. The software mainly includes camera calibration, QR code detection, transformation matrix calculation, and object pose estimation.Figure 4
Software flow chart.


## 3.2. Means of Verification

When checking the position measurement, the QR code is 4 cm∗ 4 cm, and it is pasted on a regular hexahedral surface of 6 cm∗6 cm. The measurement accuracy is obtained by measuring the position coordinates of the object in the camera coordinate system and comparing it with the position coordinates obtained by the system. It is also necessary to calibrate the position of the camera before testing the attitude measurement, display a QR code in the center of the display, and place the camera in the position of 1.2 m in front of the display. Adjust the angle of the camera until the angle of the measured shaft angle is 0.In the process of checking the pose measurement, in light of the spatial pose of the object is difficult to measure accurately, Processing3 is used to display a regular hexahedral model with 6 cm the edge length of the model. Different QR codes are displayed on the six faces of the regular hexahedral, and the axial angle of the model pose is brought out while adjusting the model. The accuracy of the measurement is obtained by comparing the pose of both the model and the measurement. The model is shown on Processing3, rotating the model around the arrow direction shown in Figure1.


## 4. Results

### 4.1. Results and Errors of Object Pose Estimation

Considering that the edge length of QR code is 4 cm, the change range ofz-axis direction is between 0.1 m and 3.0 m when x=y=0.0 m, and the step size is 0.1 m with 30 groups of data. In light of the angle and view of the camera, when z=1.2 m, the error of the x-axis and y-axis is measured, and the range of variation is between -0.50 m and 0.50 m with 21 groups of data. The error of the angle is calculated by, respectively, calculating the angle between the axis of the real value and the axis of the measured value and by making the difference. The results are shown in Figure 5.Figure 5
Pose estimation error. (a) The error of position estimation. (b) The error of attitude.
(a)(b)From the results, it can be seen that the error between the real value and the measured one is so small that the error in thez-direction is not more than 9 cm, and the error in the x and y directions is not more than 5 cm. The angle error between the true value of the axis angle and the measured one is not more than 7°, and the angle error is not more than 5°. Due to the limitation of QR code size, camera resolution, and illumination, the accuracy of pose estimation is only more accurate, and the error will increase to a certain extent with the increase of the distance from the object to the camera.


### 4.2. The Estimated Velocity of Object Position

The system performance is average in terms of accuracy [15], but it can already meet the needs of 6DoF pose estimation for objects in most cases. In terms of the calculation speed, the system has very excellent performance, and its highest main frequency of the Renesas RZ/A2M development board used in this system is 528 MHz, and its captured image resolution is 640 ∗ 480. The average time of object estimation is shown in Table 1 below. When using DRP on RZ/A2M to accelerate some image processing steps, the fastest time for pose estimation is about 22 ms.Table 1
The average time spent on object estimation.
Number of QR codesQR code recognition time (ms)Pose solution time (ms)Total calculation time (ms)Average frame rate (fps)1121022221153628.573262147Compared with most of the methods that do not use visual markers to estimate the pose of objects, this method requires less calculation power and does not need to be accelerated by GPU. By using the QR code as the visual marker, the difficulty of finding feature points is avoided, and the resolution of the input image is less required. It can be implemented on MPU or CPU with low main frequency and can still maintain a smooth and available frame rate of about 28 Hz. The velocity comparison with other object pose estimation methods is shown in Table2 below.Table 2
Velocity comparison with other object pose estimation methods.
NumberPose estimation methodCalculating platformAverage frame rate (fps)1FastQR (ours)Renesas RZ/A2M (with the DRP)282Estimation of the 3D pose of the object by 3D model of an object [16]A 2.83 GHz CPU (Q9550) and an NVIDIA GTX 280 GPU43Estimation of position and pose of the camera by AprilTag [17]PC84Pose estimation of objects using fusion visual and inertial data [18]A 3.60 GHz CPU and NVIDIA Cmadro 4000 GPU1056DoF pose estimation of objects by geometric information [19]16-core Intel(R) Xeon(R) E5-2637 CPU and GeForce GTX 1080 GPU306Seamless single shot 6D object pose prediction [20]Desktop with a Titan X GPU50


### 4.3. Position and Pose Estimation of Irregular Objects

Here, taking the XBOX handle as an example, seven QR codes are pasted on the handle to meet the requirements of Section2.4. Note that the QR code should be pasted smoothly to avoid affecting the recognition effect, and no duplicate QR code should be pasted. The pose estimation effect for single and multiple targets is shown in Figure 6 below.Figure 6
Effect diagram of the pose estimation of single and multiple irregular objects.


## 4.1. Results and Errors of Object Pose Estimation

Considering that the edge length of QR code is 4 cm, the change range ofz-axis direction is between 0.1 m and 3.0 m when x=y=0.0 m, and the step size is 0.1 m with 30 groups of data. In light of the angle and view of the camera, when z=1.2 m, the error of the x-axis and y-axis is measured, and the range of variation is between -0.50 m and 0.50 m with 21 groups of data. The error of the angle is calculated by, respectively, calculating the angle between the axis of the real value and the axis of the measured value and by making the difference. The results are shown in Figure 5.Figure 5
Pose estimation error. (a) The error of position estimation. (b) The error of attitude.
(a)(b)From the results, it can be seen that the error between the real value and the measured one is so small that the error in thez-direction is not more than 9 cm, and the error in the x and y directions is not more than 5 cm. The angle error between the true value of the axis angle and the measured one is not more than 7°, and the angle error is not more than 5°. Due to the limitation of QR code size, camera resolution, and illumination, the accuracy of pose estimation is only more accurate, and the error will increase to a certain extent with the increase of the distance from the object to the camera.


## 4.2. The Estimated Velocity of Object Position

The system performance is average in terms of accuracy [15], but it can already meet the needs of 6DoF pose estimation for objects in most cases. In terms of the calculation speed, the system has very excellent performance, and its highest main frequency of the Renesas RZ/A2M development board used in this system is 528 MHz, and its captured image resolution is 640 ∗ 480. The average time of object estimation is shown in Table 1 below. When using DRP on RZ/A2M to accelerate some image processing steps, the fastest time for pose estimation is about 22 ms.Table 1
The average time spent on object estimation.
Number of QR codesQR code recognition time (ms)Pose solution time (ms)Total calculation time (ms)Average frame rate (fps)1121022221153628.573262147Compared with most of the methods that do not use visual markers to estimate the pose of objects, this method requires less calculation power and does not need to be accelerated by GPU. By using the QR code as the visual marker, the difficulty of finding feature points is avoided, and the resolution of the input image is less required. It can be implemented on MPU or CPU with low main frequency and can still maintain a smooth and available frame rate of about 28 Hz. The velocity comparison with other object pose estimation methods is shown in Table2 below.Table 2
Velocity comparison with other object pose estimation methods.
NumberPose estimation methodCalculating platformAverage frame rate (fps)1FastQR (ours)Renesas RZ/A2M (with the DRP)282Estimation of the 3D pose of the object by 3D model of an object [16]A 2.83 GHz CPU (Q9550) and an NVIDIA GTX 280 GPU43Estimation of position and pose of the camera by AprilTag [17]PC84Pose estimation of objects using fusion visual and inertial data [18]A 3.60 GHz CPU and NVIDIA Cmadro 4000 GPU1056DoF pose estimation of objects by geometric information [19]16-core Intel(R) Xeon(R) E5-2637 CPU and GeForce GTX 1080 GPU306Seamless single shot 6D object pose prediction [20]Desktop with a Titan X GPU50


## 4.3. Position and Pose Estimation of Irregular Objects

Here, taking the XBOX handle as an example, seven QR codes are pasted on the handle to meet the requirements of Section2.4. Note that the QR code should be pasted smoothly to avoid affecting the recognition effect, and no duplicate QR code should be pasted. The pose estimation effect for single and multiple targets is shown in Figure 6 below.Figure 6
Effect diagram of the pose estimation of single and multiple irregular objects.


## 5. Conclusion

The system uses a monocular camera and multi-QR code to estimate the spatial pose of the object, which has extremely low cost, small volume, low computing power, fast speed, good robustness, and so on. This can meet the basic requirements for pose estimation. Also, at the expense of a certain precision, the calculation power requirement of object spatial pose estimation is lowered, and the calculation speed is greatly improved. It can be realized on many mobile embedded devices’ cores, such as MPU or CPU with low main frequency and small memory. Compared with the one using GPU to accelerate pose estimation, the method has a higher energy efficiency ratio and greatly expands the application scene of object pose estimation. This scheme can also be further extended to complex multijoint objects in distributed embedded computing regions.As future work, we plan to extend this method to achieve fast pose estimation on multijoint targets. By posting QR codes to each joint and connecting rod of multijoint objects, the spatial pose estimation of complex multijoint objects can be realized. We will further study to improve the accuracy of pose estimation without reducing the computing speed.



---

*Source: 9481190-2021-09-30.xml*