Proceedings of International Conference on
Technology and Social Science 2021 (ICTSS 2021)

Recognition of Lane Markings in Factories and Method for
Self-Position Estimation Using AR Markers
Kento Hisanaga1,a, Shiyuan Yang1,b, Seiichi Serikawa1,c,*1

Department of Electronic system, Kyushu Institute of Technology, 1-1 Sensui-cho, Tobata-ku,
Kitakyushu-shi, Fukuoka, 804-8550, JAPANa<kento.hisanaga426@mail.kyutech.jp>, b<yang@elcs.kyutech.ac.jp >,c<serikawa@elcs.kyutech.ac.jp>

Keywords: image processing, automatic transfer robots, AR marker, lane marking

Abstract. Currently, automations of factories are progressing, and the demand for introduction ofautomatic transfer robots is increasing. However, there are many factories that cannot introduce
automation of factories because of cost and time. In this study, we propose an automatic transfer robot
system using AR markers in addition to lane markings originally drawn in the factory, to reduce theabove problems. This makes it possible to flexibly determine and change the route without reattaching
the tape as a method using magnetic tape. Lane markings and AR markers are recognized in real timeusing a camera. Then, the distance and self-position are gotten by using the angle, position of the lane
marking and the AR marker. The self-position could be estimated from the information.
1. Introduction
An introduction of factory automation has a long history, and more than half a century has passed
since robots were well used in factories [1]. In recent years, needs of unmanned operation of factories
has increased, and a demand for automatic transfer robots is increasing. The advantages of automatictransfer robot include a reduction of cost such as labor cost and improvement of productivity. Moreover,in Japan, labor shortages due to declining birthrate and aging population are serious problem, and theneed of factory automation will increase more in the future. Currently, a method using magnetic tapeis often used as tape of line trace for automatic transfer robot. In this method, a magnetic tape isattached in advance on floor. A robot can correctly move to a destination, because line trace technology
is so simple and tough. However, it has some disadvantages. (1) If there is a layout change in thefactory, it is necessary to reattach the tape, then it takes time and cost. This greatly affects theproductivity. (2) If there is an obstacle on a path, the robot cannot avoid the obstacle [2],[3],[4].
To improve the problems, we propose an automatic transfer robot system which can estimate selfposition using AR markers in addition to lane markings originally drawn in the factory. This makes it
possible to change and add routes more flexibly than a method using magnetic tape.

2. Method
2.1 Method for detecting lane markings by image processingTo use robot system in real space, it must work real-time. Therefore, a simple processing ispreferable. In detecting lane marking, grayscale conversion is first performed. By the conversion, theamount of data is reduced to 1/3 of the original color image acquired from a camera. Next, histogram
flattening is performed on the converted image. By performing this process, the contrast of the image
can be increased, and the image can be easily analyzed. At 3rd, an edge detection and Hough transform

1

Proceedings of International Conference onTechnology and Social Science 2021 (ICTSS 2021)
are performed. Some candidates of lanes are detected by edge extraction. From the candidates, a lane
is detected more accurately using Hough transform.
2.2 Calculation of distance to lane markings
For calculating the distance from the lane marking, we explain a shooting range of the camera.
Figure 1 shows the range. Equations (1) and (2) are obtained from the image sensor size (Hi in thevertical direction and Wi in the horizontal direction), the focal length f of lens, the maximum shooting
range (Hmax in the vertical direction and Wmax in the horizontal direction), and length z between lensand an object.
𝐻𝑚𝑎𝑥 =𝑊𝑚𝑎𝑥 =𝑧 × 𝐻𝑖

(1)

𝑓𝑧 × 𝑊𝑖(2)

𝑓Next, Fig. 2 shows the positional relationship between camera and ground, and Fig. 3 shows anexample of an image after detecting lane markings. Let the height of the camera be Hc, the horizontal
distance from where the angle of view can be seen be z0. In the three-dimensional coordinate system,
the camera is placed on the y-axis form the origin, and it is placed in the z-axis direction. In Fig. 3,PICTURE_WIDTH is the number of pixels in the horizontal direction of image, andPICTURE_HEIGHT is the number of pixels in the horizontal direction. The starting point of the lane
marking at the bottom of the lane marking screen is p1(x1, y1), and the point where the lane marking is
detected at a position where p2 is shifted vertically by 20 [pixel] from p1. It means p2 (x2, y1-20). The
symbols 𝐻𝑐 , 𝐻𝑧1 , 𝐻𝑧2 ,𝑧0 , 𝑧1, 𝑧2 represent the length in the real world, and 𝐻𝑝1 , 𝐻𝑝2 , 𝑊𝑝1 , 𝑊𝑝1
show the number of pixels. The values of 𝐻𝑧1 and 𝐻𝑧2 are calculated by Eqs. (3) and (4).
𝐻𝑧1 = 𝐻𝑚𝑎𝑥
𝐻𝑧2 = 𝐻𝑚𝑎𝑥

𝑃𝐼𝐶𝑇𝑈𝑅𝐸_𝐻𝐸𝐼𝐺𝐻𝑇 – 𝑦1
𝑃𝐼𝐶𝑇𝑈𝑅𝐸_𝐻𝐸𝐼𝐺𝐻𝑇𝑃𝐼𝐶𝑇𝑈𝑅𝐸_𝐻𝐸𝐼𝐺𝐻𝑇 –(𝑦1 −20)
𝑃𝐼𝐶𝑇𝑈𝑅𝐸_𝐻𝐸𝐼𝐺𝐻𝑇

(3)
(4)

The distance 𝑧1 between the camera and the point (PICTURE_WIDTH / 2, 𝑦1) and the distance𝑧2 between the point (PICTURE_WIDTH / 2, 𝑦2 ) are calculated by Eqs. (5) and (6), respectively.
𝑧0𝑧1 = −𝐻𝑐 𝐻 −𝐻
𝑧1𝑐𝑧0𝑧2 = −𝐻𝑐 𝐻 −𝐻𝑧2𝑐(5)(6)

The maximum horizontal shooting ranges 𝑊𝑚𝑎𝑥1 and 𝑊𝑚𝑎𝑥2, when the distances to the camera
are 𝑧1 and 𝑧2 , are calculated by Eqs. (7) and (8), respectively.
𝑊𝑚𝑎𝑥1 =𝑊𝑚𝑎𝑥2 =𝑧1×𝑊𝑖
𝑓𝑧2×𝑊𝑖𝑓(7)(8)

From these values, the distances 𝑊1 and 𝑊2 between the z-axis and the points 𝑝1 , and 𝑝2 in
the real world are calculated by Eqs. (9) and (10).
𝑊𝑝1

𝑊1 = 𝑊𝑚𝑎𝑥1 𝑃𝐼𝐶𝑇𝑈𝑅𝐸_𝑊𝐼𝐷𝑇𝐻
𝑊𝑝2

𝑊2 = 𝑊𝑚𝑎𝑥2 𝑃𝐼𝐶𝑇𝑈𝑅𝐸_𝑊𝐼𝐷𝑇𝐻
2(9)
(10)

Proceedings of International Conference on
Technology and Social Science 2021 (ICTSS 2021)
Next, Fig. 4 shows each point viewed from the direction above camera. The distance W of the lane
markings in x-direction can be calculated by Eq. (11).
W=(𝑍2 𝑊1 −𝑍1 𝑊2 )𝑍2−𝑍1

Fig. 1. Shooting range of a camera.

Fig. 2. Relationship between camera andground.

Fig. 3. Image after detecting thelane markings.

Fig. 4. Each point viewed from the direction
above camera.

(11)

2.3 Detection of AR marker and estimation of position
We used a marker-type AR composed of black and white. To detect an AR marker, the surrounding
black frame recognizes is utilized, and the bit information is extracted by pattern of in the frame. In
this study, we used an AR marker with 4x4 bit information. In this experiment, the size of the AR
marker and bit information are fixed in advance. Different information is incorporated into the marker
for each installation position. Let the height of the camera be a fixed value hc. From the size and angleof the AR marker on image, the position of camera c (xc, yc, zc) in the real three-dimensional coordinates
can be estimated.
The angle 𝜃𝑐 formed by the distance 𝑙𝑐 between the AR marker and camera and unit vector n(1,0) in x-axis direction is calculated by Eqs. (12) and (13). If an AR marker does not appear in acamera, it cannot be detected, then the current position is estimated from the most recently estimatedvalue.
𝑙𝑐 = √(𝑥𝑐 2 + 𝑦𝑐 2 + 𝑧𝑐 2 ) − ℎ𝑐 2cos𝜃𝑐 =𝑥𝑐√𝑥𝑐2 +𝑦𝑐2

3(12)
(13)

Proceedings of International Conference on
Technology and Social Science 2021 (ICTSS 2021)

Fig. 5. Schematic diagram of position relationship between camera and AR marker.
3. Experiment
In this experiment, we practice two experiments. One is to calculate the distance by recognizing thelane marking, and another is to estimate the self-position using the AR marker.
3.1 Distance calculation by lane markings
Figure 6 shows a schematic diagram of the experiment using the lane markings. The speed ofcamera is 4 [cm / s], the moving distance is 200 [cm], and the width of lane marking is 2 [cm]. One isthe experiment in case that the camera is placed parallel to the lane marking. Another is the experimentin case that the camera is set to 45 degrees. When the camera is parallel to the lane marking, thedistances to the lane marking are 30 [cm], 50 [cm], 70 [cm], and when the angle of camera is 45 degrees,
the distances are 50 [cm], 60 [cm], and 70 [cm].

Fig. 6. Schematic diagram of distance measurement using lane markings.
3.2 Estimation of self-position by AR marker
Figure 7 shows a schematic diagram of the experiment using the AR marker. The speed of camera
is moved is 5 [cm / s], the moving distance is 100 [cm], the size of the AR marker is 3.4 x 3.4 [cm].
AR markers are placed in parallel to the moving direction of camera. One is the experiment in casethat the camera is placed parallel to the lane marking. Another is the experiment in case that thecamera is set to 45 degrees. When the camera is parallel to the lane marking, the distance to the lane
marking is 20 [cm], and when the angle of camera is 45 degrees, the distance is 30 [cm].
4 Experimental result
The experimental results are shown here. One is the measurement distance to lane marking, andanother is self-position estimated by using AR markers.
4Proceedings of International Conference on
Technology and Social Science 2021 (ICTSS 2021)

Fig. 7. Schematic diagram of self-position estimated by AR marker.
4.1 Measurement distance to lane marking
Figure 8 shows the results when the camera is parallel to the lane marking, and Fig 9 shows thosewhen the camera is placed at 45 degrees. When the cameras were placed in parallel, the error rate was
the largest at 16% when the distance was 30 [cm], and the distance could be calculated accurately.
When the camera was placed at 45 degrees, the calculation result between the side surface of thecamera and the lane marking had a large error, and the shortest distance had a small error. This isthought to be a human factor caused by manually moving the camera. In the future, we will install amobile robot and detect more accurately the distance from the number of revolutions of the motor.

Fig. 8. The shortest distance (left) and the distance (right) to the side when camera isplaced in parallel to lane marking.

Fig. 9. The shortest distance (left) and the distance (right) to the side at 45 degrees ofcamera.
5

Proceedings of International Conference on
Technology and Social Science 2021 (ICTSS 2021)
4.1 Self-position estimated by using AR markers
Figure 10 shows the results when the camera is placed in parallel to the AR markers, and thosewhen the camera is placed at 45 degrees. In both cases, it can be seen that the positions were
estimated because the difference the estimated value and theoretical value is small. However, selfposition is not sometimes estimated because AR marker itself is not detected. This should be
improved in the future.

Fig.10 Self-position estimation results when the cameras are parallel (left) and when thecameras are slanted (right)
5. Conclusion
In this study, we propose an automatic transfer robot system using AR markers in addition to lane
markings originally drawn in the factory in the factory. The distance to lane markings was measured,and self-position was estimated by using AR markers. As the results, it was possible to detect thedistance and estimate the self-position by processing the image taken in real time by the Web camera.
In addition, the self-position could be estimated from the value estimated immediately before, when
AR marker could not be detected. This can be expected to reduce the route miss during automatictransfer.
When the AR marker could not be detected, the self-position was estimated from value
immediately before, AR marker itself should be detected more correctly in the future.
References[1] From LNEWS: IDC Japan /AGV
https://lnews.jp/2019/05/l0514309.html

marketsize

forecast

in2023(website),[2] Juyang Weng, Paul Cohen, Marc Herniou, “Camera calibration with distortion models andaccuracy evaluation”. IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.14,
No.10, p.965-980.,1992.
[3] Hirokazu Kato, Mark Billinghurst, Koichi Asano, Keihachiro Tachibana, “Augmented reality
system based on marker tracking and its calibration”, Journal of the Virtual Reality Society ofJapan, Vol.4, No.4, pp. 607-614. 1999.
[4] Shotatsu Keiji, “[Details] Image processing programming”, Softbank Creative, 2008 (in Japanese).

6