Mitsubishi Heavy Industries Technical Review Vol. 58 No. 1 (March 2021)
1

Robust and Accurate Self-Localization Method under
Varying Lighting Conditions

KATSUMASA KITAJIMA*1 YUSUKE KINOUCHI*1
KIICHI SUGIMOTO*2

For guiding industrial AGVs (Automatic Guided Vehicle), robust self-localization method isindispensable. Visual SLAM (Visual Simultaneous Localization And Mapping, hereafter V-SLAM)
using a camera is one of the self-localization methods that can be used indoors and does notdepend on artificial landmarks. However, in a semi-outdoor environment affected by sunlight,
since the lighting conditions fluctuate rapidly, the self-localization methods process using V-SLAM
may become unstable. Therefore, Mitsubishi Heavy Industries, Ltd. (MHI) has developed a robust
and accurate self-localization method for operation under varying lightning conditions, bycombining 2D LiDAR, standard equipment of many industrial AGVs, with V-SLAM, and also
integrating internal sensors using extended Kalman filter.

|1. Introduction
MHI Group is developing mobile robot products such as disaster response robots,(1) plant
patrol inspection robots(2) and unmanned forklifts.(3) Commonly for these products, self-localization
method method, which estimates the position and orientation of the robot in real time, isindispensable.
In GNSS（Global Navigation Satellite System）-denied indoor environments, one of the mostcommon approach is using artificial landmarks (beacons, reflectors, image markers, etc.). However,there are challenges associated with operability because such approaches require systemre-construction after changing the equipment layout.
In recent years, SLAM has been attracting attention as a self-localization method that can beused indoors without artificial landmarks. SLAM is a method that simultaneously estimates
self-pose and constructs a map, and its mainstream approach is to use topographical features thatexist naturally and image features. In order to apply SLAM to navigation of industrial AGVs
operating under environments such as factories or warehouses, it is important to ensure robustness
to environmental changes. Industrial AGVs basically move indoors, but stable self-localizationneeds to be possible even in areas that are strongly affected by sunlight (semi-outdoorenvironments).
This report outlines a SLAM design that realizes accurate and robust self-localization for
operation under varying lighting conditions, by combining 2D LiDAR, standard equipment ofmany industrial AGVs, with V-SLAM, while also integrating internal sensors using extended
Kalman filter.

|2. Realization of robust self-localization process by integrating
multiple sensors
2.1

Functional configuration

This technology uses a wheel encoder and an inertial measurement unit (IMU) as the internalsensors, and a monocular camera and 2D LiDAR as the external sensors (Figure 1). The internalsensors and 2D LiDAR are not affected by lighting conditions. Therefore, even in situations where*1CIS Department, ICT Solution Headquarters, Mitsubishi Heavy Industries, Ltd.

*2

Chief Staff Manager, CIS Department, ICT Solution Headquarters, Mitsubishi Heavy Industries, Ltd.

Mitsubishi Heavy Industries Technical Review Vol. 58 No. 1 (March 2021)2

the lighting conditions fluctuate rapidly, the self-localization performance can be maintained by theinternal sensors and 2D LiDAR. The components of this technology, V-SLAM, LiDAR SLAM andextended Kalman filter (EKF), are described in detail in Section 2.2, Section 2.3 and Section 2.4,
respectively.

Figure 1

2.2

System configuration

V-SLAMA monocular camera cannot directly acquire distance information, unlike LiDAR. For thisreason, it is difficult for monocular V-SLAM to obtain a map that matches the scale of the real
space, which leads to low self-localization accuracy due to the scale drift of the map. Therefore, wehave improved the self-localization accuracy by correcting the map using 3D LiDAR. In contrast tomonocular cameras, 3D LiDAR can directly acquire distance information, has a wide field of view
and also it is robust to fluctuations in lighting conditions. For this reason, in many cases SLAM
using 3D LiDAR is capable of more accurate self-localization than monocular V-SLAM.
Therefore, by executing both V-SLAM and 3D LiDAR SLAM at the time of map construction andcomparing the self-localization results, the amount of error correction for the map constructed byV-SLAM was determined.
Figure 2 shows the relationship between coordinate systems at the time of self-localization.
{ } represents the self-pose at a certain moment (key frame) estimated by V-SLAM at the time ofmap construction. { } represents the self-localization result of 3D LiDAR SLAM corresponding
to the key frame. The difference between { } and { } is the error of the map constructed byV-SLAM, which is the amount of correction for V-SLAM. { } is the self-pose by V-SLAM basedon the relative displacement with the key frame { }. By performing correction using therelationship between { } and { }, a high-precision self-localization result { } can be obtainedwith V-SLAM.

Mitsubishi Heavy Industries Technical Review Vol. 58 No. 1 (March 2021)
3Figure 2

2.3

Relationship between coordinate systems

LiDAR SLAM

Many industrial AGVs are equipped with 2D LiDAR to prevent collisions. Self-localization
with 2D LiDAR SLAM was performed using this existing 2D LiDAR. In the case of LiDAR
SLAM, similarly to V-SLAM, a method of obtaining the self-pose by collating the data from thesensors with a map constructed in advance is generally used. However, existing 2D LiDAR is often
installed in a low position on the vehicle body and the field of view is easily occluded by dynamicobstacles. Since the camera faces toward the ceiling in our design of the V-SLAM, the field of view
can always be ensured. However, SLAM using the existing 2D LiDAR cannot always secure thefield of view and map matching may fail. Therefore, SLAM using the existing 2D LiDAR was
made able to use both a method of using a particle filter on the premise of collation with the known
map(4),(5) and a scan matching method that aligns terrain patterns by sequential optimization
processing without collation with the known map(6).

2.4Integration of sensors using extended Kalman filter

Through movement model calculation (odometry) of an AGV using internal sensors,
self-localization results with a short calculation cycle and small variance can be obtained. However,since integration operation is included, error accumulates over time. On the other hand,self-localization using external sensors (V-SLAM and LiDAR SLAM) undergoes only a small
effect of the cumulative error, while the calculation cycle is long and the variance is large.
Therefore, by integrating the information from the internal and external sensors using EKF (sensorintegration) to combine the advantages of both, self-localization with a short calculation cycle,
small variance and cumulative error can be realized.
As shown in Figure 1, we calculate linear combination of self-localization results of LiDARSLAM and V-SLAM using weighting parameters
andbefore input to BKF. Theseweights are set in advance for each location on the map and selected according to the current
position of the AGV. This design allows us to robustly localize the vehicle by setting larger valueson suitable sensors at each location. For example, in semi-outdoor environments the weight of 2D
LIDAR should be larger, while in environments with poor geometrical features the weight of thecamera should be larger and sensor integration according to the characteristics of the position canbe performed.

|3. Verification test
In order to verify the effectiveness of the design, we carried out a self-localization test using
an actual AGV. The true pose used for error evaluation was measured by tracking the target marker
attached on the AGV with a laser tracker. The test was conducted in an indoor environment and asemi-outdoor environment. The results of each test are described in Section 3.1 and Section 3.2.

Mitsubishi Heavy Industries Technical Review Vol. 58 No. 1 (March 2021)
4

3.1

Indoor self-localization test

The results of the indoor self-localization test are shown in Figure 3 to Figure 5. Figure 3
illustrates the movement path of the AGV and the sensing status of the camera and 2D LiDAR
during the test. The AGV departed from the starting position, passed through points A and B andtraveled back to the original location. While moving, the AGV switched its moving direction
multiple times to make the self-localization error of the internal sensors easier to accumulate. For
this test, a situation where the field of view of the 2D LiDAR would be frequently obstructed bydynamic obstacles and temporarily placed objects was assumed, so LiDAR SLAM (scanmatching), which is not premised on collation with the known map, was used.
The test results indicate that the self-localization result of our method achieved an error of 70[mm] or less when compared with the true pose measured by the laser tracker (Figure 4). In
addition, by comparing the accuracy between V-SLAM and LiDAR SLAM (Figure 5), it wasconfirmed that when the fluctuation of the lighting conditions is small, the accuracy increasing
method with V-SLAM described in Section 2.2 can obtain self-localization accuracy comparable toLiDAR SLAM.

Figure 3

Indoor self-localization test

Figure 4

Results of indoor self-localization test (1/2)

Mitsubishi Heavy Industries Technical Review Vol. 58 No. 1 (March 2021)
5Figure 5

3.2

Results of indoor self-localization test (2/2)

Semi-outdoor self-localization test

Figure 6 to Figure 8 give the results of the semi-outdoor self-localization test. In this test, agrid map for self-localization with 2D LiDAR (Figure 6) was created in advance and theself-localization was performed by collating it with the obtained map. Figure 7 presents a situationin which the self-localization process is executed in real time by collating the grid map with thelaser scan (the colored areas in the figure are the point cloud obtained by laser scanning with 2DLiDAR).
Figure 8 shows the results of evaluating the self-localization accuracy. In this test, the AGV
started from indoors and moved to the outdoors and the illuminance changed rapidly from about
1,000 [lx] to 80,000 [lx] due to the influence of sunlight. Even under such conditions, the result of
the self-localization using our method achieved an error of 70 [mm] or less.

Figure 6

Grid map by LiDAR SLAMMitsubishi Heavy Industries Technical Review Vol. 58 No. 1 (March 2021)
6

Figure 7

Semi-outdoor self-localization test

Figure 8Results of semi-outdoor self-localization test

|4. Conclusion
This report outlined SLAM design that realizes accurate and robust self-localization undervarying lighting conditions by combining 2D LiDAR, standard equipment of many industrial
AGVs, with V-SLAM, while also integrating internal sensors using an extended Kalman filter. We
verified the effectiveness of our design through an evaluation test using an actual AGV andconfirmed that our method can realize highly accurate and robust self-localization in indoor andsemi-outdoor environments.
We will continue to develop sensor integration technologies that can deal with various
environmental changes to further improve the reliability of self-localization method.

References(1)(2)

Yusuke Kinouchi et al., Robots utilizing ROS/Gazebo in Mitsubishi Heavy Industies, Journal of theRobotics Society of Japan, Vol.35 No.4 (2017), p.8-11
Ken Onishi, Explosion-proof Robotics Technology Contributing to Prevent and Response to theMitsubishi Heavy Industries Technical Review Vol. 58 No. 1 (March 2021)
7(3)

(4)
(5)
(6)

Flammable Gas Disaster in Petrochemical Facilities, Journal of the Robotics Society of Japan, Vol.38
No.3 (2020), p.235-238
Kiyotaka Okada, Logistics System Solution Expansion - From Sales of Items to Sales of Systems, From
Automated Operation to Unmanned Operation -, Mitsubishi Heavy Industries Technical Review, Vol.54
No.1 (2017), p.6-10
Grisetti, G. et al., Improved Techniques for Grid Mapping with Rao-Blackwellized Particle Filters, IEEE
Transactions on Robotics (2007), Vol.23, p.34-46
Dellaert, F. et al., Monte Carlo Localization for Mobile Robots, IEEE International Conference onRobotics and Automation (1999), p.1322-1328
Takeuchi, E. et al., A 3D Scan Matching using Improved 3D Normal Distributions Transform for MobileRobotic Mapping, IEEE/RSJ International Conference on Intelligent Robots and Systems (2006),
p.3068-3073