Mitsubishi Heavy Industries Technical Review Vol. 58 No. 2 (June 2021)
1

Feasibility Study of Pallet Handling in Mixed Fleet
Environment

ATSUSHI WADA*1

KENJI TAKAO *2

RYO TSUJISAKA*3

KATSUMASA KITAJIMA*4NORIYUKI HASEGAWA*5 RYOJI ARAKI*4

Automatic guided forklifts (hereafter AGF(s)) handle cargo according to a given path andpallet loading/unloading positions. Therefore, allowing an AGF to take a pallet placed by a forklift
requires the forklift operator to place the pallet according to the pallet guides or marks attached to
the floor in advance. We are developing a technology that enables an AGF to handle cargo without
using guides by allowing the AGF to identify by itself the position of a pallet to be handled andautomatically generate and track its path. As a measure to resolve the shortage of logistics
workers and reduce their burden and also with the aim of automating pallet handling that is
performed by AGFs and forklifts in a mixed manner, we will proceed with the development of thistechnology.

|1. Introduction
In recent years, with the background of a serious labor shortage in addition to the increasingneeds for logistics due to the growth of e-commerce, etc., expectations for making logistics
automated and unmanned have increased. On the other hand, at logistics sites where pallet handlingis performed, mixed work of forklifts and unmanned carrying systems such as AGFs is
indispensable and it is prerequisite to set conditions at load-passing points according to therequirements of the unmanned carrying system. This causes problems such as the moving of aforklift being hindered by the installation of the aforementioned pallet guides on the floor of thewarehouse or cargo handling area and the placing of the pallet at the position specified by a mark,etc., being required. In order to solve these problems, we have developed a function that allows an
AGF to automatically detect the deviation of the position/posture of a pallet placed randomly by aforklift and corrects the deviation to perform guidance control.

|2. Overview of new guidance control technology
The new guidance control technology consists of three items: [1] pallet detection technology(hereafter, PF (Pallet Finder))(1), [2] approach path generation technology (hereafter, PP (PathPlanning)) and [3] guidance technology. In order to develop and verify these technologies in a shortperiod of time, we implemented them with ROS (Robot OS) and constructed an interface between
the existing AGF system (laser guidance system) and the developed ROS node so that it can beeasily added to the existing system. Figure 1 is a configuration diagram of this system.

*1

Chief Staff Manager, Material Handling Solution Engineering Division, Engineering Headquarters, Mitsubishi Logisnext Co.,

*2

Chief Staff Manager, CIS Department, ICT Solution Headquarters, Mitsubishi Heavy Industries, Ltd.

*3

Material Handling Solution Engineering Division, Engineering Headquarters, Mitsubishi Logisnext Co., Ltd.

*4

CIS Department, ICT Solution Headquarters, Mitsubishi Heavy Industries, Ltd.

*5

Machinery Research Department, Research & Innovation Center, Mitsubishi Heavy Industries, Ltd.

Ltd.

Mitsubishi Heavy Industries Technical Review Vol. 58 No. 2 (June 2021)
2

Figure 1

System configuration of high-performance cargo handling system|3. Problem setting
Figure 2 sets the conditions for the problem to be handled in this report. For safety reasons,
AGFs have their forks on the backward side unlike forklifts, are operated so that they move toward
their bumper side and are often designed so that their forward moving speed is faster thanbackward moving speed. The left of Figure 2 illustrates the typical operation of an AGF. The AGF
moves forward in parallel with the target pallet to a point slightly beyond the front of the pallet(hereafter the switchback point), turns around and moves backward to approach the pallet. Thissystem uses the existing AGF moving control up to this switchback point and approaches the palletfrom the switchback point according to the position/posture of the pallet.

Figure 2 Setting conditions for problem to be handledMitsubishi Heavy Industries Technical Review Vol. 58 No. 2 (June 2021)
3|4. Developed technologies
4.1(1)Pallet detection technology (PF: Pallet Finder)

Overview
In order to realize the concept "Can be easily added to the existing system" described in
Chapter 2, this system uses two-dimensional LiDAR, which is attached to the existing AGF asa safety sensor in the front, back, left and right, as a sensor for pallet position/posture detection.
These sensors are mounted near the ground and can irradiate the height of the pallet pocket withlaser beam. In addition, while crossing in front of the target pallet before moving to theswitchback point as shown in the left of Figure 2, the AGF can detect the position/posture of
the pallet with this two-dimensional LiDAR (front and rear). The following paragraphs provide
details of the pallet detection technology.
(2) Detailed algorithm
Figure 3 depicts a detailed flow of the pallet detection technology. The processes [1] to[6] in the flow are described below.
[1] Extraction of pallet front face area
By extracting only the point cloud in the rectangular area where the pallet should exist
from all the measured LiDAR point clouds, the pallet front face is recognized.
[2] Detection of straight line
In [1], the front face of the pallet is only narrowed down to a certain area. In order tonarrow it down to a more linear shape, straight line detection is performed by the RANSAC(random sample consensus) algorithm. Specifically, the score of a straight line connecting
two randomly selected points is calculated and this calculation is repeated a certain numberof times to detect the straight line for which the score is largest.
[3] Detection of line segment
Several points adjacent to each other on the straight lines detected in [2] are grouped
and the point clouds are grouped into three segments so that they correspond to the beams of
the pallet, which are located at both edges and the center of the pallet when viewed from thefront. As a result, the point clouds can be recognized as a single pallet.
By doing this, it is possible to reduce erroneous recognition of the pallet to be detected
and the adjacent pallet.
[4] Detection of pallet edge point
By selecting two among the three line segments detected in [3] to extract a set of line
segments the length of which is the pallet width, the edge points of the pallet can be
identified.
[5] Fitting of straight line
Due to the characteristics of the LiDAR sensor, the point cloud near the pallet pocket
may not be reflected accurately, resulting in influence on the pallet detection process. The
following processing is performed to remove this influence.
First, among the point clouds included in all the line segments detected in [3], thoseexisting in the area between the pallet edges detected in [4] are extracted. From the extracted
point clouds, ones within the specified number from the end of each line segment detected in[3] are excluded. The front face of the pallet is extracted as a straight line shape byapproximation processing using the remaining point clouds and the inclination θ of the pallet
can be calculated. Then, the midpoint of the x-coordinates of the pallet edge points obtainedin [4] is set as the x-coordinate of the pallet and the y-coordinate can be obtained from theabove θ.
[6] Detection reliability
The certainty of the obtained pallet position/posture estimation result is evaluated as
the reliability and a high-reliability result is preferentially used for approach path generationand guidance control described later. Here the reliability calculation method is described.
In general, based on the angular resolution of the 2D-LiDAR sensor (available fromthe specifications shown in the brochure), it is possible to estimate in advance how many
point clouds can be measured (ideal number of measurement points) according to theMitsubishi Heavy Industries Technical Review Vol. 58 No. 2 (June 2021)
4

distance from the sensor. On the other hand, the number of LiDAR points that can be
actually measured may be less than the ideal number of measurement points due to the
influence of the relative angle between the sensor and the object. When the number of points
close to the ideal measurable point group is actually measured, the reliability of the pallet
position/posture estimated from the measurement result can be evaluated as high.

Figure 3

4.2

Pallet detection technology: Process flow

Approach path generation technology (PP: Path Planning) and guidancecontrol technology

The AGF is a forklift that cannot move in all directions and requires guidance control thattakes into account these movement restrictions(2)(3). Based on the pallet position/posture estimatedin section 4.1 PF, we have developed an AGF guidance technology that aims to allow the AGF toreach the posture in which the forks can be inserted into the pallet pocket. Specifically, wedeveloped a technology that combines the generation of an approach path from the switchback
point shown in the left of Figure 2 according to the pallet position/posture detected by the PF, thepath following control and high-precision positioning control using a servo(4).
(1) Approach path generation technology
As shown in the left of Figure 2 and as explained in section 4.1 PF, the position/postureof the target pallet can be estimated by the PF before the AGF reaches the switchback point.
Using the estimation result, the approach path to the pallet is generated.
Figure 4 presents the process flow and Figure 5 gives variables in this process flow.
This method characteristically generates a path only by combining an arc path and a straight
path, does not require optimization calculation, etc., and can generate a path quickly. Since theturning radius of the arc path and the AGF steering angle command have a one-to-one
correspondence, the steering angle command of the AGF moving on the arc path can beobtained at the same time when there is no disturbance. When the steering angle is constant, theturning radius does not depend on the forklift speed, so any forklift speed can be set within therange where slipping does not occur.
First, the final guidance start position (temporary) in Figure 5 is calculated. The
parameter
represents how much of the last straight line section is left. When the AGF
reaches the final guidance start position, its posture agrees with that of the pallet. Next, thecenter of the turning path (temporary) is calculated to determine whether additional switchback
is necessary. Using this information and the turning radius, it is determined whether additionalswitchback (detouring in upper part) as shown in Figure 4 is necessary.
When there is no need for additional switchback (detouring in upper part), the switch

Mitsubishi Heavy Industries Technical Review Vol. 58 No. 2 (June 2021)
5

back point (SB: switchback)is calculated. However, it is assumed that theswitchback point is set on the path defined in the system (original path) andagrees withthat path, so no calculation is required. Then, the center of the turning path for reaching thefinal guidance start position from the switchback pointis calculated and after that,the final guidance start position is calculated. Since both the final guidance start position andthe final guidance start position (temporary) exist on a straight line with the same inclination as
the pallet posture, the approach path from the switchback point to the pallet is completely
determined when the final guidance start position is obtained. The right of Figure 5 is an
example of the approach path generation result in the case of no need for detouring in upperpart.
On the other hand, if additional switchback is required, the center of a new turning path[2] is calculated. This new turning path is calculated so as to intersect the approach path and theinitially calculated turning path at only one certain point each and these intersections arereferred to as the switchback point and the additional switchback point, respectively. In thiscase, the AGF can be guided to the pallet by allowing it to move from the switchback point to
the final guidance start position via the additional switchback point. In this manner, thistechnology can characteristically create a complex approach path that requires switchback.

Figure 4

Approach path generation technology: Process flow

Figure 5 Approach path generation technology: Calculation example in the case of no needfor detouring in upper part

(2)

Path following control technology
The path following control is a feedback control that allows the forklift to move along a
pre-planned path. Its aim is to start the forklift near the start position of the path, move alongMitsubishi Heavy Industries Technical Review Vol. 58 No. 2 (June 2021)
6

the path and stop at the goal position. Therefore, the main goal is to keep the deviation from thepath sufficiently small. The target model and control law used in the analysis are described
below.
First, the AGF motion model that is the background of this control is going to be
described. The target of this control is an AGF with two fixed wheels and one steering wheel.
The steering wheel also acts as a driving wheel. Assuming that the turning center of the AGF is
the point where the axle center lines of the steering wheel and the fixed wheels intersect, themotion model represented by Equation 1 is obtained as ODE(2). This model describes thechanges in the position , and the posture θ with respect to the time using the steering
angle and moving speed as input information (Equation 1 in Figure 6).
Next, the above model is applied to the path following control and for the change in theerror with respect to the time , the distance error
and posture error
from the path arecontrol targets of this control (right of Figure 6). By applying this model to a coordinate systemcentered on a point on the path closest to the current position, Equation 2 in Figure 6 isobtained. In this equation, L is the wheelbase length andis the curvature of the path at thenearest neighbor point.
Finally, the control law used is going to be described. The purpose of this control is tomaintain== 0. In order to achieve that purpose, the input function:,↦needs to be designed so that the above equation has a stable equilibrium point where,= (0,0). This control adopts proportional control for feedback and uses the control law
represented by Equation 3 in Figure 6.

Figure 6 Path following system overview(3)

Servo control technology
Using an approach path alone may cause the risk that the AGF deviates from theapproach path due to the effects of disturbance or that the AGF deviates from the front of thepallet due to an error in the measured value of the pallet position/posture occurring during
generation of the approach path. Therefore, this system incorporates a newly-developed
technology that performs precision guidance using a servo with Lidar sensor (hereafter,
LiDAR-Servo) into the guidance control (Figure 7). As a result, the influence of a pathfollowing error can be eliminated and the influence of a measurement error of the pallet
position/posture can be mitigated.
This servo mechanism outputs steering command values based on the feedback of thedeviation of the detection result of the pallet position/posture estimated each time by the safety
2D-LiDAR sensor from the target position. Figure 8 presents the specific guidance processing
flow. First, the steering command value is output with priority given to the deviation of theposition and when the position converges to the target value to some extent, the servo control
for the deviation of the posture is performed. Since this servo mechanism does not pre-read, thecontrol is started from the state where the AGF almost squarely faces the pallet. However, as
shown in Figure 8, since which of the position and posture is prioritized for guidance isMitsubishi Heavy Industries Technical Review Vol. 58 No. 2 (June 2021)
7

switched according to the situation, effective operation is possible even if the posture is tilted tosome extent.
(4) Guidance mode selection function
This section describes the mechanism for switching from approach path/following toservo control. First, a map of relative position and posture that enables fork insertion by servo
control is created and registered in the system. If the pre-created map conditions include thecurrent relative position/posture estimated based on the PF result, switch to servo control isimplemented.

Figure 7

Servo control using PF

Figure 8

Guidance processing flow of fork insertion to pallet pocket by servo using PF|5. Verification with actual equipment
5.1

Forklift and environment for verification

We verified the developed high-performance cargo handling system by installing it on ourAGF (reach type). For this verification, the conventional automatic moving control system was
used.

5.2

Verification result

Figure 9 (a) is the result of real-time estimation of the pallet position/posture using the safety
2D-LiDAR sensor. It can be seen that the point cloud on the front face of the pallet was efficiently
and accurately captured and the position/posture of the pallet could be estimated accurately. Figure
9 (b) shows an approach path generated using the most reliable estimation results among the pallet

Mitsubishi Heavy Industries Technical Review Vol. 58 No. 2 (June 2021)
8

positions/postures estimated before the AGF reached the switchback point. This indicates that an
approach path with which the forks could be inserted into the pockets of the pallet and for which
the position/posture was estimated could be generated. Furthermore, when path following andservo control were performed to allow the AGF to approach the pallet, the forks could be inserted
into the pallet without any problem, so the effectiveness of the developed function was verified.

Figure 9 Verification result (pallet detection result, approach path, error at final arrival)|6. Conclusion
We have developed a system for AGFs consisting of [1] pallet detection technology , [2]approach path generation technology and [3] guidance technology and verified its effectiveness in atest environment. In the development of this system, functions could be developed andimplemented efficiently by using existing systems and sensors and ROS. In the future, this system
will be verification-tested at customer sites and brought to market. We will also develop thistechnology to promote further automation in various logistics environments where it was difficult
to do so in the past.

References
(1)
(2)
(3)
(4)

Junhao Xiao (et al.): Pallet recognition and localization using an RGB-D camera International Journal ofAdvanced Robotic Systems, 2017
Matthew R. Walter (et al.): Closed-loop Pallet Manipulation in Unstructured Environments, The 2010
IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020
Brian Paden (et al.): "A Survey of Motion Planning and Control Techniques for Self-Driving Urban
Vehicles", IEEE Transactions on Intelligent Vehicles, 1(1), pp.33 - 55)
Atsushi Ozato, et al., Position and Orientation Control of Omnidirectional Mobile Robot by Linear Visual
Servoing, Transactions of the Japan Society of Mechanical Engineering, (C) Vol. 77, No. 774, pp.215-224(2011)